Apache Kafka / Intermediate: contracts, failures and observability / Lesson 11

External effects and the outbox pattern

Analyze dual writes and place durable handoff evidence inside the authoritative transaction.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Identify the database-to-Kafka dual-write gap.
Explain outbox relay duplication and destination idempotency.

Two successful operations are not one transaction

Suppose an order service updates its database and then sends an order-confirmed event. A crash after the database commit but before sending leaves confirmed state with no emitted event. Reversing the order can publish a fact whose database transaction later fails. Kafka producer idempotence cannot close this cross-system gap because it does not enlist an arbitrary database transaction. Describe the authoritative state change and the event handoff together. The outbox pattern stores a durable event intent in the same database transaction as that state change, then lets a separate relay publish committed intents to Kafka.

Keep the intent beside the state change

The SQL sketch uses a synthetic order and stable event identifier. It is non-runnable and is not a universal schema or a tested database transaction. A real implementation needs appropriate uniqueness, serialization, isolation, permissions and relay indexing for the selected engine. The important invariant is that order confirmation and its event intent commit together or neither does. Do not insert the outbox row through a separate connection after the order transaction completes. Also avoid placing secrets or unnecessary personal data in an event intent, because its retention and access path can differ from the original business table.

sql Illustrative outbox transaction sketch

BEGIN;
UPDATE lab_orders SET status = 'confirmed' WHERE id = 17;
INSERT INTO lab_outbox (event_id, aggregate_id, event_type)
VALUES ('evt-order-17', 17, 'order-confirmed');
COMMIT;

The tables and transaction are illustrative; no database engine is contacted or validated.

The relay still needs duplicate-safe delivery

A relay can publish an outbox event successfully and crash before marking it as published. On restart it may publish again. Marking it first risks losing the event if publication fails. The normal design therefore permits retransmission and relies on stable event identity and duplicate-safe consumers or destinations. A change-data-capture relay and a polling relay have different operational requirements, source offsets and cleanup rules; neither should be assumed exactly once without checking the specific connector and destination protocol. Deleting outbox rows too early can destroy retry or reconciliation evidence needed after a prolonged outage.

Model a repeated intent at the destination

The offline fixture contains two deliveries of one intent and one delivery of another. A dictionary keyed by event identity retains two logical intents. This verifies only the chosen local identity rule; assigning into a dictionary is not a substitute for an atomic durable destination write. If the same identifier appears with conflicting payloads, silent overwrite would hide a serious contract violation. A production handler should detect that conflict and preserve evidence rather than casually choosing the last payload. Identity stability and payload consistency must be reviewed together when building reconciliation and replay tools.

python Offline consistent-identity check

deliveries = [('evt-17', 'confirmed'), ('evt-17', 'confirmed'), ('evt-18', 'confirmed')]
unique = {}
for event_id, payload in deliveries:
    if event_id in unique:
        assert unique[event_id] == payload
    unique[event_id] = payload
assert len(unique) == 2
print(len(unique))

Expected output

The duplicate must have the same payload; no durable transaction is modeled.

Reconcile using business coverage

A complete handoff design can answer which committed order confirmations have durable intents, which intents have been published and which unique events have reached the destination. These are separate checkpoints and should not be collapsed into one transport-success count. Include terminal failures and long-pending intents in the operational view. Outbox cleanup needs a documented evidence horizon and a way to prove that removing old intents does not prevent supported recovery. This lesson executes no SQL, opens no database and creates no connector. Its examples support design reasoning and a small local duplicate-identity check only.

Common errors and fixes

A committed order has no event after a crash.

Why it happens: Database update and event publication were separate uncoordinated writes.
How to fix it: Record a durable intent atomically with the authoritative change.
Prevent it next time: Review the crash between each cross-system operation.

An outbox relay publishes one event twice.

Why it happens: It restarted after publication but before progress marking.
How to fix it: Retain stable identity and enforce duplicate-safe destination handling.
Prevent it next time: Test relay restart and acknowledgment loss explicitly.

The same identifier carries different payloads.

Why it happens: Identity generation or mutation violated the event contract.
How to fix it: Quarantine the conflict and investigate the authoritative occurrence.
Prevent it next time: Check identity-payload consistency during reconciliation.

Check your understanding

1. Does adding an outbox automatically make a remote email happen exactly once?

Show worked solution

No. It closes the authoritative database-to-intent gap when implemented transactionally, but relay delivery and the email provider still have their own failure windows. The destination must support an idempotency key or another reviewed duplicate-safe protocol, and uncertain results need reconciliation using the original occurrence identity.

2. Why is marking published before sending an unsafe way to avoid duplicates?