Data Engineering / Advanced: distributed correctness and reliability / Lesson 14

Delivery semantics and transactional sinks

Define exactly which effect is at most once, at least once or transactionally coordinated. Test duplicate delivery and lost responses without promising arbitrary end-to-end exactly-once behavior.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Separate producer, transport, consumer and sink guarantees.
Coordinate a duplicate ledger with the effect in one supported boundary.

Exactly once must name the counted effect

At-most-once delivery may lose work, while at-least-once delivery permits repetition. Exactly-once claims need a stated boundary and failure model: message append, state update, output publication and an external payment are different effects. A producer’s idempotent retry does not automatically deduplicate a separate API retransmission with a new identity. Consumer offset handling does not alone coordinate an arbitrary sink. Ask which histories, identities and transactions establish the claim, and what happens after a lost response. A product feature label is not a substitute for this reasoning.

Contrast delivery attempts with unique business events

The offline fixture delivers e1 twice and e2 once. A plain list records three deliveries, while a set records two unique identities. Neither representation alone is a production sink protocol: the set drops ordering and content, and it cannot detect conflicting e1 payloads. The example makes the counted population explicit. Decide whether metrics count attempts, accepted events or effects so a retry spike is not mislabeled as new business volume. Preserve both operational attempt evidence and logical identity without confusing one with the other.

python Offline attempt-versus-event counts

deliveries = ['e1', 'e1', 'e2']
assert len(deliveries) == 3
assert len(set(deliveries)) == 2
assert deliveries.count('e1') == 2

Three attempts represent two declared identities. Content consistency and durable application are intentionally outside this model.

Read messaging guarantees with their prerequisites

Kafka documents idempotent producers and transactions within its supported protocol, including coordination of eligible records and consumer offsets. Transactional visibility depends on appropriate consumer behavior such as read_committed. This does not automatically atomically commit a remote warehouse, HTTP notification or application database that is outside that boundary. Connector and sink support must be verified for the selected versions and configuration. This lesson installs no broker or client and supplies no deployable Kafka settings; it uses the official 4.1 design documentation for terminology and explicitly bounded local models.

Commit event identity and effect together locally

The SQLite memory program records an event identity and increments its aggregate only when that identity was newly inserted. Both changes share one local transaction, and a repeated event therefore leaves the amount at twenty-five. The declared fixture uses identical repeated content; a real system must reject or resolve conflicting content for the same identity. This example demonstrates a local database pattern, not distributed commit or irreversible external delivery. The explicit isolation mode makes BEGIN ownership clear, and the connection is closed without creating a persistent file.

python Local identity/effect transaction in SQLite memory

import sqlite3

db = sqlite3.connect(':memory:', isolation_level=None)
db.executescript('CREATE TABLE seen(id TEXT PRIMARY KEY); CREATE TABLE total(amount INTEGER); INSERT INTO total VALUES(0);')
for event_id, amount in [('e1', 25), ('e1', 25)]:
    db.execute('BEGIN')
    inserted = db.execute('INSERT INTO seen VALUES(?) ON CONFLICT(id) DO NOTHING', (event_id,)).rowcount
    if inserted == 1:
        db.execute('UPDATE total SET amount=amount+?', (amount,))
    db.execute('COMMIT')
assert db.execute('SELECT amount FROM total').fetchone()[0] == 25
assert db.execute('SELECT COUNT(*) FROM seen').fetchone()[0] == 1
db.close()

Two deliveries produce one recorded event and total 25. The guarantee ends at this in-memory transaction boundary.

Design each uncertain boundary before enabling retries

Enumerate failure before effect, after effect before acknowledgment, after acknowledgment before checkpoint and during downstream publication. Decide which steps can be replayed and which require reconciliation or idempotent external APIs. Bound retry attempts and deadlines, and retain identity evidence long enough for the recovery window. A deduplication ledger that expires too early can allow an old event to apply again. Tests should inspect final business state and recorded identity, not only successful task completion. Be precise about what a verified local example supports and which distributed assumptions remain untested.

Common errors and fixes

Idempotent producer mode is called exactly-once external billing

Why it happens: A transport guarantee was extended beyond its transaction boundary.
How to fix it: Specify and implement the external effect/idempotency protocol.
Prevent it next time: Review every system boundary and lost-response case.

The deduplication record commits without its effect

Why it happens: Related state updates used separate uncoordinated commits.
How to fix it: Use one supported transaction or an explicit repairable protocol.
Prevent it next time: Test interruption between identity and effect.

An old replay becomes a new charge

Why it happens: Identity retention was shorter than replay possibility.
How to fix it: Reconcile the event and review identity lifetime.
Prevent it next time: Align replay and deduplication horizons.

Check your understanding

1. A consumer commits its offset after sending an email. Can a crash still duplicate the email?

Show worked solution

Yes, if the crash occurs after delivery but before durable progress. Offset ordering alone cannot atomically coordinate an external service. Use a stable event/output identity with an appropriate idempotent delivery or outbox/reconciliation design, and test that failure window rather than claiming a generic exactly-once guarantee.

2. Why must equal event identity with different amount not be ignored casually?