Apache Kafka / Intermediate: contracts, failures and observability / Lesson 8

Idempotence, retries and event identity

Separate producer retry protection from application duplicate handling and uncertain business outcomes.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Describe the scope of producer idempotence.
Design event-level duplicate handling for replay and retried requests.

Not every duplicate has the same origin

A producer can retry a protocol request after an uncertain network result. Separately, an application can call send twice for the same business occurrence, perhaps after an HTTP retry or a process restart. Producer idempotence addresses supported producer retry sequences through producer identity and sequence numbers; it does not deduplicate arbitrary business events merely because their payloads match. Consumer replay introduces another repetition path even if the topic contains only one record. Name these cases separately so a team does not claim end-to-end exactly-once behavior from one producer flag while the destination still repeats an external action.

Keep compatible producer constraints together

The fragment explicitly enables idempotence and requests all in-sync acknowledgments. The Kafka Java producer documents compatible limits for retries and max.in.flight.requests.per.connection; conflicting explicit settings can cause configuration failure rather than silently supplying the intended guarantee. Do not disable ordering-related constraints simply to chase throughput. Defaults and supported combinations can differ in non-Java clients, so cite the exact implementation when documenting behavior. These lines are not a complete client program and are not executed. They illustrate why a reviewed configuration set matters more than a single isolated property copied into an unknown application.

properties Illustrative compatible Java producer choices

enable.idempotence=true
acks=all
max.in.flight.requests.per.connection=5

The documented Java constraints must be reviewed with retries and version; no producer runs.

Put identity where the side effect can enforce it

An event identifier should remain stable across retransmission of one occurrence. A destination that stores both the identifier and the effect atomically can reject a repeated occurrence. The atomicity is essential: checking a separate seen set before performing an unrelated remote action leaves a crash window and a race. A bounded deduplication retention period must cover the supported replay horizon, and keys may need tenant or operation scope. Equal payloads are not sufficient identity because two legitimate scans can have identical data. Conversely, a producer-generated random identifier on every retry prevents recognition of a true retransmission.

Make the local invariant visible

The model counts event identifiers once and ignores the repeated evt-1. It demonstrates the intended business count under one single-threaded in-memory schedule. It does not prove atomicity with a database or payment provider, and the seen set disappears when the process ends. In a real destination, the durable uniqueness rule and state change must belong to one supported transaction or an equivalent destination protocol. Record the outcome of a duplicate explicitly: it may be a successful already-applied response rather than an error that causes endless retries. Preserve enough evidence to distinguish a duplicate from an identifier collision.

python Offline business-identity model

event_ids = ['evt-1', 'evt-1', 'evt-2']
seen = set()
applied = 0
for event_id in event_ids:
    if event_id not in seen:
        seen.add(event_id)
        applied += 1
assert applied == 2
print(applied)

Expected output

This model is not durable or concurrent and is not an external-effect transaction.

Bound retries and preserve uncertain outcomes

A retry policy needs a maximum duration, backoff and a terminal decision. Rapid synchronized retries can amplify a transient problem, while unlimited retries can block one partition indefinitely. Classify configuration and authorization failures separately from plausible temporary transport failures. When a deadline expires after a possible write, record uncertainty and reconcile using the original event identity. Do not generate a new event to hide the uncertainty. Tests should place failure before send, after broker acceptance but before response, and after destination effect but before commit. Offline identity checks here establish only deterministic handling of supplied identifiers.

Common errors and fixes

Enabling idempotence still leaves duplicate business actions.

Why it happens: The application or consumer replay repeated a logical occurrence.
How to fix it: Enforce stable event identity at the effect boundary.
Prevent it next time: Document protocol retry and business retry separately.

The deduplication marker exists but the effect never happened.

Why it happens: The marker and effect were stored through separate operations.
How to fix it: Use destination-supported atomicity or a reviewed reconciliation protocol.
Prevent it next time: Test crashes on both sides of the marker write.

A timed-out request is retried with a fresh identifier.

Why it happens: Uncertainty was mistaken for definite nonexecution.
How to fix it: Reconcile or retry the same occurrence with its original identity.
Prevent it next time: Persist identity before exposing a retryable operation.

Check your understanding

1. Two distinct depot scans have identical payloads. Should payload equality deduplicate them?

Show worked solution

No. They can be legitimate separate occurrences. Use an event identity whose creation rule matches the business event and keep it stable only for retransmissions of that occurrence. A content hash can be part of collision detection or integrity checking, but it does not automatically define the correct business uniqueness rule.

2. Does the offline seen set prove duplicate protection after a restart?