Duplicate effects after a consumer restart

Understand the likely causes

• The effect may have completed before an offset commit, an application request may have been sent twice, or destination deduplication may be volatile or non-atomic. These causes require different evidence.

Compare occurrence identity and physical coordinates

Collect sanitized event identifiers and full topic-partition-offset coordinates from the producer, consumer and destination evidence. One coordinate processed twice suggests replay; two coordinates with one event identity suggest retransmission or duplicate publication. Two different identities for the same business occurrence suggest the identity creation policy itself needs review. Equal payload text alone does not prove a duplicate because legitimate events can look alike. Preserve the original records and timestamps rather than editing them to make totals match, and keep destination request identifiers separate from business event identifiers.

Reconstruct the effect-to-commit interval

Place the destination acknowledgment and Kafka commit on one incident timeline. If the destination accepted the action and the worker stopped before a successful source commit, the replacement can correctly replay the record under at-least-once processing. Committing earlier would change the risk to missing effects after a crash. Producer idempotence does not solve this consumer-side interval. Kafka transactions can atomically link supported Kafka outputs and offsets, but an arbitrary notification provider remains outside that boundary. Describe the actual systems participating in atomicity before selecting a remedy.

Inspect deduplication where the effect is stored

A local seen set disappears after restart and cannot protect a durable external action. A separate durable marker can also fail if the marker and effect commit independently: marker-first can suppress missing work, while effect-first can repeat it. Prefer a destination-supported idempotency key or atomic uniqueness-and-effect transaction where available. Check that the retention horizon covers supported replay and that the key includes the required tenant and operation scope. A repeated identifier with conflicting payloads is a contract violation requiring investigation, not a normal duplicate to overwrite silently.

Test one repeated identity without contacting the provider

A tiny local model can prove that a proposed identity comparison recognizes two deliveries of evt-9 as one occurrence. It cannot establish durable destination behavior, so keep that limitation with the result. In an independently authorized integration environment, the meaningful scenario applies an effect, interrupts before source progress is recorded and then replays the same original identity. Verification must observe one final business effect and a well-defined already-applied response. Do not use real recipients or payment accounts merely to demonstrate duplicate handling in a teaching exercise.

deliveries = ['evt-9', 'evt-9']
assert len(deliveries) == 2
assert len(set(deliveries)) == 1
print(len(deliveries), len(set(deliveries)))

2 1

Keep it from recurring

• Persist occurrence identity before retryable publication, document source-commit ordering and keep destination duplicate policy with the recovery contract. Review deduplication expiry whenever retention or replay scope changes.