Apache Kafka / Intermediate: contracts, failures and observability / Lesson 10

Poison records and retry topics

Classify record failures, preserve diagnostic evidence and review how retry routing changes ordering.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish transient dependency failure from deterministic invalid input.
Design bounded retry and quarantine contracts.

Diagnose before repeating

A poison record fails repeatedly because its bytes, schema or business content cannot be processed under the current reader contract. Retrying the same invalid bytes immediately does not repair them. A temporary unavailable dependency is a different category and may succeed after bounded backoff. Classify errors with enough context to make that distinction without leaking sensitive payloads. A deserializer exception can occur before the normal business handler receives the record, so the error path must be designed at the appropriate client boundary. Do not assume every failure reaches the same application catch block or can be safely skipped.

Carry original identity into a retry envelope

A retry topic is an application pattern, not an automatic universal feature of a basic consumer loop. Its envelope should retain the original event identity and source coordinate, record a bounded attempt count and describe a failure category. Avoid copying credentials or unrestricted stack traces into a broadly readable topic. The JSON fixture preserves the original location and one attempt counter. The counter's semantics must be clear: does one mean the first failure or the first retry? A consistent convention allows operators to tell exhausted work from a newly failed event without guessing from timestamps.

json Synthetic retry-envelope contract

{"event_id":"evt-7","source":{"topic":"parcels","partition":0,"offset":17},"failed_attempts":1,"category":"temporary-dependency"}

failed_attempts counts observed failures; this data does not implement delayed delivery.

Retry routing changes the sequence

Moving a failed parcel update to a retry topic and continuing the original partition lets later parcel events overtake it. This can be acceptable for independent notifications but incorrect for state transitions requiring strict order. Blocking the partition preserves a stronger local sequence at a cost to unrelated keys sharing that partition. Alternatives include key-aware blocking, destination version rules or a redesigned event contract, each with its own complexity. Delayed retry also needs a scheduler or consumer logic; merely placing a future timestamp in a Kafka record does not make the broker delay delivery until that time.

Bound the schedule explicitly

The offline schedule calculates three synthetic delays with a fixed cap. It shows a finite policy rather than sleeping, connecting to a broker or running a load test. Real retry systems may add jitter to avoid synchronized bursts, but deterministic tests should control the random source. A maximum delay is not a maximum total retry duration; include processing time and all previous waits when checking the business deadline. Decide what happens when the budget expires: quarantine, a compensating workflow or an explicit failed status. Infinite retry is a policy too, but usually prevents a bounded recovery claim.

python Offline finite backoff budget

delays_s = [min(2 ** attempt, 8) for attempt in range(3)]
assert delays_s == [1, 2, 4]
assert sum(delays_s) == 7
print(delays_s, sum(delays_s))

Expected output

[1, 2, 4] 7

Seven seconds is waiting only; processing and transport time are outside this arithmetic.

Do not lose the record while moving it

If the consumer commits the source position before the retry record is safely accepted, a crash can lose the handoff. If it publishes the retry and crashes before committing, the handoff can repeat. Kafka transactions can link Kafka output and consumed offsets within their supported boundary; otherwise a reviewed duplicate-safe transfer protocol is necessary. Quarantine is not the end of ownership: define who reviews records, how corrections are authorized and how replay avoids a second effect. Verification should count original unique identities through success, retry and terminal states, including a crash between publication and source progress recording.

Common errors and fixes

One malformed record causes an endless hot retry loop.

Why it happens: Deterministic invalid input was classified as transient.
How to fix it: Apply a reviewed terminal path with preserved source identity.
Prevent it next time: Test malformed bytes and unsupported schemas separately.

Later parcel updates overtake a retried update.

Why it happens: Retry routing changed the ordering boundary.
How to fix it: Choose a key-order policy appropriate to the business transitions.
Prevent it next time: Include overtaking scenarios in retry design reviews.

A record vanishes between source and quarantine.

Why it happens: Source progress was committed before a safe handoff.
How to fix it: Use a supported atomic Kafka handoff or a duplicate-safe transfer protocol.
Prevent it next time: Inject conceptual crashes around publish and commit boundaries.

Check your understanding

1. Does assigning a timestamp one hour ahead make a standard Kafka consumer wait an hour?

Show worked solution

No. A record timestamp is metadata, not a general scheduled-delivery instruction. Delayed retries require explicit scheduling or reader logic and a reviewed ordering policy. The design must also explain how the source offset, retry publication and final effect remain safe across crashes and repeated delivery.

2. Why can a dead-letter topic still contain duplicate original events?