Apache Kafka / Basic: records, partitions and reader progress / Lesson 6

Retention, compaction and tombstones

Choose between bounded history and keyed-state recovery while respecting asynchronous cleanup.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish delete retention from log compaction.
Explain null-value tombstones and recovery limitations.

Retention is a data-availability policy

Kafka retains records independently of whether one group has consumed them. A delete policy removes old log segments according to configured time or size criteria, subject to implementation scheduling and segment boundaries. It is not a promise to erase one record at its exact expiry millisecond. Slow readers need enough retained coverage to recover, and a committed offset does not reserve data forever. Define both the business history requirement and the longest expected recovery delay. A topic designed for short-lived transport may be inadequate as the sole source for a quarterly rebuild, even when normal consumer lag stays low.

Compaction retains useful keyed state

Log compaction eventually removes older records for a key while retaining a recent value needed for state reconstruction. It does not turn the topic into a request-response database or guarantee there is exactly one visible record per key at every moment. Consumers can encounter several historical versions before cleaning catches up. Compacted records keep their original offsets, so gaps are normal. A key is required for the state identity. If every event receives a random key, records cannot supersede one another under the intended entity model, and compaction will not produce the desired parcel-status projection.

Distinguish a null value from text

A keyed record with a null value is a tombstone used by compaction to communicate deletion. The bytes of the string null or the JSON text {deleted:true} are ordinary non-null values unless the application gives them additional meaning. The offline projection applies P17=accepted, then P17=sorted, then a null deletion. Its final map is empty, while the input still contains three events. A real compacted log has cleaner scheduling and tombstone-retention behavior that this dictionary does not emulate. A rebuilding consumer must observe deletions within the documented coverage assumptions rather than starting from an arbitrarily stale snapshot.

python Offline tombstone projection

history = [('P17', 'accepted'), ('P17', 'sorted'), ('P17', None)]
state = {}
for key, value in history:
    if value is None:
        state.pop(key, None)
    else:
        state[key] = value
assert state == {}
assert len(history) == 3
print(len(state))

Expected output

None represents the declared null-value contract; no cleaner or broker runs.

Review combined cleanup explicitly

The properties illustrate a topic intended for keyed current-state recovery, with a declared tombstone visibility window. They are not universal production values or administrative commands. cleanup.policy can combine compact and delete, but then time or size deletion can remove data beyond compaction's key-state behavior. The chosen policy must match rebuild and privacy requirements together. Segment rolling, minimum compaction lag and cleaner capacity affect when eligible data is actually cleaned. Increasing one retention setting cannot prove complete end-to-end recoverability, especially if snapshots, connector offsets or downstream schema versions have a shorter useful lifetime.

properties Illustrative compacted-topic policy

cleanup.policy=compact
delete.retention.ms=86400000

The one-day tombstone window is a synthetic choice requiring recovery analysis, not a recommendation for every deployment.

Test a recovery narrative before choosing storage

For a parcel-status projection, specify whether a reader must reconstruct current state, every historical transition or both. A compacted state topic supports a different question from an immutable business audit archive. Test a fixture containing duplicate updates, a deletion and a later recreation of the same key. The recreation must be interpreted under the event contract rather than being blocked forever by a stale local deletion marker. Capture the oldest supported snapshot age and maximum replay duration. These constraints determine whether a rebuild can reliably see all necessary changes; cleanup configuration alone does not answer that business question.

Common errors and fixes

Compaction is expected to leave exactly one record per key immediately.

Why it happens: Asynchronous cleaning was mistaken for synchronous replacement.
How to fix it: Make consumers handle repeated versions under the event contract.
Prevent it next time: Test replay before and after conceptual cleaning.

The string null is sent as a deletion.

Why it happens: Serialized text was confused with a null record value.
How to fix it: Use the client serializer’s documented tombstone representation.
Prevent it next time: Test exact bytes and null handling at producer and consumer boundaries.

A stale snapshot misses a deletion.

Why it happens: The rebuild exceeded the tombstone visibility assumptions.
How to fix it: Reassess supported snapshot age and replay coverage.
Prevent it next time: Document snapshot, retention and replay windows together.

Check your understanding

1. Can a compacted parcel-state topic guarantee a complete audit of every status change?

Show worked solution

No. Older values for a key may be removed through compaction. If every transition must remain available, design an appropriate retained history or independent archive with its own recovery contract. Current-state reconstruction and complete historical auditing are distinct requirements even when both begin from the same producer events.

2. After the tombstone fixture, a new P17=accepted event arrives. What should the declared projection do?