Apache Kafka / Basic: records, partitions and reader progress / Lesson 5

Offsets, commits and replay

Record the next safe position and reason through crashes before resetting any reader.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish current position, completed processing and committed position.
Explain at-least-once replay and reset policy.

Three positions can differ

The consumer's current position describes where subsequent fetching proceeds under client behavior. Completed business work may trail fetched records, and the committed offset records a restart position for the group. These are different facts. Kafka commits the next offset to read, so successfully completing offset 41 means the next safe position is 42, provided all earlier relevant records are complete. A commit is not deletion from the topic and does not prevent another group reading the same history. Keep progress per partition; a single topic-wide integer cannot accurately describe independent partition histories or outstanding work.

Commit only a completed prefix

The local calculation has completed offsets 10 and 12 while offset 11 remains unfinished. Starting from 10, the safe restart position stops at 11. Committing 13 merely because it is one beyond the largest completion would skip 11 after a crash. This matters when workers process a batch concurrently: the largest completed offset is not the same as a contiguous completed prefix. The model assumes the listed offsets are the delivered sequence; real Kafka offsets can contain gaps, so production tracking must follow delivered records and client metadata rather than requiring every numerical offset to exist.

python Offline completed-prefix calculation

delivered = [10, 11, 12]
completed = {10, 12}
next_safe = 10
for offset in delivered:
    if offset not in completed:
        break
    next_safe = offset + 1
assert next_safe == 11
print(next_safe)

Expected output

The hole at 11 prevents committing 13; the fixture assumes contiguous delivered offsets.

Place crashes around the side effect

If work completes before a commit and the process crashes between them, replay can repeat the work. If a commit happens before work and the process crashes, the work can be skipped on restart. At-least-once processing chooses the first risk and requires a duplicate-safe side effect. Neither commitSync nor auto-commit makes a remote payment or database update atomic with the Kafka offset. External systems need their own transaction or deduplication design. Explain the precise crash window in a diagram or test schedule before promising delivery semantics to a caller, especially when the external action cannot be undone.

A reset policy does not override valid commits

auto.offset.reset is consulted when a usable committed position is absent or out of range under the client rules. Setting earliest does not force every restart to begin at the start while a valid group commit exists. Retention can remove the old start, so earliest means the earliest currently available position rather than the beginning of all historical time. A reviewed replay plan uses an independent group or an authorized offset change, checks destination duplicate handling and estimates retained coverage. This lesson runs no reset command and changes no group; the fragment only exposes the configuration's meaning.

properties Illustrative restart policy

enable.auto.commit=false
auto.offset.reset=earliest

earliest is a fallback when no usable committed position exists, not a forced replay setting.

Make replay observable and bounded

Before a replay, identify the business range, source topic, partitions and available offsets, then decide how results will be compared with the existing projection. Keep a distinction between raw record count and unique business event count because duplicates and filtering alter those totals. Record a stopping boundary so catching up does not become an endless ambiguous operation. Old event schemas and obsolete business rules can make replay different from normal live consumption. A dry transformation over synthetic fixtures can verify logic, but it cannot establish live retention coverage or the health of destination services.

Common errors and fixes

The last processed offset is committed instead of the next position.

Why it happens: The restart-position convention was misunderstood.
How to fix it: Commit the next safe position after the completed delivered prefix.
Prevent it next time: Test an exact crash-and-restart schedule.

Changing earliest appears to do nothing.

Why it happens: The group already has a valid committed position.
How to fix it: Inspect the intended replay plan rather than changing unrelated settings.
Prevent it next time: Separate first-start policy from explicit replay procedures.

An unfinished record disappears after restart.

Why it happens: A commit moved beyond unfinished concurrent work.
How to fix it: Track the completed delivered prefix per partition.
Prevent it next time: Make acknowledgment ordering part of worker design.

Check your understanding

1. Offsets 20 and 21 completed, but 22 did not. What next position is safe?

Show worked solution

The safe next position is 22 under that declared delivered sequence. Committing 23 would claim completion of the unfinished record. After restart, beginning at 22 repeats unfinished work while preserving completed progress. If external work for 22 may already have happened, duplicate handling must still protect its uncertain outcome.

2. Can a committed offset restore records that retention already deleted?