Apache Kafka / Advanced: atomic processing and recovery design / Lesson 14

Streams state, windows and late data

Connect Kafka Streams topology decisions to keys, time, local state and recovery.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish KStream events from KTable updates.
Explain event time, window membership and late-arrival policy.

A stream and a table answer different questions

Kafka Streams is a client library for building processing topologies over Kafka. A KStream models a sequence of records, while a KTable models updates to keyed state under its contract. Counting every parcel status event differs from counting the current number of parcels in each status: the latter must account for replacement and retraction as state changes. Choosing the abstraction affects aggregation semantics and emitted updates. State stores can be backed by changelog topics for recovery, but a local directory alone is not the whole recovery design. Application identity, topology names and internal topic compatibility require deliberate lifecycle management.

Choose the clock before the window

Event time describes the business occurrence time extracted from a record; processing time describes when the application handles it. Network delay and replay make them diverge. Kafka Streams uses stream-time concepts tied to observed record timestamps rather than assuming wall-clock passage finalizes every window. A grace period defines how much lateness a windowed operation accepts under its semantics. Larger grace can improve coverage but requires retained state and delays finality. Define whether dashboards show provisional updates or final results, and make rejected-late records observable rather than silently presenting an incomplete count as complete.

Compute a declared tumbling-window boundary

The offline arithmetic uses ten-second nonoverlapping windows aligned to zero. Timestamps nine and zero belong to the first window, while ten belongs to the next. The half-open interval convention avoids counting an exact boundary twice. This is window-membership arithmetic only; it does not implement Kafka Streams stream time, grace, suppression or state retention. Tests should include exact boundaries, late arrival and invalid timestamps. Business reporting might require calendar windows in a named time zone instead, which introduces different alignment and daylight-saving decisions. Do not substitute integer epoch buckets for a calendar requirement without discussion.

python Offline half-open window membership

def window_start(timestamp_s, width_s=10):
    return (timestamp_s // width_s) * width_s
assert window_start(0) == 0
assert window_start(9) == 0
assert window_start(10) == 10
print(window_start(9), window_start(10))

Expected output

0 10

The ten-second buckets are a declared arithmetic convention, not a Streams runtime test.

State the aggregation key and output interpretation

The second model counts three synthetic arrival events by depot and window. D1 has two arrivals in its first window, while D2 has one in the next. It treats every input as an independent event; a duplicated event identifier would increase the count unless deduplication were separately designed. A KTable input could require update-aware behavior rather than this append-count model. The output key must identify both depot and window if downstream readers store one result per reporting bucket. Otherwise, results from different windows can overwrite each other under a compacted-state interpretation.

python Offline depot-window event counts

from collections import Counter
events = [('D1', 2), ('D1', 9), ('D2', 10)]
counts = Counter((depot, timestamp // 10 * 10) for depot, timestamp in events)
assert counts[('D1', 0)] == 2
assert counts[('D2', 10)] == 1
print(counts[('D1', 0)], counts[('D2', 10)])

Expected output

2 1

Every supplied event counts once; no grace or duplicate policy is implemented.

Review topology changes as state migrations

Changing a grouping key, window definition or state-store serialization can invalidate the meaning of existing local and changelog state. A new application identity may start an independent processing history rather than safely upgrading the old one. Review internal topics, repartition steps, restore duration and output compatibility before rollout. Kafka Streams offers supported testing utilities for topology logic, but this course does not install or run them; the Python examples are explicitly smaller conceptual checks. Production readiness also requires an authorized test of failure recovery and retained source coverage, because a successful pure aggregation does not prove a state store can be restored.

Common errors and fixes

A current-state update is counted as another independent entity.

Why it happens: Table-update semantics were treated as append-only event semantics.
How to fix it: Choose KTable or KStream behavior from the business question.
Prevent it next time: Test replacement and deletion events in aggregates.

A window never finalizes when only wall-clock time advances.

Why it happens: Stream-time behavior was confused with processing time.
How to fix it: Review timestamp extraction and the operation’s documented time semantics.
Prevent it next time: Include idle inputs and late records in time tests.

An upgrade restores state with incompatible meaning.

Why it happens: Grouping or serialization changed without state migration review.
How to fix it: Plan topology and state compatibility explicitly.
Prevent it next time: Treat state-store changes as migrations.

Check your understanding

1. Does an event timestamp of10 belong to [0,10) or [10,20) in the declared model?

Show worked solution

It belongs to [10,20), because the upper boundary is excluded and the lower boundary included. This makes every valid timestamp belong to exactly one tumbling bucket. The calculation alone says nothing about whether a late record is still accepted after stream time advances beyond the window and its grace period.

2. Why should output identity include the window as well as the depot?