Data Engineering / Intermediate: change, time and coordination / Lesson 9

Event time, windows and late data

Separate event occurrence from processing progress and choose explicit correction/finalization rules. Treat watermarks as a source-informed completeness mechanism, not a universal wall clock.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Distinguish event, ingestion and processing time.
Define windows, late acceptance and result updates.

Three clocks answer different questions

Event time describes when the producer says a business event occurred, ingestion time describes arrival at a boundary, and processing time describes computation progress. Delays, clock errors and replays can separate them substantially. A daily event report and a daily arrival report therefore select different populations. Specify timezone and interval boundaries before aggregating. Sorting records by arrival does not reconstruct authoritative event order, and a quiet source does not necessarily prove all earlier events have arrived. Framework timestamp assignment and source guarantees must be understood before calling a result complete.

Expose out-of-order arrival with a small fixture

The local fixture arrives in the order e2 then e1 but sorts by event time into e1 then e2. This does not prove every event timestamp is trustworthy or that sorting a bounded batch implements streaming completeness. It shows why processing order and event order must be named separately. In a real flow, preserve the original event and arrival timestamps with enough provenance to explain corrections. Avoid silently replacing a suspicious producer timestamp with now, because that changes the dataset’s meaning and may move the event into another business interval.

python Offline event/arrival ordering distinction

events = [
    {'id': 'e2', 'event_minute': 12, 'arrival': 1},
    {'id': 'e1', 'event_minute': 10, 'arrival': 2},
]
assert [e['id'] for e in events] == ['e2', 'e1']
ordered = sorted(events, key=lambda e: (e['event_minute'], e['id']))
assert [e['id'] for e in ordered] == ['e1', 'e2']

The two orders differ under invented timestamps. No streaming engine or real clock is used.

A window boundary is not proof of complete input

A fixed window assigns events to declared intervals; triggers decide when results are emitted, and allowed lateness/correction policies govern later updates. In systems such as Apache Beam, watermarks represent progress toward input completeness under the source and runner model. They are not simply the maximum timestamp observed and are not a universal guarantee that no older event can ever appear. Early results may be provisional. Define whether later output replaces a prior total or adds a delta, because consuming repeated cumulative panes as independent increments can double-count.

State a toy late-result policy explicitly

The offline classifier labels a result provisional before window end, correctable until a declared cutoff and outside the retained correction horizon afterward. This is a simple application policy, not the precise implementation of Beam triggers or every runner’s boundary rules. The numeric units and limits are invented. A production design needs source watermark behavior, state retention, output identity and a repair path for data beyond the online horizon. Dropping late records without count and reason can make a timely dashboard look complete while hiding missing business facts.

python Offline declared correction-horizon model

def result_state(progress, window_end, correction_horizon):
    if progress < window_end:
        return 'provisional'
    if progress <= window_end + correction_horizon:
        return 'correctable'
    return 'offline-repair-required'

assert result_state(54, 60, 15) == 'provisional'
assert result_state(70, 60, 15) == 'correctable'
assert result_state(90, 60, 15) == 'offline-repair-required'

The three assertions follow only this function’s inclusive cutoff. They establish no real watermark or platform guarantee.

Balance completeness, latency and retained state

Longer correction horizons retain more state and can cost more, while shorter horizons require stronger rejection/repair handling. Deduplicate event identity independently from late timing; an on-time duplicate is still a duplicate. Monitor input lag, watermark progress, late counts, correction outputs and consumer update behavior together. Replays and backfills may require a separate bounded recomputation rather than changing live lateness blindly. Explain which results are provisional, corrected or finalized under the dataset contract. A low processing delay alone does not prove event-time completeness or correct downstream accumulation.

Common errors and fixes

Fast processing is reported as complete event data

Why it happens: Arrival/processing progress was confused with source event completeness.
How to fix it: Define the event-time and watermark/correction contract.
Prevent it next time: Test delayed and out-of-order records.

Repeated window totals are added twice

Why it happens: Cumulative updates were consumed as independent deltas.
How to fix it: Use output identity and the declared replacement/delta semantics.
Prevent it next time: Test early and corrected emissions together.

Late drops are invisible

Why it happens: Completeness loss had no recorded count or repair path.
How to fix it: Retain scoped late reasons and define bounded recomputation.
Prevent it next time: Monitor lateness against retained state/history.

Check your understanding

1. An event arrives today with yesterday’s event time. Which daily partition should own it?

Show worked solution

The dataset contract decides whether partitioning follows event day or arrival day. For an event-day report, the late event belongs to yesterday and may require a correction or backfill. Preserve both timestamps and do not silently move it to today merely because processing is easier.

2. Why is max observed event time alone not a universal watermark?