Data Engineering / Editorial solution guide
Diagnose late data, stale output and backlog
Separate publication age, source completeness, event-time progress and consumer update semantics before widening lateness or replaying history.
Read-only knowledge base · No public comments
Recognize the symptoms
A job recently succeeded but a dashboard remains stale, historical totals change after late arrivals, or retry queues grow while the scheduler appears busy. A recent output timestamp and low average task duration do not prove complete source coverage or correct downstream replacement. Preserve the affected interval and accepted version.
Understand the likely causes
- Causes include missing source partitions, an old pinned input, delayed producers, stalled watermark/checkpoint progress, a hot or oversized partition, deterministic errors retried as transient, or cumulative window results consumed as deltas. Retention can expire while consumers lag, making an automatic replay impossible. Distinguish quiet legitimate input from unavailable input.
Record the exact consumer freshness promise
Identify dataset, source interval, event/arrival timestamps, accepted checkpoint and publication version. State which clock the consumer objective uses and whether results are provisional, correctable or final. Do not relabel current processing time as event time to make a metric look fresh. Compare per-partition and oldest outstanding progress with the relevant source/runner contract. A framework watermark is not simply a global maximum timestamp, and an idle source can still require explicit completeness evidence.
Test expected coverage separately from publication age
The offline model declares two expected partitions but only one accepted partition, despite a one-minute publication age. It illustrates two independent assertions and does not query a scheduler, storage catalog or real clock. Determine whether the missing partition is late, rejected, incorrectly named or outside the actual contract before repair. An aggregate over available data can be mathematically correct while still incomplete. Keep expected/received/rejected identities visible rather than accepting a plausible total and a recent timestamp.
expected = {'day-1', 'day-2'}
accepted = {'day-1'}
publication_age_minutes = 1
missing = sorted(expected - accepted)
print(f'missing_partitions={missing}')
print(f'publication_age_minutes={publication_age_minutes}')
assert missing == ['day-2']missing_partitions=['day-2']
publication_age_minutes=1
The recent publication is still missing day-2 under the invented contract. No real freshness measurement is performed.
Check whether corrected output replaces or increments
The local fixture emits cumulative versions ten and fifteen for one window. Adding both yields twenty-five, while replacement by the latest version yields fifteen. This is not a streaming-engine implementation; it makes the consumer contract visible. Use stable output identity and version rules, including conflict handling, before applying late corrections or replayed panes. A longer lateness horizon cannot fix a consumer that adds replacements as deltas. Record late/drop counts and an offline repair path for events beyond retained online state.
updates = [('w1', 1, 10), ('w1', 2, 15)]
latest = {}
for window, version, total in updates:
if window not in latest or version > latest[window][0]:
latest[window] = (version, total)
assert sum(value for _, _, value in updates) == 25
assert latest['w1'] == (2, 15)
print(f'latest_total={latest["w1"][1]}')latest_total=15
The accepted latest total is 15, not 25. Real concurrent update and durable publication guarantees require the selected sink protocol.
Repair the evidenced bottleneck within retained history
Inspect source availability, rejects, task error classes, dependency state, skew, file counts and sink latency at matching times. Bound retries and backfill admission so repair does not starve live freshness. Confirm history still exists before promising replay; missing source retention can require a supported re-snapshot or explicit limitation. Stage a candidate version, reconcile its population and coordinate consumer replacement. This guide changes no lateness, quota, scheduler state, cloud resource or production data; corrections belong to the authorized owner’s reviewed procedure.
Verify the fix
Test late, out-of-order, duplicate, missing-partition and stalled-consumer fixtures with known expected results. Verify source coverage, accepted progress, output version and downstream totals, including early/corrected updates. Compare tails and oldest work alongside average runtime, and state which failure scenario and retention horizon the successful repair actually covers.
Keep it from recurring
- Keep event-time and arrival-time contracts explicit, monitor lag against retained history and separate transient retries from deterministic failures. Preserve lineage for scoped backfills and version consumer replacement semantics with the produced dataset.