Diagnose late stream results and replayed batches

Understand the likely causes

• Watermarks follow event-time progress under operator rules, not a fixed wall-clock sleep. Output modes determine whether results are provisional or final. Checkpoints coordinate recovery, while custom sinks still need their own idempotent or transactional commit protocol. Deleting a checkpoint changes recovery history and can replay data.

Identify the clock and output promise

Compare event_time, received time and processing progress for a small approved synthetic sequence. Append-mode window output may wait for watermark-based finality, whereas update-style output can revise a total. A short trigger interval cannot force an idle event-time source to advance its watermark. Inspect the supported operator, watermark placement and output mode before increasing resources. A far-future timestamp can also distort progress, so validate source time conventions rather than blaming every delayed window on insufficient executors. No stream is started or monitored by this guide.

Use a local timeline without overstating engine behavior

The timeline subtracts a ten-minute delay from a maximum observed event time13:40, giving a candidate13:30. It is not an observed Spark watermark and does not establish immediate state cleanup. Engine batch timing, multiple-input watermark policy and operator eligibility remain relevant. Record state rows and batch duration alongside event-time progress in an authorized environment. Data older than a watermark can have operation-specific handling; a lateness threshold is not a complete automatic rejection ledger. Keep source identities if late records need repair or a separate correction path.

maximum observed event time:13:40
configured delay:10 minutes
local candidate:13:30
engine conclusion:inspect supported query progress and output mode

Reconcile a sink timeout before replay

A sink may commit before its acknowledgment is lost. The local ledger preserves an unknown outcome for operation q2:8 rather than calling it rejected. Query identity and batch identity together must follow the chosen checkpoint history. foreachBatch is at-least-once by default unless an appropriate sink protocol prevents repeated effects. A marker written separately before payload can lose data; one written afterward can permit duplicates. Verify the supported atomic or idempotent relationship and inspect existing durable outcome evidence before another submission.

{"operation_key":"q2:8","sink_outcome":"unknown","action":"reconcile durable outcome before replay","checkpoint_reset":"not a routine fix"}

Review compatibility before changing recovery state

Checkpoint changes, source retention and stateful query modifications have separate constraints. A fresh checkpoint can replay old source data, while an expired source can make the desired recovery impossible. Do not delete state to reduce memory without deciding which late-data and duplicate guarantees are being changed. Rehearse interruption before and after sink commit, then verify source offsets, state and one intended effect per operation key. A query writing two destinations needs partial-success reconciliation because their writes are not automatically one transaction. This guide performs no reset, restart, source read or sink write.

Keep it from recurring

• Version query and checkpoint ownership, retain replayable source history, document sink semantics and monitor state alongside event-time progress. Keep a recovery runbook that resolves unknown outcomes before allowing replay or checkpoint replacement.