A rerun creates duplicate output or repeated external effects

Understand the likely causes

• The task may append on every attempt, generate new operation identity, read changing latest input or repeat an external publication whose acknowledgment was lost. Orchestration state alone cannot distinguish these outcomes.

Compare execution attempts with business operations

Collect the intended interval, source manifest version, calculation version and destination operation key for each attempt. Run IDs and try numbers identify execution history, while the business key identifies the result that should exist once under the destination contract. Two legitimate report versions are different from two unintended copies of one version. Inspect source changes as well: a retry that reads latest may compute a different result rather than a byte-identical duplicate. Preserve original artifacts and counts before editing anything, and document which version is authoritative for downstream readers.

Place the failure around publication

Reconstruct the sequence from transformation completion through destination acceptance to task-state reporting. If publication succeeded and the response was lost, Airflow can correctly retry a task whose external effect already exists. A new attempt identifier does not undo that effect. Conversely, an early success marker before publication can leave missing output. The metadata database and an arbitrary external store are not one transaction. Identify the destination's supported atomic replacement, idempotency key or reconciliation protocol instead of treating a retry-count change as a correctness repair.

Choose repeatable replacement or explicit versions

For a complete interval report, a stable partition replacement or immutable version plus authoritative pointer can be appropriate when supported by the destination. A simple append of the same rows creates duplicates and may require a reviewed correction, not another blind rerun. The offline model applies the same two-row result twice to one dictionary key and retains one logical partition. It does not prove durable atomicity, concurrent-writer safety or remote storage behavior. Review conflicting payloads under one key and define whether a late correction replaces, amends or creates a new accepted version.

rows = [('P1', 200), ('P2', 300)]
reports = {}
for _ in range(2):
    reports[('2026-10-01', 'v1')] = tuple(rows)
assert len(reports) == 1
assert len(reports[('2026-10-01', 'v1')]) == 2
print(len(reports), len(reports[('2026-10-01', 'v1')]))

1 2

Reconcile before authorizing another historical pass

List affected partitions and unique business identities, then compare expected totals with authoritative artifacts. Keep raw row count separate from unique row identity because duplicates can inflate one while leaving the other unchanged. Decide the supported correction operation and preserve evidence needed to reverse or audit it. This guide deletes no rows, changes no task states and performs no external publication. After an independently authorized repair, a meaningful test repeats one synthetic interval and interrupts after destination acceptance but before task-state confirmation, verifying that the final business result remains single and complete.

Keep it from recurring

• Make source manifests immutable, keep business operation identity stable across attempts and use destination-supported duplicate-safe publication. Review output retention and reconciliation whenever catchup, backfill range or calculation versions change.