Apache Airflow / Advanced: historical processing, deployment and recovery evidence / Lesson 13

Backfills, partitions and late data

Plan historical reprocessing as a bounded business operation with explicit interval coverage and replacement behavior.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish scheduler catchup, task retries and explicit backfill.
Review reprocessing policy, concurrency and late-correction evidence.

Name the historical result being rebuilt

A backfill creates workflow runs for historical scheduled dates under the selected time-based timetable. It differs from retrying one failed task and from scheduler catchup that creates missed intervals. State why history is being rebuilt: a corrected calculation, late source data or a missing output require different acceptance evidence. A task that always reads latest input cannot safely reconstruct a past partition just because its run has an old label. Keep source version, business interval and calculation version explicit. If the original input has expired, report the coverage gap instead of pretending that a newer source snapshot reproduces the original historical facts.

Existing run state changes the plan

Airflow 3.1 backfill reprocessing choices distinguish creating only missing dates, reconsidering failed runs and reconsidering completed or failed runs. A currently queued or running latest run is not simply duplicated under those choices. Review the exact documented policy against existing state before predicting which dates will be created. A dry-run date list is planning evidence rather than a promise of the eventual state-sensitive result. Backfill max_active_runs is independent of the DAG's ordinary max_active_runs setting in this baseline, so protecting an external system requires reviewing backfill admission as well as normal scheduling limits.

List a bounded business range offline

The Python example enumerates three declared daily partitions using an explicitly inclusive start and exclusive end. This is the course's planning convention, not the boundary contract of an Airflow CLI option. It produces a reviewable list without creating runs or contacting metadata storage. Compare the list with expected source manifests and current destination partitions before any authorized execution. Calendar requirements in local timezones need separate boundary handling; the example uses plain dates deliberately. Record a stopping boundary and a maximum expected item count so a mistaken date range cannot silently turn a small correction into years of unintended work.

python Offline bounded historical partition plan

from datetime import date, timedelta
start = date(2026, 10, 1)
end_exclusive = date(2026, 10, 4)
partitions = []
current = start
while current < end_exclusive:
    partitions.append(current.isoformat())
    current += timedelta(days=1)
assert partitions == ['2026-10-01', '2026-10-02', '2026-10-03']
print(len(partitions))

Expected output

Three planned dates are listed; this does not reproduce Airflow command boundary semantics or create runs.

Choose replacement, amendment or versioned output

Late data can change the correct report for an already published interval. Decide whether a reprocessing operation replaces that interval atomically, publishes a new report version or appends a clearly defined amendment. Blind append often duplicates original rows. Readers must know which version is authoritative and how an interrupted publication is recovered. The design record names a versioned replacement intent and stable partition identity; it does not implement remote storage atomicity. An approval or publication boundary may be needed in a real business workflow, but the task logic should still produce concrete evidence that old and corrected totals were reconciled.

json Synthetic corrected-output publication contract

{"business_partition":"2026-10-01","calculation_version":"v2","publication":"versioned-replacement","source_manifest":"lab-manifest-17","expected_previous_version":"v1"}

The fields describe business intent and must be implemented by a destination-supported publication protocol.

Verify coverage before declaring backfill complete

Run order can matter when historical partitions depend on earlier outputs; reverse order is not universally safe merely because a tool offers it. Prefer independent partition logic where possible and make remaining dependencies explicit. Acceptance compares required partitions, unique business identities, calculation versions and published artifacts, including failed and intentionally excluded intervals. A green set of runs is not enough if one expected date was never created. This course performs no backfill command, task clearing or state mutation. Only the finite date enumeration and JSON design record are checked locally, leaving orchestration and destination recovery behavior untested.

Common errors and fixes

A backfill rewrites every date with the latest source data.

Why it happens: Historical labels were not connected to input partitions.
How to fix it: Select versioned source coverage for each declared interval.
Prevent it next time: Test one old partition executed on a later day.

Backfill overloads a system despite normal DAG-run limits.

Why it happens: Independent backfill admission was omitted from the capacity review.
How to fix it: Review backfill concurrency and shared resource pools together.
Prevent it next time: Include historical workloads in the protected-resource budget.

A green backfill misses a required date.

Why it happens: Existing-run policy and intended coverage were not reconciled.
How to fix it: Compare the required partition set with actual accepted outputs.
Prevent it next time: Retain a finite coverage checklist with the recovery result.

Check your understanding

1. Why is a list of dates from a dry-run plan insufficient proof that every run will be created?

Show worked solution

Creation also depends on reprocessing policy and existing run states at execution time. A date may already have a completed, queued or running run whose handling differs under that policy. Review both the intended range and state-sensitive rules, then verify final business output coverage rather than counting planning lines.

2. Should a corrected report append the original interval rows again by default?