Data Engineering / Advanced: distributed correctness and reliability / Lesson 15

Backfills, schema evolution and cutover

Repair an identified historical population without corrupting live writes or changing meaning silently. Stage compatibility, isolated validation and a data-safe cutover boundary.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Define immutable backfill inputs, scope and output version.
Coordinate live/backfill concurrency and rollback history.

A backfill is a controlled new computation

A backfill recomputes an identified historical population under a stated transformation and schema. Record why repair is needed, which partitions/keys are affected and whether consumers receive replacement, deltas or new versions. Reading current source state may not reproduce what existed historically, so establish available source snapshots or change history. A date range alone is incomplete identity when inputs can change beneath it. Keep the original accepted output and lineage long enough to compare and recover. A rerun should not become an unreviewed broad overwrite merely because the task name includes repair.

Keep backfill identity distinct from live publication

The JSON declares one synthetic repair interval and a new output version, with publication blocked until comparison and consumer readiness are checked. It is not a runnable scheduler or table-format request. Real plans need the exact input versions, resource/admission budget, owner, overlap handling and rollback boundary. Avoid consuming unbounded capacity while live freshness degrades. A temporary branch or staging area can help isolate a candidate, but its guarantees depend on the actual engine and metadata protocol, not the label branch itself.

json Nonexecuting historical-repair plan

{
  "repair": "lab-correct-units-v2",
  "interval": ["2026-10-01", "2026-10-03"],
  "inputVersion": "lab-source-snapshot-v1",
  "outputVersion": "lab-candidate-v2",
  "publicationGate": ["key-and-value-reconciliation", "consumer-readiness"],
  "liveWrites": "require-reviewed-overlap-protocol"
}

This metadata performs no backfill or traffic change. Its gate is a requirement, not evidence of a completed comparison.

Expand readers before contracting old representations

A rolling migration can leave old and new producers/consumers active together. Add compatible reading, define new writes, backfill and verify before removing old fields or changing authority. A physically wider type may still have incompatible units, null behavior or default meaning. Avro schema resolution and Iceberg field-ID evolution illustrate different mechanisms, not a universal cross-format rule. Test the actual writer/reader versions and accepted records. If rollback requires old code, ensure the new data remains interpretable or define a tested conversion/history path rather than assuming a binary rollback restores compatibility.

Compare complete key coverage as well as totals

The local candidate has the same total as the accepted version but a different key set. The check rejects it because one old key is missing and a new unexpected key offsets its amount. This counterexample prevents a backfill approval based only on aggregate equality. Actual repair checks should include partition scope, identities, occurrence counts, values, schema and intended business corrections. A legitimate changed key set may be acceptable only under an explicit contract revision, not because a convenient reconciliation happened to pass.

python Offline equal-total/wrong-population counterexample

accepted = {'e1': 10, 'e2': 20}
candidate = {'e1': 10, 'e3': 20}
assert sum(accepted.values()) == sum(candidate.values())
assert set(accepted) != set(candidate)
assert set(accepted) - set(candidate) == {'e2'}

The total 30 conceals missing e2 and unexpected e3. The program reads only local synthetic dictionaries.

Define authority after the first new write

At cutover, establish which output version is authoritative and how consumers replace prior results without duplicate accumulation. Coordinate overlapping live writers with the storage protocol, fence obsolete producers where required and preserve source progress. Once the destination accepts unique new changes, pointing readers or writers back without reconciliation can lose or fork history. Rehearse rollback eligibility, retention and downstream repair. A successful backfill task is one evidence item; service acceptance also needs data-level comparison, consumer compatibility, freshness and an owner who approves remaining uncertainty.

Common errors and fixes

Backfill and live writer overwrite each other

Why it happens: Historical scope and concurrency/publication protocol were undefined.
How to fix it: Stage the candidate and use the supported conflict/merge process.
Prevent it next time: Test overlapping keys and commit order.

Matching totals hide changed keys

Why it happens: Reconciliation ignored population identity.
How to fix it: Compare key coverage and intended value changes.
Prevent it next time: Include offsetting missing/unexpected fixtures.

Rollback loses destination-only data

Why it happens: Endpoint/version reversal ignored new accepted history.
How to fix it: Reconcile authority and history through the rehearsed plan.
Prevent it next time: Define the last safe rollback point before cutover.

Check your understanding

1. Why can reading today’s source fail to reproduce last month’s accepted dataset?

Show worked solution

The source may have changed, deleted or corrected records since that period. Reproduction requires the identified historical snapshot/log and transformation context, not merely a date filter on current state. If that history is unavailable, report the limitation and define an accepted repair contract instead of pretending the result is identical.

2. A candidate backfill is valid but consumers add it to the old total. What contract is missing?