Data Engineering / Advanced: distributed correctness and reliability / Lesson 18
Recovery, replay and operational reliability
Prove recoverable data state through retained inputs, compatible transformations and controlled service authority. Connect recovery objectives to data acceptance rather than merely restarting jobs.
On this page
Lesson goals
- Define recovery scope, RPO/RTO and isolated validation.
- Reconcile replay, checkpoints and consumer publication after failure.
Recover a defined dataset and service outcome
Recovery point objective concerns acceptable lost history, while recovery time objective concerns time to restore the declared service outcome. A restarted scheduler is not a recovered dataset if required source events, schema or sink state are missing. Inventory raw/history retention, target snapshots, checkpoints, code/configuration, credentials and necessary keys. Define whether consumers can tolerate stale or provisional data during repair. A backup artifact is useful only with compatible restore/replay and acceptance evidence, and live replication can faithfully reproduce a mistaken delete instead of preserving a safe historical point.
Preserve enough versioned context for replay
The synthetic recovery plan names the input horizon, output version, transformation and checkpoint interpretation needed to reconstruct a state. It creates no infrastructure, backup or restore job. A real plan must also prove that the referenced inputs and credentials remain available independently of the failed source. Do not preserve a checkpoint that refers to target effects no longer present after restore without reconciliation. The correct recovery point is a coordinated data boundary, not simply the newest file timestamp or largest stored offset found on disk.
{
"exercise": "synthetic-flow-recovery",
"retainedInput": "lab-events-v1",
"targetSnapshot": "lab-accepted-output-v1",
"transform": "normalize-v1",
"checkpointRule": "must-match-restored-target-boundary",
"validation": ["known-keys", "amounts", "duplicates", "consumer-version"],
"productionResources": false
}The record describes required relationships only. It does not prove recoverability or authorize a real restore.
Rehearse failure windows without using live data
Use approved synthetic inputs and isolated destinations to exercise partial batches, stale checkpoints, duplicate replay, missing schemas and consumer replacement. Verify known keys and values, not only object counts or a process health check. Keep scheduled jobs and outbound notifications from reaching real consumers. Measure provisioning, restore, replay and validation under the stated clock boundary. Report which dependencies were simulated and which remain untested. Do not claim a production recovery SLA from an in-memory example, and do not weaken access controls to make an exercise appear successful.
Verify replayed keys and values under a declared identity rule
The local reconstruction function accepts repeated identical event identities and rejects conflicting content. Replaying the supplied duplicate therefore yields two unique events and total thirty. This is a bounded in-memory rule, not a distributed log or durable deduplication service. It makes one recovery acceptance invariant explicit. Real recovery must retain enough source history to reconstruct the promised population and must coordinate output/checkpoint state and downstream replacement. If historical code or units changed, pin or convert them under a reviewed migration rather than silently rerunning current logic against old data.
def reconstruct(events):
result = {}
for event_id, amount in events:
if event_id in result and result[event_id] != amount:
raise ValueError('CONFLICTING_EVENT_CONTENT')
result[event_id] = amount
return result
state = reconstruct([('e1', 10), ('e2', 20), ('e1', 10)])
assert state == {'e1': 10, 'e2': 20}
assert sum(state.values()) == 30
try:
reconstruct([('e1', 10), ('e1', 11)])
except ValueError as error:
assert str(error) == 'CONFLICTING_EVENT_CONTENT'
else:
raise AssertionError('conflict was not rejected')Identical replay is tolerated and conflicting content is rejected. The program contacts no system and provides no crash durability.
Resume one accepted history and verify downstream repair
Before resuming publication, identify the authoritative target, accepted source boundary and consumer update mode. Fence obsolete writers when required and prevent the repaired output from being added to the old result accidentally. A rollback after unique new writes needs history reconciliation, not merely an endpoint change. Record failed recovery checks, owners and follow-up dates, then repeat the relevant scenario after remediation. Reliability combines controlled change, retained evidence, capacity and verified data meaning. A green task or successful connection is one observation, not proof that the complete production flow is correct and recoverable.
Common errors and fixes
A restored checkpoint points beyond restored data
- Why it happens
- Progress and target recovery boundaries were not reconciled.
- How to fix it
- Restore/replay under one accepted coordinated boundary.
- Prevent it next time
- Test mismatched checkpoint/snapshot recovery.
Current transform silently changes historical meaning
- Why it happens
- Code, units or schema versions were not retained.
- How to fix it
- Pin compatible logic or apply a declared conversion.
- Prevent it next time
- Keep transformation lineage for the recovery horizon.
A recovery exercise triggers real outbound effects
- Why it happens
- Isolation omitted consumers, jobs or credentials.
- How to fix it
- Stop under the approved plan and repair the isolation boundary.
- Prevent it next time
- Verify destination and outbound isolation before replay.
Check your understanding
1. Why is restoring the largest checkpoint insufficient evidence of safe restart?
Show worked solution
It may refer to effects absent from the restored target, causing skipped work. Match the checkpoint to the accepted target snapshot and replayable source history, then verify key/value and consumer semantics. Progress metadata is useful only when its data boundary remains valid after recovery.
2. What should a successful isolated recovery report still disclose?
Show worked solution
The tested data volume, failure scenario, clock boundary, simulated dependencies and untested production assumptions. It demonstrates that exercise, not every outage or universal RPO/RTO compliance. Preserve remaining risks and owners and schedule proportionate approved follow-up rather than turning laboratory arithmetic into an operational guarantee.
Key takeaways
- Recover data, progress, code and consumers as a coordinated contract.
- Rehearsal evidence has an explicit scope and remaining uncertainty.