DevOps / Intermediate: controlled delivery systems / Lesson 11

Backups, restoration, and continuity

Define recovery objectives, protect consistent copies, and verify restoration as a user-visible capability.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Define recovery point, time, and consistency objectives.
Verify restored data and external side effects.
Plan access, dependency order, and recovery rehearsals.

Start with loss and time objectives

Recovery point objective, RPO, describes the tolerated data-loss interval; recovery time objective, RTO, describes the tolerated recovery duration under stated conditions. They are business requirements to validate, not properties automatically supplied by a backup product. The first record assigns synthetic objectives to a training service and names consistency scope. A ten-minute backup schedule does not guarantee ten-minute RPO if jobs fail or copies cannot be restored. Include detection, decision, access, provisioning, restoration, reconciliation, and verification in a realistic recovery timeline rather than timing only file transfer.

yaml Synthetic recovery agreement

service: training-bookings
rpo_minutes: 15
rto_minutes: 60
consistency_scope: bookings-and-payment-reference-ledger
restore_location: isolated-lab
external_effects: disabled
verification: integrity-and-selected-user-journeys

The objectives apply to a declared consistency scope and a lab exercise. They are not guarantees or instructions to restore an actual service.

Preserve consistent and independent copies

A copy taken while an application writes can be internally inconsistent unless the storage and application coordinate the snapshot. Use the database’s supported backup method and include required transaction logs, keys, and metadata. Replication improves some availability properties but can replicate accidental deletion or corruption, so it is not a complete backup strategy. Separate copies from the same failure and credential domain where required, protect retention against unauthorized deletion, and encrypt with a recoverable key policy. A perfectly preserved encrypted archive is useless if all authorized access to its key is lost during the disaster.

Verify content and meaning separately

Checksums detect changed bytes during storage or transfer, but application consistency requires more evidence. The second simulation serializes and reopens a synthetic dataset in memory, then checks an invariant. It does not create a real backup or demonstrate crash consistency. A restoration exercise should also verify schema, account access, expected counts, relationships, and selected user journeys in an isolated environment. Keep production integrations disabled in the recovery lab so restored jobs do not send messages, charge accounts, or mutate another environment. Record both what was checked and what remains outside the exercise.

python Harmless restoration-invariant simulation

import json
original = {"bookings": [{"id": 1, "seats": 2}, {"id": 2, "seats": 3}]}
copy_bytes = json.dumps(original, sort_keys=True).encode()
restored = json.loads(copy_bytes)
assert len({row["id"] for row in restored["bookings"]}) == 2
print(sum(row["seats"] for row in restored["bookings"]))

Expected output

The restored synthetic records have unique IDs and five seats. This in-memory round trip demonstrates a validation idea, not database backup durability.

Rehearse the sequence and access

A runbook should identify decision authority, safe copy selection, dependency order, required permissions, and rollback or abort conditions. Test that the recovery team can access tools and keys when the primary identity system is unavailable. Measure the full sequence with synthetic data and realistic constraints, without generating real external effects. Dependencies such as DNS, certificate issuance, and configuration services may be necessary before users can reconnect. Assign ownership for every handoff, and preserve enough audit evidence to explain which recovery point was selected and why.

Reconcile and learn after restoration

Restoring a copy can rewind local records while external systems retain newer transactions. Reconcile messages, payments, and idempotency records before declaring recovery complete. Communicate the actual loss and availability window rather than repeating the intended objectives. Update backup frequency, retention, runbooks, and capacity when the exercise exposes a gap. A successful backup job is only one intermediate signal; the capability is demonstrated when authorized people restore an appropriate point and verify useful service within the stated constraints. Treat recovery evidence as expiring when architecture, data scale, credentials, or dependencies change materially.

Common errors and fixes

Replication is mistaken for backup

Why it happens: The same deletion or corruption propagates to replicas.
How to fix it: Retain independent recoverable points with protected retention.
Prevent it next time: Test accidental deletion and recovery assumptions.

The archive is valid but the service is inconsistent

Why it happens: Byte integrity replaced application consistency checks.
How to fix it: Restore with supported coordination and verify domain invariants.
Prevent it next time: Include transaction and cross-system scope in backup design.

The recovery team cannot access the key

Why it happens: Recovery depended on the failed primary identity path.
How to fix it: Use the authorized emergency access procedure and audit it.
Prevent it next time: Rehearse key and tooling access during dependency failure.

Check your understanding

1. Backups run every hour, but the latest restorable copy is three hours old. What RPO evidence exists?

Show worked solution

The actual recoverable loss window is at least three hours relative to the incident point, not the one-hour schedule. Verify the selected copy and missing logs, communicate the gap, and repair backup and alerting controls. Scheduling alone is not proof of recoverability.

2. A restored database predates a payment accepted remotely. What follows before reopening writes?