SRE / Advanced: controlled failure and service readiness / Lesson 15
Recovery objectives and data integrity
Distinguish redundancy, backups, and verified restoration, then reason about recovery time, recoverable data, dependency order, and reconciliation.
On this page
Lesson goals
- Explain RTO and RPO as objectives with boundaries.
- Evaluate backup completeness and restore evidence.
- Include application invariants and residual work in recovery.
Define the recovery promise
Recovery time objective, or RTO, expresses the intended maximum restoration time for a defined service scope. Recovery point objective, or RPO, expresses the tolerable data-loss interval relative to the incident. Neither is a result until demonstrated under relevant conditions. State when the clock starts, what restored means, and which dependent systems are included. A database accepting connections is not a complete Library recovery if reservation confirmations disagree with stored records. Business owners should review the consequences of lost or delayed work, and engineering owners should identify the evidence and costs needed to support the objectives.
Separate independent protection mechanisms
Replication can improve availability but may rapidly copy an accidental deletion or corrupt application write. A backup can preserve an earlier state, but only if it is complete, accessible, protected, and restorable with the needed metadata and keys. A successful backup job proves less than a verified application restore. Consider retention, isolation, access control, consistency, and compatibility with the intended recovery software. The editorial record below names the protected state and missing prerequisites explicitly. It is not a backup command, and no data store or encryption key is accessed by the course.
recovery_scope: library-reservations
protected_state: [reservation-records, required-metadata, compatible-schema]
prerequisites: [approved-key-access, restore-tool-compatibility, dependency-order]
verification: [record-count-checks, invariant-checks, representative-journeys]
residual_work: reconcile-external-confirmations
The contract separates restoring storage from verifying the user journey and reconciling external effects. It is not executable recovery configuration.
Calculate an illustrative recovery gap
The offline timeline assumes an incident at minute sixty, a latest usable recovery point at minute fifty-five, and full verified service restoration twenty minutes after the incident. The observed recovery-point gap is five minutes and elapsed recovery is twenty minutes in this model. These are not measured RPO or RTO achievements for a real system. A backup schedule alone cannot establish the latest usable point because jobs can fail or produce unusable artifacts. Log shipping and point-in-time recovery introduce platform-specific requirements that need their own verified procedure and evidence.
incident_minute = 60
latest_usable_point = 55
verified_restoration_minute = 80
print("point_gap_minutes", incident_minute - latest_usable_point)
print("recovery_minutes", verified_restoration_minute - incident_minute)
point_gap_minutes 5
recovery_minutes 20All timestamps are synthetic minutes on one timeline. The model neither reads backups nor performs restoration or measures a service.
Verify meaning, not just bytes
Checksums can detect some artifact corruption, while application invariants check whether restored state makes sense. For reservations, every confirmation should map to a valid record, available inventory should not be negative, and replayed messages should not create duplicates. Choose checks appropriate to the actual data model and privacy constraints. Restore order matters when one system references identifiers or events from another. An apparently successful storage restore may still leave caches, indexes, external notifications, or downstream consumers inconsistent. Record the authoritative source and reconciliation process before reopening writes broadly.
Rehearse the complete authorized path
A recovery exercise should validate access, documentation, dependency order, restoration, application checks, communication, and residual work within an approved isolated environment. Keep the exercise scope and cleanup explicit, and do not use production credentials or unapproved data copies. Measure the phases separately to find the largest delay; obtaining access can dominate data transfer. Retain evidence of the tested version and limitations, then fix gaps with owners and review dates. This lesson provides only local arithmetic and an editorial plan; it makes no claim that a backup, failover, or restore procedure has been executed.
Common errors and fixes
Replication is called a backup strategy
- Why it happens
- The same harmful write can reach every replica.
- How to fix it
- Evaluate independent recoverable historical state.
- Prevent it next time
- Review deletion, corruption, and credential-loss scenarios.
A backup success message is treated as recovery proof
- Why it happens
- Restore dependencies and invariants were never tested.
- How to fix it
- Require an approved isolated end-to-end restore exercise.
- Prevent it next time
- Track usable recovery evidence and its limitations.
Recovered storage serves inconsistent user outcomes
- Why it happens
- External effects and indexes were omitted.
- How to fix it
- Reconcile authoritative state and dependent views.
- Prevent it next time
- Define application-level closure criteria.
Check your understanding
1. Backups run every fifteen minutes. Can you guarantee a fifteen-minute RPO?
Show worked solution
Not from the schedule alone. You need evidence that a sufficiently recent consistent backup or recovery stream is usable, accessible, and complete when needed. Failed jobs, missing keys, incompatible tooling, or unavailable dependencies can widen the actual recovery gap beyond the schedule.
2. The restore finishes in ten minutes but verification takes forty. What recovery time should be reported?
Show worked solution
Under a service-restored definition that requires verified user outcomes, include the full fifty minutes plus any other included phases. Report phase timings separately for improvement, but do not exclude required verification merely to claim the target was met. State the clock boundary and remaining work.
Key takeaways
- Recovery objectives need explicit service and time boundaries.
- A copy is useful only when safely restorable.
- Verify application invariants and reconcile residual effects.