Google Cloud / Advanced: Governance, Reliability, and Architecture / Lesson 17
Backup and Disaster Recovery
Define recoverable data, restoration time, failure scenarios, and application validation before calling a copy a recovery solution.
On this page
Lesson goals
- Distinguish HA, backup, replication, RPO, and RTO.
- Model recovery gaps and timing boundaries.
- Plan restoration evidence and dependency ownership.
A recovery objective starts with a failure scenario
High availability addresses continuity under selected failures, backups preserve recoverable history, and disaster recovery coordinates restoration after a defined disruption. Replication can copy a logical mistake quickly, so it is not automatically protection from deletion or corruption. Define whether the scenario is zonal outage, regional loss, credential compromise, bad migration, or accidental data change.
RPO expresses acceptable data-loss exposure relative to the failure boundary, while RTO expresses a target restoration duration under a defined service criterion. They are objectives, not automatic product guarantees. Choose them from business consequences and verify the actual service configuration and application path that could meet them.
Measure a recovery point against the incident boundary
The first offline program compares an invented failure minute with the latest recoverable minute. The fifteen-minute gap exceeds a ten-minute target. It does not infer a Cloud SQL log-retention setting, object-replication delay, or product RPO. A real gap depends on the consistent recovery point actually available, not merely the time a backup job was scheduled.
A completed backup can still be unusable because of corruption, missing encryption access, unsupported version, or an incomplete application snapshot. Keep backup status, recoverability, and application-consistent restoration as different evidence. Record which writes a restored point includes and how that boundary is verified.
failure_minute = 120
latest_recoverable_minute = 105
target_rpo_minutes = 10
gap = failure_minute - latest_recoverable_minute
assert gap >= 0
print(f"modeled_gap={gap} target={target_rpo_minutes} meets={gap <= target_rpo_minutes}")
modeled_gap=15 target=10 meets=False
The synthetic fifteen-minute gap misses the ten-minute objective; no backup is inspected.
Restoring bytes does not restore the whole service
Recovery can require infrastructure, data, identity, secrets, keys, DNS, network paths, artifact versions, and dependent-service capacity. Protect backup and restore authority separately from ordinary application writes so one compromised identity does not automatically control all copies. Keep restoration location and data classification requirements explicit.
Cross-region replication or a warm standby has service-specific promotion, consistency, quota, and client-routing behavior. Cloud SQL replica promotion, Storage version selection, and VM snapshot restoration are different procedures. Do not transfer one service’s recovery claims to another or assume every operation has an immediate inverse.
Include detection and validation in the timing model
The second program sums invented detection, restoration, validation, and traffic-switch stages. Forty minutes fit a forty-five-minute target under that model. It is not a measured recovery test. A real RTO boundary must specify when the clock starts, which user operations count as restored, and how partial service or data reconciliation is treated.
Parallel stages and uncertain durations can make a real critical path different from a simple sum. Capacity and manual approvals can dominate. Use an approved recovery exercise to collect timing and evidence, then revise the plan. No restore, failover, network switch, or cloud incident is executed by the lesson.
stages = {"detect": 5, "restore": 20, "validate": 10, "switch": 5}
target_rto_minutes = 45
modeled = sum(stages.values())
print(f"modeled_minutes={modeled} target={target_rto_minutes} fits={modeled <= target_rto_minutes}")
modeled_minutes=40 target=45 fits=True
The invented stages total forty minutes; this does not measure Google Cloud recovery time.
A restore test must validate application meaning
A recovery exercise should verify data completeness/consistency, authorized access, application compatibility, dependency health, user transactions, and the chosen time/data boundaries. Use isolated approved resources and synthetic or authorized data; never turn a theory example into an unplanned production restoration. Record runbook version, owner, evidence, gaps, and cleanup responsibility.
Retention, deletion, immutability, soft-delete/version settings, and encryption lifecycles interact. A backup that outlives the required key or credentials may not meet recovery needs. Review the plan after migrations, policy changes, and architecture updates. A backup count alone is not evidence that the current application can recover.
Common errors and fixes
Replication is called a backup against every logical error.
- Why it happens
- Copies and historical recoverability were conflated.
- How to fix it
- Define the threat and retained recovery history.
- Prevent it next time
- Test deletion/corruption scenarios conceptually and under approved exercises.
RTO starts only when data restore begins.
- Why it happens
- Detection, approval, validation, and routing were omitted.
- How to fix it
- Define the end-to-end timing boundary.
- Prevent it next time
- Record stage and critical-path evidence.
A backup survives but its encryption access does not.
- Why it happens
- Recovery dependencies have incompatible lifetimes.
- How to fix it
- Review key, identity, and retention relationships.
- Prevent it next time
- Include restore access in protection evidence.
Check your understanding
1. A backup completed at09:00, but the application-consistent recoverable point is08:50 and the incident is09:05. Which gap matters?
Show worked solution
The relevant modeled recovery-point gap is fifteen minutes from08:50 to09:05, not five minutes from job completion. Verify what data the restored point includes and which clock/reference is used. The scheduled or completed job timestamp alone does not prove consistency or the available transaction boundary. Compare the verified point with the declared RPO target.
2. Why can a successful database restore still fail the application’s RTO?
Show worked solution
The application may still lack compatible code, secrets, key access, network/DNS paths, capacity, client reconnection, or transaction validation. RTO should identify the user-visible service criterion rather than only the database operation’s success. Map dependencies and test the complete approved path, preserving measured evidence instead of extrapolating from one component.
Key takeaways
- Recovery objectives refer to defined failure and service boundaries.
- Backups, replicas, keys, identities, and applications must recover together.
- Local timelines explain assumptions without proving a restore or SLA.