SRE / Advanced: controlled failure and service readiness / Lesson 17
Dependency contracts and failure domains
Review dependency criticality, correlated failure, fallback semantics, and control-plane assumptions before claiming that redundant components produce reliable journeys.
On this page
Lesson goals
- Classify hard and optional dependencies.
- Recognize shared failure domains.
- Validate fallback correctness and operational contracts.
Identify what the user path truly requires
A hard dependency must work for the promised operation to succeed. An optional dependency can fail while a deliberately reduced but still valid outcome remains available. Library reservation storage is hard for a durable reservation; personalized recommendations may be optional. Classification is a product and correctness decision, not simply a timeout setting. If the fallback returns a success message without storing the reservation, it has changed the promise rather than preserved it. Draw the synchronous critical path and note background work whose delayed completion still matters to freshness or reconciliation.
State independence before multiplying availability
For an illustrative operation requiring two independent components, multiplying their availability fractions can estimate the joint fraction under that strong assumption. Two 99.9% components yield about 99.8001%, not an improvement over either component. The offline arithmetic makes the assumption visible. Real failures are often correlated through networks, identity, storage, deployment systems, or operators, so the product can be misleading. Redundant paths require a different model with failover and common-cause behavior. Do not turn supplier percentages into a precise service guarantee without examining boundaries, observation periods, and shared risks.
component_a = 0.999
component_b = 0.999
illustrative_joint = component_a * component_b
print(f"{100 * illustrative_joint:.4f}%")
print("assumption: independent-required-components")
99.8001%
assumption: independent-required-componentsThe result applies only to the stated independence model. It is not an availability forecast, measured supplier result, or redundancy calculation.
Look for common causes behind separate boxes
Two replicas may share a host, rack, region, account, identity provider, configuration source, or release mechanism. A single bad configuration can defeat geographical separation if it propagates everywhere. Control-plane outages can prevent scaling or recovery even when existing data-plane traffic continues. Inventory those dependencies and consider the failure modes the design is intended to withstand. Diversity can reduce some correlated risks but adds compatibility and operational complexity. The design record below names a shared identity dependency so that reviewers do not confuse two application regions with fully independent service paths.
service_paths: [region-a, region-b]
shared_dependencies: [identity-provider, configuration-distribution]
review_question: can-existing-reservations-continue-with-control-plane-unavailable
fallback_requirement: preserve-authorization-and-data-correctness
claim_limit: regional-duplication-does-not-prove-common-cause-tolerance
Separate regions still share named dependencies. The record is an architecture review aid and does not inspect or change infrastructure.
Check the meaning and cost of fallback
A cached read may provide a valid degraded result if its age and scope satisfy the user promise. A stale permission decision or inventory count may be unsafe. Define acceptable staleness, disclosure, resource use, and the return path to normal operation. A fallback can overload an older system if every caller switches simultaneously, and a supposedly optional dependency can still block threads while timing out. Verify implementation boundaries rather than relying on architecture labels. Fail-open and fail-closed choices affect security and availability differently and require explicit review for the actual operation.
Create a dependency operating contract
For important dependencies, record ownership, supported operations, quotas, timeout and retry expectations, change notification, telemetry, escalation, and recovery assumptions. Keep user-facing impact separate from blame when the dependency fails. Review the contract after incidents and significant version or workload changes. A dependency map should stay small enough to maintain but detailed enough to reveal critical shared boundaries. Combined with SLOs, capacity evidence, and recovery plans, it supports rational investment: remove an unnecessary synchronous call, isolate optional work, or improve a hard dependency instead of adding redundant boxes that share the same weakness.
Common errors and fixes
Two regions are called fully independent
- Why it happens
- Identity and configuration remained shared.
- How to fix it
- Inventory common causes and limit the claim.
- Prevent it next time
- Review control and data-plane dependencies.
A fallback reports success with stale unsafe state
- Why it happens
- Availability was prioritized without correctness semantics.
- How to fix it
- Define valid degraded outcomes and reject unsafe ones.
- Prevent it next time
- Review staleness and authorization boundaries.
An optional call still stalls the whole request
- Why it happens
- Execution isolation and deadline behavior were missing.
- How to fix it
- Bound waiting and isolate optional resource use.
- Prevent it next time
- Verify implementation behavior with suitable evidence.
Check your understanding
1. A cached catalogue is stale by five minutes. Is it a safe fallback?
Show worked solution
It depends on the catalogue promise and the operation. Browsing a disclosed stale description may be acceptable, while using stale inventory to guarantee a reservation may not be. Define freshness and correctness separately, and ensure the fallback cannot bypass authorization or create an invalid confirmation.
2. Why can a control-plane outage matter after the application stays healthy?
Show worked solution
Existing traffic may continue while scaling, credential renewal, configuration changes, replacement of failed instances, or recovery actions become unavailable. Review the required duration and dependencies of continued operation. A healthy data plane now does not prove that the service can recover from the next failure.
Key takeaways
- Dependency criticality follows the valid user outcome.
- Common causes can defeat visible redundancy.
- Fallbacks need correctness, resource, and recovery contracts.