AWS / Advanced: reliability, security, and governance / Lesson 13
Reliability, failure domains, and scaling
Design surviving useful capacity, bounded failure handling, and observable recovery across AWS service boundaries.
On this page
Lesson goals
- Explain the assumptions, responsibilities, and trade-offs of reliability, failure domains, and scaling.
- Interpret both examples and distinguish declared intent from verified evidence.
- Diagnose the listed failures and justify the worked decisions without using a real AWS account, credential, or service.
Define reliability at the user boundary
Reliability is the ability to perform the intended function under stated conditions, not simply the presence of multiple instances. Define important operations, acceptable error and latency behavior, data integrity, and recovery objectives. A service SLA and an application SLO have different scopes and purposes; neither proves that the workload architecture meets its target. The first synthetic record names a user contract and failure assumptions rather than promising zero downtime. Include configuration, identity, DNS, database, queue, and operational dependencies in the model. Redundant compute can still share one fragile dependency or one overprivileged recovery path.
journey: synthetic-valid-booking
objective: agreed-error-latency-and-integrity-contract
failure_assumption: one-reviewed-AZ-path-unavailable
surviving_capacity: measured-separately
shared_dependencies: explicitly-inventoried
recovery: authorized-bounded-runbook-and-user-checks
The record names conditions that require evidence and makes no availability percentage or AWS service guarantee. It is not a deployable failover policy.
Separate placement from surviving capacity
Multiple AZs reduce selected local failure exposure only if the remaining path can serve demand and access required state. A database, NAT path, storage placement, or network dependency can have a different scope. Service modes and Regional features determine which redundancy AWS supplies and which the customer configures. When coordinating zones across accounts, use documented zone identity rather than assuming friendly names always identify the same physical location. The second offline simulation removes one synthetic capacity unit group and compares remaining capacity with fixed demand. It models arithmetic, not AWS availability or a validated capacity experiment.
fictional_capacity = {"zone-a": 8, "zone-b": 8, "zone-c": 8}
remaining = sum(value for zone, value in fictional_capacity.items() if zone != "zone-a")
demand = 15
print(remaining)
print(remaining >= demand)
print("fixed fictional units; no latency or AWS guarantee")
16
True
fixed fictional units; no latency or AWS guaranteeSixteen fictional units remain and exceed fixed demand fifteen. Bursts, placement, shared dependencies, state, and response latency are outside this model.
Bound retries, queues, and admission
Timeouts bound waiting, while retries may repeat work and amplify dependency load. Use an overall deadline, bounded attempts, backoff and jitter where appropriate, and idempotency or reconciliation for side effects. SDK retry behavior depends on configuration, service, and version and should not be assumed universal. Queues absorb bursts but need age, size, concurrency, and recovery policy. Rejecting or degrading low-priority work can preserve an important journey when capacity is constrained. A circuit breaker or fallback also needs a defined recovery probe and correctness contract; returning stale or incomplete data is not automatically acceptable.
Scale the bottleneck with delay in mind
Auto Scaling or serverless concurrency changes desired compute, not instantaneous end-to-end capacity. Startup, image or package availability, quota, placement, connection pools, and downstream limits affect useful recovery time. Scaling the wrong layer can worsen database saturation. Keep headroom for failure, deployment overlap, and burst demand, and set maximums consistent with financial and dependency limits. Use approved representative measurements before selecting thresholds. This course does not generate load, request capacity, simulate a real AZ failure, or change a scaling policy. Existing documentation and fictional arithmetic cannot certify production survival.
Make recovery observable and reversible where possible
A recovery runbook should name decision authority, safe evidence, compatible artifacts, configuration, credentials, data checks, and user validation. Traffic movement does not undo accepted messages or remote transactions. Test selected failure transitions in an authorized isolated environment with synthetic work and stop conditions, then update assumptions when architecture or scale changes. Record what a control protects against and what remains outside scope. No real service, alarm, health check, failover, traffic shift, or administrative action is performed here. Reliability emerges from maintained contracts and verified operations rather than a diagram with two AZ boxes and a generic failover arrow.
Common errors and fixes
Two AZs are treated as an availability proof
- Why it happens
- Shared state and surviving capacity were omitted.
- How to fix it
- Trace the full surviving user path and dependencies.
- Prevent it next time
- Test named failure assumptions with bounded authorized exercises.
Retries make an outage larger
- Why it happens
- Attempts and total deadlines were unbounded.
- How to fix it
- Apply retry budgets, admission control, and safe operation identity.
- Prevent it next time
- Measure attempts per original request and dependency pressure.
Scaling applications overloads the database
- Why it happens
- Desired replicas were confused with end-to-end capacity.
- How to fix it
- Bound concurrency and identify the measured bottleneck.
- Prevent it next time
- Include downstream budgets in scaling policy.
Check your understanding
1. An application has two equal capacity groups and needs nearly both to serve normal load. Is losing one safe?
Show worked solution
Not from that design. Redundancy without sufficient surviving useful capacity can still fail the user contract. Measure demand, degradation options, provisioning delay, and dependency capacity before claiming failure tolerance.
2. Why does a timeout not prove an external write failed?
Show worked solution
The operation may commit after the caller stops waiting or its response may be lost. Use idempotency, status reconciliation, and safe retry semantics. Treat an uncertain result explicitly instead of translating every timeout into permission to repeat a side effect.
Key takeaways
- Reliability is the ability to perform the intended function under stated conditions, not simply the presence of multiple instances.
- Timeouts bound waiting, while retries may repeat work and amplify dependency load.
- A recovery runbook should name decision authority, safe evidence, compatible artifacts, configuration, credentials, data checks, and user validation.