Google Cloud / Advanced: Governance, Reliability, and Architecture / Lesson 16

Reliability, SLOs, and Failure Capacity

Define user-visible success, error budgets, dependency assumptions, and capacity under failure before claiming high availability.

About 5 min readContent reviewed 2026-10-25

On this page

Lesson goals

Distinguish an SLI, SLO, SLA, and recovery objective.
Model dependency and error-budget arithmetic explicitly.
Plan graceful degradation and safe change.

Reliability is a user-facing contract

An SLI measures a defined service outcome, an SLO sets its target and period, and an SLA is a contractual agreement with its own terms. They are not interchangeable numbers. Define eligible requests or windows, good outcomes, missing-data handling, and measurement point. A provider SLA does not automatically become the application’s end-to-end SLO.

Availability, latency, correctness, and data freshness can fail differently. A healthy VM or HTTP200 can still return wrong or stale data. Choose indicators that reflect the application’s purpose and keep the failure model concrete. No example here promises a Google service percentage or contractual remedy.

Dependency arithmetic needs dependence assumptions

The first program multiplies two invented99% availability values for a serial path under a simplified independence model. It yields98.01%, illustrating that required dependencies can reduce end-to-end success. Real failures can be correlated, and provider percentages may have different periods, exclusions, and definitions, so multiplying published SLAs is not a valid automatic application forecast.

Map identity, network, data, DNS, keys, and control-plane dependencies as well as compute. Redundant instances sharing one required database or credential path can retain a single failure point. State which failures the design tolerates and which common causes remain.

python Calculate a simplified dependency model

from decimal import Decimal

first = Decimal("0.99")
second = Decimal("0.99")
combined = first * second
print(f"independent_serial_model={combined:.2%}")

Expected output

independent_serial_model=98.01%

The invented independent serial model gives98.01%; it is not an SLA or measured service result.

An error budget links objectives to allowed bad events

For a request-based model, the allowed bad-event budget is eligible events multiplied by one minus the objective. The second program uses one hundred thousand invented requests and a99.9% objective, allowing one hundred bad requests. Observing one hundred twenty consumes120% of that model budget. This is not a Google product limit or a reason to hide failures by changing eligibility.

A rolling window can recover as old bad events leave the period, while calendar accounting behaves differently. Burn-rate alerts relate current bad-event rate to the allowed rate over specified windows. Choose objectives and response policy from user needs and engineering tradeoffs rather than setting100% to sound reliable.

python Calculate an invented request error budget

from decimal import Decimal

eligible = 100000
objective = Decimal("0.999")
observed_bad = 120
allowed_bad = Decimal(eligible) * (1 - objective)
print(f"allowed_bad={allowed_bad:.0f} remaining={allowed_bad - observed_bad:.0f} consumed={Decimal(observed_bad) / allowed_bad:.0%}")

Expected output

allowed_bad=100 remaining=-20 consumed=120%

The model permits100 bad events, leaving minus20 after observing120.

Redundancy must survive load and state transitions

Redundancy needs sufficient surviving capacity, healthy dependencies, correct routing, and state recovery. Autoscaling cannot instantly guarantee quota, startup, or downstream capacity. Avoid retry storms with bounded retries, backoff/jitter, deadlines, and idempotency appropriate to the operation. A fallback can degrade safely only if its results and limits remain acceptable.

Distinguish load-balancing health from repair triggers and detect gray failures where only some users or operations fail. Test timeouts and overload contracts at component boundaries. Do not simulate cloud incidents or run load tests from a theory lesson; define the approved evidence an operational validation would need.

Reliability includes change and incident practice

Use controlled rollouts, reversible changes where feasible, migration compatibility, and a tested incident response. Error-budget policy can guide when to slow risky releases, but it requires agreed ownership and objectives. Backups and DR address additional failure classes; high availability alone does not restore deleted or corrupted data.

Record assumptions, dependencies, evidence dates, and unresolved risks. Review actual incidents to improve both design and runbooks. The offline calculations explain arithmetic under declared models; they neither measure live reliability nor validate a failover, stress test, SLA, or recovery claim.

Common errors and fixes

A provider SLA is copied as the application SLO.

Why it happens: Different boundaries and dependencies were ignored.
How to fix it: Define the user-facing indicator and full path.
Prevent it next time: Record eligibility, period, and exclusions.

Retries amplify an overloaded dependency.

Why it happens: Failure handling has no bounded admission or deadline.
How to fix it: Use justified retry, backoff, idempotency, and overload policy.
Prevent it next time: Test the contract under approved controlled conditions.

An error budget is improved by excluding failed requests after the fact.

Why it happens: Measurement definition changes hide service failure.
How to fix it: Keep eligibility stable and report any reviewed definition change.
Prevent it next time: Audit numerator and denominator.

Check your understanding

1. Two replicas serve an application but share one required identity/key path. Why might they not provide independent availability?

Show worked solution

A failure or misconfiguration in that shared path can prevent both replicas from working. Count redundancy only after mapping common dependencies and their failure modes. Review key access, identity issuance, network, and data services as well as the replicas. Independence is an assumption needing evidence, not a consequence of having two instance names.

2. What does a consumed error budget of120% establish in the offline example?