SRE / Advanced: controlled failure and service readiness / Lesson 16

Reliability tests and bounded experiment design

Choose tests for explicit failure mechanisms and design fault experiments with authorization, containment, abort conditions, and honest evidence boundaries.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Map tests to failure risks.
Separate offline models from environment evidence.
Define a contained experiment and acceptance criteria.

Start from a failure claim

A reliability test should address a concrete risk: expired work executes, a duplicate message creates a second reservation, a missing dependency blocks every worker, or a restore omits required state. Different risks need different test boundaries. A unit test can validate a deadline comparison, while an integration test may be needed to verify that cancellation reaches a specific client library. An end-to-end test can exercise a user path but may not explore every state transition. Write the failure mechanism and expected defense before selecting a testing tool, so a passing suite has an interpretable meaning.

Use a minimal model to test an invariant

The offline example accepts synthetic work only when its deadline is strictly in the future. It tests the boundary where current time equals the deadline and confirms that expired work is rejected. This is meaningful evidence for that pure rule, not proof that a real queue propagates time correctly or cancels work. Models are valuable because they make assumptions visible and execute without external services. Keep them small enough to inspect. If a model omits concurrency, network behavior, or persistence, state those omissions rather than implying that the system has been tested under those conditions.

python Offline deadline invariant test

def may_start(now, deadline):
    return now < deadline
assert may_start(9, 10) is True
assert may_start(10, 10) is False
assert may_start(11, 10) is False
print("deadline-boundary-checks-passed")

Expected output

deadline-boundary-checks-passed

Only a pure comparison is tested. The script uses no clock, queue, network, service, or fault injection.

Choose the smallest boundary that answers the question

Use deterministic unit and contract fixtures for classifications, state transitions, and malformed inputs. Use approved isolated integration environments for protocol and persistence behavior that the pure model cannot represent. Reserve broad experiments for questions that truly require system interaction, and do not assume that more realistic automatically means more informative. A complicated test can fail for unrelated environment reasons and obscure the mechanism. Record versions, workload shape, seeds where appropriate, and evidence collection. Avoid testing only the happy path: duplicate, delayed, missing, and partially completed operations often expose the most important reliability assumptions.

Design containment before any fault

A fault experiment requires explicit authorization for its environment and action. Define a hypothesis, baseline, maximum affected scope, duration, observation method, abort conditions, recovery path, and accountable operator. The editorial plan below stays non-executable and targets an isolated synthetic model. Never infer permission to disrupt a configured service from permission to read educational material. An emergency stop must remain usable if the experiment’s own telemetry or dependency fails. Review whether the learning can be obtained by a safer offline fixture first. Controlled experiments are engineering work, not a demonstration of willingness to break things.

yaml Non-executable experiment plan

experiment_design: synthetic-dependency-unavailable
environment: isolated-offline-model-only
hypothesis: bounded-failure-does-not-admit-expired-work
maximum_scope: fixed-synthetic-event-list
abort_on: any-unexpected-external-access
recovery: discard-model-state
acceptance: invariant-and-expected-outcome-reviewed

The plan documents containment and acceptance. It does not inject a fault, access a cluster, run a load generator, or alter service configuration.

Report exactly what the evidence supports

A passing local fixture establishes the behavior of the tested code under its inputs. It does not establish production availability, disaster recovery readiness, peak capacity, or the behavior of a different runtime version. Report failed checks and untested conditions beside successful results. When a test discovers a weakness, assign remediation and preserve a regression case that targets the mechanism. Review brittle tests that mirror implementation details without protecting a meaningful invariant. Reliability confidence grows from complementary evidence and explicit limitations, not from a large undifferentiated count of green checks.

Common errors and fixes

A model result is advertised as production proof

Why it happens: The evidence boundary was omitted.
How to fix it: State tested inputs, assumptions, and missing system behavior.
Prevent it next time: Separate model, integration, and operational claims.

A fault experiment has no abort path

Why it happens: Containment was considered after execution.
How to fix it: Do not execute until stop and recovery are reviewed.
Prevent it next time: Require authorization and bounded scope in the plan.

Tests pass while duplicate writes remain possible

Why it happens: Only happy-path examples were covered.
How to fix it: Add mechanism-focused duplicate and partial-completion cases.
Prevent it next time: Map regression cases to user and data invariants.

Check your understanding

1. What does the deadline fixture prove when all assertions pass?

Show worked solution

It proves that the shown may_start function uses a strict comparison for the three supplied cases in the local Python execution. It does not prove deadline propagation, clock agreement, queue integration, or cancellation. Those require separate evidence at the relevant implementation boundary.

2. A team proposes a live dependency outage to test retries. What must be reviewed first?