SRE / Advanced: controlled failure and service readiness / Lesson 14

Canaries, compatibility, and change risk

Limit release exposure and compare meaningful cohorts while checking data compatibility, telemetry coverage, stop conditions, and recovery evidence.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Identify the full release identity.
Interpret canary evidence and limitations.
Define compatible rollout and recovery gates.

Know exactly what changed

A release includes artifact, configuration, feature exposure, schema assumptions, and sometimes dependency behavior. A Git revision alone does not identify all of that state. Record immutable artifact identity and the reviewed configuration revision, then verify what each cohort actually runs. A successful deployment API response is not evidence that users received a healthy result. Separate build correctness, rollout completion, and service outcome checks. A configuration-only change can be as risky as code, particularly when it changes deadlines, retries, routing, or authorization. Use the same review discipline for the complete behavior change.

Choose a comparable control

A canary exposes a bounded subset to the candidate while a control continues serving comparable work. Compare user-facing errors and latency with traffic counts, request mix, region, and dependency conditions. If the candidate receives only easy cached reads, a healthy result says little about reservation writes. The offline example reports synthetic control and candidate error fractions and their difference in percentage points. It does not establish statistical confidence, random assignment, or a safe rollout decision. Low sample counts and rare failure paths require additional reasoning rather than an automatic green signal.

python Offline canary cohort arithmetic

control_bad, control_total = 4, 2000
candidate_bad, candidate_total = 16, 2000
control = control_bad / control_total
candidate = candidate_bad / candidate_total
print(f"control={control:.2%}")
print(f"candidate={candidate:.2%}")
print(f"difference_pp={100 * (candidate - control):.2f}")

Expected output

control=0.20%
candidate=0.80%
difference_pp=0.60

The candidate is 0.60 percentage points worse on this synthetic fixture. The calculation is a comparison, not a statistical release evaluator.

State proceed, pause, and abort conditions

A rollout plan needs explicit exposure stages, observation requirements, decision ownership, and recovery criteria. Include absolute user-health checks as well as candidate-control comparison: both cohorts may fail because of a shared dependency. Missing or stale telemetry should prevent a confident health claim. The editorial gate below distinguishes insufficient evidence from verified failure. Do not interpret a fixed wait as sufficient coverage when the critical workflow has not run. A release can remain paused while a human reviews uncertainty instead of forcing every state into a binary success or failure.

yaml Illustrative rollout gate

release_gate: reservation-candidate
proceed_requires: [representative-work, fresh-telemetry, healthy-users, reviewed-comparison]
pause_on: [insufficient-events, missing-evidence]
abort_on: [verified-user-regression, unsafe-data-effect]
recovery_requires: compatible-previous-artifact-and-config
decision_owner: release-lead

This record is an editorial checklist, not deployment configuration. An abort decision still requires an authorized compatible recovery procedure.

Check whether rollback is truly compatible

Rolling back code does not undo a database migration, emitted message, external charge, or changed data representation. Prefer compatible staged evolution when mixed versions will coexist: add support, migrate or backfill under controlled conditions, then remove old assumptions only after consumers no longer need them. The exact method depends on the data system and cannot be inferred from this conceptual sequence. Review both forward and backward compatibility and define reconciliation for irreversible effects. A feature flag can reduce exposure, but its default, propagation delay, ownership, and removal plan also need verification.

Use release evidence after the decision

Retain the release identity, cohort assignment, evaluation interval, decision, and outcome in a safe audit record. After promotion, continue observing broader traffic because the canary cannot cover every tenant, region, data shape, or long-running task. Review aborted releases without blaming the person who noticed the regression. A canary that stops a harmful rollout is a successful defense, even though the candidate was rejected. Improve workload coverage and stop criteria from what escaped or caused false alarms. No release, deployment, migration, or live traffic experiment is executed by this lesson.

Common errors and fixes

A canary passes but writes fail after promotion

Why it happens: The cohort did not exercise representative write paths.
How to fix it: Review workload coverage and pause unsupported conclusions.
Prevent it next time: Include critical journeys and data shapes in evaluation.

Rollback restores code but breaks data reads

Why it happens: Schema and data compatibility were ignored.
How to fix it: Use the approved compatible recovery or forward repair.
Prevent it next time: Review mixed-version and irreversible effects before release.

Both cohorts fail but the comparison looks equal

Why it happens: Only relative regression was checked.
How to fix it: Check absolute user health and shared dependencies.
Prevent it next time: Require both absolute and comparative evidence.

Check your understanding

1. A candidate has zero errors after ten requests. Is that enough to promote?

Show worked solution

Not by itself. The sample may miss rare paths and may not represent the workload. Check the agreed gate, exposure risk, traffic mix, objective, and observation requirements. Keep the state insufficient evidence when the relevant journeys have not been exercised; do not invent confidence from zero observed errors.

2. What must a rollback plan identify besides the old image?