DevOps QA / Advanced: operational validation and governance / Lesson 13

Progressive-delivery validation and recovery

Connect pre-release evidence to bounded exposure, observation, and a recovery decision without assuming rollback is always simple.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Define staged exposure and explicit promotion states.
Verify recovery and mixed-version behavior before relying on them.

Treat exposure as part of the test argument

Pre-release tests run under selected assumptions; progressive delivery adds evidence from limited exposure to real operating conditions. It does not replace unit, integration, security, or compatibility checks. A stage needs a candidate identity, cohort definition, duration or sample requirement, relevant indicators, stop conditions, and responsible owner. A small traffic percentage can still affect high-value users or irreversible operations. Define blast radius in user and data terms rather than relying on one percentage. This course models decisions only and never changes traffic, deploys software, or operates a live service.

Specify a reviewable stage plan

A stage plan makes the transition criteria inspectable before execution. Separate exposure control from evidence analysis so an analysis failure does not accidentally expand traffic. The synthetic plan below is not a vendor deployment manifest and contains no service credentials. Its percentages and windows are teaching choices, not safe defaults. A real plan must consider traffic volume, business cycles, data migrations, and user consent where relevant. Include an inconclusive outcome for sparse data or broken telemetry; elapsed time alone cannot create the evidence a stage failed to collect.

json An application-defined lab exposure plan

{
  "candidate": "lab-artifact-73",
  "stages": [
    {"name": "limited", "exposurePercent": 2, "minEligibleEvents": 500},
    {"name": "expanded", "exposurePercent": 10, "minEligibleEvents": 2000}
  ],
  "missingTelemetry": "hold",
  "stopOn": ["data-integrity-failure", "confirmed-slo-regression"],
  "recoveryOwner": "delivery-maintainers"
}

No controller consumes this file in the course. Candidate labels are synthetic; real promotion must bind verified immutable artifacts and reviewed policy. Minimum event counts require a statistical and business justification for the actual service.

Test the coexistence window

During a rollout, old and new application versions may read the same database, consume the same events, and serve the same clients. Verify schema and message compatibility across that window. Expand-and-contract migrations can separate adding compatible structures from later removal, but require explicit sequencing and cleanup. Feature flags separate deployment from exposure only when both enabled and disabled paths remain compatible and observable. Test flag changes, stale clients, and retries against mixed versions. A successful canary on one instance does not prove a destructive migration can be reversed safely.

Represent promotion as a state decision

A trustworthy evaluator distinguishes harmful evidence, incomplete evidence, insufficient samples, and eligible evidence. The decision function below has no deployment capability. It gives data-integrity failure priority and holds when telemetry is missing. In a real system, policy must define which measurements count as confirmed harm and how uncertainty is handled. Never use a passing return value as a substitute for separate authorization and exposure controls. Keep candidate identity, policy version, and evidence links with every decision so a later incident can reconstruct why expansion happened.

python A pure staged-decision example

def stage_decision(*, integrity_ok, telemetry_complete, enough_samples, regression):
    if not integrity_ok:
        return "stop: integrity"
    if not telemetry_complete or not enough_samples:
        return "hold: inconclusive"
    if regression:
        return "stop: regression"
    return "eligible: next review"

All inputs are pre-validated synthetic facts; the function neither queries production nor changes exposure. enough_samples must come from a justified analysis plan, and regression must reflect the chosen evidence standard rather than an arbitrary dashboard color.

Verify recovery before it becomes urgent

Recovery can mean stopping exposure, disabling a feature, restoring compatible configuration, applying a forward fix, or using a validated rollback. Binary rollback alone may not reverse database writes or external side effects. Practice the intended recovery in an isolated environment with representative state and check how long detection and recovery take. Preserve the evidence window before cleanup. After a stage stops, classify whether the candidate, environment, telemetry, or policy failed. Feed the result into pre-release tests and the next analysis plan rather than merely retrying the same rollout until it appears green.

Common errors and fixes

A timer expands exposure despite missing metrics

Why it happens: Elapsed time is treated as sufficient evidence.
How to fix it: Hold on incomplete or insufficient observations.
Prevent it next time: Test analysis failure and sparse-traffic states.

Rollback cannot read the migrated schema

Why it happens: Mixed-version compatibility was not verified.
How to fix it: Use a reviewed compatible or forward recovery path.
Prevent it next time: Design migration and recovery together.

A small canary affects critical operations

Why it happens: Blast radius is defined only as a traffic percentage.
How to fix it: Choose cohorts by business and data risk.
Prevent it next time: Review irreversible operations before exposure.

Check your understanding

1. A low-traffic stage sees no errors in ten requests. Is promotion justified?

Show worked solution

Not by that observation alone. Compare it with the planned sample requirement, risk, and uncertainty. Hold if evidence is insufficient, and use other validated signals without pretending zero observed failures proves a low true error rate.

2. A flag disables the new UI but new database writes remain. What must recovery verify?