CI/CD / Integrity, coordination, and improvement / Lesson 17

Rollback, roll-forward, and recovery rehearsal

Choose recovery based on state compatibility and verified outcomes rather than a familiar button.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Compare package rollback, configuration reversal, and roll-forward.
Design a tabletop recovery record with explicit completion criteria.

Recover the user outcome

The purpose of recovery is restoring acceptable service while preserving required data and safety constraints. Returning to an older package is one possible action, not the definition of recovery. Parcel may fail because of a configuration change, an external carrier outage, or incompatible stored data. Reinstalling old code may help, do nothing, or worsen the situation. Identify the user impact, current adoption state, relevant timeline, and available compatible candidates before selecting an action through the incident owner’s process.

Separate reversible layers

Package rollback changes application bytes. Configuration reversal changes settings. Feature disablement changes exposure. Roll-forward introduces a corrective candidate. Data restoration changes authoritative state and has its own consistency and loss implications. These actions can be combined, but their order matters. A previous package may depend on a column already removed, and a restored database may omit external actions performed since the snapshot. State the recovery contract for each layer and avoid promising that a single pipeline button reverses every effect.

Retain the material needed to recover

A recovery candidate needs retained bytes, authenticated identity, compatible configuration, required evidence, and an owner who can authorize adoption. Keep the invocation and target observations that distinguish partial from complete rollout. The following record is a tabletop plan, not an execution script. It is intentionally explicit about state compatibility and verification. A real plan also needs environment-specific authority and operational details, but those belong to an approved runbook rather than a public lesson with guessed commands.

json A tabletop recovery decision

{"incident":"lab-incident-4","current":"candidate-r9","recovery_candidate":"candidate-r8","schema_compatibility":"review-required","user_check":"synthetic-label-journey","decision":"pending"}

No real incident, environment, or package is represented. The pending compatibility decision prevents treating retention alone as proof that an old candidate is safe to adopt.

Reject an incompatible old candidate

The complete local model evaluates a declared compatibility set. It accepts an older candidate only if the currently observed schema is supported and the package is retained. This is a narrow policy calculation, not a database check or recovery procedure. Real compatibility may involve data formats, queues, external APIs, configuration, and business invariants beyond a schema number. The model’s purpose is to show why an old package’s existence is insufficient and why recovery readiness must be maintained before the incident.

python Evaluate declared recovery compatibility

candidate = {"retained": True, "schemas": {3, 4}}
def recoverable(observed_schema):
    return candidate["retained"] and observed_schema in candidate["schemas"]
print(recoverable(4))
print(recoverable(5))

Expected output

True
False

The schema numbers are synthetic compatibility labels. This program reads no database and cannot prove real migration reversibility.

Rehearse evidence and decision boundaries

A tabletop rehearsal can walk through missing package, expired approval, partial rollout, telemetry outage, and incompatible schema without touching infrastructure. In a separately authorized disposable environment, a fuller recovery rehearsal can validate actual mechanics and timing. Keep those evidence levels distinct. Define completion as restored user behavior plus observed candidate and configuration identity, not merely a successful recovery command. After an incident, preserve a concise timeline, contributing conditions, and owned improvements; avoid reducing a systemic failure to the person who pressed the button.

Common errors and fixes

A retained old package is assumed safe to restore.

Why it happens: Availability was confused with state and configuration compatibility.
How to fix it: Evaluate current state and the candidate’s supported contracts.
Prevent it next time: Maintain compatibility evidence throughout the recovery horizon.

A successful rollback command closes the incident.

Why it happens: Requested action was substituted for restored user behavior.
How to fix it: Verify adoption identity, critical journeys, and required data invariants.
Prevent it next time: Define outcome-based completion criteria in the recovery plan.

A database restore ignores actions completed in external systems.

Why it happens: Local state recovery was treated as global business reversal.
How to fix it: Coordinate reconciliation and compensation through the incident owner.
Prevent it next time: Inventory external side effects and recovery consistency boundaries.

Check your understanding

1. Errors begin after a configuration update while package identity is unchanged. What should the first recovery hypothesis consider?

Show worked solution

Consider reversing the relevant configuration under the incident process after confirming the timeline and impact. A package rollback may be unrelated. Preserve both identities and verify the user outcome after any chosen action rather than assuming causation from one nearby event.

2. A tabletop exercise completes in five minutes. Can the team claim five-minute production recovery?