MLOps / Pipelines and controlled delivery / Lesson 12

Rollouts, shadowing, and rollback

Plan bounded exposure and recovery around the full prediction contract, not only the model file.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish shadow evaluation, canary exposure, and controlled experiments.
Assign exposure at a meaningful stable unit.
Verify rollback compatibility and evidence before relying on it.

Exposure modes answer different questions

Shadow processing computes a candidate output alongside the active path without using that candidate to make the user’s decision. It can reveal interface and resource behavior under representative inputs, but does not directly measure outcomes caused by candidate decisions. A canary exposes a bounded portion of actual operations to a change and observes specified guardrails. A controlled experiment requires a justified allocation and analysis design to estimate an effect. These techniques overlap operationally but do not provide identical evidence. State which output influences the decision, which population is exposed, and what can be inferred from each observation.

Allocation needs a unit and a stability policy

The first fixture uses a deterministic hash bucket for a synthetic entity id. Stable assignment can avoid switching one entity between paths across requests, but the correct unit may be a user, household, session, site, or another dependency group. Hashing does not automatically establish an unbiased experiment, balanced small samples, or independence. Record the allocation rule, eligible population, version, and exposure level. If treatment can influence future eligibility or interact across units, the analysis must account for those effects. Three sample ids demonstrate rule invariants only and do not represent live traffic.

python Deterministic offline standard-library fixture

import hashlib
def arm(entity_id, candidate_percent):
    bucket = int(hashlib.sha256(entity_id.encode()).hexdigest(),16) % 100
    return "candidate" if bucket < candidate_percent else "control"
ids = ["lab-a", "lab-b", "lab-c"]
first = [arm(key,10) for key in ids]
second = [arm(key,10) for key in ids]
print("stable assignments", first == second)
print("zero exposure", all(arm(key,0)=="control" for key in ids))
print("full exposure", all(arm(key,100)=="candidate" for key in ids))

Expected output

stable assignments True
zero exposure True
full exposure True

Stable hashing demonstrates a deterministic allocation rule only; three ids do not establish balanced traffic, randomized causal inference, or a real rollout.

Observe guardrails before increasing exposure

Define functional, operational, and model-related stop conditions, minimum observation requirements, label delays, and who can interpret ambiguous evidence. Fast availability and error signals can arrive before predictive outcomes mature. The second fixture applies a fixed synthetic error-rate stop rule while separately checking recovery prerequisites. Its threshold is a teaching choice, not a universal acceptable error rate. A quiet short observation window can reflect low traffic or delayed failures rather than success. Compare cohorts and measurement definitions carefully, and avoid increasing exposure solely because a dashboard has no visible alert yet.

python Deterministic offline standard-library fixture

observed = {"candidate_errors":3, "candidate_requests":20, "schema_compatible":True}
error_rate = observed["candidate_errors"]/observed["candidate_requests"]
rollback_ready = {"package_available":True, "schema_compatible":False}
stop_exposure = error_rate > 0.10
can_restore = all(rollback_ready.values())
print(f"candidate error rate={error_rate:.2f}")
print("stop candidate exposure", stop_exposure)
print("old package alone sufficient", can_restore)

Expected output

candidate error rate=0.15
stop candidate exposure True
old package alone sufficient False

The fixture separates a fixed stop rule from recovery prerequisites; thresholds are illustrative and no traffic or selection state is changed.

Rollback restores a compatible process

An earlier model package may depend on an earlier schema, feature vocabulary, runtime, or decision threshold. Repointing selection to that file cannot restore behavior if those dependencies changed incompatibly or were deleted. Keep recovery artifacts, configuration, consumer compatibility, and a known-input check under the supported recovery objective. Distinguish stopping new exposure from handling in-flight work, already-issued decisions, and delayed outcomes. Some effects cannot be undone by a model switch, so the response plan needs to state those limits. Test the recovery path in a controlled supported environment before assuming it is available.

Preserve the rollout decision and its limits

Record exact candidate and control identities, allocation, timing, eligibility, observed metrics and counts, thresholds, decision, and consumer versions. A stopped rollout is useful evidence about a specific change and context, not proof that every future candidate with the same architecture will fail. If the comparison informed further tuning, record that selection use before claiming a new independent assessment. These programs calculate hash-allocation invariants, a fixed rate, and compatibility Booleans entirely in memory. They do not shadow requests, run an experiment, change traffic, deploy an artifact, or execute a rollback.

Common errors and fixes

Shadow agreement is presented as candidate decision impact

Why it happens: Candidate outputs did not influence the observed decisions.
How to fix it: State shadow evidence limits and use an appropriate effect-evaluation design.
Prevent it next time: Record which path actually controlled each decision.

A small canary has no alerts and is expanded immediately

Why it happens: Traffic count, label maturity, and observation power were not considered.
How to fix it: Apply declared evidence and guardrail requirements.
Prevent it next time: Test low-traffic and delayed-outcome cases in the decision policy.

Rollback points to an incompatible old package

Why it happens: Dependencies changed while only model retention was planned.
How to fix it: Verify schema, transforms, runtime, and consumer recovery together.
Prevent it next time: Maintain a controlled full-process recovery fixture.

Check your understanding

1. Why does a shadow candidate not automatically reveal the outcome of using its different decisions?

Show worked solution

The active path still determined the actual action, so the observed outcome generally belongs to that action and its context. The candidate’s counterfactual outcome may differ. Shadowing can test computations and operational fit, while decision impact requires an appropriate design and assumptions.

2. The old package exists but expects a removed feature. Is rollback ready?