MLOps / Observation, recovery, and lifecycle decisions / Lesson 17

Incident response and recovery evidence

Mitigate a concrete impact while preserving enough state to diagnose the prediction process and verify recovery.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate observed impact, hypotheses, mitigation, and causal explanation.
Preserve model, data, feature, and decision identities in a timeline.
Verify recovery against functional and task-specific evidence.

An ML incident can have several failing layers

A service outage, incompatible input schema, stale feature source, wrong label mapping, unexpectedly high fallback, or deteriorating mature-label performance can require an organized response. State the observed impact and eligible population before choosing a remedy. A technically reachable endpoint does not rule out an incident in its predictions. Assign coordination, diagnosis, mitigation, and communication responsibilities under the existing operational plan. Keep uncertainty visible: an initial suspected cause is a hypothesis, not a verified explanation, and urgent impact reduction need not wait for every analytical question to be settled.

Preserve evidence before changing the process

Record times, affected cohorts, resolved model and configuration versions, feature availability, recent changes, errors, fallback behavior, and relevant target maturity. The first fixture orders a tiny event timeline and explicitly refuses to infer causation from sequence alone. A change before failure is a useful investigation lead, but another dependency or collection defect can explain the same timing. Preserve a minimal synthetic reproduction when possible, without copying private payloads or credentials into the incident record. Destructive cleanup or uncontrolled retraining can erase the state needed to distinguish competing explanations.

python Deterministic offline standard-library fixture

events = [{"time":10,"kind":"selection","version":"model-2"},
          {"time":12,"kind":"schema-error","version":"model-2"},
          {"time":13,"kind":"fallback","version":"rule-1"}]
ordered = sorted(events, key=lambda event:event["time"])
for event in ordered:
    print(event["time"], event["kind"], event["version"])
print("temporal order alone proves cause", False)

Expected output

10 selection model-2
12 schema-error model-2
13 fallback rule-1
temporal order alone proves cause False

A synthetic timeline preserves observed sequence while explicitly avoiding a causal conclusion from timing alone.

Mitigation follows the supported fallback and recovery plan

Possible actions include stopping new candidate exposure, deferring unsupported work, applying a reviewed rule, or restoring a compatible previous process. Select within the actual authorization and system contract, checking consequences for in-flight and already-issued decisions. Model rollback alone may not repair changed feature semantics or an invalid historical dataset. Keep the mitigation scope and known limitations explicit. A response can reduce impact without establishing the root cause, and that distinction should remain in the record. This lesson describes decision structure only and performs no operational action against a live system.

Recovery requires behavior, not just a green process

The second fixture lists recovery prerequisites and fails until a known-input result matches the reference. A real check should cover the relevant input contract, selected version, feature validity, output meaning, fallback, delivery health, and task-specific outcome evidence as it becomes available. Delayed labels can prevent immediate confirmation of predictive recovery even after an interface defect is repaired. State what is verified now and what must be reassessed later. Passing one fixture cannot prove that every affected population or failure mode has recovered, so retain counts and relevant slices.

python Deterministic offline standard-library fixture

recovery = {"package_available":True, "input_compatible":True,
            "features_valid":True, "known_fixture_matches":False}
def gaps(checks):
    return [name for name,passed in checks.items() if not passed]
print("initial gaps", gaps(recovery))
recovery["known_fixture_matches"] = True
print("after synthetic repair", gaps(recovery))
print("declared checks pass", all(recovery.values()))

Expected output

initial gaps ['known_fixture_matches']
after synthetic repair []
declared checks pass True

The in-memory check transition models recovery evidence; it does not restore a service or establish untested quality properties.

Turn the investigation into testable improvements

After impact is controlled, compare evidence with hypotheses and document contributing conditions, detection gaps, recovery limitations, and concrete prevention work. Focus on the system and decision context rather than unsupported blame. Each follow-up should have an owner, a completion condition, and a check that would expose the failure again. Reassess invalidated model or evaluation artifacts under the repaired contract. These programs only sort literal events and evaluate in-memory recovery Booleans. They do not page responders, send messages, access incident systems, restore infrastructure, retrain a model, or claim a completed real incident investigation.

Common errors and fixes

The most recent model change is declared the cause immediately

Why it happens: Temporal correlation replaced a tested explanation.
How to fix it: Preserve alternatives and locate the first mismatched dependency.
Prevent it next time: Keep hypotheses distinct from verified findings in the timeline.

A healthy endpoint closes a prediction incident

Why it happens: Process availability was substituted for correct model behavior.
How to fix it: Check input, feature, version, output, fallback, and later outcome evidence.
Prevent it next time: Define recovery criteria for each affected layer.

Retraining destroys the failing state before diagnosis

Why it happens: Mitigation and investigation evidence were not separated.
How to fix it: Preserve relevant identities and a safe reproduction before replacing artifacts.
Prevent it next time: Test evidence retention and recovery ownership in a controlled exercise.

Check your understanding

1. Why does the event timeline not prove model-2 caused the schema error?

Show worked solution

It establishes order only. The producer schema, consumer configuration, feature source, or another simultaneous change could explain the error. Compare exact contracts and reproduce the first violation before making a causal claim, while using the supported mitigation to limit impact if needed.

2. An input defect is repaired but labels take seven days. What can recovery verification say now?