MLOps / Observation, recovery, and lifecycle decisions / Lesson 14

Drift, label delay, and performance evidence

Distinguish distribution change, broken measurements, and observed predictive outcomes.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Define a comparison window, reference population, and drift statistic.
Join outcomes by identity under an explicit maturity rule.
Avoid equating drift alerts with model failure or automatic retraining.

A distribution difference needs a reference question

Input distributions can change because of seasonality, eligibility, collection defects, product behavior, or a new population. Choose reference and current windows with comparable units and measurement rules. A schema or unit defect is not simply ordinary population drift. Different statistics detect different changes, and aggregate summaries can miss conditional or subgroup changes. State which feature or score is compared, how missingness and categories are handled, sample counts, and why the change matters to the task. A distance alone does not identify its cause or tell an owner which action is appropriate.

A distance is not a performance measurement

The first fixture computes total variation between two tiny categorical distributions. The value 0.60 quantifies the declared difference, but it is not a p-value, an error-rate increase, or a universal alarm threshold. Sample variability, repeated monitoring, correlated observations, and many simultaneous features affect interpretation. A model can maintain its task performance after a visible input change, and performance can deteriorate through changed input-target relationships without an obvious marginal feature shift. Use drift as diagnostic evidence alongside data quality, coverage, labels, and the model’s intended population.

python Deterministic offline standard-library fixture

reference = {"a":8,"b":2}
current = {"a":2,"b":8}
keys = sorted(reference.keys() | current.keys())
p = {key:reference.get(key,0)/sum(reference.values()) for key in keys}
q = {key:current.get(key,0)/sum(current.values()) for key in keys}
total_variation = 0.5*sum(abs(p[key]-q[key]) for key in keys)
print(f"categorical total variation={total_variation:.2f}")
print("reference count", sum(reference.values()))
print("current count", sum(current.values()))

Expected output

categorical total variation=0.60
reference count 10
current count 10

A fixed categorical distance measures this distribution difference; it does not establish significance, performance loss, or the cause of the change.

Outcome monitoring waits for legitimate labels

Predictions and outcomes must join by stable identity with the correct target horizon, label definition, revision policy, and maturity time. The second fixture evaluates only labels available at its assessment time and reports coverage. It never treats an unavailable outcome as negative. Faster-arriving labels can be systematically different from delayed ones, so the mature subset may be biased. Keep cohort-based reporting that can be revised as labels mature, and distinguish prediction time from assessment time. A metric for recently mature outcomes should not be compared blindly with a fully settled historical cohort.

python Deterministic offline standard-library fixture

predictions = {"a":0.8,"b":0.7,"c":0.2}
labels = {"a":{"value":1,"available":4}, "b":{"value":0,"available":5},
          "c":{"value":1,"available":9}}
assessment_time = 6
mature = [key for key in predictions if labels[key]["available"] <= assessment_time]
correct = sum((predictions[key] >= 0.5) == bool(labels[key]["value"]) for key in mature)
print("mature ids", mature)
print(f"mature accuracy={correct/len(mature):.2f}")
print(f"label coverage={len(mature)/len(predictions):.2f}")

Expected output

mature ids ['a', 'b']
mature accuracy=0.50
label coverage=0.67

Only available targets contribute; the mature subset can differ systematically from delayed cases, so its accuracy is not automatically representative.

Investigate the first changed boundary

Check feature schema, source freshness, eligibility, preprocessing, selected version, fallback behavior, and outcome collection before blaming the model’s learned relationship. Compare important slices and counts under compatible definitions. A target-policy change can invalidate comparisons without any change to inference code. If ground truth is affected by the system’s own actions, record the selection and feedback mechanism rather than assuming the observed labels form an unbiased sample. A drift detector cannot infer the missing counterfactual outcomes. Label-quality review and domain knowledge remain part of interpreting the evidence.

Choose a response proportional to supported evidence

A data defect may require repairing input production and reassessing affected decisions; a legitimate population change may justify new evaluation; insufficient labels may require waiting or a reviewed fallback. Retraining is one possible controlled response, not an automatic cure. Preserve alert definition, reference identity, current counts, mature-label coverage, investigation, and decision. Check any revised candidate under the normal acceptance contract. The supplied programs calculate fixed categorical probabilities and a small labeled-cohort metric only; no monitoring platform, statistical detector service, external labels, model training, or real-world drift investigation is executed.

Common errors and fixes

A drift distance is reported as model accuracy loss

Why it happens: A feature statistic was substituted for outcome evidence.
How to fix it: Measure aligned mature-label performance and investigate the change separately.
Prevent it next time: Label each statistic by the property it actually measures.

Recent missing labels become negative outcomes

Why it happens: Delay was interpreted as a target value.
How to fix it: Use the declared maturity and availability policy.
Prevent it next time: Report label coverage and test late-arriving outcomes.

Retraining starts immediately after every drift alert

Why it happens: No cause or candidate acceptance was assessed.
How to fix it: Investigate data, context, and performance before choosing a response.
Prevent it next time: Keep alert, investigation, retraining, and promotion as separate events.

Check your understanding

1. What can the value 0.60 in the categorical fixture establish?

Show worked solution

It establishes the total-variation distance between the two supplied empirical probability tables. It does not establish statistical significance, why the distribution changed, whether a model failed, or which response is best. Counts and the comparison contract are needed even to interpret the distance responsibly.

2. Why should mature accuracy 0.50 be reported with coverage 0.67?