Machine Learning / Evidence and operational responsibility / Lesson 18

Monitoring, drift, and retraining

Separate data and outcome signals, investigate changes under stable definitions, and govern retraining through a verifiable release process.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Deployment changes the evidence stream
Compute a bounded input-shift summary
Retraining is a controlled learning and release decision

Deployment changes the evidence stream

After release, inputs, behavior, and target availability can differ from the development dataset. Monitor schema violations, missingness, unknown categories, score distributions, decision volume, latency, and eventual outcomes under defined windows. Preserve model, preprocessing, and policy version identifiers with safe observation metadata. Do not log sensitive raw features indiscriminately. Delayed or selectively observed labels can make apparent performance unrepresentative, especially when the system’s decisions influence which outcomes become visible. A dashboard should distinguish measured outcomes, proxy signals, and missing evidence rather than filling every gap with a reassuring aggregate.

Drift is not one interchangeable diagnosis

A feature distribution can change without the prediction relationship changing, and outcome or conditional relationships can change without a large shift in a selected marginal feature. Distinguish the distribution being compared and the assumptions behind names such as covariate or concept shift. A statistical difference does not automatically mean operational harm, while no detected difference does not prove safety. Seasonal patterns, collection bugs, unit changes, and policy updates can create similar alerts. Compare like-for-like populations, windows, and schemas before drawing conclusions. Repeated testing across many features also needs an alert policy that accounts for noise and investigation capacity.

Compute a bounded input-shift summary

The first example compares two fixed histograms using an explicitly chosen bin boundary and total-variation distance. The baseline and current samples put different mass on the two bins, giving a simple checkable shift signal. This is not a statistical significance test, a performance estimate, or proof of concept drift. Bin choice and sample size affect the summary, and the code requires matching normalization. Real monitoring needs an appropriate reference window, support counts, data-quality checks, and a response owner. Keep diagnostic calculations distinct from automatic decisions to retrain or disable a model.

python Fixed synthetic offline example

import numpy as np
reference = np.array([0.0, 0.0, 1.0, 1.0])
current = np.array([1.0, 1.0, 1.0, 1.0])
bins = np.array([-0.5, 0.5, 1.5])
p = np.histogram(reference, bins=bins)[0] / len(reference)
q = np.histogram(current, bins=bins)[0] / len(current)
print(p.tolist(), q.tolist())
print(f"fixture total variation={0.5 * np.abs(p - q).sum():.2f}")

Expected output

[0.5, 0.5] [0.0, 1.0]
fixture total variation=0.50

The two fixed bins shift from equal mass to all mass in the second bin. This is a descriptive distribution signal only.

Inspect outcomes only when their labels mature

The second example evaluates only records whose outcome-availability time falls within a declared observation cutoff. An earlier prediction date alone does not make its target available. The code prints the included count and mean error so delayed rows cannot silently be treated as correct or missing at random. A real system also needs a policy for corrections, censoring, and selective feedback. Compare the same maturity definition across model versions and groups. If an intervention prevents an outcome from being observed, ordinary supervised error on observed cases may not answer the desired counterfactual or policy question.

python Fixed synthetic offline example

import numpy as np
label_available = np.array([2, 4, 7, 8])
absolute_error = np.array([1.0, 2.0, 9.0, 9.0])
cutoff = 5
mature = label_available <= cutoff
print("included outcomes", int(mature.sum()))
print(f"mature fixture MAE={absolute_error[mature].mean():.2f}")

Expected output

included outcomes 2
mature fixture MAE=1.50

Only the first two outcomes are available under the cutoff. The later supplied errors are not used as if their labels had already matured.

Retraining is a controlled learning and release decision

Investigate the cause of an alert before retraining: a broken unit conversion can contaminate a newly fitted model too. Define retraining data eligibility, split policy, labels, baselines, evaluation gates, subgroup checks, and rollback. A newer fit should not replace a known version merely because it is newer or sees more rows. Preserve reproducibility records and compare the full procedure in an isolated evaluation before approval. Shadow or staged observation can provide additional evidence under a reviewed plan, but it does not eliminate confounding or guarantee future quality. Assign an owner for monitoring, incident response, and decisions to pause, revert, or retain the model.

Common errors and fixes

Every distribution alert triggers automatic retraining

Why it happens: Change detection was confused with a diagnosed performance problem.
How to fix it: Investigate data quality and operational meaning first.
Prevent it next time: Assign alert owners and release gates.

Recent performance looks perfect before labels arrive

Why it happens: Unavailable outcomes were ignored or counted incorrectly.
How to fix it: Use an explicit maturity and selection policy.
Prevent it next time: Report coverage and delays beside every metric.

A newer artifact replaces the model without comparison

Why it happens: Freshness was mistaken for validated improvement.
How to fix it: Evaluate the full candidate procedure and preserve rollback.
Prevent it next time: Version data, code, transforms, and decision policy.

Check your understanding

1. Why can a large input shift occur without immediate predictive harm?

Show worked solution

The shifted feature may be irrelevant, the relationship can remain stable, or the changed region may not alter the decision. Conversely, a subtle conditional change can matter greatly. Treat input monitoring as a signal for investigation and obtain appropriate mature outcome evidence rather than equating shift with failure.

2. What should happen when a unit-conversion bug is discovered in live features?