Investigate strong offline scores and weak operational outcomes

Understand the likely causes

• Possible explanations include a leaked or unrepresentative evaluation, serving-time preprocessing mismatch, changed input units, a new population, revised target definition, delayed or selectively observed labels, uncalibrated scores, or a threshold that does not fit capacity and costs.
• A healthy predictor can also be embedded in a changed human workflow. Preserve model and decision-policy versions before attributing every problem to the estimator.

1. Define the observed deterioration precisely

Name the outcome, time window, affected population, unit of aggregation, and cost or metric that changed. Separate application errors and latency from predictive loss, ranking, calibration, and human-review workload. Confirm the model, preprocessing, threshold, and label-definition versions used in each comparison. Do not compare one offline aggregate with another live metric and call the gap a model failure. Keep sensitive observations restricted and use a synthetic fixture for initial diagnosis. Preserve the original selection protocol so an apparently strong offline number can be audited for leakage and mismatched deployment assumptions.

2. Check serving parity and data quality first

Trace a known approved record through the frozen training-side and inference-side transformations. Inspect units, field order, missingness, category mappings, finite values, target inversion, and stale artifacts. A unit conversion bug can imitate drift and would contaminate a freshly trained replacement too. Compare input summaries by relevant slice, not only a global mean. A distribution difference is evidence to investigate, not proof that the prediction relationship or decision quality changed. Conversely, a small marginal shift does not rule out a harmful conditional change. Keep the inference contract and its regression fixtures visible while narrowing the hypothesis.

3. Audit label maturity and outcome selection

Determine when an outcome becomes available and which cases ever receive a label. The local program includes only outcomes mature by a declared cutoff and prints their count beside the error. It does not treat later supplied values as already observable. Compare equivalent maturity windows across groups and versions. If decisions affect who gets reviewed or which events become visible, observed-case error may be selected and not representative. Missing evidence should be reported as missing, not as success. Review corrections and censoring policies before drawing a trend line from changing coverage.

import numpy as np
label_available = np.array([2, 5, 5, 8])
error = np.array([0.2, 0.4, 0.6, 9.0])
cutoff = 5
mature = label_available <= cutoff
print("mature outcomes", int(mature.sum()))
print(f"fixture MAE={error[mature].mean():.2f}")

mature outcomes 3
fixture MAE=0.40

4. Choose and verify the justified correction

The remedy may be a data fix, restored preprocessing artifact, revised monitoring definition, calibrated decision policy, workflow change, rollback, or a properly evaluated new fit. Do not select it solely because a drift test fired. Review costs, capacity, subgroup effects, and safe fallback behavior under the actual task. Use permitted validation for any new threshold or model choice and protect independent assessment. Record the candidate, owner, approval evidence, and rollback condition. The course performs no deployment or live experiment; its examples support diagnosis of the contracts that an authorized operational process would verify.

Keep it from recurring

• Assign monitoring and release owners, version labels and decision policies, preserve training/inference parity checks, and define alert investigation before automatic action. Retraining should pass the same leakage, validation, fairness, and artifact controls as the original procedure.