Trace disagreement between offline and delivered predictions

Understand the likely causes

• Possible causes include changed feature order or units, a different fitted imputation or scaling state, category or label mappings, inference mode, historical feature availability, or a consumer retaining an older package. These differ from a new population reaching two otherwise equivalent paths.

1. Freeze the two complete functions being compared

Record the raw input with units and stable identity, feature and label definitions, decision timestamp, package identity, selected version, transform state, output interpretation, and relevant execution mode. Resolve mutable aliases to concrete versions in both records. Choose a tiny synthetic observation that exposes the suspected mismatch, rather than copying private request payloads into a public reproduction. State whether the expected agreement is exact scores, values within a justified tolerance, class labels, or downstream decisions. A vaguely similar model name and a rounded final score are not enough to define the comparison.

2. Compare representations before final predictions

Inspect parsing, selected feature names and order, units, missingness treatment, vocabulary, fitted transformation parameters, and the resulting numeric coordinates. The local fixture preserves the raw row and weights while changing one stored scale. The reference score is one and the loaded score is zero, so retraining is unnecessary to explain this discrepancy. Compare actual loaded state rather than only configuration files that an operator intended to load. If representations agree, continue to model parameters, inference mode, output mapping, and threshold logic. Preserve the first differing boundary as the diagnostic finding.

expected = {"features":["length","width"], "mean":[10.0,20.0],
            "scale":[2.0,4.0], "weights":[2.0,-1.0]}
loaded = dict(expected, scale=[4.0,4.0])
row = {"length":12.0,"width":24.0}
def score(package):
    values = [row[name] for name in package["features"]]
    z = [(x-m)/s for x,m,s in zip(values,package["mean"],package["scale"])]
    return sum(w*x for w,x in zip(package["weights"],z))
print("different fields", [key for key in expected if expected[key] != loaded[key]])
print("reference score", score(expected))
print("loaded score", score(loaded))

different fields ['scale']
reference score 1.0
loaded score 0.0

3. Reconstruct the information available at prediction time

For a historical example, distinguish feature event time from publication or usable time. A later correction may describe an old event without having been available to the original predictor. Apply the declared age and availability policy, and inspect missing or fallback features. Compare singleton and batch context under the intended model contract; a training-mode normalization layer or wrong attention axis can introduce companion dependence. Read the actual framework or feature-store semantics before assuming a generic flag proves correctness. The guide does not execute such a platform; build a supported, narrow runtime reproduction if that boundary is implicated.

4. Repair the first mismatch and reassess affected evidence

Restore the correct package, transform, temporal selection, or mapping under the supported operational procedure. Keep a known-input regression fixture and repeat both ordinary and boundary cases, including invalid inputs and threshold-near decisions. If offline evaluation consumed an unavailable historical value or another mismatched function, rebuild that evaluation rather than claiming the old score now validates the repaired path. Distinguish restored interface behavior from predictive quality that requires mature outcomes. Record the affected artifact identities and any pending label assessment so the scope of the repair remains reviewable.

Keep it from recurring

• Package learned transforms and ordered semantics with model identity, pin selection at a defined boundary, and preserve sentinel inputs that expose coordinate, unit, category, and timing changes. Review compatibility and any invalidated assessment when upstream contracts change.