MLOps / Pipelines and controlled delivery / Lesson 11

Training-serving skew and feature time

Compare the actual historical and delivered feature representations, including when each value became usable.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish feature event time, publication time, and age.
Prevent retrospective corrections from entering historical examples improperly.
Diagnose representation mismatch before attributing loss to drift.

Skew compares paths that should agree

Training-serving skew occurs when the representation used to fit or assess a model differs from the one delivered under its intended inference contract. Causes include separate transformation implementations, changed units, default values, category mappings, and feature freshness. Distribution shift is another concept: the population can change even when both paths implement the same function correctly. Diagnose whether the same raw observation and permitted context produce the same representation before treating a disappointing metric as evidence that the world changed. Keep raw meaning, transform identity, and output coordinates visible in a parity fixture.

Historical reconstruction has two time questions

A feature record has an event or effective time and a time when it was published or available to the prediction process. A backfilled correction can describe an earlier event while being unknown at that earlier prediction. The first fixture filters both time dimensions and a maximum age. An event-time-only lookup chooses a value that arrives too late. A feature-store product can implement particular timestamp and deduplication semantics, so verify its documented behavior and configuration rather than assuming that the phrase point-in-time automatically excludes every late correction. Availability metadata must itself represent the actual usable time faithfully.

python Deterministic offline standard-library fixture

history = [{"event":1, "available":1, "value":10},
           {"event":3, "available":3, "value":30},
           {"event":4, "available":7, "value":99}]
prediction_time, max_age = 5, 3
by_event = [r for r in history if prediction_time-max_age <= r["event"] <= prediction_time]
available = [r for r in by_event if r["available"] <= prediction_time]
correct = max(available, key=lambda r:r["event"])
naive = max(by_event, key=lambda r:r["event"])
print("available historical value", correct["value"])
print("event-time-only value", naive["value"])
print("feature age", prediction_time-correct["event"])

Expected output

available historical value 30
event-time-only value 99
feature age 2

The later-event record was not available at prediction time; this explicit two-time policy also applies a declared maximum age.

Freshness is a task-specific validity condition

A maximum age can exclude a stale value, but a missing result then needs an explicit policy: reject, defer, use a defined fallback, or apply a documented missingness representation. Do not silently substitute the latest current value into an old example. Likewise, a recent event timestamp does not prove recent publication or successful materialization at the consumer. Preserve retrieval time, selected feature version, source status, and any fallback indicator when needed for diagnosis. A universal age limit is inappropriate because different measurements change at different rates and support different decision horizons.

Compare fitted state and implementation together

The second fixture uses the same raw number and mean but different scales, producing different model inputs. The mismatch could come from a stale package, an independently recomputed statistic, or configuration rather than the feature source. Compare names, units, types, category levels, missingness, transform version, fitted parameters, and selected historical values before final scores. A shared transformation definition can reduce duplication, but its dependencies and execution modes still need verification. Test boundary inputs and unsupported cases as well as a normal row, since two paths can agree on easy examples and diverge elsewhere.

python Deterministic offline standard-library fixture

def transform(value, mean, scale):
    assert scale != 0
    return (value-mean)/scale
raw_value = 14.0
training = transform(raw_value, mean=10.0, scale=2.0)
serving = transform(raw_value, mean=10.0, scale=4.0)
print("training representation", training)
print("serving representation", serving)
print("same", training == serving)

Expected output

training representation 2.0
serving representation 1.0
same False

Identical raw input and feature name do not establish parity when stored transformation parameters differ.

Repair the earliest mismatched boundary

Locate whether the first discrepancy appears in source selection, temporal eligibility, normalization, encoding, batching, or model inference. Repair that boundary and retain a regression fixture that would fail under the original defect. Rebuild affected training or assessment artifacts if the mismatch invalidates prior evidence; changing only the current serving path does not repair a leaked historical evaluation. These programs perform fixed list selection and scalar transformation checks only. They do not query a feature store, materialize features, read a production stream, or validate a platform’s point-in-time implementation.

Common errors and fixes

Historical training uses a corrected value published later

Why it happens: Only event time constrained feature retrieval.
How to fix it: Apply the declared availability boundary as well as effective time.
Prevent it next time: Test a backfilled correction with distinct event and publication times.

A fresh-looking feature is unavailable to consumers

Why it happens: Event recency was mistaken for successful publication or materialization.
How to fix it: Track actual usable time and consumer retrieval evidence.
Prevent it next time: Include stale, delayed, missing, and fallback feature cases.

A raw-input match is treated as transform parity

Why it happens: Stored statistics, encoding, or defaults differ.
How to fix it: Compare full representations and fitted state before scores.
Prevent it next time: Keep known-input parity tests for both normal and boundary cases.

Check your understanding

1. Why must the value 99 be excluded at prediction time five?

Show worked solution

Its event time is four, but it is not available until seven. Under the declared usable-information policy it was unknown at five. The earlier value thirty is available and within the age window. Choosing 99 would reconstruct knowledge the predictor did not have.

2. The serving path is repaired after a historical leakage bug. Can the previous validation report remain authoritative?