MLOps / Prediction and evidence foundations / Lesson 3

Splits, availability, and leakage boundaries

Make evaluation manifests reproduce the information actually available to training and prediction.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate event time, feature availability, and label maturity.
Protect training, selection, calibration, and final-assessment boundaries.
Audit entity overlap under the intended deployment task.

A split simulates a particular future use

Random rows, unseen entities, later time periods, and new sites are different evaluation settings. Choose the setting from the prediction contract rather than whichever split produces the highest score. Stable row ids alone do not identify every dependency: several rows can describe the same device, household, event, or source. Preserve the relevant group and time metadata outside the predictive features. A manifest should identify the exact records and separation rule used in each cohort, so a later pipeline run can verify that the evaluation still answers the same operational question.

Availability time differs from an event timestamp

An event can occur before a training cutoff while its target becomes known afterward. Likewise, a feature may describe an earlier event but be published or corrected later. The first fixture uses a strict label-availability cutoff for training and prediction time for the future cohort, excluding one boundary record. It demonstrates a stated policy rather than a universal gap length. Set a gap or purging rule from target horizons, overlapping windows, and actual information dependencies. Evaluation of future outcomes must wait for the declared label-maturity rule; a missing delayed label is not automatically a negative outcome.

python Deterministic offline standard-library fixture

rows = [{"id":"a", "prediction_time":1, "label_available":3},
        {"id":"b", "prediction_time":2, "label_available":4},
        {"id":"c", "prediction_time":4, "label_available":6},
        {"id":"d", "prediction_time":5, "label_available":7}]
cutoff = 5
train = [r["id"] for r in rows if r["label_available"] < cutoff]
future = [r["id"] for r in rows if r["prediction_time"] >= cutoff]
excluded = [r["id"] for r in rows if r["id"] not in train + future]
print("train", train)
print("future cohort", future)
print("boundary exclusion", excluded)

Expected output

train ['a', 'b']
future cohort ['d']
boundary exclusion ['c']

The strict training cutoff uses label availability, while future evaluation uses prediction time; the future cohort can be scored only after its labels mature.

Learned preprocessing crosses the boundary too

Imputation values, scaling parameters, vocabularies, feature selection, calibration mappings, and thresholds all consume information. Fit each step only on the cohort its role permits, and keep learned preprocessing inside the relevant cross-validation fold. Wrapping an already globally transformed array in a pipeline does not remove the earlier leakage. Record which data informed architecture and threshold changes, including manual revisions prompted by inspected errors. A final test repeatedly used to guide those revisions has become selection evidence even if a folder still calls it test. Preserve an honest assessment plan when repairing that boundary.

Audit the dependency unit rather than only row ids

The second fixture has distinct row ids on both sides but one shared group. Under its declared unseen-group task, that overlap violates the intended separation. A known-entity prediction task can legitimately use another contract, so shared groups are not universally forbidden. Review whether the group identifier captures the actual dependence, and inspect unequal group sizes and slice counts. A split can be disjoint by a convenient column yet still share a household, site, repeated text, or collection artifact that changes the generalization question. No simple overlap assertion establishes every form of independence.

python Deterministic offline standard-library fixture

train = [{"id":"a1", "group":"x"}, {"id":"b1", "group":"y"}]
evaluation = [{"id":"a2", "group":"x"}, {"id":"c1", "group":"z"}]
train_groups = {r["group"] for r in train}
evaluation_groups = {r["group"] for r in evaluation}
overlap = sorted(train_groups & evaluation_groups)
print("shared groups", overlap)
print("valid for declared unseen-group task", not overlap)

Expected output

shared groups ['x']
valid for declared unseen-group task False

Different row ids do not prevent group overlap; this diagnostic detects one violation of an explicitly unseen-group evaluation contract.

Version the evaluation boundary as an artifact

Store cohort identities, availability rules, exclusions, target definitions, and preprocessing fit boundaries with the experiment record. Check that regenerated manifests match the intended versions, and explain any changed eligibility or late-arriving correction. Use the same baseline and candidate evaluation boundary for a fair comparison. The synthetic programs here audit small fixed records and do not train or evaluate an external model. They establish the stated manifest arithmetic only; they do not certify absence of all leakage, future representativeness, subgroup acceptability, or a useful deployed decision process.

Common errors and fixes

A historical row leaks a future label into training

Why it happens: Its event date preceded the cutoff but label availability did not.
How to fix it: Filter by the declared information-availability boundary.
Prevent it next time: Preserve event, feature-publication, and label-maturity times separately.

Disjoint row ids are reported as independent entities

Why it happens: Related rows share a group across cohorts.
How to fix it: Split and audit the dependency unit matching the task.
Prevent it next time: Keep known overlap fixtures and inspect group construction.

A pipeline is assumed to repair leaked preprocessing

Why it happens: The data was transformed using all cohorts before pipeline execution.
How to fix it: Rebuild learned transforms inside the permitted training boundary.
Prevent it next time: Trace every fitted statistic to the records it consumed.

Check your understanding

1. A prediction is made on day four, its label is available on day six, and training freezes before day five. Can that labeled row train this run?

Show worked solution

Under the strict policy shown, no. The label is not available before the cutoff even though the prediction event already occurred. Exclude it from this training cohort and record the reason. A later training snapshot can include it if its own availability and eligibility rules permit.

2. Why might a new-device evaluation require more than checking distinct observation ids?