Machine Learning / Prediction and data foundations / Lesson 3

Splits, groups, time, and leakage

Estimate generalization under the intended deployment setting and prevent information from crossing evaluation boundaries.

About 5 min readContent reviewed 2026-10-26

On this page

Lesson goals

A split simulates the future use case
Keep a repeated entity on one side
Protect the final test from selection

A split simulates the future use case

Training data fits parameters, validation data supports choices, and a final held-out test estimates the selected procedure under its declared sampling assumptions. Randomly dividing rows is appropriate only when it reflects the intended independence and deployment setting. Predicting new events differs from predicting new observations of already known events. A model evaluated on familiar entities can look strong while failing on unseen ones. Choose the unit of separation before fitting or preprocessing. No split method repairs a mislabeled target, a future-only feature, or an unrepresentative population; it defines one part of the evidence, not the whole quality guarantee.

Leakage includes more than a copied target

Leakage occurs when information unavailable to the intended training or prediction process influences fitting or evaluation. Examples include target-derived columns, duplicated people across splits, preprocessing fitted on all rows, and repeatedly tuning decisions on the final test. An unsupervised transform can still leak evaluation distribution information if it is learned globally. Keep every learned step inside the appropriate training fold. An explicitly transductive task may have a different contract, but that exception must be justified rather than used to excuse an accidental boundary violation. Record the split procedure and the data identities it separates.

Keep a repeated entity on one side

The first example separates all observations of group c from groups a and b. It prints disjoint group identities and the row indices on each side. This is a manual illustration of a group-held-out contract, not an estimate of performance from six invented rows. If groups vary in size, decide whether evaluation weights rows or groups, and report that choice. A group may be a person, household, device, site, or another dependency unit. Incorrect grouping can still leave related records across the boundary, so inspect how identifiers were constructed rather than trusting a column called group automatically.

python Fixed synthetic offline example

import numpy as np
groups = np.array(["a", "a", "b", "b", "c", "c"])
train = np.flatnonzero(groups != "c")
test = np.flatnonzero(groups == "c")
print(train.tolist(), test.tolist())
print(sorted(set(groups[train]) & set(groups[test])))

Expected output

[0, 1, 2, 3] [4, 5]
[]

The observation indices differ and no group appears on both sides. This checks the declared grouping, not general independence.

Respect feature and label availability in time

The second example requires training labels to be available strictly before a cutoff, then evaluates later prediction times. One intervening row is excluded under that explicit policy. A chronological split can still leak if labels mature after the cutoff or windows overlap across the boundary. Choose a gap or purging policy from the actual horizon and data dependencies, not a universal fixed number. Do not shuffle future records into past training merely to improve class balance. Time-respecting evaluation should also reflect retraining cadence and whether the future population can differ from the historical one.

python Fixed synthetic offline example

import numpy as np
prediction_time = np.array([1, 2, 3, 4, 5, 6])
label_available = prediction_time + 1
cutoff = 5
train = label_available < cutoff
test = prediction_time >= cutoff
print("training times", prediction_time[train].tolist())
print("test times", prediction_time[test].tolist())

Expected output

training times [1, 2, 3]
test times [5, 6]

The declared strict cutoff excludes time four because its label becomes available at five. This gap follows the fixture policy, not a universal rule.

Protect the final test from selection

Use validation or cross-validation for model and threshold choices, then evaluate the chosen procedure on the final test without treating that result as another tuning signal. If the team makes changes because of test errors, the set is no longer an untouched final assessment. Keep split manifests or stable identifiers so later runs can reproduce the same boundary. A seed can reproduce one random partition under its software assumptions but does not prove representativeness or independence. Include simple leakage checks and document known dependencies. A credible modest score under a faithful split is more useful than an impressive number under an invalid one.

Common errors and fixes

Validation is strong only when rows are randomly shuffled

Why it happens: Related entities or future observations cross the split.
How to fix it: Use the group or temporal boundary that matches deployment.
Prevent it next time: Audit overlap by entity and availability time.

A scaler is fitted before cross-validation

Why it happens: Held-out folds influence learned preprocessing.
How to fix it: Fit preprocessing separately within each training fold.
Prevent it next time: Treat all learned transforms as model steps.

The final test guides repeated revisions

Why it happens: Selection has consumed the assessment set.
How to fix it: Use a separate selection protocol and obtain honest final evidence.
Prevent it next time: Record which data informed every decision.

Check your understanding

1. Why can two photos of the same object on opposite sides invalidate a new-object evaluation?

Show worked solution

The model can exploit object-specific information rather than generalize to a new object. Split by the object identity and keep its dependent observations together. Then decide how to weight objects and inspect whether other shared sources, such as site or time, also create relevant dependence.

2. A label is finalized seven days after each prediction. What must a historical training cutoff consider?