Machine Learning / Prediction and data foundations / Lesson 4

Preprocessing without leakage

Learn transformations from training data, preserve the feature schema, and apply the identical fitted policy to future inputs.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

A fitted transformation is part of the model
Fit scale on training rows only
Encode categories under a stable vocabulary

A fitted transformation is part of the model

Scaling, imputation, category vocabularies, feature selection, and dimensionality reduction can all learn from data. They therefore belong inside the training boundary and every cross-validation training fold. Applying a transform to held-out inputs is legitimate; fitting its statistics from those inputs under an ordinary inductive evaluation is not. Keep a fitted artifact that records the schema and learned values, then reuse it at inference. A pipeline can encode this discipline in a library, but the name pipeline does not prove that an earlier global transformation was safe. Trace where every learned statistic originated.

Choose scaling from meaning and algorithm sensitivity

Subtracting a training mean and dividing by a training scale changes a feature’s numeric representation, not its underlying information. Distance-based and penalized linear methods can be sensitive to relative scales, while a threshold tree has different behavior. Standardization does not make a distribution Gaussian or remove outliers. Decide how to handle a zero training scale, and monitor a feature that was constant during fitting but later varies. Keep target scaling separate and invert it before reporting losses in original units where required. Scaling choices should follow the algorithm and measurement contract rather than a universal instruction to normalize everything.

Fit scale on training rows only

The first example fits column means and standard deviations on a fixed training array, replaces a zero denominator with the declared value one, and transforms one separate row. The second column is constant in this fixture, so its unchanged future value transforms to zero. This is an explicit denominator policy, not proof that every constant feature should always be retained. NumPy’s population-style standard deviation is used as coded; another implementation may use a different convention. Serialize the exact policy and feature order so deployment does not silently recompute statistics from each incoming batch or swap columns.

python Fixed synthetic offline example

import numpy as np
train = np.array([[1.0, 10.0], [3.0, 10.0], [5.0, 10.0]])
future = np.array([[7.0, 10.0]])
mean = train.mean(axis=0)
scale = train.std(axis=0)
scale = np.where(scale == 0, 1.0, scale)
transformed = (future - mean) / scale
print(mean.tolist())
print(np.round(transformed, 3).tolist())

Expected output

[3.0, 10.0]
[[2.449, 0.0]]

Only training rows determine the mean and denominator. The constant second feature uses the declared denominator one.

Impute with a justified learned value

The second example learns medians from training observations and uses them for missing evaluation entries. The unusually large observed evaluation value is not included in those medians and is not replaced merely for being large. An all-missing training column needs a defined fallback or rejection rather than a silent nonfinite statistic. Consider a missingness indicator when the absence itself carries information, while checking that its meaning is stable in deployment. A simple median is a useful baseline, not a universal model of missingness. Do not fill a future-dependent field with information that would only become available after the prediction.

python Fixed synthetic offline example

import numpy as np
train = np.array([[1.0, np.nan], [3.0, 4.0], [np.nan, 6.0]])
future = np.array([[np.nan, 100.0], [5.0, np.nan]])
medians = np.nanmedian(train, axis=0)
assert np.isfinite(medians).all()
filled = np.where(np.isnan(future), medians, future)
print(medians.tolist())
print(filled.tolist())

Expected output

[2.0, 5.0]
[[2.0, 100.0], [5.0, 5.0]]

Training medians are two and five. The observed hundred remains observed; only missing entries use the learned values.

Encode categories under a stable vocabulary

A category code such as zero, one, or two can accidentally imply an ordering or distance the domain does not possess. One-hot representations separate known categories, but the training vocabulary and unknown-category policy must be preserved. A category absent during fitting should not shift every later column. Define whether unknown values are rejected, mapped to a reserved indicator, or represented under a documented fallback. Check types, order, units, finite values, and dimension at inference. Keep preprocessing and estimator artifacts together; a correctly fitted model receiving a different representation is effectively being asked a different question.

Common errors and fixes

Validation influences imputation values

Why it happens: Statistics were fitted on all rows.
How to fix it: Fit each learned transform on training data only.
Prevent it next time: Keep preprocessing inside the selection pipeline.

Inference changes when batch composition changes

Why it happens: Scaling is refitted on each incoming batch.
How to fix it: Reuse the recorded training transformation.
Prevent it next time: Test one record alone and within a batch.

Unknown categories shift feature columns

Why it happens: Vocabulary and column order were rebuilt at inference.
How to fix it: Freeze the mapping and define unknown handling.
Prevent it next time: Validate encoded shape and semantics before prediction.

Check your understanding

1. Why can a scaler fitted without targets still leak validation information?

Show worked solution

Its means and scales can incorporate the held-out feature distribution, which the ordinary training procedure should not see. Fit it within each training fold and transform the held-out fold with those values. Being unsupervised does not remove the evaluation boundary.

2. Design a policy for a categorical value never observed in training.