Machine Learning / Models and controlled selection / Lesson 12

Cross-validation and tuning

Separate training, selection, and final assessment while treating preprocessing, thresholds, and stopping as parts of the chosen procedure.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Cross-validation estimates a defined procedure
Fit a baseline independently in each fold
Report selection uncertainty and resource costs

Cross-validation estimates a defined procedure

Cross-validation repeatedly fits on training portions and evaluates on held-out portions under a chosen split design. It does not make dependent observations independent or automatically simulate future deployment. Choose group, temporal, or ordinary folds according to the task. Every learned step, including imputation, feature selection, calibration where applicable, and early stopping, needs its proper internal data boundary. Fold scores are evidence about the entire fitted procedure under those assumptions, not a permanent property of a model name. Preserve the split identities and explain whether the final deployed fit uses all permitted development data after selection.

Hyperparameters are selected, not learned in the same way

Model depth, regularization strength, preprocessing variants, and threshold policies can all be selection choices. Searching more alternatives increases the opportunity to select noise from validation results. Keep the search space, metric, budget, and stopping policy explicit before inspecting outcomes where practical. A random search or grid is a search strategy, not a guarantee of fair comparison. If evaluation data influences feature engineering or exclusions, that influence is part of selection too. Use nested evaluation or another honest held-out assessment when estimating the performance of a tuning procedure rather than reporting the best inner score as an unbiased final estimate.

Fit a baseline independently in each fold

The first example uses three fixed folds and recomputes a constant training mean separately for each. It prints the held-out MAE values and their unweighted fold mean. The observations are invented independent teaching records, not a time-series protocol or a real benchmark. If fold sizes differ, an unweighted mean of fold metrics can differ from a pooled per-observation metric; declare the aggregation. Do not compute one mean from all targets before looping. That shortcut would give held-out labels to every fitted baseline and invalidate the intended comparison even though no complex estimator is involved.

python Fixed synthetic offline example

import numpy as np
y = np.arange(6, dtype=float)
indices = np.arange(len(y))
folds = np.array_split(indices, 3)
losses = []
for test in folds:
    train = np.setdiff1d(indices, test)
    prediction = np.full(test.shape, y[train].mean())
    losses.append(float(np.abs(y[test] - prediction).mean()))
print(losses)
print(f"mean fixture MAE={np.mean(losses):.3f}")

Expected output

[3.0, 0.5, 3.0]
mean fixture MAE=2.167

Each fold learns its own mean from only its training indices. Equal fold sizes make the demonstrated unweighted mean easy to interpret.

Keep candidate choice separate from the final score

The second example selects a candidate using a supplied validation-loss table and only then reads that candidate’s separately supplied test loss. These are fictional bookkeeping values, not measurements from a fitted external model. Their purpose is to show that the test table must not choose the candidate. Tie handling, multiple metrics, and constraints such as latency should be declared, not improvised after seeing results. If the test result causes a new search, document that the assessment has been consumed and obtain an appropriate new evaluation plan. A variable named test does not preserve independence when the workflow repeatedly looks at it.

python Fixed synthetic offline example

import numpy as np
names = ["small", "medium", "large"]
validation_loss = np.array([0.8, 0.6, 0.7])
separate_test_loss = np.array([0.9, 0.75, 0.65])
chosen = int(np.argmin(validation_loss))
print("chosen", names[chosen])
print(f"reported fixture test loss={separate_test_loss[chosen]:.2f}")

Expected output

chosen medium
reported fixture test loss=0.75

Selection uses only the validation array. The fictional test values do not change the chosen medium candidate, even though another test entry is smaller.

Report selection uncertainty and resource costs

Report more than the winning mean: include fold or repeat variation, sample structure, tested alternatives, and relevant subgroup outcomes. Fold scores are typically correlated because training sets overlap, so their standard deviation divided by the square root of the number of folds is not automatically a valid confidence interval. Keep runtime, memory, and reproducibility metadata as part of model selection. Refitting the chosen pipeline on all permitted development data is a separate step whose artifact must preserve the selected settings. The final result should describe the procedure and evidence honestly rather than imply that a tiny score difference establishes a universal winner.

Common errors and fixes

The reported score is the best tuning-fold result

Why it happens: Selection and independent assessment were conflated.
How to fix it: Evaluate the selected procedure under an honest outer or held-out protocol.
Prevent it next time: Record all data that influenced the search.

A global transform is reused across validation folds

Why it happens: Held-out distributions influenced every fold’s representation.
How to fix it: Refit learned preprocessing within each training split.
Prevent it next time: Test the full pipeline boundary.

A tiny fold-score gap is called conclusive

Why it happens: Sampling and selection uncertainty were ignored.
How to fix it: Report variation and the structure of the comparison.
Prevent it next time: Avoid unsupported confidence intervals from correlated folds.

Check your understanding

1. Why can a feature-selection step invalidate cross-validation even if the estimator is refitted in every fold?

Show worked solution

If selection used all labels before splitting, each held-out target already influenced the representation. Refit selection inside the fold together with preprocessing and the estimator. Evaluate the whole learned procedure, not only the final coefficient fit.

2. What happens when a final test score motivates another round of tuning?