Machine Learning / Evidence and operational responsibility / Lesson 13

Evaluation uncertainty and model comparisons

Report the uncertainty and sampling unit behind a score, and compare models through matched evidence rather than isolated winning numbers.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

A score is an estimate under assumptions
Inspect resampled statistic arithmetic
Communicate evidence without unsupported precision

A score is an estimate under assumptions

A held-out metric summarizes finite observations from a defined sampling process. It is not a permanent constant of an algorithm, and a small test set can support very uncertain conclusions. Record the number of observations, independent entities, positive outcomes, time span, and weighting unit. A thousand correlated records from ten devices do not necessarily provide the same information as a thousand independent devices. Separate uncertainty from evaluation sampling, random training, hyperparameter selection, and future distribution change. One interval computed under one source of variation does not automatically cover the others. State what was held fixed and what was resampled or refitted.

Choose a resampling unit that matches dependence

A bootstrap resamples observed units with replacement under its assumptions to explore a statistic’s variability. Ordinary independent-row resampling is inappropriate when the relevant units are groups or dependent temporal blocks. A test-set bootstrap of fixed predictions does not reproduce the uncertainty of training an entirely new model. Small or degenerate samples can make resampling intervals unstable, and rare outcomes can disappear from resamples. Use the domain’s sampling design and an appropriate statistical method rather than one universal recipe. Report limitations and support counts. A broad interval is informative evidence, not a defect to hide by running more favorable resamples.

Inspect resampled statistic arithmetic

The first example computes means from four explicitly supplied resample-index rows. It is a transparent arithmetic demonstration, not a Monte Carlo interval or a coverage claim. A full bootstrap procedure would generate a justified set of resamples under a declared random and sampling policy. Keeping the original loss vector separate makes it clear which observations are repeated in each draw. Repeated index zero does not create another independent observation. The resulting spread illustrates that a sample-derived summary can change with composition, while the tiny fixed collection is insufficient for a meaningful real-world confidence statement.

python Fixed synthetic offline example

import numpy as np
loss = np.array([0.0, 1.0, 2.0, 3.0])
resample_indices = np.array([[0, 1, 2, 3], [0, 0, 1, 1], [2, 2, 3, 3], [0, 2, 0, 2]])
means = loss[resample_indices].mean(axis=1)
print(means.tolist())

Expected output

[1.5, 0.5, 2.5, 1.0]

Four fixed resamples produce four checkable means. They illustrate resampling arithmetic and are not used to claim a confidence interval.

Compare alternatives on the same cases

The second example computes per-observation loss differences for two supplied predictors on matched outcomes. A paired comparison preserves which cases were easy or hard for both. Comparing two unrelated means can discard that information. The mean improvement and number of improved cases answer different questions: a method may improve many small errors while worsening one costly case. Keep weights, exclusions, thresholds, and prediction availability identical or explain the difference. If the same test was used to select one model from many, account for selection before calling the resulting comparison independent. These fixture losses are invented and establish no external model ranking.

python Fixed synthetic offline example

import numpy as np
loss_a = np.array([1.0, 2.0, 1.0, 2.0])
loss_b = np.array([0.5, 2.5, 0.5, 1.5])
assert loss_a.shape == loss_b.shape
difference = loss_b - loss_a
print(f"fixture mean B-minus-A={difference.mean():.3f}")
print("improved cases", int(np.sum(difference < 0)))

Expected output

fixture mean B-minus-A=-0.250
improved cases 3

Lower loss is better under this fixture’s convention. B improves three matched cases and has a mean difference of minus one quarter.

Communicate evidence without unsupported precision

Use a meaningful number of reported digits and describe the split, population, sample structure, and uncertainty method. Do not label the standard deviation of overlapping cross-validation folds a universal confidence interval. For operational decisions, include subgroup support, error costs, and whether the observed difference is useful at all. Repeatedly collecting test results until a desired significance appears changes the procedure and must be handled under an explicit sequential design. A final report can conclude that the evidence does not distinguish candidates reliably. Prefer that honest statement to a fragile winner selected from a tiny numerical gap.

Common errors and fixes

Many correlated rows are treated as independent support

Why it happens: The resampling and uncertainty unit is wrong.
How to fix it: Use the relevant group or temporal design.
Prevent it next time: Report independent entities and dependence assumptions.

A fixed-prediction bootstrap is called total model uncertainty

Why it happens: Training and selection were held fixed but not acknowledged.
How to fix it: State the exact source of variation assessed.
Prevent it next time: Separate fitting, sampling, and deployment uncertainty.

A tiny gap is promoted to a universal winner

Why it happens: Finite-sample and multiple-selection uncertainty were ignored.
How to fix it: Report the paired comparison and its limitations.
Prevent it next time: Keep decision usefulness distinct from numerical ordering.

Check your understanding

1. Why might a device-level bootstrap be more appropriate than resampling every sensor row?

Show worked solution

Rows within a device can share environment, calibration, and history, so independent-row resampling understates that dependence. Resample at the unit that matches the intended generalization claim, preserving relevant within-unit structure. The design still needs enough representative units and an honest statement of its assumptions.

2. What does a negative mean B-minus-A loss difference establish on a fixed fixture?