Machine Learning / Evidence and operational responsibility / Lesson 14

Calibration and decision costs

Assess probability meaning separately from ranking, and connect thresholds to declared costs and a representative population.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

A useful ranking can have misleading probabilities
Inspect a reliability-table fixture
Monitor probabilities and decisions as separate outputs

A useful ranking can have misleading probabilities

A model can rank cases well while its stated probabilities are systematically too high or too low. Calibration asks whether outcomes agree with predicted probabilities under a specified population and conditioning view. It is not the same as classification accuracy or perfect prediction of every individual. A score of 0.8 does not mean a particular event is guaranteed, and a small bucket with two cases cannot establish reliable frequency agreement. Inspect both discrimination and calibration because each supports different decisions. Distribution or prevalence changes can alter probability behavior even when some ranking patterns remain similar.

Fit calibration without recycling evaluation targets

Calibration methods such as sigmoid or isotonic mappings learn from outcomes and must use a legitimate calibration split or cross-validated construction. Calibrating a predictor on the same outcomes it overfit can give misleading confidence. Method flexibility, data size, and class support matter. Temperature scaling is another documented approach in particular settings, but a research result on selected architectures is not a guarantee for every model or population. Preserve the calibration artifact and its training provenance alongside the base predictor. The final assessment should evaluate the combined fitted procedure, including any selected threshold, rather than the raw estimator alone.

Inspect a reliability-table fixture

The first example divides four supplied probabilities into two explicitly defined bins and reports count, mean probability, and observed positive rate. The numbers show a discrepancy in each tiny bucket, but they cannot establish a stable calibration estimate. Different bin edges and sample sizes can change the visual impression, so report counts and uncertainty rather than only a curve. Empty bins require a defined display policy. Proper scoring rules such as log loss or Brier score summarize probability quality under their assumptions, but they also reflect aspects beyond calibration alone; do not interpret one lower score as a complete diagnosis.

python Fixed synthetic offline example

import numpy as np
p = np.array([0.1, 0.2, 0.7, 0.8])
y = np.array([0, 1, 1, 1])
for mask in [p < 0.5, p >= 0.5]:
    print(f"n={int(mask.sum())} mean_p={p[mask].mean():.2f} observed={y[mask].mean():.2f}")

Expected output

n=2 mean_p=0.15 observed=0.50
n=2 mean_p=0.75 observed=1.00

The table is directly calculated from four fixed cases. Two observations per bin are not evidence for reliable population calibration.

Derive a threshold from an explicit cost model

The second example assumes calibrated probabilities for the decision population, zero cost for correct decisions, false-positive cost one, and false-negative cost four. Comparing the two expected costs gives a positive decision at probability at least one fifth under the stated tie policy. This is a mathematical fixture, not a universal threshold recommendation. Capacity limits, intervention effects, unequal benefits, changing prevalence, and human review can change the optimization problem. If the probabilities are not appropriate for the deployment population, the cost comparison does not magically repair them. Validate the operational assumptions before applying a formula.

python Fixed synthetic offline example

import numpy as np
p = np.array([0.1, 0.3, 0.6, 0.9])
cost_fp, cost_fn = 1.0, 4.0
positive_cost = (1.0 - p) * cost_fp
negative_cost = p * cost_fn
choice = positive_cost <= negative_cost
print(f"fixture threshold={cost_fp / (cost_fp + cost_fn):.2f}")
print(choice.astype(int).tolist())

Expected output

fixture threshold=0.20
[0, 1, 1, 1]

The expected-cost rule depends on calibrated probabilities and the explicitly simplified costs. Equality chooses positive in this fixture.

Monitor probabilities and decisions as separate outputs

Store the base model, calibration mapping, class convention, threshold, and cost policy as coordinated artifacts. Test exact threshold behavior and explain any abstain or review region. Monitor probability distributions, delayed outcomes, calibration slices, alert volume, and realized error costs with an appropriate time window. A threshold change can alter workload and subgroup error rates even when the predictor is unchanged. Keep the final test out of repeated threshold tuning, and do not infer that global calibration guarantees calibration in every subgroup. A trustworthy decision report states which assumptions were checked and which uncertainties remain.

Common errors and fixes

A high AUC is called calibrated probability evidence

Why it happens: Ranking and probability reliability were conflated.
How to fix it: Evaluate calibration and proper probability scores separately.
Prevent it next time: State the output scale and intended decision.

A calibration mapping overfits the predictor’s training outcomes

Why it happens: No independent calibration construction was used.
How to fix it: Use permitted held-out or cross-validated calibration.
Prevent it next time: Record calibration data provenance and support.

A cost-derived threshold is deployed under different assumptions

Why it happens: Costs, prevalence, or probability meaning changed.
How to fix it: Revalidate the complete decision policy.
Prevent it next time: Monitor alert volume and subgroup consequences.

Check your understanding

1. Why can one globally calibrated predictor still need subgroup calibration review?

Show worked solution

Aggregate agreement can average over different errors within groups. Inspect appropriate conditional slices with support and uncertainty, while recognizing the limited sample size and the chosen fairness or decision objective. Global calibration alone does not establish every conditional property or a complete fairness guarantee.

2. What changes if a decision has a hard review-capacity limit?