Machine Learning / Prediction and data foundations / Lesson 6

Classification scores and metrics

Separate ranking, probabilities, decisions, and error rates so a classification result has an interpretable operational meaning.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

A class decision is not the whole prediction
Compute one thresholded fixture explicitly
Keep ranking evaluation and decision policy separate

A class decision is not the whole prediction

A classifier can produce a category, a ranking score, or an estimated probability depending on its interface. These outputs support different decisions. A score threshold turns ordered values into labels, while calibration asks whether probability values agree with observed frequencies under a defined population. Do not call an arbitrary margin a probability or assume a default threshold matches the task’s error costs. Define the positive class explicitly, because precision, recall, and several rates depend on that convention. The examples here use fixed binary labels solely to check calculations; their scores are supplied synthetic values, not the result of a claimed trained model.

Name each confusion-matrix quantity

True positives are positive outcomes predicted positive; false positives are negative outcomes predicted positive. False negatives miss positives, while true negatives correctly reject negatives. Precision divides true positives by predicted positives, and recall divides true positives by actual positives. Specificity uses actual negatives, and F1 combines precision and recall under its definition. Each denominator can be small or even zero, so state the undefined-case policy rather than silently interpreting a library’s default as a scientific conclusion. Count both outcomes and decisions to verify the metric implementation. Rates without their sample counts can hide weak evidence.

Compute one thresholded fixture explicitly

The first example applies a stated greater-than-or-equal threshold of 0.5 and obtains one of each confusion-matrix outcome. The printed precision and recall are both one half. Equality behavior matters when scores can equal the threshold, so keep it consistent between evaluation and inference. The simple code verifies shape before comparison to prevent accidental array broadcasting. A production system also needs a stable class-label mapping and handling for missing or invalid scores. A correct matrix on four invented rows establishes arithmetic, not expected performance, fairness, or calibration on a real population.

python Fixed synthetic offline example

import numpy as np
y = np.array([0, 1, 1, 0])
score = np.array([0.2, 0.8, 0.4, 0.7])
assert y.shape == score.shape
pred = score >= 0.5
tp = int(np.sum((y == 1) & pred))
fp = int(np.sum((y == 0) & pred))
fn = int(np.sum((y == 1) & ~pred))
tn = int(np.sum((y == 0) & ~pred))
print(tp, fp, fn, tn)
print(f"precision={tp / (tp + fp):.2f} recall={tp / (tp + fn):.2f}")

Expected output

1 1 1 1
precision=0.50 recall=0.50

The threshold creates one true positive, false positive, false negative, and true negative. Both demonstrated denominators are nonzero.

Compare accuracy with a meaningful baseline

The second example predicts the majority class for a deliberately imbalanced fixture. It obtains ninety-percent accuracy while detecting none of the positive cases. That number is a transparent property of ten invented labels, not an impressive benchmark. Whether this baseline is useful depends on the costs, prevalence, and intended decision. Balanced accuracy, precision-recall measures, class-specific rates, and a confusion matrix can reveal different aspects, but no metric is universally best. Evaluate the agreed primary objective and important secondary constraints rather than searching after the fact for whichever score looks strongest.

python Fixed synthetic offline example

import numpy as np
y = np.array([0] * 9 + [1])
pred = np.zeros_like(y)
accuracy = np.mean(pred == y)
recall = np.sum((pred == 1) & (y == 1)) / np.sum(y == 1)
print(f"fixture accuracy={accuracy:.2f} recall={recall:.2f}")

Expected output

fixture accuracy=0.90 recall=0.00

The majority baseline is correct on nine negatives but misses the single positive. The result illustrates the denominator choice, not model quality outside this fixture.

Keep ranking evaluation and decision policy separate

ROC and precision-recall curves vary a threshold and summarize ranking behavior in different ways; their interpretation depends on the task and class distribution. AUC does not directly specify the deployed threshold, the number of alerts, or the cost of errors. For a ranked review queue, capacity at the top of the list may matter more than a global summary. For a probabilistic decision, proper scoring rules and calibration deserve separate attention. Select thresholds on permitted validation data, preserve the final assessment boundary, and report counts and uncertainty. Later lessons connect these scores to calibration, subgroup analysis, and operational monitoring.

Common errors and fixes

High accuracy hides missed positives

Why it happens: The evaluation ignored class prevalence and error type.
How to fix it: Inspect the baseline and class-specific counts under the task’s costs.
Prevent it next time: Choose metrics before comparing models.

A ranking score is reported as a probability

Why it happens: Its numerical scale was never calibrated or defined that way.
How to fix it: Use the estimator’s documented output contract and assess calibration separately.
Prevent it next time: Label scores and probabilities distinctly.

Precision is silently zero when no positives are predicted

Why it happens: An undefined denominator policy is hidden.
How to fix it: Report the case and its count explicitly.
Prevent it next time: Test zero-denominator cases before interpreting summaries.

Check your understanding

1. A model emits no positive predictions. What can be said about precision and recall?

Show worked solution

Precision has a zero predicted-positive denominator and is undefined under the direct formula. If actual positives exist, recall is zero. State the reporting convention and support counts rather than presenting an arbitrary replacement value as a measured property. If no actual positives exist either, recall also needs an undefined-case policy.

2. Why should a threshold be selected before the final test is used?