Deep Learning / Tensor and learning foundations / Lesson 3

Outputs, logits, and stable losses

Match the target type, output representation, and loss reduction while keeping arithmetic numerically stable.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish logits, probabilities, and decisions.
Calculate stable classification losses.
Choose reduction denominators that match the task.

The target determines the output contract

A regression target is a measured quantity with units; a class target identifies a category under a labeling policy. A multiclass model with mutually exclusive classes commonly emits one unconstrained logit per class. Softmax converts those logits into normalized scores, but a normalized score is not automatically calibrated for deployment. Independent binary labels require a different contract from one exclusive class. Write target shape, class order, missing-label policy, and loss input expectations together. A numerically valid loss can still train the wrong task if a class index is interpreted as a continuous target.

Separate probability calculation from a threshold

Logits are real-valued scores before a probability transformation. A binary logit z corresponds to sigmoid(z); changing a decision threshold can change predicted labels without refitting z. Cross-entropy evaluates the assigned probability of the observed outcome, whereas accuracy evaluates thresholded or argmax decisions. Two candidates can have identical accuracy and different losses because their confidence differs. Some library losses expect logits and combine transformations stably; others expect probabilities. Applying softmax before a logits-based loss changes the objective. Read the exact input contract rather than inferring it from the loss name.

Avoid forming large exponentials directly

Softmax is unchanged when the same constant is subtracted from every logit. Subtracting the maximum therefore avoids overflow from large positive scores while preserving the distribution. The first fixture computes log-sum-exp and the negative log probability of class zero without taking the logarithm of a rounded probability. The scores are deliberately large to expose the numerical reason for this transformation. This is a scalar teaching implementation for finite inputs. A reviewed numerical library also needs dtype, nonfinite-input, gradient, and device behavior that this small program does not implement.

python Deterministic standard-library calculation

from math import exp, log
z = [1000.0, 999.0]
m = max(z)
log_partition = m + log(sum(exp(v - m) for v in z))
loss = log_partition - z[0]
probabilities = [exp(v - log_partition) for v in z]
print([round(p, 6) for p in probabilities])
print(f"loss={loss:.6f}")

Expected output

[0.731059, 0.268941]
loss=0.313262

Subtracting the common maximum preserves relative scores and avoids exp(1000).

Use a stable binary-logit identity

Binary cross-entropy from a logit can be written max(z,0) - z*y + log1p(exp(-abs(z))). The second fixture checks confident correct and confident incorrect predictions as well as an uncertain prediction. A wrong confident score receives a large penalty even when thresholded accuracy only records one error. Loss reduction is a separate decision: summing all tokens, averaging examples, and averaging valid tokens assign different weights. For padded sequences, divide by the number of valid targets under the declared objective, and handle a zero-valid-target batch explicitly instead of dividing by zero.

python Deterministic standard-library calculation

from math import exp, log1p
def binary_loss(z, y):
    assert y in (0, 1)
    return max(z, 0) - z * y + log1p(exp(-abs(z)))
for z, y in [(1000.0, 1), (1000.0, 0), (0.0, 1)]:
    print(f"{z:.1f}, {y}: {binary_loss(z, y):.6f}")

Expected output

1000.0, 1: 0.000000
1000.0, 0: 1000.000000
0.0, 1: 0.693147

The stable identity remains finite for these large finite logits; printed zero reflects display precision.

Test objective invariants before comparing models

Keep a tiny set of expected cases: equal logits yield equal probabilities; a common logit offset preserves softmax; a correct confident prediction lowers cross-entropy; masked targets do not affect the chosen numerator or denominator. Record whether class weighting or label smoothing changes the target objective. Such choices may help a particular task, but they also change the interpretation of reported losses and sometimes probability estimates. Compare candidates under the same explicit evaluation definition. A decreasing training loss proves progress on that calculation, not better calibration, acceptable subgroup behavior, or useful decisions outside the fixture.

Common errors and fixes

Loss increases after adding a probability layer

Why it happens: A logits-based loss received probabilities and transformed them again.
How to fix it: Pass raw logits or select the documented probability-based loss.
Prevent it next time: Test one known logit against an independent scalar calculation.

Cross-entropy becomes infinite for large correct scores

Why it happens: Exponentiation overflowed or a rounded probability reached zero.
How to fix it: Use the stable combined logit formulation.
Prevent it next time: Include extreme finite scores and finite-value checks.

Long sequences dominate the reported mean unexpectedly

Why it happens: The reduction denominator differs from the intended unit of weighting.
How to fix it: Define whether examples or valid tokens receive equal weight.
Prevent it next time: Report counts and test uneven sequence lengths.

Check your understanding

1. Why does adding 50 to every multiclass logit leave probabilities unchanged?

Show worked solution

Each exponential gains the same multiplicative factor exp(50), which cancels between numerator and denominator. Subtracting the maximum uses exactly this invariance. The class ordering and cross-entropy are unchanged, apart from finite-precision effects in a particular implementation.

2. Two valid targets have losses 1 and 3, and two padding positions contain arbitrary values. What is the valid-token mean?