PyTorch / Tensor, gradient, and module foundations / Lesson 4

Losses, targets, and reductions

Match a PyTorch loss input contract to target meaning, dimensions, dtype, and weighting.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish raw logits from probabilities and final decisions.
Specify class-index, binary, and regression target shapes.
Inspect unreduced losses and intentional reduction denominators.

A loss interface encodes a specific task

CrossEntropyLoss commonly receives unnormalized multiclass logits and integer class-index targets under the contract demonstrated here. For a batch of B examples and C exclusive classes, logits have shape (B,C) and index targets have shape (B,) with an appropriate integer dtype such as torch.long. The API also documents a separate probability-target contract, which requires deliberate distribution semantics and validation; do not infer that any floating target is appropriate. Class order and target availability remain part of the data contract. A successful call cannot prove the label indices refer to the intended categories.

Use logits-based losses according to their documentation

CrossEntropyLoss combines the relevant log-probability and negative-log-likelihood calculation, so applying softmax beforehand changes the function it receives. BCEWithLogitsLoss instead combines sigmoid and binary cross-entropy for a same-shape floating target contract, including independent binary labels when that is the task. It is not interchangeable with an exclusive multiclass index target. Regression losses such as MSELoss compare numeric values under an intended paired shape and unit. A probability output, an argmax decision, and a differentiable training score have different roles, even when one is computed from another.

Inspect per-example loss before choosing aggregation

The illustrative program uses two float64 logit rows and two long class indices, then requests reduction="none". Both rows favor their correct class by two, so both losses equal log(1+exp(-2)), approximately 0.126928. The loss shape is (2,), preserving one value per example. This unweighted fixture avoids class weights and ignored labels so the arithmetic is independently clear. In a spatial or sequence task, additional dimensions can remain before reduction. Inspect the exact unreduced shape rather than assuming that a scalar mean guarantees one correctly paired term per observation.

python Illustrative CPU PyTorch 2.14 API program; torch unavailable, expected output derived

import torch
from torch import nn
logits = torch.tensor([[2.0,0.0],[0.0,2.0]], dtype=torch.float64, device="cpu")
target = torch.tensor([0,1], dtype=torch.long, device="cpu")
losses = nn.CrossEntropyLoss(reduction="none")(logits, target)
print("loss shape", tuple(losses.shape))
print("per example", [round(x,6) for x in losses.tolist()])
print(f"mean={losses.mean().item():.6f}")

Expected output

loss shape (2,)
per example [0.126928, 0.126928]
mean=0.126928

Class-index targets identify the correct logit column. The log-sum-exp arithmetic gives the displayed values, but no loss module was executed.

The denominator defines the weighting question

The standard-library reference computes one stable class loss and a mean over two valid positions, excluding an arbitrary padded value. A mean over all tensor elements, a mean over examples, and a mean over valid tokens assign different weights. Class weights and ignored targets can change a library loss’s documented denominator, so compare its formula rather than dividing by a convenient batch count afterward. Handle no-valid-target input under an explicit policy. A change in sequence padding should not improve a valid-token metric merely by adding zeros to the denominator.

python Executed deterministic standard-library companion

from math import exp, log
z = [2.0,0.0]
m = max(z)
class_loss = m+log(sum(exp(x-m) for x in z))-z[0]
values, valid = [0.2,1.4,99.0], [True,True,False]
numerator = sum(x for x,ok in zip(values,valid) if ok)
denominator = sum(valid)
assert denominator > 0
print(f"class loss={class_loss:.6f}")
print(f"valid mean={numerator/denominator:.6f}")

Expected output

class loss=0.126928
valid mean=0.800000

The executed scalar reference checks stable classification arithmetic and a declared valid-position denominator; it does not test PyTorch loss weighting or ignore_index behavior.

Test semantic cases as well as numerical stability

Keep fixtures for equal logits, a confident correct class, a confident incorrect class, invalid indices, unequal sequence lengths, and threshold-sensitive decisions where relevant. Verify label mapping and finite input assumptions before tuning the optimizer. A lower training loss is evidence about the programmed objective, not automatically better calibration, subgroup behavior, or external decision quality. The framework example is illustrative because torch is absent. Only the supplied scalar companion is executed; no CrossEntropyLoss, BCEWithLogitsLoss, MSELoss, masking implementation, or model fit was run during review.

Common errors and fixes

CrossEntropyLoss receives already-softmaxed scores

Why it happens: The logits contract was confused with a probability display.
How to fix it: Pass documented raw logits for the chosen loss mode.
Prevent it next time: Compare one known row with an independent log-sum-exp calculation.

A binary target is treated as a multiclass index vector

Why it happens: Different task and shape contracts were conflated.
How to fix it: Choose the loss from label semantics and validate shape and dtype.
Prevent it next time: Test one complete labeled fixture before adding batching.

Padding changes the reported valid-target mean

Why it happens: The numerator mask and denominator policy did not match.
How to fix it: Aggregate the intended valid terms and valid count.
Prevent it next time: Test unequal lengths and a no-valid-target boundary explicitly.

Check your understanding

1. Why are target values [0,1] torch.long in the illustrated program?

Show worked solution

They are indices selecting the correct class in each of two logit rows, not probabilities to regress toward. The first row selects column zero and the second selects column one. A probability-target variant would have another shape and distribution contract; changing dtype alone does not define that task.

2. Two valid token losses are 0.2 and 1.4, with one padded position. What is their mean, and what would dividing by three do?