Locate nonfinite loss and divergent updates

Understand the likely causes

• Possible causes include malformed or nonfinite inputs, an unstable probability calculation, invalid target indices, mismatched loss inputs, an unintended reduction, or steps that are too large for the objective scale. Low-precision overflow and erroneous gradient accumulation introduce other boundaries. No single symptom uniquely identifies one of these causes.

1. Reconstruct one deterministic transition

Capture a minimal synthetic reproduction with input and target shapes, feature units, target convention, parameter values, mode, and the exact loss reduction. Disable random augmentation and use a supported ordinary-precision reference where practical. Inspect values after preprocessing, after each relevant layer, before reduction, after backward computation, and after the update. Stop at the earliest violation of the intended finite-value or shape contract. Do not keep stepping after the model is invalid and then use that corrupted state to judge a proposed repair.

2. Separate arithmetic stability from gradient scale

Check whether a logits-based loss received raw scores or an already transformed probability. For a scalar reproduction, compare with a stable log-sum-exp or binary-logit identity. The fixture deliberately pairs a stable large-score loss with an independent large finite derivative caused by a feature scale of 100. A finite result does not establish a sensible step size, and a large derivative does not prove overflow. Calculate the predicted parameter change before applying it. This separates a forward-formula problem from an optimizer-scale hypothesis and supplies a concrete expectation for the next experiment.

from math import exp, log, isfinite
logits = [1000.0, 998.0]
maximum = max(logits)
loss = maximum + log(sum(exp(z-maximum) for z in logits)) - logits[0]
residual, feature = -3.0, 100.0
gradient = residual * feature
print("finite stable loss", isfinite(loss))
print(f"loss={loss:.6f}, gradient={gradient:.1f}")
print(f"update at 0.1={-0.1*gradient:.1f}")

finite stable loss True
loss=0.126928, gradient=-300.0
update at 0.1=30.0

3. Audit denominators and precision order

Inspect the unreduced loss and the count of valid examples or tokens. A switch from a mean to a sum can scale derivatives with batch length, while a zero-valid-target batch can create division by zero. For accumulation, combine the intended numerator and denominator before one update. If loss scaling is involved, verify nonfinite handling, unscaling, and clipping order against the actual framework documentation. Keep the clipping bound meaningful relative to the unscaled combined gradient. Clipping cannot repair a wrong target or reconnect a detached parameter.

4. Verify the hypothesis with a small controlled trace

For an unstable exponential formula, the repair should make the same large finite logits produce a finite independently checked loss. For excessive update size, hold the derivative calculation fixed and compare justified smaller-step trajectories. For a masking defect, changing padding must stop changing the valid objective. Test one change at a time and retain the pre-repair failure as a regression fixture. Once the tiny case behaves as predicted, reintroduce stochastic features and larger batches gradually; do not infer real-world model quality from successful local fitting.

Keep it from recurring

• Maintain extreme-finite-score cases, uneven valid counts, a zero-valid-target policy, and a deterministic one-update fixture. Record objective scaling, feature units, accumulation, precision, and optimizer settings together so a future change cannot silently alter the diagnostic baseline.