Deep Learning / Architectures and training dynamics / Lesson 12

Diagnosing training dynamics

Use boundary checks, controlled fitting, and gradient evidence to isolate a failed training procedure.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate data, objective, derivative, and update failures.
Interpret clipping and zero-gradient summaries.
Design a controlled tiny-set learning investigation.

Preserve the first failing boundary

A training failure can begin in data decoding, feature scaling, a layer operation, the objective, backward computation, or the optimizer update. Preserve the first nonfinite value or unexpected shape, not only the final invalid loss. Record the exact synthetic input, parameters, mode, reduction, and update state needed to reproduce it. Check input and target alignment before adjusting architecture. A constant loss may come from a disconnected graph or unchanged parameters, but it may also reflect display rounding or an already-flat local region. Turn an observation into competing testable hypotheses rather than a single automatic diagnosis.

Fit a tiny deterministic set as a controlled test

A small fixed training set with stochastic transformations disabled can reveal whether the pipeline can reduce its own objective. Use a task the chosen architecture can represent, inspect initial predictions, and compare an independent loss calculation. Failure to fit directs attention toward implementation, scale, or optimization, but success does not prove generalization or eliminate leakage. Keep one data order and enough printed precision to see changes. After the controlled case works, add minibatches, random transforms, normalization state, and scheduling one at a time so the first behavioral change has an interpretable cause.

Understand what gradient clipping changes

Global norm clipping rescales a finite gradient vector when its Euclidean norm exceeds a chosen bound. The first fixture preserves direction while shrinking a length-five vector to length two. Elementwise clipping changes coordinates independently and is a different operation. Clipping can limit an update contribution and is useful in some procedures, but cannot restore a missing gradient, correct labels, or make a nonfinite vector finite reliably. Its bound has meaning relative to loss scaling, accumulation, and precision. State whether clipping happens after accumulation and any required unscaling, before the optimizer consumes the gradient.

python Deterministic standard-library calculation

from math import sqrt
gradient = [3.0, 4.0]
limit = 2.0
norm = sqrt(sum(g*g for g in gradient))
scale = min(1.0, limit/norm)
clipped = [g*scale for g in gradient]
print(f"before={norm:.1f}")
print([round(g,2) for g in clipped])
print(f"after={sqrt(sum(g*g for g in clipped)):.1f}")

Expected output

before=5.0
[1.2, 1.6]
after=2.0

Global norm clipping rescales the entire vector and preserves its direction for a finite nonzero gradient.

Inspect distributions and paths rather than one norm

A single global gradient norm can hide a layer with all-zero derivatives or a small subset with extreme values. The second fixture prints finite and all-zero checks for two named synthetic groups. Inspect activation saturation, rectifier activity, parameter changes, and expected graph connections to interpret the zero group. An intentionally frozen group should remain unchanged; an accidentally detached trainable group should not. Norms alone cannot distinguish these cases. Also compare gradients with actual parameter updates because optimizer state, weight decay, or an omitted parameter group can change the relationship between them.

python Deterministic standard-library calculation

from math import isfinite
layers = {"hidden": [0.0, 0.0], "output": [0.25, -0.5]}
for name, values in layers.items():
    print(name, "finite", all(isfinite(v) for v in values),
          "all_zero", all(v == 0 for v in values))

Expected output

hidden finite True all_zero True
output finite True all_zero False

These summaries locate a synthetic zero-gradient boundary; they do not identify its cause without tracing the forward graph.

Make each repair predict an observable result

If the hypothesis is a detached hidden layer, restoring its graph path should create the expected hidden gradients in a known fixture. If the hypothesis is an excessive step size, a controlled smaller-step trace should change the divergence pattern while leaving the derivative check intact. If the hypothesis concerns padding, appending padding should stop changing valid predictions after the repair. Preserve before-and-after evidence and avoid changing several mechanisms together. Re-enable validation only after the training calculation is understood, then assess generalization honestly. These diagnostics establish local correctness evidence, not a certification that a real model is safe or useful.

Common errors and fixes

Clipping is used as the first response to invalid loss

Why it happens: The original nonfinite operation was never located.
How to fix it: Check forward boundaries and reproduce the first invalid value.
Prevent it next time: Keep finite-value assertions on a tiny deterministic pipeline.

A large global norm hides a frozen hidden group

Why it happens: Aggregation concealed the distribution of gradients and updates.
How to fix it: Inspect named parameter groups and their expected graph paths.
Prevent it next time: Test gradient presence and actual mutation separately.

A tiny-set fit is reported as model quality

Why it happens: Memorization of training examples was confused with generalization.
How to fix it: Use it only as a controlled implementation diagnostic.
Prevent it next time: Keep separate training-correctness and held-out evaluation records.

Check your understanding

1. For gradient (6,8) and global norm limit five, calculate the clipped vector.

Show worked solution

The norm is ten, so multiply both coordinates by 0.5 to get (3,4), whose norm is five. Direction is preserved. Clipping each coordinate independently to five would instead yield (5,5), a different direction and norm.

2. A hidden group has zero gradients while the output group changes. Give a useful next test.