PyTorch / Editorial solution guide

Diagnose missing gradients and unchanged parameters

Separate registration, graph participation, gradient accumulation, and optimizer membership before changing a learning rate.

About 4 min readContent reviewed

Read-only knowledge base · No public comments

Recognize the symptoms

A named value remains unchanged, its .grad is None, a displayed intermediate has no gradient, or the loss changes while an intended layer never updates. Preserve one fixed CPU input, target, module state, gradient mode, objective, and optimizer configuration before adding more training iterations.

Understand the likely causes

  • Possible causes include unregistered layers, an optimizer referencing an old Parameter, intentional freezing, a detached expression, reconstruction with torch.tensor, a no-grad context, unused branches, or inspection of a non-leaf tensor whose gradient was not retained. These causes require different checks; a smaller or larger learning rate cannot reconnect a missing graph path.

1. Reconstruct one expected derivative and update

Choose a tiny synthetic case whose objective and at least one parameter derivative can be calculated independently. State the loss reduction, valid count, dtype, CPU device, module mode, and gradient context. Keep a pre-update snapshot appropriate to the storage contract. Decide which parameters should receive derivatives and which should remain frozen. A scalar loss value alone cannot identify the missing path. In the supported runtime, inspect the actual tensor expression passed to backward rather than only a logged Python number. This guide provides an investigation plan; it does not execute that runtime inspection.

2. Compare registration, optimizer membership, and gradients

Inspect named_parameters and registered containers, then compare the actual Parameter identities supplied to the optimizer. Replacing a module attribute after optimizer construction can leave the optimizer referring to another object. The companion’s synthetic hidden weight has a derivative but lacks optimizer membership, while its head bias belongs to the optimizer but has no derivative contribution. Neither finding alone explains the other. A plain list of layers and an ordinary tensor attribute also have different registration behavior from ModuleList, Parameter, or a registered buffer. Test the role actually required rather than assuming a familiar name guarantees it.

python Executed deterministic standard-library companion 
parameters = {"hidden.weight":{"registered":True,"requires_grad":True,"in_optimizer":False,"gradient":-4.0},
              "head.bias":{"registered":True,"requires_grad":True,"in_optimizer":True,"gradient":None}}
for name,state in parameters.items():
    findings = []
    if not state["registered"]: findings.append("unregistered")
    if not state["in_optimizer"]: findings.append("missing optimizer membership")
    if state["gradient"] is None: findings.append("no gradient contribution")
    print(name, findings)
Expected output
hidden.weight ['missing optimizer membership']
head.bias ['no gradient contribution']

Synthetic records distinguish optimizer membership from gradient presence. The program does not inspect real Parameters, autograd graphs, modules, or optimizer state.

3. Trace the parameter-to-objective path and mode

Check requires_grad on the intended leaf parameters and inspect whether no_grad, inference_mode, detach, item, or a constructor severed a needed dependency. module.eval is a separate behavior switch and does not disable autograd by itself. A non-leaf intermediate can have a valid backward role without accumulating a displayed .grad unless retention is requested appropriately; do not diagnose every missing intermediate buffer as a disconnected model. Compare None with an explicit zero derivative under the chosen optimizer, because skipped parameters and zero-gradient updates can interact differently with state or decay.

4. Verify one repaired transition before expanding

Clear gradients at the intended accumulation boundary, compute a fresh objective, differentiate, and inspect actual parameter changes after the intended optimizer step. Do not reuse a freed graph accidentally or print a stale pre-update loss as a new measurement. Check parameter, buffer, and mode policies separately if a supposedly frozen model changes behavior. Retain a regression fixture for the identified defect and an unrelated valid path. Once the one-step result matches the independent expectation, add batching and stochastic features gradually; successful local fitting remains distinct from held-out model-quality evidence.

Verify the fix

Verification identifies the failed registration, membership, derivative, or update boundary and matches one controlled transition to its independent expectation. Only the supplied standard-library record classifier was executed here. No PyTorch parser, Parameter inspection, backward graph, optimizer step, model, or gradient-mode runtime was tested.

Keep it from recurring

  • Keep named-state and optimizer-membership checks with a small derivative fixture, preserve None-versus-zero semantics, and review structural changes before reusing an optimizer. Record the actual execution environment and distinguish intended freezing from accidental disconnection in diagnostics.

Official references