PyTorch / Training state and architecture interfaces / Lesson 7

Optimizers, parameter groups, and state

Make the optimizer’s parameter membership, per-group options, and continuation state explicit.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Construct stable parameter groups with intentional options.
Distinguish a missing gradient from an explicit zero gradient.
Explain why weights alone may not reproduce the next optimizer step.

An optimizer owns references to selected parameters

Construct an optimizer from the parameters that the intended update should change, after establishing the module structure and relevant dtype or device choices. Registered module parameters are discoverable, but the optimizer only receives what was actually supplied. A newly created Parameter is not automatically inserted into an existing optimizer because an attribute name looks familiar. Use deterministic parameter collections rather than unordered sets, and inspect names and identities during setup. Shared parameters require deliberate membership so a single mathematical value is not accidentally treated as independent copies. The optimizer does not decide whether the selected objective is statistically valid.

Parameter groups override declared defaults

A group contains a params collection and supported optimizer options, such as a learning rate or decay policy. The illustrated SGD program has one weight group at 0.1 and one bias group at 0.01, overriding the outer default. Its half-squared loss has gradients minus four and minus two, producing weight 1.4 and bias 0.02. This is a mechanics fixture, not a recommendation that every bias needs a smaller rate. Different algorithms give decay and regularization options their own documented semantics; do not infer equivalence merely from a common keyword.

python Illustrative CPU PyTorch 2.14 API program; torch unavailable, expected output derived

import torch
from torch import nn
w = nn.Parameter(torch.tensor(1.0,dtype=torch.float64,device="cpu"))
b = nn.Parameter(torch.tensor(0.0,dtype=torch.float64,device="cpu"))
optimizer = torch.optim.SGD([{"params":[w],"lr":0.1},{"params":[b],"lr":0.01}],lr=1.0)
optimizer.zero_grad(set_to_none=True)
loss = 0.5*(2*w+b-4)**2
loss.backward()
optimizer.step()
print(f"weight={w.item():.6f}; bias={b.item():.6f}")
print("group rates", [group["lr"] for group in optimizer.param_groups])

Expected output

weight=1.400000; bias=0.020000
group rates [0.1, 0.01]

The two explicit parameter groups consume gradients minus four and minus two with different rates. No optimizer or parameter group was executed in PyTorch.

Missing gradients and zero gradients differ

zero_grad(set_to_none=True) resets buffers to None. After backward, a parameter that received no gradient can remain None, which torch.optim handles differently from a tensor full of zeros. With a zero derivative, optimizer state or decay may still affect the update according to the algorithm; a missing gradient can cause the parameter’s step to be skipped. Do not turn every None into zero simply to make a logging function work. First decide whether the parameter is intentionally unused or frozen, or whether the graph path was broken accidentally. Report those cases distinctly.

State is part of the update procedure

Momentum, adaptive moments, counters, and group settings can change the next step even when weights and current gradients match. The standard-library companion contrasts a restored momentum buffer with an absent buffer under a stated scalar rule. The initial absent buffer takes the first gradient, matching the initialization convention described for ordinary PyTorch SGD without the additional variants used elsewhere. Dampening, Nesterov behavior, and adaptive optimizers require their exact documented equations. A weights-only artifact may be appropriate for inference while being insufficient for continuing the same training trajectory.

python Executed deterministic standard-library companion

def step(parameter, buffer, gradient):
    next_buffer = gradient if buffer is None else 0.9*buffer+gradient
    return parameter-0.1*next_buffer, next_buffer
restored, restored_buffer = step(1.0,2.0,-4.0)
weights_only, new_buffer = step(1.0,None,-4.0)
print(f"restored parameter={restored:.6f}; buffer={restored_buffer:.6f}")
print(f"weights-only parameter={weights_only:.6f}; buffer={new_buffer:.6f}")

Expected output

restored parameter=1.220000; buffer=-2.200000
weights-only parameter=1.400000; buffer=-4.000000

This executed scalar rule uses an initial momentum buffer equal to the first gradient when absent. It illustrates state dependence without validating a torch optimizer implementation.

Verify membership and restoration at a known boundary

Compare expected named parameters, group options, gradient presence, and actual parameter changes on a small deterministic case. When restoring optimizer state, preserve compatible group structure and follow the documented load ordering with any scheduler; names alone are not a universal state-matching mechanism. Dynamic unfreezing requires an explicit supported parameter-group policy. Keep the distinction between a conceptual scalar update and a tested framework operation. The torch program is unexecuted, and the companion tests only its own arithmetic; no optimizer serialization, device transfer, scheduler restoration, or model adaptation was performed.

Common errors and fixes

A new trainable parameter never changes

Why it happens: The existing optimizer does not reference the replacement or added Parameter.
How to fix it: Reconcile optimizer membership with the intended module parameters.
Prevent it next time: Inspect parameter identities and a known update after structural changes.

An unused parameter changes after None is replaced with zero

Why it happens: The change altered skip semantics and allowed optimizer state or decay to act.
How to fix it: Preserve the documented missing-gradient policy and diagnose its cause.
Prevent it next time: Test absent and explicit-zero gradients separately for the chosen optimizer.

A resumed run takes another next step despite equal weights

Why it happens: Momentum, counters, group settings, or other optimizer state were omitted.
How to fix it: Restore the relevant compatible state and verify one transition.
Prevent it next time: Keep a next-update comparison against uninterrupted execution.

Check your understanding

1. Why does the bias move by 0.02 rather than 0.2 in the framework fixture?

Show worked solution

Its gradient is minus two and its group rate is 0.01. SGD subtracts 0.01*(-2), producing positive 0.02 from zero. The weight uses another group rate, 0.1. The outer default rate one is overridden by both explicit groups and is not used for those updates.

2. Why do the two scalar continuation cases end at 1.22 and 1.4?