PyTorch / Training state and architecture interfaces / Lesson 8

Accumulation, clipping, and schedules

Preserve objective weighting and optimizer-step boundaries when changing microbatch processing.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Accumulate sample-weighted derivatives at fixed parameters.
Apply gradient-norm clipping to the intended combined gradient.
Advance a scheduler using its documented update clock.

Microbatches do not redefine the objective automatically

A large logical batch can be processed as several smaller microbatches while parameters remain fixed. To reproduce a mean per-example objective, accumulate the intended summed losses divided by the total valid example count, or use an algebraically equivalent weighting. Equal averaging of unequal microbatch means gives another objective. The fixture divides both microbatch sums by three, including a final one-example batch. This equivalence concerns the declared calculation: batch-dependent normalization, stochastic operations, and other state can prevent a complete model from behaving exactly like one larger forward pass.

Reset and update at the logical boundary

The illustrated scalar model starts at zero and processes three fixed inputs in microbatches of two and one. zero_grad occurs once before accumulation, and optimizer.step occurs once after both backward calls. Updating between them would evaluate the second derivative at another parameter state, creating a different trajectory. The full-batch derivative is minus fourteen divided by three. Keep the actual valid count for a partial final accumulation window rather than a fixed number of microbatch calls. Record whether a skipped update consumes data or advances any schedule under the chosen policy.

python Illustrative CPU PyTorch 2.14 API program; torch unavailable, expected output derived

import torch
w = torch.nn.Parameter(torch.tensor(0.0,dtype=torch.float64,device="cpu"))
x = torch.tensor([1.0,2.0,3.0],dtype=torch.float64,device="cpu")
y = x.clone()
optimizer = torch.optim.SGD([w],lr=0.1)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer,step_size=1,gamma=0.5)
optimizer.zero_grad(set_to_none=True)
for start,end in [(0,2),(2,3)]:
    (0.5*((w*x[start:end]-y[start:end])**2).sum()/len(x)).backward()
print(f"accumulated={w.grad.item():.6f}")
norm = torch.nn.utils.clip_grad_norm_([w],max_norm=1.0,error_if_nonfinite=True)
optimizer.step()
scheduler.step()
print(f"preclip norm={norm.item():.6f}; weight={w.item():.6f}")
rate = optimizer.param_groups[0]["lr"]
print(f"next rate={rate:.6f}")

Expected output

accumulated=-4.666667
preclip norm=4.666667; weight=0.100000
next rate=0.050000

Microbatch sums use the full three-example denominator, then one combined clip and update; displayed values are rounded derived expectations, not executed torch results.

Norm clipping transforms gradients before consumption

clip_grad_norm_ computes a norm across supplied parameter gradients as a combined vector and modifies those gradients in place. Its return value reports the total norm used before clipping, not the final parameter update. The fixture requests a finite-norm error and a bound of one. Display rounding makes its derived step 0.100000; implementation stabilization and floating precision mean exact mathematical equality should not be assumed. Clipping each element independently is another transformation. Clipping does not repair invalid targets, a missing graph path, or an already nonfinite forward calculation.

A scheduler has a specific event and state

The program uses StepLR with one optimizer update per scheduling event and halves the rate after that update for the next one. Other schedulers use different clocks or a validation metric; read the actual API and call-order contract. Advancing once per microbatch would run this schedule too quickly. Save the relevant optimizer and scheduler state for continuation and verify their restored rates with a known transition. If loss scaling is used in another procedure, unscale at its documented boundary before applying a clipping bound meant for unscaled gradients; no scaler or mixed-precision runtime is used here.

Use arithmetic to expose an accidental weighting change

The companion compares the intended sample mean with an equally weighted mean of two microbatch means and calculates ideal scalar clipping. It does not verify PyTorch’s internal stabilization, counters, or gradient buffers. Inspect valid counts, gradient norms, actual parameter changes, and the rate applied at each optimizer update in a supported runtime. Include an uneven final window, no-valid-target policy, and an intentionally absent gradient. The framework program is unexecuted because torch is unavailable; only the supplied standard-library calculation was run, with no training throughput or memory measurement.

python Executed deterministic standard-library companion

gradients = [-1.0,-4.0,-9.0]
combined = sum(gradients)/len(gradients)
mean_of_means = ((gradients[0]+gradients[1])/2+gradients[2])/2
scale = min(1.0,1.0/abs(combined))
clipped = combined*scale
print(f"sample mean={combined:.6f}")
print(f"mean of microbatch means={mean_of_means:.6f}")
print(f"mathematical clipped step={-0.1*clipped:.6f}")

Expected output

sample mean=-4.666667
mean of microbatch means=-5.750000
mathematical clipped step=0.100000

The companion checks exact objective weighting and ideal scalar norm clipping. It does not reproduce implementation epsilon, scheduler counters, or framework gradient storage.

Common errors and fixes

Accumulation disagrees with the intended logical-batch mean

Why it happens: Unequal microbatch means received equal weight.
How to fix it: Accumulate the intended numerator using the actual valid denominator.
Prevent it next time: Test two-plus-one observations against an explicit full objective.

A scheduler decays before the intended updates occur

Why it happens: It was advanced for every microbatch rather than the declared event.
How to fix it: Advance it according to its documented update or metric clock.
Prevent it next time: Log both microbatch and optimizer-step counters.

Clipping appears inconsistent after a precision-policy change

Why it happens: The bound was applied to a differently scaled gradient.
How to fix it: Check accumulation, any unscaling, clipping, and update order together.
Prevent it next time: Compare a tiny known combined gradient under the explicit policy.

Check your understanding

1. Why is the unweighted mean of microbatch means minus 5.75 rather than minus 14/3?

Show worked solution

The first two derivatives average to minus 2.5 and the singleton derivative is minus nine. Giving these two means equal weight yields (-2.5-9)/2=-5.75. Observation weighting instead sums minus fourteen and divides by three. Both are arithmetic, but only one matches the declared per-example objective.

2. What does the clipping function’s returned norm tell you in this fixture?