PyTorch / Tensor, gradient, and module foundations / Lesson 2

Autograd and gradient modes

Trace which operations enter a backward graph and separate gradient accumulation from parameter updates.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Identify leaf tensors and scalar backward objectives.
Explain accumulation into .grad across fresh graphs.
Distinguish requires_grad, no_grad, inference_mode, and module evaluation mode.

Autograd follows supported executed operations

PyTorch records operations in a backward graph when the relevant inputs require gradients under grad mode. A leaf tensor created with requires_grad=True can receive accumulated derivatives in its .grad field; intermediate tensors normally serve the chain rule rather than retaining a displayed gradient buffer automatically. Integer indices and hard decisions do not become ordinary differentiable parameters merely because they appear in a program. Define the scalar objective and the parameter path before diagnosing missing gradients. Automatic differentiation computes derivatives of the implemented calculation, not proof that its target, labels, or evaluation design are correct.

Backward and accumulation have separate lifetimes

The illustrative program creates a scalar leaf, evaluates a half-squared loss, and calls backward. At w=2, the residual is five and the local derivative of 3*w is three, so the gradient is fifteen. A fresh forward graph is then built and differentiated again, accumulating another fifteen into the same leaf buffer. This is different from calling backward twice on the same released graph. Clearing a buffer does not change the parameter; an optimizer update is another operation. Later lessons distinguish None from an explicit zero gradient when an optimizer interprets skipped parameters.

python Illustrative CPU PyTorch 2.14 API program; torch unavailable, expected output derived

import torch
w = torch.tensor(2.0, dtype=torch.float64, device="cpu", requires_grad=True)
loss = 0.5*(3*w-1)**2
loss.backward()
print("first gradient", w.grad.item())
(0.5*(3*w-1)**2).backward()
print("accumulated gradient", w.grad.item())
w.grad = None
with torch.no_grad():
    output = 3*w
print("no-grad output requires gradient", output.requires_grad)

Expected output

first gradient 15.0
accumulated gradient 30.0
no-grad output requires gradient False

Each backward call uses a newly constructed scalar objective, so leaf gradients accumulate without reusing a freed graph. No PyTorch backward or grad mode was executed during review.

Gradient modes control graph recording

torch.no_grad disables recording for computations in its context while allowing many resulting values to be used later in ordinary grad mode. torch.inference_mode has a stronger contract with additional restrictions on tensors created inside it, so do not assume it is a drop-in replacement for every no-grad block. requires_grad controls which parameters participate in a gradient calculation; it is not a training-versus-evaluation flag. module.eval changes module behavior such as dropout or running-statistic use and does not by itself disable autograd. Choose mode and module state independently according to the actual computation.

Check a small smooth derivative independently

The standard-library companion calculates the analytic derivative and a central finite difference of the same quadratic. Its tolerance check verifies that scalar calculation at a smooth point. It does not validate PyTorch’s engine or establish correctness for a network with branching, masking, or nondifferentiable operations. Numerical checks need suitable precision, perturbation size, and deterministic inputs; extremely tiny differences can suffer cancellation. If a framework gradient is absent, first inspect whether a detach, plain-number conversion, no-grad context, or unsupported path severed the intended dependency before altering the optimizer.

python Executed deterministic standard-library companion

def loss(w):
    return 0.5*(3*w-1)**2
w, h = 2.0, 1e-5
analytic = 3*(3*w-1)
central = (loss(w+h)-loss(w-h))/(2*h)
print(f"analytic={analytic:.6f}")
print(f"central={central:.6f}")
print("within tolerance", abs(analytic-central) < 1e-8)

Expected output

analytic=15.000000
central=15.000000
within tolerance True

The executed companion checks one smooth scalar derivative independently; it does not exercise autograd graphs, gradient buffers, or PyTorch context managers.

Keep differentiation and mutation visibly ordered

Finish the derivatives needed from the forward state before mutating parameters or saved values. Logging loss.item after constructing the objective is different from replacing a needed tensor expression with a Python number before backward. Detaching deliberately stops a gradient path, and it is not a substitute for copying independent storage. Validate which named parameters should have gradients and which are intentionally frozen. The framework program remains illustrative with derived outputs because torch is unavailable; the executed companion supports arithmetic reasoning only and makes no graph-memory, framework, model-training, or device-performance claim.

Common errors and fixes

A second step unexpectedly doubles its gradient

Why it happens: Leaf buffers accumulated because the intended reset boundary was omitted.
How to fix it: Clear gradients at the declared accumulation boundary before the next independent update.
Prevent it next time: Test fresh-graph derivatives and buffer reset separately.

Calling eval is assumed to disable gradients

Why it happens: Module behavior was confused with autograd recording.
How to fix it: Choose eval state and an appropriate gradient context independently.
Prevent it next time: Assert both module training state and intended gradient presence.

The loss is a number with no backward path

Why it happens: A detach or plain-number conversion removed the dependency.
How to fix it: Keep the differentiable tensor expression until differentiation is complete.
Prevent it next time: Test a known parameter-to-objective path before adding logging or wrappers.

Check your understanding

1. Why does the second backward in the fixture produce thirty rather than fifteen?

Show worked solution

The second objective is computed from a fresh graph at the same parameter value, so its derivative is another fifteen. PyTorch adds that contribution to the existing fifteen in the leaf .grad field. No optimizer step changed w. Resetting the buffer before the fresh backward would instead leave one contribution.

2. A module is in eval mode but its floating parameters require gradients. Can an ordinary forward still build a gradient graph?