Deep Learning / Tensor and learning foundations / Lesson 4

Chain rule and reverse accumulation

Trace a scalar computational graph and accumulate all paths that contribute to a parameter gradient.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Derive local and total derivatives.
Accumulate gradients through a shared node.
Distinguish differentiability from trainability.

A graph records how a value was computed

A computational graph represents operations and their dependencies. For prediction p = w*x + b followed by loss L = (p-y)^2/2, the forward pass computes p and the residual before the loss. The backward pass applies the chain rule in reverse dependency order. It needs relevant saved values from the forward calculation, not merely the final scalar loss. Automatic differentiation applies these local derivative rules to the executed computation; it does not estimate gradients by trying random perturbations, solve the optimization problem, or verify that the programmed objective matches the intended task.

Derive a gradient before automating it

The local derivative of the half-squared loss with respect to p is p-y. The derivative of p with respect to w is x, so dL/dw = (p-y)*x; dL/db = p-y. The first program makes these steps visible and performs one gradient-descent update. A gradient describes local sensitivity with other inputs held fixed. Its sign and size depend on the loss scaling, feature units, and current parameter values. A large gradient is not itself proof of an implementation defect, and a zero gradient is not by itself evidence that the whole task is solved.

python Deterministic standard-library calculation

x, y = 2.0, 5.0
w, b = 1.0, 0.0
prediction = w * x + b
residual = prediction - y
gw, gb = residual * x, residual
w -= 0.1 * gw
b -= 0.1 * gb
print(f"loss={0.5 * residual**2:.2f}, dw={gw:.2f}, db={gb:.2f}")
print(f"w={w:.2f}, b={b:.2f}")

Expected output

loss=4.50, dw=-6.00, db=-3.00
w=1.60, b=0.30

The forward residual is reused for both parameter derivatives; the update is distinct from differentiation.

Every downstream use contributes

If an intermediate value influences the loss along two branches, its total derivative is the sum of the branch contributions. For a=w*x and L=a*a+a, dL/da=2a+1. The two uses inside a*a each contribute a, and the direct a branch contributes one. The second fixture illustrates accumulation without a framework. Replacing addition with assignment when processing one branch loses other paths and silently produces an incorrect gradient. Shared parameters in recurrent networks and tied embeddings follow this same principle: reuse changes the number of contributions, not the identity of the parameter.

python Deterministic standard-library calculation

w, x = 2.0, 3.0
a = w * x
branch_square = 2 * a
branch_direct = 1.0
da = branch_square + branch_direct
dw = da * x
print(f"a={a:.1f}, loss={a*a+a:.1f}")
print(f"da={da:.1f}, dw={dw:.1f}")

Expected output

a=6.0, loss=42.0
da=13.0, dw=39.0

Both downstream branches contribute before multiplication by the local derivative x.

Differentiation follows the executed path

A program can contain control flow, nondifferentiable selections, integer indices, detached values, and operations with undefined derivatives. Automatic differentiation only follows dependencies supported by the chosen system. A hard threshold generally does not provide a useful derivative for learning a classifier score. A deliberate stop-gradient operation can be part of an algorithm, but accidentally converting a tracked value to a plain number can sever a needed path. State whether the intended derivative is through a data transformation, a model parameter, or a separate optimization procedure. Those are different mathematical questions even when the forward numbers coincide.

Separate graph lifetime, storage, and updates

Backward computation may release saved intermediates to save memory; repeated derivatives can require an explicit retained or rebuilt graph in a framework. Gradient buffers may accumulate across backward calls, whereas model parameters change only when an update is applied. These are separate lifecycles. Design a tiny derivative test before adding batching, stochastic layers, or stateful modules. Preserve the parameter value at which a derivative was computed until all dependent gradients are ready. The manual programs demonstrate reverse-chain arithmetic only; they do not emulate a complete autodiff engine, higher-order differentiation, graph storage policy, or framework buffer semantics.

Common errors and fixes

A shared parameter receives too small a gradient

Why it happens: One branch contribution overwrote another.
How to fix it: Add all contributions before applying an update.
Prevent it next time: Test a graph with at least two paths to the same parameter.

The loss changes but a parameter has no gradient

Why it happens: The computational path was detached or used a nondifferentiable decision.
How to fix it: Trace the parameter to the loss and retain differentiable score operations.
Prevent it next time: Check expected gradient presence on a tiny known graph.

Parameters change during the backward trace

Why it happens: The update happened before all gradients used the original forward values.
How to fix it: Finish the derivative calculation before mutating parameters.
Prevent it next time: Keep backward and optimizer steps visibly separate.

Check your understanding

1. For x=3, y=4, w=1, b=0 under half-squared error, calculate both gradients.

Show worked solution

The prediction is three and the residual is minus one. Therefore dL/dw=(-1)*3=-3 and dL/db=-1. The loss is one half. These derivatives describe the original parameters and should not be recomputed from a partially updated forward state.

2. Why is a shared weight not updated separately as each branch is visited?