Deep Learning / Tensor and learning foundations / Lesson 5

Gradient descent and numerical checks

Use a controlled quadratic to distinguish derivative correctness from step-size behavior.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Predict when a scalar update contracts error.
Compare an analytic derivative with central differences.
Interpret optimization traces without universal tuning rules.

An optimizer uses a gradient to choose a step

Gradient descent updates a parameter by subtracting a learning rate times the gradient. For L(w)=(w-3)^2/2, the derivative is w-3 and the error after one update is (1-eta) times its previous value. In this scalar quadratic, convergence requires 0<eta<2. This interval follows from the stated curvature of one, and must not be advertised as a general learning-rate recommendation for neural networks. Rescaling the objective or changing curvature changes the stable range. An optimizer transforms derivative information into a trajectory; correct gradients alone do not ensure that every chosen step decreases loss.

Inspect a trajectory rather than one endpoint

The first fixture starts at zero and applies four updates with eta=0.5. The distance to three halves at each step, so the printed loss shrinks predictably. An update with eta=2 would oscillate between equally distant points here, and a larger value would expand the error. Real objectives can have different curvature in different directions, noisy batches, and nonconvex regions. A trace should record the objective definition, step number, learning rate, and any change in reduction. Compare such traces before deciding whether slow progress comes from scale, derivatives, or an incompatible schedule.

python Deterministic standard-library calculation

w = 0.0
for step in range(1, 5):
    gradient = w - 3.0
    w -= 0.5 * gradient
    loss = 0.5 * (w - 3.0)**2
    print(f"{step}: w={w:.4f}, loss={loss:.6f}")

Expected output

1: w=1.5000, loss=1.125000
2: w=2.2500, loss=0.281250
3: w=2.6250, loss=0.070312
4: w=2.8125, loss=0.017578

This deterministic quadratic permits an exact contraction argument; the loss is measured after each update.

Numerical differentiation is an independent check

A central difference approximates a derivative by evaluating f(w+h) and f(w-h) and dividing their difference by 2h. It is useful for a tiny smooth deterministic objective because it checks the whole forward calculation independently of the derivative formula. The second fixture compares it with the known derivative at w=1. Numerical differentiation is expensive for many parameters and is not how ordinary reverse-mode autodiff trains a network. Select a point away from activation kinks, disable stochastic transformations, use adequate precision, and compare with a stated absolute and relative tolerance rather than exact floating-point equality.

python Deterministic standard-library calculation

def loss(w):
    return 0.5 * (w - 3.0)**2
w, h = 1.0, 1e-5
finite_difference = (loss(w+h) - loss(w-h)) / (2*h)
analytic = w - 3.0
print(f"analytic={analytic:.6f}")
print(f"central={finite_difference:.6f}")
print(abs(analytic - finite_difference) < 1e-8)

Expected output

analytic=-2.000000
central=-2.000000
True

The tolerance verifies this smooth scalar fixture without claiming exact binary floating-point equality.

Choose a useful perturbation scale

A very large finite-difference step measures nonlocal curvature rather than a local derivative. A very small step subtracts nearly equal floating-point values and can amplify cancellation. There is no one step size that is best for every dtype, parameter scale, or objective. Check several nearby perturbations on representative coordinates and inspect both loss values when a comparison fails. For nondifferentiable points, a symmetric finite difference may disagree with a framework subgradient convention without indicating a faulty implementation. Separate these mathematical cases from genuine path breaks or missing branch contributions before editing the model.

Record the update state as part of the experiment

Momentum and adaptive methods introduce state in addition to parameters; restarting only the weights can therefore change the trajectory. Training reproducibility also depends on sample order, random operations, numerical precision, and software behavior. Keep a small deterministic baseline that can be rerun before introducing these sources of variation. Monitor parameter updates relative to parameter scale, gradient finiteness, and the intended validation measure. Reducing the learning rate can be an experiment when steps diverge, but it cannot repair swapped labels or an incorrect loss. Make one hypothesis-driven change at a time and preserve the evidence that motivated it.

Common errors and fixes

A correct derivative still produces increasing loss

Why it happens: The step is too large for the local curvature or the objective changed.
How to fix it: Reproduce a deterministic trace and test a justified smaller step.
Prevent it next time: Log objective scaling and post-update loss on a tiny fixture.

A gradient check fails only at a ReLU boundary

Why it happens: The sampled point is not differentiable in the ordinary sense.
How to fix it: Check a nearby smooth point and document the chosen subgradient.
Prevent it next time: Include smooth and kink cases with different expectations.

A resumed run takes a different next step

Why it happens: Optimizer state or sample order was not restored.
How to fix it: Restore the full relevant state and input position.
Prevent it next time: Define resume equivalence explicitly and test the next update.

Check your understanding

1. For the stated quadratic, what happens with eta=1 and with eta=2?

Show worked solution

At eta=1 the error multiplier is zero, so the next parameter is exactly three in real arithmetic. At eta=2 the multiplier is minus one, so an initial nonzero error alternates sign without shrinking. These conclusions use this objective and do not transfer unchanged to other curvatures.

2. A derivative check worsens when h changes from 1e-5 to an extremely tiny value. What should you inspect?