Deep Learning / Architectures and training dynamics / Lesson 8

Minibatches, accumulation, and schedules

Preserve objective weighting and update counts when changing how examples are processed.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Calculate sample-weighted gradient accumulation.
Distinguish epochs, microbatches, and optimizer updates.
Define schedule indexing and resume state precisely.

A minibatch estimates a declared objective

For a mean per-example objective, a minibatch gradient averages the derivatives of its examples at the current parameter state. Sampling policy affects this estimate: balanced-class sampling, repeated examples, or a truncated last batch can change effective weights. A larger batch changes noise, work per update, and often update counts per epoch. It does not automatically justify multiplying the learning rate by a universal factor. Record the sampling unit, replacement policy, final-batch policy, and objective denominator before comparing two runs. An epoch is one defined pass through data, while an optimizer step is one parameter update.

Accumulation combines derivatives before mutation

Gradient accumulation processes several microbatches while keeping parameters fixed, then combines their contributions before one update. For unequal microbatch sizes, an average of their means differs from the mean over all examples. The first fixture exposes that difference using three scalar gradients grouped into batches of two and one. Reproduce the declared numerator and denominator rather than dividing by a fixed number of accumulation calls. Accumulation can match a larger-batch mean under suitable conditions, but batch-dependent normalization, stochastic operations, and differing random draws can prevent exact equivalence of the full computation.

python Deterministic standard-library calculation

gradients = [[1.0, 3.0], [5.0]]
weighted = sum(sum(g) for g in gradients) / sum(map(len, gradients))
unweighted = sum(sum(g)/len(g) for g in gradients) / len(gradients)
print(f"sample mean={weighted:.1f}")
print(f"mean of batch means={unweighted:.1f}")

Expected output

sample mean=3.0
mean of batch means=3.5

Unequal microbatches require sample-count weighting to reproduce a sample mean at one parameter state.

Do not confuse accumulation with sequential updates

If parameters change after every microbatch, later derivatives are evaluated at new states. That is a different optimization trajectory even if the same examples are seen. Stateful optimizers may also update their moments and step counters at each call, amplifying the distinction. Zero gradient buffers at the intended accumulation boundary, apply any clipping to the intended combined gradient, and advance the schedule at the documented event. A final partial accumulation window needs an explicit normalization policy. Simply skipping its update or using the full-window denominator changes how those examples contribute.

A schedule is a function of an explicit clock

A learning-rate schedule can depend on optimizer updates, epochs, processed examples, or a validation event. The second fixture defines six optimizer-update rates with a two-update warmup and a cosine decline. Its endpoint and repeated peak follow the chosen indexing and are not a claimed optimal recipe. Calling a schedule once per microbatch instead of once per optimizer update advances it too quickly when accumulating. Save its counter and any metric-dependent state along with optimizer state. Check the first, transition, and final values directly before running a long experiment.

python Deterministic standard-library calculation

from math import cos, pi
base, warmup, total = 0.1, 2, 6
for step in range(total):
    if step < warmup:
        rate = base * (step + 1) / warmup
    else:
        progress = (step - warmup) / (total - warmup - 1)
        rate = base * 0.5 * (1 + cos(pi * progress))
    print(f"step={step}, lr={rate:.3f}")

Expected output

step=0, lr=0.050
step=1, lr=0.100
step=2, lr=0.100
step=3, lr=0.075
step=4, lr=0.025
step=5, lr=0.000

The explicit six-update schedule defines its indexing and reaches zero at the final listed update.

Compare runs on more than an epoch label

Two experiments with the same epoch count can perform different numbers of optimizer updates after a batch-size change. Compare processed examples, updates, objective weighting, and the schedule at a matching point. Throughput and memory constraints matter, but measure them in the actual execution environment; this course makes no device performance claims. Preserve sample identifiers or a deterministic order fixture so a resumed calculation can recover the next batch. When results change, first determine whether the mathematical training procedure changed before attributing the difference to hardware, randomness, or model capacity.

Common errors and fixes

Accumulation disagrees with a full-batch mean

Why it happens: Unequal microbatches were averaged with equal weight.
How to fix it: Weight each contribution by its valid sample count.
Prevent it next time: Test a partial final microbatch against an explicit full numerator.

Warmup ends before the first intended update

Why it happens: The clock advanced on every microbatch.
How to fix it: Advance it at the documented optimizer-step boundary.
Prevent it next time: Log both microbatch and optimizer counters.

The final examples have weaker influence

Why it happens: A partial window used a full-window divisor or was dropped.
How to fix it: Normalize by the actual valid count and apply the intended update.
Prevent it next time: Include uneven dataset lengths in the training-loop fixture.

Check your understanding

1. Microbatches of sizes two and one have mean gradients 2 and 5. What is their combined sample mean?

Show worked solution

The numerator is 2*2+1*5=9 and the denominator is three, giving three. Averaging 2 and 5 gives 3.5 and gives the single example as much weight as the first two combined. Parameters must remain fixed while those derivatives are computed.

2. A run accumulates four microbatches per update and uses a 100-update warmup. When should the warmup counter advance?