PyTorch / Training state and architecture interfaces / Lesson 12

Randomness, reproducibility, and resumption

Specify the state and comparison required for replay instead of treating one seed as a complete experiment.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish a seed from an advanced generator state.
Separate stochastic module behavior from gradient recording.
Define exact, numerical, and statistical reproducibility boundaries.

A seed identifies one starting random state

Random initialization, dropout, sampling, data augmentation, and worker processes can consume random draws in different places. Setting a seed does not capture how many draws have already occurred or every source used by a program. An explicit torch.Generator can give a supported operation a deliberate stream without relying only on global state. Keep its device and the operation’s generator support consistent. The illustrative fixture stays on CPU and checks continuation through get_state and set_state. It does not claim a universal printed random sequence across releases, platforms, or different generator libraries.

Replay compares a defined next transition

After four draws, the program saves the generator state, makes three more draws, restores the state, and requests the same three draws again. The expected equality concerns that same generator configuration and operation sequence. A checkpoint for training can additionally need model parameters, persistent buffers, optimizer moments, scheduler state, input position, and partially accumulated gradients. Restoring the initial seed is not generally the same as restoring a generator midway through a run. Define a clean checkpoint boundary or explicitly preserve partial state, then compare the next intended update with uninterrupted execution.

python Illustrative CPU PyTorch 2.14 API program; torch unavailable, expected output derived

import torch
from torch import nn
generator = torch.Generator(device="cpu").manual_seed(17)
first = torch.rand(4,generator=generator,dtype=torch.float64,device="cpu")
state = generator.get_state()
next_a = torch.rand(3,generator=generator,dtype=torch.float64,device="cpu")
generator.set_state(state)
next_b = torch.rand(3,generator=generator,dtype=torch.float64,device="cpu")
dropout = nn.Dropout(p=1.0)
values = torch.ones(3,dtype=torch.float64,device="cpu")
dropout.train()
training = dropout(values)
dropout.eval()
evaluation = dropout(values)
print("same next draws", torch.equal(next_a,next_b))
print("first draw count", first.numel())
print("mode outputs", training.tolist(), evaluation.tolist())

Expected output

same next draws True
first draw count 4
mode outputs [0.0, 0.0, 0.0] [1.0, 1.0, 1.0]

Generator restoration defines a same-environment next-draw expectation, while dropout probability one exposes mode behavior without printing random values. No torch RNG or module was executed.

Module mode and randomness interact through specific layers

The fixture uses Dropout with probability one to produce deterministic extreme outputs for teaching: zeros in training mode and unchanged ones in evaluation mode. This is not a useful universal training configuration. eval controls the layer’s documented mode behavior, while no_grad controls graph recording independently. A frozen parameter set can still produce variable outputs under stochastic training behavior or changing normalization buffers. Record the relevant module state as part of inference and evaluation contracts instead of assuming that requires_grad=False makes a whole model behaviorally frozen.

Reproducibility has several legitimate meanings

Bitwise identity, values within a justified tolerance, the same downstream decision, and similar distributions over independent runs are different goals. PyTorch documents limits to reproducibility across environments and releases; selecting deterministic algorithms where supported can change execution or raise errors for unsupported operations. It does not automatically control data provenance, external libraries, concurrency, or the scientific validity of a split. The standard-library companion illustrates its own generator continuation and deliberately makes no claim that Python random matches torch. Record the actual runtime, libraries, hardware, data, and comparison rule when performing real experiments.

python Executed deterministic standard-library companion

import random
generator = random.Random(17)
first = [generator.random() for _ in range(4)]
state = generator.getstate()
next_a = [generator.random() for _ in range(3)]
generator.setstate(state)
next_b = [generator.random() for _ in range(3)]
print("same next draws", next_a == next_b)
print("first draw count", len(first))
print("all in unit interval", all(0.0 <= value < 1.0 for value in first))

Expected output

same next draws True
first draw count 4
all in unit interval True

This executed Python generator example checks its own state continuation only. Its sequence is not asserted to match PyTorch or any other generator.

Preserve selection history when repeating experiments

A favorable seed chosen after inspecting many validation results is a selection decision, not automatically a typical or independent assessment. Report the intended replication design, number of runs, and relevant variation rather than only the best result. Compare resumed behavior at a small known boundary before a long run, including data order and mode. The framework program remains illustrative with derived invariants because torch is unavailable. Only Python’s standard-library generator was executed; no PyTorch random stream, dropout, deterministic-kernel setting, checkpoint continuation, worker process, or model-training reproducibility was tested.

Common errors and fixes

A resumed run repeats the beginning of its random stream

Why it happens: Only the original seed was restored, not the saved continuation state.
How to fix it: Restore the relevant generator and execution state at the chosen boundary.
Prevent it next time: Compare the next draws and next update with uninterrupted execution.

A supposedly frozen model still has stochastic outputs

Why it happens: Parameter gradient flags were mistaken for module mode.
How to fix it: Specify parameter, buffer, and stochastic-mode policies separately.
Prevent it next time: Test repeated inference on a fixed input under the intended mode.

One repeated seed is described as universal reproducibility

Why it happens: Environment, algorithm, and experiment boundaries were omitted.
How to fix it: State the exact supported comparison and its limitations.
Prevent it next time: Record versions, inputs, generators, and actual tests performed.

Check your understanding

1. Why does restoring the initial seed after four draws not generally resume at the fifth draw?

Show worked solution

A seed reconstructs the stream’s starting state. Without consuming the same first four draws again, the next output comes from the beginning. A saved generator state after those draws describes the continuation boundary directly. Other random sources and training state still require their own capture.

2. Does the fixture prove PyTorch and Python random produce identical values for seed seventeen?