Deep Learning / Representation and acceptance evidence / Lesson 17

Reproducibility and checkpoint contracts

Specify what a resumed calculation must preserve and verify it with the next update.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish repeatability, resume equivalence, and statistical replication.
Identify parameter, optimizer, data-order, and mode state.
Verify a deterministic round trip without unsafe object loading.

Name the kind of reproducibility required

Bitwise repeatability, numerically close results, equivalent next updates, and similar distributions over independent runs are different goals. A fixed seed may reproduce random draws under one software configuration, but it does not force all device kernels or parallel reductions to behave identically. Declare tolerances and the environment boundary before calling a run reproducible. For scientific comparisons, variation over legitimate independent runs may matter more than one repeated draw. This lesson uses deterministic scalar programs so the resume test has a clear exact expectation; it makes no cross-platform neural-training reproducibility guarantee.

A checkpoint is more than learned weights

Resuming training can require parameters, optimizer moments, schedule counters, random states, data-order position, accumulation progress, precision-scaler state, and mutable model buffers. Inference needs a different subset plus architecture, preprocessing, label mapping, and mode. Decide what the artifact is intended to do before choosing its contents. The first fixture serializes one scalar momentum state in memory and compares the next update before and after restoration. JSON is sufficient for this tiny numeric mapping, but real tensor artifacts require framework-specific formats and a reviewed loading policy described in their dedicated tracks.

python Deterministic standard-library calculation

import json
state = {"weight": 1.0, "velocity": 0.5, "step": 3}
def update(s):
    gradient = s["weight"] - 3.0
    velocity = 0.9*s["velocity"] + gradient
    return {"weight": s["weight"]-0.1*velocity,
            "velocity": velocity, "step": s["step"]+1}
restored = json.loads(json.dumps(state, sort_keys=True, allow_nan=False))
print(update(state) == update(restored))
print(update(restored))

Expected output

True
{'weight': 1.155, 'velocity': -1.55, 'step': 4}

The same next update follows after a complete in-memory JSON round trip of this deterministic scalar momentum state.

Test the next transition rather than only the file

A load operation can succeed while omitting a counter, changing a dtype, or attaching the wrong optimizer state. Compare the next input, forward result, loss, gradient, and update with a continued reference calculation under the declared tolerance. A checkpoint taken halfway through gradient accumulation needs either the partial accumulated state or a documented restart boundary. Random transforms and shuffled input order also require the correct continuation state if exact next-step equivalence is intended. Preserve enough metadata to detect an incompatible model or dataset instead of silently accepting matching tensor sizes.

Content identity is useful but limited evidence

The second fixture hashes a deliberately canonical JSON representation of a simple feature contract. Object-key order is ignored, while feature-list order remains meaningful and changes the digest. This distinguishes artifact identity from correctness: a hash can identify a consistently wrong contract. An untrusted file and an untrusted matching digest do not establish trusted origin. Avoid loading unknown executable object formats merely to inspect a checkpoint. Keep sensitive data and credentials out of teaching artifacts; this fixture contains only synthetic names and small numbers and performs no file or network operations.

python Deterministic standard-library calculation

import hashlib, json
a = {"features": ["length", "width"], "scale": [2, 4]}
b = {"scale": [2, 4], "features": ["length", "width"]}
c = {"features": ["width", "length"], "scale": [2, 4]}
def digest(contract):
    raw = json.dumps(contract, sort_keys=True, separators=(",", ":"), allow_nan=False).encode()
    return hashlib.sha256(raw).hexdigest()
print("key order ignored", digest(a) == digest(b))
print("feature order detected", digest(a) != digest(c))

Expected output

key order ignored True
feature order detected True

Canonicalization is explicitly defined for these simple JSON values; a digest checks identity, not semantic correctness or trusted origin.

Record enough context to interpret a result

Keep the objective, reductions, sample boundaries, initialization rule, configuration, software versions, and the exact validation selection rule together with the result. When two runs disagree, compare contracts and state transitions before blaming randomness. If only statistical replication is required, report variation and the number of runs without selecting the most favorable seed as if it were typical. If strict repeatability is required, test it in the supported environment and document its limits. The separate MLOps course handles operational artifact promotion and monitoring; this lesson establishes the mathematical and experimental meaning of checkpoint evidence.

Common errors and fixes

A loaded model resumes with a different trajectory

Why it happens: Weights were restored without the optimizer, counter, or input state.
How to fix it: Restore the state required by the declared resume boundary.
Prevent it next time: Compare the very next update against uninterrupted execution.

Equal tensor sizes hide an incompatible artifact

Why it happens: Feature order or architecture meaning changed.
How to fix it: Validate explicit schema and configuration identities.
Prevent it next time: Include known-input predictions and contract checks.

A matching hash is treated as proof of model correctness

Why it happens: Byte identity was confused with scientific validity or trusted provenance.
How to fix it: Verify the objective, origin, and evaluation separately.
Prevent it next time: State exactly what each integrity check establishes.

Check your understanding

1. In the scalar momentum fixture, why would restoring velocity as zero change the next update?

Show worked solution

The intended velocity is 0.9*0.5+(1-3)=-1.55, giving weight 1.155. Starting velocity at zero gives -2 and weight 1.2. Both calculations can load the same weight successfully, yet they are different training continuations.

2. Why must feature-list order remain significant when canonicalizing the contract?