MLOps / Prediction and evidence foundations / Lesson 5

Reproducible training boundaries

Define what a training rerun must preserve and how its input manifest explains the resulting artifact.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate deterministic replay from statistical replication.
Identify data, code, transforms, configuration, and environment inputs.
Use content identity within an explicitly canonical representation.

Reproducibility is a declared comparison

Exact bytes, close predictions, equivalent next updates, and similar performance over independent runs are different reproducibility goals. Choose the required comparison before evaluating a rerun. A seed is only one input to a stochastic procedure and does not capture data order, software, numerical libraries, or all device behavior. An operational replay may need to recover the artifact behind a particular decision, while a scientific replication may assess variation over legitimate independent runs. Do not call both goals identical or promise strict replay without testing the supported execution boundary.

A training request should name all relevant inputs

Record the dataset snapshot and eligibility policy, split manifest, fitted preprocessing definition, target mapping, code revision, parameters, dependency environment, and relevant randomness. Separate configuration values from mutable defaults that a later library release can change. If a process reads an external resource, its identity and availability become part of the reproducibility problem; this course avoids those resources entirely. Preserve the run’s input manifest before execution and link its outputs afterward. A path named latest_data is not an immutable snapshot and cannot explain which values an earlier training run actually consumed.

Start with a deterministic replay fixture

The first program fits a constant baseline by averaging three supplied targets and records their ids. Repeating that fully specified calculation produces an equal result. The fixture is intentionally simple enough to verify independently: the sum is twelve and the count is three. This checks a training procedure and manifest boundary without installing a framework or asserting that a larger neural procedure is deterministic. Compare learned preprocessing as well as final parameters when the actual procedure contains it. A rerun producing the same scalar can still differ in eligibility or data provenance, so values alone do not establish identical evidence.

python Deterministic offline standard-library fixture

def fit_constant(rows):
    assert rows and all("id" in row and "target" in row for row in rows)
    return {"kind":"constant", "value":sum(row["target"] for row in rows)/len(rows),
            "training_ids":[row["id"] for row in rows]}
rows = [{"id":"a", "target":2.0}, {"id":"b", "target":4.0}, {"id":"c", "target":6.0}]
first = fit_constant(rows)
second = fit_constant(rows)
print("constant", first["value"])
print("repeat equal", first == second)
print("training ids", first["training_ids"])

Expected output

constant 4.0
repeat equal True
training ids ['a', 'b', 'c']

This tiny deterministic training procedure is fully specified by supplied rows; its repeatability does not imply general framework or hardware repeatability.

Hash a defined representation rather than a vague name

The second program defines a canonical JSON encoding for small values and hashes it. Reordering mapping keys does not change identity, while changing one dataset value does. List order remains meaningful under this contract. A digest can support cache keys and integrity checks, but it does not prove semantic correctness, authorized origin, or sufficient provenance. Complex artifacts need explicit rules for types, nonfinite values, encoding, and file composition. Keep the actual manifest and artifact available under the intended retention policy; a hash with no retrievable content cannot reconstruct the experiment.

python Deterministic offline standard-library fixture

import hashlib, json
def identity(value):
    raw = json.dumps(value, sort_keys=True, separators=(",",":"), allow_nan=False).encode()
    return hashlib.sha256(raw).hexdigest()
a = {"code":"lab-r1", "dataset":[["a",2],["b",4]], "split":"split-v1"}
b = {"split":"split-v1", "dataset":[["a",2],["b",4]], "code":"lab-r1"}
c = {"code":"lab-r1", "dataset":[["a",2],["b",5]], "split":"split-v1"}
print("mapping order ignored", identity(a) == identity(b))
print("changed data detected", identity(a) != identity(c))

Expected output

mapping order ignored True
changed data detected True

Canonicalization is explicitly defined for these simple JSON values; the digest identifies the declared inputs, not their correctness or trusted origin.

Handle failure and resumption as part of the procedure

A failed training attempt should not leave a partial artifact indistinguishable from a completed one. Record status and publish outputs only after the declared validation boundary. A resumed optimizer can need more state than weights, including counters, moments, data position, and partial accumulation. Decide whether the supported behavior is restart from a clean boundary or exact continuation and test that choice. These fixtures execute only a local constant fit and JSON identity checks. They do not run a scheduler, restore framework checkpoints, build environments, or claim cross-platform training equivalence.

Common errors and fixes

A rerun uses different data under the same path

Why it happens: A mutable location was treated as a snapshot identity.
How to fix it: Preserve explicit content and eligibility identity for each run.
Prevent it next time: Reject manifests that rely only on a latest-style name.

Matching seeds are treated as complete replay evidence

Why it happens: Other stochastic, software, and input state was omitted.
How to fix it: Define and capture the relevant execution boundary.
Prevent it next time: Test the required output or transition comparison explicitly.

A partial output is mistaken for a completed model

Why it happens: Failure status and publication boundaries were unclear.
How to fix it: Keep incomplete artifacts distinct until required checks finish.
Prevent it next time: Test failure and restart behavior on a tiny local procedure.

Check your understanding

1. Why can two equal constant predictions fail to prove equivalent training evidence?

Show worked solution

Different datasets can have the same target mean, or an exclusion policy can change without changing that mean. Compare dataset identity, ids, eligibility, target definition, split, and procedure as well as the learned value. Equal outputs answer a narrower question than equal provenance.

2. What does the digest fixture prove when it detects the changed target?