Investigate restored-model state and mode disagreement

Understand the likely causes

• Possible causes include a state mapping retaining live references, omitted persistent buffers, partial key loading, an incompatible architecture or label mapping, dropout or running statistics in the wrong mode, or a consumer using another package. Matching tensor sizes or restricted loading options alone cannot distinguish these cases.

1. Define the intended restoration boundary

Decide whether the artifact is for inference or exact training continuation. Record feature order, units, learned preprocessing, architecture, state keys, output meaning, module mode, gradient context, and supported runtime. A training continuation can additionally need optimizer moments, scheduler counters, random state, input order, and accumulation progress. Use a self-created synthetic artifact when building the reproduction. Do not open an unknown executable object format merely to inspect it in a process with valuable access. This guide executes no torch loader and does not claim that a file’s origin or compatibility was verified.

2. Compare snapshots and persistent state

A state_dict describes registered parameters and persistent buffers, not only the optimizer’s trainable list. Its tensor references also require a deliberate snapshot policy when retaining an earlier best state. The companion demonstrates the analogous problem with nested lists: a shallow mapping remembers a weight of three after the live value changes, while the independent snapshot retains two. Scores nine and six then reflect different stored functions, not random inference. In a supported torch investigation, compare actual named values and buffer policy, not just an object variable called best_model.

import copy
live = {"weight":[2.0],"center":[1.0]}
remembered = dict(live)
snapshot = copy.deepcopy(live)
live["weight"][0] = 3.0
def score(state,x):
    return state["weight"][0]*(x-state["center"][0])
print("remembered weight", remembered["weight"][0])
print("snapshot weight", snapshot["weight"][0])
print("scores", score(remembered,4.0),score(snapshot,4.0))

remembered weight 3.0
snapshot weight 2.0
scores 9.0 6.0

3. Check mode and input interpretation independently

eval changes only modules with mode-dependent behavior and does not disable gradients itself. no_grad or inference_mode controls another boundary, while a fixed requires_grad policy does not freeze running buffers or stochastic training behavior automatically. Compare the same raw observation through both preprocessing paths before blaming the loaded weights. A class-order or unit mismatch can keep every tensor shape valid. Verify strict or deliberately partial key handling and the actual consumer version. For torch.load, preserve explicit supported options and trusted provenance; do not remove restrictions reflexively when a format is unsupported.

4. Verify behavior and document pending evidence

After repairing the first mismatch, run known ordinary and boundary inputs through the complete original and restored functions under the declared modes. Compare named state, output shape, scores, and downstream decisions with justified tolerances. If exact training resumption is required, compare the next update rather than treating one inference match as sufficient. An additional compile or export path needs its own capture and compatibility evidence. Keep the original failure fixture and record any invalidated assessment. A successful structural round trip does not establish statistical usefulness, future portability, or production performance.

Keep it from recurring

• Preserve immutable snapshots, required buffers, architecture and semantic metadata, and explicit mode policies together. Test known-input restoration before long experiments, keep loading trust separate from behavioral acceptance, and state the exact versions and runtime checks actually performed.