Diagnose resume and frozen-adaptation state changes

Understand the likely causes

• Common causes include omitted optimizer moments or schedule counters, changed input order, an incomplete accumulation window, mutable normalization buffers, low-precision rounding, a different base artifact, or stale preprocessing-dependent caches. A successful load and equal parameter shapes do not distinguish these causes.

1. Specify the expected continuation

Record the exact pre-update parameter and buffer state, next synthetic input, objective reduction, optimizer rule, schedule event, and relevant random or data-position state. Decide whether the checkpoint was taken at a clean optimizer boundary. If it was taken during accumulation, restoring only weights and moments omits the partial gradient numerator and denominator. Create an uninterrupted reference for one transition before investigating a long divergent curve. Avoid using arbitrary executable object loading to inspect unknown artifacts; a small trusted numeric fixture is enough to test the state principle.

2. Compare forward, derivative, and optimizer results

First compare preprocessing and forward outputs. If they agree, compare the objective and derivatives, then the optimizer state and resulting parameter change. The fixture shows two restorations with identical starting weights and gradients but different velocity. Their next parameters differ, locating the discrepancy inside the update rule rather than the representation. If the first discrepancy instead appears in forward outputs, inspect mode, buffers, preprocessing, base identity, and precision before changing optimizer settings. Keep every comparison tied to the same original state so a partial update cannot contaminate later checks.

def next_step(weight, velocity, gradient):
    velocity = 0.8*velocity + gradient
    return weight - 0.1*velocity, velocity
reference = next_step(2.0, 1.0, -2.0)
weights_only = next_step(2.0, 0.0, -2.0)
print("full state", tuple(round(x,2) for x in reference))
print("weights only", tuple(round(x,2) for x in weights_only))
print("same", reference == weights_only)

full state (2.12, -1.2)
weights only (2.2, -2.0)
same False

3. Audit what the freeze policy actually preserves

List trainable weights, persistent buffers, random-layer behavior, and caches separately. A frozen parameter policy does not necessarily stop running statistics or stochastic training behavior. Compare named snapshots before and after the controlled step, and confirm that only the intended head or adaptation factors change. For a low-rank update, verify the base weight identity and factor orientation; compatible dimensions can still apply a transposed or mismatched update. Invalidate cached features when preprocessing or the encoder changes, even if the cache remains readable and its row count matches.

4. Restore the full contract and retest boundaries

Repair the earliest identified mismatch, restore the required state, and rerun the one-transition comparison with the declared exactness or tolerance. Check schedule indexing and whether a skipped nonfinite update should advance the clock under the chosen policy. Then compare several further transitions and a fixed inference fixture, including class order and feature units. If exact continuation is unsupported, state that limitation and evaluate the intended statistical alternative rather than claiming equivalence. No amount of matching artifact hashes proves that the underlying objective or evaluation is appropriate.

Keep it from recurring

• Save checkpoints at documented boundaries with configuration and schema identity. Test resume behavior before long experiments, keep caches keyed to representation and preprocessing identity, and separate inference artifacts from training-continuation artifacts so their different state requirements remain visible.