Investigate batch, padding, and inference disagreement

Understand the likely causes

• Possible causes include a wrong batch or feature axis, batch-derived statistics in the wrong mode, dropout still active, missing normalization buffers, unmasked padding, cross-example attention, recurrent state leakage, or changed preprocessing and label order. These possibilities can produce similar final symptoms and require separate boundary tests.

1. Preserve one observation and vary one context

Keep the original feature values, units, identifier, preprocessing output, and model state fixed. Compare the observation alone, with two deliberately different companion sets, and under a permitted permutation of the same batch. For sequences, compare the same valid prefix with different padding lengths and conspicuous padding values. Record intermediate shapes and known sentinel values so a transpose or flattening error becomes visible. Use the documented numerical tolerance rather than display-rounded equality. If the intended model legitimately uses shared context, define exactly which dependencies are allowed before interpreting a difference as a defect.

2. Inspect mode-dependent computation and buffers

Compare actual normalization axes, stored statistics, and random-layer behavior. The fixture recomputes mean and variance from the batch, so the first observation has a different result alone. Two displayed values can round to the same number while their internal values differ; compare the appropriate unrounded quantities. A real framework may reject certain training-mode singleton shapes rather than run this formula. This scalar program therefore explains the mathematical coupling only. Verify the framework inference mode and the restoration of persistent buffers using that framework’s documented API, not a guessed flag.

from math import sqrt
def batch_normalized_first(values):
    mean = sum(values)/len(values)
    variance = sum((v-mean)**2 for v in values)/len(values)
    return (values[0]-mean)/sqrt(variance+1e-5)
print(f"companions A={batch_normalized_first([2.0,4.0]):.4f}")
print(f"companions B={batch_normalized_first([2.0,20.0]):.4f}")
print(f"alone={batch_normalized_first([2.0]):.4f}")

companions A=-1.0000
companions B=-1.0000
alone=0.0000

3. Trace masks at every dependency boundary

For recurrent models, ensure padding does not update a state later used as a prediction. For pooled outputs, exclude invalid positions from both the numerator and denominator. For attention, check the allowed key positions for every query and verify the mask polarity and broadcast shape. A padding loss mask cannot remove an earlier dependency already mixed into valid representations. Cross-example mixing can also arise from applying attention over the batch axis. Keep sequence identity and reset rules explicit when evaluating interleaved streams.

4. Compare the complete restored inference function

After repairing the identified boundary, compare preprocessing output, selected intermediate tensors, final scores, class names, and threshold decisions with the reference. Test restoration with the architecture and label mapping that produced the artifact. A state dictionary can have compatible sizes while assigning a different meaning to an input coordinate. Include a threshold-near fixture if small numeric changes affect actions. Preserve the companion and padding tests as focused regression checks; passing one ordinary batch is weak evidence for a contract that must hold across batching choices.

Keep it from recurring

• Version axis contracts, normalization buffers, input transformations, and mode settings with the model. Keep non-square batches and unequal sequence lengths in a small consistency suite, and verify unsupported empty or all-masked cases under an explicit policy.