Trace shape, broadcasting, and attention-mask errors

Understand the likely causes

• A target vector may broadcast against a prediction column into pairwise errors, a batch axis may be mistaken for sequence, or a Boolean mask may use the opposite convention from the receiving API. Padding can also affect representations before a loss-only mask is applied. These are separate information and alignment defects.

1. Write the complete axis and target contract

State what every dimension means at the dataset, collation, module, prediction, and unreduced loss boundaries. Use a non-square fixture and distinct observation ids so a transpose or cross-example combination becomes visible. For a class-index loss, distinguish the class axis from the example axis and verify label dtype and mapping. For regression, require the intended paired prediction and target shapes explicitly. Do not infer correct pairing from a scalar mean: aggregation can erase evidence that an extra axis was created. Inspect known values and identity as well as dimensions.

2. Compare paired and expanded arithmetic independently

The companion gives identical predictions and targets, so a legitimate paired squared-error mean is zero. A full matrix of all prediction-target pairs instead averages to 1.333333. That mismatch is possible even though both calculations are finite and have plausible numeric outputs. In a supported PyTorch reproduction, inspect the actual unreduced tensor and the operation that introduced the extra axis. Align the intended shape rather than reshaping blindly until a warning disappears. Keep class weights, ignored targets, valid lengths, and denominator policy visible when checking the resulting reduction.

predictions, targets = [1.0,2.0,3.0], [1.0,2.0,3.0]
paired = [(a-b)**2 for a,b in zip(predictions,targets)]
pairwise = [[(a-b)**2 for b in targets] for a in predictions]
blocked = [[j>i for j in range(3)] for i in range(3)]
wrong_direct = [j for j,value in enumerate(blocked[0]) if value]
converted = [j for j,value in enumerate(blocked[0]) if not value]
print(f"paired mean={sum(paired)/len(paired):.6f}")
print(f"pairwise mean={sum(map(sum,pairwise))/9:.6f}")
print("wrong direct first keys", wrong_direct)
print("converted first keys", converted)

paired mean=0.000000
pairwise mean=1.333333
wrong direct first keys [1, 2]
converted first keys [0]

3. Enumerate the allowed attention positions

Name the exact API. MultiheadAttention’s Boolean attn_mask uses True for blocked positions, while direct scaled_dot_product_attention uses True for participation. The fixture’s first blocked row incorrectly permits future keys one and two if passed unchanged to the direct rule; inversion permits only key zero. Float masks use additive score semantics and cannot be treated as arbitrary Boolean inversions. Verify broadcast shape and any padding-mask combination separately. A key mask does not automatically define invalid query outputs or their loss eligibility, and a fully excluded query needs an explicit supported policy.

4. Retest padding, mode, and information flow

Compare the same valid prefix under different padding lengths and values, and vary future values that must not affect a causal earlier output. Check module versus functional layouts, including head dimensions and batch_first scope. Direct attention dropout follows dropout_p, so eval on an enclosing module is not sufficient unless its wrapper changes that argument appropriately. Preserve the first repaired axis or dependency boundary as a regression fixture. Evaluate the whole affected calculation again; a corrected mask cannot retroactively validate an earlier report produced with leaked future context.

Keep it from recurring

• Keep asymmetric shape fixtures, identifiable sequence values, explicit mask polarity, and uneven valid lengths in component checks. Version the loss and information-flow contracts, and separate correct numeric execution from legitimate prediction-time context and external model quality.