Deep Learning / Architectures and training dynamics / Lesson 11

Attention, masks, and position

Trace scaled dot-product attention and identify the information that masks and position mechanisms supply.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Calculate attention scores, weights, and values.
Apply masks before normalization.
Distinguish content interactions from positional information.

A query retrieves a weighted combination

Attention compares a query vector with key vectors, turns compatibility scores into weights, and combines corresponding value vectors. Keys determine where to assign weight; values determine what is combined. Their dimensions therefore have different contracts even when a model chooses equal widths. In self-attention these vectors derive from the same sequence through separate transformations; cross-attention obtains queries and key/value pairs from different sources. Neither operation inherently enforces causal availability. The permitted context must be defined separately according to the prediction task and encoded in the mask.

Scale scores and inspect one complete query

Scaled dot-product attention divides a query-key dot product by the square root of the key dimension before softmax. Under common variance assumptions this moderates score growth with dimension; it is not a guarantee against all saturation. The first fixture uses two-dimensional fixed vectors and scalar values so every stage can be checked. Its weighted output lies between the two values because the weights are nonnegative and sum to one. A later output projection or residual addition need not preserve that scalar range. Multihead attention repeats related operations in several learned subspaces and then combines their results.

python Deterministic standard-library calculation

from math import exp, sqrt
query = [1.0, 0.0]
keys = [[1.0, 0.0], [0.0, 1.0]]
values = [2.0, 8.0]
scores = [sum(a*b for a,b in zip(query,k))/sqrt(2) for k in keys]
m = max(scores)
weights = [exp(s-m)/sum(exp(t-m) for t in scores) for s in scores]
print([round(w,6) for w in weights])
print(f"context={sum(w*v for w,v in zip(weights,values)):.6f}")

Expected output

[0.669762, 0.330238]
context=3.981431

One query produces normalized weights over two keys and a weighted scalar value; there is no learned model or tokenization.

Mask the set of allowed dependencies

A padding mask excludes positions that are storage artifacts, while a causal mask excludes future positions for a particular query. The second fixture removes a disallowed score before normalization; masking probabilities afterward without renormalizing changes their sum and therefore the weighted computation. A row with no allowed key needs a defined handling policy, because softmax of entirely excluded values is not a meaningful distribution. Mask polarity and shape differ across libraries, so test actual weights for a tiny case rather than inferring semantics from a Boolean variable name.

python Deterministic standard-library calculation

from math import exp
scores = [1.0, 2.0, 100.0]
allowed = [True, True, False]
maximum = max(s for s, ok in zip(scores, allowed) if ok)
unnormalized = [exp(s-maximum) if ok else 0.0 for s,ok in zip(scores,allowed)]
weights = [v/sum(unnormalized) for v in unnormalized]
print([round(w,6) for w in weights])
print("masked weight", weights[2])

Expected output

[0.268941, 0.731059, 0.0]
masked weight 0.0

The disallowed high score is excluded before normalization and receives exactly zero weight in this finite fixture.

Content matching alone does not encode order

Without a position mechanism, unmasked self-attention has a permutation-equivariant structure: permuting input positions and their corresponding queries, keys, and values permutes outputs. Sequence order needs information supplied by position embeddings, relative-position rules, rotary transformations, masks, or another architectural mechanism. These mechanisms are not interchangeable decorations; they change how relative or absolute location enters the score or representation. A causal mask supplies an ordering constraint but does not by itself make every desired distance relation explicit. State the maximum or extrapolated context assumptions and evaluate length changes rather than assuming unlimited sequence generalization.

Account for context cost and interpretation limits

Conventional dense self-attention forms interactions between every pair of sequence positions, so a score matrix grows quadratically in sequence length. Alternative implementations can reduce materialization or change the interaction pattern; actual memory and runtime depend on the algorithm, shapes, and hardware. This course measures none of those systems. Attention weights describe one internal weighted combination and are not automatically a causal explanation of a final prediction. Test dependency masks and relevant controlled changes, and keep the distinction between numerical correctness, architectural possibility, and demonstrated task performance explicit.

Common errors and fixes

A future target affects an earlier prediction

Why it happens: The causal mask is missing, inverted, or broadcast on the wrong axes.
How to fix it: Inspect the allowed key set for each query and correct mask semantics.
Prevent it next time: Assert zero weight for future keys in a tiny unequal-length fixture.

Attention weights sum to less than one after masking

Why it happens: Probabilities were zeroed after softmax without renormalization.
How to fix it: Exclude scores before normalization using the documented operation.
Prevent it next time: Check row sums and all-masked behavior.

Reordering tokens preserves an unintended behavior

Why it happens: No suitable position information entered the computation.
How to fix it: Add a justified positional mechanism and evaluate order-sensitive cases.
Prevent it next time: Maintain pairs with identical content and different order.

Check your understanding

1. If two allowed keys have equal scores and values two and six, what is the context?

Show worked solution

Softmax assigns one half to each allowed key, so the weighted value is four. Adding a common constant to both scores preserves this answer. A third masked key must contribute zero regardless of its stored score or value.

2. Why does an attention visualization not by itself explain the final decision causally?