PyTorch / Training state and architecture interfaces / Lesson 10

Embeddings, sequences, and lengths

Keep token identity, learned lookup state, padding, and recurrent layout contracts distinct.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Use integer ids with an explicit embedding vocabulary and padding policy.
Pool or pack only declared valid positions.
Read batch_first and recurrent hidden-state shapes according to the specific API.

An embedding table maps ids rather than numeric magnitudes

nn.Embedding stores learned rows and retrieves them using supported integer indices. The id is a vocabulary position, not a measured amount whose numeric distance defines similarity. Preserve the token-to-id mapping, unknown-category policy, and vocabulary version with the model. Gradients update selected floating table values rather than making the integer indices ordinary differentiable inputs. padding_idx supplies a specific gradient and initialization convention for one row; it does not automatically mask every later pooling, attention, recurrent update, or loss. A manually loaded padding row can also differ from the default zero initialization.

Inspect the full batch and lookup result

The illustrative program explicitly assigns four two-dimensional embedding rows, including a zero padding row. Two padded sequences have valid lengths two and one, so the lookup shape is (2,3,2). An explicit mask removes invalid positions before a per-sequence mean, producing [2,3] and [5,6]. The denominator is each sequence’s valid length, not the padded width. This is a fixed table calculation rather than a trained semantic representation. A zero-length input would require another supported policy; the shown fixture assumes positive lengths and does not divide by zero.

python Illustrative CPU PyTorch 2.14 API program; torch unavailable, expected output derived

import torch
from torch import nn
from torch.nn.utils.rnn import pack_padded_sequence
embedding = nn.Embedding(4,2,padding_idx=0,dtype=torch.float64,device="cpu")
with torch.no_grad():
    embedding.weight.copy_(torch.tensor([[0.0,0.0],[1.0,2.0],[3.0,4.0],[5.0,6.0]],dtype=torch.float64,device="cpu"))
tokens = torch.tensor([[1,2,0],[3,0,0]],dtype=torch.long,device="cpu")
lengths = torch.tensor([2,1],dtype=torch.long,device="cpu")
values = embedding(tokens)
valid = torch.arange(tokens.shape[1],device="cpu")[None,:] < lengths[:,None]
pooled = values.masked_fill(~valid[:,:,None],0.0).sum(dim=1)/lengths[:,None]
packed = pack_padded_sequence(values,lengths,batch_first=True,enforce_sorted=False)
print("embedding shape", tuple(values.shape))
print("valid means", pooled.tolist())
print("packed data shape", tuple(packed.data.shape))

Expected output

embedding shape (2, 3, 2)
valid means [[2.0, 3.0], [5.0, 6.0]]
packed data shape (3, 2)

Fixed embedding rows, positive CPU lengths, and explicit masking determine these shapes and means. No embedding, packing routine, or recurrent model was executed.

Packing is an interface with ordering constraints

pack_padded_sequence consumes padded values and lengths according to batch_first and sorting options. When lengths are supplied as a tensor, the documented CPU requirement matters even if another part of a model uses a different device; this course stays entirely on CPU. enforce_sorted=False permits the routine’s internal sorting policy rather than requiring the caller’s descending order. PackedSequence.data has one row per valid position here, so its shape is (3,2). Do not treat its storage order as the original rectangular batch or assume that a separate target list can be flattened arbitrarily. Preserve the relevant alignment through the documented unpacking and indexing interfaces.

Recurrent layout flags have a bounded meaning

For recurrent modules, batch_first changes the documented layout of input and output sequences, but does not generally change the hidden-state layout into a batch-first form. Read the particular RNN, GRU, or LSTM contract, including layer and direction dimensions and any distinct cell state. Bidirectional context can be legitimate for a fully observed sequence and inappropriate for a causal prediction before future input exists. Packing and masks are representations of valid context, not proof of prediction-time legitimacy. Keep sequence ids, reset boundaries, and the intended last valid state explicit when constructing a classifier or stream processor.

Test lengths and padding as first-class data

The list companion uses conspicuous padding values to verify that valid means and counts ignore storage artifacts. Include unequal lengths, invalid length metadata, an unsupported empty case, and the same valid prefix with different padding. Check whether a later operation has already mixed padding into a state before masking only the loss. The framework program is an unexecuted PyTorch 2.14 expectation because torch is unavailable. The executed companion validates only list selection and arithmetic; it does not test embeddings, packing order, recurrent hidden states, tokenization, or learned sequence quality.

python Executed deterministic standard-library companion

sequences = [[1.0,3.0,99.0],[5.0,99.0,99.0]]
lengths = [2,1]
assert all(0 < n <= len(row) for row,n in zip(sequences,lengths))
means = [sum(row[:n])/n for row,n in zip(sequences,lengths)]
flattened = [value for row,n in zip(sequences,lengths) for value in row[:n]]
print("valid means", means)
print("valid positions", len(flattened))
print("padding excluded", 99.0 not in flattened)

Expected output

valid means [2.0, 5.0]
valid positions 3
padding excluded True

The companion checks explicit valid-position selection on lists; it does not reproduce PackedSequence ordering, embedding gradients, or a recurrent-state computation.

Common errors and fixes

A sequence mean changes after adding padding

Why it happens: The padded width or invalid values entered aggregation.
How to fix it: Mask values and use the intended valid-length denominator.
Prevent it next time: Compare identical valid prefixes under different padding lengths and values.

A packed target no longer matches its output

Why it happens: Storage ordering or sorting metadata was ignored.
How to fix it: Align targets through the documented packing or unpacking contract.
Prevent it next time: Keep a tiny unequal-length fixture with identifiable positions.

A hidden state is reshaped using batch_first blindly

Why it happens: The flag’s sequence-layout scope was assumed to cover hidden states.
How to fix it: Read and preserve the actual layer, direction, batch, and hidden dimensions.
Prevent it next time: Assert hidden-state shapes independently of input layout.

Check your understanding

1. Why does the first pooled embedding equal [2,3]?

Show worked solution

Its valid token ids are one and two, whose fixed rows are [1,2] and [3,4]. Adding them gives [4,6] and dividing by valid length two gives [2,3]. The padding row is excluded. Dividing by padded width three would define another, padding-dependent result.

2. Does padding_idx=0 remove the need for attention or loss masks?