Deep Learning / Architectures and training dynamics / Lesson 10

Sequences, recurrence, and padding

Define temporal state, valid lengths, and target alignment before training a sequence model.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Trace state reuse over time.
Keep padding out of both state and loss.
Respect prediction-time information boundaries.

A sequence has an order and a state contract

A recurrent model updates a state using the previous state and the current input. Reusing the same update parameters across time lets the representation depend on a prefix of the sequence. Record whether time is the first or second tensor axis, what starts a new sequence, and how the initial state is chosen. Carrying state between unrelated observations creates an unintended information path. In a streaming task, retaining state can be legitimate, but its ownership, reset rule, and handling of missing observations must match evaluation. The ordering is semantic, not just a storage preference.

Padding is storage rather than an observation

Batches often pad shorter sequences to a common length. A padding value should not silently become a real event. Masking only the final loss can be insufficient when padding has already changed the hidden state used for a sequence-level prediction. The first fixture uses an additive recurrence and a conspicuous padding value of 99. Valid lengths gate the state update, preserving the prefix sum. Real recurrent cells also contain nonlinearities and learned transformations, but the same distinction remains: an input-position mask, a state-update policy, and a loss mask have separate responsibilities.

python Deterministic standard-library calculation

sequences = [[2, 3, 99], [4, 99, 99]]
lengths = [2, 1]
for sequence, length in zip(sequences, lengths):
    state = 0
    for t, value in enumerate(sequence):
        if t < length:
            state = state + value
    print(state)

Expected output

5
4

Masking the state update prevents the deliberately nonzero padding value from entering the final state.

Define the target at every time position

For next-step prediction, the input prefix ending at time t must align with the target at t+1, without exposing that target among the permitted inputs. Sequence classification instead may use a final valid state, an explicit pool, or another aggregation. Bidirectional models can use both past and future context for a fully observed sequence, but that is invalid for a causal streaming prediction that must be made before the future exists. State clearly whether the sequence is fully available at decision time. A correct tensor shape cannot establish that this information boundary is respected.

Masking and averaging are different decisions

The second fixture masks padded loss entries and computes a mean over valid tokens. It also computes an equally weighted mean of per-sequence means. Their equality in this chosen example is accidental; replace the second valid loss by eight and the results differ. Decide which unit should receive equal weight, and test unequal lengths with unequal losses. Empty sequences or a batch with no valid targets require an explicit skip, rejection, or defined aggregation policy. Keeping a padded zero in the denominator can reward adding more padding even when no new observation was supplied.

python Deterministic standard-library calculation

losses = [[1.0, 3.0, 100.0], [2.0, 100.0, 100.0]]
lengths = [2, 1]
numerator = sum(sum(row[:n]) for row, n in zip(losses, lengths))
valid_count = sum(lengths)
print(f"valid-token mean={numerator/valid_count:.1f}")
print(f"example mean={sum(sum(row[:n])/n for row,n in zip(losses,lengths))/2:.1f}")

Expected output

valid-token mean=2.0
example mean=2.0

This particular fixture gives equal means under two different weighting rules; equality here does not make the rules interchangeable.

Long dependency paths require measured diagnosis

Backpropagation through a repeated state update multiplies local derivative factors along time, which can weaken or amplify distant contributions. Gated recurrent units and long short-term memory introduce structured state paths, but do not guarantee that any desired long dependency is learned. Truncated backpropagation deliberately limits derivative history while potentially retaining forward state; that changes the training procedure and must be stated. Inspect valid lengths, state resets, gradient norms, and alignment before choosing a more complex cell. The fixtures demonstrate masking arithmetic only and make no trained language, audio, or forecasting performance claim.

Common errors and fixes

Predictions change when extra padding is appended

Why it happens: Padding entered the recurrent state or pooling operation.
How to fix it: Gate updates and aggregate only valid positions under the model contract.
Prevent it next time: Test equal prefixes with different padding lengths and values.

A streaming evaluation improves after adding future context

Why it happens: A bidirectional path violated prediction-time availability.
How to fix it: Use a causal model or explicitly redefine the fully observed task.
Prevent it next time: Audit every context source at the decision timestamp.

Unrelated sequences influence one another

Why it happens: Hidden state persisted across sequence boundaries.
How to fix it: Reset or route state by the correct sequence identity.
Prevent it next time: Test isolated and interleaved sequence processing.

Check your understanding

1. Replace the second valid loss in the fixture by eight. Calculate token and sequence means.

Show worked solution

The valid losses are one, three, and eight, so their token mean is twelve divided by three, or four. The sequence means are two and eight, whose equal-weight mean is five. Neither rule is universally correct; the intended evaluation unit decides.

2. Why can a zero padding value still alter a recurrent state?