Deep Learning / Architectures and training dynamics / Lesson 7

Initialization, normalization, and residual paths

Reason about signal scale and gradient paths without treating architectural techniques as universal repairs.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Relate initialization scale to fan-in and activation.
Distinguish normalization axes and stored state.
Explain the identity path in a residual block.

Signal scale depends on the whole layer

A layer sums several weighted inputs before applying its activation. Under simplifying independence and variance assumptions, more summed terms increase preactivation variance unless weight scale compensates. Initialization schemes use fan-in, fan-out, and activation assumptions to keep signals in a useful range. These assumptions are approximations, especially when inputs are correlated or architectures branch. Zeroing every hidden weight can preserve unwanted symmetry between units. Random initialization can break that symmetry, but a random seed only identifies a draw under stated software behavior; it does not guarantee good conditioning, convergence, or reproducibility on every device.

Read an initialization rule as a conditional argument

The first fixture computes sqrt(2/fan_in), a familiar standard-deviation scale for a zero-mean ReLU-oriented initialization. It does not sample weights or train a model. The factor two reflects an approximate variance argument involving the rectifier, and other activations or architectural conventions call for other rules. Bias policy, normalization, residual scaling, and parameter sharing can alter the effective signal path. Inspect layer activation and gradient summaries on a tiny controlled batch before assuming a named initializer solves the problem. Replacing an initializer cannot repair a wrong target contract or invalid evaluation split.

python Deterministic standard-library calculation

from math import sqrt
for fan_in in [2, 8, 32]:
    print(f"fan_in={fan_in}, relu_std={sqrt(2/fan_in):.4f}")

Expected output

fan_in=2, relu_std=1.0000
fan_in=8, relu_std=0.5000
fan_in=32, relu_std=0.2500

A common ReLU-oriented variance rule shrinks individual weight scale as the number of summed inputs grows.

Normalization axes define which values interact

Normalizing a feature across a training batch makes one observation depend on its companions. Batch normalization also commonly maintains running statistics for inference, whose exact update and variance conventions are implementation-specific. Layer normalization instead normalizes selected feature dimensions within an observation. The second fixture normalizes each two-feature row using population variance and an epsilon, omitting learned gain and bias. Its equal outputs do not mean the original rows were equal. Normalization intentionally discards some information; whether later learned scales or other paths preserve what the task needs is an architectural question.

python Deterministic standard-library calculation

from math import sqrt
rows = [[1.0, 3.0], [10.0, 14.0]]
for row in rows:
    mean = sum(row) / len(row)
    variance = sum((x-mean)**2 for x in row) / len(row)
    normalized = [(x-mean)/sqrt(variance+1e-5) for x in row]
    print([round(x, 5) for x in normalized])

Expected output

[-1.0, 1.0]
[-1.0, 1.0]

Statistics are computed within each row, using population variance and no learned scale or shift.

A residual connection offers an identity route

A same-shape residual block writes y=x+F(x). Its derivative with respect to x contains an identity contribution in addition to the derivative of F. This can provide a useful route for signals and gradients in deep compositions, but does not mathematically guarantee stable gradients or successful learning for every F. When spatial size or feature width changes, the skip branch requires an explicit compatible projection or another documented operation. Silent broadcasting is not a substitute for architectural alignment. Check both branch shapes and their scales before interpreting the resulting sum.

Observe before changing several mechanisms

Compare preactivation scale, activation sparsity or saturation, gradient norms, and train-versus-inference behavior layer by layer. Keep the input and parameter state fixed while comparing two modes. A small batch can make batch statistics noisy; replacing batch normalization with layer normalization changes the model and should be evaluated rather than called an equivalent repair. When a residual branch dominates the skip signal, inspect scaling and initialization instead of adding further mechanisms blindly. Record the normalization axes, epsilon, trainable gain and bias, stored statistics, and initialization rule in the model contract so later framework implementations can be checked against it.

Common errors and fixes

Hidden units remain indistinguishable

Why it happens: Symmetric initialization gives units identical inputs and updates.
How to fix it: Use an appropriate symmetry-breaking initialization.
Prevent it next time: Inspect distinct parameters and activations at the first update.

A prediction changes when unrelated examples join its batch

Why it happens: Training-mode batch statistics couple observations.
How to fix it: Use the intended inference statistics and verify mode state.
Prevent it next time: Test the same example alone and with different companions.

A residual sum succeeds with the wrong feature width

Why it happens: Broadcasting hid a branch-shape mismatch.
How to fix it: Use an explicit compatible projection or correct the branch.
Prevent it next time: Assert complete branch shapes before addition.

Check your understanding

1. For fan-in 18, calculate the shown ReLU-oriented standard deviation.

Show worked solution

The scale is sqrt(2/18)=sqrt(1/9)=1/3. It describes a proposed distribution standard deviation, not a requirement that every weight equal one third. Check the activation, parameter layout, and initialization convention before applying the formula to an actual framework layer.

2. Why is y=x+F(x) not a guarantee against exploding gradients?