Deep Learning / Tensor and learning foundations / Lesson 2

Affine maps, nonlinearities, and representation

Explain what extra layers can represent and why nonlinear operations change the model family.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Compose affine maps explicitly.
Use a hidden representation to separate a simple pattern.
Relate activation behavior to gradients and units.

Depth without nonlinearity still collapses

An affine map combines a linear transformation with a bias. Composing two affine maps gives another affine map: (XW1 + b1)W2 + b2 equals X(W1W2) + b1W2 + b2. The intermediate representation can change optimization or factorization constraints, but does not create an arbitrary nonlinear decision boundary by itself. This distinction prevents the misleading claim that adding any layers automatically adds expressive nonlinearity. It also explains why a final linear prediction head can be appropriate when earlier layers already construct nonlinear features. Always distinguish the mathematical function family from how its parameters are stored.

A hidden feature can change separability

Consider binary inputs where the desired output is one exactly when the two inputs differ. No single straight boundary separates these four corners under the usual thresholded affine classifier. Two rectified features, max(0, x1-x2) and max(0, x2-x1), sum to their absolute difference on these inputs. This is a concrete representation change, not evidence that every learned hidden unit has a clean human interpretation. Real networks distribute useful information across coordinates, and several parameter settings can represent equivalent functions. Explain a representation through tested input-output behavior before assigning semantic names to individual neurons.

Construct a small nonlinear feature map

The first fixture evaluates the two hidden rectifiers on all four binary inputs. It uses fixed hand-chosen weights, so it demonstrates representational capacity rather than successful training. The output reaches one at the off-diagonal corners and zero at the equal-input corners. ReLU returns zero for negative inputs and leaves positive inputs unchanged. At zero its ordinary derivative is not uniquely defined; implementations choose a convention for backpropagation. Such conventions matter in numerical gradient checks and should be stated rather than treating every kink as a smooth point.

python Deterministic standard-library calculation

inputs = [(0, 0), (0, 1), (1, 0), (1, 1)]
for a, b in inputs:
    h = [max(0, a - b), max(0, b - a)]
    print((a, b), h, sum(h))

Expected output

(0, 0) [0, 0] 0
(0, 1) [0, 1] 1
(1, 0) [1, 0] 1
(1, 1) [0, 0] 0

Fixed hidden features represent the four-corner difference rule without claiming a training result.

Saturation changes local sensitivity

A sigmoid maps a real input into the open interval between zero and one. Its derivative s(1-s) becomes small near either extreme, so a long chain of such local derivatives can transmit a small gradient. The second calculation displays that local property at zero and ten. It does not show that sigmoid is always a poor choice; a sigmoid is meaningful for a binary probability output under an appropriate objective. Hidden activations, output transformations, and loss implementations have different roles. Match each role deliberately rather than using one activation everywhere because its output looks convenient.

python Deterministic standard-library calculation

from math import exp
for z in [0.0, 10.0]:
    s = 1.0 / (1.0 + exp(-z))
    print(f"{z:.1f}: value={s:.6f}, slope={s * (1-s):.6f}")

Expected output

0.0: value=0.500000, slope=0.250000
10.0: value=0.999955, slope=0.000045

The same bounded activation has very different local sensitivity in its central and saturated regions.

Choose capacity with a testable purpose

More hidden units allow more possible functions, but a useful function still requires suitable data, optimization, and evaluation. A feature may encode a shortcut that works only under the training collection process. Examine performance under the intended split and meaningful perturbations, and compare with a simpler baseline. Network width and depth also affect storage, numerical behavior, and computational cost. This lesson uses a manually constructed feature map so every intermediate value is inspectable. It establishes an architectural idea without implying universal approximation at practical size, successful optimization, causal explanation, or reliable behavior outside the shown inputs.

Common errors and fixes

Several linear layers fail to learn a curved boundary

Why it happens: Their composition remains affine when no nonlinearity intervenes.
How to fix it: Introduce a justified nonlinear hidden representation.
Prevent it next time: Write the composed function before assuming depth changes expressiveness.

A hidden sigmoid barely changes during training

Why it happens: Preactivations lie in a saturated region with small local derivatives.
How to fix it: Inspect activation and gradient distributions before changing scale or activation.
Prevent it next time: Monitor saturation and use initialization suited to the chosen layer.

A hand-built successful map is reported as trained accuracy

Why it happens: Fixed weights on four inputs were confused with learning and evaluation.
How to fix it: Describe it as a representational fixture and evaluate learned models separately.
Prevent it next time: Separate architecture demonstrations from empirical evidence.

Check your understanding

1. Collapse f(x)=2x+3 followed by g(h)=4h-1 into one affine map.

Show worked solution

Substitution gives g(f(x))=4(2x+3)-1=8x+11. Both the slope and bias change, but the function is still affine. Adding a rectifier between the maps would generally make this algebraic collapse invalid across the rectifier boundary.

2. Why should a binary output sigmoid not be removed solely because hidden sigmoids can saturate?