Deep Learning / Representation and acceptance evidence / Lesson 15

Transfer, freezing, and constrained adaptation

Treat reuse as a hypothesis about representation compatibility and test what remains fixed.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish a frozen feature extractor from fine-tuning.
Audit parameter updates and mutable model state.
Explain a low-rank additive update and its limits.

Transfer assumes useful structure is reusable

Transfer learning reuses parameters or representations learned for another objective or population. A new task may share useful structure, but its input preprocessing, label meanings, and decision requirements can differ. Start with a baseline and state why the source representation should help. A pretrained artifact also needs provenance, permitted use, architecture, and preprocessing metadata; a convenient filename is not sufficient evidence. This course downloads no artifact and uses fixed synthetic features to explain the procedure. The mathematical idea of reuse can be understood without claiming any external model quality or incurring network or accelerator requirements.

A frozen representation and a trainable head differ

A linear probe holds the representation fixed and fits a restricted head, testing what information is accessible to that predictor. Fine-tuning changes some or all of the representation and therefore evaluates a different procedure. The first fixture updates a two-coordinate head using half-squared error on a fixed feature table. It checks that features remain unchanged. A good probe result does not prove the representation is optimal, while a poor result can reflect the restricted head as well as missing information. Keep evaluation splits and selection budgets comparable when assessing reuse against training from scratch.

python Deterministic standard-library calculation

features = [[1.0, 2.0], [2.0, 1.0]]
targets = [3.0, 3.0]
head = [0.0, 0.0]
gradient = [sum((sum(w*x for w,x in zip(head,row))-y)*row[j]
                for row,y in zip(features,targets))/len(targets) for j in range(2)]
updated = [w-0.1*g for w,g in zip(head,gradient)]
print("gradient", gradient)
print("head", updated)
print("features unchanged", features == [[1.0,2.0],[2.0,1.0]])

Expected output

gradient [-4.5, -4.5]
head [0.45, 0.45]
features unchanged True

A fixed feature table acts as the frozen representation; only the two-coordinate linear head is updated.

Freezing gradients does not freeze every state

A model can contain trainable parameters, persistent statistics, random behavior, and caches. Preventing parameter gradients does not automatically stop running-statistic updates in a training-mode normalization layer. Likewise, a frozen parameter still present in an optimizer can interact with its state or decay semantics depending on implementation. Define which values may change and verify both parameter and buffer snapshots. Changing the output label mapping requires replacing or reinterpreting the head correctly. A numerically compatible head can still assign scores to the wrong class names if source and destination class orders differ.

Constrain the update rather than the whole model

A low-rank adaptation writes a weight change as a product of smaller factors, such as A*B, while retaining the original weight. The second fixture uses an outer product to make a rank-at-most-one update to a two-by-two matrix. This restricts the possible update family, not necessarily the rank of the final adapted matrix. Actual parameter savings depend on dimensions and rank: factorization is beneficial only when r(m+n) is smaller than mn for the relevant matrix. The example demonstrates a constraint without claiming that a particular rank suffices for a real task.

python Deterministic standard-library calculation

W = [[1, 0], [0, 1]]
a = [1, -1]
b = [2, 3]
update = [[ai*bj for bj in b] for ai in a]
adapted = [[W[i][j]+update[i][j] for j in range(2)] for i in range(2)]
print("rank-one update", update)
print("adapted", adapted)

Expected output

rank-one update [[2, 3], [-2, -3]]
adapted [[3, 3], [-2, -2]]

The outer product gives a rank-at-most-one additive update; no training or pretrained artifact is involved.

Validate adaptation as a complete new procedure

Choose which blocks to adapt, learning-rate groups, stopping rules, and input transformations using permitted training and validation evidence. Inspect whether improvements persist on relevant slices and whether the reused system loses required behavior elsewhere. Freezing can reduce update storage but may still require forward activations or gradients through a frozen block depending on what is being optimized. Recompute any cached features if preprocessing or upstream parameters change. Package the final architecture, base identity, adaptation factors, label mapping, and inference settings together conceptually; the separate MLOps track develops lifecycle operations beyond this mathematical lesson.

Common errors and fixes

A supposedly frozen model changes its predictions

Why it happens: Mutable statistics or mode-dependent behavior still changed.
How to fix it: Audit buffers and inference mode as well as parameter updates.
Prevent it next time: Compare named state snapshots and fixed-input outputs.

A reused head predicts the wrong class names

Why it happens: The class order differs from the source mapping.
How to fix it: Replace or remap the head using the destination label contract.
Prevent it next time: Test named classes with explicit score indices.

Cached features stay in use after changing preprocessing

Why it happens: The cache no longer represents the current input-to-feature function.
How to fix it: Invalidate and recompute features under the new contract.
Prevent it next time: Key caches by data, preprocessing, and encoder identity.

Check your understanding

1. For a 100 by 80 weight and rank four factors, compare trainable update entries with a dense update.

Show worked solution

A dense update has 8000 entries. Factors of shapes (100,4) and (4,80) have 400+320=720 entries, before any extra adapter parameters. This is a parameter-count comparison only; activation memory, compute, and final quality require separate analysis.

2. Why is setting parameter gradients off insufficient to prove a feature extractor is frozen?