Deep Learning / Tensor and learning foundations / Lesson 6

Generalization, regularization, and honest selection

Connect network capacity and regularization to a protected evaluation process.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Interpret training and validation curves.
Calculate explicit decay and dropout expectations.
Keep model selection separate from final assessment.

Training fit answers a limited question

A network can fit the examples used for optimization without predicting new observations usefully. Generalization concerns behavior under a declared new-data setting. Repeated entities, future measurements, or preprocessing learned from evaluation rows can make a held-out score misleading even when the optimization is correct. Begin with a legitimate split and a simple baseline. Training loss and validation loss should use compatible definitions if they are compared directly; dropout, augmentation, and regularization terms can otherwise make their scales differ. An apparent gap is evidence to interpret in context, not an automatic diagnosis of one cause.

Choose checkpoints using validation only

Validation can guide architecture, regularization, schedules, and early stopping. Repeatedly choosing among many candidates adapts decisions to that validation evidence, so an untouched final assessment has a separate role. The first fixture selects the epoch with the lowest synthetic validation loss rather than the lowest training loss. Its tiny table is invented and establishes selection arithmetic only. Decide ties and the primary metric in advance. Preserve the selected checkpoint itself; stopping at a later epoch and reporting an earlier score does not recover the earlier parameter state.

python Deterministic standard-library calculation

train = [0.8, 0.5, 0.3, 0.2]
validation = [0.9, 0.6, 0.55, 0.7]
best = min(range(len(validation)), key=lambda i: validation[i])
print("selected epoch", best + 1)
print("validation", validation[best])
print("training-only choice", min(range(len(train)), key=lambda i: train[i]) + 1)

Expected output

selected epoch 3
validation 0.55
training-only choice 4

The validation choice and training-fit choice differ; no final test participates.

Regularization changes the learning procedure

An L2 penalty adds a coefficient times a squared parameter norm to an objective. Plain gradient descent on a half-scaled squared penalty creates a shrinkage term. Decoupled weight decay applies an explicit shrinkage step; its equivalence to a penalty depends on optimizer details and is not universal for adaptive methods. Parameter groups may exclude biases or normalization parameters under a stated design. Regularization strength interacts with loss reduction and learning rate, so copying a coefficient without its context is weak reasoning. More regularization can also underfit rather than improve useful prediction.

Dropout has a training and inference contract

In inverted dropout, a retained activation is divided by its retention probability during training. This preserves that activation’s expectation under the stated Bernoulli mask; inference uses the full activation without that random mask. The second fixture enumerates both outcomes for one value rather than sampling, giving an exact expectation check. It does not prove that the nonlinear network output equals the output obtained by averaging masks. Dropout introduces stochastic training behavior and is distinct from data augmentation, which changes inputs under a task-preserving transformation policy. Both need task-specific justification and evaluation.

python Deterministic standard-library calculation

activation, keep = 6.0, 0.75
kept_output = activation / keep
expected = keep * kept_output + (1 - keep) * 0.0
w, learning_rate, decay = 2.0, 0.1, 0.2
shrunk = w * (1 - learning_rate * decay)
print(f"kept={kept_output:.1f}, expected={expected:.1f}")
print(f"decay-only weight={shrunk:.2f}")

Expected output

kept=8.0, expected=6.0
decay-only weight=1.96

The dropout expectation is exact for one activation, and the separate decay calculation contains no data gradient.

Make the selection budget and limitations visible

Record how many candidates and revisions used validation feedback, what augmentations were allowed, and which objective selected the checkpoint. Check relevant groups and input conditions rather than only an average. A transformation that preserves an image label in one task can destroy a numeric measurement or directional label in another. Regularization does not repair leakage, annotation errors, or a changing deployment population. Report the final assessment with uncertainty and the intended use boundary. The local examples do not train a network or establish any external performance; they make selection and regularization calculations auditable.

Common errors and fixes

Validation loss exceeds training loss

Why it happens: Different modes, augmentation, overfitting, or population differences may contribute.
How to fix it: Compare compatible definitions and inspect the split before diagnosing.
Prevent it next time: Record data, mode, and regularization terms for every curve.

A saved model differs from the reported best epoch

Why it happens: Only the last parameters were kept.
How to fix it: Restore the checkpoint selected under the validation rule.
Prevent it next time: Test checkpoint identity and selected epoch together.

Weight decay coefficients behave differently across optimizers

Why it happens: Penalty and decoupled shrinkage semantics were assumed identical.
How to fix it: Read the update rule and state the parameter-group policy.
Prevent it next time: Treat optimizer, reduction, and decay as one experiment definition.

Check your understanding

1. Why should early stopping avoid using final-test loss?

Show worked solution

Stopping is a model-selection choice. If final-test outcomes determine the epoch, the test has influenced the learned procedure and no longer provides an untouched assessment. Use validation for stopping, preserve the chosen parameters, and evaluate the final choice once under the declared assessment plan.

2. An activation is 10 and dropout keeps it with probability 0.5. List the training outcomes and expectation.