Machine Learning / Models and controlled selection / Lesson 7

Optimization and regularization

Connect a fitted objective to gradient updates, numerical behavior, and controlled model complexity.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Separate the model, objective, and optimizer
Verify one logistic update by hand
Use learning behavior as diagnostic evidence

Separate the model, objective, and optimizer

A model family defines possible predictions, a loss measures disagreement, and an optimizer searches for parameters under an objective. These choices are related but not identical. Logistic regression uses a linear score with a logistic link for a binary probability model; least squares uses a different target and loss contract. Training loss can improve while held-out behavior worsens. Diagnose whether a failure comes from an unsuitable representation, excessive flexibility, an optimization problem, or invalid evaluation before changing a learning rate. A converged solution is a solution to the chosen objective, not proof the objective represents the real decision.

A gradient step has a scale and direction

Gradient descent moves parameters opposite the objective’s gradient under its learning-rate convention. The step size interacts with feature scale, curvature, batch size, and loss normalization. Too large a step can oscillate or diverge; too small a step can make progress impractically slow. Check finite inputs and losses and compare a simple expected gradient on a tiny fixture. Use numerically stable expressions for logarithms and exponentials, such as logaddexp for the demonstrated logistic loss. More iterations cannot fix leaked validation data or a mislabeled target. Record stopping criteria and verify what the reported convergence flag actually means.

Verify one logistic update by hand

The first example starts one coefficient at zero for two symmetric observations. Both initial probabilities are one half, the average gradient is negative one half, and a step size of one half gives a positive coefficient of one quarter. The new probabilities and loss are printed after that single step. This is not a fully fitted classifier or a claim of calibrated probabilities. The fixed values keep exponentials in a safe small range. A library implementation may include an intercept, penalty, weights, and a different loss scaling, so match its documented objective before comparing numeric updates directly.

python Fixed synthetic offline example

import numpy as np
x = np.array([-1.0, 1.0])
y = np.array([0.0, 1.0])
w = 0.0
p = 1.0 / (1.0 + np.exp(-x * w))
gradient = np.mean(x * (p - y))
w -= 0.5 * gradient
z = x * w
print(f"w={w:.2f}")
print(np.round(1.0 / (1.0 + np.exp(-z)), 3).tolist())
print(f"loss={np.mean(np.logaddexp(0.0, z) - y * z):.3f}")

Expected output

w=0.25
[0.438, 0.562]
loss=0.576

One fixed step yields w=0.25. The resulting probabilities are a calculation under the stated model, not empirical calibration evidence.

Regularization changes the preferred solution

The second example solves a small ridge objective with no intercept: squared residual sum plus alpha times squared coefficient norm. The identity design makes the shrinkage independently checkable. A positive penalty trades training fit against coefficient magnitude; it does not guarantee better future predictions for every dataset. The meaning of alpha depends on scaling and the objective’s normalization. L1 and L2 penalties have different geometry and sparsity behavior, and intercept handling is implementation-specific. Fit preprocessing and choose regularization strength inside the validation protocol. Do not compare raw coefficient magnitudes across arbitrary feature units as though they were a universal importance scale.

python Fixed synthetic offline example

import numpy as np
X = np.eye(2)
y = np.array([2.0, 4.0])
alpha = 1.0
plain = np.linalg.lstsq(X, y, rcond=None)[0]
ridge = np.linalg.solve(X.T @ X + alpha * np.eye(2), X.T @ y)
print(plain.tolist())
print(ridge.tolist())

Expected output

[2.0, 4.0]
[1.0, 2.0]

For this explicit unnormalized objective and identity design, alpha one halves both coefficients. No intercept is included.

Use learning behavior as diagnostic evidence

Compare training and validation losses as data size, complexity, or penalty changes under a valid split. High errors on both can reflect insufficient representation, noise, or optimization trouble; a gap can reflect overfitting but is not a unique diagnosis. More features, regularization, or data are hypotheses to test, not automatic prescriptions. Keep the same metric and sampling protocol across comparisons and account for selection after many trials. Document any early-stopping data as part of the learned procedure. The final test must assess the selected training-and-stopping policy, not a model repeatedly revised using that test’s outcomes.

Common errors and fixes

Training diverges after scaling changes

Why it happens: The learning rate no longer suits the objective’s scale.
How to fix it: Inspect finite gradients, normalization, and feature scale.
Prevent it next time: Keep a tiny hand-checkable update test.

A convergence flag is reported as model validity

Why it happens: Optimization success was confused with task correctness.
How to fix it: Validate the representation and held-out decision objective.
Prevent it next time: Report fitting and evaluation evidence separately.

Penalty strengths are compared across incompatible conventions

Why it happens: Loss normalization or feature scaling differs.
How to fix it: Match the documented objective before interpreting alpha.
Prevent it next time: Record solver, scaling, and penalty settings together.

Check your understanding

1. Why can a smaller training loss be the wrong model-selection criterion?

Show worked solution

A more flexible model can fit noise or exploit leaked information. Compare the complete learned procedure on permitted validation data using the task’s metric and constraints. Keep final assessment separate and inspect optimization diagnostics without treating fit alone as evidence of generalization.

2. What must be recorded to reproduce a regularized linear fit?