Machine Learning / Prediction and data foundations / Lesson 5

Regression, loss, and linear models

Fit a transparent regression example and evaluate errors in meaningful units without confusing training fit with generalization.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Choose the target and loss together
Fit a small exact line transparently
Inspect residual structure and deployment limits

Choose the target and loss together

Regression predicts a numeric outcome, but the best point prediction depends on the loss and decision. Squared error emphasizes larger deviations, while absolute error has different sensitivity and units. A target may be bounded, skewed, noisy, or only observed for selected cases. Document those properties before selecting a formula. Predicting attendance counts with an unconstrained line can produce negative values outside its fitted region, so inspect domain behavior as well as average error. A convenient numeric representation does not make extrapolation safe. Compare with a training-only baseline under the same held-out protocol and target definition.

A linear model is linear in its coefficients

A linear regression combines a feature vector with coefficients and an intercept. Feature transformations can make the input representation nonlinear in an original measurement while the fitted coefficient model remains linear. Ordinary least squares minimizes squared residuals on the supplied training data under its formulation. Collinearity or rank deficiency can make coefficients unstable or nonunique even when predictions are similar. Use a numerically appropriate solver rather than explicitly inverting a normal-equation matrix. A coefficient is an association conditional on the representation and other features, not an automatically causal effect or a universal importance ranking.

Fit a small exact line transparently

The first example adds an intercept column and solves a fixed training system with NumPy’s least-squares routine. The synthetic relation is exactly one plus twice x, so the rounded fitted coefficients and a prediction at four can be checked by hand. This deliberately easy construction explains the solver contract; it is not evidence that linear regression predicts an external dataset perfectly. The new point is within the numeric neighborhood of the fixture but still an explicitly chosen demonstration. A real workflow fits preprocessing only on training rows and evaluates the complete procedure on separate representative outcomes.

python Fixed synthetic offline example

import numpy as np
x = np.array([0.0, 1.0, 2.0, 3.0])
y = np.array([1.0, 3.0, 5.0, 7.0])
design = np.column_stack([np.ones_like(x), x])
coef, *_ = np.linalg.lstsq(design, y, rcond=None)
print(f"intercept={coef[0]:.1f} slope={coef[1]:.1f}")
print(f"prediction={np.array([1.0, 4.0]) @ coef:.1f}")

Expected output

intercept=1.0 slope=2.0
prediction=9.0

The exact synthetic line gives intercept one and slope two after rounding. The held demonstration point therefore predicts nine.

Compare errors with clearly named metrics

The second example computes MAE and RMSE from the same fixed residuals. MAE averages absolute deviations, while RMSE is the square root of mean squared deviations and has the target’s units. MSE instead has squared units. Metrics need their denominator and aggregation unit stated, especially when groups or observations receive different weights. R-squared can be negative on held-out data and needs care for a constant target; it is not a universal percentage of predictions that are correct. Keep a few concrete residual checks so a broadcasting or unit error cannot hide behind a plausible summary.

python Fixed synthetic offline example

import numpy as np
y = np.array([1.0, 3.0, 5.0])
pred = np.array([2.0, 2.0, 5.0])
assert y.shape == pred.shape
error = y - pred
print(f"fixture MAE={np.abs(error).mean():.3f}")
print(f"fixture RMSE={np.sqrt(np.mean(error ** 2)):.3f}")

Expected output

fixture MAE=0.667
fixture RMSE=0.816

The residual magnitudes are one, one, and zero. Shape equality is checked before computing both metrics.

Inspect residual structure and deployment limits

Plot or summarize residuals by prediction level, time, and relevant slices. A low average error can hide systematic underprediction for large outcomes or a subgroup. Training residuals alone are optimistic evidence after fitting, and choosing a model from many validation trials adds selection uncertainty. If error spread changes with the target level, a single constant uncertainty band may be inappropriate. Distinguish prediction intervals from confidence about an average model parameter. Document extrapolation limits and fallback behavior. This course’s calculations are deterministic arithmetic fixtures, not claims about statistical coverage or guaranteed performance in a new environment.

Common errors and fixes

A good training fit is announced as future performance

Why it happens: Evaluation reused the fitting outcomes.
How to fix it: Measure the complete procedure on the agreed held-out data.
Prevent it next time: Separate fitting, selection, and final assessment records.

Regression error has unexplained scale changes

Why it happens: Target units or inverse transformation were inconsistent.
How to fix it: Evaluate predictions and targets in the same declared units.
Prevent it next time: Include hand-checkable residuals.

Coefficients change wildly with small input changes

Why it happens: Collinearity or an unstable solving strategy is present.
How to fix it: Inspect rank, representation, and regularization needs.
Prevent it next time: Use an appropriate solver and validate stability.

Check your understanding

1. A target is converted from dollars to cents. How should MAE and RMSE change for equivalent predictions?

Show worked solution

Both scale by one hundred because they have the target’s units. MSE scales by ten thousand because it uses squared errors. Keep the conversion consistent on truth and predictions and report the units; otherwise two valid-looking scores can describe incompatible quantities.

2. Does a slope of two prove that increasing the feature causes the target to rise by two?