Machine Learning / Models and controlled selection / Lesson 11

Dimensionality reduction and selection

Distinguish a compact representation from a supervised feature choice, and keep both fitted inside the learning boundary.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Reduction changes what the model can see
Inspect a rank-one reconstruction
Evaluate the downstream effect honestly

Reduction changes what the model can see

Dimensionality reduction constructs a representation with fewer coordinates, while feature selection retains a subset of original inputs. Both can reduce cost or redundancy, but either can discard information useful for the task. Fewer dimensions do not automatically imply better generalization or easier explanation. State whether the goal is compression, visualization, denoising, or predictive efficiency, because those objectives need different evidence. A two-dimensional plot is a projection, not a complete map of the original data’s distances or structure. Keep the original schema and fitted transform so future observations can be processed under the same representation.

PCA follows variance under a chosen scaling

Principal component analysis finds orthogonal directions that summarize variance in centered data under its formulation. Scaling features changes the geometry and therefore the components. High variance is not the same as relevance to the prediction target, and a low-variance direction can carry important signal. Fit centering, optional scaling, and the component basis on permitted training rows only. A component’s sign can flip without changing its subspace or reconstruction, and repeated eigenvalues can make individual directions nonunique. Compare meaningful subspaces or reconstructed values rather than requiring one arbitrary signed vector in every numerical environment.

Inspect a rank-one reconstruction

The first example centers three points on a line, obtains a basis through singular-value decomposition, and reconstructs them using one component. All meaningful variance lies in one direction in this exact fixture, so the rounded variance fraction and reconstruction check can be verified geometrically. This does not prove that one component is adequate for a real prediction task. The code tests reconstruction with a tolerance and does not assert an exact component sign. Singular values determine the demonstrated variance proportions after squaring; a zero-total-variance input would need a separate defined policy rather than division by zero.

python Fixed synthetic offline example

import numpy as np
X = np.array([[-2.0, -2.0], [0.0, 0.0], [2.0, 2.0]])
mean = X.mean(axis=0)
centered = X - mean
_, singular, basis = np.linalg.svd(centered, full_matrices=False)
variance_fraction = singular ** 2 / np.sum(singular ** 2)
reconstructed = (centered @ basis[:1].T) @ basis[:1] + mean
print(np.round(variance_fraction, 6).tolist())
print(np.allclose(reconstructed, X))

Expected output

[1.0, 0.0]
True

The fixed points lie on one line, so one component reconstructs them. The test is insensitive to an arbitrary component sign flip.

Selection that uses targets is supervised learning

The second example ranks two training features by absolute correlation with a numeric target and keeps one. It is a deliberately simple selection rule, not a recommended universal screening method. Correlation can miss nonlinear relationships and interactions, be distorted by outliers, or select redundant proxies. Since labels influence the choice, it must be recomputed inside each training fold during evaluation. The future row is sliced using the recorded selected index, not reranked from its own values. Test constant features, missingness, and multiple comparisons under the selection protocol, and keep the data identities that informed the choice.

python Fixed synthetic offline example

import numpy as np
train = np.array([[0.0, 0.0], [1.0, 1.0], [2.0, 0.0], [3.0, 1.0]])
y = np.array([0.0, 1.0, 2.0, 3.0])
assert np.all(train.std(axis=0) > 0) and y.std() > 0
correlations = [abs(np.corrcoef(train[:, j], y)[0, 1]) for j in range(train.shape[1])]
selected = int(np.argmax(correlations))
future = np.array([[4.0, 100.0]])
print(selected)
print(future[:, [selected]].tolist())

Expected output

0
[[4.0]]

Training labels determine the selected index. The separate future row uses that frozen selection without influencing it.

Evaluate the downstream effect honestly

Compare the full transform-and-model pipeline with an unreduced baseline on the agreed held-out task. A high explained-variance fraction alone does not establish good target prediction, calibrated probabilities, fairness, or robustness. Reconstruction error and predictive loss answer different questions. If the transform is used for monitoring or explanation, document which original information is no longer visible. Store the mean, scale, basis or selected indices, feature order, and compatible software assumptions together. A dimension mismatch at inference should produce a controlled rejection or fallback rather than an accidental matrix product with a new interpretation.

Common errors and fixes

A high variance fraction is called predictive success

Why it happens: Compression and target performance were conflated.
How to fix it: Evaluate the complete downstream task separately.
Prevent it next time: Report reconstruction and prediction evidence distinctly.

Feature screening uses all labels before cross-validation

Why it happens: Held-out targets influenced the chosen representation.
How to fix it: Refit supervised selection inside every training fold.
Prevent it next time: Include selection in the learned pipeline.

A PCA test fails only because signs changed

Why it happens: An arbitrary basis sign was treated as model meaning.
How to fix it: Compare subspaces, reconstructions, or sign-aligned quantities.
Prevent it next time: Use appropriate numeric tolerances.

Check your understanding

1. Can discarding low-variance directions remove predictive information?

Show worked solution

Yes. PCA ranks variance without using the prediction target, so a small-variance feature direction can still separate outcomes. Validate the transform as part of the full model pipeline against an unreduced baseline. A compression criterion alone cannot decide which information the task needs.

2. What must accompany a stored PCA basis for consistent inference?