Machine Learning / Models and controlled selection / Lesson 9

Ensembles and error diversity

Understand averaging, bagging, and boosting as distinct procedures whose benefit depends on member errors and valid selection.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Combining models does not erase their assumptions
Inspect a vote and its limited meaning
Validate the aggregate, not just its ingredients

Combining models does not erase their assumptions

An ensemble combines predictions from several fitted members. Averaging regression outputs or voting over class decisions can reduce some variability when members make sufficiently different errors, but common bias and shared leakage remain. Ten copies of the same predictor do not create ten independent sources of evidence. Evaluate the entire ensemble under the same split and metric as its components and baseline. Its operational cost includes memory, latency, preprocessing, and version coordination across members. A more complicated aggregate is worthwhile only when a measured benefit under the actual decision contract justifies those costs.

Bagging and random forests create diversity differently

Bagging fits members on resampled training observations, often using bootstrap samples with replacement. Random forests add tree-specific randomness such as considering subsets of features at splits under their implementation. Out-of-bag observations can support a particular internal assessment when its assumptions fit, but they do not automatically solve group dependence, temporal leakage, or all model-selection bias. A row can be absent from one bootstrap sample while a related observation still informs that tree. Keep the proper external evaluation boundary. Record seeds, member count, sampling policy, and preprocessing rather than treating the family name as a complete reproducibility specification.

Inspect a vote and its limited meaning

The first example combines three fixed binary member decisions for four observations. A majority rule gives one aggregate class per observation, and the mean vote fraction is also printed. That fraction measures agreement among these supplied decisions; it is not automatically a calibrated event probability. A tie policy becomes necessary with an even number of members, and weighted voting requires a documented weighting rule learned without final-test contamination. The example contains no fitting step and makes no accuracy claim. It isolates the combination arithmetic so member identity, class mapping, and row alignment can be checked independently.

python Fixed synthetic offline example

import numpy as np
members = np.array([[0, 1, 1, 0], [1, 1, 0, 0], [0, 1, 1, 1]])
vote_fraction = members.mean(axis=0)
prediction = vote_fraction > 0.5
print(np.round(vote_fraction, 3).tolist())
print(prediction.astype(int).tolist())

Expected output

[0.333, 1.0, 0.667, 0.333]
[0, 1, 1, 0]

Three members avoid ties in this fixture. Vote fractions describe agreement, not empirically calibrated probabilities.

Boosting adds stages under a chosen objective

Boosting builds an additive procedure whose later stages respond to the chosen loss and current predictions. For squared-error gradient boosting, residuals motivate the next correction, while other losses require their own gradient treatment. The second example applies one explicitly supplied weak correction with shrinkage and compares training loss before and after. The correction is a fixture, not a complete learned boosting implementation. Learning rate, stage count, depth, and early stopping interact, and lower training loss after another stage does not guarantee better held-out behavior. Evaluate the complete stage-selection policy within the proper validation protocol.

python Fixed synthetic offline example

import numpy as np
y = np.array([1.0, 3.0, 5.0])
initial = np.full(y.shape, y.mean())
weak_correction = np.array([-1.0, 0.0, 1.0])
updated = initial + 0.5 * weak_correction
print(f"initial fixture MSE={np.mean((y - initial) ** 2):.3f}")
print(f"updated fixture MSE={np.mean((y - updated) ** 2):.3f}")

Expected output

initial fixture MSE=2.667
updated fixture MSE=1.500

One supplied correction reduces this fixed training loss. It provides no evidence about an unseen population or an automatically trained ensemble.

Validate the aggregate, not just its ingredients

A stacking model needs out-of-fold or otherwise valid base predictions for its meta-learner; fitting the meta-model on base training predictions can leak overfit information. Calibrate the final decision output where required rather than assuming member calibration transfers unchanged through every combination. Inspect slices, stability, and correlated failure modes, and verify that all members receive the same intended schema. Compare inference cost under representative input sizes. Document a fallback if one artifact is missing or incompatible. Ensemble diversity is a hypothesis supported by member errors and held-out results, not an automatic quality certificate conferred by a larger model count.

Common errors and fixes

More members repeat the same failure

Why it happens: Their errors share bias or leaked information.
How to fix it: Inspect diversity and evaluate the full procedure honestly.
Prevent it next time: Compare against components and a baseline.

Vote agreement is announced as calibrated risk

Why it happens: Member decisions were averaged without calibration evidence.
How to fix it: Assess final probability behavior separately.
Prevent it next time: Label vote fraction and probability distinctly.

A stacking meta-model learns overfit base outputs

Why it happens: Base training predictions were used as independent evidence.
How to fix it: Use a valid out-of-fold or held-out construction.
Prevent it next time: Keep every selection stage inside the protocol.

Check your understanding

1. Why do identical duplicated predictors not provide the usual intuition of averaging independent errors?

Show worked solution

Their errors move together, so averaging preserves the same prediction and shared bias. Diversity must be assessed from actual error behavior under the intended distribution. More artifacts can increase cost without adding useful information or improving the decision.

2. Can a falling boosting training loss justify adding stages until it reaches zero?