Machine Learning / Models and controlled selection / Lesson 8

Trees and partitioning

Understand splits, leaf predictions, and capacity controls without treating a fitted tree as an automatic explanation or probability guarantee.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

A tree partitions feature space
Compare two stump splits explicitly
Control complexity and inspect stability

A tree partitions feature space

A decision tree applies a sequence of feature tests and assigns a prediction at a leaf. For classification, a leaf can summarize class frequencies; for regression, it can summarize target values under the chosen loss. Axis-aligned thresholds create piecewise regions and can model interactions without an explicit product feature. This flexibility does not remove the need for representative data or valid evaluation. Small changes in observations can alter a fitted tree’s structure. A readable sequence of tests is a description of that model’s rule, not proof the rule is causal, stable, or appropriate for an individual decision.

A split criterion is a local fitting rule

Classification trees can choose splits by reducing an impurity measure such as Gini impurity or entropy. The gain compares the parent with a weighted combination of child impurities. It does not directly optimize every operational metric or guarantee the best complete tree. Greedy local search can miss a different global structure. Regression criteria have their own target-loss interpretation. State minimum leaf size, depth, and other stopping rules because unrestricted partitions can memorize small samples. A leaf with one example has little evidence for a reliable probability even if it is perfectly pure on training data.

Compare two stump splits explicitly

The first example computes weighted Gini impurity for two thresholds on four fixed observations. Splitting at 2.5 separates the labels perfectly in this fixture, while 1.5 leaves a mixed right child. Empty children are rejected by the helper because they do not define a valid split under this demonstration. This calculation explains the objective used to compare candidate partitions; it does not claim that searching these two thresholds implements a production tree learner. Real implementations must consider feature handling, missing values, tie policy, weights, stopping conditions, and numerical or memory constraints.

python Fixed synthetic offline example

import numpy as np
x = np.array([1.0, 2.0, 3.0, 4.0])
y = np.array([0, 0, 1, 1])
def gini(values):
    p = np.mean(values == 1)
    return 1.0 - p * p - (1.0 - p) ** 2
def split_impurity(threshold):
    left = x <= threshold
    assert left.any() and (~left).any()
    return left.mean() * gini(y[left]) + (~left).mean() * gini(y[~left])
for threshold in [1.5, 2.5]:
    print(f"threshold={threshold:.1f} impurity={split_impurity(threshold):.3f}")

Expected output

threshold=1.5 impurity=0.333
threshold=2.5 impurity=0.000

The weighted impurity is one third for the first candidate and zero for the perfectly separating fixture split.

Apply a learned threshold with the same convention

The second example applies a fixed stump using x <= 2.5 for the left region and the complementary right region. The exact equality convention is part of the model artifact, just like feature order and units. A different unit at inference moves records across the boundary even if the code runs successfully. Categorical and missing-value handling are estimator-specific; a numeric category code should not be assumed to express a meaningful order. Trees often require less conventional scale normalization for threshold ordering than distance-based models, but transformations and finite precision can still affect a particular implementation and should be documented.

python Fixed synthetic offline example

import numpy as np
future = np.array([1.5, 2.5, 3.5])
labels = np.where(future <= 2.5, 0, 1)
print(labels.tolist())

Expected output

[0, 0, 1]

Equality follows the declared left-branch rule. This is application of one fixed stump, not a newly trained classifier or quality claim.

Control complexity and inspect stability

Choose depth, leaf sizes, pruning, and related controls using permitted validation data, not the final test. Compare with a simple baseline and inspect whether small perturbations change important decisions. An impurity-based feature importance can favor certain feature types and can split importance among correlated variables; it is not a causal ranking. Leaf frequencies also need calibration assessment under representative data before being used as reliable risk estimates. Evaluate groups and error types, and keep deployment size and latency in view. A compact tree can be a useful model, but interpretability is a task-dependent property that requires more than drawing its nodes.

Common errors and fixes

A pure training leaf is treated as certain risk

Why it happens: Few observations were mistaken for reliable probability evidence.
How to fix it: Assess support and calibration on representative held-out data.
Prevent it next time: Report leaf size and uncertainty where decisions rely on it.

A deep tree memorizes the fixture

Why it happens: Capacity was not controlled under validation.
How to fix it: Choose complexity and stopping rules through the split protocol.
Prevent it next time: Compare train and validation behavior with a baseline.

A drawn path is called a causal explanation

Why it happens: Model rules were confused with intervention effects.
How to fix it: Describe the path as model behavior and state its limits.
Prevent it next time: Keep causal claims under a separate evidence standard.

Check your understanding

1. Why does a split with zero training impurity not prove good generalization?

Show worked solution

It can isolate noise, tiny samples, or leaked information. Evaluate the complete tree-building procedure on independent permitted data, control complexity, and inspect relevant slices. The split criterion is a fitting objective, not an external guarantee that future cases follow the same partition.

2. What must a deployed stump artifact preserve beyond the threshold number?