Machine Learning / Evidence and operational responsibility / Lesson 16

Explanations and causal limits

Explain the behavior of a specific model under explicit assumptions without turning associations or perturbations into causal guarantees.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Specify what the explanation is explaining
Measure a fixed model’s perturbation response
Validate explanations as part of a workflow

Specify what the explanation is explaining

An explanation can describe a prediction, a model’s global behavior, a training-data pattern, or the effect of an intervention. Those are different objects. State the audience and purpose before choosing a method: debugging, reviewing an individual outcome, or making a scientific claim needs different evidence. A feature attribution explains a relationship inside a chosen model and representation, not automatically a person’s real cause or a valid action they can take. Keep the base model’s held-out quality and scope visible. Explaining a weak or leaked model precisely does not make its predictions trustworthy.

Global importance depends on the question

Permutation importance measures a model-score change after disturbing a feature under a chosen evaluation and permutation policy. It depends on the fitted model, metric, data distribution, and the way dependence is handled. Correlated inputs can substitute for one another, making individual importance appear small even when the information is useful. Arbitrary permutations can also create unrealistic combinations. Impurity-based tree importance has other biases and is not interchangeable with a held-out permutation measure. Report variability across justified repeats and keep the test-selection boundary intact. A ranking should be presented with its assumptions, not as a universal ordering of real-world causes.

Measure a fixed model’s perturbation response

The first example uses a known linear rule on four synthetic feature rows. It reverses one feature column while leaving the model fixed and compares mean absolute error. The result demonstrates a perturbation calculation, not a statistically stable importance estimate or a recommended permutation scheme for dependent data. The supplied target was generated from the same rule, so the zero baseline is constructed rather than an external achievement. A real analysis should use appropriate held-out outcomes, repeated valid perturbations, and an interpretation that acknowledges the dependency structure and metric. Do not refit the model during a standard fixed-model permutation check unless the question explicitly changes.

python Fixed synthetic offline example

import numpy as np
X = np.array([[0.0, 0.0], [1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
coef = np.array([1.0, 2.0])
y = X @ coef
changed = X.copy()
changed[:, 0] = changed[::-1, 0]
print(f"constructed baseline MAE={np.abs(y - X @ coef).mean():.1f}")
print(f"fixed perturbation MAE={np.abs(y - changed @ coef).mean():.1f}")

Expected output

constructed baseline MAE=0.0
fixed perturbation MAE=1.0

The baseline is exactly constructed from the rule. Reversing feature zero changes each prediction by one in this fixture.

Local additive terms depend on representation

The second example decomposes one linear prediction into intercept and feature terms, then rescales one feature and inversely rescales its coefficient. The prediction and corresponding contribution remain the same, while the raw coefficient changes. This exposes why coefficient magnitude alone is not a unit-independent importance measure. More complex local attribution methods also depend on a reference or background distribution and assumptions about missing or intervened features. Different legitimate references can answer different questions. A locally accurate explanation need not be globally stable, and a counterfactual suggestion can be infeasible or impermissible in the real domain.

python Fixed synthetic offline example

import numpy as np
x = np.array([2.0, 3.0])
coef = np.array([4.0, 5.0])
intercept = 1.0
terms = x * coef
scaled_x = x * np.array([100.0, 1.0])
scaled_coef = coef / np.array([100.0, 1.0])
print(terms.tolist())
print(float(intercept + terms.sum()))
print(np.allclose(terms, scaled_x * scaled_coef))

Expected output

[8.0, 15.0]
24.0
True

The original feature terms are eight and fifteen. A unit change alters a coefficient but preserves the corresponding contribution and prediction.

Validate explanations as part of a workflow

Test explanation stability under small justified input changes, alternative representations, and relevant slices. Verify that displayed feature names, units, and values match the actual inference pipeline. Do not expose sensitive training examples or private attributes through a debugging interface without a reviewed access policy. When causality is the question, use an appropriate causal design with explicit assumptions rather than reading it from a feature-importance chart. Record the model version, explanation method, background data identity, and known limitations. Human readability is valuable, but a persuasive story is not a substitute for fidelity to the model or evidence about the world.

Common errors and fixes

An importance chart is presented as causal effect

Why it happens: Model dependence was confused with intervention evidence.
How to fix it: State the explanation target and causal limitations.
Prevent it next time: Keep causal claims under a separate design.

Permutation creates impossible records

Why it happens: Feature dependence was ignored.
How to fix it: Choose a justified perturbation or qualify the interpretation.
Prevent it next time: Inspect domain validity and correlated substitutes.

Displayed explanations use the wrong units

Why it happens: The interface and fitted preprocessing are inconsistent.
How to fix it: Trace the exact inference representation.
Prevent it next time: Version explanation metadata with the model pipeline.

Check your understanding

1. Why can two correlated features each have low permutation importance?

Show worked solution

The model can retain similar predictive information through the unpermuted substitute, so disturbing one column need not greatly change its score. This does not prove both are irrelevant. Interpret the fixed-model question and dependency assumptions, and consider justified grouped or conditional analyses where appropriate.

2. Does changing a displayed influential feature guarantee a desired real-world outcome?