Data Analysis / Advanced: Defensible and Reproducible Decisions / Lesson 15
Resampling, Tests, and Multiple Comparisons
Use resampling to illuminate uncertainty under stated assumptions and keep exploratory selection separate from confirmatory evidence.
On this page
Lesson goals
- Distinguish bootstrap sampling from label permutation.
- Interpret p-values and error rates without probability-of-truth claims.
- Account for a defined family of comparisons.
Resampling does not create new source information
Resampling reuses observed data under a specified procedure to approximate or enumerate a reference distribution. Bootstrap samples draw observations with replacement to study estimator variation; permutation methods rearrange labels under an exchangeability or randomization argument. The procedures answer different questions and require different assumptions. Increasing resample count reduces simulation noise, not selection bias in the original sample.
For clustered or time-dependent data, ordinary independent-row resampling can break the structure that determines uncertainty. A cluster or block procedure may be needed, and its design must match the target. Tiny datasets can make bootstrap distributions highly discrete and uninformative about unseen tails. Treat computational convenience as separate from inferential validity.
Enumerate a tiny bootstrap exactly
The first program enumerates every ordered size-two resample of values1 and3. There are four equally weighted resamples under this toy empirical distribution, with means1,2,2,3. The average of those means is2. This is a transparent arithmetic illustration, not a demonstration that a two-observation bootstrap interval has reliable population coverage.
A realistic workflow records the resampling unit, statistic, interval method, seed, number of draws, and software versions. Do not choose an interval formula solely because it is easy to calculate. Assess whether independence, sample size, and estimator behavior support that method, and compare with domain-relevant sensitivity checks.
from itertools import product
from statistics import mean
values = [1, 3]
means = [mean(sample) for sample in product(values, repeat=2)]
print("means", ",".join(map(str, means)))
print(f"resamples={len(means)} average={mean(means):.1f}")
means 1,2,2,3
resamples=4 average=2.0
The four ordered draws produce means1,2,2,3; no Monte Carlo randomness is needed.
Specify the null randomization distribution
The second program considers all six ways to assign two of four fixed outcomes to groupA. It counts absolute mean differences at least as large as the observed difference of two. Two assignments meet that condition, giving a two-sided exact permutation proportion of one third. The calculation is meaningful only under a justified equal-allocation null reference, not because any two observational groups can always be shuffled.
A p-value describes extremeness under the specified null procedure. It is not the probability that the null is true, the probability the finding is a mistake, or a measure of practical effect size. State the statistic and sidedness, and include ties according to the declared rule. Monte Carlo approximations need their own finite-simulation policy.
from itertools import combinations
from statistics import mean
values = [1, 2, 3, 4]
observed = 2.0
differences = []
for indices in combinations(range(4), 2):
chosen = set(indices)
first = [value for index, value in enumerate(values) if index in chosen]
second = [value for index, value in enumerate(values) if index not in chosen]
differences.append(abs(mean(second) - mean(first)))
extreme = sum(value >= observed for value in differences)
print(f"assignments={len(differences)} extreme={extreme} p={extreme / len(differences):.4f}")
assignments=6 extreme=2 p=0.3333
Two of the six equally weighted assignments have absolute mean difference at least two.
Many attempts change the error opportunity
Testing many outcomes, subgroups, time windows, or transformations creates a family of opportunities for a chance-looking finding. Choosing only the smallest p-value after exploration does not retain the same error interpretation as one predeclared test. Define the family and purpose before selecting an adjustment strategy. Family-wise error control and false-discovery-rate control target different quantities.
Bonferroni compares each of m valid p-values with alpha/m and provides a conservative family-wise bound without requiring independence. It cannot repair invalid p-values, hidden outcome selection, or a badly defined family. Sequential monitoring also needs a suitable plan; repeatedly checking an ordinary fixed-sample threshold until it passes changes its behavior.
Connect inference to effect and decision
Report effect estimates, uncertainty, sample design, test specification, and every relevant comparison, including unexciting results. A large p-value can reflect weak information rather than equivalence, and a small one can accompany a negligible effect. Separate a confirmatory conclusion from a hypothesis generated for a future study.
Reproducibility includes the exact resampling algorithm and random-stream context, not a seed alone. Use deterministic enumeration for tiny teaching examples and local generators for simulations. Check numerical stability and simulation error, but do not let extensive computation obscure the assumptions that make the reference distribution relevant.
Common errors and fixes
More bootstrap draws are claimed to fix a biased sample.
- Why it happens
- Simulation precision is confused with source representativeness.
- How to fix it
- Address sampling/selection assumptions separately.
- Prevent it next time
- Report resampling unit and source limitations.
A p-value is called the probability of no effect.
- Why it happens
- A conditional reference probability is reversed.
- How to fix it
- State the null procedure and report effect uncertainty separately.
- Prevent it next time
- Review interpretation text as carefully as code.
Only the best subgroup result is reported.
- Why it happens
- Exploratory selection hides the comparison family.
- How to fix it
- Report the search and apply a justified multiplicity/validation plan.
- Prevent it next time
- Predeclare primary outcomes where possible.
Check your understanding
1. Four valid tests form a predeclared family with alpha0.05. What is the Bonferroni per-test threshold, and what does it not guarantee?
Show worked solution
The threshold is0.05/4=0.0125. Under valid individual p-values, the method bounds the probability of at least one false rejection in that family by the chosen alpha. It does not prove a rejected effect is large, fix confounding, or legitimize extra undisclosed tests. State the family and effect estimates alongside adjusted decisions.
2. Why should ordinary independent-row bootstrap not automatically be used for daily observations with strong serial correlation?
Show worked solution
Resampling isolated days destroys the dependence pattern and can make estimator variability look too small or otherwise wrong. Choose a justified time-series or block-based procedure with a suitable block design, or use another model matching the process. More draws of the wrong resampling unit only approximate the wrong reference more precisely.
Key takeaways
- Bootstrap and permutation procedures need distinct validity arguments.
- A p-value is not a probability that a hypothesis is true.
- Multiplicity, selection, and effect size belong in the reported inference.