Data Analysis / Advanced: Defensible and Reproducible Decisions / Lesson 13
Comparisons, Experiments, and Causal Claims
Distinguish an observed contrast from the effect of an intervention and identify the design assumptions needed to connect them.
On this page
Lesson goals
- State a causal estimand and assignment unit.
- Explain confounding, randomization, and interference.
- Interpret before/after contrasts without automatic causal language.
An effect compares alternatives for the same target
A causal question asks what would differ under an intervention compared with a specified alternative. Observational groups can differ for reasons that also affect the outcome: baseline difficulty, time, selection, or another common cause. A measured association alone does not reveal the counterfactual outcome for the same units under the other condition.
Define the intervention, comparison, eligible units, outcome, timing, and aggregation rule. An average effect over accounts differs from one over events when accounts generate different event counts. Causal language should name the design and assumptions, not emerge merely because a chart shows two separated lines.
A difference in changes still needs assumptions
The first program subtracts the control group’s before/after change from the treatment group’s change. It produces five units as a descriptive contrast. Reading that number as a causal difference-in-differences estimate additionally requires assumptions such as comparable untreated trends, no relevant simultaneous group-specific shock, and a well-defined treatment. Two time points cannot establish those assumptions.
Compare several pre-intervention periods where available and inspect how groups were selected. A good-looking pretrend check does not prove every future assumption. Keep measurement changes and shifting group composition visible, because they can create an apparent effect even when the intervention has no influence.
control_before, control_after = 20, 23
treatment_before, treatment_after = 20, 28
control_change = control_after - control_before
treatment_change = treatment_after - treatment_before
contrast = treatment_change - control_change
print(f"control_change={control_change} treatment_change={treatment_change} contrast={contrast}")
control_change=3 treatment_change=8 contrast=5
The arithmetic gives8 minus3 equals5; the program does not validate a causal design.
Random assignment is a design operation
Random assignment can make treatment allocation independent of potential outcomes by design, supporting a causal comparison under the experiment’s other requirements. Random sampling instead concerns who enters the study; neither automatically supplies the other. The second program creates a small balanced assignment with a local seeded generator and verifies partition properties, not a treatment effect.
A seed makes this synthetic allocation reproducible in the stated environment. It does not guarantee perfect balance on every covariate or make the experiment representative of another population. Specify the unit randomized: individual, account, classroom, or cluster. Analysis and uncertainty must respect that assignment unit.
from random import Random
units = list(range(6))
treated = set(Random(7).sample(units, 3))
control = set(units) - treated
assert treated.isdisjoint(control)
assert treated | control == set(units)
print(f"treated={len(treated)} control={len(control)} overlap={len(treated & control)}")
treated=3 control=3 overlap=0
Three units enter each group, every unit is covered, and no unit belongs to both.
Protect the assignment and observation process
Noncompliance, attrition, contamination, and interference can complicate interpretation after randomization. An intention-to-treat contrast follows assigned groups and answers a different question from comparing only those who complied. If one unit’s treatment changes another unit’s outcome, an individual no-interference assumption may fail. Design clustering or another explicit strategy when needed.
Choose primary outcomes and observation windows before examining results. Repeated peeking, selective exclusions, or trying many outcomes can inflate false discoveries unless the design accounts for them. Record operational failures and missing outcomes by assigned group instead of deleting inconvenient records after seeing the effect.
Report magnitude, uncertainty, and scope together
A statistically detectable effect may be practically small, while a non-significant result may be too imprecise to exclude an important effect. Report the effect scale, interval or uncertainty analysis, sample/cluster counts, and design limitations. Do not translate failure to reject into proof of equivalence; equivalence needs a specified margin and suitable design.
For observational work, state the result as an association unless a justified identification strategy supports more. Sensitivity analysis can show how conclusions respond to plausible confounding or missingness assumptions, but it cannot prove an assumption true. Communicate what additional evidence or experiment would resolve the decision-relevant uncertainty.
Common errors and fixes
A before/after increase is called the intervention effect.
- Why it happens
- Other time changes lack a comparison or identification argument.
- How to fix it
- State the descriptive contrast and required causal assumptions.
- Prevent it next time
- Plan the comparison before observing outcomes.
A randomized study is assumed representative.
- Why it happens
- Random assignment was confused with random sampling.
- How to fix it
- Separate internal comparison from target-population generalization.
- Prevent it next time
- Document recruitment and assignment independently.
Only compliant participants are compared after assignment.
- Why it happens
- Post-assignment selection changes the groups.
- How to fix it
- Define the estimand and analyze under its justified policy.
- Prevent it next time
- Report attrition and compliance by assigned group.
Check your understanding
1. An experiment assigns treatment by classroom but reports each student as an independent randomized unit. What is wrong?
Show worked solution
The assignment unit is the classroom, and students within one classroom can share conditions and correlated outcomes. Treating every student as an independent assignment can overstate effective information and understate uncertainty. Use an analysis appropriate to the cluster design and report classroom as well as student counts. Randomization does not erase dependence inside a randomized cluster.
2. The difference in changes is5. What must an honest short report add before calling it a causal effect?
Show worked solution
Name the units, outcomes, periods, and group-selection rule; state the parallel-untreated-trend and other identification assumptions; examine available pre-period and composition evidence; and report uncertainty appropriate to the design. Without that support, label5 as a descriptive adjusted change, not a demonstrated intervention effect. A neat subtraction cannot supply the missing design.
Key takeaways
- Association, random sampling, and random assignment are different concepts.
- Causal claims require a design and explicit identification assumptions.
- Report practical magnitude and uncertainty without selective outcomes or exclusions.