Data Analysis / Advanced: Defensible and Reproducible Decisions / Lesson 18

Communication and Responsible Decisions

Present evidence with its denominators, uncertainty, and limits so a decision-maker can assess what the analysis actually supports.

About 5 min readContent reviewed 2026-10-25

On this page

Lesson goals

Connect claims to auditable metrics and populations.
Separate observation, inference, recommendation, and value judgment.
Assess subgroup harm, privacy, and decision sensitivity.

Lead with a qualified answer to the actual question

A useful report answers the declared question, gives the supporting result, and explains the limits that could change its interpretation. Separate what was observed, what was inferred under assumptions, and what action is recommended. A recommendation also depends on costs, benefits, and values that a dataset may not measure. Do not disguise those choices as a purely statistical conclusion.

State the population, period, unit, numerator, denominator, and comparison near the claim. Use an effect scale understandable to the audience. A percentage-point change and a relative percentage change are different quantities; report the baseline so readers cannot confuse them.

Make absolute and relative change explicit

The first program compares rates of10% and15% with equal denominators. The absolute increase is five percentage points, while the relative increase is50%. Both are valid transformations, but saying “up50%” without a baseline can exaggerate or obscure practical magnitude. The counts show that the illustration involves ten versus fifteen events among one hundred opportunities each.

For unequal populations, report the actual denominators and design before comparing. A large-looking relative change from a tiny baseline may have a small absolute impact and wide uncertainty. Rounding and selected chart axes should not substitute for a clear explanation of scale.

python Two honest scales for the same contrast

before_events, before_total = 10, 100
after_events, after_total = 15, 100
before = before_events / before_total
after = after_events / after_total
print(f"before={before:.0%} after={after:.0%}")
print(f"percentage_points={(after - before) * 100:.1f} relative_change={(after / before - 1):.0%}")

Expected output

before=10% after=15%
percentage_points=5.0 relative_change=50%

The change is five percentage points and fifty percent relative to the ten-percent baseline.

Describe uncertainty without turning it into a slogan

Use intervals and sensitivity results with their assumptions. A confidence procedure’s repeated-sampling coverage is not a guarantee that this dataset is unbiased or that a chosen action is safe. Distinguish statistical uncertainty from measurement gaps, missingness, model assumptions, and uncertainty about future conditions. Some limitations cannot be compressed into one error bar.

Avoid equating “not statistically significant” with “no meaningful difference.” A wide interval can include important positive and negative effects. State what magnitude the evidence can rule out, what remains unresolved, and which additional collection would improve the decision. Keep exploratory findings labeled as such.

Test whether a choice depends on uncertain assumptions

The second program compares a predeclared decision threshold with a synthetic estimate interval. The interval crosses the threshold, so the rule reports uncertainty rather than treating the point estimate as sufficient. This is a communication illustration, not a universal decision rule or a probability calculation. Real decisions may use asymmetric consequences and additional evidence.

A responsible analysis can say that the data are insufficient for the requested distinction. Specify what would change the recommendation, and avoid selectively choosing a favorable endpoint or scenario. A sensitivity table is useful when several plausible definitions or missingness assumptions produce materially different actions.

python A threshold-sensitive interpretation

estimate = 0.12
interval_low, interval_high = 0.08, 0.16
threshold = 0.10
if interval_low > threshold:
    message = "above threshold under this interval"
elif interval_high < threshold:
    message = "below threshold under this interval"
else:
    message = "threshold remains unresolved"
print(f"estimate={estimate:.2f} interval=({interval_low:.2f},{interval_high:.2f})")
print(message)

Expected output

estimate=0.12 interval=(0.08,0.16)
threshold remains unresolved

The interval0.08..0.16 contains0.10, so the example does not claim a resolved threshold decision.

Audit who is represented and who bears the consequences

An overall metric can hide subgroup errors, unequal coverage, or burdens placed on people absent from the data. Inspect relevant subgroup denominators and measurement quality without inventing categories or exposing small sensitive cells. Fairness and acceptable tradeoffs require context and stakeholder judgment; no single metric supplies a universal ethical certificate.

Minimize personal data, use authorized purposes, and choose aggregation and disclosure controls appropriate to the actual setting. A chart or synthetic analysis does not establish legal compliance. Keep a review trail connecting claims to methods and limitations, and update conclusions when evidence changes. Responsible communication includes saying what was not measured.

Common errors and fixes

A five-point change is described as five percent.

Why it happens: Percentage points and relative percent are confused.
How to fix it: Report both with a baseline and counts.
Prevent it next time: Test the wording against a numeric example.

An interval crossing an important threshold is hidden.

Why it happens: The point estimate alone drives the recommendation.
How to fix it: State the unresolved range and sensitivity of the action.
Prevent it next time: Predeclare meaningful decision scales.

An overall improvement is called universally beneficial.

Why it happens: Coverage and subgroup consequences are unexamined.
How to fix it: Inspect relevant groups and disclose limitations.
Prevent it next time: Review affected populations with domain stakeholders.

Check your understanding

1. A rate rises from1% to2%. Write a precise statement and one missing piece of evidence.

Show worked solution

The rate increased by one percentage point, equivalent to a100% relative increase from the1% baseline. Add the actual event and exposure counts, time window, eligible population, and uncertainty before judging importance. The relative doubling alone does not reveal whether the difference is precise, practically large, or caused by a particular intervention.

2. A recommendation changes when missing outcomes are treated under two plausible scenarios. What should the report do?