Data Analysis / Intermediate: Relationships and Uncertainty / Lesson 12

Standard Errors, Confidence Intervals, and Assumptions

Explain what varies across samples, calculate a mean standard error under an explicit model, and communicate interval coverage without overstating what was learned.

About 5 min readContent reviewed 2026-10-25

On this page

Lesson goals

Distinguish population parameter, estimate, SD, and SE.
Interpret a confidence procedure through repeated sampling.
State dependence, distribution, selection, and sensitivity limits.

Identify the uncertain estimate and its target

A population mean is a target parameter; a sample mean is an estimate calculated from observed data. Under a sampling or probability model, another sample can produce another estimate even when the target parameter stays fixed. Uncertainty statements describe that estimation procedure under its assumptions, not an apology for a weak chart. Begin by naming the population, variable, observation unit, and estimator.

Descriptive precision and inferential validity are different. A computer can print many decimals for an estimate from a biased or dependent dataset. A standard error does not automatically account for coverage error, nonresponse, measurement mistakes, or model misspecification. State what the available data and design allow before selecting an interval formula.

Separate spread among values from variability of a mean

Sample SD measures spread among observations. For independent identically distributed observations with finite variance, the estimated standard error of the sample mean is s divided by square root of n. SE describes variability of the estimator across hypothetical samples, not typical differences between individuals. Its units remain the measurement's units. Dividing SD by n rather than square root of n overstates precision.

The first program uses sample SD with denominator n-1, equivalent to ddof=1 in a NumPy comparison. NumPy's default std uses a different denominator, so cross-tool checks must specify the convention. Four values produce SD about2.582 and SE about1.291. This is a worked arithmetic demonstration, not evidence that four arbitrary observations justify a reliable normal confidence interval.

python Calculate sample SD and a model-based mean SE

from math import sqrt
from statistics import mean, stdev

values = [8.0, 10.0, 12.0, 14.0]
n = len(values)
sample_sd = stdev(values)  # sample denominator n - 1
standard_error = sample_sd / sqrt(n)  # assumes independent common-distribution observations
print(f"mean {mean(values):.3f}")
print(f"sample_sd {sample_sd:.3f}")
print(f"mean_se {standard_error:.3f}")

Expected output

mean 11.000
sample_sd 2.582
mean_se 1.291

The sample variance is20/3. Its square root is about2.582, and dividing by sqrt(4) gives SE about1.291.

Attach confidence to the procedure, not a fixed-parameter probability

A95% confidence procedure is designed to cover the fixed target in about95% of repeated samples under its assumptions. Once one interval is calculated, it either contains that fixed value or does not; the frequentist statement is not a95% probability assigned to the parameter being inside this particular observed interval. It also does not mean95% of individual values lie inside it.

The second program uses an explicitly known population SD of4 and independent normal observations, the setting for a normal critical-value mean interval. It holds a hypothetical estimate of50 fixed while comparing n100 and400. The margin halves. When SD is estimated, a Student t procedure under suitable normal assumptions differs; do not automatically copy the known-SD formula into a small unknown-SD problem.

python Known-SD normal intervals under stated assumptions

from math import sqrt
from statistics import NormalDist

estimate = 50.0
known_population_sd = 4.0
critical = NormalDist().inv_cdf(0.975)
for n in (100, 400):
    margin = critical * known_population_sd / sqrt(n)
    print(f"n={n} margin={margin:.3f} interval={estimate-margin:.3f},{estimate+margin:.3f}")

Expected output

n=100 margin=0.784 interval=49.216,50.784
n=400 margin=0.392 interval=49.608,50.392

The97.5th normal percentile gives a two-sided95% procedure here. Holding estimate and SD fixed, quadrupling n halves the margin; no additional sample was collected.

Check dependence and design before using row count

Repeated measurements from one device, clustered respondents, time dependence, unequal sampling probabilities, and a large sampling fraction can require different variance calculations. A thousand rows from ten devices are not automatically a thousand independent units. Identify what was sampled and how dependence arises. The simple s/sqrt(n) calculation is not a universal uncertainty engine.

Small samples and severe skew or influential extremes can undermine an assumed approximation. Design-aware methods, appropriate models, or carefully justified resampling may be needed, but none repairs arbitrary selection bias by itself. Distinguish uncertainty caused by finite sampling from systematic uncertainty about measurement or representativeness. Document which assumptions are checked and which remain substantive judgments.

Report estimate, method, and sensitivity together

Present the point estimate, interval, confidence level, units, sample unit, effective design information, and method. Compare conclusions under defensible alternatives such as a different missing-data rule, subgroup definition, or plausible dependence model. Do not select the narrowest interval after trying many methods and report it as if prespecified. The same transparency applies to exploratory endpoint selection.

A wider confidence level generally produces a wider interval for the same method and data. A narrower interval does not automatically mean a more accurate or less biased estimate. Distinguish an interval for a mean from a prediction interval for a future observation. Explain practical magnitude and remaining limitations alongside precision so the reader can assess what decision the evidence supports.

Common errors and fixes

SE was interpreted as variation between people.

Why it happens: The estimator's sampling variability was confused with observation spread.
How to fix it: Label SD and SE separately with units and assumptions.
Prevent it next time: Specify the target and denominator in the analysis contract.

A95% interval was called a95% probability for the fixed parameter.

Why it happens: A repeated-sampling guarantee was assigned to one observed interval.
How to fix it: Explain long-run procedural coverage under the assumptions.
Prevent it next time: Use a fixed-parameter versus varying-interval diagram or verbal example.

Thousands of repeated rows produced implausibly tiny intervals.

Why it happens: Row count was treated as independent sample size.
How to fix it: Identify clusters or repeated units and use an appropriate variance design.
Prevent it next time: Audit the sampling unit before applying s/sqrt(n).

Check your understanding

1. Under the simple independent-observation model, SD is10. What are mean SEs for n100 and n400?

Show worked solution

They are10/sqrt(100)=1 and10/sqrt(400)=0.5. Quadrupling sample size halves SE while SD describes observation spread and is held fixed in this comparison. This does not guarantee the same reduction under clustering, changing populations, or systematic bias.

2. What does a95% confidence procedure promise, and what does it not promise about100 future intervals?