Data Analysis / Basic: Questions and Trustworthy Tables / Lesson 5
Descriptive Statistics and Distributions
Describe location and variation together and choose summary definitions that match the data and the question.
On this page
Lesson goals
- Compare mean, median, spread, and distribution shape.
- Distinguish sample and population variance conventions.
- Specify quantile and outlier policies explicitly.
A center alone does not describe a dataset
A distribution describes how values are arranged, not only where an average lies. Report a suitable center alongside spread, sample size, and shape. A symmetric compact group and a long-tailed group can share a mean while representing very different experiences. Examine a plot or sorted values before compressing everything into one number.
The arithmetic mean uses every value and is sensitive to extreme observations. The median describes a middle position and is often less sensitive to a very long tail. Neither is automatically the correct answer: the mean can match a total-per-unit question, while the median can describe a typical ordered observation. State which question each answers.
Explain why two centers disagree
The first program includes one large value among three small ones. Its mean is26.5 and median2.5. The disagreement is evidence about the distribution, not a reason to hide one statistic. Removing100 solely to make the average look typical would change the population under study. Check whether it is a unit error or a genuine event before deciding.
Always show the count and contextual units in a report. A mean from four observations deserves different uncertainty treatment from the same mean across a carefully designed large sample. The descriptive calculation itself does not establish representativeness or a causal explanation for the tail.
from statistics import mean, median
values = [1, 2, 3, 100]
print(f"n={len(values)} mean={mean(values):.2f} median={median(values):.2f}")
n=4 mean=26.50 median=2.50
The large observation pulls the arithmetic mean while the two middle values determine the median.
Name the denominator in a spread calculation
Population variance divides squared deviations by N when describing the complete finite set under that convention. The usual unbiased sample-variance estimator uses N minus one under appropriate independent-sampling assumptions. NumPy var defaults to ddof=0; statistics.variance uses the sample convention. A library default is not a choice of analytical purpose.
The second example sets ddof explicitly so the difference is visible. Standard deviation is in the original measurement units, while variance is in squared units. Standard error of a mean is a different quantity describing estimator variability under assumptions, not the spread of individual records. Later lessons develop that distinction.
import numpy as np
values = np.array([1.0, 2.0, 3.0], dtype=np.float64)
population = np.var(values, ddof=0)
sample = np.var(values, ddof=1)
print(f"population={population:.2f} sample={sample:.2f}")
population=0.67 sample=1.00
With NumPy1.26.4, the same squared deviations use denominators3 and2, yielding0.67 and1.00.
Quantiles need an interpolation definition
A percentile summarizes an ordered position, but finite-sample algorithms differ in interpolation and endpoint rules. Python statistics.quantiles offers named methods, and numerical libraries expose their own documented choices. Record the method when comparing reports, especially with small samples. Two tools can disagree without either miscalculating its selected definition.
Range, interquartile range, and robust spread measures describe different features. A box-plot fence is a review convention, not proof that every point beyond it is erroneous. Multiple modes or subgroup mixtures can make a single center misleading. Show subgroup distributions when the data-generating process suggests meaningful differences.
Use precision that the evidence supports
Do not report many decimal places merely because software prints them. Match display precision to measurement resolution and the intended comparison, while preserving enough internal precision for calculations. Rounding can make small groups appear tied or percentages fail to sum exactly to100; explain that presentation effect rather than silently altering values.
Before computing statistics, validate finite values, missingness, units, and the observation grain. NaN can have surprising ordering behavior, and all-missing or too-small groups need a no-result policy. A summary table should include count and relevant missing counts, and any inferred population statement should wait for a justified sampling design.
Common errors and fixes
Two variance reports disagree.
- Why it happens
- They use different ddof or population/sample conventions.
- How to fix it
- Record the selected denominator and compare equivalent definitions.
- Prevent it next time
- Test a small hand-calculated dataset.
A typical-value claim uses only a mean from a skewed group.
- Why it happens
- Distribution shape is omitted.
- How to fix it
- Show median, spread, and a distribution view as appropriate.
- Prevent it next time
- Inspect raw or sorted values before summarizing.
An outlier fence becomes an automatic deletion rule.
- Why it happens
- A descriptive convention is treated as error proof.
- How to fix it
- Investigate and report sensitivity.
- Prevent it next time
- Preserve flagged observations and reasons.
Check your understanding
1. For values[1,2,3,100], explain what the mean and median each communicate without declaring either false.
Show worked solution
The mean26.5 is the total106 divided by four and is useful for total-per-observation reasoning. The median2.5 lies halfway between the two middle ordered values and reflects the central rank. The large value is real in the synthetic data, so the disagreement should prompt a distribution explanation rather than selective reporting. Neither statistic establishes a population conclusion.
2. Why is standard error not a substitute for standard deviation in describing individual wait times?
Show worked solution
Standard deviation describes variation among individual measurements; standard error describes uncertainty in an estimated mean under a sampling model. A large sample can make the mean precise while individual waits remain highly variable. Report the quantity matching the question, with units, sample size, and assumptions, rather than using a smaller number to make service variability look low.
Key takeaways
- Describe center, spread, count, and shape together.
- Variance and quantile conventions must be explicit.
- Descriptive summaries do not establish sampling validity or causation.