Data Analysis / Intermediate: Relationships and Uncertainty / Lesson 11

Sampling Frames, Selection, and Weights

Connect the population of interest to the records that can actually be selected, and distinguish sampling variability from coverage and response bias.

About 5 min readContent reviewed 2026-10-25

On this page

Lesson goals

Distinguish target population, frame, selected sample, and respondents.
Explain known nonzero inclusion probabilities.
Use weights with explicit targets and limitations.

An available table is not automatically the population

The target population is the set about which the question is asked. The sampling frame is the operational list or mechanism from which units can be selected. An app-user directory may omit people who never installed the app, even if every listed account is measured perfectly. Duplicate entries and outdated eligibility can also distort coverage. Define the unit and reference period before selecting records.

Trace the chain from target population to frame, selected sample, respondents, and final valid observations. Each transition can exclude people differently. Filtering to complete records is another selection step, not harmless preparation. A random sample from an incomplete frame supports a frame-based claim under its assumptions; it does not automatically generalize to excluded units.

Describe the selection mechanism, not just a seed

A probability design gives frame units known nonzero inclusion probabilities through a specified random mechanism. Simple random sampling without replacement of two from four assigns equal probability to each two-unit subset. The first program enumerates all six possible subsets and finds that every unit appears in three, so its inclusion probability is one-half under that design. Enumeration is not a collected survey.

A fixed random seed can make a software draw reproducible but does not make a convenience frame representative. Probability sampling can use unequal inclusion probabilities, stratification, or clusters; “random” does not always mean equal weights. Record the design and any dependence between selected units so later uncertainty estimates reflect how the observations were obtained.

python Enumerate a small probability design

from itertools import combinations

frame = ("a", "b", "c", "d")
possible_samples = list(combinations(frame, 2))
inclusions = [sum(unit in sample for sample in possible_samples) for unit in frame]
probabilities = [count / len(possible_samples) for count in inclusions]
print("samples", len(possible_samples))
print("inclusions", inclusions)
print("probabilities", probabilities)

Expected output

samples 6
inclusions [3, 3, 3, 3]
probabilities [0.5, 0.5, 0.5, 0.5]

There are six equally likely two-unit subsets under the stated design. Every frame unit appears in three, giving inclusion probability0.5.

Separate participation from the original draw

Selected units may not respond, and response can depend on the outcome or related circumstances. A voluntary satisfaction form may attract unusually pleased or upset users. A high response rate is useful operational information but not proof of low bias; the difference between respondents and nonrespondents matters. Record unit nonresponse separately from unanswered items within otherwise responding records.

Compare participation and complete-record retention by available relevant groups, while recognizing that observed characteristics may not capture the actual bias mechanism. More responses from the same selective process can reduce a standard error while leaving systematic error. Do not describe a large opt-in table as a probability sample merely because it contains many records or has a random row order.

State the population represented by a weighted estimate

Base weights commonly use inverse inclusion probability. The second program has population stratum sizes200 and800 but sample counts40 and10, giving base weights5 and80. Its unweighted sample mean is14, while the weighted estimate is26 using observed stratum means10 and30. The weighted target reflects known population shares rather than the disproportionate sample allocation.

This arithmetic does not prove either sample mean represents its stratum. Nonresponse adjustments or calibration can address specified differences under assumptions, not recover arbitrary missing groups or unmeasured selection mechanisms. Inspect extreme weights and report any trimming policy and sensitivity. Variance estimation must account for the design; pretending weighted observations form a simple independent sample of the raw row count can be misleading.

python Weight a disproportionate stratified sample

strata = [
    {"population": 200, "sample": 40, "mean": 10.0},
    {"population": 800, "sample": 10, "mean": 30.0},
]
weights = [s["population"] / s["sample"] for s in strata]
unweighted = sum(s["sample"] * s["mean"] for s in strata) / sum(s["sample"] for s in strata)
weighted = sum(s["sample"] * w * s["mean"] for s, w in zip(strata, weights)) / sum(
    s["sample"] * w for s, w in zip(strata, weights)
)
print("weights", weights)
print(f"unweighted {unweighted:.1f}")
print(f"weighted {weighted:.1f}")

Expected output

weights [5.0, 80.0]
unweighted 14.0
weighted 26.0

Sample shares80%/20% yield mean14; the specified population shares20%/80% yield26. Weighting changes the represented target under stated assumptions.

Report uncertainty and selection limits together

A sampling standard error describes variability under a sampling or model assumption. It does not include every coverage, measurement, processing, or nonresponse error. Describe the frame, design, response process, exclusions, weights, and remaining limitations alongside the estimate. Avoid treating a narrow interval as evidence that every source of bias is negligible.

Before collecting more data, ask whether the key limitation is random variation or a systematic exclusion. Improving frame coverage or follow-up can be more relevant than increasing a selective sample's size. A defensible report distinguishes what the design supports, what adjustments assume, and which unobserved populations remain outside the evidence. Those qualifications are part of the result rather than an optional disclaimer.

Common errors and fixes

A large convenience sample was called representative.

Why it happens: Size was confused with a valid coverage and selection design.
How to fix it: Describe the actual frame and restrict or redesign the population claim.
Prevent it next time: Audit coverage and selection before calculating intervals.

Nonrespondents were silently excluded from the report.

Why it happens: Response was assumed unrelated to relevant outcomes.
How to fix it: Report response and retention by known groups and assess plausible differences.
Prevent it next time: Preserve eligibility and response indicators.

A weighted mean was presented as automatically unbiased.

Why it happens: Known weights were assumed to repair every missing or selective process.
How to fix it: State the target, adjustment assumptions, weight diagnostics, and sensitivity.
Prevent it next time: Separate design weights from unsupported bias claims.

Check your understanding

1. A survey samples app accounts randomly but targets all residents. What limitation survives the random draw?

Show worked solution

Residents without an eligible app account are uncovered by the frame and have zero chance of entering that draw. Randomness among accounts does not restore them. Broaden the frame or restrict the population claim, and investigate duplicate accounts if the intended unit is a person.

2. A sample has40 observations averaging10 and10 averaging30, while population shares are20% and80%. Compute both means.