Observability / Advanced: bias, trust, and evidence engineering / Lesson 13

Sampling and selection bias

Explain what retained traces represent, compare early and late selection, and protect population measurements from biased diagnostic samples.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish sampling purpose from population counting.
Compare head and tail sampling constraints.
Explain why biased samples cannot yield raw failure rates.

Decide what the sample is for

Sampling controls how much telemetry is recorded or retained, often to balance diagnostic usefulness, processing cost, and storage. A uniform representative sample and an error-focused diagnostic sample answer different questions. Keeping every slow or failed trace can make rare mechanisms easier to inspect, but the retained fraction of failures no longer estimates the raw service failure rate. Use trustworthy population metrics for that question unless a valid weighting and inclusion model is established. Document the policy, stage, and revision beside trace searches so a responder knows which operations may be absent and why.

Compare early and late decisions

Head sampling decides near the beginning of a trace, before the final outcome is known. Tail sampling can use more of the completed trace’s behavior, but needs buffering, decision timing, and access to relevant spans. Distributed routing can split spans across samplers and weaken that decision unless the design preserves the required grouping. Late or unusually long spans may arrive after a decision. A rejected early recording decision cannot necessarily be recovered by a later sampler because the missing spans may never have been exported. Review actual SDK and collector semantics rather than treating sampling stages as interchangeable switches.

Demonstrate the bias with fixed outcomes

The offline population contains ninety successful and ten failed operations. A diagnostic policy retains every failure and only nine successes. The retained set reports more than half failures even though the full population failure rate is ten percent. No randomness is needed to see the bias. This fixture does not implement a statistically valid sampler or an estimator. It demonstrates why a trace-search result count needs policy context. A real weighting method would require known inclusion probabilities and assumptions that remain valid across outcomes, services, and changing policies.

python Offline selection-bias fixture

population_success, population_failure = 90, 10
retained_success, retained_failure = 9, 10
print("population_failure", f"{population_failure / (population_success + population_failure):.1%}")
print("retained_failure", f"{retained_failure / (retained_success + retained_failure):.1%}")

Expected output

population_failure 10.0%
retained_failure 52.6%

The retained set is deliberately outcome-biased. The two ratios describe different populations and must not be presented as the same service rate.

Preserve the sampling contract across signals

The editorial review below separates diagnostic retention from population accounting and calls out late-span behavior. Metrics, logs, and traces may each have different selection policies, so a cross-signal link does not imply matching completeness. An exemplar or example trace is a navigation aid, not proof that every metric observation has a retained trace. Keep privacy restrictions independent: an operation being interesting or failed does not justify retaining sensitive payloads. When policy changes, annotate the transition so volume and outcome comparisons do not mistake changed selection for changed service behavior.

yaml Illustrative sampling contract

sampling_review: parcel-diagnostic-traces
purpose: retain-useful-failure-examples
population_rates: separate-unsampled-outcome-metrics
required_context: [policy-version, selection-stage, late-span-policy]
claim_limit: retained-failure-fraction-is-not-service-failure-rate
privacy: same-field-policy-for-retained-and-discarded-paths

This record implements no sampler and prescribes no universal percentage. It states the evidence boundary and the required policy context.

Test which stories remain visible

Use synthetic successful, failed, slow, long-running, partial, and late-arriving traces to review selection behavior. Check whether all required spans reach the decision point and whether policy costs fit the expected workload. Retain a small unbiased or otherwise justified baseline when population comparisons are needed, and document its limits. This lesson runs only fixed arithmetic; it sends no spans, allocates no collector buffer, and tests no sampling implementation. A useful sampling design makes its omissions legible and preserves the evidence needed for its declared diagnostic questions.

Common errors and fixes

Trace search reports a huge service failure rate

Why it happens: Outcome-biased retention was treated as a population sample.
How to fix it: Use valid population metrics or a justified estimator.
Prevent it next time: Display sampling context beside trace counts.

Tail sampling cannot find the missing spans

Why it happens: Earlier recording or routing removed required evidence.
How to fix it: Review the complete sampling and routing path.
Prevent it next time: Test partial and late trace cases.

A policy change looks like a traffic surge

Why it happens: Selection revisions were not annotated.
How to fix it: Compare like policies and retain transition metadata.
Prevent it next time: Version sampling contracts with consumers.

Check your understanding

1. A policy retains all errors and one percent of successes. Can retained counts estimate raw error percentage directly?

Show worked solution

No. Errors and successes have different inclusion probabilities. Raw retained counts are biased toward errors. Use independent population evidence or a carefully justified estimator with known probabilities and limitations. Do not assume the policy remained uniform across services, time, or incomplete traces.

2. Why can a tail-sampling design need trace-aware routing?