Observability / Intermediate: instrumentation and interpretation / Lesson 10

Query populations and valid aggregation

Build a query contract that keeps event scope, time windows, weighting, missing data, and label alignment explicit before trusting a result.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Compute a weighted population ratio.
Recognize invalid averages and joins.
Separate no traffic, zero, and missing evidence.

Name what belongs in the result

A telemetry query is a measurement definition, not just syntax accepted by a backend. State which services, environments, operations, outcomes, and time intervals belong to the result. A numerator that includes one region and a denominator that includes all regions can produce a plausible but false percentage. Review units and temporality before applying rates or sums. Keep query variables and default filters visible because a dashboard selection can change the population silently. A correct query over the wrong tenant or environment remains wrong for the user’s question and may also violate access expectations.

Aggregate counts before dividing

For a population success ratio, sum compatible good counts and eligible counts separately, then divide. Averaging per-instance percentages gives small and large instances equal weight even when their traffic differs greatly. The offline example has a ten-request cohort at fifty percent and a nine-hundred-ninety-request cohort at one hundred percent. The simple mean is seventy-five percent, while the combined population is 99.5%. Both are mathematically defined quantities, but only one answers the stated event-weighted question. Label any alternative weighting explicitly rather than presenting it as the same service ratio.

python Offline weighted-ratio comparison

cohorts = [(5, 10), (990, 990)]
mean_of_ratios = sum(good / total for good, total in cohorts) / len(cohorts)
population_ratio = sum(good for good, total in cohorts) / sum(total for good, total in cohorts)
print(f"mean_of_ratios={mean_of_ratios:.1%}")
print(f"population_ratio={population_ratio:.1%}")

Expected output

mean_of_ratios=75.0%
population_ratio=99.5%

The combined count is 995 good out of 1,000 eligible events. This fixture validates arithmetic, not any backend query language.

Keep interval and evaluation semantics visible

A rate, interval increase, rolling average, and instant value answer different questions. Query step and range selection can smooth or hide a short burst. Late data may revise a past interval, and two panels evaluated at different times can disagree while using similar expressions. Document whether timestamps refer to source events, ingestion, or evaluation. The editorial contract below fixes population, interval, weighting, and empty-input behavior without claiming product syntax. Verify exact backend rules for staleness, interpolation, reset handling, and range boundaries before translating the contract into a production expression.

yaml Illustrative query contract

query_contract: parcel-valid-update-success
population: same-service-environment-and-operation-family
window: explicitly-selected-utc-interval
aggregation: sum-good-divided-by-sum-eligible
empty_population: no-traffic-not-perfect-success
missing_source: unknown-with-coverage-warning

This record is not PromQL or SQL. It defines interpretation before choosing a backend expression and preserves distinct empty and missing states.

Review matching keys before joining signals

Joining metadata with metric streams can duplicate values, drop unmatched series, or attach the wrong identity if keys are not unique at the intended boundary. A display name is not necessarily a stable key. Review one-to-one and one-to-many expectations and what should happen to missing metadata. Do not suppress unmatched records merely to make a panel cleaner without understanding the affected population. Cross-signal links are often navigation aids rather than statistical joins; trace sampling and log filtering mean that a linked set may not represent the full metric population.

Test meanings with small known cases

Create fixtures for uneven traffic, no traffic, missing sources, counter resets, label changes, and delayed observations. Compute expected results independently before encoding the query. A green syntax check cannot detect the wrong denominator or a misleading display unit. Compare the query contract with the actual panel, alert, and export that consume it, because each may add filters or transformations. This lesson executes only a local ratio calculation and no query engine. Its expected values are exact for the synthetic list and provide no evidence about data-source permissions, backend performance, or installed query semantics.

Common errors and fixes

A service ratio is the average of instance ratios

Why it happens: Unequal traffic received equal weight.
How to fix it: Aggregate compatible counts before division.
Prevent it next time: Keep weighting explicit in query fixtures.

A join doubles a traffic total

Why it happens: Metadata keys matched several records.
How to fix it: Check uniqueness and intended matching cardinality.
Prevent it next time: Test duplicate and missing keys.

An empty result is rendered as success

Why it happens: No traffic and missing evidence were collapsed.
How to fix it: Preserve distinct states and coverage context.
Prevent it next time: Review empty-input behavior in every consumer.

Check your understanding

1. Two regions have 9/10 and 90/100 good events. What is the combined ratio?

Show worked solution

The combined count is 99/110, or 90%. In this particular case the simple average also gives 90% because both regional ratios are equal. That coincidence does not validate averaging percentages in general; use a deliberately uneven fixture as well to protect the intended weighting.

2. A panel and an alert use the same metric name but disagree. What should you compare?