Prometheus / Intermediate: PromQL and rule contracts / Lesson 7

Aggregation and meaningful error ratios

Use grouping and weighted counts to derive a service ratio while preserving eligibility, units, and the distinction between absent series and measured zero.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain by and without grouping choices.
Build compatible numerator and denominator populations.
Interpret ratios with unequal traffic and missing outcomes.

Choose the population before collapsing labels

Aggregation reduces series along declared dimensions. sum by(operation) retains the chosen grouping labels and drops others, while without(instance) removes named labels and can retain additional dimensions that appear later. Neither is universally safer: an unnoticed new label can fragment one result, while an overly narrow by list can merge environments. Define the intended population and source selectors first. Keep operation, environment, or tenant boundaries when the question requires them. A shorter expression is not necessarily a clearer measurement contract, especially when consumers assume labels that the aggregation removed.

Divide compatible rates rather than percentages

The error-ratio expression sums selected error rates by operation and divides by all eligible request rates with the same grouping. The source metric contract must establish eligibility; PromQL cannot infer whether malformed client input should count. The ratio is dimensionless because both terms have the same per-second event unit. Multiplying by one hundred changes presentation to percent, not the underlying population. Do not add an arbitrary constant to the denominator merely to avoid empty results. A zero denominator or absent numerator requires an explicit policy consistent with actual event and instrumentation semantics.

promql Illustrative operation error ratio

sum by (operation) (
  rate(warehouse_requests_total{job="warehouse-lab",outcome="error"}[5m])
)
/
sum by (operation) (
  rate(warehouse_requests_total{job="warehouse-lab"}[5m])
)

The unexecuted expression uses the same selected job and rate window for both terms. It assumes the source counts the agreed eligible attempts.

Show why averages of ratios mislead

The offline fixture compares two synthetic instances: one has one error among ten requests and another has one among nine hundred ninety. Averaging instance error percentages gives about 5.05%, while the combined population error fraction is 0.2%. Equal weighting is a valid separate statistic but does not answer the event-weighted service question. Rate-based aggregation has related population requirements, including compatible windows and reset handling. This Python calculation uses complete fixed-window counts and does not reproduce Prometheus extrapolation or evaluate the preceding query.

python Offline error-ratio weighting fixture

instances = [(1, 10), (1, 990)]
mean_error = sum(bad / total for bad, total in instances) / len(instances)
combined_error = sum(bad for bad, total in instances) / sum(total for bad, total in instances)
print("mean_instance_error", f"{mean_error:.2%}")
print("population_error", f"{combined_error:.2%}")

Expected output

mean_instance_error 5.05%
population_error 0.20%

The two errors belong to 1,000 total requests. The calculation shows weighting only, not a tested recording rule or PromQL engine result.

A missing error stream is not always a zero

An instrument may create an outcome series only after its first event. That can leave a ratio empty when no error series exists, even though eligible traffic is present. A collection failure can produce similar absence with a very different meaning. Review supported client-library initialization and query semantics rather than coercing every gap to zero. Keep expected traffic and telemetry completeness separate. A broad fallback expression can conceal a broken source or attach the wrong label set. Test no traffic, no errors, absent source, and newly created operation cases independently before using the ratio for paging.

Keep correctness visible through consumer changes

Review numerator and denominator together whenever outcome labels, eligibility, grouping, or instrumentation changes. A dashboard can display the same percent sign while selecting a different population after a rollout. Preserve the contract revision, query window, units, and source scope beside the result. Compare meaningful regional or operation slices when a global ratio can hide concentrated harm. This lesson executes only fixed arithmetic; no query engine, rule evaluator, data source, or monitoring service is accessed. The unexecuted expression’s limitations remain visible instead of claiming a measured reliability result.

Common errors and fixes

The service ratio is an average of instance ratios

Why it happens: Unequal traffic received equal weight.
How to fix it: Aggregate compatible event counts or rates before division.
Prevent it next time: Keep an uneven-traffic fixture.

A new label fragments a without aggregation

Why it happens: Retained dimensions changed without consumer review.
How to fix it: Compare the grouping contract with the new label set.
Prevent it next time: Test added and removed labels.

All empty results become healthy zero

Why it happens: Absent source and no-error states were conflated.
How to fix it: Define separate absence and no-traffic policies.
Prevent it next time: Verify collection completeness alongside the ratio.

Check your understanding

1. Why can two regions with identical error percentages be averaged without changing this particular answer?

Show worked solution

Equal percentages produce the same mean and combined ratio, so that fixture does not expose weighting mistakes. It does not justify the method for unequal percentages and traffic. Use an intentionally uneven case to protect the event-weighted contract and retain region scope when regional impact matters.

2. What must be checked before adding an or fallback to an empty ratio?