Prometheus / Intermediate: PromQL and rule contracts / Lesson 9

Classic histograms and quantile estimates

Read cumulative bucket series, preserve le during aggregation, and distinguish directly counted threshold fractions from interpolated quantile estimates.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain _bucket, _sum, and _count roles.
Aggregate compatible classic buckets before quantile estimation.
State bucket-resolution and population limitations.

A classic histogram is a related series family

A classic duration histogram exposes cumulative bucket counters distinguished by le, plus observation count and sum series under the usual exposition convention. Each bucket includes observations at or below its upper bound, so larger buckets contain earlier observations too. The positive-infinity bucket represents the total count. Do not sum all bucket values to get request volume. The _sum of nonnegative durations and _count can support an average through compatible rates, but that average does not reveal a slow tail. Keep observation boundary, unit, and eligibility consistent across the family.

Preserve the bucket boundary label

The unexecuted expression computes per-bucket rates, aggregates by le and operation, then estimates the ninety-fifth percentile. Removing le before histogram_quantile destroys the classic bucket structure the function needs. Aggregating compatible buckets first is different from averaging instance p95 values, which cannot reconstruct a population percentile. The estimate depends on bucket resolution and function interpolation rules. A value displayed with many decimals is not evidence of equivalent measurement precision. Check the pinned classic-histogram behavior and do not copy this series-family expression to native histograms without reviewing their distinct representation.

promql Illustrative classic-histogram p95 expression

histogram_quantile(0.95,
  sum by (le, operation) (
    rate(warehouse_request_duration_seconds_bucket{job="warehouse-lab"}[5m])
  )
)

The non-runnable expression preserves le and assumes compatible second-based buckets. No percentile result was evaluated by a query engine.

Use an exact boundary for a threshold question

If the question asks what fraction completes within half a second, a bucket exactly at that boundary can give a direct count relative to the compatible total. The offline fixture has five durations and three at or below 0.5 seconds, giving sixty percent. It also calculates the mean, which answers a different question. The model does not generate exposition or estimate a Prometheus quantile. A coarse bucket spanning the threshold cannot establish the exact within-threshold count without more information; interpolation is an estimate and must remain labeled as such.

python Offline histogram threshold fixture

durations = [0.05, 0.2, 0.4, 0.8, 1.4]
cumulative = [sum(value <= bound for value in durations) for bound in (0.1, 0.5, float("inf"))]
print("cumulative_buckets", cumulative)
print("within_half_second", f"{cumulative[1] / cumulative[-1]:.0%}")
print("mean_seconds", f"{sum(durations) / len(durations):.2f}")

Expected output

cumulative_buckets [1, 3, 5]
within_half_second 60%
mean_seconds 0.57

The bucket counts overlap cumulatively and total five observations at infinity. The exact fixture fraction is 60%; no histogram query engine is used.

Check the population before combining buckets

Compare bucket boundaries, units, metric meaning, source versions, and time windows before aggregation. A migration from milliseconds to seconds or a changed boundary set can produce incompatible families even when names look similar. Failed requests and timeouts may be absent from a duration instrument unless its contract includes them, making a healthy-looking percentile incomplete. Sparse traffic can move a high percentile sharply. Preserve count and source coverage beside the estimate, and distinguish no observations from a measured fast distribution. The chosen quantile does not establish correctness or availability by itself.

Review resolution and series cost together

Every classic bucket adds series for each label combination, and _sum and _count add their own streams. More boundaries improve resolution only where they are placed and increase collection, storage, and query work. Choose boundaries from actual diagnostic and objective questions rather than a universal list. Keep unbounded labels out of the histogram family because bucket multiplication amplifies their cost. This lesson executes a fixed list calculation only. No client histogram, scrape, query engine, native-histogram feature, or measured workload is created or tested.

Common errors and fixes

All buckets are summed as total traffic

Why it happens: Cumulative overlap was treated as disjoint counts.
How to fix it: Use the compatible total count or infinity bucket.
Prevent it next time: Test a small known distribution.

histogram_quantile receives buckets without le

Why it happens: Aggregation removed the required classic boundary label.
How to fix it: Retain le until the quantile function consumes the buckets.
Prevent it next time: Review grouping in rule fixtures.

Instance percentiles are averaged into service p95

Why it happens: Non-additive summaries replaced the underlying distribution.
How to fix it: Aggregate compatible buckets before estimation.
Prevent it next time: State resolution and sample count.

Check your understanding

1. Cumulative bucket counts at 0.1, 0.5, and infinity are 2, 8, and 10. How many lie above 0.1 and at most 0.5?

Show worked solution

There are 8 − 2 = 6 observations in that interval. The total is ten, not twenty. This uses cumulative upper-bound semantics and assumes one compatible family. A direct threshold fraction at 0.5 is 8/10, or eighty percent.

2. Why can a precise-looking p95 be misleading after a histogram migration?