Observability / Basic: signals and measurement meaning / Lesson 5

Histograms and honest latency summaries

Choose distribution boundaries for a question, distinguish bucket representations, and avoid invalid percentile aggregation or unsupported precision.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Classify observations into explicit intervals.
Distinguish disjoint and cumulative bucket counts.
Explain estimation and compatible aggregation.

A mean cannot describe the whole population

Two processing workloads can have the same average but very different slow tails. A histogram summarizes how observations fall across value ranges, allowing questions about the proportion within a deadline or the shape of latency. Keep failures and timeouts in the measurement policy: measuring only successful completions can make a failing system look faster. Record the observation boundary and unit before designing buckets. A duration measured around one storage call is not interchangeable with end-to-end parcel-update delay, even when both are expressed in seconds and displayed on the same dashboard.

Put boundaries where decisions need them

Explicit bucket boundaries determine the resolution available near an important value. If a decision depends on completing within half a second, a matching boundary can support a direct count instead of interpolating between distant bounds. More buckets can improve resolution but increase data volume. OpenTelemetry also defines other histogram representations; backend and exporter support must be checked rather than assuming lossless conversion. The editorial record below states the quantity, boundary policy, and overflow expectation. It is a design example, not an SDK view or a collector configuration.

yaml Illustrative histogram design

distribution: parcel-processing-seconds
explicit_upper_bounds: [0.1, 0.5, 1.0]
includes_overflow: true
question: fraction-at-or-below-0.5-seconds
compatibility_check: unit-boundaries-population-and-temporality

A boundary at 0.5 seconds supports the stated threshold question. The record is not an OpenTelemetry API declaration.

Know whether the counts overlap

In a disjoint explicit-bucket representation, each observation belongs to one interval. Some exported formats instead represent cumulative counts at each upper bound, where the lower observations also appear in larger buckets. The offline example creates disjoint counts and derives cumulative counts from the same five observations. Summing cumulative bucket counts would count many observations repeatedly. Read the actual signal format before writing a query or transformation. The total count provides a useful consistency check, but it does not reveal whether the events were eligible or whether an upstream pipeline already dropped observations.

python Offline bucket representation model

values = [0.05, 0.2, 0.4, 0.8, 1.4]
bounds = [0.1, 0.5, 1.0]
disjoint = [0, 0, 0, 0]
for value in values:
    index = next((i for i, bound in enumerate(bounds) if value <= bound), len(bounds))
    disjoint[index] += 1
cumulative = [sum(disjoint[:i + 1]) for i in range(len(disjoint))]
print("disjoint", disjoint)
print("cumulative", cumulative)
print("within_half_second", f"{cumulative[1] / len(values):.0%}")

Expected output

disjoint [1, 2, 1, 1]
cumulative [1, 3, 4, 5]
within_half_second 60%

The last bucket is overflow. Three of five values meet the 0.5-second threshold, giving 60%; no telemetry engine is used.

Merge compatible evidence before estimating

Histograms can support population aggregation when their representations and semantics are compatible. Check units, boundaries or scale rules, event population, temporality, and time interval. Adding unlike buckets merely because they have the same array length can produce a plausible but meaningless distribution. Do not average per-instance p95 values to obtain a service p95; percentiles are not additive. Preserve sample count and estimation limitations. When conversion or rebucketing reduces resolution, make that loss explicit so a later reader does not interpret an interpolated tail value as an exact measured deadline.

Show what the distribution can establish

A histogram can establish the count in represented ranges, while an estimated percentile depends on the query method and available resolution. Sparse samples need particular caution: one slow observation can move a high percentile sharply. Compare traffic mix and missing data before attributing a distribution shift to a release. Keep threshold compliance and exploratory percentiles distinct when they answer different decisions. This lesson runs only a local bucket classifier; it does not test OpenTelemetry aggregation, Prometheus histogram queries, backend conversion, or a measured latency workload.

Common errors and fixes

Bucket totals exceed the observation count

Why it happens: Cumulative counts were summed as disjoint intervals.
How to fix it: Use the format’s correct count semantics.
Prevent it next time: Test a known small distribution before production queries.

Two histograms merge into a false distribution

Why it happens: Units or boundaries differ.
How to fix it: Align compatible representations or keep them separate.
Prevent it next time: Version histogram contracts with consumers.

Service p95 is the mean of instance p95

Why it happens: Non-additive percentiles were treated as totals.
How to fix it: Aggregate compatible underlying distributions first.
Prevent it next time: Document estimation methods and sample counts.

Check your understanding

1. Cumulative counts at 0.1, 0.5, and infinity are 2, 7, and 10. How many lie above 0.1 and at most 0.5?

Show worked solution

Subtract the lower cumulative count from the upper: 7 − 2 = 5 observations. The total is ten, not nineteen. This interpretation assumes the stated cumulative upper-bound semantics and one compatible observation population.

2. Can a histogram with boundaries 0.1 and 1.0 prove the exact fraction within 0.5?