Prometheus / Advanced: scale, persistence, and safe validation / Lesson 13

Cardinality, churn, and ingestion limits

Review the number and turnover of series, estimate label multiplication, and distinguish protective scrape limits from a valid dimensional design.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Estimate classic-histogram series combinations.
Separate active cardinality from churn over time.
Explain the failure scope of selected scrape limits.

Count combinations rather than labels alone

Three bounded labels can still create a large series set because their values combine. A classic histogram multiplies each combination by its bucket and auxiliary series. The offline estimate uses twelve operations, four outcomes, three regions, and ten histogram-related streams per combination, producing 1,440 possible streams. Real counts depend on emitted combinations, metric families, exporter behavior, and target identity. A multiplication bound is not a memory or disk forecast. Use it to challenge an instrumentation proposal and identify which dimension answers a necessary diagnostic question.

python Offline histogram-cardinality estimate

operations, outcomes, regions = 12, 4, 3
classic_bucket_and_auxiliary_series = 10
series_bound = operations * outcomes * regions * classic_bucket_and_auxiliary_series
print("bounded_series", series_bound)
print("with_100_build_values", series_bound * 100)

Expected output

bounded_series 1440
with_100_build_values 144000

The ten streams are a declared model input, not a universal histogram layout. Adding 100 build values multiplies possible combinations, not measured storage.

A stable active count can hide heavy turnover

Churn is the creation and retirement of identities over time. A short-lived instance label or per-request identifier can create many historical series even if only a small set is active at one instant. Review new-value rate, lifecycle, retention, and query requirements together. Do not remove instance blindly if it is needed to diagnose a faulty target, but do not add build, URL, or error-text values without bounds and purpose. Privacy risk and cost often share the same source: unnecessary fine-grained identity. Keep individual investigation detail in a suitable authorized event system where appropriate.

Protect the server without hiding limit failures

The illustrative job fragment sets a sample limit and label-count limit, with synthetic numbers chosen only to show syntax. Under the pinned configuration semantics, exceeding such limits can fail the scrape rather than retaining a convenient first subset as a valid complete observation. Review the exact point where each limit applies, including metric relabeling, and inspect generated collection health. A sample cap does not solve all churn or query cost. It can also remove coverage suddenly, so choose it with expected workloads, owners, and an explicit response to limit failures.

yaml Illustrative per-job protective-limit fragment

sample_limit: 5000
label_limit: 20

This incomplete fragment is not loaded or recommended for a real service. The values illustrate fields and do not establish safe capacity or partial-scrape behavior.

Find the dimension that changed

When ingestion or memory pressure changes after a release, compare metric families, label value sets, target count, and instance turnover through approved bounded evidence. A new raw path label can dominate even when request throughput is unchanged. Do not scan private label values or export a full production series inventory into a public document. Attribute the growth to a concrete contract change before proposing a limit increase. Removing a dimension must preserve valid identity or use explicit aggregation; merging distinct samples by label deletion is not an equivalent cost optimization.

Measure usefulness after a bounded change

An authorized remediation should compare series count, new-series rate, collection success, query usefulness, and important diagnostic slices. Lower volume alone is not success if regional failures or ownership disappear. Version the metric and consumer migration and define rollback criteria. This lesson executes only fixed multiplication and supplies an unexecuted configuration fragment. It performs no series scan, memory benchmark, server tuning, scrape, data deletion, or production investigation. Actual capacity and limit behavior require the correct installed version and isolated authorized evidence.

Common errors and fixes

A low active count proves cardinality is harmless

Why it happens: Historical churn and retention were ignored.
How to fix it: Review identity turnover and consumer needs.
Prevent it next time: Budget new values as well as current combinations.

A sample limit is treated as safe partial ingestion

Why it happens: The scrape-failure semantics were misunderstood.
How to fix it: Read the pinned limit behavior and expose collection failure.
Prevent it next time: Test limit boundaries with the appropriate isolated tool.

Dropping a label is called equivalent aggregation

Why it happens: Distinct identities collide without a sum operation.
How to fix it: Review final uniqueness and query semantics.
Prevent it next time: Keep migration fixtures for collisions and missing slices.

Check your understanding

1. Why can a request-ID label be costly even if only one request is active at a time?

Show worked solution

Each new identifier can create a new series identity, leaving substantial churn and retained history. Low instantaneous concurrency does not bound the historical series set. Use a bounded metric dimension and appropriate restricted event evidence for individual requests.

2. What must be verified after reducing histogram buckets to control cost?