Prometheus / Intermediate: PromQL and rule contracts / Lesson 10

Recording rules and derived series contracts

Design a recording rule as a versioned materialized measurement with explicit grouping, evaluation cadence, dependencies, and consumer expectations.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain what a recording rule stores.
Review rule identity and evaluation dependencies.
Validate derived meaning independently of shorter queries.

A recording rule creates another measurement contract

A recording rule periodically evaluates an expression and stores its result under a configured metric name. It can make expensive or widely reused calculations easier to consume, but it also creates another series family with its own labels and cadence. The rule does not repair a wrong source definition or automatically populate all earlier history. Consumers need to know whether they read raw counters, derived per-second rates, or a ratio. Name and document the aggregation and unit so a later query does not accidentally apply rate again to an already derived rate.

Read the rule as a typed, scoped interface

The non-runnable YAML fragment records an operation-level request rate for the synthetic warehouse job. The colon-separated name follows a descriptive convention rather than creating enforcement of correct aggregation. The configured thirty-second rule interval is illustrative and independent of the range used inside rate. The source selector, retained labels, window, and result unit all belong in review. A rule label can overwrite a result label, so treat added labels as semantic changes rather than harmless decoration. This fragment is not loaded into a server or claimed to pass promtool.

yaml Illustrative Prometheus v3.5.0 recording rule

groups:
  - name: warehouse-lab-rates
    interval: 30s
    rules:
      - record: operation:warehouse_requests:rate5m
        expr: sum by (operation) (rate(warehouse_requests_total{job="warehouse-lab"}[5m]))

The expression stores requests per second grouped by operation. Configuration, scheduling, and query evaluation remain untested platform behavior.

Check the output label shape with fixed values

The offline fixture takes three synthetic per-instance rates and aggregates them to two operations. It shows that instance identity is intentionally removed and that lookup becomes five requests per second. It does not calculate rates from counters or evaluate a recording rule. Verify both values and output dimensions in a real authorized rule test, including added labels and empty inputs. A correct total with incorrect labels can still break vector matching, alert routing, or dashboard navigation. Keep the example’s assumed rate unit explicit rather than treating every input float as interchangeable.

python Offline derived-label-shape fixture

source_rates = [("lookup", "a", 2.0), ("lookup", "b", 3.0), ("reserve", "a", 2.0)]
recorded = {}
for operation, instance, rate_value in source_rates:
    recorded[operation] = recorded.get(operation, 0.0) + rate_value
for operation in sorted(recorded):
    print(operation, recorded[operation])

Expected output

lookup 5.0
reserve 2.0

The inputs are already modeled rates. The script checks summation and grouping only, with no counters, timestamps, PromQL, or rule scheduler.

Make rule dependencies and freshness visible

Rules within a group have documented sequential evaluation behavior, but separate groups should not be treated as a universal dependency scheduler. A derived rule consuming another derived series adds freshness and ordering assumptions. Review the pinned semantics, group intervals, evaluation errors, and whether a consumer can see stale or missing upstream results. Avoid cycles and hidden chains that nobody can explain. A shorter dashboard expression may conceal several layers of aggregation and time windows. Preserve the dependency graph and meaning so responders can trace a surprising value back to original observations.

Review cost, compatibility, and retirement

Recording every possible grouping can increase storage and evaluation work rather than save resources. Choose reused calculations with a clear owner and consumer need, then test semantic changes before rollout. Renaming or regrouping a rule creates a migration problem for dashboards and alerts; old stored series do not disappear merely because a definition changes. Keep a compatibility and retirement plan under the authorized data lifecycle policy. This lesson executes only a local grouping fixture, not a rule engine, configuration reload, backfill, data deletion, or performance benchmark.

Common errors and fixes

rate is applied to a recorded per-second rate

Why it happens: Consumers lost the derived metric’s unit contract.
How to fix it: Read the rule definition and use appropriate aggregation.
Prevent it next time: Document unit and window in names and metadata.

A renamed rule breaks every alert

Why it happens: Consumer migration was omitted.
How to fix it: Update references with a reviewed transition plan.
Prevent it next time: Version rules and consumers together.

Separate groups are assumed to evaluate in dependency order

Why it happens: Scheduling assumptions exceeded documented guarantees.
How to fix it: Review actual group semantics and simplify dependencies.
Prevent it next time: Test upstream missing and stale results.

Check your understanding

1. A rule aggregates by operation and removes instance. Can a downstream query recover each instance’s rate from it?

Show worked solution

No. The aggregation discarded that dimension. Keep the raw or separately appropriate derived evidence if instance diagnosis is required. Do not attach arbitrary instance labels later and present them as recovered detail; labels cannot reconstruct information removed by aggregation.

2. What should a recording-rule review include besides whether the expression parses?