Prometheus / Basic: collection and time-series meaning / Lesson 6

Counter rates, increases, and resets

Interpret counter-derived rates with reset and window semantics, and avoid treating a simple subtraction or Python model as the PromQL engine algorithm.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain rate, increase, and irate purposes.
Apply rate before aggregating independent counters.
State sparse-sample and reset uncertainty.

A counter total is not a traffic rate

A cumulative request counter answers how many increments the instrument has accumulated since its start, subject to resets. rate estimates a per-second average rate over a range; increase expresses a window increase using related rate semantics; irate uses the last two relevant samples and is more responsive to short changes. Consult the pinned function definitions for exact behavior. Do not compare raw totals across instances with different lifetimes as if they measured current demand. A newly restarted busy process can have a smaller total than an old idle process while serving more traffic now.

Derive each rate before aggregation

The illustrative query computes each selected counter’s rate, then sums by operation. This preserves the ability to handle resets in individual series before their identities are collapsed. Summing counters first can hide one instance’s decrease behind another instance’s increase and distort reset interpretation. The five-minute range is an example, not a universal recommendation; it needs enough observations and suitable responsiveness. The result’s unit is requests per second when the source counts requests. Keep outcome and eligibility policy clear because a syntactically correct rate over the wrong event set remains misleading.

promql Illustrative rate-before-sum expression

sum by (operation) (
  rate(warehouse_requests_total{job="warehouse-lab"}[5m])
)

The expression is reference-based PromQL for the fixed baseline and is not executed. It sums per-series rates while retaining operation as a grouping label.

Use a small reset model without overclaiming

The offline fixture sees values 100, 110, 4, and 10. Under its explicit assumption that a decrease means reset to zero, observed adjusted increments total twenty rather than minus ninety. It cannot recover increments that occurred and disappeared between observations. Prometheus rate and increase also use timing and extrapolation rules that this toy loop omits; their results can be fractional even when event counts are integers. Label the model honestly. It tests one reset-aware accounting idea, not an equivalent implementation or an exact count of every real request.

python Offline simplified counter-reset model

values = [100, 110, 4, 10]
adjusted = 0
for previous, current in zip(values, values[1:]):
    adjusted += current - previous if current >= previous else current
print("naive_difference", values[-1] - values[0])
print("modeled_adjusted_increase", adjusted)

Expected output

naive_difference -90
modeled_adjusted_increase 20

The loop assumes resets to zero and omits timing, extrapolation, missed increments, and stale markers. It is explicitly not PromQL rate or increase.

Review window and sample availability

A range with too few usable samples cannot support the intended rate estimate. Missed scrapes, irregular timing, process churn, and label changes can reduce continuity. A very short window may be noisy, while a long one can smooth urgent changes. Choose according to the diagnostic or alerting decision, and inspect the collection interval and actual coverage rather than applying one fixed multiple blindly. No traffic, a zero estimated rate, absent data, and a newly created series are distinct states. Do not fill all of them with a reassuring zero merely to keep a graph continuous.

Keep rate interpretation tied to the event contract

A higher completion rate can reflect more demand, retries counted as independent attempts, or changed eligibility rather than better service. Pair rates with outcome and latency evidence when diagnosing user experience. Negative observations in a quantity expected to be monotonic deserve investigation of type, units, and instrumentation, not automatic application of reset logic. This lesson runs only the simplified local loop. No Prometheus binary, query API, scrape, or real counter is exercised, and no numerical output is claimed for the unexecuted PromQL expression.

Common errors and fixes

Raw counter totals are compared as current load

Why it happens: Process lifetime and resets were ignored.
How to fix it: Use an appropriate counter-derived rate with coverage context.
Prevent it next time: Document quantity and time semantics.

A restart creates a huge or negative traffic spike

Why it happens: Naive subtraction or aggregation hid reset behavior.
How to fix it: Handle each counter before aggregation using documented functions.
Prevent it next time: Include reset cases in rule fixtures.

A Python loop is called an exact rate implementation

Why it happens: Timing and extrapolation were omitted from the claim.
How to fix it: Limit the claim to the modeled accounting rule.
Prevent it next time: Keep engine validation separate from illustrative arithmetic.

Check your understanding

1. Why should rate normally be applied before sum for independently restarting instances?

Show worked solution

Reset detection belongs to each original counter sequence. Aggregating first can conceal a decrease in one sequence behind growth in another. Compute compatible per-series rates, then aggregate to the intended population. Preserve enough grouping to keep unrelated jobs or operations separate.

2. Why can increase return a non-integer for a request counter?