Prometheus / Intermediate: PromQL and rule contracts / Lesson 11

Alert rules and state lifecycles

Read alert activation as vector membership, distinguish pending from firing, and keep stable identity, descriptions, and notification delivery as separate concerns.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain expression presence and for duration.
Separate labels from annotations.
Test missing, recovering, and unstable-identity cases.

Presence matters more than a boolean-looking number

An alert rule becomes active for vector elements returned by its expression. A filtering comparison such as up == 0 can return a series whose numeric value is zero and still represent an active alert. Do not treat the output like a Python truth test. Adding the bool comparison modifier can retain nonmatching series with zero-valued results, which is often wrong for an alert intended to select failures only. Review expression membership and labels explicitly. The rule’s existence, successful evaluation, active condition, firing state, and delivered notification are distinct milestones.

Read a rule with its observation boundary

The illustrative rule selects failed scrapes for the synthetic warehouse job and requires two minutes of continued activity before firing. That duration is a teaching choice, not a recommended incident threshold. It observes collection failure rather than user availability. Labels define identifying and routing context; annotations hold descriptive material that can change without becoming a new identity. Never place credentials or raw customer payloads in either. The fragment is non-runnable and has not been loaded, evaluated, or connected to an Alertmanager. Its apparent simplicity does not remove the need for ownership and a response plan.

yaml Illustrative Prometheus v3.5.0 alert rule

groups:
  - name: warehouse-lab-alerts
    rules:
      - alert: WarehouseTargetScrapeFailed
        expr: up{job="warehouse-lab"} == 0
        for: 2m
        labels:
          severity: training
        annotations:
          summary: Synthetic warehouse collection failure

The rule selects failed collection and uses a stable training label. It is not operational paging configuration or evidence of a fired alert.

Model elapsed activity without inventing a scheduler

The offline classifier takes declared activity and elapsed continuous time. It reports pending before the synthetic hold duration, firing at the boundary, and inactive when the condition is absent. It does not calculate elapsed time from samples or reproduce restart, evaluation-gap, or recovery behavior. Actual for semantics depend on successful repeated evaluations and stable alert identity under the documented implementation. Two evaluations do not automatically mean two minutes. Keep time-based expectations explicit in tests, including the exact evaluation timestamps rather than just a list of true and false values.

python Offline alert-state boundary model

def state(active, continuous_seconds, hold_seconds):
    if not active:
        return "inactive"
    return "firing" if continuous_seconds >= hold_seconds else "pending"
print(state(True, 60, 120))
print(state(True, 120, 120))
print(state(False, 0, 120))

Expected output

pending
firing
inactive

The elapsed times are fixture inputs, not measured or scheduled intervals. No Prometheus alert state, notification, or persistence behavior is tested.

Keep changing values out of identity labels

A label containing the latest error count or formatted current value can create a new alert identity at every evaluation. That can reset pending behavior, fragment grouping, and produce confusing notifications. Put suitable changing explanation in annotations instead, while keeping service, environment, and severity labels stable under a deliberate routing contract. Added labels can overwrite expression labels, so review target identity preservation. A missing source can remove the expression result without proving recovery. Define separate coverage and user-impact conditions rather than relying on one alert to explain every kind of absence.

Test firing, recovery, and delivery independently

Use synthetic cases for healthy, failed, transient, sustained, missing, and recovering input. Verify expected labels and annotations as well as timing. A rule unit test can validate engine behavior when the correct authorized binary is available; this course uses only the local classifier and keeps all platform snippets false. Notification routing needs its own safe test plan that does not contact real responders without authorization. After incidents, review whether the condition selected the right scope and prompted a useful action, rather than measuring quality by the number of rules installed.

Common errors and fixes

A returned zero is assumed to mean no alert

Why it happens: Vector membership was confused with numeric truthiness.
How to fix it: Inspect filtering semantics and returned label sets.
Prevent it next time: Test zero-valued active elements.

Pending never reaches firing

Why it happens: A changing label creates a new identity each evaluation.
How to fix it: Move changing descriptions into annotations.
Prevent it next time: Assert stable identifying labels.

Disappearing data is called recovery

Why it happens: Source absence removed the selected series.
How to fix it: Check coverage and expected activity separately.
Prevent it next time: Include missing-source cases in tests.

Check your understanding

1. Why can up == bool 0 be inappropriate as a direct failure alert expression?

Show worked solution

The bool modifier can retain both matching and nonmatching vector elements with one or zero values. Alert activation is based on returned elements, not ordinary boolean truth of their numeric value. Use the documented filtering expression for the intended failure population and test returned identities.

2. What evidence is still missing after an alert rule unit test passes?