Grafana / Intermediate: transformations, configuration, and alerts / Lesson 11

Alert queries, expressions, and instances

Define a supported alert measurement independently of panel formatting, with explicit query references, reduction, label identity, timing, and no-data policy.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish Grafana-managed and data-source-managed rules.
Review query and server-expression capabilities.
Separate condition, pending state, and delivery.

Choose the rule’s execution and ownership model

Grafana-managed rules and data-source-managed rules have different ownership, execution, and supported operations. Not every source that renders a panel supports server-side alert evaluation, and external rule-management capabilities depend on the integration. Review backend plugin support, data-source credentials, edition, hosting, and version. A browser user successfully opening a dashboard does not prove a background evaluator can run the same query. Define who owns the measurement, the rule, and the response. Keep source-managed rule semantics separate from a Grafana-managed expression chain rather than treating all alert editors as one execution engine.

Make query references and reductions explicit

A Grafana-managed design can query a supported source, reduce a time series to a value, and evaluate a condition using supported server-side expressions. Query references identify stages in that chain. The editorial plan below names A, B, and C without claiming provisioning syntax. Panel transformations, units, colors, and browser-only logic are not automatically part of this evaluation. Review the exact label relationship and timestamp handling between stages. A math expression combining mismatched series can lose or mispair evidence if the supported matching rules are misunderstood.

yaml Editorial alert-expression chain

alert_design: transit-delay-lab
A: supported-server-side-source-query
B: reviewed-per-route-reduction-of-A
C: reviewed-condition-on-B
identity: stable-route-and-environment-labels
separate_policies: [pending, recovery, no-data, query-error, notification]

This is a design record, not Grafana provisioning or an expression API model. It creates no rule and specifies no operational threshold.

Each intended label set needs a clear outcome

Multi-dimensional evaluation can create distinct alert instances for different label sets, such as transit routes. Preserve stable identity and avoid putting a changing measurement into an identifying label. The local model treats one complete route as above a synthetic threshold and another incomplete route as unknown. It is not Grafana’s no-data or missing-series state machine. A whole query returning no data, an execution error, and one route disappearing can require different handling. Define the actual supported policies rather than assuming all unknown cases should become healthy or all should page.

python Offline per-route condition fixture

series = {"r1": [5.0, 40.0], "r2": [10.0, None]}
for route in sorted(series):
    values = series[route]
    if any(value is None for value in values):
        print(route, "unknown-model-input")
    else:
        print(route, "condition", max(values) > 30)

Expected output

r1 condition True
r2 unknown-model-input

The strict missing-value policy and threshold are synthetic. The program models neither Grafana reductions nor alert-instance state transitions.

Timing is part of the alert contract

Query range, evaluation interval, pending period, and supported recovery settings affect when a condition becomes actionable and clears. They are not interchangeable with dashboard refresh. A transient spike can appear in a panel while not satisfying a sustained condition, and an old last value can keep an appearance without fresh evidence. Define expected timestamps and state transitions in test cases, including boundary times and missing data. Rule identity changes may create new instances, so review labels during migrations. A condition becoming true is still separate from notification grouping, routing, and receiver delivery.

Test the supported path without claiming a live alert

Use known-value fixtures for normal, threshold, sustained, transient, missing, error, and recovery cases. Verify exact output labels, units, and expected evaluation times with the correct authorized integration when available. No real recipient should be contacted by an editorial exercise. This lesson executes only synthetic Python and creates no rule, query, evaluator, data source, alert instance, or notification. The fixed reference explains concepts, not current entitlements or a tested deployment. Record which engine and version, if any, actually ran before presenting a validation claim.

Common errors and fixes

A panel color is treated as the alert condition

Why it happens: Presentation settings replaced the independent evaluation contract.
How to fix it: Define the supported query and server expression.
Prevent it next time: Test panel and alert paths separately.

The evaluator cannot query a source the browser can display

Why it happens: Plugin or identity capabilities differ.
How to fix it: Review backend alert support and credentials.
Prevent it next time: Include execution identity in the environment contract.

A changing delay becomes an identifying label

Why it happens: Each value creates a different instance identity.
How to fix it: Keep stable dimensions and use suitable descriptions for values.
Prevent it next time: Test label stability and missing-series behavior.

Check your understanding

1. Why can a panel show a threshold breach while a rule remains normal or pending without either being defective?

Show worked solution

The query range, reducer, condition, evaluation time, and pending period can differ. Compare exact scope and source freshness before judging the state. A visual threshold is not the rule definition, and a transient display value may not satisfy a sustained server-side condition.

2. Should one missing route automatically be treated the same as a whole query error?