Grafana / Editorial solution guide

The panel and alert disagree

Compare independent query, expression, identity, timing, and unknown-data contracts before assuming that a panel threshold controls alert evaluation.

About 3 min readContent reviewed

Read-only knowledge base · No public comments

Recognize the symptoms

A panel is red while the rule is normal or pending, or an alert fires while a stat appears healthy. The difference can follow a reducer, source, timing, or plugin change. A firing rule may also exist without a delivered notification, which is a separate problem.

Understand the likely causes

  • The panel and rule can use different data sources, credentials, ranges, reducers, transformations, evaluation intervals, pending periods, or missing-data policies. A source may support interactive rendering but lack the required server-side alert path. Display colors and field mappings do not automatically define the rule’s measurement.

Record both contracts side by side

Capture rule management type, selected Alertmanager scope, Grafana and plugin versions, hosting or edition, source UID, query references, absolute time context, and current evidence freshness. Compare the actual rule definition with the panel’s query and display chain. Do not infer the rule from a screenshot. Keep private query values, credentials, and notification destinations out of shared notes. This guide reads no rule or account and performs no alert preview, test notification, query, or source operation.

Separate range, reduction, and pending time

The offline dictionary comparison finds differences between synthetic panel and rule contracts: reducer, window, and hold duration. It proves only that the supplied settings differ. A current last value and a maximum over a longer window can disagree legitimately; a transient peak may not satisfy a sustained condition. Panel refresh is not the evaluator interval. Review actual supported semantics, including timezone, source delay, and overrides. Do not change every duration to the same number merely to force agreement if the views intentionally answer different decisions.

python Offline panel-versus-rule contract comparison 
panel = {"reducer":"last", "window_seconds":300, "hold_seconds":0}
rule = {"reducer":"max", "window_seconds":900, "hold_seconds":120}
differences = sorted(key for key in panel if panel[key] != rule[key])
print("contract_differences", differences)
Expected output
contract_differences ['hold_seconds', 'reducer', 'window_seconds']

The records are synthetic and not Grafana schema. The script evaluates no condition, pending state, query, or actual alert instance.

Check the server-side path and label identity

Verify that the chosen source and plugin support the rule’s server-side query or expression. Interactive credentials or browser-only processing may not apply to background evaluation. Panel transformations and unit formatting are not automatically reused. Compare label sets and multi-dimensional instance identity, especially after a rename or a disappearing route. Whole-query no data, one missing series, and execution error can have different configured handling. Keep those distinctions rather than mapping every unknown to normal or every gap to an urgent alert.

Repair the intended measurement and then verify delivery

Choose a reviewed measurement correction supported by evidence, or document why the panel and rule intentionally differ. Preserve source scope, units, identity, and missing-data meaning. Only after rule state is understood should notification policy, grouping, suppression, contact integration, and delivery be checked as separate stages. A successful receiver test does not establish policy matching for real alert labels. This guide changes no rule, dashboard, policy, contact point, silence, credential, or account and contacts no person or integration.

Verify the fix

Use known-value cases for normal, threshold, sustained, transient, missing-series, no-data, error, and recovery behavior in a separately authorized isolated integration. Verify exact instance labels and evaluation times. State whether only this local contract comparison ran. A matching screenshot or local arithmetic result cannot establish the supported evaluator, notification path, or production outcome.

Keep it from recurring

  • Version panel and alert measurements with explicit source, expression, labels, units, timing, and unknown-state rules. Keep notification tests separate and synthetic unless real contact is authorized. Review plugin upgrades and source-identity changes across both consumers so a display improvement cannot silently weaken a critical alert or conceal lost coverage.

Official references