Prometheus / Advanced: scale, persistence, and safe validation / Lesson 17

Alertmanager routing, grouping, and suppression

Follow a firing alert through routing and grouping while distinguishing deduplication, inhibition, silences, and receiver delivery from rule evaluation.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Read a route tree and stable grouping labels.
Compare grouping, inhibition, and silences.
Define safe delivery verification without contacting responders.

A firing alert begins another system’s work

Prometheus evaluates alert rules and sends alert state to configured Alertmanager endpoints. Alertmanager then applies its own routing, grouping, deduplication, and notification behavior. A green rule evaluation does not prove that this handoff succeeded, and an accepted alert does not prove receiver delivery. Keep identities and external-label assumptions visible across replicas. Descriptive labels such as team can help routing but must come from a reviewed contract; they are not proof of a person’s availability or authorization. Name an owner for both the rule and the delivery path.

Read routes from broad defaults to matching branches

The non-runnable fragment has a default receiver and a child route for a bounded team label. Its receivers contain no notification integrations, so it provides no real delivery destination. The example omits timing settings to avoid presenting universal wait or repeat values. Review child matching, sibling traversal, and continue behavior under the pinned version before extending a route tree. A broad default can be useful for review, but it must not silently discard urgent alerts in an actual system. Test unknown, missing, and conflicting labels as well as the expected branch.

yaml Illustrative Alertmanager v0.28.1 routing fragment

route:
  receiver: training-default
  group_by: [alertname, job]
  routes:
    - matchers: ['team="stock-team"']
      receiver: training-stock
receivers:
  - name: training-default
  - name: training-stock

The receivers deliberately define no email, webhook, chat, or paging integration. The fragment is not loaded and sends no message.

Test the declared label decision locally

The offline router chooses one of two synthetic names from a fixed team value. It demonstrates the expected branch and missing-label default, not the full Alertmanager matcher language, route traversal, grouping, or delivery semantics. Keep that boundary explicit when reviewing results. A real route test must preserve the exact alert label set and configuration revision, and it should avoid contacting real responders unless explicitly authorized. Do not infer that a successful local branch selection validates credentials, network access, receiver templates, or notification retries.

python Offline routing-decision fixture

def receiver(labels):
    return "training-stock" if labels.get("team") == "stock-team" else "training-default"
print(receiver({"team": "stock-team", "alertname": "WarehouseTargetScrapeFailed"}))
print(receiver({"alertname": "WarehouseTargetScrapeFailed"}))
print(receiver({"team": "other-team"}))

Expected output

training-stock
training-default
training-default

The function evaluates fixed dictionaries and returns names only. It is not an Alertmanager test and has no notification side effects.

Different suppression mechanisms need different evidence

Grouping batches related alerts into notifications; deduplication manages repeated notifications for matching alert identity; inhibition suppresses a target alert when a source condition and equality rules apply; a silence matches alerts during a specified interval. These mechanisms do not repair user harm or stop rule evaluation. Missing labels can broaden matching in unexpected ways, particularly when equality assumptions are weak. Keep inhibition scope narrow and reviewed. Time-limit silences with a reason and owner under the actual operating policy, and verify that coverage returns rather than leaving a forgotten suppression after maintenance.

Verify useful delivery and recovery safely

A complete authorized test plan should cover known and unknown routes, grouping keys, duplicate alert sources, inhibited and non-inhibited cases, silence expiry, message content, and receiver failure. Keep secrets and private payloads out of annotations and links. Review whether a recipient gets enough scope and next-step context to act. This lesson executes only a local routing function; it starts no Alertmanager, sends no alert, creates no silence, and contacts no person or integration. The pinned configuration remains teaching material with no operational delivery guarantee.

Common errors and fixes

A firing alert is reported as a delivered notification

Why it happens: The handoff and receiver path were not verified.
How to fix it: Check each authorized stage separately.
Prevent it next time: Maintain rule and delivery evidence with separate owners.

Grouping removes the distinction needed for response

Why it happens: Grouping keys were chosen for appearance rather than action.
How to fix it: Keep the scope needed to make a useful decision.
Prevent it next time: Test multi-instance and multi-service cases.

A silence hides a later incident

Why it happens: Suppression had no reviewed expiry or ownership.
How to fix it: Restore coverage and assess the hidden interval.
Prevent it next time: Require bounded reasoned suppression and return checks.

Check your understanding

1. Why should missing team be included in a routing fixture?

Show worked solution

A producer, rule, or relabeling change can omit the label. The route must then follow an explicitly reviewed default or failure path rather than an accidental destination. A correct expected-team example alone does not prove the system handles unknown ownership or preserves urgent coverage.

2. Does an inhibition rule prove the suppressed symptom is harmless?