Observability / Intermediate: instrumentation and interpretation / Lesson 12

Alerts and measurement quality

Distinguish a user-impact alert from a telemetry-coverage alert and review evaluation, missing data, routing context, and recovery as separate contracts.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Define the action behind an alert condition.
Model unknown and stale input explicitly.
Test firing, non-firing, and recovery cases.

A condition should lead to a useful decision

An alert condition evaluates evidence; a notification asks someone to respond. These are separate steps with separate failure modes. For parcel processing, a sustained rise in update age may justify investigating current user harm, while an absent emitter may indicate loss of coverage. The same person may own both, but the diagnosis differs. Start with the intended action, affected population, urgency, and owner. Do not page on every interesting anomaly. A trend useful for capacity planning can belong in a report or ticket rather than interrupting a responder who has no immediate safe action.

A quiet signal can mean several things

No eligible events, a measured zero, a missing series, and a stale last value are different states. An alert that treats them all as healthy can miss incidents; an alert that treats them all as failure can create noise during valid inactivity. Define expected activity and independent coverage evidence. The offline classifier separates fresh measured values from unknown evidence before checking a threshold. Its toy threshold is not an operational recommendation. Actual query engines and alert systems have their own absence and staleness rules, which must be verified for the deployed version.

python Offline quality-aware condition model

def evaluate(value, fresh, threshold):
    if value is None or not fresh:
        return "unknown-evidence"
    return "condition-true" if value > threshold else "condition-false"
print(evaluate(0, True, 10))
print(evaluate(None, True, 10))
print(evaluate(20, False, 10))
print(evaluate(20, True, 10))

Expected output

condition-false
unknown-evidence
unknown-evidence
condition-true

The function checks fixed synthetic inputs only. It does not evaluate an alert rule, send a notification, or claim a useful production threshold.

Review timing and identity through the rule lifecycle

An alert can require a condition to persist before firing, and it may resolve according to later evaluations. Evaluation interval, lookback window, pending duration, and data delay all influence detection and recovery. Stable alert identity matters for grouping and deduplication; placing a changing value into an identifying label can create a new alert each evaluation. Put explanatory values in appropriate descriptive fields instead. Check exact product semantics rather than assuming every system implements pending or recovery identically. A correct expression can still be operationally ineffective if its timing exceeds the useful response window.

Carry enough context without exposing payloads

The editorial alert contract below distinguishes service impact from coverage loss, names the query definition, and requires a safe verification link. Include service, environment, scope, evaluation time, owner, and next diagnostic step. Do not include credentials, customer addresses, or raw request content in a page or its URL. Routing should reach an accountable responder and have an escalation path; labels that look correct in a panel may still fail notification matching. Keep the message understandable to someone who did not author the query and who is reading under time pressure.

yaml Illustrative alert evidence contract

alert_design: parcel-update-age
impact_condition: reviewed-user-freshness-query
coverage_condition: separate-source-freshness-check
identity: stable-service-environment-and-rule
context: [query-contract, evaluation-time, safe-verification-link, owner]
message_policy: no-customer-payload-or-credentials

This design record is not a Prometheus or Grafana rule. It separates evidence quality from user impact and performs no routing or notification.

Test the complete reasoning path safely

Use synthetic fixtures for healthy, failing, sparse, missing, stale, delayed, and recovering evidence. Verify the expected state and message content without contacting real responders. Where an authorized isolated rule engine is available, test its syntax and lifecycle separately from the pure model. After actual incidents, review whether the alert arrived in time, selected the right scope, and prompted a useful action. Remove or redesign rules that repeatedly produce no decision. This lesson executes only the local classifier; no monitoring engine, scheduler, notification integration, or pager is started or contacted.

Common errors and fixes

An absent source is reported as zero failures

Why it happens: Missing evidence was coerced into a numeric zero.
How to fix it: Represent coverage uncertainty independently.
Prevent it next time: Test absence and source freshness explicitly.

A new alert appears every evaluation

Why it happens: A changing value was part of identifying labels.
How to fix it: Use stable identity and descriptive fields for values.
Prevent it next time: Review grouping and deduplication semantics.

A rule fires but nobody receives it

Why it happens: Condition evaluation was confused with notification delivery.
How to fix it: Inspect routing and ownership in an authorized test path.
Prevent it next time: Verify evaluation and delivery as separate contracts.

Check your understanding

1. A telemetry source is intentionally idle overnight. Should absence always page?

Show worked solution

No. Define expected activity, service criticality, and independent reachability or coverage evidence. An idle event stream is not automatically broken, but stale expected heartbeat evidence may be a different condition. Preserve those meanings and test the schedule and timezone assumptions rather than suppressing all overnight problems.

2. Why test recovery as well as firing?