SRE / Intermediate: response and operational evidence / Lesson 7
Actionable burn-rate alerts
Connect a page to sustained user harm and a response, using burn-rate windows and explicit low-traffic and missing-data behavior.
On this page
Lesson goals
- Calculate normalized burn rate.
- Explain paired long and short windows.
- Define routing and actionable response context.
Start with an action, then choose the signal
A page should interrupt a responder because a meaningful action can reduce current or imminent user harm. A ticket can track slower work, and a dashboard can support exploration without interrupting anyone. For Library reservations, a page might report sustained bad eligible outcomes with a link to the relevant runbook and ownership route. A CPU threshold alone is usually insufficient context: the responder needs to know the affected service and safe next decision. Review whether each alert has a distinct action; several correlated dependency alarms may otherwise create many interruptions for one failure.
Normalize to the reliability allowance
Burn rate is the observed bad fraction divided by the allowed bad fraction. For a 99.9% objective, a one-percent bad fraction burns at ten times the allowance. The offline calculation also estimates how much of a twenty-eight-day budget one hour at that rate would consume under constant event-rate assumptions: about 1.49%. The estimate is useful for reasoning about urgency, not a universal alert threshold. Actual request-weighted budget use depends on traffic distribution. Keep the objective, interval, and numerator definition attached to the rate so that a later target change does not silently change its meaning.
allowed_bad_fraction = 0.001
observed_bad_fraction = 0.01
burn = observed_bad_fraction / allowed_bad_fraction
window_hours = 28 * 24
print(f"burn={burn:.1f}x")
print(f"one_hour_budget={100 * burn / window_hours:.2f}%")
burn=10.0x
one_hour_budget=1.49%The budget-use approximation assumes constant eligible event rate. No alert is evaluated against a monitoring service.
Use paired windows for a reason
A long window can establish that the problem consumes enough budget to matter, while a short window checks that it is still happening. Requiring both can improve reset behavior after recovery compared with a long-window condition alone. Several window pairs may support different urgency levels, but thresholds must be evaluated against the serviceโs traffic and response needs. The illustrative policy below deliberately omits product syntax. Do not copy a number from another organization without checking detection delay, false pages, low-volume behavior, and the time required for a responder to act.
alert: reservation-budget-burn
condition: long-window-high-burn AND short-window-high-burn
minimum_evidence: explicit-low-traffic-policy
route: library-primary-then-secondary
context: [sli-definition, impact, windows, runbook, last-release]
missing_data: separate-telemetry-condition
This is a conceptual contract, not executable Prometheus or paging configuration. The chosen thresholds require service-specific evaluation.
Treat sparse and missing evidence honestly
One failed request among two attempts produces a fifty-percent bad fraction, but the operational meaning differs from fifty thousand failures. A minimum event policy can reduce noise while potentially delaying detection for a small service. Independent synthetic checks or different objectives may be appropriate for critical low-volume journeys. Never turn an empty denominator into proof of health. Missing telemetry also needs its own routing and diagnosis because a blind monitoring path can coincide with an outage. Test these edge cases on historical or synthetic fixtures before deciding how the alert should page.
Operate the alert as a maintained product
Test firing, non-firing, recovery, routing, and notification content without contacting real responders during editorial exercises. After real incidents, ask whether the alert arrived soon enough, named the right scope, and led to a useful action. Group related notifications while retaining distinct user impact. Time-limit silences and record why they exist, because permanent suppression can hide future harm. Remove alerts that have no owner or response rather than leaving them to train people to ignore pages. Alert quality is measured by useful decisions and protected user experience, not the total number of rules.
Common errors and fixes
The page stays active long after recovery
- Why it happens
- Only a long-window condition was used.
- How to fix it
- Evaluate a short-window recovery condition with the long window.
- Prevent it next time
- Test recovery timing on incident fixtures.
One sparse failure repeatedly wakes the team
- Why it happens
- Traffic volume and criticality were ignored.
- How to fix it
- Define a low-volume strategy and independent evidence.
- Prevent it next time
- Review false pages together with missed incidents.
A silence hides a later outage
- Why it happens
- Suppression had no expiry or accountable owner.
- How to fix it
- Restore coverage and assess the hidden interval.
- Prevent it next time
- Require bounded silences and review ownership.
Check your understanding
1. For an allowed bad fraction of 0.0005 and observed 0.002, calculate burn.
Show worked solution
The rate is 4x. That identifies error pace relative to the objective, not an automatic instruction to page. Choose windows and urgency using expected budget loss, traffic, user criticality, and response time, then test both outage and recovery scenarios.
2. What should be included in a useful page?
Show worked solution
Include the service, user-facing symptom, affected scope, start or evaluation time, objective and window context, safe dashboard and runbook links, and ownership escalation. Avoid secrets or raw customer payloads. The recipient should be able to verify impact and choose a bounded first action.
Key takeaways
- A page needs a meaningful response.
- Burn-rate thresholds need traffic and window context.
- Test absence, low volume, and recovery explicitly.