DevOps QA / Advanced: operational validation and governance / Lesson 14

SLOs, error budgets, and delivery decisions

Use user-centered reliability indicators to inform delivery without confusing operational targets with test-suite pass rates.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Define eligible events, good events, and the measurement window.
Interpret budget consumption and burn rate with explicit data quality.

Define reliability from the user task

An SLI measures an aspect of service behavior relevant to users, such as successful eligible booking attempts completed within an agreed time. An SLO sets a target over a defined window. A test pass rate measures a different population and cannot be substituted for production reliability. Specify eligible traffic, success semantics, latency boundary, exclusions, and data source. A request returning HTTP 200 can still fail the user task, while a rejected invalid booking may be a correct outcome. Definitions should reflect the actual promise rather than whichever server counter is easiest to collect.

Calculate a budget with explicit assumptions

For a request-based SLO, the allowed bad-event fraction is one minus the target. Multiplying it by eligible events gives the corresponding event budget for that window. This arithmetic does not define an availability-time budget unless the SLO itself uses time-based accounting. Data completeness and event classification matter as much as the formula. The example uses synthetic counts and validates impossible inputs. A negative remaining budget indicates the target has been exceeded; it should not be silently clamped to zero if the magnitude is useful for policy review.

python Request-based budget arithmetic over synthetic counts

def event_budget(total, bad, target):
    if total <= 0 or not 0 <= bad <= total or not 0 < target < 1:
        raise ValueError("INVALID_SLO_INPUT")
    allowed_bad = total * (1 - target)
    return {"allowed_bad": allowed_bad, "remaining": allowed_bad - bad}

example = event_budget(100000, 80, 0.999)

The target is a teaching value, not a recommended SLO. Floating-point arithmetic is sufficient to illustrate the formula; production accounting should define rounding and measurement semantics. No service metrics are queried and the example does not authorize a release.

Read burn rate as consumption speed

Burn rate compares observed bad-event fraction with the fraction allowed by the SLO. A value of one spends budget at the target’s sustainable average rate under the chosen accounting model; larger values consume faster. Short windows detect rapid harm but can be noisy, while long windows provide context but may react slowly. Multiwindow approaches combine these tradeoffs. Keep the same event definition and adequate traffic across windows. A low-volume interval with no failures does not establish safety, and a telemetry outage should not produce an artificial zero burn rate.

Separate reliability policy from a deployment command

An error-budget policy can slow risky changes when reliability is degraded and prioritize repair, while still allowing changes that reduce an active incident. It needs accountable ownership and explicit exception handling. The synthetic record below states missing-data behavior and distinguishes reliability work from ordinary feature delivery. It is not a standard SRE schema or executable vendor configuration. The gate must verify current trustworthy indicators and candidate risk; it should not automatically release an untested fix merely because that change is labeled reliability. Policy expresses a decision process, not immunity from judgment.

json An application-defined error-budget policy

{
  "policy": "lab-booking-reliability",
  "window": "rolling-28-days",
  "whenBudgetExhausted": "hold-risk-increasing-features",
  "reliabilityFixes": "review-with-targeted-evidence",
  "missingIndicators": "inconclusive",
  "exceptionOwner": "service-maintainers"
}

Twenty-eight days is an illustrative window. A real policy defines SLIs, target, data validation, exception approval, and review cadence with stakeholders. The JSON contains no deployment instructions and does not inspect a live service.

Validate the indicator before trusting the gate

Use known synthetic event sets to verify good and bad classification, late data, duplicates, exclusions, and denominator handling. Compare backend indicators with a user-journey perspective where practical. Record measurement changes because moving the boundary can improve a number without improving users’ experience. SLOs guide reliability tradeoffs and monitoring; they do not cover all security, privacy, data integrity, or accessibility requirements. Keep those controls explicit. Review incidents where the SLO stayed green despite harm, and revise the indicator rather than dismissing the incident as irrelevant to quality.

Common errors and fixes

No metrics are reported as zero errors

Why it happens: Missing series are converted to healthy values.
How to fix it: Require data completeness and report inconclusive.
Prevent it next time: Test telemetry outages in policy evaluation.

Test pass rate is called the service SLO

Why it happens: Different event populations are conflated.
How to fix it: Define user-centered eligible and good service events.
Prevent it next time: Keep delivery and service indicators labeled separately.

An urgent fix bypasses all evidence

Why it happens: The reliability label is treated as automatic authorization.
How to fix it: Use the documented incident exception and targeted checks.
Prevent it next time: Review emergency decisions and close follow-up gaps.

Check your understanding

1. An SLO target is 99.9% and a window contains 100,000 eligible events with 80 bad events. Explain the budget.

Show worked solution

The allowed fraction is 0.1%, corresponding to about 100 bad events under request-based accounting. About 20 remain. This says nothing by itself about security or the next candidate’s correctness, and the result depends on valid complete event classification.

2. The service is within budget but a release corrupts one private note. Should the green SLO settle the decision?