SRE / Intermediate: response and operational evidence / Lesson 8

On-call readiness and runbook contracts

Design a sustainable responder rotation and a runbook that helps an authorized unfamiliar engineer verify impact, escalate, and choose a bounded action.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Define primary, secondary, and escalation responsibilities.
Review a runbook for preconditions and verification.
Measure operational burden and training gaps.

Coverage is an agreement between people

On-call means a named person or team can respond within an agreed expectation when service risk requires attention. Define coverage hours, acknowledgement expectations, escalation, and handoff. A schedule entry does not prove readiness: responders need current access, system understanding, communication channels, and permission to escalate early. Pair a primary with a secondary so one person is not the only path to recovery. Avoid promising round-the-clock coverage without enough staffing and recovery time. Product and engineering leaders must understand the operational cost of the service promises they approve.

Transfer state rather than just the pager

A handoff should cover active incidents, temporary mitigations, expiring silences, risky changes, and known measurement gaps. The illustrative record names an unresolved concern and its next decision, keeping assumptions separate from facts. The incoming responder should confirm that they understand the state and can reach the secondary. Avoid storing credentials or private customer details in the handoff document. An empty handoff should still confirm that no known active concerns remain. This creates continuity across time zones and reduces the chance that an important exception becomes invisible when its original owner leaves the rotation.

yaml Synthetic handoff record

handoff: library-training-shift
active_incident: none
known_gap: regional-probe-data-delayed
next_decision: reassess-coverage-at-agreed-checkpoint
mitigation_expiry: explicit-owner-and-time-required
secondary_contact: approved-internal-directory

The record carries operational state and ownership without exposing contact details. It does not change a schedule or notify anyone.

Write a decision path with safety boundaries

A runbook should state the symptom it addresses, scope, prerequisites, read-only checks, expected observations, possible branches, authorized mitigations, stop conditions, and verification. Explain why a command is relevant instead of presenting an unexplained block to paste. If target identity or expected state is ambiguous, the reader should stop and escalate. Distinguish a reversible configuration change from an operation that can destroy data. Link the service model and owners so the responder can reason beyond the happy path. A runbook cannot substitute for appropriate access approval or platform-specific operational training.

Use a safe tabletop to find missing branches

Practice with synthetic observations before an unfamiliar responder faces a real incident. The offline state classifier chooses escalation when identity is unknown, and a read-only verification step when impact remains unconfirmed. It does not perform an operational repair. Ask the learner to explain each decision and what evidence would change it. A tabletop may reveal an inaccessible dashboard, outdated dependency name, or missing secondary owner. Record those gaps as engineering work. Rehearsal builds shared understanding without requiring live faults, privileged access, or unapproved changes to a running service.

python Offline runbook branching model

def first_step(identity_known, impact_confirmed):
    if not identity_known:
        return "stop-and-escalate"
    if not impact_confirmed:
        return "verify-user-impact-read-only"
    return "follow-reviewed-mitigation-branch"
print(first_step(False, True))
print(first_step(True, False))

Expected output

stop-and-escalate
verify-user-impact-read-only

The function evaluates synthetic booleans only. Its final branch still requires a reviewed implementation-specific runbook and authorization.

Review the human system

Track interruptions, actionable pages, sleep disruption, recovery time, and the recurring work exposed by the rotation. A responder who is continually firefighting has little capacity to remove the causes. Escalate unsustainable load as a service design and staffing problem rather than treating endurance as a professional virtue. Keep training progression explicit: shadowing, reverse shadowing, tabletop practice, then supported ownership. Review runbooks after incidents and service changes. A healthy rotation distributes knowledge, protects recovery time, and turns operational observations into improvements that make future response less demanding.

Common errors and fixes

A new responder cannot access essential evidence

Why it happens: Schedule assignment was treated as readiness.
How to fix it: Verify access and supported practice before primary duty.
Prevent it next time: Maintain a readiness review with a secondary owner.

The runbook says restart everything

Why it happens: It lacks target, rationale, and stop conditions.
How to fix it: Replace the instruction with bounded evidence-driven branches.
Prevent it next time: Review side effects and verification for each action.

The team normalizes continuous overnight pages

Why it happens: Operational burden is invisible to planning.
How to fix it: Measure interruptions and prioritize reliability work.
Prevent it next time: Review sustainable coverage with leadership.

Check your understanding

1. What should an incoming responder ask about a temporary mitigation?

Show worked solution

Ask which user harm it addresses, its scope and owner, side effects, verification evidence, expiry, and the condition for removal. A mitigation without an expiry review can become an undocumented permanent change. Confirm what to do if the symptom returns or the original owner is unavailable.

2. A runbook action targets a cluster name that no longer exists. What next?