SRE / Intermediate: response and operational evidence / Lesson 11

Postmortems and effective follow-up actions

Write a learning review that reconstructs context, explains contributing conditions, and produces owned verifiable improvements without blaming the nearest person.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Reconstruct the incident with evidence and uncertainty.
Distinguish trigger, mechanism, and contributing conditions.
Design actions with owners and acceptance evidence.

Review the system people worked within

A postmortem explains how an incident developed and how the organization can reduce future harm. A blameless approach examines the information, incentives, tools, and constraints available to participants at the time. It does not erase accountability for improving the system. “An engineer pressed the wrong button” stops analysis before asking why the interface allowed an ambiguous destructive action, why review missed it, and why recovery was difficult. Invite the people who operated the system to explain their reasoning. Psychological safety helps expose weak signals and near misses that punishment would drive underground.

Build a factual account before a tidy story

Include user impact, scope, duration, detection, response, mitigation, and residual consequences. Align timestamps and preserve links to safe evidence. Label uncertain intervals and conflicting observations rather than inventing precision. Record what went well as well as what failed: a tested recovery path or a clear handoff may have limited harm. Avoid retrospective certainty when participants faced incomplete information. The synthetic record separates the release trigger from a queue-retention mechanism and two contributing conditions. It illustrates a causal explanation to test, not a claim about a real service.

yaml Synthetic causal account

trigger: candidate-release-enabled-new-retry-path
mechanism: expired-work-remained-in-unbounded-queue
contributing_conditions:
  - deadline-not-propagated
  - queue-age-not-visible-in-review
impact: reservation-deadlines-missed
uncertainty: precise-first-expired-item-time

A trigger starts the sequence, while the mechanism explains how harm occurred. Contributing conditions identify defenses that can be improved.

Specify an observable completion condition

An action should name its owner, intended risk reduction, scope, priority, and acceptance evidence. “Be more careful” is neither a testable safeguard nor a durable system change. “Reject expired queued work and prove it with an offline deadline fixture” gives a reviewer something concrete to inspect. Not every action needs a new tool: simplifying a configuration boundary or removing a fragile feature can be more effective. Balance prevention, detection, mitigation, and recovery work. Prefer a small number of high-value actions over a long list that cannot realistically receive engineering capacity.

Track outcomes rather than closed tickets

The offline review checks whether synthetic action records contain an owner and acceptance evidence. It is a narrow completeness check, not a judgment of action quality. After implementation, verify the intended failure defense and review whether operational burden actually changed. A ticket can be closed because code merged even when the rollout or runbook training remains incomplete. Give overdue high-risk work a visible escalation route. If an action is cancelled, record the accepted risk and decision owner instead of leaving an unexplained checkbox that future responders cannot interpret.

python Offline follow-up completeness check

actions = [
    {"id": "deadline-guard", "owner": "library-team", "evidence": "expiry-fixture-passes"},
    {"id": "dashboard-review", "owner": "", "evidence": ""},
]
for action in actions:
    complete = bool(action["owner"] and action["evidence"])
    print(action["id"], "specified" if complete else "needs-definition")

Expected output

deadline-guard specified
dashboard-review needs-definition

The checker flags missing fields only. It cannot establish that the guard works, the evidence is genuine, or the action reduces risk.

Look across incidents for recurring conditions

Periodic review can reveal repeated dependency confusion, missing ownership, unsafe defaults, or recovery gaps across otherwise different incidents. Compare mechanisms and defense failures rather than ranking teams by incident count. More reports can indicate a healthier reporting culture rather than worse reliability. Share a sanitized lesson with the people who can use it and update service design reviews accordingly. Keep reviews proportionate: a minor near miss may need a concise record, while a serious event deserves deeper reconstruction. The goal is sustained learning that changes engineering decisions before similar conditions combine again.

Common errors and fixes

The report ends at human error

Why it happens: Analysis ignored the decision context and safeguards.
How to fix it: Examine interface, information, review, and recovery conditions.
Prevent it next time: Ask how the system made the action plausible.

Many actions remain ownerless

Why it happens: The review produced wishes rather than commitments.
How to fix it: Assign accountable owners and acceptance evidence.
Prevent it next time: Review capacity and priority before approving actions.

Closed actions do not prevent recurrence

Why it happens: Completion was measured by ticket state alone.
How to fix it: Verify the intended defense with meaningful evidence.
Prevent it next time: Revisit recurring mechanisms across incidents.

Check your understanding

1. Rewrite “the operator must avoid mistakes” as a useful action.

Show worked solution

Identify the concrete hazardous condition first. For ambiguous target selection, require explicit environment identity and a reviewed preview that refuses an unknown target, then verify with a negative fixture. Assign an owner and acceptance evidence. This changes a defense rather than asking a person to become infallible.

2. Why record an effective incident handoff in the review?