SRE / Advanced: controlled failure and service readiness / Lesson 18
Production readiness and reliability planning
Combine user objectives, operational evidence, dependency risks, and recovery ownership into a review that produces concrete decisions and a sustainable improvement plan.
On this page
Lesson goals
- Assemble a risk-based readiness record.
- Choose an explicit engagement and ownership boundary.
- Prioritize improvements with acceptance evidence.
Readiness is an evidence conversation
A production readiness review asks whether a service can meet its promises and be operated responsibly within an agreed scope. It is not a certificate that failures cannot occur. Bring the user journey, SLIs and objectives, workload assumptions, dependency map, release plan, observability, on-call readiness, security boundaries, and recovery evidence. Scale the review to user consequences and system complexity. A small internal tool and a critical reservation system need different depth. Begin early enough that architecture can change; a review held only on launch day can become a rushed approval ritual rather than useful engineering.
Separate verified, assumed, and missing
For every important claim, state the evidence, its scope, date, and limitations. A passed unit test supports a local invariant; it does not establish regional failover readiness. A supplier SLA is relevant context but does not measure the integrated user journey. The editorial record marks recovery evidence as incomplete and assigns the resulting decision instead of hiding the gap behind a green checklist. Unknowns can be accepted for a bounded low-risk scope when the accountable owner understands them, but they should remain visible and time-limited. Avoid inventing evidence to make a launch appear complete.
review: library-synthetic-readiness
verified: [offline-sli-fixtures, documented-ownership]
assumed: [future-workload-mix]
missing: [isolated-end-to-end-recovery-evidence]
decision: limit-scope-until-required-evidence-reviewed
risk_owner: accountable-service-owner
review_trigger: evidence-or-scope-changes
The record does not claim production approval. It distinguishes known evidence from missing operational validation and names a decision owner.
Agree how SRE and product teams work together
SRE engagement can include early design consultation, readiness review, shared projects, or operational ownership with a clear agreement. The appropriate model depends on staffing, service criticality, maturity, and the ability to reduce recurring work. Do not transfer an unstable service to another team without transferring knowledge, engineering capacity, and decision rights. Define responsibilities for releases, incidents, data correctness, dependencies, and toil reduction. Name the conditions for revisiting or ending the arrangement. A written boundary prevents reliability work from becoming an unlimited queue of requests that no team can prioritize coherently.
Prioritize mechanisms with concrete completion evidence
Choose work by expected harm reduction, urgency, effort, and dependencies, while keeping uncertainty visible. The offline example orders a small fictional backlog by a simple risk score. The score is only a discussion aid: a regulatory obligation or prerequisite can override the arithmetic, and ordinal scores are not monetary expected loss. Attach a meaningful acceptance condition to each chosen improvement. For example, a deadline guard should reject expired work in a fixture, while a recovery project needs approved isolated restore evidence. Reserve capacity for this work rather than assuming it will happen between incidents.
backlog = [
("deadline-guard", 9),
("panel-colour-change", 1),
("restore-evidence-gap", 8),
]
for name, score in sorted(backlog, key=lambda item: (-item[1], item[0])):
print(name, score)
deadline-guard 9
restore-evidence-gap 8
panel-colour-change 1The fixed scores are synthetic and ordinal. Sorting cannot establish business priority, legal obligations, effort, or actual risk reduction.
Revisit the agreement as the service evolves
A readiness decision ages when traffic, dependencies, teams, data models, or user expectations change. Set review triggers and a sustainable cadence, then compare objectives, incidents, operational load, action outcomes, and new risks. Retire controls that no longer help and strengthen defenses that repeatedly fail. Preserve concise decision records so future owners can understand why a tradeoff was accepted. The course ends with an operating loop: define a user promise, measure it honestly, respond to harm, learn from evidence, and invest in improvements. No final checklist removes the need for continuing engineering judgment.
Common errors and fixes
A green checklist substitutes for missing evidence
- Why it happens
- Readiness was treated as document completion.
- How to fix it
- Mark unknowns and obtain or explicitly accept bounded evidence gaps.
- Prevent it next time
- Require claim scope and acceptance ownership.
Operational ownership moves without engineering capacity
- Why it happens
- The engagement boundary was incomplete.
- How to fix it
- Agree responsibilities, capacity, and escalation before transfer.
- Prevent it next time
- Review sustainability and exit conditions.
The reliability backlog never receives time
- Why it happens
- Improvements depended on spare time between incidents.
- How to fix it
- Reserve capacity and choose risk-reducing actions.
- Prevent it next time
- Review outcomes and recurring toil with leadership.
Check your understanding
1. A low-risk launch lacks a complete restore rehearsal. What should the review record?
Show worked solution
Record the missing evidence, affected promise, accepted exposure limit, accountable risk owner, compensating constraints, and the condition and date for review. Whether launch is acceptable requires the actual authorized decision process. Do not claim the restore works or hide the gap in a completed checkbox.
2. How would you assess whether the SRE engagement helped after three months?
Show worked solution
Compare the same serviceโs user objectives, incident mechanisms, actionable paging, recurring work, recovery evidence, and delivered risk-reducing improvements. Include workload and team changes. Avoid judging success only by fewer reports or more automation; ask whether outcomes improved and the operating agreement remains sustainable.
Key takeaways
- Readiness is scoped evidence and accountable risk acceptance.
- Engagement needs capacity and clear responsibilities.
- Reliability improvement is a continuing operating loop.