SRE / Intermediate: response and operational evidence / Lesson 10

Hypothesis-driven troubleshooting

Use a service model, falsifiable hypotheses, and low-risk observations to narrow failures without confusing correlation, mitigation, and causal explanation.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Frame a precise failure statement.
Choose observations that distinguish hypotheses.
Separate restoration evidence from causal evidence.

Describe the difference from expected behavior

Begin with what fails, for whom, since when, and under which conditions. “The service is slow” is weaker than “valid reservations in region west exceed the confirmation deadline after the new release, while read-only catalogue requests remain normal.” Compare affected and unaffected populations using the same measurement definition. Include known telemetry gaps. This framing narrows the search and makes a hypothesis testable. Keep the user impact visible even when investigating an internal component, because a technically interesting anomaly may be unrelated to the incident that needs mitigation.

Draw the path and its queues

Map request admission, application work, connection acquisition, storage, and response delivery. Ask where work can wait or be lost. A long database span could represent connection waiting, lock contention, network delay, or query execution depending on instrumentation. Name those boundaries rather than jumping directly to database tuning. The editorial hypothesis record links a predicted observation to an alternative explanation. A useful model need not include every implementation detail; it should expose the state and dependencies necessary to distinguish plausible causes and identify a bounded safe observation.

yaml Synthetic falsifiable hypothesis

hypothesis: connection-pool-wait-dominates
prediction: acquisition-time-rises-before-query-time
comparison: same-route-and-region-before-after-release
alternative: instrumentation-boundary-changed
next_observation: read-only-pool-wait-and-version-evidence

The prediction distinguishes acquisition delay from query execution and checks an instrumentation alternative. It performs no database operation.

Choose discriminating evidence

Prefer an observation whose possible outcomes separate competing explanations. If only the new release fails under the same traffic class and region, compare release-specific behavior before changing shared infrastructure. The offline table computes synthetic bad fractions for two cohorts. The difference supports further investigation, but unequal request shapes or assignment bias could still explain it. Do not read causal certainty from the arithmetic. Record what result would weaken the hypothesis, and stop collecting repetitive evidence once a more informative boundary is available. The best next check is often smaller than a broad diagnostic dump.

python Offline cohort comparison

cohorts = {"control": (2, 1000), "candidate": (40, 1000)}
for name, (bad, eligible) in cohorts.items():
    print(f"{name}: {bad / eligible:.3%}")

Expected output

control: 0.200%
candidate: 4.000%

The candidate has a higher synthetic bad fraction. The calculation does not randomize traffic or establish causation.

Treat changes as experiments with risk

A production change made during an incident can both mitigate harm and destroy evidence. Decide whether restoration urgency justifies it, keep scope bounded, record the prior state, and define verification and reversal. Avoid changing timeout, retry count, pool size, and replica count together: even if the graph improves, attribution becomes difficult and side effects may remain. Read-only observations are usually preferable when they can answer the question promptly, but they still need appropriate access and privacy boundaries. This course executes only synthetic local calculations, not operational experiments against a running system.

Close the reasoning loop

After an observation or mitigation, compare the predicted result with the actual result and update the hypothesis. A restart that helps establishes that some reset state mattered; it does not explain which state or why it accumulated. Preserve unresolved questions for later analysis and distinguish confidence levels. Build a minimal reproduction from sanitized or synthetic conditions when practical, then test the suspected mechanism safely. Document dead ends that would otherwise be repeated. Good troubleshooting leaves a clearer system model and a defensible decision trail, even when the complete causal chain remains uncertain.

Common errors and fixes

The latest release is declared guilty immediately

Why it happens: Temporal sequence was treated as causal proof.
How to fix it: Compare cohorts and test a predicted mechanism.
Prevent it next time: Record alternatives and evidence that could falsify the claim.

A restart is written down as the root cause

Why it happens: A mitigation result was mistaken for an explanation.
How to fix it: Investigate which state reset and why it became harmful.
Prevent it next time: Separate restoration notes from causal analysis.

Many simultaneous tunings obscure the outcome

Why it happens: No bounded experiment or change log existed.
How to fix it: Stabilize, record state, and test one justified change at a time.
Prevent it next time: Define expected effects and stop conditions.

Check your understanding

1. Queue age rises while CPU remains low. Propose two hypotheses and a discriminating observation.

Show worked solution

Workers may be waiting for a constrained downstream resource, or work may not be assigned to available workers. Compare in-flight worker states, connection acquisition delay, and assignment progress using approved read-only evidence. Low CPU alone cannot distinguish those possibilities or justify adding replicas.

2. A rollback restores latency. What should the causal report say?