Software QA / Advanced: strategy, resilience, and leadership / Lesson 15

Performance experiments and failure recovery

Design bounded performance experiments with realistic workload assumptions and correctness checks. Use synthetic measurements to reason about latency percentiles, saturation, and recovery without sending load to live systems.

About 4 min readContent reviewed 2026-10-23

On this page

Lesson goals

Define workload, measurements, and acceptance conditions.
Interpret percentiles and errors together.
Check correctness during degraded and recovering behavior.

A performance claim needs conditions

“The portal supports 1,000 users” is incomplete: concurrent sessions, arrival rate, requests per journey, data size, think time, and hardware all affect meaning. Define a representative workload with read/write proportions and realistic datasets, then agree latency and error targets for specific operations. Distinguish load testing at expected demand, stress testing beyond expected capacity, and endurance testing over sustained time. This course uses bounded synthetic examples only. In practice, obtain scope and resource limits for an isolated environment before any load experiment, and stop if the experiment harms shared dependencies.

Worked case: a hidden slow tail

Consider 20 synthetic latencies: nineteen requests take 100 milliseconds and one takes 2,100 milliseconds. The mean is 200 milliseconds, while the maximum is 2,100. Under the nearest-rank convention, p95 is the nineteenth sorted value, 100 milliseconds; p99 is the twentieth, 2,100. State the percentile convention and sample size. This tiny dataset illustrates why a mean can hide a slow request and why sparse tail estimates are unstable. It does not establish a production percentile or show that the slow result is acceptable.

python Synthetic percentile calculation; no network traffic

from math import ceil
values = sorted([100] * 19 + [2100])
def nearest_rank(percent):
    return values[ceil(percent / 100 * len(values)) - 1]
assert sum(values) / len(values) == 200
assert nearest_rank(95) == 100
assert nearest_rank(99) == 2100

With only 20 observations, p99 is simply the maximum under this convention. Larger representative samples are needed for meaningful tail estimates.

Check the experiment before explaining results

Record configuration, dataset, cache state, warm-up policy, duration, and generator limits. A saturated load generator can make the service appear faster by failing to send intended work. Closed-loop clients slow their own arrivals when responses slow, which can conceal the demand pattern a fixed arrival-rate workload would expose. Inspect achieved arrival rate, throughput, errors, timeouts, and resource signals together. Excluding failed requests from latency reports without reporting failure frequency creates a deceptively healthy view. Keep functional correctness assertions active so a fast incorrect answer is not counted as a successful optimization.

Worked case: timeout with a completed write

In a local controlled scenario, a reservation write commits but the dependency response times out. The client retries. Expect at most one reservation, a consistent seat count, and an understandable status after recovery. Measure time to a stable usable outcome as well as first response latency. Separately simulate a dependency unavailable before any write and expect a safe rejection or queued state according to the contract. These cases distinguish resilience of user outcomes from raw availability of a page, and they require explicit state inspection after the fault is removed.

Compare results without overclaiming

Compare builds under equivalent conditions and inspect variability before attributing a small difference to the change. A threshold breach is evidence to investigate, not automatically proof of a particular bottleneck. Correlate slow operations with relevant resource or trace signals, then run a focused follow-up experiment that can distinguish hypotheses. Record limitations such as smaller datasets or simulated providers. W3C navigation timing offers browser timing concepts, but server response, rendered usability, and completed business operation are different endpoints. Select the measurement that matches the user promise and label draft specifications accurately.

Common errors and fixes

Only average latency is reported

Why it happens: Tail experience and errors are hidden.
How to fix it: Report sample size, percentiles, failures, and workload together.
Prevent it next time: Define measurement endpoints and conventions before execution.

Fast error responses count as performance success

Why it happens: Correctness is excluded from the success definition.
How to fix it: Require valid business outcomes and report errors separately.
Prevent it next time: Include functional assertions in performance scenarios.

Recovery is judged only by a responding homepage

Why it happens: Persistent side effects and user completion are ignored.
How to fix it: Inspect recovered state and complete the affected task.
Prevent it next time: Define recovery oracles for each critical operation.

Check your understanding

1. A service has lower mean latency but twice as many timeouts. Is it improved?

Show worked solution

Not enough evidence supports that claim. Check whether timed-out requests were omitted from latency, compare the same achieved workload, inspect tail values, and evaluate successful task completion. A lower average among survivors can coexist with worse user experience.

2. What should a reservation recovery case observe after a retry?