Software Architecture / Evolution: capacity, migration, validation, and review / Lesson 13

Reason about performance before changing topology

Connect latency paths, concurrency, chatty I/O, saturation, and workload-scoped evidence.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Build a bounded latency or concurrency model and state its assumptions.
Choose an investigation that distinguishes a bottleneck from a symptom.

Trace the critical path rather than counting machines

A request can wait in admission queues, application code, database calls, provider calls, and response transfer. Sequential waits add to the critical path; parallel work contributes according to its dependencies and completion rule. Adding replicas will not remove a serialized external call or an inefficient per-row query loop. Map one representative operation and identify where elapsed time is measured. Distinguish CPU work from waiting and queueing. A high utilization chart can be relevant, but it does not identify which user path is slow or why.

Use small calculations to expose assumptions

The following model sums fictional sequential stages. It can help ask whether optimizing local rendering could materially improve a request dominated by database waiting. It does not predict tail latency, overlapping execution, retries, or contention. A real measurement should preserve stage boundaries and compare the same workload before and after a change. Never add percentiles from independent stage distributions and call the result the exact request percentile; the observations and their correlations matter. Simple models are useful because their omissions can be named and tested.

python A fictional sequential latency budget

stages_ms = {"admission": 20, "application": 10, "database": 120, "encoding": 5}
total = sum(stages_ms.values())
print(f"sequential_total_ms={total}")
print(f"without_encoding_ms={total - stages_ms["encoding"]}")

Expected output

sequential_total_ms=155
without_encoding_ms=150

The declared stages total 155 ms. Eliminating the five-millisecond encoding stage would leave 150 ms in this model, so it cannot address most of the elapsed time.

Reduce unnecessary round trips before adding capacity

A workshop list that fetches the organizer separately for every row can create many database or network calls. A bounded join, batched lookup, or suitable query projection may reduce round trips, but each changes payload size and consistency assumptions. Compare query plans and representative data in an isolated environment rather than declaring batching always faster. Avoid replacing many small calls with one unlimited export that exhausts memory. The desired shape is bounded work per request with pagination or another explicit result-size contract when the dataset can grow.

Connect throughput, time in system, and work in progress

Under stable conditions and consistent measurement boundaries, Little’s law relates average work in progress to average throughput multiplied by average time in the system. Units must agree. The example uses two completed requests per second and half a second of average system time, giving one request in progress on average. This is not a thread-pool sizing rule, nor does it hold as a steady-state forecast for an endlessly growing queue. Variability, safety margin, memory use, and downstream limits still require evidence.

python A unit-consistent steady-state calculation

throughput_per_second = 2
average_seconds_in_system = 0.5
average_in_progress = throughput_per_second * average_seconds_in_system
print(f"average_in_progress={average_in_progress:.1f}")
print("assumption: stable system and matching boundaries")

Expected output

average_in_progress=1.0
assumption: stable system and matching boundaries

The output is 1.0 average request in progress under the stated assumptions. It neither generates load nor selects operational concurrency settings.

Change the factor implicated by evidence

Form a hypothesis such as repeated organizer queries dominate list latency, then compare a bounded alternative using the same synthetic dataset and isolated test method. Record errors, correctness, memory, and tail behavior alongside throughput; faster wrong results are not an improvement. Stop increasing load when the safe lab scope is reached. Production load tests, database tuning, and infrastructure resizing are outside this lesson. Report only the small calculations actually executed and describe deployment experiments as proposed work requiring their own authorized environment.

Common errors and fixes

Replicas are added while the shared database remains saturated.

Why it happens: A symptom was treated as a capacity location.
How to fix it: Trace the request and identify the constrained shared resource.
Prevent it next time: Compare stage timing and saturation before topology changes.

Per-stage p95 values are summed as an exact request p95.

Why it happens: Distribution correlation was ignored.
How to fix it: Measure end-to-end observations or preserve their joint trace.
Prevent it next time: State whether a result is a model, bound, or observation.

A batched query returns unlimited data.

Why it happens: Round-trip reduction ignored memory and response bounds.
How to fix it: Add a meaningful result-size and pagination contract.
Prevent it next time: Test representative large lab inputs as well as small cases.

Check your understanding

1. A stage takes 2 ms out of a measured 500 ms request. What should a proposal to halve it acknowledge?

Show worked solution

Under the same sequential conditions it saves about 1 ms, leaving about 499 ms. The optimization may serve another goal, but it cannot explain a major latency improvement alone. Investigate the dominant stages and any overlapping behavior before prioritizing work. Preserve the original operation and measurement boundary in the comparison.

2. Why does an average work-in-progress value not specify the maximum safe worker count?