Linux performance / Advanced: topology, observability, and controlled tuning / Lesson 17

Design capacity tests without misleading results

Specify demand generation, workload realism, safety bounds, and acceptance criteria before any isolated load experiment.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Distinguish open-loop and closed-loop demand models.
Define warm-up, run duration, replication, and failure handling.
Estimate headroom from measured curves rather than a universal threshold.

Capacity is conditional on the service objective

Capacity means sustainable useful demand while the stated latency, errors, correctness, and recovery conditions remain acceptable. A maximum observed request rate without those conditions is not a production capacity promise. Define the request mix, input distribution, data size, cache state, connection reuse, concurrency model, and dependency behavior. Distinguish a single-instance test from a distributed service whose coordination changes as it grows. Keep the test environment isolated and disposable. This lesson prepares plans and synthetic arithmetic only; it never runs a load generator or sends traffic to any host.

Demand generation affects the result

A closed-loop client waits for completion before issuing more work, so offered rate can fall when the service slows. An open-loop model attempts arrivals according to an external schedule and can expose queue growth under independent demand. Neither is universally realistic; choose the one that matches the question and state the consequences. Record attempted arrivals, actual submissions, completions, rejections, and timeouts separately. The generator can itself become CPU, connection, or bandwidth limited, so a flat measured rate may describe the test client rather than the target’s capacity.

Write stop conditions before starting

A controlled plan names authorized endpoints, maximum duration, maximum offered demand, resource and error stop conditions, observers, data cleanup, and rollback. Verify that downstream services are also isolated or explicitly approved. A synthetic account can still trigger email, billing, or durable writes if dependencies are real. Prefer a representative data copy and safe sinks. Changes to security, durability, or validation must not be hidden inside the test. Preserve the plan and versioned configuration with results so another engineer can distinguish a real capacity finding from a different experiment.

text Read a non-executable load-test contract

target: isolated synthetic report service
arrivals: declared schedule, bounded by approved ceiling
measure: client latency, successful completions, errors, queue age
stop: deadline reached OR approved error/pressure boundary crossed
cleanup: preserve evidence, restore fixture data, verify baseline

This text intentionally contains no traffic-generation command or universal resource threshold. The actual ceiling and stop criteria must be chosen for the authorized lab and service objective.

Observe the curve, not one successful point

A useful capacity curve compares several controlled demand levels and repeated windows. Throughput can flatten while latency and queue age continue rising, indicating that extra offered work mostly creates waiting or errors. Identify the region that meets the objective with enough reserve for bursts, dependency variance, and recovery. A short peak can consume cache or buffer reserves without being sustainable. Include steady behavior and cleanup between trials, and avoid selecting only the best repetition. Confidence depends on workload realism, sample size, environmental stability, and the generator’s own health.

bash Check completed work against offered demand

awk 'BEGIN { completed=5400; seconds=60; offered=100; rate=completed/seconds; printf "synthetic_success=%.1f/s completion_fraction=%.2f\n",rate,rate/offered }'

Expected output

synthetic_success=90.0/s completion_fraction=0.90

The declared synthetic minute completes 90 jobs/s from 100 offered jobs/s. The missing fraction needs outcome and queue accounting; this is not a benchmark result or proof that ten percent were rejected.

Forecast with explicit uncertainty

Projecting capacity requires a demand model, growth assumptions, and knowledge of shared bottlenecks. Doubling instances need not double throughput if they share storage, database locks, or an external quota. Repeated tests can establish conditional evidence, but production traffic may include rare expensive inputs absent from the fixture. Record limitations and plan ongoing verification after any later authorized rollout. Keep a conservative recovery reserve based on measured behavior rather than a fashionable utilization percentage. The final recommendation should state what was tested, what objective passed, and which untested conditions could invalidate it.

Common errors and fixes

Reporting only peak requests per second

Why it happens: Latency, errors, and sustainability were omitted.
How to fix it: Use objective-passing intervals and full outcomes.
Prevent it next time: Define acceptance before generating demand.

The load generator becomes the bottleneck

Why it happens: Client health and submission rate were not observed.
How to fix it: Inspect generator limits and actual arrivals.
Prevent it next time: Monitor both sides of the experiment.

Assuming linear scale-out

Why it happens: Shared dependencies and coordination were ignored.
How to fix it: Test multiple instance counts with shared-resource evidence.
Prevent it next time: State scaling assumptions and uncertainty.

Check your understanding

1. Offered rate rises, throughput stays flat, and queue age grows. What does the extra demand accomplish?

Show worked solution

It mainly adds unfinished work or failures at the observed boundary. Inspect outcomes and the limiting stage, and use the objective-passing region rather than the largest offered rate as the capacity recommendation.

2. Why can a 30-second clean run be insufficient for a service with periodic writeback?