Linux performance / Basic: measurement and resource signals / Lesson 2

Interpret latency, throughput, and queues

Use distributions and queueing relationships to distinguish faster individual work from a system that sustains useful demand.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Separate service time, waiting time, and client latency.
Interpret percentiles and aggregate populations correctly.
Apply queueing arithmetic only under stated assumptions.

A latency boundary is part of the metric

Client latency can include name lookup, connection establishment, proxy waiting, application work, and response transfer. Service time covers only a chosen processing stage; queueing time is the wait before that stage begins. A server histogram that starts after admission can miss the slowest part of the client experience. Define start and end events and say whether failures and timeouts are included. For a multi-stage report, keep both the total and stage timings so a faster CPU stage cannot hide a growing database or admission queue.

Averages do not describe every user

A mean summarizes total duration divided by count and is useful for some accounting questions. A percentile identifies a position in an ordered population and exposes a different property. A small fraction of very slow requests can violate a tail objective while the mean looks stable. Report sample count and observation window with each percentile; a tiny sample cannot support a strong claim about rare events. Do not average percentile values across hosts or time buckets to obtain a global percentile. Combine compatible underlying observations or merge appropriate histogram counts instead.

bash Compute a mean from a declared small population

printf '%s\n' 10 10 10 10 60 | awk '{sum+=$1;n++} END {printf "synthetic_mean=%.1f ms\n",sum/n}'

Expected output

synthetic_mean=20.0 ms

The synthetic mean is 20 ms, although one of five requests takes 60 ms. This example demonstrates why the mean alone does not represent the slowest request.

Offered demand differs from completed work

Offered load is the work clients attempt to submit. Throughput is completed work per interval, and successful throughput should exclude rejected or failed requests according to the contract. Raising concurrency can increase throughput until a resource or serialization point becomes limiting; beyond that point it may mostly increase waiting. Record in-flight work and queue depth alongside completion rate. An asynchronous acknowledgement is not necessarily durable completion. If the user needs a generated report, counting accepted jobs as finished hides the backlog and misstates the useful result.

Use Little’s law with matching populations

For a stable system and a consistently defined population, average in-system work equals arrival or departure rate multiplied by average time in the system. Keep units consistent and include the same waiting boundary in all terms. The relationship is an accounting check, not a promise that adding workers reduces latency. A rapidly growing queue violates the simple steady-state comparison, and dropped work changes the relevant population. Use the equation to detect mismatched metrics or estimate average concurrency under an explicitly hypothetical stable workload before designing a lab experiment.

bash Check stable-workload concurrency arithmetic

awk 'BEGIN { rate=40; latency_seconds=0.25; printf "synthetic_inflight=%.1f\n",rate*latency_seconds }'

Expected output

synthetic_inflight=10.0

At a hypothetical 40 completions per second and 0.25 seconds average total latency, average in-flight work is 10. This is a stated model, not measured host capacity.

Preserve stalls and censored observations

A closed-loop client waits for a reply before sending more work, so its offered rate falls during stalls. A histogram collected only from completions can then underrepresent the experience of demand that would have arrived independently. Timeouts also censor exact completion duration; dropping them from a chart makes the chart look healthier. Record the generation model, attempted arrivals, queueing policy, timeout count, and timing boundary. Interpret percentiles together with errors and backlog rather than treating a smooth chart as proof that the service met its objective.

Common errors and fixes

Averaging host percentiles

Why it happens: Different distributions were collapsed into incompatible summaries.
How to fix it: Merge compatible histogram counts or raw observations.
Prevent it next time: Keep aggregation metadata with every latency series.

Counting accepted jobs as completed

Why it happens: The completion boundary stops before useful output exists.
How to fix it: Measure final successful work and queue age.
Prevent it next time: Define completion from the user’s requirement.

Ignoring timeouts in the result

Why it happens: Only successful replies entered the latency population.
How to fix it: Report failures and censored durations alongside latency.
Prevent it next time: Specify outcome inclusion before collecting data.

Check your understanding

1. A stable service completes 20 jobs/s with mean total residence time 0.5 s. Estimate average in-flight work.

Show worked solution

Average in-flight work is 10 jobs if both measurements describe the same stable population and boundary. This does not imply there are ten busy CPU threads: in-flight jobs can be waiting on queues, storage, or dependencies.

2. A chart’s p99 improves while timeout count doubles. What conclusion is justified?