SRE / Intermediate: response and operational evidence / Lesson 12

Capacity, demand, and queue behavior

Reason about concurrency, utilization, queue age, and failure headroom using explicit assumptions rather than selecting a universal CPU target.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Relate throughput, time, and in-flight work.
Distinguish capacity from demand forecasts.
Plan for bursts and reduced-capacity states.

Define the work unit and limiting resource

Capacity is the amount of useful work a system can sustain while meeting its required behavior. Requests per second alone is incomplete without request shape, latency, correctness, and resource limits. A catalogue lookup and a reservation transaction can consume different resources. Identify the limiting resource under the relevant workload: CPU, memory, worker slots, connection pool, storage throughput, or a downstream quota. Do not assume the resource with the most visible chart is the bottleneck. A capacity statement should name the workload, environment, evidence source, and conditions under which it ceases to apply.

Relate work in progress to time

Under stable flow and matching boundaries, Little’s Law relates average in-flight work to arrival rate multiplied by average time in the system. At one hundred requests per second and two hundred milliseconds average residence time, the model predicts twenty in-flight requests. If average time rises to one second at the same arrival rate, the average rises to one hundred. This arithmetic does not prove stability, and it is not a queueing simulator. Use consistent units and populations. When arrivals exceed completions for a sustained interval, a growing backlog violates the steady-flow assumption and needs a separate transient analysis.

python Offline steady-flow concurrency model

arrival_per_second = 100
for residence_seconds in (0.2, 1.0):
    average_in_flight = arrival_per_second * residence_seconds
    print(f"{average_in_flight:.0f}")

Expected output

20
100

The results are average in-flight work under explicit stable-flow assumptions. No requests are sent and no service capacity is measured.

Inspect age as well as length

A queue can absorb a short burst, but it cannot create processing capacity. Waiting consumes the user’s deadline and may keep memory or other resources occupied. Queue length alone is ambiguous because work items can differ in cost; oldest age and completion rate add useful context. Bound queue size and consider whether expired work should still execute. An unbounded queue can turn a temporary slowdown into memory exhaustion and prolonged recovery. For asynchronous work, define freshness objectives and backlog clearance expectations separately from the acknowledgement latency of accepting a job.

Plan for the state after a failure

Headroom must cover the failure state you intend to survive, not merely normal operation. A fictional pool of four equal workers loses twenty-five percent of its raw processing slots when one worker is unavailable. If healthy demand already consumes nearly all four, moving traffic to the remaining three can overload them. Real workers may not be equal and shared dependencies can limit all of them. The design record lists the evidence needed before asserting failure tolerance. Redundancy without spare effective capacity may improve an architecture diagram while doing little to protect users during a failure.

yaml Illustrative capacity review

capacity_review: reservation-workers
workload: agreed-request-mix
failure_case: one-worker-unavailable
evidence_required:
  - remaining-workers-meet-user-deadline
  - downstream-quota-remains-sufficient
  - backlog-clears-within-freshness-objective
forecast: demand-range-with-explicit-uncertainty

The record asks for evidence instead of claiming a tested capacity. This course runs no load test and provisions no workers.

Use forecasts as conditional inputs

Forecast demand as a range with assumptions about growth, seasonality, request mix, and launch events. Compare lead time for adding effective capacity with the time remaining before a constraint is likely to matter. Autoscaling has detection and startup delay, so it cannot be assumed to handle every sudden burst. Define admission and degradation behavior for demand beyond the safe envelope. Review observed demand and bottlenecks after significant software changes. Capacity planning is a recurring risk discussion that connects workload evidence, user objectives, failure scenarios, and operating cost; it is not a one-time machine-count calculation.

Common errors and fixes

A long queue is treated as extra capacity

Why it happens: Buffering was confused with processing.
How to fix it: Inspect completion rate, age, and deadline waste.
Prevent it next time: Bound waiting and define overload behavior.

Normal operation passes but failover overloads survivors

Why it happens: No headroom was reserved for the failure state.
How to fix it: Review useful capacity after the intended loss.
Prevent it next time: Include reduced-capacity scenarios in design evidence.

A forecast is presented as a measured limit

Why it happens: Demand prediction and capacity evidence were conflated.
How to fix it: Label assumptions and obtain appropriate workload evidence.
Prevent it next time: Version workload mix and uncertainty.

Check your understanding

1. A stable service completes 50 operations per second with 0.4 seconds average residence time. Estimate in-flight work.

Show worked solution

The average is 50 × 0.4 = 20 operations under matching-boundary steady-flow assumptions. This is not a maximum concurrency setting. Variance, bursts, resource limits, and tail latency still matter, and a continuously growing backlog means the stable model does not apply.

2. Why might doubling application replicas fail to double throughput?