DevOps / Advanced: reliability, trust, and governance / Lesson 15
Capacity, overload, and cost
Relate demand to resource budgets, protect the service under overload, and evaluate cost against useful outcomes.
On this page
Lesson goals
- Relate demand, concurrency, queueing, and resource budgets.
- Explain dependency-aware scaling and overload protection.
- Evaluate cost per useful outcome within reliability constraints.
Define demand and the useful unit
Capacity planning begins with workload shape, not a guessed instance count. Identify request mix, concurrency, data size, burst behavior, dependency limits, and useful completion criteria. Average CPU alone cannot describe tail latency, memory retention, queue delay, or a saturated connection pool. The first simulation applies the steady-state relationship between arrival rate, average time in the measured system, and average work in that system. It is arithmetic over a stable hypothetical workload, not a load test or proof that twenty-four workers are sufficient. Boundaries and units must match for the relationship to be meaningful.
arrival_per_second = 120
average_seconds_in_system = 0.2
print(arrival_per_second * average_seconds_in_system)
24.0The hypothetical average work in the measured system is 24.0. This does not determine a safe queue, worker count, tail latency, or burst capacity.
Bound admission and queueing
A queue absorbs temporary mismatch but does not create capacity. If arrival exceeds sustainable completion for long enough, waiting time and retained work grow. Set limits on active operations, queued items, item size, and total deadline. Choose explicit overload behavior such as rejecting low-priority work or degrading an optional feature. Retries can amplify overload, so use budgets, backoff, jitter, and idempotency where appropriate. A failed dependency should not cause every instance to launch more parallel calls in an attempt to recover throughput. Protect critical journeys before optimizing an overall average.
Scale the bottleneck, not the symptom
More application replicas can worsen a database or rate-limited upstream bottleneck. Autoscaling needs a signal related to demand, a delay model, minimum safe capacity, and an upper bound consistent with dependencies and budget. Resource requests influence placement; limits and platform quotas affect runtime behavior. Keep headroom for bursts, failures, and rollout overlap, and include the time required to obtain capacity. Representative capacity experiments belong in an authorized isolated environment with bounded scope and stop conditions. This course performs none. Use existing approved measurements rather than manufacturing load against shared services.
Connect cost to useful work
Total spending includes idle capacity, redundancy, storage, data transfer, telemetry, build infrastructure, and operational effort. The second synthetic record uses fictional cost units to separate a useful unit cost from a raw machine count. Cost per successful operation can rise when retries and errors increase even if infrastructure spending is flat. Allocation needs a consistent scope and time window; shared platform costs require an explicit policy. Do not infer current cloud prices from this record. Compare published provider pricing and actual billing only within an authorized financial review, and account for reliability obligations before removing redundancy.
window: synthetic-one-day
cost_units: 2000
successful_bookings: 10000
cost_units_per_success: 0.2
included: compute-storage-telemetry-and-shared-allocation
pricing_source: fictional-teaching-data
2,000 divided by 10,000 is 0.2 fictional units per success. The declared scope is needed for comparison, and these are not provider prices or a real bill.
Make efficiency a controlled decision
Rightsizing, cache tuning, retention changes, and scheduled nonproduction capacity can reduce waste, but each changes a contract. Shorter backup retention may weaken recovery; aggressive scale-down may miss a burst; cheaper storage may delay restoration. State the expected benefit, affected SLO and security controls, observation window, and reversal condition. Evaluate several comparable periods rather than one quiet sample. The goal is sustainable value under reliability and risk constraints, not the smallest bill in isolation. Document the decision so another team can distinguish intentional headroom from forgotten resources and avoid undoing an important safeguard.
Common errors and fixes
Adding replicas worsens errors
- Why it happens
- An unchanged dependency limit became more overloaded.
- How to fix it
- Bound calls and identify the actual limiting resource.
- Prevent it next time
- Include dependency capacity in scaling policy.
A queue grows until memory is exhausted
- Why it happens
- Waiting was unbounded and callers already expired.
- How to fix it
- Apply admission, queue, size, and deadline limits.
- Prevent it next time
- Monitor queue age and reject obsolete work.
Cost savings weaken recovery
- Why it happens
- Retention or redundancy was removed without risk review.
- How to fix it
- Restore the required control and reassess unit economics.
- Prevent it next time
- Link efficiency proposals to SLO and recovery constraints.
Check your understanding
1. A stable system completes 50 operations per second with average time 0.4 seconds inside its boundary. Compute average work present.
Show worked solution
The average is 50 Γ 0.4 = 20 operations within the same measured boundary. It is not a maximum and does not describe burst or tail behavior. Confirm stable flow and consistent units before using it to interpret capacity measurements.
2. Spending stays constant while successful operations halve. What happens to cost per success?
Show worked solution
It doubles under the same accounting scope and window. Investigate errors, retries, and demand changes rather than concluding the platform is efficient because the bill is unchanged. Include user value and reliability when selecting the repair.
Key takeaways
- Capacity planning begins with workload shape, not a guessed instance count.
- More application replicas can worsen a database or rate-limited upstream bottleneck.
- Rightsizing, cache tuning, retention changes, and scheduled nonproduction capacity can reduce waste, but each changes a contract.