Kubernetes / Advanced: extensibility, resilience, and lifecycle / Lesson 16
Autoscaling and capacity limits
Relate workload metrics, replica decisions, scheduling, and dependency budgets without expecting autoscaling to create immediate unlimited capacity.
On this page
Lesson goals
- Explain the model, assumptions, and boundaries of autoscaling and capacity limits.
- Interpret both examples and distinguish declared intent from established evidence.
- Diagnose the listed failures and justify the worked recovery decisions without executing cluster operations.
Separate replica and node scaling
Horizontal workload scaling changes desired replicas; node scaling supplies or removes machines according to a separate implementation and policy. More desired replicas can remain Pending when nodes, storage, quotas, or placement rules cannot satisfy them. Vertical changes alter resource sizing and have their own disruption and compatibility implications. Identify the actual bottleneck before selecting a control loop. A database connection limit or upstream quota can make application scale-out harmful. Record minimum useful capacity, maximum permitted demand on dependencies, provisioning delay, and budget so scaling remains a bounded operational decision rather than a reaction to one high graph.
Understand metric availability and denominator
The autoscaling/v2 HPA can use supported resource, custom, or external metrics through the required APIs and adapters. CPU utilization targets are relative to resource requests, not an absolute percentage of the whole node. Missing requests or metrics affect calculation and need diagnosis. The first manifest targets an existing Deployment and assumes resource metrics and appropriate CPU requests. It supplies neither the metric pipeline nor the target workload. A stale or unavailable adapter should not be interpreted as evidence of zero demand. Protect the metric path and keep its units and aggregation aligned with the user workload.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: {name: booking-api, namespace: training}
spec:
scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: booking-api}
minReplicas: 2
maxReplicas: 6
metrics:
- type: Resource
resource:
name: cpu
target: {type: Utilization, averageUtilization: 60}
The target workload, requests, and metrics API must already exist. Sixty percent is a teaching value, not a measured recommendation, and this object is never applied.
Read the simplified proportional model carefully
A basic model proposes the ceiling of current replicas multiplied by observed metric divided by target. The second simulation computes that arithmetic only. Real HPA behavior includes tolerance, missing observations, readiness treatment, stabilization, and configured scaling policies. Therefore a mathematical result is not proof of the exact immediate action. Bounds also matter: minReplicas and maxReplicas constrain intent, while scheduling determines feasibility. Do not copy a target percentage as universal tuning; measure useful throughput, tail latency, resource pressure, and scaling delay for the chosen application before setting the control contract.
current_replicas = 2
observed_percent = 80
target_percent = 50
proposal = (current_replicas * observed_percent + target_percent - 1) // target_percent
print(proposal)
print("arithmetic only; real controller policy omitted")
4
arithmetic only; real controller policy omittedThe ceiling of 2 Γ 80 / 50 is four. This omits real HPA tolerance, observations, bounds, readiness, and stabilization and cannot predict an actual cluster action.
Coordinate ownership and oscillation controls
If a GitOps reconciler continuously resets replicas while an HPA changes them, the controllers can fight. Assign ownership of the scale field explicitly. Stabilization and rate policies reduce oscillation but can delay response, so analyze both burst and scale-down behavior. Scale-down must respect safe termination and application state, and reduced idle capacity may weaken failure or rollout headroom. A queue metric can reflect useful demand only when item size, deadlines, and abandoned work are understood. Retry storms may inflate apparent demand and should be contained rather than rewarded with unlimited downstream calls.
Validate useful capacity and cost
Evaluate autoscaling with authorized isolated experiments, synthetic demand, bounded scope, and stop conditions. This course performs none and contacts no metrics endpoint or cloud API. Check how long a new replica takes to become useful, how dependencies respond, and whether scale-down preserves accepted work. Include node, storage, telemetry, and provider costs rather than counting only Pods. Keep a reviewed fallback for unavailable metrics and a maximum that protects dependency and budget limits. Autoscaling is a feedback mechanism with delay and assumptions; it cannot replace efficient code, bounded admission, compatible state, or a recovery plan.
Common errors and fixes
CPU target is interpreted as node utilization
- Why it happens
- The resource-request denominator was ignored.
- How to fix it
- Inspect admitted requests and metric availability.
- Prevent it next time
- Document metric units and scope.
Desired replicas rise but ready capacity does not
- Why it happens
- Placement or startup cannot supply useful replicas.
- How to fix it
- Trace scheduling, storage, readiness, and dependency constraints.
- Prevent it next time
- Include provisioning delay and headroom in policy.
Replica count oscillates between tools
- Why it happens
- Scale-field ownership conflicts with reconciliation.
- How to fix it
- Assign one controller responsibility for the field.
- Prevent it next time
- Review GitOps and autoscaler integration explicitly.
Check your understanding
1. With three replicas, observed 90 and target 60, what does the simple model propose?
Show worked solution
The proportional value is 4.5, so the ceiling is five. Actual HPA output can differ because of missing metrics, tolerance, stabilization, policies, and bounds. State the assumptions before comparing the arithmetic with observed behavior.
2. Why can doubling API replicas worsen latency?
Show worked solution
The extra concurrency can overload a fixed database, queue, or external rate limit, while new replicas may also be warming. Measure the actual bottleneck and bound admission and dependency calls. Replica count is not synonymous with useful end-to-end capacity.
Key takeaways
- Horizontal workload scaling changes desired replicas; node scaling supplies or removes machines according to a separate implementation and policy.
- A basic model proposes the ceiling of current replicas multiplied by observed metric divided by target.
- Evaluate autoscaling with authorized isolated experiments, synthetic demand, bounded scope, and stop conditions.