Kubernetes / Advanced: extensibility, resilience, and lifecycle / Lesson 14
Availability and disruption budgets
Combine replica placement, voluntary disruption controls, termination, and application recovery into a realistic availability contract.
On this page
Lesson goals
- Explain the model, assumptions, and boundaries of availability and disruption budgets.
- Interpret both examples and distinguish declared intent from established evidence.
- Diagnose the listed failures and justify the worked recovery decisions without executing cluster operations.
Identify the failure domain
Three replicas on one node do not provide the same resilience as replicas spread across independent failure domains. Node, zone, storage, network, and control-plane failures have different correlations. Define which failures the service should tolerate and how users observe degradation. A replica count alone does not prove quorum, safe failover, or sufficient surviving capacity. The first synthetic plan separates desired replicas, zone placement, and backend requirements. Keep dependencies in the model: spreading application Pods cannot protect a single database or a shared configuration system from its own failure.
replicas: 3
failure_domains: 3-labelled-zones
surviving_capacity: must-be-measured
backend: separately-reviewed-availability
accepted_work: drain-or-idempotent-retry
claim: no-zero-downtime-guarantee
The record names independent requirements instead of turning three replicas into a guarantee. It is not a Kubernetes manifest or measured capacity result.
Use disruption budgets for their real scope
A PodDisruptionBudget constrains supported voluntary evictions for selected Pods. It does not prevent every involuntary node failure, enforce a database quorum, or replace Deployment rollout strategy. The second policy/v1 example asks for at least two selected Pods to remain available under its accounting. An empty selector has important API-specific meaning, so use an explicit narrow match. A restrictive budget can block node maintenance when too few healthy replicas remain. Review the owning controller, health semantics, and actual available count before concluding the budget is faulty or bypassing it.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: booking-availability, namespace: training}
spec:
minAvailable: 2
selector:
matchLabels: {app: booking-api}
The budget applies to matching Pods and supported evictions. It does not stop node crashes, provide extra capacity, or override rollout and application compatibility requirements.
Design topology and capacity together
Topology spread and affinity rules can distribute replicas, but hard constraints need enough eligible capacity. A failed zone may leave too little room for replacement even when an autoscaler is configured. Storage topology, quotas, taints, and image constraints still apply. Balance desired separation with scheduling feasibility and the workload’s latency or data needs. Do not turn a critical anti-affinity rule into a preference without understanding why it exists. Conversely, a rule requiring more domains than the installation has can make all new replicas Pending and prevent an otherwise useful rollout.
Coordinate maintenance with accepted work
Graceful termination and readiness interact with eviction, endpoint changes, and long-lived connections. A permitted eviction can still interrupt a transaction if the application does not drain or resume safely. Plan checkpoints, retries, idempotency, and external reconciliation. Maintenance is also a capacity event: remaining replicas must handle traffic while a node or zone is unavailable. No drain, eviction, deletion, or maintenance command is executed here. In a real authorized operation, preserve the selected context and disruption evidence and respect the platform’s documented procedures rather than using force as a routine shortcut.
Verify service-level survival
An availability exercise should verify user outcomes, capacity, data integrity, and recovery time for the chosen failure domain, using synthetic work in an authorized isolated environment. Passing a PDB check alone is insufficient. Distinguish planned voluntary maintenance from surprise loss and document which controls cover each. Preserve recovery evidence and update assumptions after scale, topology, or dependency changes. Kubernetes can coordinate replacement, but the service owner must establish that surviving replicas can do useful correct work and that lost or uncertain operations are reconciled. Avoid universal claims of zero downtime based only on replica and budget numbers.
Common errors and fixes
A PDB is assumed to prevent node loss
- Why it happens
- Voluntary eviction control was confused with all failure protection.
- How to fix it
- Design placement, capacity, and application recovery separately.
- Prevent it next time
- Map controls to named failure domains.
Maintenance is blocked by unhealthy replicas
- Why it happens
- The healthy count no longer satisfies the budget.
- How to fix it
- Investigate availability and use the authorized maintenance decision.
- Prevent it next time
- Rehearse degraded-state maintenance and recovery.
Spread rules make replacement impossible
- Why it happens
- Required topology exceeds eligible remaining capacity.
- How to fix it
- Review constraints and measured failure headroom.
- Prevent it next time
- Validate topology with storage and scheduling together.
Check your understanding
1. Three healthy selected Pods have minAvailable two. Does that guarantee two survive a node crash?
Show worked solution
No. The budget governs supported voluntary disruptions, not involuntary loss. If all three share one failing node, all can disappear. Check placement, dependency resilience, and surviving capacity rather than treating a budget as physical redundancy.
2. Why can three-zone spreading still fail after one zone is lost?
Show worked solution
The remaining zones may lack schedulable capacity, required storage, or dependency availability. Replacement also takes time. Measure the intended failure scenario and preserve admission or degradation policy so surviving work remains useful while recovery proceeds.
Key takeaways
- Three replicas on one node do not provide the same resilience as replicas spread across independent failure domains.
- Topology spread and affinity rules can distribute replicas, but hard constraints need enough eligible capacity.
- An availability exercise should verify user outcomes, capacity, data integrity, and recovery time for the chosen failure domain, using synthetic work in an authorized isolated environment.