Kubernetes / Intermediate: rollout, policy, and evidence / Lesson 8
Probes, startup, and graceful termination
Assign different meanings to startup, readiness, and liveness, then coordinate shutdown with routing and accepted work.
On this page
Lesson goals
- Explain the model, assumptions, and boundaries of probes, startup, and graceful termination.
- Interpret both examples and distinguish declared intent from established evidence.
- Diagnose the listed failures and justify the worked recovery decisions without executing cluster operations.
A health check is an operational decision
A probe should correspond to a specific action the platform can take. Readiness decides whether the Pod should receive new Service traffic; liveness detects a condition where restarting a container is useful; startup can protect legitimate initialization. A single endpoint that checks every remote dependency often mixes those purposes. A temporary database outage may require admission control rather than restarting every process. Define the application’s internal health contract and failure modes before choosing paths or timing values. A probe response is not a complete business transaction and cannot establish data integrity by itself.
Gate initialization without hiding failure
A configured startup probe must succeed before liveness and readiness operate according to documented behavior. Give initialization a measured bounded allowance, not an unlimited exemption. Image pull and scheduling delays are different from a process that started and is initializing. The first container fragment uses separate endpoints and an illustrative startup allowance. It is not a full Pod or a universal tuning prescription. If initialization never completes, preserve its reason and fix the prerequisite rather than repeatedly increasing thresholds. Version and application-specific probe behavior must be checked before adapting the fragment.
startupProbe:
httpGet: {path: /startup, port: http}
periodSeconds: 5
failureThreshold: 24
readinessProbe:
httpGet: {path: /ready, port: http}
periodSeconds: 5
livenessProbe:
httpGet: {path: /live, port: http}
periodSeconds: 10
The surrounding container must define the http port and endpoints. Approximately 120 seconds of startup allowance is illustrative and timing-dependent, not a tested safe production value.
Interpret failure evidence precisely
An HTTP probe can fail because the process is unready, the path is wrong, a named port is unresolved, or a timeout is too short under measured conditions. Exec probes introduce command and resource overhead; TCP checks establish only limited transport evidence. Distinguish the probe’s observation from a container exit reason. The second synthetic matrix separates a running unready process from repeated termination. CrashLoopBackOff is restart backoff and not proof that a liveness setting is wrong. Record probe type, endpoint, timing, container state, and recent changes while avoiding credential-bearing logs.
container running + readiness false -> inspect admission contract
container exited before probe -> inspect startup exit evidence
shared database unavailable -> avoid automatic fleet restart
termination requested -> drain accepted work within grace
Different observations imply different investigations. The rows are independent lab scenarios and do not authorize a restart or override a real recovery policy.
Coordinate termination with request ownership
Deletion or replacement initiates termination with a grace period. Lifecycle hooks, signal handling, endpoint updates, and connection draining need an application-aware contract. A preStop hook consumes part of the available grace period; it is not extra unlimited time. The process should stop admission, finish or cancel accepted work safely, release resources, and exit before forced termination. Existing connections and external load balancers may have their own delays. Do not assume removing readiness instantly erases every in-flight request. Track long-lived connections and jobs separately, and use idempotency when retry can follow interruption.
Test the action, not only the endpoint
In an authorized isolated environment, verify slow startup, dependency failure, readiness removal, graceful termination, and forced-deadline behavior with synthetic work. Confirm that liveness restarts only the intended failure and does not amplify a shared outage. Measure startup and request durations before setting thresholds. Keep probe and termination changes in review because they alter availability and recovery. No probe, command, process signal, cluster request, or network operation is performed here. Static YAML validation confirms only parseable structure; it cannot show that an endpoint exists or that accepted work survives termination.
Common errors and fixes
Dependency failure restarts every replica
- Why it happens
- Liveness depends on shared remote availability.
- How to fix it
- Separate restart usefulness from admission readiness.
- Prevent it next time
- Rehearse dependency outages with no restart storm.
A hook exceeds shutdown time
- Why it happens
- preStop was treated as extra grace time.
- How to fix it
- Budget hook, drain, and cleanup inside the total window.
- Prevent it next time
- Measure termination with accepted work.
Threshold increases hide invalid startup
- Why it happens
- A prerequisite failure was mistaken for slowness.
- How to fix it
- Inspect the first startup error and fix the input.
- Prevent it next time
- Distinguish waiting, initialization, and permanent failure.
Check your understanding
1. Should a remote database timeout always fail liveness?
Show worked solution
No. Ask whether restarting the process can improve the condition. A shared dependency outage can make mass restarts harmful. Use bounded readiness or degradation policy as appropriate, and keep a separate liveness signal for restart-worthy local failure.
2. A total termination window is thirty seconds and preStop uses ten. What follows?
Show worked solution
The hook consumed part of the total budget, leaving roughly twenty seconds for later signal handling and shutdown, subject to documented timing details. Design a smaller bounded drain and cleanup rather than assuming another full thirty seconds follows the hook.
Key takeaways
- A probe should correspond to a specific action the platform can take.
- An HTTP probe can fail because the process is unready, the path is wrong, a named port is unresolved, or a timeout is too short under measured conditions.
- In an authorized isolated environment, verify slow startup, dependency failure, readiness removal, graceful termination, and forced-deadline behavior with synthetic work.