Deployment fails: distinguish crashes, readiness, and rollback safety

Understand the likely causes

• Common causes include invalid startup configuration, missing files, wrong identity, incompatible architecture, resource pressure, an incorrect probe, or unavailable dependencies. CrashLoopBackOff is restart backoff after failure, not a diagnosis. Readiness failure alone should not be confused with the reason a process exited or a Pod could not be scheduled.

Fix the observation context

Identify the cluster context, namespace, workload owner, intended image identity, configuration revision, and rollout time from approved records before inspecting anything. Use only authorized read access and redact sensitive logs. Separate controller progress, Pod scheduling and image events, container termination, and probe results. The first synthetic record shows a configuration exit before any readiness decision can matter. Preserve the first failure and previous termination evidence before replacing instances, since repeated restarts can obscure the useful message. No cluster or container is accessed by these examples.

workload: lab-bookings
image_id: lab-build-42
container_state: terminated
exit_code: 1
safe_log_message: Missing API_ENDPOINT
restart_display: CrashLoopBackOff
first_action: verify-required-setting-reference

Follow the failing branch

For pending work, inspect scheduling constraints and resource or storage availability. For image failures, inspect the exact image reference, architecture, and authorized registry access without printing credentials. For a crash, inspect exit information and safe startup logs; for a running but unready process, compare the probe with the documented application endpoint and dependency policy. Do not remove probes or raise all limits simply to change the displayed status. Each repair should address an observed violated contract, and a shared dependency outage should not trigger a fleet-wide liveness restart loop.

Choose a compatible recovery action

Pause expansion according to the release policy while impact is assessed. Compare the previous artifact with current schema, messages, configuration, and credential versions before considering rollback. The second review record blocks a return that cannot read the current data representation. If compatibility is unknown, preserve evidence and use the accountable recovery decision process rather than issuing a blind undo. A configuration correction or forward repair may be safer. Platform rollback can change workload intent but cannot reverse already accepted writes or remote side effects.

candidate_recovery: lab-build-41
current_schema: expanded-v3
previous_reader_compatibility: not-verified
credential_version: current-approved-reference
decision: block-blind-rollback
next_step: accountable-compatibility-review

Repair through the reviewed path

Change the smallest justified input in the approved source of truth and preserve the artifact and configuration association. If a GitOps reconciler owns the workload, an unmanaged manual edit can be reversed; use the documented scoped emergency exception when required. Separate startup allowance, traffic readiness, and restart liveness. Confirm timing from measured behavior rather than copying universal thresholds. Record the repair owner and decision, and keep the previous evidence available so post-incident review can explain why the chosen action reduced risk instead of merely suppressing a symptom.

Keep it from recurring

• Test startup, health semantics, termination, mixed-version state, and required settings before promotion. Preserve release identity and first-failure evidence. Rehearse a compatible recovery in an authorized isolated environment; a documented rollback command alone is not a recovery capability.