Kubernetes / Advanced: extensibility, resilience, and lifecycle / Lesson 18
GitOps ownership and incident diagnosis
Coordinate declarative automation and human response while preserving evidence, scoped authority, and application recovery.
On this page
Lesson goals
- Explain the model, assumptions, and boundaries of gitops ownership and incident diagnosis.
- Interpret both examples and distinguish declared intent from established evidence.
- Diagnose the listed failures and justify the worked recovery decisions without executing cluster operations.
Identify the authoritative control loops
A GitOps controller can reconcile versioned desired state while workload controllers, autoscalers, Operators, and admission systems manage other concerns. Define which system owns each field and resource. A manual edit may disappear because the declared source reasserts it, not because Kubernetes randomly changed configuration. Repository write access can indirectly exercise the reconciler’s authority, so review permissions and artifact references as security boundaries. Git history makes intent reviewable but does not prove a release is healthy or make external effects reversible. Controller and tool-specific APIs need separate version and support checks.
Plan safe exceptions before an incident
Emergency stabilization may need a scoped pause or reviewed desired-state change, according to the selected controller’s documented behavior. Name an accountable owner, expiry, evidence, and return-to-normal procedure. Disabling every reconciler or changing cluster-wide permissions is not a routine recovery step. The first synthetic record separates an emergency exception from the approved durable fix. A paused controller can allow drift to accumulate, so monitor the affected scope and preserve the reason. Do not restore revoked credentials or incompatible schema merely to make an old manifest match its historical state.
incident: lab-booking-7
scope: one-reviewed-workload
exception_owner: accountable-response-lead
exception_expiry: explicit-review-time
durable_fix: reviewed-desired-state-revision
closure: user-contract-and-reconciliation-verified
The fields describe governance, not a controller command or permission grant. No reconciliation is paused and no incident communication is sent.
Diagnose the user path in order
Start with impact, affected cohort, revision, namespace, and time. Classify API rejection, scheduling, image access, initialization, crash, readiness, Service selection, DNS, policy, transport, and application behavior as separate branches. Preserve UID and first-failure evidence, and distinguish observation from hypothesis. The second synthetic decision record says that DNS success with no ready endpoints points toward selection or workload readiness, not an immediate resolver repair. Avoid arbitrary exec, port forwarding, broad scans, or force deletion as a substitute for a supported explanation. Read-only operations still require authorized context and safe handling of sensitive data.
observation: service DNS resolves
observation_2: zero eligible ready endpoints
hypothesis: selector-or-readiness-contract
next_evidence: selected-Pod-labels-and-readiness
not_justified: change-DNS-or-disable-all-policy
The record narrows the next question without proving a cause. It avoids a broad network repair when endpoint eligibility already lacks evidence.
Recover the service rather than one status
Choose a bounded action under the incident authority and verify the intended user contract afterward. A restart can hide evidence without correcting configuration or data; a traffic rollback can leave accepted side effects behind. Check compatibility among image, configuration, credential, schema, and persistent state before returning to a previous revision. Reconcile queues and uncertain operations before full closure. Keep incident coordination, technical action, and user communication clear enough that multiple responders do not fight the same field. No message is sent and no recovery operation is performed by this lesson.
Turn incidents into better contracts
A post-incident review should identify missing evidence, conflicting ownership, unsafe defaults, and recovery friction rather than blaming the last person who touched a manifest. Assign a small set of concrete actions with owners, deadlines, and verification. Examples include a selector exclusion test, a credential-adoption check, or a field-ownership rule for HPA integration. Restore normal reconciliation and verify the declared revision matches actual useful behavior. The course ends with read-only reasoning and isolated simulations, not a cluster exercise. Reliable Kubernetes operation depends on accountable people and application contracts as much as on declarative controllers.
Common errors and fixes
Manual repair is immediately undone
- Why it happens
- An active reconciler still owns the desired field.
- How to fix it
- Use the approved scoped exception or source change.
- Prevent it next time
- Document ownership and emergency procedures.
Restart success is declared full recovery
- Why it happens
- User outcomes and state consequences were not checked.
- How to fix it
- Verify journeys, integrity, queues, and compatibility.
- Prevent it next time
- Define incident exit evidence before changing status.
Several responders fight the same field
- Why it happens
- Coordination and action ownership were unclear.
- How to fix it
- Assign a coordinator and record scoped actions.
- Prevent it next time
- Rehearse handoff and return-to-normal ownership.
Check your understanding
1. A reconciler reverts a manual replica edit while HPA also changes replicas. What needs review?
Show worked solution
Identify ownership among the source reconciler, autoscaler, and responder. Delegate the scale field according to the supported integration and use a scoped incident exception if needed. Repeating edits does not resolve the control-loop conflict.
2. A new revision is synchronized but bookings still fail. Can the incident close?
Show worked solution
No. Synchronization concerns the controller’s model, while service recovery requires correct user outcomes and state. Trace the remaining failure boundary and preserve the release association. Close only when the declared recovery criteria and residual-task ownership are satisfied.
Key takeaways
- A GitOps controller can reconcile versioned desired state while workload controllers, autoscalers, Operators, and admission systems manage other concerns.
- Start with impact, affected cohort, revision, namespace, and time.
- A post-incident review should identify missing evidence, conflicting ownership, unsafe defaults, and recovery friction rather than blaming the last person who touched a manifest.