DevOps / Advanced: reliability, trust, and governance / Lesson 18
GitOps, platform boundaries, and governance
Design a reconciled delivery model with explicit authority, safe exceptions, and a platform that serves teams rather than hiding risk.
On this page
Lesson goals
- Explain desired-state reconciliation and authority.
- Handle drift and emergency exceptions safely.
- Define a usable platform with explicit governance boundaries.
Understand the reconciliation contract
GitOps applies a declared desired state that is versioned and retained, obtained automatically by agents, and continuously reconciled with the target. A pipeline that runs imperative commands from a Git repository does not necessarily satisfy that model. The value is a reviewable state history and a controlled reconciliation boundary, not a claim that Git can describe every business operation. Identify the source of truth, reconciler identity, managed scope, and convergence evidence. Different controllers and versions provide different behavior, so verify their documented semantics before relying on health, ordering, pruning, or suspension features.
Review repository and controller authority
A user who can change desired state may indirectly exercise the reconciler’s privileges. Protect review paths, identities, repository access, and artifact references accordingly. Avoid storing plaintext secrets in the desired-state repository. The first teaching record separates application-team intent from platform-owned guardrails and a narrowly scoped reconciler. A central platform should not require every team to understand every cluster detail, but abstraction must expose meaningful constraints and failure evidence. Multi-tenant boundaries need explicit authorization and policy; a folder naming convention alone does not isolate workloads or credentials.
desired_state_owner: application-team
policy_owner: platform-security-partnership
reconciler_scope: approved-namespace-and-resource-kinds
artifact_reference: immutable-reviewed-identity
emergency_exception: accountable-and-time-limited
completion: convergence-plus-user-contract-evidence
The fields describe authority and evidence. They are not an Argo CD, Flux, or Kubernetes configuration and grant no actual permissions.
Interpret drift before correcting it
Drift is a difference between actual and declared state, not automatically malicious behavior. It may reflect an authorized emergency, controller defaulting, an unmanaged field, or an unexplained mutation. The second in-memory simulation compares a tiny declared and observed record and reports changed keys without contacting a target. Real controllers need documented ownership and comparison rules. Automatic correction can undo an emergency mitigation if no exception procedure exists. Define how to pause or narrow reconciliation through the approved controller policy, preserve evidence, and return to reviewed desired state when the incident is stabilized.
desired = {"revision": "lab-build-42", "replicas": 3}
observed = {"revision": "lab-build-41", "replicas": 4}
changed = sorted(key for key in desired if desired[key] != observed.get(key))
print(",".join(changed))
print("review before reconciliation")
replicas,revision
review before reconciliationThe comparison reports replicas and revision. It diagnoses a difference only; it does not authorize a repair or change any workload.
Make the platform a usable product
A platform offers supported paths for common needs: build identity, deployment configuration, telemetry, secrets access, and recovery. Learn from teams’ actual work and measure adoption, feedback time, reliability, and support burden. A template is useful when teams understand its assumptions and can evolve beyond it through a reviewed extension path. Avoid mandatory complexity that merely shifts operational work into application teams. Governance should provide policy and evidence at the right boundary, with documented exceptions and ownership, rather than a ticket queue for every harmless change. Retire unsupported templates and keep compatibility promises clear.
Verify convergence and retain human accountability
A reconciler reporting synchronization may establish only that declared objects match its model, not that users can complete their work. Check readiness, rollout policy, data compatibility, and user outcomes separately. Record the exact desired-state revision and running artifact identities for recovery. Reverting a Git commit changes intent but cannot undo deleted data or every external effect. Keep approved emergency access and incident communication available when the Git or reconciliation service is unavailable. GitOps is one implementation of controlled feedback, while accountability for risk, security, and useful service remains with people and their agreed operating model.
Common errors and fixes
Git review indirectly grants excessive access
- Why it happens
- Repository permissions ignored reconciler authority.
- How to fix it
- Align review and controller scope with least privilege.
- Prevent it next time
- Threat-model desired-state changes and artifact references.
Automation reverses emergency mitigation
- Why it happens
- Reconciliation had no approved exception process.
- How to fix it
- Use the documented scoped pause and reconcile the intended fix.
- Prevent it next time
- Rehearse emergency ownership and return-to-normal steps.
Synchronized objects are called healthy service
- Why it happens
- Controller convergence replaced user outcome checks.
- How to fix it
- Verify readiness, data, and the actual journey separately.
- Prevent it next time
- Define completion beyond the controller status.
Check your understanding
1. A responder manually changes replicas during an incident. Why might the change vanish?
Show worked solution
A reconciler can restore the declared count because it owns that field. Use the approved emergency exception or reviewed desired-state update, preserve the reason and owner, and restore normal reconciliation afterward. Repeated manual edits create a contest rather than a reliable recovery plan.
2. A platform template hides storage deletion in an innocuous setting. What should improve?
Show worked solution
Expose the destructive consequence in the contract and review evidence, validate dangerous combinations, and require the appropriate recovery and approval policy. An abstraction should reduce cognitive burden while preserving meaningful risk visibility, not conceal irreversible effects behind a convenient interface.
Key takeaways
- GitOps applies a declared desired state that is versioned and retained, obtained automatically by agents, and continuously reconciled with the target.
- Drift is a difference between actual and declared state, not automatically malicious behavior.
- A reconciler reporting synchronization may establish only that declared objects match its model, not that users can complete their work.