Argo CD / Generated systems and controlled recovery / Lesson 16

Rollback, desired-state correction, and recovery

Choose a compatible recovery path across source, automation, configuration, and durable business state.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish a Git desired-state correction from a historical Argo CD rollback operation.
Evaluate data compatibility and partial side effects before choosing an older candidate.

Recover the user outcome under a known state contract

Recovery means restoring acceptable service while preserving required data and safety constraints. Returning to an earlier source revision is one possible action, not the definition of recovery. Parcel can fail because of a package, values change, external carrier, schema transition, or an incorrect health customization. Preserve desired and observed identities and the timeline before selecting a response. Repeated sync of the same incompatible intent may increase disruption rather than restore service. The incident owner must know which layer changed and which recovery material remains usable.

Desired-state correction and historical rollback differ

A reviewed source correction or revert changes the durable desired input and allows reconciliation under effective policy. An Argo CD historical rollback operation selects prior application deployment history under its supported behavior. The release-3.1 documentation states that rollback cannot be performed with automated sync enabled. That is a versioned operational constraint, not an instruction from this course to disable automation. For generated Applications, the parent ownership layer also matters. Choose a reviewed process that does not immediately fight the declared source or silently change authority.

Old manifests do not restore every side effect

A previous Deployment can be incompatible with a contracted schema or a changed message format. Hooks may have completed external actions that resource rollback cannot undo. Pruning and finalizers can affect retained data and resource identity. Distinguish source, generated manifests, workload package, environment values, secrets, and authoritative data recovery. Restoring the Argo CD control plane is another process and does not automatically restore application databases. Retain appropriate protected recovery evidence and rehearsal results without exposing credentials or claiming that a backup filename proves recoverability.

Keep a bounded recovery decision record

The JSON below is a synthetic tabletop record, not an Argo CD command, export, import, or live incident. It leaves compatibility pending and names user verification as a completion requirement. A real recovery decision needs authorized ownership, actual retained identities, state observations, and an explicit data-loss or compensation contract where relevant. Do not copy sensitive cluster or repository credentials into the record. Separate request acceptance from convergence so a lost client response does not create an unnecessary duplicate action.

json A synthetic recovery review record

{"incident":"parcel-lab-incident","desired":"lab-r8","observed":"lab-r8-partial","recovery_candidate":"lab-r7","state_compatibility":"pending-review","completion":"verify-label-journey"}

All labels are declared teaching data. No source is reverted, history selected, automation toggled, resource restored, or control plane exported or imported.

Reject retained but incompatible material

The complete local model checks that a synthetic older candidate is retained and supports the currently observed schema label. It rejects a newer schema outside its declared support set. This is not a database test or a proof of real migration reversal. Actual compatibility includes data semantics, queues, APIs, values, and completed external actions. An offline tabletop can reveal missing decisions; actual mechanics and timing require separately authorized suitable rehearsal. State exactly which evidence level was collected and close recovery only after the relevant observed identities and user outcome agree.

python A synthetic recovery-compatibility predicate

candidate = {"retained": True, "schemas": {6, 7}}
def compatible(observed_schema):
    return candidate["retained"] and observed_schema in candidate["schemas"]
print(compatible(7))
print(compatible(8))

Expected output

True
False

Only declared data is evaluated. No database, Application history, backup, cluster, or external business system is accessed.

Common errors and fixes

A live rollback is assumed durable while desired source remains unchanged.

Why it happens: Recovery action and reconciliation ownership were not aligned.
How to fix it: Review the desired-state correction path and effective automation with the owner.
Prevent it next time: Rehearse source, historical operation, and generated-child behavior separately.

A retained old candidate is assumed compatible with current data.

Why it happens: Availability was confused with state compatibility.
How to fix it: Evaluate schema, data, configuration, and external-effect contracts before recovery.
Prevent it next time: Maintain compatibility evidence throughout the recovery horizon.

A successful recovery command closes the incident without user verification.

Why it happens: Request completion replaced restored service evidence.
How to fix it: Verify observed identities, relevant health, and the affected user journey.
Prevent it next time: Define outcome-based recovery completion criteria.

Check your understanding

1. Why does an old source revision not automatically reverse a completed carrier purchase hook?

Show worked solution

The external business action has its own state and side effects. Resource or source reversal does not undo it. Preserve the original operation identity and use the owner’s reviewed reconciliation or compensation process alongside application recovery.

2. A tabletop exercise completes in four minutes. Can that be reported as production recovery time?