Ansible / Advanced: orchestration, platforms, tests, and diagnosis / Lesson 13

Define failure, retries, and recovery honestly

Use failed_when, changed_when, block/rescue/always, and bounded retries without treating ignored errors or rescue as automatic rollback.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Classify task failure, unreachable state, and invalid input separately.
Explain block rescue boundaries and truthful aggregate outcomes.
Retry only a justified transient condition with bounded effects.

A failed task is not every possible execution error

Ansible distinguishes task failure, unreachable targets, skipped work, and errors in parsing or evaluating the automation. A rescue block handles the documented failure path of its block tasks, not every unreachable or malformed definition. ignore_errors permits continuation for applicable task failures; it does not make a failed operation correct or guarantee that later prerequisites exist. Preserve the original error category, target identity, task, and relevant result fields. A clear failure classification is more useful than adding broad ignore settings until a run finishes green. Keep sensitive fields redacted while retaining enough context to identify the actual operation.

Failure and change predicates must preserve the real contract

failed_when can express a module or command’s application-specific failure meaning, while changed_when controls change reporting. They are not permission to redefine a broken outcome as success. A YAML list of predicates has the documented conjunction behavior; use an explicit expression when OR is intended. Conditions should evaluate to Booleans under the supported engine. Inspect result availability before using fields, especially after skip or check mode. Keep the primary failure distinct from a later diagnostic or cleanup error so the report does not obscure what first prevented useful completion.

Rescue is a recovery branch, not a transaction engine

A block groups tasks and shared directives, rescue describes eligible failure handling, and always describes work that should run along the relevant block lifecycle. A successful rescue can allow execution to continue, but it does not automatically undo prior changes or establish application health. Define what recovery actually restores, how its own failures are reported, and whether the overall workflow should remain failed. Do not claim atomicity for a sequence of remote modules. A rejected configuration, an unreachable target, and a partially completed migration can require fundamentally different responses.

yaml Non-runnable block, rescue, and always illustration

- name: Illustrate a bounded recovery structure
  hosts: lab_nodes
  gather_facts: false
  vars:
    lab_ready: true
  tasks:
    - name: Describe validation and recovery boundaries
      block:
        - name: Validate a synthetic input
          ansible.builtin.assert:
            that:
              - lab_ready is boolean
              - lab_ready
      rescue:
        - name: Describe recovery intent
          ansible.builtin.debug:
            msg: Synthetic recovery branch only
      always:
        - name: Describe final evidence collection
          ansible.builtin.debug:
            msg: Synthetic lifecycle complete

The YAML uses only synthetic validation/debug actions but was not executed. Parsing does not prove rescue eligibility, engine behavior, or any real recovery.

Bound retries and require a safe repeat operation

until with retries and delay can handle a documented transient condition, but repeating an operation can duplicate side effects if it is not safe to retry. Distinguish polling a read-only status from resubmitting a transaction. Preserve attempts, final status, and the condition that justified another try. A timeout is not proof the earlier operation had no effect. Do not retry malformed input, permission denial, or an unsupported module indefinitely. The offline model below stops at a declared success within a finite list; it is not Ansible’s retry engine or a live polling loop.

python Model bounded retry intent offline

outcomes = ['transient', 'transient', 'ok']
limit = 3
final = 'failed'
for attempt, outcome in enumerate(outcomes[:limit], 1):
    if outcome == 'ok':
        final = 'success'
        break
print(f'attempts={attempt} final={final}')

Expected output

attempts=3 final=success

The synthetic result succeeds on the third allowed observation. It invokes no command, API, timer, module, or target and does not demonstrate safe retries for real state-changing work.

Recovery needs independent postconditions

A recovery plan should define durable state, configuration compatibility, required application behavior, and whether remaining hosts may proceed. Verify the intended postcondition rather than only the rescue task’s exit result. Test failure before effects, partial effects, retry exhaustion, recovery failure, and unreachable targets on disposable infrastructure when authorized. Keep the first error and subsequent actions in a concise timeline. The course performs no fault injection or remote recovery; it validates only offline structure and models, preserving the distinction between a reviewed design and an executed operational proof.

Common errors and fixes

ignore_errors is used as success handling

Why it happens: Continuation was confused with a correct postcondition.
How to fix it: Classify and handle the documented failure explicitly.
Prevent it next time: Assert useful completion independently of task continuation.

rescue is expected for unreachable hosts

Why it happens: Distinct execution categories were conflated.
How to fix it: Use the documented unreachable-host policy separately.
Prevent it next time: Test failure classes individually.

A timed-out transaction is retried blindly

Why it happens: Unknown completion was treated as no side effect.
How to fix it: Check the application’s repeat/idempotency contract.
Prevent it next time: Separate status polling from transaction submission.

Check your understanding

1. Why can successful rescue still leave an unacceptable deployment?

Show worked solution

Earlier tasks may have changed durable or partial state that rescue did not restore. The application may also remain unhealthy. Define and verify recovery postconditions and decide whether the overall workflow should still report failure.

2. What makes a retry condition defensible?