Ansible / Advanced: orchestration, platforms, tests, and diagnosis / Lesson 17

Build layered automation tests and honest evidence

Separate parser, schema, renderer, module, convergence, application, and recovery tests while keeping every execution boundary controlled.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Choose tests that verify public behavior rather than incidental task layout.
Explain the evidence supplied by each validation layer.
Plan disposable convergence and recovery tests without running hosts here.

Each validation layer answers a different question

YAML parsing checks data syntax. Project schema checks validate required inputs and types. Ansible-aware syntax and lint tooling can add language and style diagnostics, while actual rendering exercises the pinned templating engine. Disposable target tests examine module behavior and convergence. Application checks verify useful results, and recovery tests examine interruption or partial-state handling. No single green check substitutes for all layers. Record which tools, versions, fixtures, and execution scope were used. This course has PyYAML but no ansible-core or Jinja2, so its performed evidence is intentionally limited to parser and offline model checks.

A test environment must not inherit production scope

A test harness should use explicit synthetic inventory, configuration, dependency versions, and non-secret inputs. Merely naming a test file does not stop Ansible from discovering a local configuration or inventory source. Some sources, lookups, or plugins can contact services during evaluation, so review them before execution. Prefer a disposable target environment with a documented cleanup and recovery owner for later authorized integration tests. Do not install complex services just to claim a runtime test. A meaningful offline fixture is better evidence than an inaccurately labeled production-like experiment whose effects were not controlled.

A second-run test must observe real state and reporting

For an authorized integration test, apply the same desired state to a controlled target, verify the result, then repeat without changing inputs. Unnecessary change should stop according to the module and project contract. Do not satisfy the test by forcing changed_when: false while effects continue. Include deterministic template output and handler activation counts where they are public behavior. A second run can legitimately change if external state or inputs changed, so preserve those assumptions. Also verify the application independently: convergence alone does not prove availability, data correctness, or restart recovery.

Make acceptance requirements visible in the play design

An assertion can express an invariant and fail clearly when a declared result does not meet it. Use it to check a real public requirement, not merely mirror the same internal condition under another name. The following task checks synthetic input shape only and does not probe an application. A complete workflow may additionally need supported read-only health checks, configuration validation, or data checks on the isolated target. Each of those operations has its own scope and credentials. Keep diagnostic messages specific but avoid printing secrets or full result objects unnecessarily.

yaml Non-runnable assertion design

- name: Illustrate explicit input acceptance
  hosts: lab_nodes
  gather_facts: false
  vars:
    report_contract:
      workers: 2
      enabled: true
  tasks:
    - name: Validate the declared synthetic contract
      ansible.builtin.assert:
        that:
          - report_contract.workers is integer
          - report_contract.workers > 0
          - report_contract.enabled is boolean
        fail_msg: Invalid synthetic report contract

The YAML was parsed only. It does not execute Ansible tests, inspect targets, or prove that the deployed role validates the same interface.

Report actual tests and keep missing evidence explicit

A useful test report lists fixture scope, exact assertions, outcomes, and untested layers. An offline state-transition model can demonstrate an idempotence concept, but it must not be called a second Ansible run. Keep intentional failure cases and verify the harness itself returns failure when assertions fail. Retain examples for undefined variables, wrong types, partial loop failure, handler timing, tag prerequisites, and delegated concurrency. Broaden testing only when a new risk or failure justifies it. The goal is a compact reliable contract suite, not a large set of tests that simply restates implementation details.

python Test a small synthetic convergence contract offline

def converge(observed, desired):
    return desired.copy(), observed != desired
desired = {'workers': 2, 'enabled': True}
state, first_changed = converge({'workers': 1, 'enabled': True}, desired)
state, second_changed = converge(state, desired)
assert first_changed and not second_changed and state == desired
print('PASS synthetic convergence contract')

Expected output

PASS synthetic convergence contract

This Python function copies dictionaries only. It validates the toy model’s public postconditions and is explicitly not an Ansible module or two real target runs.

Common errors and fixes

Parser success is called integration coverage

Why it happens: Untested engine and target layers were omitted from the report.
How to fix it: Name each exercised and untested boundary accurately.
Prevent it next time: Keep evidence layers separate in release notes.

A test double hides all failure paths

Why it happens: The fixture always returns success or desired data.
How to fix it: Add controlled missing, malformed, and partial-result cases.
Prevent it next time: Test the contract’s rejection behavior too.

Idempotence is faked through reporting overrides

Why it happens: A green summary replaced state evidence.
How to fix it: Observe real effects and truthful changes on disposable targets.
Prevent it next time: Separate convergence, handlers, and application acceptance.

Check your understanding

1. What should a report say when only PyYAML and Python models ran?

Show worked solution

State that YAML syntax and the declared offline assertions passed, and explicitly say Ansible execution, Jinja rendering, connection behavior, module effects, and target idempotence were not tested. This is useful honest evidence, not a reason to invent runtime proof.

2. Why can a public-behavior test survive role refactoring better than a task-count assertion?