RabbitMQ / Advanced: ordered ownership, retained replay and accepted recovery / Lesson 18

Failure schedules and recovery acceptance

Use explicit crash windows and business invariants to decide what a recovery test actually proves.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish pure model checks from broker/client integration evidence.
Define accepted loss, duplicate behavior and recovery time in observable terms.

Start with a named failure and invariant

A meaningful test states where failure occurs and what must remain true afterward. For a label job, one invariant might be that every accepted unique operation reaches either one valid result or an explicit terminal state. That differs from requiring every delivery callback to occur once. Publisher acceptance, routing, delivery, external effect and acknowledgment have separate crash windows. Use a small schedule with known identities before introducing concurrency. A test that merely repeats the implementation or observes a green connection cannot establish the intended business property. State assumptions about source coverage, queue type and destination behavior alongside the expected result.

Put uncertainty between concrete milestones

The JSON schedule describes a worker effect that completes before connection failure prevents a certain acknowledgment. On replacement, the same operation can be delivered again. A correct destination rule recognizes the original identity and avoids another effect. A different schedule places failure before broker acceptance, and another after acceptance but before confirm observation; they require different evidence even if the client sees a similar timeout. Do not simulate a dangerous real process kill or network partition in a configured environment merely to illustrate the idea. A reasoned schedule is useful teaching evidence when its limitations are explicit.

json Synthetic crash-window acceptance schedule

{"operation_id":"label-P17-v1","schedule":["deliver","apply-effect","connection-fails-before-ack-certainty","redeliver-same-operation"],"required_destination_result":"one-accepted-effect"}

The steps describe a conceptual failure and required invariant; no connection is interrupted.

Check unique completion against repeated deliveries

The local model feeds evt-a twice and evt-b once into a synthetic effect ledger keyed by immutable operation identity. It verifies two unique completed identities despite three deliveries. The dictionary is volatile and single-threaded, so the result does not prove atomic database writes or durable restart safety. Production evidence must exercise the actual supported destination transaction or idempotency protocol. Include conflicting payloads, expired deduplication evidence and a terminal invalid input in the broader plan. Record both raw deliveries and unique business outcomes so redelivery is visible without being automatically counted as a second legitimate operation.

python Offline repeated-delivery completion invariant

deliveries = ['evt-a', 'evt-a', 'evt-b']
effect_ledger = {}
for operation_id in deliveries:
    effect_ledger.setdefault(operation_id, 'completed')
assert len(deliveries) == 3
assert set(effect_ledger) == {'evt-a', 'evt-b'}
print(len(deliveries), len(effect_ledger))

Expected output

3 2

Three deliveries map to two local identities; the volatile dictionary proves no durable transactional effect.

Include verification in the recovery clock

Recovery point describes acceptable missing business history, while recovery time describes the interval until required service behavior is restored. Merely restarting a broker does not prove queued work, reply state, source manifests or external results are complete. The plan should account for detection, safe restoration, backlog drain and business verification. A separate recovery copy and its restore restrictions matter for failure classes that replication cannot cover. Preserve original versions, node identities where relevant and effective policies in the evidence. A result should say exactly which failure schedule was exercised rather than generalizing one passing case into universal resilience.

Report executed scope and untested boundaries

This course executes only isolated standard-library Python models and parses JSON fixtures. Client Java fragments, SQL sketches and all broker configuration remain non-runnable illustrations. No broker, database, cloud service, administrative command, scan or load test is used. Those limits do not make the models useless: they isolate identity, arithmetic, routing assumptions and acceptance invariants before a real integration plan. A deployment-readiness claim still requires a separately authorized compatible environment with synthetic data and exact client/server versions. Preserve expected and observed results plus the excluded behaviors so future reviewers cannot mistake a local computation for an actual failover certification.

Common errors and fixes

A local set check is reported as a successful broker failover test.

Why it happens: The tested boundary was not stated.
How to fix it: Report the exact fixture, runtime and invariant actually checked.
Prevent it next time: Keep pure-model and integration evidence separately labeled.

Recovery is declared before unique business coverage is reconciled.

Why it happens: Process health replaced service acceptance.
How to fix it: Verify required operations, terminal outcomes and destination state.
Prevent it next time: Include verification and backlog drain in the recovery budget.

A retry test omits failure after a completed external effect.

Why it happens: Only definite pre-execution failure was covered.
How to fix it: Add the effect-to-ack uncertainty schedule with original identity.
Prevent it next time: Derive test cases from every responsibility transfer.

Check your understanding

1. Why does the local ledger not prove duplicate protection after a process restart?

Show worked solution

It exists only in memory and has no transactional relationship with an external effect. Restart loses its evidence, and concurrent workers are not modeled. A durable claim needs the real destination’s supported identity and effect protocol, tested at the precise crash boundary in an authorized environment.

2. Does surviving one node failure prove recovery from accidental deletion of every queue?