RabbitMQ / Intermediate: replicated work, retry safety and operational evidence / Lesson 9

Duplicate-safe workers and the outbox

Close the authoritative-state-to-message gap while treating relay duplication and destination effects explicitly.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Identify the database/message dual-write crash window.
Explain atomic identity-and-effect handling at the destination.

Two operations are not one distributed transaction

Suppose a synthetic order is confirmed in an authoritative database and a label job is then published. A crash between those steps leaves confirmed state without a job. Reversing the order can publish work for a database transaction that later fails. Publisher confirms do not enlist an arbitrary database transaction, and consumer acknowledgments do not close the producer-side gap. An outbox records the intended event in the same supported database transaction as the state change, then a relay publishes committed intents. The important invariant is joint commitment of business state and durable publication intent, not merely performing both calls eventually.

Keep the original operation key in the intent

The SQL sketch writes order confirmation and one outbox row inside an illustrative transaction. It assumes suitable tables and constraints that are not created or tested here. A real implementation needs engine-specific isolation, unique identity, serialization and a bounded cleanup policy. Recording the outbox row on a separate connection after commit reintroduces the gap. A relay can publish successfully and crash before marking the intent sent, so duplicate publication remains possible. Marking sent first risks loss instead. Preserve the same operation identifier across relay restarts and treat confirm uncertainty as a reconciliation case.

sql Illustrative authoritative-state/outbox sketch

BEGIN;
UPDATE lab_orders SET status = 'confirmed' WHERE id = 17;
INSERT INTO lab_outbox (operation_id, order_id, operation_type)
VALUES ('label-P17-v1', 17, 'calculate-label');
COMMIT;

No database is contacted; the transaction requires actual engine constraints and a separate relay design.

Deduplicate where the business effect commits

A consumer that checks a separate seen set and then sends a remote notification still has a race and a crash window. The identity marker and effect must be atomic under a destination-supported transaction or idempotency protocol, or be reconciled through an explicit alternative. A volatile Python set disappears on restart. Deduplication retention must cover the supported retry and replay horizon, and identity may require tenant and operation scope. Equal payloads are not necessarily one occurrence: two legitimate scans can look identical. Conflicting payloads under one stable operation key should be investigated rather than silently overwritten.

Check duplicate identity and payload consistency locally

The local model receives one operation twice with the same result and one separate operation. It retains two unique results while asserting that a duplicate key agrees with its previous payload. This checks a chosen identity invariant in a single-threaded in-memory schedule, not durable concurrency or a payment-provider contract. A real system must also decide how to respond when a duplicate is already applied: often a stable already-completed result is preferable to an error that triggers endless retries. Keep transport-delivery counts separate from unique-operation counts so reconciliation does not mistake legitimate retransmission for additional completed business work.

python Offline identity-and-payload consistency model

deliveries = [('label-P17-v1', 'ready'), ('label-P17-v1', 'ready'), ('label-P18-v1', 'ready')]
results = {}
for operation_id, result in deliveries:
    if operation_id in results:
        assert results[operation_id] == result
    results[operation_id] = result
assert len(results) == 2
print(len(results))

Expected output

The duplicate agrees with prior content; no durable atomic external effect is implemented.

Account for each handoff independently

A complete design can identify authoritative changes with intents, intents accepted by RabbitMQ and unique destination effects. These checkpoints can diverge during recovery and should not be compressed into one success counter. Include pending and terminal intents in operational review, with a retention horizon that preserves required evidence. A consumer should acknowledge only after its supported responsibility boundary completes. This lesson executes no SQL, creates no outbox and contacts no destination. Its local identity model is a narrow test, while cross-system atomicity and relay recovery remain design requirements that need separate authorized integration evidence.

Common errors and fixes

A committed order has no published job after a crash.

Why it happens: Database change and publication were uncoordinated writes.
How to fix it: Store a durable intent atomically with authoritative state.
Prevent it next time: Review the crash between every cross-system step.

The relay publishes an intent twice.

Why it happens: Publication succeeded before relay progress was recorded.
How to fix it: Preserve stable identity and use duplicate-safe destination handling.
Prevent it next time: Test restart after acceptance but before the sent marker.

A durable seen marker suppresses an effect that never happened.

Why it happens: Marker and effect were committed separately.
How to fix it: Use supported atomicity or an explicit reconciliation protocol.
Prevent it next time: Test both sides of the marker/effect boundary.

Check your understanding

1. Does an outbox automatically make a notification happen exactly once?

Show worked solution

No. It links authoritative state with a durable publication intent when implemented transactionally. Relay delivery and notification execution still have their own failure windows. Use a destination-supported idempotency or reconciliation protocol and retain original operation identity through uncertain sends and consumer redelivery.

2. Why does the local duplicate check compare payloads as well as identifiers?