Software Architecture / Interactions: distribution, failure, and operation / Lesson 9

Handle cross-boundary consistency without imaginary atomicity

Explain dual-write gaps, transactional outbox delivery, and business compensation using a fictional reservation workflow.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Identify failure windows between a durable write and an external message.
Design explicit workflow states and compensation rather than assuming distributed rollback.

Two successful APIs do not create one transaction

Suppose booking commits a reservation and then publishes ReservationConfirmed. A crash between those actions leaves a reservation with no event. Reversing the order can publish an event for a reservation that never commits. Catching exceptions does not close the crash window, and a network timeout can leave publication uncertain. The architecture must connect the durable business change with a durable intention to publish or use another explicitly supported transaction mechanism. State which storage and broker capabilities the chosen design actually relies on.

Record the publication intention with the business change

A transactional outbox stores the reservation and an outgoing event record in the same local transaction. A separate relay later publishes pending records. This closes one dual-write gap but introduces relay concurrency, cleanup, ordering, and duplicate-delivery decisions. The relay can crash after publication but before marking a record sent, so consumers still need safe duplicate handling. Do not describe the outbox as exactly-once business processing. Its useful guarantee is narrower: a committed business change retains the associated publication intention under the stated database assumptions.

text A proposed outbox protocol

BEGIN local transaction
  insert reservation r-18 as confirmed
  insert outbox event e-18 for reservation r-18
COMMIT
relay publishes e-18 later
consumer records e-18 before applying its repeat-sensitive effect

This is pseudocode, not executable database syntax. The consumer’s identity record and effect also need an appropriate atomic boundary or an explicitly idempotent external operation.

Model intermediate states instead of hiding them

If a reservation workflow spans inventory and payment, a single local transaction cannot automatically cover both independent owners. Define states such as awaiting payment, confirmed, cancellation requested, and cancelled, together with allowed transitions. Name the owner that decides the final business outcome. A pending state must have an expiry or investigation path so work cannot disappear indefinitely. The UI should describe uncertainty honestly and allow outcome retrieval. A state diagram is valuable only when persistence, event handling, and operator procedures agree with it.

Compensation is a new business action

A refund or released seat does not erase history as a database rollback would. Another actor may have seen a confirmation, a provider may charge a fee, and compensation itself can fail. Specify compensating actions with stable identities, permitted transitions, and retry or escalation behavior. Some actions are irreversible and need manual resolution or a different workflow order. The sample transition table exposes these decisions. It is not a universal payment design and performs no financial operation; all states and identifiers are synthetic.

yaml A partial workflow transition table

awaiting_payment:
  payment_succeeded: confirmed
  payment_failed: cancellation_requested
cancellation_requested:
  seat_released: cancelled
  release_failed: requires_review
confirmed:
  duplicate_payment_succeeded: confirmed

Repeated success leaves the confirmed state unchanged. requires_review has an owner to define in the implementation. The table omits many product-specific rules and must not be mistaken for a complete payment protocol.

Exercise the crash windows and reconciliation path

A useful verification matrix injects failure before commit, after commit before publish, after publish before relay acknowledgement, and during consumer handling. Check both business state and durable publication state after recovery. Reconciliation compares authoritative records and unresolved workflow transitions with a documented decision process; it must not blindly replay irreversible actions. Preserve an audit trail of transitions without recording sensitive payloads unnecessarily. This lesson describes the test design only and claims no broker, transaction engine, payment provider, or failure-injection execution.

Common errors and fixes

A committed booking has no notification event.

Why it happens: Business write and publication were independent crash-prone writes.
How to fix it: Store the publication intention in the intended atomic boundary.
Prevent it next time: Test the after-commit, before-publish failure window.

An outbox event produces the same effect twice.

Why it happens: Relay redelivery was assumed impossible.
How to fix it: Use consumer deduplication or an idempotent effect with stable identity.
Prevent it next time: Include relay crashes after successful publication.

Compensation is described as erasing history.

Why it happens: Business reversal was confused with local rollback.
How to fix it: Model a new owned action and its own failure states.
Prevent it next time: Document irreversible effects before choosing workflow order.

Check your understanding

1. Why does marking an outbox row sent before publication risk losing the event?

Show worked solution

A crash after the mark but before publication leaves no pending row for the relay to retry. Publishing first avoids that loss window but permits duplicates if the later mark fails. Choose at-least-once delivery with safe consumer effects or another precisely verified mechanism; do not claim both loss prevention and duplicate elimination from ordering alone.

2. A seat-release compensation repeatedly fails. What should happen?