Spring Boot / Advanced: observability and resilient releases / Lesson 17

Reliable workflows and the outbox boundary

Coordinate committed database state, durable operation identity, delivery retries, and consumer deduplication without claiming local events provide exactly-once remote effects.

About 5 min readContent reviewed 2026-10-26

On this page

Lesson goals

Separate local atomicity from remote delivery.
Explain durable idempotency and payload equivalence.
Use events as hints rather than the only recovery record.

Identify which effects one transaction can protect

An order row and an outbox row can share one database transaction when they use the same supported transactional resource and manager. An HTTP notification, email, or payment is another boundary and cannot be undone by ordinary JPA rollback. Sending it before commit can publish an order that never exists; sending only from volatile memory after commit can lose work on a crash.

The outbox stores delivery intent with the business change. A worker later reads durable pending events, delivers under a bounded policy, and records progress. Acknowledgement can be lost after delivery, so duplication remains possible. The consumer needs its own durable idempotency or reconciliation contract. Do not describe the pattern as universal exactly-once transport or as a replacement for authorization and data retention.

Persist intent through the managed transaction boundary

The fragment shows a custom OrderStore capability and OutboxStore capability under one managed transaction. insertOrReplay and ensureOrderCreated are application contracts, not magic Spring Data method names. Their implementations must enforce scoped unique request/event identity, payload equivalence, and the intended database atomicity. The code is unexecuted and does not establish those guarantees merely by annotation.

A request key must identify the same intended operation, not any request with a convenient repeated string. Include trusted tenant scope and every operation-defining field. On replay, return the original committed result only after verifying equivalence; reject a conflicting payload. If commit confirmation is lost, preserve the original key and reconcile instead of inventing a new operation that can be accepted twice.

java Illustrative local atomic business/outbox intent

package training.catalog;

import org.springframework.context.ApplicationEventPublisher;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

record OutboxHint(long orderId) {}

@Service
class OrderApplication {
    private final OrderStore orders;
    private final OutboxStore outbox;
    private final ApplicationEventPublisher events;
    OrderApplication(OrderStore orders, OutboxStore outbox, ApplicationEventPublisher events) {
        this.orders=orders; this.outbox=outbox; this.events=events;
    }
    @Transactional
    public Receipt place(TrustedOrderCommand command) {
        Receipt receipt=orders.insertOrReplay(command);
        outbox.ensureOrderCreated(receipt, command.operationKey());
        events.publishEvent(new OutboxHint(receipt.orderId()));
        return receipt;
    }
}

The custom stores must share the actual transaction and implement durable scoped identity. Publishing the local hint is not the durable record itself.

A local after-commit hint is only an optimization

The listener fragment signals a dispatcher after a supported transaction commits. If the process exits before that signal is handled, a later durable scan must still find the outbox event. The signal can reduce latency, but it cannot be the sole source of delivery truth. Events published outside a transaction also have configured listener semantics that must be understood rather than assumed.

A worker needs bounded admission, claim/lease ownership where appropriate, retry deadlines, poison-event handling, and safe shutdown. Several application instances must not accidentally apply the same effect without consumer protection. Transaction advice for recording progress and delivery confirmation needs explicit design. Do not start a scheduled worker or call any external destination merely to demonstrate this framework fragment.

java Illustrative after-commit wake-up hint

package training.catalog;

import org.springframework.stereotype.Component;
import org.springframework.transaction.event.TransactionPhase;
import org.springframework.transaction.event.TransactionalEventListener;

@Component
class OutboxWakeup {
    private final OutboxDispatcher dispatcher;
    OutboxWakeup(OutboxDispatcher dispatcher) { this.dispatcher=dispatcher; }
    @TransactionalEventListener(phase=TransactionPhase.AFTER_COMMIT)
    void committed(OutboxHint hint) { dispatcher.signal(); }
}

A durable scanner remains required. No transaction event or dispatcher was executed, and signal is a custom capability rather than a delivery guarantee.

Show why in-memory deduplication cannot establish durability

The complete Java model processes e1,e1,e2 using a local set, then simulates losing only that set while the modeled applied count remains. Replaying e1 raises the count from two to three. This is an actual deterministic counterexample to treating volatile memory as a durable deduplication record; it is not a crash simulation or a database transaction test.

A real consumer can store a unique event identity with its business effect in one supported transaction, or use another reviewed reconciliation design. Define payload mismatch handling and retention so an old legitimate replay is not mistaken for a new action after identity records expire. Protect event content and avoid storing whole personal records when a smaller approved payload suffices.

java Dependency-free counterexample to volatile dedup durability

public class Main {

 public static void main(String[] args) throws Exception {
var seen=new java.util.HashSet<String>();
int applied=0;
for(String event:java.util.List.of("e1","e1","e2")) if(seen.add(event)) applied++;
System.out.println("first_pass "+applied);
seen.clear();
if(seen.add("e1")) applied++;
System.out.println("after_memory_loss "+applied);
 }
}

Expected output

first_pass 2
after_memory_loss 3

Clearing only dedup memory permits one repeated effect in the model. No actual crash, persistence resource, or external delivery is exercised.

Test each uncertain interval, not only the happy path

Verify failure before commit, after commit before wake-up, after delivery before progress acknowledgement, duplicate request/event identity, mismatched payload, worker restart, and bounded retry exhaustion. Check the final business state and delivery intent, not only a log saying “published.” Framework events and transaction proxies must be exercised through the managed boundary in an approved isolated integration environment.

No database, broker, email service, or payment system is started here. Keep architecture assumptions explicit, including which store owns atomicity and which side effects require reconciliation. A robust workflow tells the caller whether work is rejected, accepted, committed, pending delivery, completed, or uncertain; collapsing these into one success flag makes recovery and user communication unreliable.

Common errors and fixes

An after-commit event disappeared on restart

Why it happens: Only an in-memory signal carried delivery intent.
How to fix it: Persist intent with business state and rescan pending work.
Prevent it next time: Test the commit-to-dispatch gap.

A consumer repeated a business mutation

Why it happens: Delivery acknowledgement and durable dedup/effect coordination were incomplete.
How to fix it: Use a scoped stable identity and reviewed atomic consumer design.
Prevent it next time: Test delivery-before-ack failure.

A duplicate key returned another operation's result

Why it happens: Scope or payload equivalence was not enforced.
How to fix it: Validate trusted scope and operation-defining fields before replay.
Prevent it next time: Test mismatched payload and cross-tenant keys.

Check your understanding

1. Why does an outbox not automatically give exactly-once remote delivery?

Show worked solution

The recipient can apply the event before its acknowledgement is lost, causing a retry. Durable intent helps recovery but the receiver still needs idempotency or reconciliation. State the transport and business-effect guarantees separately.

2. A client loses commit confirmation. Should it submit a new operation key?