RabbitMQ / Advanced: ordered ownership, retained replay and accepted recovery / Lesson 14
RPC correlation, timeouts and replies
Design request/reply as a distributed operation with explicit identity, response lifetime and uncertain outcomes.
On this page
Lesson goals
- Distinguish correlation identity from business idempotency identity.
- Explain reply queues, Direct Reply-to and timeout limitations.
A request is not a local function call
An RPC client publishes a request, a worker processes it and a reply returns through messaging. Network delay, queueing, retries and worker failure can occur between every stage. A request timeout means the caller stopped waiting under its deadline; it does not prove the server did no work or cancel a completed effect. Name the business operation and define how a caller checks an uncertain result. For long work, an asynchronous job identifier and status query can make lifecycle clearer than holding one request open. Choose the interaction from the user requirement rather than hiding remote work behind an apparently cheap function call.
Correlate a reply without redefining the operation
AMQP 0-9-1 reply_to identifies the reply destination and correlation_id helps the client associate a response with pending work. A stable application operation key addresses duplicate execution; it can differ from a per-request correlation identifier. The JSON fixture keeps both concepts visible. If a caller retries an uncertain operation with a new correlation identifier, it should still preserve the business idempotency identity where the contract requires one occurrence. Validate response type and version before accepting the payload, and bound the pending-request map so missing replies cannot accumulate indefinitely. Unknown or late replies need an explicit handling policy.
{"reply_to":"lab.rpc.responses","correlation_id":"request-17-attempt-a","operation_id":"label-P17-v1","request_type":"calculate-label-v1"}correlation_id matches a response; operation_id names the intended business occurrence across supported retries.
Choose reply storage from disconnection requirements
A dedicated reply queue can be shared across several requests from a client, avoiding a new topology object for every call. Its exclusivity, durability and cleanup must match whether replies should survive client disconnection and how long results remain useful. Direct Reply-to avoids a normal reply queue but has transient, at-most-once response behavior and specific same-channel/consumer-mode constraints in the AMQP 0-9-1 path. It is unsuitable when replies need buffering through a disconnected caller. The word direct still involves a RabbitMQ node; it does not create a peer-to-peer network connection from worker to client.
Accept only a known pending response
The local map contains one correlation identifier. Receiving its response removes the pending entry and exposes the associated operation key. A second receipt no longer matches pending work. This is in-memory matching only: it does not implement durable result retention, timeouts or transport acknowledgment. A late response after timeout may still contain useful reconciliation evidence, so production code should follow its documented lifecycle instead of blindly treating every unmatched reply as malicious or every repeated reply as new work. A server can repeat a reply after losing its request acknowledgment, even when business execution was duplicate-safe.
pending = {'request-17-attempt-a': 'label-P17-v1'}
correlation_id = 'request-17-attempt-a'
operation_id = pending.pop(correlation_id, None)
assert operation_id == 'label-P17-v1'
assert pending.pop(correlation_id, None) is None
print(operation_id)label-P17-v1The response consumes one supplied pending entry; no message delivery or timeout is simulated.
Observe request, effect and reply separately
Acceptance should track broker request acceptance, server business completion, reply publication and client result acceptance as distinct facts. A publisher confirm cannot stand in for a valid reply. Check duplicate requests, a lost reply, a caller restart and an unknown correlation identifier in an authorized integration plan. This course creates no reply queue, client or RPC server and runs no Direct Reply-to subscription. It parses a property fixture and executes local correlation bookkeeping only. Actual client API behavior and response durability remain conditional on the chosen protocol, queue design and deployment version.
Common errors and fixes
A caller timeout is treated as cancellation of server work.
- Why it happens
- Waiting policy was confused with remote execution state.
- How to fix it
- Reconcile the original operation or use a supported explicit cancellation protocol.
- Prevent it next time
- Document uncertain outcomes before allowing automatic retries.
Every RPC call creates and deletes a new reply queue.
- Why it happens
- Response correlation was replaced by unnecessary topology churn.
- How to fix it
- Review a reusable reply queue or suitable Direct Reply-to design.
- Prevent it next time
- Choose reply lifetime and storage deliberately.
Direct Reply-to is assumed to retain replies during disconnection.
- Why it happens
- Its transient response contract was ignored.
- How to fix it
- Use a suitable reply queue or durable result/status design.
- Prevent it next time
- Test caller loss and late-response behavior explicitly.
Check your understanding
1. Should retrying an uncertain operation always create a new business operation identity?
Show worked solution
No. A new correlation identifier may identify a new request attempt, while the stable operation key should continue identifying the same intended occurrence. Otherwise an accepted first attempt and a later retry can both execute as apparently independent work. Use the destinationโs supported idempotency and result-reconciliation contract.
2. Does a positive request publisher confirm prove the client has received the computed label?
Show worked solution
No. It concerns broker acceptance of the request under its messaging contract. Server execution, reply publication and client validation remain separate stages. Track them explicitly, and preserve the original operation identity when any stage becomes uncertain rather than reporting end-to-end completion from transport acceptance alone.
Key takeaways
- Correlation, business identity and reply storage have distinct purposes.
- A timeout limits waiting; it does not establish nonexecution.