RabbitMQ / Intermediate: replicated work, retry safety and operational evidence / Lesson 10

Heartbeats, connections and recovery

Detect failed transport promptly while preserving channel state and uncertain message outcomes during recovery.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain heartbeat negotiation and false-positive tradeoffs.
Separate transport recovery, topology recovery and message reconciliation.

An open TCP object is not proof of a healthy peer

A network path can fail without immediately closing the local socket. Heartbeats help peers detect an unresponsive connection sooner than relying on ordinary operating-system TCP failure detection alone. They do not prove the business handler is making progress or the destination database is healthy. Choose a detection budget that accounts for scheduling pauses and transient congestion; an excessively small timeout can create false failures and recovery churn. Keep heartbeat timeout separate from publication deadline, consumer acknowledgment timeout and application request timeout. Each clock describes a different boundary and should have its own evidence and consequence.

Model the documented maintained-client rule

The local function models the negotiation described for RabbitMQ-maintained Java, .NET and Erlang clients: if either proposal is zero, use the larger; otherwise use the smaller. Both zero requests disable heartbeats under that rule. This is not a universal promise for every third-party client or protocol. Confirm the selected library's implementation before choosing configuration. The synthetic six- and ten-second inputs yield six, while zero and ten yield ten. The example performs no wait and opens no connection. It illustrates agreement arithmetic only, leaving actual failure detection timing and event-loop responsiveness untested.

python Offline maintained-client negotiation model

def negotiated(server_s, client_s):
    return max(server_s, client_s) if 0 in (server_s, client_s) else min(server_s, client_s)

assert negotiated(6, 10) == 6
assert negotiated(0, 10) == 10
assert negotiated(0, 0) == 0
print(negotiated(6, 10), negotiated(0, 10))

Expected output

6 10

The arithmetic follows the stated client-family rule and does not measure heartbeat failure detection.

Reopening transport does not restore every guarantee

Connection failure closes the old channels. A supported client recovery feature may recreate connections, channels, topology and subscriptions, but exact coverage and callbacks depend on the library. Protocol errors such as incompatible queue declarations still need correction; reconnecting repeatedly does not repair them. Outstanding delivery tags and confirm sequence context belong to the old channel. Preserve stable application identities separately. Initial connection failure can also have a different retry path from recovery after an established connection is lost. Review lifecycle documentation rather than assuming an auto-recovery flag handles every startup, shutdown and protocol-error case.

Keep a bounded recovery plan

The JSON record describes retry delay, a finite startup deadline and the required reconciliation of unconfirmed operations. It is an application policy sketch rather than a client-library configuration schema. Backoff and jitter can reduce synchronized reconnect storms, but they do not fix bad credentials or a missing virtual-host grant. Distinguish permanent configuration errors from plausible temporary transport failure. A consumer effect that completed before connection loss can be redelivered when its acknowledgment was uncertain. A publisher send can also have succeeded without an observed confirm. Recovery must therefore restore ownership and reconcile business outcomes, not merely obtain another socket.

json Synthetic connection-recovery policy

{"startup_deadline_seconds":30,"retry_delay_seconds":2,"reconcile_unconfirmed":true,"restore_subscriptions":"client-supported","preserve_business_identity":true}

Fields document intended behavior; they are not constructor options for every RabbitMQ client.

Use lifecycle evidence before changing timeouts

Correlate disconnect time, negotiated heartbeat, blocked-connection signals, application event-loop delays and the first channel shutdown reason. A resource alarm or a blocked client thread can look different from a broken network path. Avoid disabling heartbeats as a generic cure for frequent reconnects; diagnose why the agreed budget is missed. This lesson reads no live connection state, changes no kernel setting and runs no reconnect loop. It executes negotiation arithmetic and parses a design record only. Actual client recovery, topology restoration and duplicate-safe destination behavior require separate authorized tests with the selected runtime and version.

Common errors and fixes

Frequent reconnects are treated only by disabling heartbeats.

Why it happens: Detection symptoms were changed without diagnosing pauses or network conditions.
How to fix it: Correlate negotiated timeout, event-loop health and transport evidence.
Prevent it next time: Review each timeout’s scope and expected failure response.

Auto-recovery repeatedly encounters PRECONDITION_FAILED.

Why it happens: An incompatible topology declaration remains unchanged.
How to fix it: Correct the ownership and declaration contract before retrying.
Prevent it next time: Separate protocol/configuration faults from transient transport errors.

Old delivery tags are reused after reconnect.

Why it happens: Channel-scoped state was treated as durable identity.
How to fix it: Use new channel context and original application operation keys.
Prevent it next time: Test uncertain acknowledgment and confirm outcomes explicitly.

Check your understanding

1. Does a negotiated heartbeat of 10 seconds prove a label job finishes within 10 seconds?

Show worked solution

No. The heartbeat concerns peer connection responsiveness, not business execution duration. Publication, processing and request deadlines need separate budgets and enforcement. A responsive client can still be waiting on a slow external dependency, so measure task completion rather than inferring it from transport liveness.

2. Why can reconnect success still leave an uncertain publication outcome?