RabbitMQ / Intermediate: replicated work, retry safety and operational evidence / Lesson 7

Quorum replication and write availability

Evaluate replicated queue safety against member availability and declared failure domains.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain majority agreement and queue-specific membership.
Separate queue availability from cluster process health and independent recovery.

Replication belongs to the queue’s member set

A quorum queue uses a leader and follower members with Raft-based agreement. Its member set is a concrete queue property, not simply every node that happens to belong to the RabbitMQ cluster. A cluster can contain more nodes than a particular queue has replicas. Adding a node does not justify assuming every existing queue immediately gains a healthy copy there. Review actual membership and placement separately from intended replication size. Spread members across the failure domains the design needs to tolerate, because three replicas in one lost domain provide a different risk profile from three independently placed copies.

Count majority before promising availability

The offline function computes floor(N/2)+1 for a declared member count. Three members require two, and five require three. This illustrates quorum arithmetic only, not leader election, network partitions or disk behavior. A reachable node in a minority cannot be treated as evidence the queue can safely accept the same writes as a healthy majority. Availability also needs a suitable leader and functioning resources. Larger groups can tolerate some additional member failures but increase replication and coordination costs. Choose membership from an explicit failure and capacity model rather than assuming more replicas are always a free improvement.

python Offline majority threshold calculation

def majority(members):
    if members < 1:
        raise ValueError('INVALID_MEMBER_COUNT')
    return members // 2 + 1

assert majority(3) == 2
assert majority(5) == 3
print(majority(3), majority(5))

Expected output

2 3

The result describes required counts only and does not test a replicated queue.

Confirmed data and current service are different questions

Publisher confirms for quorum queues are tied to the documented replicated acceptance behavior. A later temporary loss of majority can make the queue unavailable without proving previously confirmed data was deleted. Conversely, reaching an endpoint and seeing cluster processes alive does not prove the queue has a usable quorum. Keep data-loss evidence, availability evidence and client uncertainty separate. If a connection fails before a confirm is observed, retrying the original operation can still produce duplicates. Replication does not remove the application’s stable-identity and destination-idempotency requirements at the consumer effect boundary.

Review a synthetic placement inventory

The JSON record lists three members and their separate synthetic domains. It is a review fixture, not a command to add members, rebalance leaders or change a cluster. Compare intended placement with actual evidence in an authorized operational review. A failed member’s data directory should never be casually recreated as an incident shortcut: membership and recovery procedures are version-sensitive and can affect recoverability. Quorum queues are durable replicated work queues, while Streams serve retained replay use cases with different consumption semantics. Classic mirrored-queue recipes do not apply to this RabbitMQ 4.1 baseline.

json Synthetic member-placement review

{"queue":"lab.labels","type":"quorum","members":[{"node":"lab-a","domain":"zone-a"},{"node":"lab-b","domain":"zone-b"},{"node":"lab-c","domain":"zone-c"}]}

The inventory states intended isolation domains; no membership is discovered or modified.

Plan recovery beyond one healthy node

An incident review should identify affected queues, remaining members, confirm failures, consumer progress and the first resource or network symptom. Do not change membership or discard replicas merely because a dashboard is red. Independent recovery against logical mistakes or total-domain loss needs separate retained evidence and a supported restoration plan. This lesson performs no cluster query, member operation or node restart. It executes only majority arithmetic and parses a placement fixture, so actual quorum availability, durability and recovery behavior remain untested rather than being inferred from the local model.

Common errors and fixes

Every cluster node is assumed to hold this queue’s data.

Why it happens: Cluster membership was substituted for queue membership.
How to fix it: Inspect the queue-specific replica inventory through authorized evidence.
Prevent it next time: Keep intended and actual membership separately documented.

A minority endpoint is called a healthy writable queue.

Why it happens: Network reachability replaced majority and leader evidence.
How to fix it: Review queue availability and client outcomes together.
Prevent it next time: Include majority-loss scenarios in the failure model.

Three replicas disappear in one domain failure.

Why it happens: Placement did not separate the relevant correlated risk.
How to fix it: Review placement and independent recovery coverage.
Prevent it next time: Name the actual failure domains in design acceptance.

Check your understanding

1. With five declared members, do two available members form a majority?

Show worked solution

No. The arithmetic requires three. This does not itself diagnose an actual queue, because leader state, member health and timing also matter. A minority becoming reachable is useful connectivity evidence but cannot replace the queue’s documented agreement and availability rules.

2. Does temporary quorum unavailability prove all confirmed messages were lost?