RabbitMQ / Intermediate: replicated work, retry safety and operational evidence / Lesson 12
Monitoring ready, unacked and backpressure
Interpret queue metrics through message ownership, rates and business freshness before changing capacity.
On this page
Lesson goals
- Distinguish ready backlog from unacknowledged deliveries and completed effects.
- Correlate queue growth with consumer behavior and resource alarms.
Ready and unacked describe different responsibility states
Ready messages wait for delivery, while unacknowledged messages have been delivered under manual acknowledgment and remain outstanding. Neither count alone states how many unique business actions are incomplete. A worker can have completed an effect but not yet acknowledged it, or can acknowledge too early while downstream work remains. Queue depth also does not directly measure elapsed age. Include a business freshness signal and stable operation counts when users care about report delay or duplicate jobs. Define metric source and sampling interval explicitly so comparisons are not made across stale snapshots or unrelated virtual hosts.
Compare accepted arrivals with removals
The local arithmetic starts with one hundred synthetic ready messages, adds eighty accepted arrivals and removes sixty delivered messages under a deliberately simplified schedule. Ready backlog becomes one hundred twenty. It does not model acknowledgments returning credit, redelivery, expiry or dead-lettering. Those paths must be included in a real accounting explanation. A growing ready count can reflect increased arrivals rather than a stopped consumer. A high unacked count can reflect long processing, missing acknowledgments or excessive prefetch. Follow the responsibility transition that is actually stalled before deciding that more consumers or larger windows are the answer.
ready_before = 100
accepted_arrivals = 80
delivered = 60
ready_after = ready_before + accepted_arrivals - delivered
assert ready_after == 120
print(ready_after)120The model excludes requeues, expiry and dead-lettering and does not represent complete queue accounting.
Backpressure can be deliberate protection
Memory and disk alarms can block publishing connections to protect broker resources. A memory watermark is an alarm threshold, not a hard guarantee that memory never exceeds it. Flow control and a resource alarm also have different meanings from a failed routing key. Preserve publisher blocked/unblocked evidence, confirm latency, node resource signals and consumer completion trends together. Raising a watermark to make an alarm disappear can remove safety margin without fixing backlog growth. Separate publishing and consuming connection behavior where the client design requires it, and review the actual documented alarm effects for the deployed version.
Estimate recovery from net completion capacity
The second offline calculation has a backlog of three hundred, arrivals of fifty per second and completions of sixty-five. Under constant comparable work, net drain is fifteen and catch-up is twenty seconds. If arrivals meet or exceed completion, this model has no finite catch-up time. Real estimates need size, processing-time, redelivery and partition or queue-skew evidence, not one average. Additional consumers can saturate the label provider and reduce effective completion. Treat the model as a way to expose assumptions before an authorized capacity study, not as proof of measured RabbitMQ throughput or a promised recovery time.
backlog = 300
arrival_per_s = 50
complete_per_s = 65
net_drain = complete_per_s - arrival_per_s
assert net_drain > 0
assert backlog / net_drain == 20
print(backlog / net_drain)20.0Twenty seconds follows from the supplied assumptions only and is not a broker performance measurement.
Connect the metric to one bounded intervention
An incident timeline should name vhost, queue, queue type, consumer count, prefetch, publish/ack rates and the first failure or alarm. Compare aligned windows before and after a reviewed change. Do not purge a backlog or acknowledge unfinished work simply to clear a dashboard. Monitoring itself has cost, so collect the evidence needed without high-frequency broad polling or a load experiment. This lesson queries no broker or monitoring endpoint and changes no resource setting. It executes only supplied arithmetic, while actual service health, alarm behavior and destination completion remain separate operational evidence requirements.
Common errors and fixes
Low queue depth is reported as proof of completed business effects.
- Why it happens
- Early acknowledgments or downstream work were ignored.
- How to fix it
- Measure accepted destination results and business freshness separately.
- Prevent it next time
- Document the responsibility boundary for each metric.
A memory watermark is treated as a hard memory cap.
- Why it happens
- Alarm behavior was confused with physical enforcement.
- How to fix it
- Review actual memory usage, limits and alarm semantics together.
- Prevent it next time
- Preserve operational reserve rather than tuning only to silence alerts.
More workers worsen completion latency.
- Why it happens
- A shared downstream dependency became saturated.
- How to fix it
- Review sustainable end-to-end capacity and bounded admission.
- Prevent it next time
- Correlate consumer changes with destination health.
Check your understanding
1. With arrival 70/s and completion 65/s, can dividing a 300-message backlog by 65 predict catch-up?
Show worked solution
No. The backlog grows by five per second under those assumptions, so there is no finite catch-up while they continue. Dividing by completion ignores new arrivals. Review input policy, sustainable capacity and dependency limits rather than presenting a mathematically incomplete estimate as a recovery promise.
2. Why can high unacked with low ready indicate a consumer-side issue?
Show worked solution
Deliveries have left ready state but remain outstanding, so processing or acknowledgment may be delayed. Inspect handler duration, external waits, prefetch and completion accounting. The metric does not by itself prove a bug: long legitimate work is another possibility, and actual business effects can already exist before acknowledgment.
Key takeaways
- Ready, unacked and unique business completion describe different stages.
- Use rates and resource evidence before removing backpressure.