RabbitMQ / Advanced: ordered ownership, retained replay and accepted recovery / Lesson 15

Streams, retention and offset replay

Choose retained non-destructive reading when independent replay is required and keep protocol-specific guarantees explicit.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish stream reading from destructive queue consumption.
Review retention, consumer offsets and producer deduplication scope.

A stream retains history independently of one reader

RabbitMQ Streams are persistent replicated append-only data structures with non-destructive consumption. Independent consumers can read retained messages and replay them until retention removes the relevant history. This differs from a work queue where successful consumption releases the message for deletion. Streams complement queues rather than replacing every work-sharing pattern. Specify whether the application needs independent event projections, a retained audit range or competing job workers. A design selected for replay still needs business duplicate handling when an external effect is repeated, and retention must cover the promised recovery horizon rather than only the normal delay between producer and reader.

The dedicated protocol exposes stream-specific behavior

Streams can be accessed through supported RabbitMQ paths, including a dedicated protocol and clients with stream-specific capabilities. Do not assume AMQP 0-9-1 channel acknowledgments, prefetch or queue TTL map identically onto that protocol. Review client connection locality, offset APIs and feature support for the selected implementation. Super streams partition a logical stream and introduce routing and ownership decisions; ordering remains scoped to the designed partition path rather than globally across partitions. A conventional queue declaration with a different type does not transform an existing populated queue into a stream in place. Plan topology and consumer migration explicitly.

An offset is a position, not a backup

A consumer offset records progress under its client and application convention. It does not preserve messages beyond retention or prove that the downstream projection is complete. Size and age retention operate through stream storage and segment behavior, so do not promise exact per-record deletion at a single instant. Offset gaps and retained-start movement must be handled through a documented restart policy. A reader resuming from expired history needs an explicit missing-coverage decision or another source, not a silent claim of complete replay. Keep source history, stored progress and business effect acceptance as separate evidence.

Select a declared retained range offline

The local fixture has retained offsets ten through thirteen and chooses records at or beyond the declared restart position twelve. Two records are selected without removing them from the source list. This models non-destructive selection only and does not implement RabbitMQ offset tracking, segment retention or flow control. The JSON design record names a dedicated-stream-protocol reader and a recovery range. Real producer deduplication support uses its own producer identity and publishing-sequence contract; it is not arbitrary payload deduplication or a substitute for consumer-side effect safety. Verify that feature separately against the chosen official client documentation.

python Offline non-destructive replay selection

retained = [(10, 'evt-a'), (11, 'evt-b'), (12, 'evt-c'), (13, 'evt-d')]
restart_at = 12
selected = [event_id for offset, event_id in retained if offset >= restart_at]
assert selected == ['evt-c', 'evt-d']
assert len(retained) == 4
print(selected)

Expected output

['evt-c', 'evt-d']

Two supplied records are selected while all four source entries remain; no stream protocol runs.

json Synthetic stream replay contract

{"stream":"lab.parcel-history","reader_protocol":"dedicated-stream-protocol","restart_offset":12,"stop_offset_inclusive":13,"destination_policy":"stable-operation-identity"}

The range and offset convention are declared application assumptions, not a live client configuration.

Verify replay coverage and the resulting projection

A replay plan should identify stream, partitions, retained start, stopping boundary, event versions and destination duplicate policy. Compare unique operation identities and final projection state, not only the number of callbacks. Include a deletion or correction event when the business contract supports one, and verify how an old snapshot is brought current. This course starts no stream plugin or client and changes no retention setting. Its local selection and JSON fixture are safe representation checks only. Actual offset persistence, replica behavior and producer deduplication remain untested platform properties requiring separately authorized evidence.

Common errors and fixes

A stored offset is assumed to retain its source messages forever.

Why it happens: Progress metadata was confused with retention.
How to fix it: Review retained coverage and the documented missing-history policy.
Prevent it next time: Align retention with supported replay and recovery duration.

Queue acknowledgment rules are copied into a dedicated stream client unchanged.

Why it happens: Protocol-specific APIs and semantics were ignored.
How to fix it: Use the selected stream client’s official offset and flow-control contract.
Prevent it next time: Record protocol and client version with every example.

Replay repeats an external business effect.

Why it happens: Non-destructive reading was mistaken for duplicate-free effects.
How to fix it: Enforce stable operation identity at the destination boundary.
Prevent it next time: Test the same retained range twice against expected final state.

Check your understanding

1. Does selecting the same stream range twice necessarily create two independent business occurrences?

Show worked solution

No. It can be replay of the same original events. Consumers must preserve event identity and apply the destination’s duplicate-safe rule when effects are involved. A projection may intentionally recompute state, but that is an explicit application contract rather than a guarantee supplied by retaining messages.

2. Can an offset restore records already removed by retention?