Redis / Intermediate: application correctness and coordination / Lesson 11

Streams, consumer groups and replay

Use stream history and pending-entry state without confusing delivery, acknowledgment and exactly-once side effects. Define retention and recovery before consumer failure.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Explain stream IDs, consumer groups and pending entries.
Design idempotent replay and bounded history lifecycle.

A stream records ordered entries with stable identities

A Redis stream is an append-oriented structure with entry IDs and field/value content. It can support event history and consumer-group delivery, but its durability still depends on persistence and replication choices. Entry order does not establish an external event’s true business timestamp or globally synchronize every producer clock. Include the event identity and any required occurrence time explicitly. Consumer groups track delivery progress and pending entries for their consumers; they are not automatic acknowledgments that an external system completed the job. Choose one or multiple independent groups according to the service’s consumption requirements.

Separate new delivery from acknowledgment and deletion

Assume the named stream and group do not exist. The explicit entry ID keeps the expected teaching identity deterministic. Creating a group at zero allows the earlier entry to be delivered. Reading with > asks for new entries for this group, and acknowledgment removes the entry from its pending list rather than from the stream itself. XLEN therefore remains one after a successful acknowledgment. These commands are only a non-runnable sequence; no group, consumer, key or worker is created here. Existing fixture state would change the outcomes and can cause errors.

redis Illustrative single-entry group lifecycle

XADD lab:events 1-0 kind created
XGROUP CREATE lab:events workers 0
XREADGROUP GROUP workers worker-a COUNT 1 STREAMS lab:events >
XACK lab:events workers 1-0
XLEN lab:events

Expected output

Delivered ID: 1-0
Acknowledged pending entries: 1
Remaining stream length: 1

The read delivers 1-0, XACK returns 1 for that pending entry, and XLEN remains 1. Acknowledgment is not stream deletion.

Pending work needs ownership and replay rules

If a consumer fails after delivery, its entry can remain pending. XPENDING and supported claiming commands expose or transfer pending ownership according to the protocol. XAUTOCLAIM was introduced in Redis 6.2 and does not by itself prove that the original worker has stopped or that a side effect never occurred. Define idle thresholds, consumer identity and duplicate handling from the workload, not a universal copied timeout. A slow worker and a dead worker are different cases. Retrying an entry after takeover may repeat an already completed external action, so stable business idempotency remains necessary.

Model acknowledgment state without losing history

The offline dictionaries separate stream history from a group’s pending set. Acknowledging an entry removes only the pending marker in this model, preserving the original event. Real Redis group metadata, replay, trimming and deletion have more detailed versioned behavior and must be checked against the deployed command set. The simple assertion is useful because it catches an application misconception that acknowledgment equals destructive removal. It does not implement distributed delivery, persistence, ownership transfer or exactly-once processing, and it creates no actual stream.

python Offline separation of history and pending state

history = {'1-0': {'kind': 'created'}}
pending = {'1-0'}
pending.remove('1-0')  # local model of acknowledgment
assert '1-0' not in pending
assert history['1-0']['kind'] == 'created'
assert len(history) == 1

The entry remains in the history while its pending marker is gone. Only local synthetic state is checked.

Retention, lag and rebuild belong to the same design

Trimming or deleting stream history can make replay impossible or remove payloads needed by pending consumers. Newer versions add commands and options for interactions between stream entries and group references; use the exact baseline rather than assuming every deployment supports them. Monitor oldest pending work, consumer lag, retention horizon and memory together. Choose how to rebuild from an authoritative source if required history is lost. Pub/Sub is a different delivery primitive and does not replace retained stream history. A group name or high acknowledgment rate is not proof that business effects are complete or recoverable.

Common errors and fixes

An acknowledged entry is assumed deleted

Why it happens: Pending state was confused with stream history.
How to fix it: Review XACK semantics and separate cleanup/retention policy.
Prevent it next time: Test history and pending counts independently.

Claimed work produces duplicate side effects

Why it happens: Ownership transfer was treated as proof that processing never happened.
How to fix it: Use stable event identity and an idempotent effect protocol.
Prevent it next time: Test crash after effect but before acknowledgment.

Consumers cannot replay required old events

Why it happens: Retention was shorter than tolerated lag or recovery time.
How to fix it: Use the documented rebuild path and review retention capacity.
Prevent it next time: Monitor the gap between consumer progress and history horizon.

Check your understanding

1. A consumer handles an entry and crashes before XACK. What can a recovery consumer safely conclude?

Show worked solution

The entry may remain pending, but that does not prove its business effect was absent. A recovery consumer must use the event’s stable identity and the designed effect/checkpoint protocol before reapplying it. Claiming and acknowledging support delivery management; they do not atomically coordinate an external system with Redis.

2. After XACK, XLEN remains unchanged. Is that evidence acknowledgment failed?