Redis / Basic: keys and bounded data structures / Lesson 4

Lists, work queues and delivery

Use ordered lists for bounded work while distinguishing removal from acknowledgment. Design retry and recovery around failure windows instead of assuming exactly-once processing.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Explain list-end operations and blocking-client behavior.
Identify loss and duplicate windows in queue workflows.

Choose ends to define the queue order

Redis lists hold ordered string elements and support operations at either end. LPUSH plus RPOP can express one FIFO convention, while reversing the ends changes that convention. Equal job payloads can appear more than once because a list is not a set. Keep job identity separate from payload text. A blocking pop can wait for work without busy polling, but occupies a client connection and needs a shutdown and timeout policy. Queue length, processing capacity and retention remain application responsibilities even when individual pushes and pops are atomic.

Reason about a declared FIFO convention

Assume the synthetic list is empty. LPUSH job-a then job-b at the head leaves job-b nearest the head; RPOP removes from the tail, so job-a is returned first. The expected next removal is job-b. No Redis process or blocking consumer is started here. A successful pop means the element was removed, not that a worker completed a side effect or that a durable audit record exists. Explain this distinction before using a bare list pop for work that cannot be lost.

redis Illustrative FIFO sequence on an empty list

LPUSH lab:jobs job-a
LPUSH lab:jobs job-b
RPOP lab:jobs
RPOP lab:jobs

Expected output

job-a
job-b

Expected pop order is job-a followed by job-b. Push replies count current list elements and are not worker acknowledgments.

Removal and completion are different events

If a worker pops a job and crashes before recording completion, a bare list no longer contains the job. Moving work atomically to a processing list can retain recoverable ownership, but requires a separate acknowledgment, stale-work policy and duplicate-safe processing. LMOVE and BLMOVE are available from Redis 6.2 and help implement supported move patterns; they do not supply the complete queue protocol. In Redis Cluster, participating keys must satisfy slot constraints. External side effects can happen before an acknowledgment is persisted, so recovery can legitimately repeat a job.

Model duplicate-safe completion explicitly

This offline model records completed synthetic job identities and suppresses a second application of the same event. It demonstrates why payload delivery count and business-effect count differ. The set is only process memory here; a real implementation needs durable coordination appropriate to the effect, including atomic state transition or an idempotent downstream service. A crash between effect and completion can still break a naive two-step implementation. Use the model to state the desired invariant, not as a claim that an in-memory set solves production exactly-once delivery.

python Offline event-identity model

completed = set()
effects = []
def apply_once(job_id):
    if job_id in completed:
        return 'duplicate'
    effects.append(job_id)
    completed.add(job_id)
    return 'applied'

assert apply_once('lab-job-1') == 'applied'
assert apply_once('lab-job-1') == 'duplicate'
assert effects == ['lab-job-1']

One local effect is recorded for two calls. The example has no crash durability, external side effect or Redis transaction.

Bound backlog and choose the right primitive

An unbounded producer can consume memory faster than workers drain jobs. Define admission, retry delays, failed-job handling and alerts from the service’s capacity and loss requirements. List trimming discards elements and must not be a casual cure for a durable backlog. Streams and consumer groups provide different acknowledgment and history primitives, discussed later, but they also require idempotency and retention design. Choose whether Redis is a cache of work or an authoritative queue, then align persistence, replication, recovery and monitoring with that decision rather than relying on the word queue alone.

Common errors and fixes

A popped job disappears after worker failure

Why it happens: Removal was treated as durable completion.
How to fix it: Use a reviewed processing/acknowledgment and recovery protocol.
Prevent it next time: Test crashes before and after side effects and acknowledgment.

Jobs run in the opposite order

Why it happens: Push and pop ends do not implement the intended convention.
How to fix it: Define and test the exact end operations.
Prevent it next time: Use a two-job fixture to prove ordering explicitly.

A backlog grows until memory pressure

Why it happens: Producer rate and retry policy are not bounded by processing capacity.
How to fix it: Introduce reviewed admission/backpressure and failure handling.
Prevent it next time: Monitor oldest work and queue length with a retention policy.

Check your understanding

1. A worker sends a notification and crashes before acknowledging. What must a retry-safe design assume?

Show worked solution

The work may be delivered again even though the notification already happened. Preserve a stable event identity and use an appropriate idempotent delivery or durable coordination protocol. Merely placing the job in a processing list does not atomically coordinate Redis with the external notification service. Test that failure window explicitly.

2. Is LTRIM an acceptable automatic response to every queue backlog?