NestJS / Advanced: lifetime and distributed-work evidence / Lesson 15

Queues, replay-safe jobs and scheduled work

Define durable job identity, retry outcomes and resource ownership independently of decorators or in-process timers.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate enqueue acceptance, processing, completion and external effects.
Explain why scheduling and job IDs do not automatically prevent duplicate business work.

Enqueueing is a state transition with its own outcome

A queue integration can decouple request latency from background processing, but enqueue acceptance is not completion of the business action. Return a stable job or operation identity and define how callers inspect eventual success, failure or cancellation. Keep job payloads minimal, versioned and free of credentials. A worker must apply current domain policy appropriate to its service authority rather than retaining an entire stale HTTP principal forever. Nest provides integration patterns; the selected queue engine and store supply persistence, locking and delivery semantics that need their own versioned review.

Name identity, scope and retention explicitly

The JSON describes a synthetic export job. operationId identifies the logical work, while a queue’s delivery attempt is a separate transport detail. The payload names tenant, course and output schema version without embedding a filesystem path or secret. A real worker must verify eligibility and resolve a server-controlled output location. Define retention and duplicate behavior before choosing job-ID options; once a queue removes an old job, its former ID may no longer provide the deduplication a caller assumes. This record creates no queue and does not provision its backing service.

json Synthetic versioned export job

{"payloadVersion":1,"operationId":"export-17","tenantId":"lab-a","courseId":"c1","outputSchema":1,"attemptIdentity":"managed-separately","credentials":"not-in-payload"}

The record distinguishes business operation and delivery attempt. No job is enqueued or processed.

Make side effects safe across lost acknowledgment

A worker can commit its effect and fail before marking the job complete. Redelivery then repeats execution unless the domain effect is replay-safe. Bind operation identity to tenant, action and canonical input, and record durable outcomes under an atomic source protocol. A process-local seen set is not sufficient across restarts or workers. External delivery requires its own idempotency or reconciliation policy. Choose bounded attempts, backoff and a visible failed-work path; malformed input should not consume infinite retries while valid jobs wait. Respect shutdown and cancellation without claiming they undo a completed effect.

Verify one logical effect in a local model

The local map records the result for one logical operation and returns it again on replay. Different input under the same identity is rejected. It is synchronous and memory-only, so it cannot prove durable deduplication, lock renewal or concurrent consumer safety. Its narrow role is to make the intended semantic distinction visible before choosing queue APIs. A real implementation needs principal scope, retention and recovery after partial failure. Do not store full sensitive payloads just to simplify fingerprint comparisons; use the reviewed canonicalization and privacy policy appropriate to the application.

javascript Complete offline logical-job replay

const done=new Map(); let count=0;
function job(id,fingerprint) {
  if (done.has(id)) {
    const old=done.get(id); if (old.fingerprint !== fingerprint) throw new Error("OPERATION_CONFLICT");
    return old.result;
  }
  const result={sequence:++count}; done.set(id,{fingerprint,result}); return result;
}
console.assert(job("export-17","lab-a:c1:v1").sequence === 1);
console.assert(job("export-17","lab-a:c1:v1").sequence === 1);
console.assert(count === 1);

The identical job replay produces one local effect. No queue, lock, durable store or concurrent worker is tested.

Treat timers, schedules and worker rollout separately

An in-process timer or cron decorator can run in every application instance and does not create a globally unique durable schedule. Decide who owns scheduling, leader coordination or a shared job trigger when multiple replicas exist. Handle overlapping runs, timezone policy, missed intervals and source deadlines deliberately. Keep consumer concurrency and queue backlog bounded by actual source capacity. Verify fail/retry/drain behavior with synthetic jobs in an authorized isolated environment. This course parses the job record and runs only the replay map, with no scheduler, queue engine, backing store or real background task execution.

Common errors and fixes

Every replica runs the same scheduled effect

Why it happens: A process-local schedule was assumed globally unique.
How to fix it: Define shared trigger or coordination under a reviewed protocol.
Prevent it next time: Test overlapping instances and retained operation identity.

A retried job repeats an external effect

Why it happens: Completion acknowledgment was confused with effect atomicity.
How to fix it: Implement durable domain replay and external reconciliation.
Prevent it next time: Test effect success followed by acknowledgment loss.

Old job IDs stop deduplicating after cleanup

Why it happens: Retention semantics were omitted from the design.
How to fix it: Align business replay retention with queue-record removal.
Prevent it next time: Test replay after the documented retention window.

Check your understanding

1. Does enqueue success mean the export is ready to download?

Show worked solution

No. It confirms only the queue operation’s documented acceptance outcome. The worker may still fail, retry or be canceled. Return an operation identity and a safe status workflow, and verify completed output eligibility before download. Do not let the HTTP response imply completion that the worker has not recorded.

2. Why is a local seen set not production deduplication?