Diagnose duplicate jobs, uncertain writes and drain failures

Understand the likely causes

• Common causes include effect commit before lost acknowledgment, process-local deduplication, queue job cleanup that removed an assumed replay guard, duplicate schedulers across replicas, unbounded retries, or releasing dependencies before accepted work drains. Delivery attempt IDs and logical operation IDs are not interchangeable.

Build one operation timeline from safe evidence

Collect the logical operation ID, payload fingerprint/version, tenant scope, attempt identities and source outcome without logging credentials or full private payloads. Separate enqueue acceptance, worker start, effect commit, result recording and acknowledgment. A failure between commit and acknowledgment predicts replay even when the first attempt did useful work. Do not combine unrelated job IDs into one timeline simply because their display names match. Confirm whether the output belongs to this operation and current scope before advising the client to retry or declaring a duplicate caused by the broker.

Review durable identity and the actual retention window

Bind the operation key to action, principal/tenant scope and canonical input. Reusing it with changed input must not silently return an unrelated outcome. Check whether deduplication survives restarts and concurrent workers and whether queue cleanup removes its protection. Business replay retention may need to outlast the queue’s job record. A source transaction should coordinate uniqueness, effect and recorded result under the chosen storage protocol. External effects still need their own idempotency or reconciliation. Do not repair the incident with an in-memory seen set and call it durable exactly-once processing.

Expose the commit-before-acknowledgment case locally

The local model records a synthetic committed outcome, then simulates a retry that retrieves that outcome without incrementing the effect. It deliberately does not emulate a crash, durable store, broker lock or concurrency. Its purpose is to make the desired distinction explicit: an attempt can lack acknowledgment while the logical operation already succeeded. A real implementation needs atomic source rules and retention. Test changed-input conflicts, simultaneous duplicates and restart recovery in a separately authorized isolated environment rather than extrapolating from this single-process map.

const outcomes=new Map(); let effects=0;
function attempt(operation) {
  if (outcomes.has(operation)) return outcomes.get(operation);
  const result={revision:++effects}; outcomes.set(operation,result); return result;
}
const committed=attempt("op-17");
const replay=attempt("op-17");
console.assert(committed === replay && effects === 1);

Inspect scheduling, admission and cleanup ownership

Check whether every replica independently schedules the same job and whether overlapping runs have a shared operation identity. During shutdown, stop producing new work, drain or cancel accepted work under a deadline, then release source dependencies. app.close does not automatically end every independent interval, and cancellation may not reach a driver that ignores it. Preserve uncertain writes when a drain deadline expires. This guide runs only the map model; it does not terminate a process, operate a queue or test a source. Real retry and shutdown verification must use synthetic jobs in an authorized isolated deployment.

Keep it from recurring

• Use explicit operation IDs, durable replay outcomes, bounded retries and one scheduling owner. Document queue and business retention independently. Keep worker shutdown and source resource ownership in release fixtures, including old/new payload versions and restart recovery.