Node.js / Advanced: reliable Node.js services / Lesson 16

Reliability and graceful shutdown

Set bounded failure policies and stop a service while accounting for accepted work and owned resources.

About 4 min readContent reviewed 2026-10-23

On this page

Lesson goals

Explain the concepts and trade-offs behind reliability and graceful shutdown.
Predict and verify both examples, including their outputs and cleanup behavior.
Diagnose the three common failures and apply the worked solutions to related inputs.

Budget the entire operation

A request needs an overall deadline, not just a separate generous timeout for every dependency and retry. Allocate that budget across connection acquisition, network work, and processing. Distinguish inactivity timeouts from total deadlines; a peer sending one byte periodically can defeat a purely idle-based policy. Propagate cancellation where supported and release resources when the request ends. Capacity limits, deadlines, and bounded queues work together: allowing an unlimited queue can consume the entire deadline before useful work starts and retain memory for callers who have already given up.

Retries need limits and semantics

The retry example retries one classified transient failure and succeeds on the second attempt. Its delay is tiny and deterministic for teaching; deployed retries normally use bounded exponential backoff with jitter and a shared deadline. Never retry all errors indiscriminately. An unavailable dependency can be made worse by synchronized retries from every instance. A circuit breaker may temporarily reject calls after sustained failures, but it needs an explicit recovery probe policy. These mechanisms complement rather than replace idempotency for operations that can change remote state.

Shutdown is a state transition

A service moves from starting to ready to draining to stopped. During draining, fail readiness so the routing layer stops assigning new requests, then stop accepting new connections and allow accepted work to finish within a deadline. Close background consumers, database pools, workers, and telemetry exporters in an intentional order. The HTTP example performs one local request, initiates a small idempotent shutdown function twice, and waits for the same closing promise. A real deployment also coordinates signal handling and its platform termination grace period; timing those transitions is an operational contract.

javascript Complete example: Shutdown is a state transition

import { setTimeout as delay } from 'node:timers/promises';
let attempts = 0;
async function dependency() {
  attempts++;
  if (attempts === 1) { const error = new Error('Temporary'); error.code = 'TRANSIENT'; throw error; }
  return 'ok';
}
async function run() {
  for (let attempt = 1; attempt <= 3; attempt++) {
    try { return await dependency(); }
    catch (error) {
      if (error.code !== 'TRANSIENT' || attempt === 3) throw error;
      await delay(1);
    }
  }
}
console.log(await run(), attempts);

Expected output

ok 2

Only TRANSIENT is retryable and the attempt count is capped. A production helper also accepts a deadline and cancellation signal.

Drain connections deliberately

server.close stops new listening work and waits for relevant connections according to the selected runtime’s HTTP behavior. Long-lived upgraded connections need their own tracking and close protocol. Set a final shutdown deadline and document what forced termination means for unfinished work. Do not call process.exit immediately after receiving SIGTERM because that can cut off responses, logs, and database cleanup. A second signal or expired deadline can trigger the documented force policy. Clean up the shutdown timer when draining finishes so the timer itself does not keep the process alive.

javascript Complete example: Drain connections deliberately

import { createServer } from 'node:http';
import { once } from 'node:events';
let ready = true;
let closing;
const server = createServer((req, res) => { res.end('ok'); });
function shutdown() {
  if (!closing) {
    ready = false;
    closing = new Promise((resolve, reject) => server.close(e => e ? reject(e) : resolve()));
  }
  return closing;
}
server.listen(0, '127.0.0.1'); await once(server, 'listening');
try {
  const response = await fetch(`http://127.0.0.1:${server.address().port}`);
  console.log(await response.text());
} finally {
  const a = shutdown(); const b = shutdown();
  console.log(ready, a === b); await a;
}
console.log('stopped');

Expected output

ok
false true
stopped

Draining changes readiness and reuses one closing promise. The single local response is consumed before the listener closes.

Exercise failure without real traffic

Use isolated tests for a slow dependency, a failing worker, duplicate shutdown calls, and a request accepted just before draining. Assert no new work is accepted after the transition and that resources close on both normal and error paths. Run capacity experiments only in an authorized controlled environment with bounded inputs. Recovery planning includes external restart, data reconciliation, and rollback, not only catching exceptions. Publish clear readiness and liveness semantics so an orchestrator does not restart a merely busy process in a loop or send traffic to a process that cannot serve it.

Common errors and fixes

SIGTERM cuts off responses

Why it happens: The signal handler exited before draining resources.
How to fix it: Transition readiness, stop admission, drain, then exit.
Prevent it next time: Test shutdown with an in-flight request.

Retries worsen an outage

Why it happens: Unbounded synchronized retries amplify dependency load.
How to fix it: Cap attempts and use backoff, jitter, and an overall deadline.
Prevent it next time: Monitor attempts per original request.

The process remains alive after closing HTTP

Why it happens: A timer, worker, pool, or upgraded connection is still owned.
How to fix it: Track and close every long-lived resource.
Prevent it next time: Keep an explicit resource lifecycle inventory.

Check your understanding

1. A platform gives 30 seconds before force termination. May every retry have a 30-second timeout?

Show worked solution

No. Reserve time for draining and cleanup, then fit all remaining attempts inside one smaller total budget. Separate per-attempt limits cannot exceed the overall deadline. Stop scheduling new attempts once shutdown begins and honor caller cancellation.

2. Two different components request shutdown at nearly the same time. What invariant should hold?