Express.js / Advanced: resource, write and release contracts / Lesson 17

Observability, health and graceful shutdown

Measure meaningful request outcomes and coordinate readiness, draining and resource cleanup under a bounded shutdown contract.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate request latency, source delay, errors and disconnects.
Model readiness and drain state without executing process signals or listeners.

Choose bounded labels and meaningful outcomes

Record a safe route identifier, method, status class, duration and application outcome without logging private bodies, tokens or full query strings. Raw URLs with object IDs create high-cardinality metrics and can reveal user data. A 2xx status alone may hide a partial domain result, while a disconnect differs from a completed error response. Measure pool queueing and source time separately when needed. Concurrent spans overlap, so adding every duration is not a valid wall-clock calculation. Preserve controlled correlation for investigation rather than turning routine telemetry into an unrestricted copy of application data.

Distinguish process liveness from serving readiness

A live process may not be ready to accept traffic, and an essential source outage does not necessarily mean restarting the process will repair it. Define liveness and readiness according to the deployment contract. Keep health responses small, public-safe and bounded; do not expose secrets or perform an expensive full database scan on every probe. The synthetic policy note below separates startup, serving and drain states without configuring an orchestrator. Readiness should become false before the application stops accepting intended new work during a coordinated shutdown.

json Synthetic lifecycle policy

{"states":["starting","serving","draining","stopped"],"readyOnlyIn":"serving","healthBody":"safe-bounded-summary","shutdown":"stop-admission-drain-cleanup-exit"}

The record is a local design artifact, not a live health endpoint, process manager setting or signal handler.

Stop admission before releasing dependencies

On a coordinated termination request, stop admitting new work, allow owned in-flight work to finish within a deadline, then release source pools and other resources. Closing a listener alone does not define what happens to long-lived streams, background jobs or upgraded connections. Give each an owner and a bounded completion or cancellation policy. Avoid accepting a request after its required source pool has already closed. If the drain deadline expires, follow an explicit forced-stop policy and preserve uncertainty for writes that may have committed. A shutdown log line cannot prove every business effect completed.

Check admission and drain invariants locally

The complete local state model accepts work only while serving. Starting drain changes readiness and admission immediately, while already-counted work can finish before stopped is reached. It is synchronous teaching logic and sends no signal or network operation. A real server needs atomic coordination for its own request lifecycle, exception-safe cleanup and a deadline. The model nevertheless makes two important invariants explicit: draining must not add new active work, and dependencies should not be declared fully stopped while accepted work remains. Test these decisions before wiring operating-system events.

javascript Complete offline drain-state model

const state={phase:"serving",active:0};
function accept() {if (state.phase !== "serving") return false; state.active++; return true;}
console.assert(accept());
state.phase="draining";
console.assert(!accept() && state.active === 1);
state.active--;
if (state.active === 0) state.phase="stopped";
console.assert(state.phase === "stopped");

One accepted item finishes; no item is accepted after drain begins. No real listener, signal or resource pool is controlled.

Verify deploy, rollback and cleanup together

A release needs compatible application and client contracts, readiness behavior, resource capacity and rollback evidence. Multiple workers may improve use of available CPU but do not make in-memory sessions or replay maps shared. Keep sticky routing assumptions explicit instead of treating them as durable state. Verify draining during safe synthetic requests and long-lived work in an authorized environment, without using production traffic as an experiment. This course parses the policy and runs only the local state model; process termination, health probes, load balancing and real resource cleanup remain unexecuted.

Common errors and fixes

A health probe leaks source credentials

Why it happens: Detailed internal diagnostics were exposed publicly.
How to fix it: Return a bounded safe summary and protect detailed telemetry.
Prevent it next time: Review exact health output keys.

New work arrives after the pool closes

Why it happens: Shutdown order released dependencies before admission stopped.
How to fix it: Coordinate readiness, admission, drain and cleanup in order.
Prevent it next time: Test accepted-work and late-arrival cases.

More workers lose session consistency

Why it happens: Process-local state was assumed shared.
How to fix it: Use reviewed shared state protocols where required.
Prevent it next time: Include multi-instance state assumptions in release review.

Check your understanding

1. Why should readiness change before dependencies close?

Show worked solution

Traffic should stop reaching a worker before it loses the resources needed for accepted requests. Drain existing work under a deadline, then release dependencies. Exact ingress propagation and connection behavior depend on deployment integration, so a local state change is not proof that upstream traffic has actually stopped.

2. Does a normal server finish event prove the user processed the result?