Express.js / Advanced: resource, write and release contracts / Lesson 14

Streaming, backpressure and client cancellation

Bound buffered work and own stream cleanup after partial delivery, disconnect or source failure.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Explain why streaming does not automatically bound memory or guarantee delivery.
Separate response finish, connection closure and cancellation propagation.

Streaming moves the failure boundary earlier

Streaming can deliver data incrementally instead of materializing a whole export before sending it. That can reduce peak buffering when every stage respects pressure, but headers and early bytes may commit before the full source succeeds. A later error cannot become a clean replacement JSON response with a different status. Choose a representation that can communicate truncation or provide an independent completion contract when the use case needs one. Do not treat the first chunk or a server-side finish event as proof that the user received, parsed and accepted a complete business document.

Use a reviewed pipeline and understand its HTTP consequences

Node’s pipeline coordinates stream forwarding and error cleanup, but piping directly to an HTTP response can destroy the socket on failure. Review that behavior before assuming catch can send another response. The fragment illustrates one source-to-response transfer and intentionally leaves source construction and request authorization outside. It is not executed here. Cancellation must be connected to the actual source and client-disconnect lifecycle through APIs supported by the chosen Node version and adapter. A local abort signal by itself cannot interrupt an external database driver that ignores it.

javascript Illustrative transfer fragment

async function streamExport(source,res,pipeline) {
  res.setHeader("Content-Type","text/csv; charset=utf-8");
  await pipeline(source,res);
}

pipeline and source are injected and unexecuted. A failure may have already closed the HTTP connection; this is not a recipe for sending JSON from catch.

Honor the writable signal and bound upstream work

A writable write returning false asks the producer to pause until capacity becomes available, commonly through drain. Continuing to enqueue every result defeats backpressure and can grow memory despite using streams. Also bound the source fetch window, transform buffers and per-item size; one enormous record can exceed a reasonable memory budget on its own. Object-mode highWaterMark counts objects rather than total bytes, so it is not a universal byte ceiling. Choose limits from the actual data contract and inspect every stage, not only the final response stream.

Model a local admission ceiling for pending chunks

This arithmetic model allows a pending chunk only when its declared bytes fit the remaining local budget. It rejects an oversized chunk even when nothing is queued. The teaching values are synthetic; the function neither implements Node backpressure nor measures resident memory. Real buffers carry overhead and several stages can hold data simultaneously. The model makes a useful precondition visible: limiting chunk count alone is insufficient when chunk sizes vary. Test actual stream behavior, source cancellation and cleanup in a separately authorized isolated integration before making operational claims.

javascript Complete offline pending-byte model

function fits(pendingBytes,nextBytes,limit) {
  return nextBytes >= 0 && pendingBytes + nextBytes <= limit;
}
console.assert(fits(4,6,10));
console.assert(!fits(4,7,10));
console.assert(!fits(0,11,10));

The declared budget accepts 4+6 and rejects 4+7 and an 11-byte item. This is arithmetic, not a memory benchmark.

Own disconnect and completion exactly once

A response can finish normally, close early or fail during source work. Distinguish these outcomes in bounded telemetry and make cleanup idempotent so several lifecycle signals cannot release the same resource incorrectly. Remove listeners and release cursors or file handles under the adapter’s documented ownership. For resumable exports, define cursor scope, retention and authoritative completion rather than restarting blindly after partial delivery. The course runs only the pending-byte model; no stream server, file transfer, source cursor or network disconnection is exercised. Honest scope prevents a small calculation from being mistaken for proven end-to-end delivery.

Common errors and fixes

Memory grows while responses are streamed

Why it happens: An upstream stage ignored pressure or buffered the whole dataset.
How to fix it: Bound source windows, item sizes and every transform buffer.
Prevent it next time: Inspect all stages under isolated representative fixtures.

A catch block tries JSON after socket destruction

Why it happens: Pipeline failure was treated as a pre-commit error.
How to fix it: Follow documented pipeline/HTTP failure and cleanup behavior.
Prevent it next time: Test failure before and after initial bytes separately.

Source work continues after the client leaves

Why it happens: Disconnect did not reach source cancellation.
How to fix it: Propagate supported cancellation and release owned resources.
Prevent it next time: Verify actual source interruption rather than only caller rejection.

Check your understanding

1. Why is an object-count buffer limit not a byte limit?

Show worked solution

Objects can differ greatly in size, and object-mode buffering counts entries. One entry can contain a large string or binary value. Bound individual payload size and account for concurrent buffers and overhead. A pure arithmetic ceiling helps define policy but does not measure the memory footprint of actual JavaScript objects or a running pipeline.

2. Can a client retry a partially delivered export without additional design?