Node.js / Intermediate: application boundaries / Lesson 10

Streams and backpressure

Process bounded chunks, preserve failure propagation, and respect a consumer that is slower than its producer.

About 4 min readContent reviewed 2026-10-23

On this page

Lesson goals

Explain the concepts and trade-offs behind streams and backpressure.
Predict and verify both examples, including their outputs and cleanup behavior.
Diagnose the three common failures and apply the worked solutions to related inputs.

Choose a streaming representation

A stream exposes data over time instead of requiring the whole input in memory. Readable streams produce data, writable streams consume it, and transforms do both. This helps with large files, network transfers, and incremental processing, but only if each stage avoids accumulating everything internally. A final array of all chunks would reintroduce the original memory problem. A streaming JSON-lines format can be easier to process incrementally than one enormous JSON array, but its parser must still bound line length and handle malformed records explicitly.

Chunks are transport units

A chunk is not necessarily a complete line, character, or application record. A UTF-8 character can span byte chunks. Use a suitable decoder when turning arbitrary byte chunks into text, and retain only the incomplete portion needed by a parser. Object mode changes the unit from bytes to JavaScript values; it does not make those values small or safe. Pick one consumption style per readable, such as async iteration or a pipeline, rather than mixing flowing data listeners with competing reads. Clear ownership prevents missing or duplicated processing.

Let pipeline connect the lifecycle

The first example connects a readable, an uppercase transform, and a writable using the promise-based pipeline helper. Awaiting pipeline observes completion and propagates errors through the chain. The sink collects two small demonstration chunks only to make the output visible; a large-file sink should write to its destination instead. Every custom stream callback must be invoked exactly once, including on failure. A transform that throws inside unrelated asynchronous work cannot rely on pipeline to discover an exception it never reported through its callback or destruction.

javascript Complete example: Let pipeline connect the lifecycle

import { Readable, Transform, Writable } from 'node:stream';
import { pipeline } from 'node:stream/promises';
const collected = [];
await pipeline(
  Readable.from(['ab', 'cd']),
  new Transform({ transform(chunk, encoding, done) { done(null, chunk.toString().toUpperCase()); } }),
  new Writable({ write(chunk, encoding, done) { collected.push(chunk.toString()); done(); } })
);
console.log(collected.join(''));

Expected output

ABCD

pipeline carries both chunks through the transform and resolves only after the writable completes. Collection is limited to this tiny fixture.

Respect a full writable buffer

When writable.write returns false, the producer must pause until drain before writing more. false means the internal buffer reached its threshold; it does not mean the chunk was rejected. The second example uses a tiny highWaterMark to make that signal visible deterministically and waits for drain before ending. The threshold is not a strict maximum for every allocation, and one oversized chunk still needs memory. Usually prefer pipeline over hand-written coordination, especially when cancellation and errors must race with drain. Unbounded writing turns a slow consumer into a memory outage.

javascript Complete example: Respect a full writable buffer

import { Writable } from 'node:stream';
import { once } from 'node:events';
const sink = new Writable({ highWaterMark: 1,
  write(chunk, encoding, done) { setImmediate(done); }
});
const acceptedMore = sink.write(Buffer.from('ab'));
console.log(acceptedMore);
if (!acceptedMore) await once(sink, 'drain');
const finished = once(sink, 'finish');
sink.end(); await finished;
console.log('finished');

Expected output

false
finished

The two-byte chunk crosses a one-byte threshold. Waiting for drain allows the buffer to clear before the stream is ended.

Handle failure and cancellation together

A destination failure should stop upstream work and release descriptors or sockets. pipeline provides coordinated destruction, but an HTTP response pipeline can destroy a connection before an application sends a custom error body, so choose that behavior deliberately. Pass a signal when cancellation is supported, bound per-record work, and test a failing source and a slow sink. Measure memory under representative input sizes rather than assuming the stream API guarantees bounded usage. Backpressure controls the rate between stages; it does not replace total request limits, deadlines, or concurrency limits across many simultaneous streams.

Common errors and fixes

Memory rises with a slow client

Why it happens: The producer ignored false from write.
How to fix it: Use pipeline or wait for drain with error and cancellation handling.
Prevent it next time: Test slow consumers and memory budgets.

Characters are corrupted between chunks

Why it happens: Each arbitrary byte chunk was decoded independently.
How to fix it: Use an incremental decoder or setEncoding for the readable.
Prevent it next time: Test multibyte text split at every byte boundary.

A pipeline never completes

Why it happens: A custom write or transform omitted its callback.
How to fix it: Invoke the callback once on every path.
Prevent it next time: Test success and failure with bounded deadlines.

Check your understanding

1. What does write returning false require, and was the chunk accepted?

Show worked solution

The chunk was accepted into the stream’s processing or buffer. Stop producing more until drain, while also observing error and cancellation. Retrying the same chunk would duplicate data. A pipeline normally manages this coordination for you.

2. A stream parser stores every processed row in an array. Is total memory bounded by backpressure?