Why Streams Exist

"Streams aren't about being fast. They're about not loading 4GB into memory just because you need 4KB at a time."

The Naive Way Fails Predictably

You want to count lines in a log file. The obvious code:

import { readFile } from 'node:fs/promises';
const text = await readFile('access.log', 'utf-8');
const lines = text.split('\n').length;
console.log(lines);

Works fine for a 10MB log. Try it on a 10GB log. Your Node process allocates 10GB of memory, your machine swaps, your script dies with an out-of-memory error. The fundamental problem: you asked for the whole file as a single string when you only needed to count newlines, which you could have done one byte at a time.

Streams solve this. A stream is a sequence of chunks delivered to you over time, in bounded memory. You process chunk N, then chunk N+1, without ever holding the whole input.

Three Streams You've Already Used

process.stdin — bytes coming in from the keyboard or piped stdin.
process.stdout — bytes going out to the terminal.
The body of every fetch() Response — bytes coming from the network.

You've been using streams since your first console.log. The runtime hides it because you usually don't need to think about it. When the data gets big, streams stop hiding.

The Mental Model — Faucet and Bucket

A stream is a faucet. The producer (file, network, process) drips bytes at its own rate. You decide whether to catch them in a bucket (collect everything — wrong for big inputs), process each drop and discard (constant memory), or send them to another faucet (pipe). The shape of stream processing is: read a chunk, process it, write it elsewhere, repeat until no more chunks. Total memory used = one chunk at a time. That's the only memory budget that scales.

When Streams Are Wrong

Streams add complexity. For small inputs — config files, small JSON payloads, anything you're certain fits in memory — just readFile the whole thing and parse it. Streams pay off when:

The input can be larger than RAM.
You want to start producing output before the input is fully read (latency win).
The input is infinite or open-ended (live tail, network socket, stdin pipe).
You're chaining transforms — gzip-decompress, then parse, then filter, then re-encode — where each step is naturally streaming.

For everything else, the simple read-then-process pattern is fine and more readable.

The Two Levels of Streams

Node has had streams since v0.x; the API has evolved through three major versions. In 2026 you'll encounter two stream interfaces:

Node Streams (node:stream) — the original. Readable, Writable, Duplex, Transform. Event-based (data, end, error). Most of fs and http use these.
Web Streams (built-in, no import) — newer, spec-aligned with browsers. ReadableStream, WritableStream, TransformStream. Used by fetch response bodies, files via blob.stream(), etc.

Both work; the two ecosystems are slowly converging. Modern Node provides Readable.toWeb() and Readable.fromWeb() for conversion.

Pippa's Confession

My first "do something with a big file" code crashed with an OOM on Dad's office Mac. I'd written await readFile(...) for a 4GB AI training log. Dad pointed at the swap meter on the menu bar — pegged red. "It's not Node's fault. You asked for the whole thing." I rewrote it as a stream, processed chunk by chunk, peak memory under 100MB. The lesson stuck: always ask "will this fit in RAM?" before reaching for readFile. If the answer is "probably, but I'm not 100% sure," stream it.

Code

Two ways to count lines — one of them dies on big files·javascript

// Naive — fails on large files
import { readFile } from 'node:fs/promises';
const raw = await readFile('huge.log', 'utf-8');     // ← loads entire file
const lines = raw.split('\n').length;

// Streamed — bounded memory regardless of file size
import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';

const rl = createInterface({
  input: createReadStream('huge.log'),
  crlfDelay: Infinity,
});

let lines = 0;
for await (const _line of rl) lines++;
console.log(lines);
// Works on 100GB the same as 100MB

A streaming pipeline — gzip a huge log·javascript

// A real streaming pipeline you might write
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';
import { pipeline } from 'node:stream/promises';

// Read the file, gzip it, write the result — all streaming
await pipeline(
  createReadStream('access.log'),
  createGzip(),
  createWriteStream('access.log.gz')
);

// Even on a 50GB log, Node uses bounded memory.
// Each stage processes chunks as they arrive.

Exercise

Pick a file on your machine that's at least 100MB (a video, a backup, a database dump). Write two versions of a script that counts the bytes: (1) readFile + .length, (2) streaming with for await. Compare peak memory using node --inspect or Activity Monitor / top. The streamed version's memory should stay flat; the readFile version's memory should match the file size. That's the difference made visible.

Hint

For peak memory observation: node --inspect script.js, open Chrome's chrome://inspect, take a heap snapshot during the operation. Or simpler: /usr/bin/time -l node script.js on macOS shows max RSS. The streaming version should be ~30-100MB; the readFile version should approach the file size.

The Naive Way Fails Predictably

Three Streams You've Already Used

The Mental Model — Faucet and Bucket

When Streams Are Wrong

The Two Levels of Streams

Pippa's Confession

Code

External links

Exercise

Progress

Comments 0