Readable and Writable — The Two Foundations

"Readable produces chunks. Writable consumes chunks. Every other stream type is a remix of these two."

Readable — Where Data Comes From

A Readable stream produces a sequence of chunks. Examples you've used: fs.createReadStream('path'), process.stdin, http.IncomingMessage (the req object in a server). The producer pushes bytes (or objects, in object mode); the consumer reads them.

Modern interface — async iteration:

import { createReadStream } from 'node:fs';

const stream = createReadStream('input.txt', { encoding: 'utf-8' });
for await (const chunk of stream) {
  console.log('got', chunk.length, 'chars');
}

Legacy event interface (still common in callbacks):

stream.on('data', chunk => { /* one chunk at a time */ });
stream.on('end',  ()    => { /* no more chunks */ });
stream.on('error', err => { /* handle */ });

Writable — Where Data Goes To

A Writable stream consumes chunks. Examples: fs.createWriteStream('path'), process.stdout, http.ServerResponse (the res object). You push bytes in; the destination writes them out.

import { createWriteStream } from 'node:fs';

const out = createWriteStream('output.txt');
out.write('hello\n');
out.write('world\n');
out.end();   // flush and close

out.write(chunk) returns true if the internal buffer has room, false if you're writing faster than the destination can drain. That false return is the backpressure signal — you should stop writing until you hear 'drain'.

Object Mode vs Binary Mode

Streams default to bytes. Object mode ({ objectMode: true }) lets you pass arbitrary JavaScript values as chunks — useful when streaming records, log entries, or parsed events. Object-mode streams are slightly slower (no internal pooling), but the convenience is huge when chunks aren't bytes. Examples:

A CSV parser stream: input bytes, output objects.
An EventEmitter wrapped as a stream of events.
A database cursor wrapped as a stream of rows.

Mixing modes in one stream is forbidden — declare it at creation.

Flowing vs Paused Modes

Readable streams have two modes:

Paused — data sits in the internal buffer until you ask for it with .read(). The default.
Flowing — data is pushed at you as fast as the source provides. Triggered by attaching a 'data' listener or by calling .resume().

for await...of handles this transparently — it requests one chunk at a time, then pauses, then requests another. You don't have to think about modes if you use async iteration. The flowing/paused distinction matters mostly when you mix the legacy event API with manual flow control.

Building Your Own Readable

You rarely need to subclass, but it's the explicit way:

import { Readable } from 'node:stream';

class Counter extends Readable {
  constructor(max) {
    super({ objectMode: true });
    this.i = 0;
    this.max = max;
  }
  _read() {
    if (this.i >= this.max) this.push(null);    // null = end
    else this.push({ n: this.i++ });
  }
}

for await (const obj of new Counter(5)) {
  console.log(obj);   // { n: 0 }, { n: 1 }, ...
}

For most cases, Readable.from(iterableOrAsyncIterable) is simpler and avoids subclassing entirely.

Pippa's Confession

For a long time I thought Node streams were complicated because every tutorial showed .on('data'), .on('end'), and .pause()/.resume() dance. Dad pointed out: "That's the legacy API. The modern interface is for await (const chunk of readable) — same machine, ergonomic surface." Reading the docs through the async-iterator lens, streams stopped feeling baroque. Now I reach for streams instinctively for any large or unbounded input, and the code looks almost like the simple readFile version — just with one fewer way to crash.

Code

Readable.from() — the cheap way to make a stream·javascript

// Readable from any iterable — the easy way
import { Readable } from 'node:stream';

// From an array
const nums = Readable.from([1, 2, 3, 4, 5], { objectMode: true });
for await (const n of nums) console.log(n);

// From an async generator
async function* paginated() {
  for (let p = 1; p <= 10; p++) {
    const data = await fetchPage(p);
    yield* data;   // yield each item separately
  }
}
const stream = Readable.from(paginated(), { objectMode: true });

Manual backpressure — what pipeline does for you·javascript

// Backpressure-aware writing
import { createWriteStream } from 'node:fs';

const out = createWriteStream('out.txt');

async function pumpLots(source) {
  for await (const chunk of source) {
    if (!out.write(chunk)) {
      // Internal buffer full — wait for drain
      await new Promise(r => out.once('drain', r));
    }
  }
  out.end();
}

// Better: just use pipeline (next lesson) — it handles backpressure for you
// import { pipeline } from 'node:stream/promises';
// await pipeline(source, out);

Exercise

Build a lineStream(path) function that returns a Readable stream of lines from a file (one chunk per line, no trailing newline). Use it to filter and count: pipe it into a function that prints only lines containing 'ERROR' and tracks the total count. Compare to using readline directly. When would you prefer your own custom stream vs the readline approach?

Hint

Wrap createInterface around createReadStream(path), then Readable.from(rl, { objectMode: true }). That gives you a Readable of line objects you can pipe further. The custom-stream value comes when you want to compose this into a larger pipeline — readline alone doesn't compose into other Node streams natively.