TypeScript Streaming with fetch

The async generator pattern

In Node 20+ and modern browsers, the ReadableStream from fetch().body + TextDecoder + a manual newline-buffer is the canonical way to consume Ollama's NDJSON. Wrapping it in an async function* generator turns the stream into something you can iterate with for await.

The wrapping pays for itself because the consumer then knows nothing about HTTP or buffers. Printing to a terminal, appending into a DOM node, piping into another stream — all of them are the same for await. It's the same slot Python's iter_lines() pattern occupies, which is why the two clients end up shaped alike.

The buffer pattern (don't skip this)

Network chunks don't align with JSON lines. A single read() might return:

'{"a":1,"b":2}\n{"c":3,' — last line is incomplete
'4}\n{"e":5}\n' — first part finishes the previous line

So the order is fixed: accumulate into a buffer, split on \n, parse every complete line, and keep the partial tail in the buffer. Dropping that last step is the common mistake, and it hides well — a short answer arrives in a single read, so no tail is ever left over and the bug sails through your test. Then answers get longer and tokens start disappearing silently. This is the single most-bug-prone piece of TypeScript Ollama clients.

Code

Async generator with proper buffer handling·typescript

type Msg = { role: "system" | "user" | "assistant" | "tool"; content: string };

async function* streamChat(
  model: string,
  messages: Msg[],
  baseUrl = "http://localhost:11434",
): AsyncGenerator<{ content?: string; done?: boolean; raw?: any }> {
  const r = await fetch(`${baseUrl}/api/chat`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ model, messages, stream: true }),
  });
  if (!r.ok || !r.body) throw new Error(`Ollama ${r.status}`);

  const reader = r.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split("\n");
    buffer = lines.pop() ?? ""; // keep incomplete tail

    for (const line of lines) {
      if (!line.trim()) continue;
      const chunk = JSON.parse(line);
      if (chunk.done) {
        yield { done: true, raw: chunk };
        return;
      }
      yield { content: chunk.message?.content ?? "", raw: chunk };
    }
  }
}

// Usage
for await (const tok of streamChat(
  "qwen2.5:7b",
  [{ role: "user", content: "Explain GGUF in 3 sentences." }],
)) {
  if (tok.done) {
    const r = tok.raw;
    const tps = r.eval_count / (r.eval_duration / 1e9);
    console.log(`\n[${r.eval_count} tokens @ ${tps.toFixed(1)} tok/s]`);
    break;
  }
  process.stdout.write(tok.content ?? "");
}

Browser version (same skeleton)·typescript

// Same code works in browsers when you've configured CORS or are proxying through your own backend.
// The only difference: use textContent instead of process.stdout, and append into a DOM node.

async function streamIntoDOM(target: HTMLElement, model: string, messages: Msg[]) {
  for await (const tok of streamChat(model, messages)) {
    if (tok.done) break;
    target.textContent += tok.content ?? "";
  }
}

The async generator pattern

The buffer pattern (don't skip this)

Code

External links

Exercise

Progress

Comments 0