Generating Text — Streaming, Stopping, and Stop Tokens

generate() vs stream_generate()

The two-liner from lesson 1 used generate(), which gives you the full completion as one string when it's done. For UI work — chat bubbles, terminal streaming, anything where you want tokens to appear as they're produced — you want stream_generate(). It's a Python generator that yields one chunk per token.

Both are real APIs in mlx-lm. generate() is implemented as a thin wrapper around stream_generate() that joins the chunks. Use the streaming version when latency matters and the buffering version when it doesn't.

What stream_generate yields

Each yield is a GenerationResponse object with rich metadata, not just a raw token. The fields you'll actually use are:

.text — the new text fragment for this step (often a whole token, sometimes part of a Unicode glyph being streamed in pieces).
.finish_reason — None while still generating, then 'stop' (hit a stop token) or 'length' (hit max_tokens) on the last yield.
.generation_tokens — running count of tokens generated so far.
.generation_tps — measured tokens-per-second since generation started; useful for live throughput display.
.peak_memory — peak GPU memory in bytes during this generation.

The richness of this metadata is one of mlx-lm's quiet strengths — you don't need a separate profiler to see throughput, and you don't have to guess why generation stopped.

Stop conditions, demystified

Generation stops when one of three things happens:

The model emits the EOS token for its tokenizer (e.g. <|eot_id|> for Llama 3, <|im_end|> for Qwen). The tokenizer knows its own EOS; mlx-lm picks it up from there. finish_reason='stop'.
You hit max_tokens. finish_reason='length'. The model would have kept going if you'd let it.
You manually stop by breaking out of the generator. The model is mid-sentence, no special finish_reason — just no more yields.

The trap most quickstarts don't mention: chat-tuned models depend on the EOS token to stop on every turn. If you forget to apply the chat template (lesson 5), the model often won't emit its EOS at the natural stopping point, and you'll see runaway generation that only stops when max_tokens kicks in.

Code

stream_generate — print as tokens arrive·python

from mlx_lm import load, stream_generate

model, tok = load("mlx-community/Llama-3.2-1B-Instruct-4bit")

print("Streaming: ", end="", flush=True)
for chunk in stream_generate(model, tok, prompt="Count 1 to 5:", max_tokens=30):
    print(chunk.text, end="", flush=True)
print()

# Verified output (2026-05-03):
#   Streaming: 1, 2, 3, 4, 5
#   Count 6 to 10:
#   ...

Inspect the GenerationResponse metadata on the last chunk·python

from mlx_lm import load, stream_generate

model, tok = load("mlx-community/Llama-3.2-1B-Instruct-4bit")

last = None
for chunk in stream_generate(model, tok, prompt="Hello.", max_tokens=20):
    last = chunk

print("text fragment   :", repr(last.text))
print("finish_reason   :", last.finish_reason)         # 'stop' or 'length'
print("generation_tokens:", last.generation_tokens)    # how many tokens were produced
print("generation_tps  :", round(last.generation_tps, 1), "tokens/sec")
print("peak_memory MB  :", round(last.peak_memory / 1024 / 1024, 1))

Manual early stop — break out of the generator·python

from mlx_lm import load, stream_generate

model, tok = load("mlx-community/Llama-3.2-1B-Instruct-4bit")

# Stop the moment we've seen a period
buf = ""
for chunk in stream_generate(model, tok, prompt="One sentence about MLX:", max_tokens=200):
    buf += chunk.text
    print(chunk.text, end="", flush=True)
    if "." in chunk.text:
        break
print()
print("---")
print("Stopped early. Captured:", repr(buf))

Exercise

Run the second code block (the metadata inspector) on a fresh prompt. Note the generation_tps on your machine — for the 1B Q4 model on Apple Silicon, you should be in the hundreds (likely 200–800 tps depending on chip). Then re-run with max_tokens=200 and compare — is the tps roughly the same, or higher/lower? KV cache growth makes later tokens slightly slower, so longer generations usually show a small tps drop. Two sentences on what you observed.