Chat Templates — When the Model Cares How You Wrap the Words

The single most under-appreciated detail

An instruct-tuned model was fine-tuned on conversations wrapped in a specific format — special tokens marking turn boundaries, special tokens marking system / user / assistant roles, special tokens marking end-of-turn. If you send the model a raw prompt without that wrapping, it has no idea you want a chat response. It might continue your text. It might output the wrapping itself. It might never stop generating. Most "the model is broken" complaints I see are actually "the chat template is missing."

The fix is one helper — tokenizer.apply_chat_template. The tokenizer that came back from load() already knows the right template for the model. You give it a list of role-keyed messages; it gives you back a properly-wrapped prompt string.

The shape

Input is a list of dicts with role and content:

messages = [
    {"role": "system", "content": "You are a terse assistant."},
    {"role": "user",   "content": "Capital of France?"},
]

Output (when called with tokenize=False, add_generation_prompt=True) is a single string ready to feed to generate:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a terse assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

Capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Send that to generate and the model knows it's the assistant's turn, knows the conversation context, and knows where to stop.

Multi-turn — just keep appending

For follow-up turns, append the assistant's previous response to the messages list and re-render the template. mlx-lm doesn't track conversation state for you — you maintain the message list, render fresh on each turn. That sounds like work; it's actually a feature, because it gives you full control over context window management.

What goes wrong when you skip this

Model answers your prompt but then keeps going, generating fake user / assistant turns.
Model outputs literal <|eot_id|> as text instead of stopping.
Model gives confused, non-instruction-following responses (because it's in completion mode, not chat mode).
Model never stops until max_tokens kicks in.

All four of these are the same bug — missing or wrong chat template. Always render the template before generate on instruct models.

Code

apply_chat_template — the canonical pattern·python

from mlx_lm import load, generate

model, tok = load("mlx-community/Llama-3.2-1B-Instruct-4bit")

messages = [
    {"role": "system", "content": "You are a terse assistant."},
    {"role": "user",   "content": "Capital of France?"},
]

prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

print("--- rendered prompt ---")
print(prompt)
print("--- generation ---")
print(generate(model, tok, prompt=prompt, max_tokens=20, verbose=False))

# Verified output (2026-05-03):
#   --- rendered prompt ---
#   <|begin_of_text|><|start_header_id|>system<|end_header_id|>
#
#   Cutting Knowledge Date: December 2023
#   Today Date: 03 May 2026
#
#   You are a terse assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
#
#   Capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
#
#   --- generation ---
#   Paris.

Multi-turn — append the assistant reply, re-render·python

from mlx_lm import load, generate

model, tok = load("mlx-community/Llama-3.2-1B-Instruct-4bit")

messages = [
    {"role": "system", "content": "You are a terse assistant."},
    {"role": "user",   "content": "Capital of France?"},
]

# Turn 1
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
reply  = generate(model, tok, prompt=prompt, max_tokens=20, verbose=False)
print("Turn 1:", reply)
messages.append({"role": "assistant", "content": reply})

# Turn 2 — follow-up question, with prior turns in context
messages.append({"role": "user", "content": "And of Germany?"})
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
reply  = generate(model, tok, prompt=prompt, max_tokens=20, verbose=False)
print("Turn 2:", reply)

Inspect the raw template (debugging tool)·python

from mlx_lm import load
model, tok = load("mlx-community/Llama-3.2-1B-Instruct-4bit")

# The Jinja template the tokenizer will use to render messages.
# Useful when you're debugging "why does my prompt look weird".
print("Chat template (first 600 chars):")
print(tok.chat_template[:600])

Exercise

Run the multi-turn block. Confirm both turns answer correctly. Then run the same multi-turn conversation but skip apply_chat_template entirely — pass the raw user message string directly to generate. Watch what happens (it'll either continue your text, output the system prompt, or run away). Two sentences on what you observed; this is the bug template to recognize forever.