Sampling Without Mythology — temperature, top-p, repetition_penalty

The three knobs everyone has opinions about

temperature, top_p, repetition_penalty. These three sampling parameters live at the top of every chat-API call signature, accumulate Twitter folklore at a furious rate, and are mostly understood through hearsay. Let's actually look at what each one does to the logits the model produces, and when each is helpful, harmful, or just superstition.

temperature — sharpens or softens the distribution

Right before sampling, the model produces a logit (real-valued score) for every token in its vocabulary. Softmax converts those into a probability distribution. temperature divides the logits before softmax — lower temperature makes the distribution sharper (the most probable token dominates more), higher temperature makes it flatter (rarer tokens get a real shot).

temperature=0 is greedy decoding — always pick the argmax. Reproducible, sometimes repetitive, rarely surprising.
temperature=0.7–1.0 is the default zone for natural-feeling generation. Most chat APIs default here.
temperature ≥ 1.5 starts to feel creative-adjacent, then quickly becomes incoherent. Useful for brainstorming; bad for instruction-following.

top_p — truncate the distribution before sampling

top_p (also called nucleus sampling) keeps only the smallest set of tokens whose cumulative probability is at least p, then samples from that set. Setting top_p=0.95 means "sample from the top tokens that together cover 95% of the probability mass; ignore the long tail." It's a complementary lever to temperature — temperature reshapes the distribution; top_p clips it.

The combination most chat APIs default to is temperature=0.7, top_p=0.95 — moderate sharpening + a defensive trim of the worst-of-the-tail nonsense.

repetition_penalty — discourage echoing recent tokens

If the model loves a word and starts saying it on every line, repetition_penalty divides the logit of recently-emitted tokens by a small factor (typical values 1.05–1.15). Use it on small instruct models that loop; don't use it on creative writing where some repetition is intentional.

What this lesson asks you to do

Run the same prompt three times, with different sampling configurations. Notice that the temperature=0 output is reproducible, deterministic, and a little dull. Notice that the high-temperature output is creative or unhinged depending on luck. Notice that adding top_p tames the worst of the high-temperature output without flattening it back to greedy. The mythology dies when you watch it happen.

Code

Three sampling regimes — same prompt, watch the variance·python

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tok = load("mlx-community/Llama-3.2-1B-Instruct-4bit")
prompt = "Write one short imaginative sentence about clouds:"

print("--- temp=0 (greedy, reproducible) ---")
print(generate(model, tok, prompt=prompt, max_tokens=25,
               sampler=make_sampler(temp=0.0), verbose=False))

print()
print("--- temp=0.7, top_p=0.95 (the default zone) ---")
print(generate(model, tok, prompt=prompt, max_tokens=25,
               sampler=make_sampler(temp=0.7, top_p=0.95), verbose=False))

print()
print("--- temp=1.5, top_p=0.95 (creative-adjacent) ---")
print(generate(model, tok, prompt=prompt, max_tokens=25,
               sampler=make_sampler(temp=1.5, top_p=0.95), verbose=False))

# Verified outputs (2026-05-03, Llama-3.2-1B-Instruct-4bit):
#   temp=0    : "As the sun set over the rolling hills, a lone cloud drifted lazily..."  (cliche, but stable)
#   temp=0.7  : something natural-sounding, varies per run
#   temp=1.5  : "They danced on the sun-kissed rooftops in intricate waltz patterns."   (creative)

Reproducibility: seed first, then sample·python

import mlx.core as mx
from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tok = load("mlx-community/Llama-3.2-1B-Instruct-4bit")
sampler = make_sampler(temp=0.7, top_p=0.95)

mx.random.seed(123)
out1 = generate(model, tok, prompt="One sentence about Mars:", max_tokens=20, sampler=sampler, verbose=False)

mx.random.seed(123)
out2 = generate(model, tok, prompt="One sentence about Mars:", max_tokens=20, sampler=sampler, verbose=False)

assert out1 == out2, "Seeded sampling should be reproducible"
print("Seeded twice, identical output:")
print(out1)

Exercise

Run the three-regime block. Then add a fourth call with temp=2.5, top_p=1.0 (aggressive temperature, no top-p clip) and observe the output — is it creative, or just incoherent? Now add a fifth with temp=2.5, top_p=0.9 and compare. The exercise is to feel that top_p is a guardrail you can tune independently of temperature, and that very high temperatures need a tighter top_p to stay readable.