Language Models Are Probability Samplers

What an LLM Actually Computes

Every time a language model produces a token, it's not "deciding what to say." It's:

Compute a probability distribution over the entire vocabulary (50K+ tokens).
Optionally apply temperature, top-k, top-p adjustments.
Sample one token from the resulting distribution.
Append the sampled token to the context. Repeat.

That's it. That's the whole inference loop. Everything fancy in modern LLMs (chain-of-thought, agentic behavior, tool use) emerges from this loop running on top of a vast learned distribution.

Sampling Knobs

temperature $T$ : divides logits before softmax. $T = 0$ → argmax (deterministic). $T = 1$ → model's natural distribution. $T > 1$ → flatter, more random. $T < 1$ → sharper, more confident.
top-k: keep only the top k highest-probability tokens, zero out the rest, renormalize.
top-p (nucleus): keep the smallest set of tokens whose cumulative probability ≥ p, zero out the rest, renormalize.

Why I Sound Different at Different Temperatures

Same model, same prompt — but at $T = 0$ I'm boring and predictable, at $T = 0.7$ I'm conversational, at $T = 1.5$ I might say something delightful or completely off the rails. The model's underlying distribution doesn't change; what changes is how aggressively the sampler picks from the tails.

Every LLM token is a sample. Tone, creativity, "personality" are properties of the sampling parameters as much as the model itself.

Track Reward

The world is stochastic; AI is built on top of that fact. You sample, language models sample, your decisions sample. Bayes is how to update beliefs as new samples roll in. From here on, when you see "softmax," "sampling," or "temperature," you'll see the probability machine underneath.

Code

Temperature in action·python

import numpy as np

# Toy logits over a 5-token vocabulary
logits = np.array([2.5, 1.0, 0.5, -0.5, -1.5])

def sample_with_temperature(logits, T):
    scaled = logits / T
    probs = np.exp(scaled - scaled.max())
    probs = probs / probs.sum()
    return np.random.choice(len(probs), p=probs)

np.random.seed(0)
for T in [0.1, 1.0, 2.0]:
    picks = [sample_with_temperature(logits, T) for _ in range(20)]
    print(f"T={T}: {picks}")
# Low T: almost always 0. High T: spread across the vocabulary.

Exercise

Use the snippet above. Plot a histogram of 1000 samples for T=0.1, T=1.0, T=2.0. Notice: low T concentrates on the top token; high T spreads across the vocabulary.

Hint

The shape of the histogram changes dramatically with T. This visualization is the most efficient way to internalize what temperature does.

Language Models Are Probability Samplers

What an LLM Actually Computes

Sampling Knobs

Why I Sound Different at Different Temperatures

Track Reward

Code

External links

Exercise

Progress

Comments 0