Language Models Are Probability Samplers

The Model Builds a Distribution; the Decoder Chooses a Token

A language model processes the current context with a neural network and produces a logit for each vocabulary token. Softmax converts those logits into a conditional next-token distribution, and a decoding algorithm chooses a token to append.

Run the context through the network to compute next-token logits.
Optionally apply temperature, repetition penalties, and candidate restrictions such as top-k or top-p.
Choose a token using greedy argmax, sampling, beam search, or another task-appropriate rule.
Append the token to the context and run the model again.

This is the outer generation loop, not a complete explanation of inference, tool use, reasoning, or learned representations. The network computation creates the distribution; the decoder makes the final local choice.

Controls on the Distribution

Temperature $T > 0$ divides logits by $T$ . Values below 1 sharpen the distribution; values above 1 flatten it. Temperature 0 is normally handled as a separate greedy-selection path.
Top-k keeps the k highest-scoring candidates and renormalizes.
Top-p keeps the smallest highest-probability set whose cumulative probability reaches p, then renormalizes.

These controls change logits or restrict candidates, affecting diversity and stability. They do not define an entire personality. System instructions, context, model training, routing, and tool results also shape a reply.

The model creates a distribution; the decoder selects a path. Separating those layers makes clear what temperature and top-p can—and cannot—change.

Closing the Track

Probability is a grammar for uncertainty. It connects representative sampling, the distinction between probability and likelihood, Bayesian updating, and next-token decoding. When a system is called stochastic, ask who creates the distribution and which rule makes the selection.

Code

Temperature in action·python

import numpy as np

# Toy logits over a 5-token vocabulary
logits = np.array([2.5, 1.0, 0.5, -0.5, -1.5])

def sample_with_temperature(logits, T):
    scaled = logits / T
    probs = np.exp(scaled - scaled.max())
    probs = probs / probs.sum()
    return np.random.choice(len(probs), p=probs)

np.random.seed(0)
for T in [0.1, 1.0, 2.0]:
    picks = [sample_with_temperature(logits, T) for _ in range(20)]
    print(f"T={T}: {picks}")
# Low T: almost always 0. High T: spread across the vocabulary.

Exercise

Use the snippet above. Plot a histogram of 1000 samples for T=0.1, T=1.0, T=2.0. Notice: low T concentrates on the top token; high T spreads across the vocabulary.

Hint

The shape of the histogram changes dramatically with T. This visualization is the most efficient way to internalize what temperature does.

The Model Builds a Distribution; the Decoder Chooses a Token

Controls on the Distribution

Closing the Track

Code

External links

Exercise

Progress

Comments 0