Decoding Strategies: Greedy, Beam, Top-k, Top-p

The model produces logits at every step. Decoding turns those logits into a chosen next token. Five common strategies, each with different bias/variance tradeoffs.

Greedy

Always pick the highest-probability token. Deterministic, fast, repetitive. Good for classification or where the answer is unambiguous; bad for creative text.

Beam search

Maintain k partial sequences ("beams") at each step. At every step, expand each beam by all possible next tokens, score each candidate, keep the top k. Better than greedy for translation and structured output, but expensive (k× compute) and tends to produce overly bland text on open-ended tasks.

Top-k sampling

Restrict the candidate set to the k highest-probability tokens, renormalize, sample. Adds randomness while filtering tail noise. k=40 is common.

Top-p (nucleus) sampling

Restrict to the smallest set of tokens whose cumulative probability exceeds p. Adapts dynamically: when one token dominates, the set is small; when uncertainty is high, more tokens enter. p=0.9 is the typical default.

Temperature

Scale logits by 1/T before softmax. T=0 is greedy. T<1 sharpens, T>1 flattens. T=0.7 is a common default for chat. (Next lesson covers temperature in detail.)

Code

Top-p sampling in 8 lines·python

def sample_top_p(logits, p=0.9, temperature=1.0):
    logits = logits / temperature
    sorted_logits, sorted_idx = logits.sort(descending=True)
    probs = F.softmax(sorted_logits, dim=-1)
    cumulative = probs.cumsum(dim=-1)
    cutoff = cumulative > p
    cutoff[..., 1:] = cutoff[..., :-1].clone()
    cutoff[..., 0] = False
    sorted_logits = sorted_logits.masked_fill(cutoff, float('-inf'))
    next_idx = sorted_idx.gather(
        -1, torch.multinomial(F.softmax(sorted_logits, dim=-1), 1)
    )
    return next_idx

Decoding Strategies: Greedy, Beam, Top-k, Top-p

Greedy

Beam search

Top-k sampling

Top-p (nucleus) sampling

Temperature

Code

External links

Exercise

Progress

Comments 0