Temperature: How Much Should the Model Improvise?

~8 min · temperature, sampling

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Temperature rescales the model's logits before softmax, controlling how 'sharp' or 'flat' the resulting distribution is.

P(token_i) = softmax(logit_i / T)

T = 0: equivalent to greedy. The highest-logit token gets all the probability mass.
T = 0.3 – 0.7: focused but not deterministic. Good for code, math, factual QA, agentic tasks.
T = 1.0: use the raw distribution. Reasonable default for general chat.
T > 1.0: more random and creative. T=1.5 or 2.0 can sometimes break out of repetitive patterns; very high temperatures produce incoherent text.

Temperature interacts with top-p. With T=0 it doesn't matter what p is — the same token always wins. With high temperature, top-p prevents wild outliers from dominating. The two together give you a tunable balance between stability and creativity.

Code

Temperature in action·python

import torch.nn.functional as F

logits = torch.tensor([8.0, 7.0, 6.0, 5.0, 4.0])

for T in [0.1, 0.5, 1.0, 2.0]:
    probs = F.softmax(logits / T, dim=-1)
    print(f"T={T:>3}  probs = {probs.tolist()}")

# T=0.1: [0.998, 0.001, 0.000, ...]   nearly deterministic
# T=0.5: [0.844, 0.114, 0.015, ...]   focused
# T=1.0: [0.643, 0.237, 0.087, ...]   raw
# T=2.0: [0.371, 0.225, 0.137, ...]   diffuse

External links

OpenAI — temperature parameter explained

Exercise

Set up a single-prompt benchmark: 'Write a one-sentence book recommendation.' Run it 20 times each at T=0.1, 0.7, 1.0, 1.5, 2.0. Manually score the outputs for (a) coherence, (b) diversity. Plot mean and variance. Where's the productive zone?

Progress

Progress is local-only — sign in to sync across devices.

← PreviousDecoding Strategies: Greedy, Beam, Top-k, Top-p Next →Repetition Penalty and Stop Conditions

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.