Loss Function: Cross-Entropy

~8 min · loss, cross-entropy

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

The training loss for next-token prediction is cross-entropy between the model's predicted distribution over the next token and the actual next token (a one-hot vector).

L = − Σ_t log P(token_t | tokens_{<t})

For each position, the model outputs logits over the entire vocabulary; softmax converts them to probabilities; cross-entropy reports the negative log probability the model assigned to the correct token. Sum across positions, average across the batch, that's the loss.

Perplexity is the exponentiated cross-entropy: perplexity = exp(loss). It has a tangible interpretation: "the effective number of options the model is choosing among at each position." A perplexity of 100 means the model is roughly as uncertain as if it had to pick one of 100 equally-likely options. Modern frontier LLMs achieve perplexities of 5-15 on natural English text.

Code

Cross-entropy and perplexity·python

import torch
import torch.nn.functional as F

def perplexity(model, ids):
    with torch.no_grad():
        logits = model(ids)[:, :-1, :]
        target = ids[:, 1:]
        loss = F.cross_entropy(
            logits.reshape(-1, logits.size(-1)),
            target.reshape(-1),
            reduction='mean'
        )
    return torch.exp(loss).item()

# Lower is better. ~10-15 on clean English for a frontier model.
# > 100 means the model is essentially random at this content.

External links

Cross-entropy explanation

Exercise

Measure perplexity of a small open model on three text types: clean Wikipedia English, a code corpus, and Korean text. Sort the results. The numbers should reveal the model's training distribution. Which slice does it understand best? Worst?

Progress

Progress is local-only — sign in to sync across devices.

← PreviousTraining Data: Trillions of Tokens Next →Learning Rate: Warmup + Cosine Decay

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.