Pre-Training Objectives: Next-Token vs Masked LM

The pre-training objective decides what "learning" means for the model. Choose one objective and run trillions of tokens through it; the model that comes out shapes the entire ecosystem.

Next-Token Prediction (Causal LM)

Given a prefix, predict the next token. Loss is cross-entropy at every position simultaneously, with the causal mask preventing future-token leakage. Used by GPT, Llama, Mistral, Claude, Gemini — every modern decoder-only LLM. The objective scales naturally to generation: a model trained to predict t+1 from t can produce arbitrary continuations by sampling and appending.

Masked Language Model (MLM)

Randomly mask ~15% of tokens and train the model to predict them from the bidirectional context. Used by BERT, RoBERTa, DeBERTa. Excellent for producing rich text representations for classification and retrieval, but cannot generate text autoregressively without major architecture changes.

Why next-token won

Three reasons. (1) The objective and the deployed task are the same — generation, with no train/inference mismatch. (2) Bidirectional MLM forces a separate "decoder" to be bolted on for generation; causal LM has its decoder built-in. (3) Causal LM scales straightforwardly to multi-modal (image, audio) sequences, while MLM is tied to text-style mask-and-fill semantics. By 2022 essentially every frontier-shape model was decoder-only causal LM.

Code

Causal LM loss in PyTorch·python

import torch.nn.functional as F

def causal_lm_loss(logits, target_ids):
    # logits: (B, L, vocab_size); target_ids: (B, L)
    # Shift: position t predicts target at t+1
    pred = logits[:, :-1, :]
    target = target_ids[:, 1:]
    return F.cross_entropy(
        pred.reshape(-1, pred.size(-1)),
        target.reshape(-1),
        ignore_index=-100,
    )

# This is the entire training loss for GPT, Llama, etc.
# Combined with a causal mask in attention, training in parallel
# is mathematically equivalent to predicting one token at a time.

Exercise

Train two tiny models on the same dataset: one with causal LM (next-token) and one with MLM (mask 15%, predict). After equal compute, compare downstream task scores: text classification, fill-in-the-blank, free generation. Which task each is good at should be obvious. What does that tell you about objective selection?