Encoder, Decoder, Encoder-Decoder — Three Shapes, Three Roles

The 2017 Transformer was an encoder-decoder. By 2024 the field has cleanly split into three shapes, and almost all frontier work uses just one of them.

Shape	Attention pattern	Pre-train objective	Examples	Best for
Encoder-only	Bidirectional (every token sees every other)	Masked LM (predict hidden tokens)	BERT, RoBERTa, DeBERTa	Classification, NER, retrieval embeddings
Decoder-only	Causal (each token sees only previous)	Next-token prediction	GPT-4/5, LLaMA 4, Claude, Mistral	Generation, chat, code, agents
Encoder-decoder	Bidirectional encoder + causal decoder + cross-attention	Span corruption / denoising	T5, BART, Whisper, NLLB	Translation, summarization, ASR

Why decoder-only won the LLM race

Three reasons. First, generation is built in: a model trained to predict the next token can generate by sampling. No separate decoder pretraining needed. Second, you can do classification or retrieval as generation tasks too, just by structuring the prompt. Third, the architecture is simpler — no encoder/decoder split, no cross-attention sub-layer, fewer hyperparameters to retune.

Encoder-only models did not vanish. BGE, E5, and other dense retrieval embeddings used inside RAG systems are encoder-only. But for "the model you talk to," decoder-only is the universal answer.

Code

Causal mask — what makes decoder-only 'decoder-only'·python

import torch
n = 6
mask = torch.triu(torch.ones(n, n), diagonal=1).bool()
# True where we should NOT attend (future positions)
# Position 0 sees nothing future; position n-1 sees everything past + itself.

scores = Q @ K.transpose(-2, -1) / (d_k ** 0.5)
scores = scores.masked_fill(mask, float('-inf'))
weights = torch.softmax(scores, dim=-1)
out = weights @ V

Encoder, Decoder, Encoder-Decoder — Three Shapes, Three Roles

Why decoder-only won the LLM race

Code

External links

Exercise

Progress

Comments 0