Decoder-Only: GPT to Llama, the Universal Shape

The decoder-only paradigm — causal self-attention plus next-token prediction — is the universal shape of modern language AI. Every conversational AI you can name in 2026 uses this architecture.

Lineage highlights

GPT-1 (2018). 117M params, 12 layers. Demonstrated unsupervised pretraining + supervised fine-tuning works across NLP tasks.
GPT-2 (2019). 1.5B params, 48 layers. Showed coherent multi-paragraph generation; OpenAI initially withheld weights citing misuse concerns.
GPT-3 (2020). 175B params, 96 layers, d_model=12,288. Scale unlocked in-context learning — solving new tasks from prompt examples without fine-tuning.
Llama 1 (2023). Meta's first open release. 7B-65B variants. First open-weight model competitive with proprietary frontiers.
Llama 3 (2024). 8B and 70B. 128K vocab, GQA, RoPE, SwiGLU — defined the modern open-weight architecture.
Llama 4 (2025). Scout (109B/17B active), Maverick (400B/17B active). Mixture-of-Experts, iRoPE, multimodal. Pushes 10M context.

The architectural drift from GPT-1 to Llama 4 is small. The unit cell — Pre-LN + multi-head attention + residual + Pre-LN + FFN + residual — is unchanged. What scaled was the dimensions, the data, and the post-training stack.

Code

Decoder-only is just a stack of one block type·python

class DecoderOnly(nn.Module):
    def __init__(self, vocab, d_model, n_layers, n_q_heads, n_kv_heads, d_ff):
        super().__init__()
        self.tok_emb = nn.Embedding(vocab, d_model)
        self.blocks = nn.ModuleList([
            LlamaBlock(d_model, n_q_heads, n_kv_heads, d_ff)
            for _ in range(n_layers)
        ])
        self.final_norm = RMSNorm(d_model)
    def forward(self, ids):
        x = self.tok_emb(ids)
        for blk in self.blocks:
            x = blk(x)
        x = self.final_norm(x)
        return x @ self.tok_emb.weight.T   # weight-tied LM head

# That's the entire architecture. Llama 3, Mistral, Qwen, Phi —
# they all instantiate this with different (vocab, d_model, n_layers, ...).

Decoder-Only: GPT to Llama, the Universal Shape

Lineage highlights

Code

External links

Exercise

Progress

Comments 0