The LLaMA Family — The Open-Weight Backbone of the Era

Why Llama matters disproportionately

Meta's Llama series is not the most capable family on every benchmark, but it is the most influential dense backbone in the open-weight world. A huge fraction of fine-tuned models, RAG systems, and agent stacks in 2024–2026 are built on a Llama derivative. Picking it apart is how you understand the dense paradigm in practice, not in theory.

Llama 3 (April 2024)

Introduced 8B and 70B with Grouped Query Attention (GQA, 8 KV heads), 128K vocabulary, RoPE positional encoding, and SwiGLU activations. The 70B in particular became the de facto reference for "competent open-weight 70B-class dense" — the thing every smaller fine-tune wanted to match.

Llama 3.1 (July 2024)

Added 405B — one of the largest openly available dense models ever released — and pushed context to 128K across the family. The 405B is rarely served at scale because of its FLOP cost, but it made dense scaling laws visible at frontier size.

Llama 3.2 / 3.3

3.2 added small (1B, 3B) and multimodal (11B, 90B) variants. 3.3 ships a refined 70B-Instruct that approaches 405B-level behavior at a fraction of the compute. The 70B-Instruct model is arguably the highest-leverage dense Llama for 2025–2026 production use.

Llama 4 (April 2025) — the MoE pivot

Llama 4 is Meta's first MoE generation (Scout, Maverick, Behemoth). It does not replace Llama 3.x dense — it sits alongside it. Many practitioners still default to Llama 3.3 70B for predictable serving while Llama 4 covers the frontier-capacity slot. The dense family is not deprecated; it is differentiated.

Code

Llama 3 dense lineup at a glance·python

llama_3_dense = [
    # name,            params, layers, d_model, ctx,    attention
    ("Llama 3.2 1B",     1,    16,     2048,   "128K",  "GQA"),
    ("Llama 3.2 3B",     3,    28,     3072,   "128K",  "GQA"),
    ("Llama 3 8B",       8,    32,     4096,   "8K",    "GQA (8 KV)"),
    ("Llama 3.1 70B",    70,   80,     8192,   "128K",  "GQA (8 KV)"),
    ("Llama 3.1 405B",   405,  126,    16384,  "128K",  "GQA (8 KV)"),
    ("Llama 3.3 70B",    70,   80,     8192,   "128K",  "GQA (8 KV) + better post-training"),
]

Exercise

Open both the Llama 3.1 70B and Llama 3.3 70B model cards on Hugging Face. The architecture sections are nearly identical. Find the section that describes how 3.3 differs from 3.1 — the answer lives almost entirely on axis 2 (post-training), not axis 1. Note one specific recipe difference in your own words.