The Llama Lineage: Open-Weight Frontier

Meta's Llama series democratized large language model research. Each release pushed forward what 'open weights' could mean.

Release	Date	Variants	Context	Notable
Llama 1	Feb 2023	7B / 13B / 33B / 65B	2K	First competitive open-weight LLM (research-only license)
Llama 2	Jul 2023	7B / 13B / 70B	4K	Commercial license, chat fine-tunes
Llama 3	Apr 2024	8B / 70B	8K → 128K	128K vocab, GQA, SwiGLU, RoPE — modern architecture template
Llama 3.1	Jul 2024	8B / 70B / 405B	128K	405B flagship matches GPT-4-class proprietary models
Llama 3.2	Sep 2024	1B / 3B / 11B vision / 90B vision	128K	Smaller text models + first vision models
Llama 3.3	Dec 2024	70B	128K	Refined post-training; 39.3M H100 GPU-hours
Llama 4 Scout	Apr 2025	109B / 17B active	10M	MoE, iRoPE, multimodal native
Llama 4 Maverick	Apr 2025	400B / 17B active	1M	128 experts + 1 shared expert
Llama 4 Behemoth	(announced)	~2T total / 288B active	—	Frontier-class teacher model (in training as of 2025)

Llama 3.3 70B's architecture: 80 layers, d_model=8192, 64 Q heads with 8 KV heads (GQA group size 8), SwiGLU activation, RMSNorm, RoPE positional encoding, 128K context window. This is the template every modern open-weight team starts from.

Code

Loading and inspecting Llama 3.3 70B·python

from transformers import AutoConfig
cfg = AutoConfig.from_pretrained("meta-llama/Llama-3.3-70B-Instruct")
for k, v in vars(cfg).items():
    if not k.startswith('_'):
        print(f"  {k} = {v}")
# d_model = 8192, num_hidden_layers = 80,
# num_attention_heads = 64, num_key_value_heads = 8,
# intermediate_size ≈ 28672, vocab_size = 128256, ...

The Llama Lineage: Open-Weight Frontier

Code

External links

Exercise

Progress

Comments 0