Scaling laws describe how loss decreases as you scale parameters, data, and compute. Two foundational papers:
- Kaplan et al. (2020). Loss follows power laws in N (parameters), D (training tokens), and C (compute). Originally suggested allocating ~73% of compute to params, ~27% to data — leading to large, undertrained models like GPT-3.
- Chinchilla (Hoffmann et al., 2022). Updated: scale params and data together. Compute-optimal ratio: ~20 training tokens per parameter. Many older models (GPT-3 included) were severely undertrained.
What changed after Chinchilla
Production-oriented teams realized training compute is paid once but inference compute is paid forever. So they over-train smaller models for cheaper serving:
| Model | Params | Tokens | Tokens / param |
|---|---|---|---|
| GPT-3 (2020) | 175B | 300B | 1.7 (severely under-trained) |
| Chinchilla optimum | — | — | ~20 |
| Llama 2 (2023) | 7B | 2T | ~286 |
| Llama 3 (2024) | 8B | 15T | ~1,875 |
| Llama 3.1 (2024) | 70B | 15T | ~214 |
Llama 3 8B at 1,875 tokens/param is ~94× past Chinchilla-optimal. The "wasted" training compute is recouped many times over by the cheaper inference of a smaller model serving billions of tokens.