The Scaling Hypothesis and What It Got Right

The scaling hypothesis is the empirical claim that — within the Transformer architecture — pouring in more parameters, more data, and more compute reliably yields more capability, in a way that is smooth and predictable rather than sporadic.

Two key papers anchor the field:

Kaplan et al. (2020). Loss decreases as a power law in model size, dataset size, and compute. The law gives quantitative guidance on how to allocate compute.
Chinchilla (Hoffmann et al., 2022). Updated Kaplan's allocation rule: scale parameters and tokens together. The compute-optimal ratio is roughly 20 training tokens per parameter. Many earlier models (GPT-3 included, with about 1.7 tokens/param) were undertrained.

The post-Chinchilla pivot

By 2024 the field had moved past Chinchilla-optimal. LLaMA 3 (8B) was trained on 15T tokens — 1,875 tokens/param, ~94× past Chinchilla. The reasoning: training cost is paid once, but inference cost is paid forever. A smaller, over-trained model is cheaper to serve at the same quality, even if its training compute is "wasted" by Chinchilla's lights.

This shift is why the open weights ecosystem now favors aggressively over-trained 7-70B models over Chinchilla-optimal 200-400B ones for most production deployments.

Code

The Chinchilla compute-optimal rule of thumb·python

# Compute (FLOPs) ≈ 6 × N × D
#   N = parameters, D = training tokens
# Compute-optimal: D ≈ 20 × N
# So if your compute budget is C FLOPs:
#   N* = sqrt(C / 120)  (approximately)
#   D* = 20 × N*

# Example: 10^22 FLOPs budget
import math
C = 1e22
N_star = math.sqrt(C / 120)         # ~ 9.1e9 = 9.1B params
D_star = 20 * N_star                 # ~ 1.8e11 = 180B tokens

The Scaling Hypothesis and What It Got Right

The post-Chinchilla pivot

Code

External links

Exercise

Progress

Comments 0