Parameter Count and Scaling

The empirical scaling laws

For language models, Kaplan et al. (2020) and Chinchilla (Hoffmann et al., 2022) showed that loss decreases as a smooth power law in parameters and tokens. Roughly: optimal training compute is split evenly between scaling parameters and scaling tokens. A 70B-parameter model trained on 1.4 trillion tokens (Chinchilla-optimal) outperforms a 175B-parameter model trained on 300 billion tokens, despite using less compute.

The lesson: bigger models alone don't help if you don't have proportionally more data. Many pre-Chinchilla 'huge model, modest data' efforts were actually under-trained.

Tip: 'How big should my model be?' is the wrong question. The right question is: 'given my compute budget, what split of parameters and tokens minimizes loss?' Chinchilla-style scaling laws give surprisingly precise answers.

What scaling actually buys you

Lower training loss — predictable, follows a power law.
Emergent capabilities — some abilities (multi-step reasoning, code generation) appear suddenly above certain scales.
Better few-shot / zero-shot performance — large models generalize from few examples.
Diminishing returns at the very top — going from 100B to 200B parameters costs much more than the accuracy gain, in most current setups.

For application work

You almost never train at frontier scale. You pick a pretrained model from the menu (3B, 7B, 13B, 32B, 70B, ...) and fine-tune. The scaling-laws intuition matters because it tells you which model size to start with — usually the smallest that meets your accuracy requirement, because inference cost scales with parameter count.

Principle: Bigger isn't always better. Pick the smallest model size that hits your accuracy target. Inference cost compounds at every request — you live with that cost forever, not once.

Code

Parameter count as a model-size signal·python

def count_params(model):
    total = sum(p.numel() for p in model.parameters())
    trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
    return total, trainable

# Common LLM sizes (parameters):
# 1.5B   - small chat / on-device
# 3B     - small with reasoning
# 7B     - sweet spot for most local fine-tuning
# 13B    - middle ground
# 32B-70B - serious fine-tune capacity
# 100B+  - frontier; usually only inference

total, trainable = count_params(model)
print(f"params: {total/1e9:.2f}B total, {trainable/1e9:.2f}B trainable")

The empirical scaling laws

What scaling actually buys you

For application work

Code

External links

Exercise

Progress

Comments 0