Training Data: Trillions of Tokens

~8 min · data, scale

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Modern LLM pretraining is data-bound at a scale that is hard to picture.

Model	Year	Training tokens
BERT	2018	~3.3B
GPT-3	2020	300B
Llama 1	2023	1.4T
Llama 2	2023	2T
Llama 3 (8B / 70B)	2024	15T
DeepSeek-V3	2024	14.8T
Llama 3.3 (70B)	2024	15T+

Sources include filtered Common Crawl, books, code repositories (GitHub), Wikipedia, scientific papers (arXiv, S2ORC), and a long tail of curated datasets. Quality matters as much as scale: deduplication, near-duplicate removal, language identification, toxicity filtering, and code-quality filtering all visibly improve model quality at fixed compute.

The data is the secret sauce. Two models with identical architecture, identical hyperparameters, and identical compute can produce quite different downstream behavior depending on their data mixture. This is why "what was it trained on?" is one of the questions most often dodged in model cards.

Code

Reading data scale from model cards·python

# What you're looking for in a model card:
#  - Tokens trained on (e.g., 'pretrained on 15T tokens')
#  - Data sources (Common Crawl? code? specific datasets?)
#  - Cutoff date (most recent data the model has seen)
#  - Filtering / dedup methodology
#  - Tokens-per-parameter ratio (Chinchilla check)

# Example: Llama 3 8B
tokens = 15e12
params = 8e9
print(f"tokens/param = {tokens/params:.0f}")     # ~1875
# Way past Chinchilla (~20). Aggressive over-training for inference cost.

External links

Exercise

Pick three open-weight models. From their tech reports, extract: (a) total training tokens, (b) source dataset names, (c) data cutoff date, (d) any reported deduplication/filtering steps. Make a comparison table. Which model has the best-documented data pipeline?

Progress

Progress is local-only — sign in to sync across devices.

← PreviousAutoregressive Generation: One Token at a Time Next →Loss Function: Cross-Entropy

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.