Tokens Are Not Words — BPE in Sixty Seconds

The model reads fragments

A token can be a word, part of a word, punctuation, whitespace, or a piece of Unicode. Tokenizers are optimized around training-data frequency, not human intuition. English prose, code, JSON, Korean, URLs, and stack traces all spend tokens differently — sometimes by an order of magnitude.

BPE in sixty seconds

Most modern LLMs use a byte-pair encoding (BPE) variant. The recipe: start with a vocabulary of single bytes (256 symbols). Find the most frequent adjacent pair across the training corpus. Merge that pair into a new token, add it to the vocabulary. Repeat thousands of times until you hit your target vocab size (typically 50K-200K). The result: common English phrases collapse into single tokens; obscure terms or non-Latin scripts get fragmented.

Stop estimating by eye

Use the provider tokenizer when the budget matters. Rough estimates are fine for a paragraph. They are not fine for a 200K-token working session — you will be 30% off and only notice when the bill or the truncation tells you.

Token cost is empirical. Count it when the work is large enough to matter, and treat estimates as a guess until measurement confirms them.

Code

Measure with tiktoken (OpenAI / GPT family)·python

from tiktoken import encoding_for_model

enc = encoding_for_model("gpt-5")
text = open("lesson.md", "r", encoding="utf-8").read()
print(f"{len(enc.encode(text))} tokens")

Count with Anthropic count_tokens·python

from anthropic import Anthropic
client = Anthropic()

count = client.messages.count_tokens(
    model="claude-sonnet-4-7",
    messages=[{"role": "user", "content": open("lesson.md").read()}],
)
print(count.input_tokens)

Quick gut-check ratios·text

English prose:                ~4 chars / token
English code:                 ~3 chars / token
JSON / structured text:       ~3.5 chars / token
Korean / Japanese / Chinese:  ~1-1.5 chars / token  (2-3x the English cost)
Emoji / unusual Unicode:      ~0.25-0.5 chars / token

Tokens Are Not Words — BPE in Sixty Seconds

The model reads fragments

BPE in sixty seconds

Stop estimating by eye

Code

External links

Exercise

Progress

Comments 0