Multilingual Tokenization and the Efficiency Gap

A tokenizer learns its merges from a training corpus. If 80% of that corpus is English, the resulting vocabulary heavily favors English subwords. The cost shows up at inference time: every non-English token tends to cost 2-4× more pieces.

What this looks like concretely

An average English word is ~1.3 tokens in cl100k_base (GPT-4). The same content in Korean averages ~3-4 tokens per syllable in the same tokenizer. So a 100-word English paragraph might be ~130 tokens, while a 100-syllable Korean paragraph could be ~350 tokens. At per-token API pricing, that is a real cost asymmetry.

Mitigations

Larger vocabularies. GPT-4o's o200k_base (200K) and Gemma 3's 262K vocab are explicit attempts to close this gap by giving multilingual subwords first-class slots.
Multilingual training data. Llama 3 pretrained on 5% non-English content; Llama 4 pushes that further with early-fusion multilingual data and explicit support for 200 languages.
Language-specific fine-tunes. Models like Solar (English/Korean), Qwen (English/Chinese), and Aya (multilingual) explicitly retrain or extend the tokenizer for their target languages.

The gap is real and shrinking, but for now: if you serve non-English users, measure your actual tokens-per-message in production rather than assuming English-equivalent costs.

Code

Measure your own token-per-language ratio·python

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")

samples = {
    "English": "The Transformer architecture uses self-attention to model long-range dependencies in sequences.",
    "Korean":  "트랜스포머 아키텍처는 시퀀스에서 장기 의존성을 모델링하기 위해 self-attention을 사용한다.",
    "Chinese": "Transformer 架构使用自注意力机制来建模序列中的长距离依赖关系。",
    "Japanese": "Transformerアーキテクチャは、シーケンス内の長距離依存関係をモデル化するために自己注意機構を使用する。",
}
for lang, text in samples.items():
    n = len(enc.encode(text))
    chars = len(text)
    print(f"{lang:>9}  {n:>4} tokens  {chars:>3} chars  {n/chars:.2f} tok/char")

Multilingual Tokenization and the Efficiency Gap

What this looks like concretely

Mitigations

Code

External links

Exercise

Progress

Comments 0