Tokenizer Surprises: CJK and Code

Same meaning, very different cost

CJK languages — Korean, Japanese, Chinese — typically cost 2-3x more tokens per equivalent semantic content than English. The reason is mundane: most BPE tokenizers were trained on English-heavy corpora, so CJK characters get fragmented more aggressively. A single Hangul syllable can fragment into 2-4 tokens.

For a bilingual product, that asymmetry compounds across long sessions: your Korean users effectively pay 2-3x the API bill for the same conversation length. Either pre-budget for it or use a tokenizer-aware pricing model. Pretending it does not exist is the common, expensive mistake.

Code is expensive too

Source code is denser than prose because the tokenizer rarely sees the same identifier twice. A function name like handleAuthenticationCallback can fragment into 4-6 tokens, repeated everywhere it is called. Long descriptive variable names burn budget at every reference site.

Practical tactics

If cost matters, prefer common, short variable names in code you send to the model — userId tokenizes more efficiently than theCurrentlyAuthenticatedUserIdentifier. Strip unnecessary whitespace and trailing comments before bulk uploads. For CJK, consider sending the original-language source plus a short English summary instead of full bilingual content twice.

CJK gotcha: if you set a token budget assuming English ratios and then run the same workflow in Korean, you will hit limits at roughly one-third the visible length. Plan for it explicitly.

Code

Measure CJK vs English·python

from tiktoken import encoding_for_model
enc = encoding_for_model("gpt-5")

en = "The model reads fragments."
ko = "모델은 조각으로 읽는다."

print(f"EN chars={len(en)} tokens={len(enc.encode(en))}")
print(f"KO chars={len(ko)} tokens={len(enc.encode(ko))}")
# KO often shows ~2-3x tokens per character vs EN.

Identifier-cost example·python

import tiktoken
enc = tiktoken.encoding_for_model("gpt-5")
short = "userId"
long = "theCurrentlyAuthenticatedUserIdentifier"
for name in [short, long]:
    print(name, "->", len(enc.encode(name)), "tokens")

Exercise

Take a 200-word essay in English and the same 200 words translated into Korean (or any CJK language you use). Count both. Compute the per-character token ratio for each. Use the result to set a CJK-aware budget for one of your real workflows.

Hint

If your budget assumed 4 chars/token and your real content is 1.5 chars/token, your real ceiling is ~38% of what you thought.

Same meaning, very different cost

Code is expensive too

Practical tactics

Code

External links

Exercise

Progress

Comments 0