Tokenizer Quirks: Why Your LLM Can't Count R's in 'strawberry'

Many "stupid" LLM failures are not stupid — they are tokenization artifacts. Once you know the failure modes, they become predictable rather than mysterious.

The strawberry problem

Ask GPT-4 "how many r's are in strawberry?" and it can get it wrong. Tokenized through cl100k_base, "strawberry" splits into ['str', 'aw', 'berry']. The model never directly sees the individual letters s, t, r, a, w, b, e, r, r, y. To answer the question it must infer the letter composition of subword pieces — a fundamentally lossy operation when the question is about character-level identity.

Other systematic quirks

Arithmetic on long numbers. "1234567" might tokenize as ['12', '345', '67']. The model doesn't see digits — it sees three subword chunks, which is why arithmetic on long numbers is unreliable.
Leading-space sensitivity. "Paris" and " Paris" (with leading space) are usually different tokens. Prompts that put words at sentence-start vs mid-sentence can hit different embeddings.
Whitespace shape in code. Two spaces vs four spaces are different tokens. Some LLMs subtly prefer 4-space Python because that's how they were trained.
CJK efficiency cliff. A Korean sentence may take 2-4× more tokens than its English translation in an English-centric tokenizer, which makes per-token API pricing meaningfully more expensive in Korean.

These quirks are not bugs — they are direct consequences of "the model sees subword pieces, not characters." Once you internalize this, you can predict which questions an LLM will be reliable on and which it will struggle with.

Code

See it for yourself with tiktoken·python

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")

for word in ["strawberry", "1234567", "Paris", " Paris"]:
    ids = enc.encode(word)
    pieces = [enc.decode([i]) for i in ids]
    print(f"{word!r:>14} -> {pieces}")

# 'strawberry' -> ['str', 'aw', 'berry']
# '1234567'    -> ['123', '456', '7']
# 'Paris'      -> ['Paris']           (no leading space)
# ' Paris'     -> [' Paris']          (different token!)

Tokenizer Quirks: Why Your LLM Can't Count R's in 'strawberry'

The strawberry problem

Other systematic quirks

Code

External links

Exercise

Progress

Comments 0