Character vs Word vs Subword: Three Strategies, One Winner

There are three fundamental ways to split text. Each has been tried at scale, and the winner is decisive.

Character-level

One token per character (or per UTF-8 byte). Tiny vocabulary (~256 for bytes), no out-of-vocabulary problem ever, but sequences become very long. The word "tokenization" becomes 12 tokens. Your context window evaporates and your model has to learn from scratch that 't', 'h', 'e' tend to mean "the." Used by ByT5 and a few research models, but rarely in production.

Word-level

One token per whitespace-separated word. Sequences are short, but vocabulary explodes — English alone has 100K+ inflected forms, and you must invent an [UNK] token for everything you didn't see during training. Misspellings, named entities, and code identifiers all collapse to [UNK]. Used pre-2018; effectively dead now.

Subword

One token per sub-word fragment, with common words kept whole and rare words split. Vocabulary stays bounded (30K-200K), sequences stay short, and there is never an OOV — anything can be encoded as a sequence of fragments. Every modern LLM uses subword tokenization; the only debates are which algorithm (BPE, WordPiece, Unigram, BBPE) and how big the vocabulary should be.

Strategy	Vocab	Seq length	OOV?	Use today
Character / byte	~256	Very long	Never	Niche (ByT5, Charformer)
Word	100K+	Short	Yes	Effectively retired
Subword	30K-200K	Moderate	Never	All modern LLMs

Code

Same string, three tokenizations·python

text = "transformer-based tokenization"

# 1. Character-level (UTF-8 bytes)
chars = list(text.encode('utf-8'))
print(len(chars))   # ~30 tokens

# 2. Word-level (whitespace + punctuation)
import re
words = re.findall(r"\w+|\S", text)
print(words)        # ['transformer', '-', 'based', 'tokenization']

# 3. Subword (BPE via tiktoken)
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")
print(enc.encode(text))  # ~5-7 tokens; common subwords kept whole

Character vs Word vs Subword: Three Strategies, One Winner

Character-level

Word-level

Subword

Code

External links

Exercise

Progress

Comments 0