SentencePiece: Language-Agnostic Tokenization

SentencePiece is Google's tokenization library that fixes a structural assumption baked into BPE and WordPiece: that whitespace is a special boundary character. SentencePiece treats input as a raw stream of Unicode characters, with no language-specific preprocessing — whitespace is encoded as the visible character ▁ (U+2581) and learned as a regular token.

This sounds like a small detail. It isn't. It means a single tokenizer trained on multilingual data behaves identically for English (where words are space-separated), Chinese/Japanese (where they are not), Korean (where boundaries are mixed), and code (where whitespace carries syntactic meaning).

SentencePiece supports two algorithms: BPE (the bottom-up method we already saw) and Unigram. Unigram works top-down: start with a huge vocabulary and iteratively prune tokens that contribute least to the corpus likelihood until you hit the target size. Llama 1/2 and T5 use SentencePiece + Unigram; Llama 3 switched to byte-level BPE; Gemma 3 uses SentencePiece + Unigram with 262K vocabulary covering 140+ languages.

Code

SentencePiece preserves whitespace as ▁·python

import sentencepiece as spm

# Train (one-liner) — needs a small text corpus on disk first
spm.SentencePieceTrainer.train(
    input='corpus.txt', model_prefix='m', vocab_size=1000,
    model_type='unigram'
)

sp = spm.SentencePieceProcessor(model_file='m.model')
print(sp.encode_as_pieces("Hello world"))
# ['▁Hello', '▁world']

print(sp.encode_as_pieces("안녕 세상"))
# ['▁', '안녕', '▁세상']  (no language-specific assumptions)

SentencePiece: Language-Agnostic Tokenization

Code

External links

Exercise

Progress

Comments 0