WordPiece: BERT's Tokenizer

~12 min · wordpiece, bert

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

WordPiece is Google's subword algorithm, developed for Japanese/Korean voice search in 2012 and famously used by BERT. It is BPE-shaped — bottom-up merges — but the merge criterion is different.

Instead of merging the most frequent pair, WordPiece merges the pair that maximizes the likelihood of the training data under a unigram language model. Concretely, it picks the pair (a, b) that maximizes count(ab) / (count(a) × count(b)). This tends to prefer merges that capture morphological boundaries rather than just frequent character co-occurrences.

The output marks continuation pieces with the prefix ##. So "unbelievable" becomes ['un', '##believ', '##able']. Detokenization is exact: drop the ## on continuation pieces, concatenate, and you get the original word back.

Code

WordPiece in action (BERT)·python

from transformers import BertTokenizer
tok = BertTokenizer.from_pretrained("bert-base-uncased")

words = ["unbelievable", "tokenization", "transformers", "embeddings"]
for w in words:
    print(w, "->", tok.tokenize(w))

# unbelievable  -> ['un', '##believable']
# tokenization  -> ['token', '##ization']
# transformers  -> ['transformers']     (kept whole; common)
# embeddings    -> ['em', '##bed', '##ding', '##s']

External links

Exercise

Run BertTokenizer over the same paragraph you used for the BPE exercise. Compare the token boundaries — where does WordPiece split words differently from BPE? Find at least three cases where the difference is morphologically meaningful (e.g., a prefix, a suffix, an inflection).

Progress

Progress is local-only — sign in to sync across devices.

← PreviousByte-Pair Encoding (BPE) Step by Step Next →SentencePiece: Language-Agnostic Tokenization

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.