Tokenizers In Depth: BPE, WordPiece, SentencePiece

Three families, one job

Tokenizers split text into the integer ids the model embeds. There are three families in current use:

Byte-Pair Encoding (BPE) — merges frequent byte/char pairs. Used by GPT-2, GPT-3, GPT-4, Llama, Qwen, Mistral, most modern LLMs. Operates on bytes (no UNK), handles all Unicode.
WordPiece — greedy longest-match. Used by BERT, DistilBERT. Marks subwords with ##.
SentencePiece (Unigram or BPE) — treats input as a raw byte stream including whitespace (▁ marks word boundaries). Used by T5, mBART, ALBERT, XLM-R.

What you actually need to know

Every tokenizer has a fast Rust-backed version (the default). Always use the fast variant unless a model only ships the slow Python one. Performance difference is often 10x.

You'll touch four methods 99% of the time:

tokenizer(text) — returns {input_ids, attention_mask}.
tokenizer.encode(text) / .decode(ids) — round-trip.
tokenizer.apply_chat_template(messages) — renders chat history per the model's chat template (Jinja in the tokenizer config).
tokenizer.batch_encode_plus(...) / passing a list to tokenizer(...) — batched.

Code

Inspect tokenizer behavior·python

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

text = "Hugging Face is the GitHub of AI."
ids = tok.encode(text, add_special_tokens=False)
toks = tok.convert_ids_to_tokens(ids)

print("ids:  ", ids)
print("tokens:", toks)
print("vocab size:", tok.vocab_size)
print("model_max_length:", tok.model_max_length)
print("special tokens:", tok.special_tokens_map)

Apply a chat template (the right way to build prompts)·python

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

messages = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "Explain Hugging Face in one sentence."},
]

prompt = tok.apply_chat_template(
    messages,
    tokenize=False,             # return string for inspection
    add_generation_prompt=True, # adds the assistant header
)
print(prompt)

# Same call, tokenized for the model:
input_ids = tok.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
print("shape:", input_ids.shape)

Exercise

Pick three different model families (Llama, Mistral, Qwen, T5, BERT). For each, load the tokenizer and compare on the same input string: vocab size, number of tokens for a 100-word English paragraph, special tokens, and what apply_chat_template produces. Note which families share a tokenizer family vs which are unique.