tiktoken and Hugging Face Tokenizers — Tools of the Trade

In production you almost never implement BPE yourself. Two libraries cover most of the field.

tiktoken (OpenAI)

Rust-backed BPE tokenizer for the OpenAI encodings: p50k_base (GPT-3), cl100k_base (GPT-4), o200k_base (GPT-4o), o200k_harmony (GPT-5). Fast — about 1M tokens/second on a single core. The Python interface is one import and one method:

Hugging Face `tokenizers`

A Rust-backed library that supports BPE, WordPiece, and Unigram via SentencePiece. It implements the full pipeline: pre-tokenization (e.g., split on whitespace), normalization (lowercase, NFKC), the model itself (BPE/WordPiece/Unigram), and post-processing (add special tokens, build chat templates). Used internally by AutoTokenizer for almost every Hugging Face model.

Speed: 10-100× faster than pure-Python tokenizers. For training, it can build a vocabulary from a corpus in minutes that would take hours in Python.

Code

tiktoken — counting tokens before an API call·python

import tiktoken

# Pick the encoding for the model you'll send to
enc = tiktoken.encoding_for_model("gpt-4o")

prompt = build_prompt()  # your prompt construction
tokens = enc.encode(prompt)
print(f"Prompt length: {len(tokens)} tokens")

# Use this BEFORE sending to estimate cost and to truncate
# safely when the prompt exceeds the model's context window.

Hugging Face tokenizers — train and load·python

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

tok = Tokenizer(BPE(unk_token="[UNK]"))
tok.pre_tokenizer = Whitespace()
trainer = BpeTrainer(vocab_size=8000,
                     special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]"])
tok.train(files=["corpus.txt"], trainer=trainer)
tok.save("my-bpe.json")

# Reload anywhere:
tok = Tokenizer.from_file("my-bpe.json")
print(tok.encode("hello world").tokens)

Exercise

Wire tiktoken into your existing OpenAI/Anthropic client wrapper so every request logs (a) prompt token count, (b) max_tokens you allowed, (c) measured response token count. After a day's traffic, look at the distribution. Are you over-budgeting context? Under-budgeting? Where are the long-tail outliers coming from?