Why the tokenizer is not an afterthought
A model never sees text — it sees integers. The tokenizer is the bridge: it chops a string into sub-word pieces and maps each to an ID from a fixed vocabulary the model was trained on. That last clause is the whole story. The vocabulary is baked into the weights, so the tokenizer and the model are a married pair — swap one and the IDs point at the wrong embeddings, and you get fluent-looking garbage with no error to warn you.
Three algorithms, one job
Different model families tokenize differently: WordPieceTokenizer (BERT), BytePairTokenizer / BPE (GPT family), and SentencePieceTokenizer (T5, Llama, Gemma). They all solve the same problem — represent any string, including words the vocabulary has never seen, by falling back to smaller pieces — they just split differently. In modern KerasHub these run pure-Python with no TensorFlow dependency, so a tokenizer is no longer a reason to drag the whole TF stack into a torch or JAX project.
The practical upshot (see the Code section): you almost never construct a tokenizer by hand. You load a Preprocessor.from_preset(), and the matched tokenizer rides along inside it. Building one from a raw vocabulary= is for the rare case where you're training a brand-new vocab from scratch.