Why Tokenization? The Bridge from Text to Tensors

Neural networks operate on numbers — specifically on dense matrices of floating-point values. They cannot directly consume the letter "A" or the word "cat." Tokenization is the bridge: a deterministic procedure that converts raw text into a sequence of integers (token IDs) drawn from a fixed vocabulary, ready for embedding lookup.

The pipeline is three steps:

Split the text into tokens — characters, words, or subwords.
Map each token to a unique integer ID via a vocabulary table.
Embed each ID into a dense vector by indexing into the embedding matrix (Track 3).

The split step is where everything interesting happens. Choose poorly and your model wastes capacity learning that "the" and " the" should mean the same thing, or that "preprocessing" should be related to "process." Choose well and the vocabulary itself becomes a useful inductive bias.

Why this is treated as a separate stage

Tokenization is decided once, frozen into a tokenizer file, and then used identically at training and inference time. You cannot change the tokenizer of a trained model without retraining it — the embedding matrix indices would no longer mean the same things. That's why every model card publishes its tokenizer alongside the weights, and why "what tokenizer does this model use?" is a fundamental question, not a trivium.

Code

The full pipeline at a glance·python

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")

text = "Tokenization decides what the model sees."
ids = tok.encode(text, add_special_tokens=False)
print(ids[:8])         # e.g. [3404, 2065, 6276, 374, ...]
print(tok.decode(ids)) # round-trips to original text

# Embedding lookup happens inside the model:
# embedding[ids] -> (seq_len, d_model) tensor

The frozen-tokenizer invariant·python

# This is why you cannot mix-and-match tokenizers:
ids_llama = tokenizer_llama.encode("hello world")
ids_gpt   = tokenizer_gpt.encode("hello world")
# Same string, different integers, different vocab sizes.
# Feeding ids_llama into a GPT model is undefined behavior.

Exercise

Pick three open-weight models with different tokenizer families (e.g., Llama 3, Mistral 7B, Qwen 2.5). Tokenize the same English paragraph in all three and record the token counts. Then do the same with a Korean paragraph. Which tokenizer is most efficient for English? For Korean? What does that imply about per-token API pricing?

Why Tokenization? The Bridge from Text to Tensors

Why this is treated as a separate stage

Code

External links

Exercise

Progress

Comments 0