Special Tokens: Scaffolding for the Model

Special tokens are reserved IDs in the vocabulary that have structural rather than lexical meaning. They are not part of the input text the user wrote — they tell the model how to interpret the rest of the sequence.

Token	Role	Used by
[CLS]	Classification — last-layer hidden state at this position represents the whole input	BERT
[SEP]	Boundary between two segments (question / context)	BERT
[PAD]	Padding to align variable-length inputs in a batch	Almost every encoder
[MASK]	Placeholder for masked-LM training	BERT
<\|endoftext\|>	Document boundary	GPT-2/3/4
<s> / </s>	Beginning / end of sequence	Llama, T5
<\|im_start\|> / <\|im_end\|>	Chat message role boundaries (system, user, assistant)	OpenAI Harmony, Llama chat

Why this matters for you

Modern chat models use chat templates made out of special tokens. When you call tokenizer.apply_chat_template(messages) the library inserts the right special tokens around each role. Bypassing the template — concatenating "User: ..." and "Assistant: ..." manually — is a top-three source of "the model is acting weird" bugs. The model trained on tokens like <|im_start|>user, not on the literal string "User:".

Code

Always use the official chat template·python

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

messages = [
    {"role": "system",    "content": "You are a helpful assistant."},
    {"role": "user",      "content": "What is a Transformer?"},
]
prompt_ids = tok.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_tensors="pt",
)
# Right way. Library inserts <|begin_of_text|>, <|start_header_id|>system,
# <|end_header_id|>, role contents, <|eot_id|>, etc.

Exercise

Take a 3-message conversation (system + user + assistant) and tokenize it three ways: (1) using apply_chat_template for Llama 3, (2) for Mistral 7B Instruct, (3) by hand-concatenating 'system: ... user: ... assistant: ...'. Decode each back. Compare token counts and where special tokens land. The hand-concat version will look obviously wrong.

Special Tokens: Scaffolding for the Model

Why this matters for you

Code

External links

Exercise

Progress

Comments 0