Loading Models & Tokenizers

~22 min · model-loading, quantization, chat-template

Level 0Observer

0 XP0/43 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Loading a base model

Two main paths — full precision (fp16/bf16) for full fine-tuning or LoRA, and 4-bit quantized for QLoRA.

Tokenizer setup

The tokenizer is just as important as the model. Two non-obvious things you must do:

Set a padding token: many base tokenizers don't have one. Use tokenizer.pad_token = tokenizer.eos_token as a sane default.
Use apply_chat_template(): every modern model ships its own chat template (Jinja2). Always format with apply_chat_template instead of hand-crafting the prompt — it handles special tokens, system messages, and model-specific formatting correctly.

Code

Standard fp16 / bf16 loading·python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "meta-llama/Llama-3.1-8B-Instruct"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="flash_attention_2",  # if available
)

4-bit QLoRA loading·python

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

Tokenizer + chat template·python

tokenizer = AutoTokenizer.from_pretrained(model_name)

# Important: set padding token (many models don't ship one)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# Use the model's built-in chat template
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"},
]
formatted = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
print(formatted)
# <|begin_of_text|><|start_header_id|>system<|end_header_id|>
# You are a helpful assistant.<|eot_id|>...

External links

Exercise

Load Llama 3.1 8B Instruct in fp16 and in 4-bit. Compare VRAM usage with nvidia-smi. Print the chat template, then format a 3-message conversation with apply_chat_template. Verify the output has the right special tokens (<|begin_of_text|>, <|start_header_id|>, etc.).