Base vs Instruct Models

The two starting points

For every open-weights model family, you typically have two flavors to fine-tune from.

Base models

Trained only on next-token prediction over enormous text corpora. They complete text beautifully but do not naturally follow instructions or hold a conversation. Examples: meta-llama/Llama-3.1-8B, mistralai/Mistral-7B-v0.3, Qwen/Qwen3-7B. A blank canvas.

Instruct / chat models

Base models that have been further trained with supervised fine-tuning and preference optimization (RLHF, DPO, ORPO) to follow instructions, refuse harmful requests, and engage in conversation. Examples: meta-llama/Llama-3.1-8B-Instruct, mistralai/Mistral-7B-Instruct-v0.3.

Which should you fine-tune?

Start from	Best for	Trade-off
Instruct	Most projects. You want conversation + your specialization.	Easier, preserves chat ability, recommended for beginners.
Base	Maximum control, unusual output formats, non-chat tasks (classification, extraction).	Harder, needs more data, may lose alignment if you are not careful.

The default recommendation

Start with the Instruct variant unless you have a specific reason not to. You inherit all the alignment work the model creator already did — safety filters, system-prompt awareness, multi-turn coherence — and you only need to add your specialization on top. Going from Instruct to a domain-specialized chat model is a much shorter trip than going from Base.

Code

Loading either flavor with the same call·python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Base model — blank canvas
base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Instruct model — already chat-tuned
chat = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# The chat-tuned tokenizer ships a chat_template; the Base does NOT.
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
print(tok.chat_template[:200])  # Jinja template that formats messages

Exercise

Pick one model family (Llama 3.1, Mistral, Qwen 3, Gemma 3 — your choice). Read both the Base and Instruct model cards on the Hub. Write a one-paragraph decision: for your most likely fine-tuning use case, which would you start from and why? Cite at least one specific element of the Instruct training that informs your choice.