SFTTrainer, Chat Templates, and Merging

~26 min · training, trl, sft

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

SFTTrainer is the right Trainer for instruction tuning

trl.SFTTrainer wraps Trainer with conveniences for chat-format data:

Accepts datasets with a messages column directly.
Applies the tokenizer's chat template automatically.
Sets up DataCollatorForCompletionOnlyLM by default if you give it a response_template.
Plugs into PEFT for LoRA / QLoRA in two extra args.

The dataset format

Either of these works:

Conversational: each row has messages: [{"role":"user","content":...}, {"role":"assistant","content":...}].
Single-turn text: each row has a text column already containing the formatted prompt + response.

Conversational is the modern default. The trainer applies the model's chat template; you don't hand-format strings.

Code

SFTTrainer + LoRA + chat dataset·python

from trl import SFTConfig, SFTTrainer
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import LoraConfig
import torch

ds = load_dataset("HuggingFaceH4/no_robots", split="train")  # conversational

base = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16)

lora_cfg = LoraConfig(
    r=16, lora_alpha=32,
    target_modules="all-linear",
    bias="none", task_type="CAUSAL_LM",
)

cfg = SFTConfig(
    output_dir="./sft-out",
    num_train_epochs=1,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=20,
    save_steps=500,
    max_seq_length=2048,
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tok,
    args=cfg,
    train_dataset=ds,
    peft_config=lora_cfg,
)
trainer.train()
trainer.save_model("./sft-adapter")

Push merged model + tokenizer + adapter to Hub·python

from peft import AutoPeftModelForCausalLM

m = AutoPeftModelForCausalLM.from_pretrained("./sft-adapter")
merged = m.merge_and_unload()
merged.save_pretrained("./qwen-sft-merged")
tok.save_pretrained("./qwen-sft-merged")

# Push (private)
merged.push_to_hub("yourname/qwen-sft", private=True)
tok.push_to_hub("yourname/qwen-sft", private=True)

External links

Exercise

Take a small chat dataset (HuggingFaceH4/no_robots or your own JSONL). SFT a 1B-3B model with SFTTrainer + LoRA. Save adapter. Test inference: prompt the unmerged model (base + adapter), then the merged model. Outputs should be similar.