QLoRA and the PEFT Library

~26 min · training, peft, qlora

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

QLoRA: LoRA on a quantized base

Quantize the base model to 4-bit (NF4 specifically — a quantization scheme designed for normally distributed weights), keep it frozen, and train LoRA adapters on top. Result: you can fine-tune a 70B model on a single 48GB GPU. The base stays in 4-bit; only the small adapters (in fp16/bf16) train.

The pieces

BitsAndBytesConfig — declares the 4-bit quant scheme.
prepare_model_for_kbit_training — freezes base, enables grad checkpointing, casts as needed.
LoraConfig — same as before.
get_peft_model — wraps the model with adapters.

The PEFT method zoo

PEFT is more than LoRA. Other methods supported: IA3, LoHa, LoKr, OFT, X-LoRA, VeRA. Most teams default to LoRA + QLoRA. The exotic ones win on specific axes (memory, expressivity), but the marginal quality difference is small for typical SFT.

Code

QLoRA setup·python

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch

bnb_cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

base = "meta-llama/Llama-3.1-8B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, quantization_config=bnb_cfg, device_map="auto")
model = prepare_model_for_kbit_training(model)

lora_cfg = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    bias="none", task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_cfg)
model.print_trainable_parameters()
# Now train with Trainer / SFTTrainer

External links

Exercise

Set up a QLoRA fine-tune on a 7B model. Compare GPU memory: full bf16 (won't fit), LoRA bf16 (might fit), QLoRA. Train for 200 steps. Save adapter. Inference: load the 4-bit base + adapter and generate. Pin a SHA for the base so the result is reproducible.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousLoRA: Low-Rank Adaptation Next →SFTTrainer, Chat Templates, and Merging

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.