QLoRA: Fine-Tune 70B on One GPU

The three innovations

QLoRA (Dettmers et al. 2023) combines LoRA with 4-bit quantization. Three key ideas:

1. NF4 (NormalFloat4) quantization

A 4-bit data type optimized for normally distributed weights. Neural network weights are typically Gaussian, so NF4 allocates quantization levels proportional to the Gaussian distribution — more precision near zero where most weights cluster.

2. Double quantization

The quantization scaling constants are themselves quantized (FP32 → FP8), saving an additional ~0.37 bits per parameter.

3. Paged optimizers

Uses CUDA unified memory to handle GPU memory spikes during gradient computation. When GPU memory fills up, optimizer states are temporarily paged to CPU RAM.

Memory savings

Model	Full FT (fp16)	LoRA (fp16)	QLoRA (4-bit)
7B	~60 GB	~18 GB	~6 GB
13B	~120 GB	~32 GB	~10 GB
70B	~600 GB	~160 GB	~36 GB

QLoRA put 7B fine-tuning on a single RTX 3090/4090 (24 GB) and 70B on a single A100 (80 GB). It democratized fine-tuning more than any other technique.

Code

QLoRA setup: bitsandbytes + PEFT·python

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # NormalFloat4
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,      # double quantization
)

# Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

# Prepare quantized model for training (enables grad on right layers,
# sets up checkpointing, ensures numeric stability)
model = prepare_model_for_kbit_training(model)

# Add LoRA adapters
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",  # recommended for QLoRA
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: ~42M || all params: ~8B || trainable%: 0.52%

QLoRA: Fine-Tune 70B on One GPU

The three innovations

1. NF4 (NormalFloat4) quantization

2. Double quantization

3. Paged optimizers

Memory savings

Code

External links

Exercise

Progress

Comments 0