Parameter-Efficient Tuning: LoRA

Why LoRA exists

Full fine-tuning of a 7B-parameter LLM means storing 7B updated parameters per task — 14GB just for the weights. LoRA (Low-Rank Adaptation, Hu et al. 2021) freezes the original weights and adds tiny trainable low-rank matrices alongside each linear layer. The fine-tuned 'update' for that layer is W' = W + (BA) where A: [d, r] and B: [r, d] with r << d.

The result: maybe 0.1% extra parameters trained, comparable accuracy to full fine-tuning, and you can swap LoRA adapters in and out without reloading the base model. This is the technology that made personal LLM fine-tuning feasible.

Tip: LoRA's typical rank is r=8 to r=64. Lower rank = fewer parameters = faster + cheaper + more risk of underfitting. Higher rank = closer to full fine-tune. Start at r=16, tune from there.

QLoRA — LoRA on a quantized base

QLoRA (Dettmers et al., 2023) keeps the frozen base model in 4-bit quantized form and trains the LoRA adapters in BF16. Result: a 70B model fine-tunes on a single 80GB H100 instead of needing eight. This is what makes serious LLM fine-tuning feasible for individuals and small teams.

The PEFT library

Hugging Face's peft library implements LoRA, QLoRA, prefix tuning, prompt tuning, IA3, and friends with a uniform API. Wrap any pretrained model with get_peft_model(model, lora_config), train normally, save the adapter (small file), load anywhere.

Principle: LoRA is the modern fine-tuning default. Full fine-tuning is now reserved for cases where the LoRA accuracy gap matters and the compute is available. For everyday adaptation, LoRA wins.

Code

LoRA fine-tuning with PEFT in twelve lines·python

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import LoraConfig, get_peft_model, TaskType

base = "meta-llama/Llama-3.1-8B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16",
                                             device_map="auto")

config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],
)
model = get_peft_model(model, config)
model.print_trainable_parameters()
# trainable params: 4,194,304 || all params: 8,034,455,552 || trainable%: 0.052%

Why LoRA exists

QLoRA — LoRA on a quantized base

The PEFT library

Code

External links

Exercise

Progress

Comments 0