LoRA: Low-Rank Adaptation

What LoRA does

Instead of updating every weight matrix W (which is huge), LoRA learns a low-rank delta ΔW = B A where B is d×r and A is r×k for a small r (typically 8-64). The original W is frozen; only B and A train. The number of trainable parameters drops by 100-1000x. Inference: W' = W + B A, applied at runtime or merged into W.

Why LoRA wins

Trainable params: 0.1-1% of full fine-tune.
VRAM: training a 7B with LoRA fits on a single 24GB GPU; full fine-tune needs 8x A100.
Composability: you can swap LoRA adapters at inference (different style per request) without reloading the base model.
Storage: a LoRA adapter is ~50MB for a 7B model. Full fine-tune is 14GB.

The hyperparameters

r (rank) — capacity. 8 for very small adaptations, 16 default, 32-64 for harder tasks.
lora_alpha — scaling factor. Effective LR scaling = alpha / r. Common: alpha = 2*r.
target_modules — which layers to adapt. q_proj, k_proj, v_proj, o_proj for attention; gate_proj, up_proj, down_proj for MLP. "all-linear" targets every Linear in the model.
lora_dropout — usually 0.05-0.1.

Code

LoRA fine-tuning with PEFT·python

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments
import torch

base = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16)

lora_cfg = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_cfg)
model.print_trainable_parameters()
# trainable params: ~5M / 1.5B  (~0.3%)

# Train as you would Trainer normally — see Lesson 0

Save adapter only, then merge for serving·python

# After training:
model.save_pretrained("./qwen-lora-adapter")

# Inference with adapter on top of base
from peft import AutoPeftModelForCausalLM
inf = AutoPeftModelForCausalLM.from_pretrained("./qwen-lora-adapter")

# Merge adapter into base weights for production serving (no PEFT runtime cost):
merged = inf.merge_and_unload()
merged.save_pretrained("./qwen-merged")

What LoRA does

Why LoRA wins

The hyperparameters

Code

External links

Exercise

Progress

Comments 0