LoRA: The Breakthrough

The core idea

LoRA (Low-Rank Adaptation, Hu et al. 2021) is the single most important PEFT technique in practice. Instead of updating a full weight matrix W (size d×d), LoRA decomposes the update into two small matrices:

W' = W + BA

W is the original frozen weight matrix (d × d)
B is a small matrix (d × r)
A is a small matrix (r × d)
r is the rank — typically 8, 16, 32, or 64

For a matrix with d=4096 and r=16:

Full update: 4096 × 4096 = 16.7M parameters
LoRA update: (4096 × 16) + (16 × 4096) = 131K parameters — a 128× reduction

Why it works

The key hypothesis: weight updates during fine-tuning have low intrinsic rank. The changes needed to adapt a model to a new task don't require modifying every dimension of the weight space — they can be captured in a much lower-dimensional subspace.

Initialization

Matrix A is initialized with random Gaussian values, B is initialized to zeros. So at the start of training, BA = 0, the model is identical to the pre-trained one. Training gradually learns the right update.

Code

Wrap a base model with LoRA in PEFT·python

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

config = LoraConfig(
    r=16,                          # rank — higher = more capacity
    lora_alpha=32,                 # scaling factor (alpha/r)
    target_modules="all-linear",   # apply to all linear layers
    lora_dropout=0.05,             # regularization
    bias="none",                   # don't train biases
    task_type="CAUSAL_LM",
)

model = get_peft_model(base, config)
model.print_trainable_parameters()
# trainable params: 13,631,488 || all params: 8,043,651,072 || trainable%: 0.17%

The core idea

Why it works

Initialization

Code

External links

Exercise

Progress

Comments 0