Train 0.1% of a 7B model and get most of the benefit
Full fine-tuning a billion-parameter model means storing a billion gradients, a billion optimizer states, and saving a billion-parameter checkpoint. That doesn't fit on a single consumer GPU and produces giant artifacts. LoRA (Low-Rank Adaptation) sidesteps the problem.
The LoRA idea
Freeze the original weights. For each linear layer you want to adapt, add a tiny "delta" of the form ΔW = B @ A, where A is (rank, in_features) and B is (out_features, rank). With rank=8 and in/out=4096, that's 8*4096 + 4096*8 = 65,536 trainable parameters per layer — versus 4096*4096 = 16,777,216 for full fine-tune. Roughly 0.4% the size.
The adapter weights start at zero so the model behaves identically to the pretrained one at step 0. Training updates only the adapters; inference can either use them as-is (small overhead) or merge them back into the original weights (zero overhead).
The PEFT library
Hugging Face's peft library implements LoRA, AdaLoRA, IA³, prompt tuning, and more. It integrates cleanly with transformers — wrap the model, train normally, save just the adapter (a few MB).
Why this changed everything
Fine-tune a 7B-param Llama on a single 24GB consumer GPU.
Adapter checkpoints are MBs not GBs — easy to share, version, A/B test.
Stack multiple adapters for multi-task models without exploding parameter count.
The combination of "frozen base + small adapter" is what made hosted-LLM fine-tuning economically viable.
Code
Wrap a model with LoRA — minimal example·python
# pip install peft transformers
from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForSequenceClassification
base = AutoModelForSequenceClassification.from_pretrained(
"bert-base-uncased", num_labels=2,
)
lora_cfg = LoraConfig(
task_type=TaskType.SEQ_CLS,
r=8, # rank — bigger = more capacity, more params
lora_alpha=32, # scaling factor
lora_dropout=0.1,
target_modules=["query", "value"], # adapt Q and V in attention
)
model = get_peft_model(base, lora_cfg)
model.print_trainable_parameters()
# trainable params: 294,912 || all params: 109,777,410 || trainable%: 0.27%
Save and load just the adapter·python
from peft import PeftModel
from transformers import AutoModelForSequenceClassification
# After training:
model.save_pretrained("my-lora-adapter") # tiny — usually a few MB
# Load on a fresh base model
base = AutoModelForSequenceClassification.from_pretrained(
"bert-base-uncased", num_labels=2,
)
model = PeftModel.from_pretrained(base, "my-lora-adapter")
# For inference — merge the adapter into the base for zero overhead
merged = model.merge_and_unload()
# `merged` is now a vanilla transformers model with ΔW baked in
Picking target_modules — the architecture matters·python
from peft import LoraConfig
# LLaMA-family — adapt these projection names
llama_lora = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
task_type="CAUSAL_LM",
)
# BERT — query / value
bert_lora = LoraConfig(
r=8, lora_alpha=32,
target_modules=["query", "value"],
task_type="SEQ_CLS",
)
# Or, let PEFT discover all linear layers automatically
auto_lora = LoraConfig(
r=16, target_modules="all-linear",
task_type="CAUSAL_LM",
)
Wrap bert-base-uncased with a LoRA config (r=8, query/value adapters). Compare model.print_trainable_parameters() to a full fine-tune (just count len(list(base.parameters())) for params with requires_grad=True). The ratio should be roughly 1:400.
Progress
Progress is local-only — sign in to sync across devices.