LoRA — Why You Don't Train the Whole Model Anymore

The math that broke full fine-tuning

Fully fine-tuning a 70B model means computing gradients for 70 billion parameters, holding optimizer state for all of them (Adam adds 2× the parameter count in moments), and storing activations through the forward pass. The total memory budget is roughly 4–6× the model's inference footprint — for a 70B model that's hundreds of gigabytes of GPU memory you don't have.

LoRA (Low-Rank Adaptation) sidesteps the whole problem. You freeze every original weight and add tiny trainable adapter matrices on top of select layers. Train only the adapters; ignore the rest. The trainable parameter count drops by 99% or more, and with it the memory budget — you can fine-tune a 7B model on a 16 GB MacBook, a 70B model on the office Mac.

The decomposition trick, briefly

For a weight matrix W with shape (d × d), LoRA approximates the update ΔW as the product of two skinny matrices: ΔW ≈ A · B, where A has shape (d × r), B has shape (r × d), and r (the LoRA rank) is small — typically 8 or 16. Instead of training d² parameters, you train 2 · d · r — a fraction of the original count.

At inference, you compute (W + A · B) · x, where the original W is unchanged and the adapter A · B is the learned correction. Two matrices, one rank, that's the whole architectural change.

Why a small adapter rivals full fine-tuning

Empirically, the changes a fine-tune actually wants to make to a pretrained model live in a low-dimensional subspace. The full d² space of possible weight updates is overkill for most downstream tasks; the rank-16 subspace LoRA gives you covers most of what fine-tuning needs to express. On almost every benchmark of interest, LoRA gets you 90%+ of the quality of a full fine-tune at 1% of the parameter cost.

That number is real and load-bearing. It's why nobody in the open-weights world fully fine-tunes anymore for typical instruction-following or domain-adaptation tasks; the math just doesn't argue for it.

What this track will do

Six lessons, end-to-end LoRA workflow on Apple Silicon — prepare data (lesson 2), pick the training-config knobs (lesson 3), graduate to QLoRA / DoRA when warranted (lesson 4), merge the adapter for cheap inference (lesson 5), then a complete walkthrough you can repeat with your own dataset (lesson 6). The whole pipeline runs comfortably on a 32 GB Mac with a 7B base model.

Code

The minimum LoRA training command·bash

# Train a LoRA adapter on a small base model with a HF-hosted dataset.
# This is the literal minimum to start; lesson 3 walks through every flag.
python -m mlx_lm lora \
  --model mlx-community/Llama-3.2-1B-Instruct-4bit \
  --train \
  --data mlx-community/wikisql \
  --iters 100 \
  --adapter-path ./my-adapter

# Output during training (verbose; mlx-lm reports loss every --steps-per-report iters):
#   Iter 10: Train loss 1.832, Learning Rate 1.000e-05, ...
#   Iter 20: Train loss 1.521, ...
#   ...

Inspect the adapter directory after training·bash

ls ./my-adapter/
# adapters.safetensors    — the trained LoRA weights (small file!)
# adapter_config.json     — rank, alpha, layers, fine-tune type, etc.
# 0000100_adapters.safetensors  — checkpoint at iter 100 (if --save-every fired)

# The adapter file is tiny compared to the base model:
du -sh mlx-community/Llama-3.2-1B-Instruct-4bit/  # ~700 MB base
du -sh ./my-adapter/                              # tens of MB at most

Exercise

Without running it yet, predict — for a Llama-3.2-1B base model fine-tuned with rank-8 LoRA on the default 16 layers, roughly how many trainable parameters are there? Use the rough formula 2 · d_model · r · num_layers (the d_model for Llama-3.2-1B is 2048). Then check the actual count by inspecting adapter_config.json after running a short training. Two sentences on whether your prediction matched.