The Parameter Problem

~18 min · full-fine-tuning, vram, memory

Level 0Observer

0 XP0/43 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Why full fine-tuning is impractical

A 7B parameter model has 7 billion floating-point numbers. In full fine-tuning, you need to:

Store the model weights (7B × 2 bytes in fp16 = 14 GB)
Store the gradients (another 14 GB)
Store optimizer states (Adam needs 2 states per param = 28 GB)
Plus activations, batch data, framework overhead...

Total: ~60–80 GB VRAM just for a 7B model. A 70B model needs 600+ GB — multiple A100s.

Model size	Full FT VRAM (fp16)	GPU needed
1B	~12 GB	RTX 3090 / 4090
7B	~60 GB	A100 80 GB
13B	~120 GB	2× A100 80 GB
70B	~600 GB	8× A100 80 GB

This is where Parameter-Efficient Fine-Tuning (PEFT) comes in. Instead of updating all 7 billion parameters, you update a tiny added subset — often less than 1% — while keeping the rest frozen.

Code

Count trainable params before vs after PEFT·python

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
total = sum(p.numel() for p in base.parameters())
print(f"Base trainable: {total:,}")

config = LoraConfig(r=16, lora_alpha=32,
                    target_modules="all-linear", task_type="CAUSAL_LM")
lora_model = get_peft_model(base, config)
lora_model.print_trainable_parameters()
# trainable params: ~13M || all params: ~8B || trainable%: 0.17%

External links

Exercise

Pick a model size you care about (1B, 7B, 13B, or 70B). Calculate full-FT VRAM, LoRA VRAM, and QLoRA VRAM (Track 4 lesson 4 has the QLoRA numbers). Compare to the GPUs you can actually rent — which method is feasible for you?