LoRA Hyperparameters

The five knobs that matter

Parameter	What it does	Recommended
`r` (rank)	Dimensionality of A, B. Higher = more expressive, more memory.	8–64 (start with 16)
`lora_alpha`	Scaling factor. Update is scaled by `alpha/r`.	2× rank (e.g. r=16 → alpha=32)
`target_modules`	Which layers get LoRA.	`"all-linear"` for QLoRA; `["q_proj","v_proj"]` for budget LoRA
`lora_dropout`	Dropout on LoRA layers.	0.05–0.1
`bias`	Whether to train bias terms.	`"none"` typically

Target modules explained

A typical Llama-style transformer block has these linear layers:

q_proj, k_proj, v_proj, o_proj: attention layers
gate_proj, up_proj, down_proj: MLP / FFN layers

Using "all-linear" applies LoRA to every linear layer. This gives the best results for QLoRA but uses more memory. For budget LoRA, starting with just ["q_proj", "v_proj"] is the original LoRA paper recipe and a reasonable balance.

The alpha/rank ratio

The effective learning rate for LoRA parameters is scaled by alpha/r. Common practice: set alpha = 2 * r. If using use_rslora=True (rsLoRA, lesson 5), alpha is automatically scaled by 1/√r, which works better at higher ranks.

Code

Conservative vs aggressive LoRA configs·python

from peft import LoraConfig

# Conservative — beginner-friendly, lower memory
beginner_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

# Aggressive — best quality, more memory
quality_config = LoraConfig(
    r=64,
    lora_alpha=128,
    target_modules="all-linear",
    lora_dropout=0.1,
    bias="none",
    task_type="CAUSAL_LM",
    use_rslora=True,   # better scaling at high ranks
)

The five knobs that matter

Target modules explained

The alpha/rank ratio

Code

External links

Exercise

Progress

Comments 0