LoRA Variants & Comparison

The 2024–2025 LoRA family

Since the original LoRA paper, several variants have been published. Each fixes a specific weakness.

Variant	Idea	Benefit
DoRA	Decompose weight into magnitude + direction; LoRA on direction only.	Closes the gap with full FT. Use `use_dora=True`.
LoRA+	Different learning rates for A and B matrices.	~2× faster convergence.
rsLoRA	Scales alpha by 1/√r instead of 1/r.	Better at high ranks (r ≥ 64). Use `use_rslora=True`.
AdaLoRA	Dynamically allocates rank per layer using SVD.	Auto-finds optimal rank per layer.
VeRA	Shared random A,B + learnable scaling vectors.	~10× fewer params than LoRA.
LoRA-FA	Freeze A, only train B.	Halves activation memory.

Method comparison vs full fine-tuning

Method	Trainable %	Quality vs full FT	Memory
Full fine-tuning	100%	Baseline	Very high
LoRA	~0.1–1%	~95–98%	Low
QLoRA	~0.1–1%	~93–97%	Very low
DoRA	~0.1–1%	~97–99%	Low–medium
Prompt tuning	~0.01%	~85–90%	Very low
Prefix tuning	~0.1%	~88–93%	Very low

Code

DoRA + rsLoRA configs in PEFT v0.17+·python

from peft import LoraConfig

# DoRA — closes the gap with full FT
dora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",
    use_dora=True,                # enable DoRA
    task_type="CAUSAL_LM",
)

# rsLoRA — better at high ranks
rslora_config = LoraConfig(
    r=64,
    lora_alpha=64,
    target_modules="all-linear",
    use_rslora=True,              # better scaling at high ranks
    task_type="CAUSAL_LM",
)

# Combined: DoRA + rsLoRA at high rank
strongest_config = LoraConfig(
    r=64,
    lora_alpha=64,
    target_modules="all-linear",
    use_dora=True,
    use_rslora=True,
    lora_dropout=0.1,
    task_type="CAUSAL_LM",
)

Exercise

Train three small (200-example) LoRA jobs on the same model + dataset: vanilla LoRA, DoRA, and DoRA+rsLoRA at high rank. Compare validation loss curves. Which one is genuinely better? Often the gap is smaller than you'd expect — the point is to verify the lift on your specific task before defaulting to the fanciest variant.

The 2024–2025 LoRA family

Method comparison vs full fine-tuning

Code

External links

Exercise

Progress

Comments 0