QLoRA & DoRA — When Vanilla LoRA Isn't Enough

Two enhancements to vanilla LoRA

LoRA gets you most of the way there for most tasks. Two named refinements extend its reach in specific directions — QLoRA for fitting bigger base models in less memory, DoRA for closing the quality gap to full fine-tuning. mlx-lm supports both with minimal extra ceremony.

QLoRA — train an adapter on top of a quantized base

QLoRA is the technique of quantizing the base model to 4-bit during training, then training the (full-precision) LoRA adapter on top of it. The base model takes ~25% of the memory it would in fp16; the adapter is small either way. Result: you can fine-tune a 70B model on hardware that couldn't fit a 70B in fp16 even at inference.

In mlx-lm, QLoRA is the implicit name for what happens when you point --model at an already-quantized base (e.g. mlx-community/Mistral-7B-Instruct-v0.3-4bit). There's no --qlora flag — quantization-aware training is automatic when the base model is quantized. The training math is the same as vanilla LoRA; the memory savings come from the base model's quantization, which mlx-lm handles transparently.

DoRA — better quality at the same parameter count

DoRA (Weight-Decomposed Low-Rank Adaptation) decomposes the weight update into two parts: a magnitude vector and a direction matrix. Train both. Empirically, DoRA closes most of the remaining quality gap between LoRA and full fine-tuning at the same parameter budget — the cost is a small increase in compute and memory per step.

In mlx-lm, you switch on DoRA with --fine-tune-type dora. Same data format, same other flags, drop-in replacement.

When to reach for each

Default — vanilla LoRA on a quantized base (effectively QLoRA). Covers 90% of cases.
Going bigger than your machine should allow — quantize the base more aggressively (Q4 with mixed-precision). Pure QLoRA territory.
LoRA quality is plateauing and you've already raised rank — try DoRA. Same training command with one flag changed.
You're considering full fine-tuning — try DoRA at higher rank first; the math usually argues against full.

What you don't need to think about

You don't pick "LoRA" vs "QLoRA" explicitly — the latter is just LoRA on top of a quantized base, which mlx-lm handles automatically. You only explicitly switch fine-tune type when going to DoRA or (rarely) full.

Code

QLoRA — just point at a quantized base·bash

# QLoRA is implicit: --model points at an already-Q4 base, mlx-lm
# trains a full-precision adapter on top, base stays quantized in memory.
python -m mlx_lm lora \
  --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --train --data ./my-data \
  --fine-tune-type lora \
  --num-layers 16 \
  --batch-size 4 --iters 200 --learning-rate 5e-5 \
  --adapter-path ./my-qlora-adapter

# Memory footprint stays at the base-model's quantized size + a small adapter.
# Same workflow as vanilla LoRA — no QLoRA-specific flags.

DoRA — drop in --fine-tune-type dora·bash

# Same training run, but using DoRA's weight decomposition for slightly
# better quality at the same trainable-parameter count.
python -m mlx_lm lora \
  --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --train --data ./my-data \
  --fine-tune-type dora \
  --num-layers 16 \
  --batch-size 4 --iters 200 --learning-rate 5e-5 \
  --adapter-path ./my-dora-adapter

# Per-step memory and compute are slightly higher than LoRA;
# converged quality is usually slightly better.

Exercise

Run the same dataset through three quick fine-tunes (50 iterations each, same other flags) — vanilla LoRA on a fp16 base, vanilla LoRA on a Q4 base (= QLoRA), and DoRA on a Q4 base. Compare the train loss curves at iter 50 and the per-step time. The exercise is to feel — concretely — what each enhancement costs and what it buys you on your hardware and your data.