Merging the Adapter — Inference Without the LoRA Round-Trip

Why merge at all

After training, you have two separate things: the base model and the trained adapter. At inference time, the model computes (W + A · B) · x — the base weights plus the adapter contribution, applied at every layer the adapter touches. That extra matrix multiply per touched layer per token costs you on every generation.

The fix is to merge — fold the adapter into the base weights once, producing a new model file where W already includes the learned A · B. After merging, inference is a normal forward pass with no adapter overhead. Same quality as the unmerged setup; cheaper per-token cost.

The merge command — mlx_lm.fuse

One command. Point it at the base model and the adapter directory; get back a new model directory ready for mlx_lm.load.

What you give up by merging

Merged models are static — the adapter is baked in, you can't swap it out. If your workflow involves multiple adapters for different tasks (one for SQL, one for haiku, one for summarization), keeping them separate and loading them on demand is more flexible than fusing. You only fuse when you've decided the model + this specific adapter is the deployment artifact.

The disk cost of fused models is roughly the same as the base model — you're not adding new parameters, you're just folding the adapter's contribution into the existing weights.

The optional GGUF export

mlx_lm.fuse can also export the fused model to GGUF format with --export-gguf. This is the cleanest path from "fine-tuned in MLX" to "running in llama.cpp/Ollama on a non-Apple machine" — fuse first, then export. Not the common path, but useful when you need to ship the result to a non-Mac deployment target.

The optional HF upload

--upload-repo your-username/repo-name pushes the merged model to a Hugging Face repo, just like mlx_lm.convert's upload flag (lesson 2 of convert). Use it when you want others to be able to mlx_lm.load your fine-tune by repo id.

Code

Fuse the adapter into the base — produce a deployable model·bash

# Folds ./my-adapter into the base model, writes the result to ./my-fused.
python -m mlx_lm fuse \
  --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --adapter-path ./my-adapter \
  --save-path ./my-fused

# After this, ./my-fused looks like a normal MLX model directory:
#   config.json  model.safetensors  tokenizer.json  ...
# And you can load it without referencing the adapter at all:
python -c "from mlx_lm import load, generate; m, t = load('./my-fused'); print(generate(m, t, prompt='hi', max_tokens=20))"

Fuse + push to your Hugging Face repo (one command)·bash

# Requires `huggingface-cli login` first.
python -m mlx_lm fuse \
  --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --adapter-path ./my-adapter \
  --save-path ./my-fused \
  --upload-repo your-username/Mistral-7B-Instruct-v0.3-MyDomain-4bit

# Others can then pull your fine-tune by repo id:
#   model, tok = load("your-username/Mistral-7B-Instruct-v0.3-MyDomain-4bit")

Fuse + export to GGUF (for non-Apple deployment)·bash

# Less common path: cross-format export so the result can run in llama.cpp / Ollama
# on Linux/Windows machines.
python -m mlx_lm fuse \
  --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --adapter-path ./my-adapter \
  --save-path ./my-fused \
  --export-gguf \
  --gguf-path ./my-fused.gguf

# ./my-fused.gguf is now a single file you can copy to any GGUF-capable runtime.

Exercise

Take the adapter you trained earlier in this track and fuse it into a deployable model with mlx_lm.fuse. Time a generation against (a) the base + adapter loaded separately at inference vs (b) the fused model. The fused model should be slightly faster per token because there's no adapter math at every layer. Two sentences on the latency difference and whether you'd consider it worth the loss of adapter swappability for your use case.