Merging & Export

~18 min · merge, export, deployment-pipeline

Level 0Observer

0 XP0/43 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

The deployment pipeline

Merge the LoRA adapter into the base model.
Convert to deployment format (GGUF, GPTQ, AWQ).
Quantize for the target hardware.
Serve via your chosen platform (Ollama, vLLM, TGI).

Why merge?

A merged model has zero PEFT runtime overhead and is byte-identical in shape to the base model. Any serving framework can use it directly — no PEFT dependency at inference time. Merging is the right move for any single-adapter production deployment.

Code

Merge LoRA, save merged model·python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Step 1: load base in full precision (so merge math is accurate)
base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.float16,
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

# Step 2: load LoRA adapter on top
model = PeftModel.from_pretrained(base, "./my-lora-adapter")

# Step 3: merge — adapters fold into base; PeftModel wrapper goes away
merged = model.merge_and_unload()

# Step 4: save the merged model
merged.save_pretrained("./merged-model")
tok.save_pretrained("./merged-model")
print("Merged model ready for vLLM / TGI / Ollama serving.")

External links

PEFT — merge_and_unload reference

Exercise

Take a LoRA adapter you've trained. Merge it and save. Load the merged model and the original (base + adapter) setup separately. Verify they produce identical outputs on 5 test prompts. Confirm the merged model has no PEFT dependency.

Progress

Progress is local-only — sign in to sync across devices.

← 📚 Fine-Tune ProjectsQuiz · 5 questions Next →Local Deployment: Ollama & llama.cpp

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.