Saving, Pushing & Converting

~22 min · hub, gguf, ollama, conversion, quantization

Level 0Observer

0 XP0/43 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Push to Hugging Face Hub

The Hub is the model registry. Push your LoRA adapter (small) or your merged model (larger) and document it with a model card.

Convert to GGUF for Ollama / llama.cpp

To run your fine-tuned model in Ollama or llama.cpp, you need GGUF format. Two-step process: merge LoRA into base, then convert HF format → GGUF using llama.cpp's converter.

GGUF quantization choices

Format	Size (7B)	Quality	Speed
Q2_K	~2.7 GB	Low	Fastest
Q4_K_M	~4.1 GB	Good	Fast
Q5_K_M	~4.8 GB	Very good	Medium
Q6_K	~5.5 GB	Excellent	Slower
Q8_0	~7.2 GB	Near-lossless	Slowest

Q4_K_M is the most-used balance. Q6_K when you have the RAM and want quality.

Code

Push LoRA + merged model to Hub·python

# Push LoRA adapter (small)
trainer.push_to_hub("your-username/my-lora-adapter")

# Or push merged model (larger)
merged = model.merge_and_unload()
merged.push_to_hub("your-username/my-merged-model")
tokenizer.push_to_hub("your-username/my-merged-model")

Convert merged model to GGUF + quantize·bash

# Step 1: clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# Step 2: convert HF model to GGUF
python convert_hf_to_gguf.py /path/to/merged-model \
    --outtype f16 \
    --outfile model-f16.gguf

# Step 3: quantize for efficient inference
./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M
# Or for Apple Silicon with plenty of RAM:
./llama-quantize model-f16.gguf model-Q6_K.gguf Q6_K

Deploy with Ollama·bash

# Create a Modelfile
cat > Modelfile << 'EOF'
FROM ./model-Q4_K_M.gguf

TEMPLATE """<|begin_of_text|><|start_header_id|>system<|end_header_id|>
{{ .System }}<|eot_id|><|start_header_id|>user<|end_header_id|>
{{ .Prompt }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""

PARAMETER temperature 0.7
PARAMETER stop "<|eot_id|>"
SYSTEM "You are my fine-tuned assistant."
EOF

# Build and run
ollama create my-finetuned -f Modelfile
ollama run my-finetuned

External links

Exercise

Take the merged model from the previous lesson, convert it to GGUF, quantize to Q4_K_M, and run it locally with Ollama. Compare its output on 5 prompts to the original HF-format model. Quality drops should be small at Q4_K_M; document any you notice.