GGUF Export for Ollama / llama.cpp

~24 min · ops, gguf, ollama

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Why GGUF

GGUF (formerly GGML) is the format llama.cpp invented and Ollama wraps. It's optimized for: CPU-first inference, Metal acceleration on Apple Silicon, cross-platform binary distribution. A single .gguf file contains weights + tokenizer + metadata, ready to ollama pull.

How to produce a GGUF

The canonical path is via llama.cpp's convert_hf_to_gguf.py. Clone llama.cpp, point the script at your HF repo, pick a quantization (Q4_K_M, Q5_K_M, Q8_0, F16). Output: a .gguf file you can ship.

The naming convention

By community convention, GGUF files on the Hub are named {model}-{params}.{quant}.gguf: Llama-3.1-8B-Instruct-Q4_K_M.gguf. The quant tier (Q4_K_M is the typical "balanced" choice) tells you the bit-width and rounding scheme.

Code

Convert a HF model to GGUF·bash

# One-time setup
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
pip install -r requirements.txt

# Download HF model
huggingface-cli download "Qwen/Qwen2.5-1.5B-Instruct" --local-dir ./qwen-1.5b

# Convert to F16 GGUF
python convert_hf_to_gguf.py ./qwen-1.5b --outfile qwen-1.5b-f16.gguf

# Quantize to Q4_K_M (build the quantize binary first via cmake)
./build/bin/llama-quantize qwen-1.5b-f16.gguf qwen-1.5b-Q4_K_M.gguf Q4_K_M

Push a GGUF to a Hub repo and pull via Ollama·bash

# Create a model repo on the Hub
huggingface-cli upload "yourname/qwen-1.5b-gguf" qwen-1.5b-Q4_K_M.gguf --repo-type=model

# Then pull via Ollama (modelfile pattern)
cat > Modelfile <<EOF
FROM hf.co/yourname/qwen-1.5b-gguf:Q4_K_M
EOF

ollama create qwen-1.5b-cwk -f Modelfile
ollama run qwen-1.5b-cwk

External links

Exercise

Convert a 1-3B Hub model to GGUF (Q4_K_M). Push the GGUF to a private Hub repo. Pull via Ollama using the Modelfile pattern. Run inference and compare output quality vs the unquantized HF model on 5 prompts.