Local Deployment: Ollama & llama.cpp

~24 min · ollama, llama-cpp, gguf, local, apple-silicon

Level 0Observer

0 XP0/43 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

The local serving path

For local inference (your laptop, an internal server, an air-gapped box), the Ollama + llama.cpp stack is the right pick. GGUF is the format both speak natively.

The pipeline

Merge LoRA into base (previous lesson).
Convert HF format → GGUF using llama.cpp's converter.
Quantize the GGUF for your target memory budget.
Wrap with a Modelfile and serve via Ollama.

Code

HF → GGUF + quantize·bash

# Convert merged HF model to GGUF
cd llama.cpp
python convert_hf_to_gguf.py ../merged-model \
    --outtype f16 --outfile model-f16.gguf

# Quantize (pick based on hardware)
./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M
# For Apple Silicon with plenty of RAM:
./llama-quantize model-f16.gguf model-Q6_K.gguf Q6_K

Ollama Modelfile + run·bash

# Create Modelfile
cat > Modelfile << 'EOF'
FROM ./model-Q4_K_M.gguf

TEMPLATE """<|begin_of_text|><|start_header_id|>system<|end_header_id|>
{{ .System }}<|eot_id|><|start_header_id|>user<|end_header_id|>
{{ .Prompt }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""

PARAMETER temperature 0.7
PARAMETER num_ctx 4096
PARAMETER stop "<|eot_id|>"
SYSTEM "You are my fine-tuned assistant."
EOF

# Build and run
ollama create my-model -f Modelfile
ollama run my-model "Hello, how can you help me?"

External links

Exercise

Take a merged 7B model and run it locally with Ollama at Q4_K_M. Compare its outputs on 5 prompts to the unquantized HF version. Document any quality drops. Then try Q6_K and re-evaluate — is the quality lift worth the extra ~1.4 GB of RAM?