llama.cpp Server

~22 min · serving, llamacpp, gguf

Level 0Downloader

0 XP0/41 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Why llama.cpp server

llama.cpp is the foundational C/C++ inference engine. Ollama uses it under the hood. llama-server (built from the llama.cpp tree) exposes the same engine as a standalone HTTP server with full OpenAI-compatible chat completions plus llama.cpp-specific endpoints.

When you reach for it directly

You have a specific GGUF Ollama doesn't ship and you don't want to build a Modelfile + repository.
You need flag-level control — exact num_ctx, batch size, GPU split, mmap settings, KV cache type.
You're embedding the engine in your own product and want one binary, no daemon, no model registry.
You want to run multiple variants on different ports — one server per model, no shared state.

Build, serve, hit

llama.cpp builds with one CMake invocation. The server binary is llama-server. Point it at any GGUF file and it listens on the port you choose with the OpenAI-compat endpoint live by default.

Code

Build, serve a GGUF, hit it·bash

# Build llama.cpp (Apple Silicon Metal build)
git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp
cd ~/llama.cpp && cmake -B build && cmake --build build --config Release -j

# Download a GGUF you want to serve directly
huggingface-cli download bartowski/Qwen2.5-7B-Instruct-GGUF \
  Qwen2.5-7B-Instruct-Q4_K_M.gguf --local-dir ~/models

# Serve on port 8080 with 8K context, all GPU layers
./build/bin/llama-server \
  -m ~/models/Qwen2.5-7B-Instruct-Q4_K_M.gguf \
  -c 8192 \
  -ngl 999 \
  --port 8080

# Hit it with OpenAI-compatible client
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-direct",
    "messages": [{"role":"user","content":"Hello!"}],
    "stream": false
  }'

Use the OpenAI SDK against llama-server·python

from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-used",   # llama-server ignores it; SDK requires non-empty
)

resp = client.chat.completions.create(
    model="qwen-direct",   # llama-server uses whatever name you gave; some builds accept any
    messages=[{"role": "user", "content": "Define GGUF in one sentence."}],
    stream=False,
)
print(resp.choices[0].message.content)

External links

Exercise

Build llama.cpp on your machine. Download one GGUF directly from HuggingFace (try a quant Ollama doesn't ship). Serve it on port 8080. Hit it with the OpenAI Python SDK. Compare answer quality and tok/s to the same model in Ollama.