GGUF and MLX Formats

GGUF — the universal local format

GGUF (GPT-Generated Unified Format) is a single-file format from the llama.cpp project. One .gguf file holds weights, tokenizer, and metadata together. It's the de facto standard for local inference because:

• Runs on CUDA (NVIDIA), ROCm/Vulkan (AMD), Metal (Apple Silicon), and CPU AVX paths.
• Supports 40+ model architectures (Llama, Qwen, Gemma, Mistral, Phi, DeepSeek, etc.).
• Most open models get a community GGUF within a day of release.
• Ollama internally stores models as GGUF blobs.

MLX — Apple's native format

MLX is Apple's machine-learning framework. Models are stored as a directory of safetensors files plus a config.json. Quantization uses fine group quantization — every 64 weights share a scale and bias — and kernels are written specifically for the Apple GPU.

• Apple Silicon only (no NVIDIA/AMD path).
• 3,000+ pre-converted models on the mlx-community HuggingFace org.
• Tends to win on decode throughput on Apple hardware; GGUF tends to win on prefill latency.
• As of Ollama v0.19, Ollama uses MLX under the hood on Apple Silicon (preview), so the format choice matters less from a user perspective.

Other formats you'll meet

• Safetensors — the HuggingFace standard for full-precision weights. Inference engines convert from safetensors to GGUF / MLX for local use.
• GGML — the predecessor to GGUF. Don't pull GGML files in 2026; they're deprecated.
• ONNX — cross-framework runtime format, common in classical ML, rare in the LLM local-inference world.