GGUF and MLX Formats

GGUF — the universal local format

GGUF is a single-file format from the llama.cpp project. One .gguf file holds weights, tokenizer, and metadata together. It's the de facto standard for local inference because:

• Runs on CUDA (NVIDIA), ROCm/Vulkan (AMD), Metal (Apple Silicon), and CPU AVX paths.
• Supports 40+ model architectures (Llama, Qwen, Gemma, Mistral, Phi, DeepSeek, etc.).
• Most open models get a community GGUF within a day of release.
• Ollama internally stores models as GGUF blobs.

MLX — Apple's native format

MLX is Apple's machine-learning framework. Models are stored as a directory of safetensors files plus a config.json. Quantization uses fine group quantization — every 64 weights share a scale and bias — and kernels are written specifically for the Apple GPU.

• Built for Apple Silicon, and fastest there — but not confined to it. pip install mlx[cuda12] (or cuda13) builds the CUDA backend for Linux/NVIDIA, and mlx[cpu] a CPU-only one. There is no AMD/ROCm path.
• 3,000+ pre-converted models on the mlx-community HuggingFace org.
• Tends to win on decode throughput on Apple hardware; GGUF tends to win on prefill latency.
• Ollama has an MLX engine on Apple Silicon, previewed in v0.19, but it is opt-in per model rather than a silent swap: you pull an MLX-built tag (gemma4:31b-mlx, qwen3.5:35b-a3b-coding-nvfp4) and the preview asked for more than 32 GB of unified memory. Plain tags still ship GGUF — ollama show gemma4:31b reports format: gguf while the -mlx sibling reports safetensors. So the format choice is still yours to make.

Other formats you'll meet

• Safetensors — the HuggingFace standard for full-precision weights. Server-side engines such as vLLM and TGI load it directly; Ollama and llama.cpp want a GGUF conversion, and MLX its own directory layout.
• GGML — the predecessor to GGUF. Don't pull GGML files in 2026; they're deprecated.
• ONNX — cross-framework runtime format, common in classical ML, rare in the LLM local-inference world.