mlx_lm.convert — Pulling a HF Model into MLX-Native Form

Why convert at all

Most LLMs on Hugging Face are stored in PyTorch's safetensors format with PyTorch-shaped weight names. MLX wants the same weights but with MLX-shaped names and (often) quantization applied. mlx_lm.convert is the tool that reads a Hugging Face model and writes out a directory ready for mlx_lm.load.

The mlx-community Hugging Face org is full of pre-converted models — for popular base models you usually don't need to convert anything yourself. You convert when (1) the model you want isn't in mlx-community, (2) you want a specific quantization that nobody uploaded yet, or (3) you've fine-tuned a model and want to ship the result.

The single command

The CLI is a one-liner. Pull a Hugging Face model, convert it (optionally quantizing on the way), write to disk:

The flags worth knowing

--hf-path (alias --model) — the Hugging Face repo id (e.g. meta-llama/Llama-3.2-1B-Instruct) or a local directory.
--mlx-path — where to write the MLX-format output. Defaults to a sensible path under the current directory.
-q / --quantize — quantize the weights during conversion. Without this flag, you get the original precision (typically bf16/fp16).
--q-bits — bits per weight (typically 4 or 8). Default is 4 when -q is set.
--q-group-size — granularity of quantization. Smaller group size means better quality but larger file. Common values are 32, 64, 128.
--q-mode — affine (the classic), or one of the newer MX formats (mxfp4, nvfp4, mxfp8) for hardware-accelerated inference on the right chips.
--quant-predicate — mixed-precision recipe (e.g. mixed_3_4 = some layers at 3-bit, others at 4-bit). Lesson 4 covers this.
--dtype — for non-quantized weights, the dtype to save (float16, bfloat16, float32).
--upload-repo — if set, pushes the converted model to a Hugging Face repo. Requires huggingface-cli login.
-d / --dequantize — go the other way; turn a quantized MLX model back into full-precision weights.
--trust-remote-code — required for some models with custom modeling code (use carefully).

The silent-fail trap to know about

If the source model's config.json is missing a field mlx-lm needs (e.g. an unusual rope-scaling config, or a non-standard model_type value), conversion may complete, but loading fails later with a confusing key error. The defense is to test-load the converted model before trusting it: python -c "from mlx_lm import load, generate; m, t = load('./my-converted'); print(generate(m, t, prompt='hi', max_tokens=5))". Five seconds; saves an hour.

Code

Convert a small Llama model from HF, quantize to Q4 (sketch — runs but skip in mlx-quest demo)·bash

# This downloads the original model from meta-llama (requires HF auth + license accept)
# and writes a Q4 MLX-format copy locally. Don't run unless you actually want to convert.
python -m mlx_lm convert \
  --hf-path meta-llama/Llama-3.2-1B-Instruct \
  --mlx-path ./Llama-3.2-1B-Instruct-MLX-Q4 \
  --quantize \
  --q-bits 4 \
  --q-group-size 64

# After conversion, the local directory looks just like the inspected files in lesson 1:
ls ./Llama-3.2-1B-Instruct-MLX-Q4/
# config.json  model.safetensors  tokenizer.json  tokenizer_config.json  ...

Quick smoke test — does it actually load and generate?·python

# Always test-load before trusting a freshly converted model.
from mlx_lm import load, generate

model, tok = load("./Llama-3.2-1B-Instruct-MLX-Q4")
print(generate(model, tok, prompt="Hi.", max_tokens=10, verbose=False))

# If this raises a key error, the conversion's config.json is missing something
# the loader expects — diff against a known-working mlx-community config.

Optional — push the converted model to your own HF repo·bash

# Requires `huggingface-cli login` first.
python -m mlx_lm convert \
  --hf-path meta-llama/Llama-3.2-1B-Instruct \
  --mlx-path ./Llama-3.2-1B-Instruct-MLX-Q4 \
  --quantize --q-bits 4 \
  --upload-repo your-username/Llama-3.2-1B-Instruct-MLX-Q4

# The upload includes a generated README with the source repo,
# the conversion command, and the quantization config — so others
# can reproduce or audit the conversion.

Exercise

Pick a small instruct model that's NOT in mlx-community (or any model you'd like to have at a custom quantization). Convert it locally with mlx_lm.convert at Q4 and Q8 — produce two output directories. Compare their disk sizes and run a quick generate on each with the same prompt. Note the size ratio (Q8 is roughly 2× the disk size of Q4) and whether the outputs feel meaningfully different in quality. Two sentences on what you found.