ONNX and MLX Conversion

~22 min · ops, onnx, mlx

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

ONNX: cross-runtime portability

ONNX Runtime runs the same exported model on Windows, Linux, macOS, mobile, web (via WebAssembly). For deploy targets where you can't bring PyTorch, ONNX is usually the right pick. The optimum.onnxruntime wrappers do the conversion in one method call.

MLX: Apple Silicon native

MLX is Apple's native ML framework: unified memory, Metal kernels, lazy evaluation. mlx-lm wraps Llama-style architectures and runs HF checkpoints natively on Apple Silicon. Throughput often beats PyTorch + MPS by 2-3x at the same memory budget.

The decision

If your deployment surface includes mobile / browser / non-PyTorch servers → ONNX. If you're shipping for Apple Silicon (M-series Macs, iPad/iPhone) → MLX. Both can sit alongside the original PyTorch checkpoint in a single Hub repo.

Code

Export to ONNX with optimum·python

from optimum.onnxruntime import ORTModelForCausalLM
from transformers import AutoTokenizer

repo = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(repo)
ort = ORTModelForCausalLM.from_pretrained(repo, export=True)

ort.save_pretrained("./qwen-onnx")
tok.save_pretrained("./qwen-onnx")

# Inference
inputs = tok("Hello", return_tensors="pt")
out = ort.generate(**inputs, max_new_tokens=20)
print(tok.decode(out[0], skip_special_tokens=True))

Convert + run via MLX on Apple Silicon·bash

# pip install mlx-lm
# Convert a HF model in-place to MLX format
python -m mlx_lm.convert --hf-path Qwen/Qwen2.5-1.5B-Instruct --mlx-path ./qwen-mlx

# Generate
python -m mlx_lm.generate \
  --model ./qwen-mlx \
  --prompt "Hello, MLX." \
  --max-tokens 50

External links

Exercise

Convert a 1-3B model to ONNX with optimum. Run inference and compare per-token latency vs PyTorch+MPS or +CUDA. If you have Apple Silicon, also convert to MLX (mlx_lm.convert) and run there. Note: where each format wins.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousGGUF Export for Ollama / llama.cpp Next →Accelerate: Multi-Device, Mixed Precision, FSDP

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.