mlx-vlm — Vision-Language Models in MLX · MLX Quest

Vision-Language Models in MLX

mlx-vlm is the MLX-native package for vision-language models — models that take an image (or video frames) and a text prompt, and produce text. The current best-of-class for the Apple-Silicon-friendly tier are Qwen-VL family models, LLaVA derivatives, and the smaller multimodal Llamas.

The shape is parallel to mlx-lm's — load returns model + processor; generate takes prompt + image and returns text. The only real new piece is the image preprocessing, which is the part most VLM tutorials get wrong on the first try.

Install + minimum loop

pip install mlx-vlm in your mlx env. The CLI is python -m mlx_vlm.generate; the Python API mirrors mlx_lm's shape with a processor object that handles images.

The image preprocessing detail everyone gets wrong

VLMs are trained on a specific image resolution and a specific normalization (mean/std). If you pass an image at the wrong resolution or skip the normalization, the model still produces output — usually plausibly worded, completely hallucinated. The processor object that mlx-vlm returns from load() handles all of this for you when you use its API; the bug appears when people pre-process images themselves and bypass the processor.

Always feed images through the processor. Don't try to be clever with PIL or torchvision transforms — the processor knows the model's expectations.

What VLMs are good at (today)

OCR-like tasks — reading text in images, including handwriting and tables.
Chart and diagram interpretation — "what's the trend in this graph?" works surprisingly well.
Visual question answering — "how many people are in this photo?" "what's the broken part?" — the bread-and-butter.
UI screenshot reasoning — "what does this app do?" "is this button enabled?" — increasingly strong as agentic UI workflows develop.

What they're still weak at

Spatial precision — "how far apart are these two objects in pixels?" or anything requiring exact geometry. VLMs are describing images, not measuring them.
Counting many objects — works well up to ~5 instances, degrades quickly past that.
Small text in large images — resolution matters; if text is below ~10 pixels per character at the model's input resolution, OCR fails.

Code

Install mlx-vlm·bash

conda activate mlx
pip install mlx-vlm

# Verify
python -c "import mlx_vlm; print('mlx_vlm:', mlx_vlm.__version__)"
# Re-verified 2026-09-15 on mlx_vlm 0.7.1 by running the next block end to end
# (first checked 2026-08-07 on 0.6.10): `load`, `generate`,
# `apply_chat_template` and `load_config` all still work unchanged.

Ask a question about an image — minimum API surface·python

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load a small VLM (under ~5 GB on disk)
model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
model, processor = load(model_path)
config = load_config(model_path)

# Image path or URL
image = ["path/to/your/image.jpg"]
prompt = "What's in this image? Be concise."

# Apply the chat template (includes the image placeholder)
formatted_prompt = apply_chat_template(processor, config, prompt, num_images=len(image))

output = generate(model, processor, formatted_prompt, image, max_tokens=120, verbose=False)
print(output.text)   # generate() returns a GenerationResult; .text is the answer

CLI variant of the same call·bash

python -m mlx_vlm.generate \
  --model mlx-community/Qwen2-VL-2B-Instruct-4bit \
  --image path/to/your/image.jpg \
  --prompt "What's in this image? Be concise." \
  --max-tokens 120

Exercise

Pick three images that test different VLM capabilities — one screenshot of a UI, one chart or diagram, one photograph with multiple objects. Ask the same VLM the same question about each ("describe what you see in two sentences"). Note where the model is sharp, where it's vague, and where it confidently invents details that aren't there. Two sentences on the failure mode you found most surprising.

mlx-vlm — Vision-Language Models in MLX