mlx-lm ships a built-in HTTP server that exposes /v1/chat/completions with the same request and response shape as OpenAI's API. That means anything that speaks the OpenAI client protocol — the official openai Python SDK, LangChain, LlamaIndex, your shell aliases — can point at your local mlx-lm server and Just Work.
This is the laziest possible production deployment for a single MLX model. Run one command, get a stable HTTP endpoint, talk to it from any tool that already knows OpenAI. The serious caveats — concurrency, queuing, multiple models — come up in prod.lesson1; this lesson is about the simple case.
Starting the server
Two-line setup. The model is loaded once at startup and held in memory across requests; subsequent requests pay only the inference cost.
Talking to it from openai-python
Point the OpenAI client at http://localhost:8080/v1 (the default port; configurable). Pass any non-empty string as the API key — the local server doesn't authenticate. Then call it the way you'd call OpenAI.
What works and what doesn't
Works: chat-completions endpoint, streaming SSE responses, model name routing (you can serve multiple models if you start multiple instances on different ports), basic sampling parameters (temperature, top_p, max_tokens).
Doesn't apply: embeddings (mlx-lm is a text-generation server, not an embeddings server — use a different package for that), function calling (depends on the model and template; OpenAI's exact JSON schema isn't always supported), fine-tuning endpoints (mlx-lm has its own LoRA workflow — see Track 4).
Single-process: the built-in server is single-process. For real concurrent serving with queuing and multiple workers, you'll write a thin FastAPI wrapper around mlx-lm — that's prod.lesson1.
When the built-in server is enough
For local development, a single-user demo, or an internal tool where you and maybe one teammate are the load, the built-in server is enough. For anything user-facing, anything with concurrency requirements, anything that needs auth or rate-limiting, you graduate to the FastAPI wrapper. The decision is about traffic and operational requirements, not about MLX itself.
Code
Start the server (in one terminal)·bash
# In a terminal with the `mlx` env activated:
conda activate mlx
# Start mlx-lm's built-in OpenAI-compatible HTTP server.
# Default port is 8080. The model is loaded once at startup.
python -m mlx_lm server \
--model mlx-community/Llama-3.2-1B-Instruct-4bit
# You'll see startup logs ending with something like:
# Starting httpd at 127.0.0.1:8080
#
# Leave this terminal running; the next code block talks to it from a second terminal.
Talk to it from openai-python (second terminal)·python
# In a separate terminal with openai-python installed:
# pip install openai
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8080/v1",
api_key="not-used-but-required", # any non-empty string is fine
)
resp = client.chat.completions.create(
model="mlx-community/Llama-3.2-1B-Instruct-4bit",
messages=[
{"role": "system", "content": "You are a terse assistant."},
{"role": "user", "content": "Capital of France?"},
],
max_tokens=20,
temperature=0.7,
)
print(resp.choices[0].message.content)
# → "Paris." (or close to it)
Streaming responses with the same client·python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="x")
stream = client.chat.completions.create(
model="mlx-community/Llama-3.2-1B-Instruct-4bit",
messages=[{"role": "user", "content": "Count 1 to 5:"}],
max_tokens=30,
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
print()
Start the server in one terminal. Hit it with the openai-python client in a second terminal — both single-shot and streaming. Then point Cursor (or any OpenAI-compatible client you already use) at http://localhost:8080/v1 with the same model name. Confirm a real chat works end-to-end. The exercise is to feel that mlx-lm slots into the OpenAI ecosystem without translation, with one config tweak per client.
Progress
Progress is local-only — sign in to sync across devices.