OpenAI-Compatible Endpoints

~22 min · inference, openai-compat

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Why OpenAI compatibility matters

Most clients in the wild — LangChain, LlamaIndex, your custom shim, the OpenAI SDK itself — speak OpenAI's wire format. HF's Inference Providers expose an OpenAI-compatible endpoint at https://router.huggingface.co/v1/.... You point the OpenAI SDK at that URL with your HF token, and any model on any HF-routed provider becomes accessible through code that thinks it's calling OpenAI.

What's the same

The chat completions, embeddings, and (some) image endpoints. JSON shapes for messages, tools, response_format, streaming chunks. Authorization: Bearer ${HF_TOKEN} in place of OPENAI_API_KEY.

What's different

Model ids are HF repo ids, not OpenAI model names. Some response fields (provider-specific metadata) carry over, others don't. Rate limits are HF's, not OpenAI's.

Code

Use the OpenAI SDK against HF·python

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://router.huggingface.co/v1",
    api_key=os.environ["HF_TOKEN"],
)

resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",  # HF repo id
    messages=[{"role": "user", "content": "Hi from the OpenAI SDK pointed at HF."}],
    max_tokens=60,
)
print(resp.choices[0].message.content)

Streaming through the OpenAI SDK·python

from openai import OpenAI
import os

client = OpenAI(base_url="https://router.huggingface.co/v1", api_key=os.environ["HF_TOKEN"])

stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "List three open-source LLMs."}],
    max_tokens=200,
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

External links

Exercise

Take an existing piece of code that calls OpenAI (or write one). Switch it to call HF's router with a Llama-3 instruct model. Diff: how many lines changed (should be 2: base_url and model). Run both versions on the same prompt, compare outputs.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousStreaming, Tool Use, and Structured Output Next →Cost, Rate Limits, and Retry Strategy

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.