InferenceClient: One Object, Many Backends

~26 min · inference, client

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

The client is the abstraction

huggingface_hub.InferenceClient is a single Python class that talks to: HF's hosted Inference API, third-party providers routed through HF (Together AI, Fireworks, Replicate, …), text-generation-inference (TGI) servers you run yourself, and any OpenAI-compatible endpoint. You construct it with provider= and model=, and call methods like chat_completion, text_generation, image_generation.

Why it matters

The same client code runs against a managed Hub endpoint in dev and against a TGI box in prod. Migration is changing two arguments. Compare to a hard-coded OpenAI client: you'd be rewriting the call site.

Code

InferenceClient against HF-hosted inference·python

from huggingface_hub import InferenceClient

client = InferenceClient(
    model="meta-llama/Llama-3.1-8B-Instruct",
    provider="hf-inference",   # HF's own routing
)

resp = client.chat_completion(
    messages=[{"role": "user", "content": "Explain Hugging Face in one sentence."}],
    max_tokens=120,
)
print(resp.choices[0].message.content)

Same code, different backend (Together AI)·python

from huggingface_hub import InferenceClient

client = InferenceClient(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
    provider="together",
)

resp = client.chat_completion(
    messages=[{"role": "user", "content": "Same prompt, larger model."}],
    max_tokens=120,
)
print(resp.choices[0].message.content)

External links

Exercise

Make the same chat_completion call against three different providers (e.g. hf-inference, together, and one other you have access to) for the same model id (or the closest match). Time each. Note: which provider is fastest, and is there a region-related variation.

Progress

Progress is local-only — sign in to sync across devices.

← 📦 Datasets LibraryQuiz · 5 questions Next →Provider Routing and the Pricing Surface

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.