Provider Routing and the Pricing Surface

~24 min · inference, providers

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

The Hub as a router

HF's Inference Providers is a routing layer: you pick a model, the platform shows you which providers serve it, and your InferenceClient(provider=...) request hits that provider via HF's edge. You pay HF, who pays the provider. Auth is your HF token.

Why route through HF and not direct

One auth layer — one token, not five provider-specific keys.
One billing surface — one invoice instead of five.
Provider abstraction — if Provider A goes down, switching is changing one string.
Free tier per provider — HF passes through provider-specific free credits for some accounts.

When to go direct

If you need provider features HF doesn't surface (custom JSON modes, adapter routing, region pinning), you'll occasionally drop down to the provider's native SDK. The InferenceClient escape hatch is the OpenAI-compatible mode — next lesson.

Code

Discover which providers serve a model·python

from huggingface_hub import HfApi

api = HfApi()
info = api.model_info("meta-llama/Llama-3.1-8B-Instruct")

# Inference providers list shows up under .inference, when present
print("inference field:", getattr(info, "inference", None))

# More reliably, check the provider tags
providers = [t for t in (info.tags or []) if t.startswith("inference-providers:")]
print(providers)

Provider switch with one string change·python

from huggingface_hub import InferenceClient

def ask(provider):
    client = InferenceClient(model="meta-llama/Llama-3.1-8B-Instruct", provider=provider)
    return client.chat_completion(
        messages=[{"role": "user", "content": "ping"}],
        max_tokens=10,
    ).choices[0].message.content

for p in ["hf-inference", "together", "fireworks-ai"]:
    try:
        print(p, "->", ask(p))
    except Exception as e:
        print(p, "->", type(e).__name__)

External links

Exercise

Pick a popular model. Identify three providers that serve it. Run the same prompt 5 times against each. Track: avg latency, p95 latency, and any output differences (response shape, content). Compute a $/1k-tokens estimate based on each provider's published price.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousInferenceClient: One Object, Many Backends Next →Streaming, Tool Use, and Structured Output

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.