Streaming, Tool Use, and Structured Output

~30 min · inference, streaming

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Streaming is one keyword

client.chat_completion(..., stream=True) returns a generator yielding ChatCompletionStreamOutput objects. Each chunk has .choices[0].delta.content. The shape mirrors the OpenAI streaming format, so the same UI code that handles OpenAI streams handles HF.

Tool calling

Pass tools=[{...}] as JSON-schema-shaped dicts. The model responds with tool_calls in the assistant message. You execute them locally and append the results as {"role": "tool", ...} messages, then loop. The contract is OpenAI-compatible: same dict shapes.

Structured output

For JSON-mode, three approaches: (1) prompt the model and validate, (2) response_format={"type": "json_object"} if the provider supports it, (3) a Pydantic-driven library like outlines or instructor. Approach 3 is the most reliable across providers.

Code

Streaming chat·python

from huggingface_hub import InferenceClient

client = InferenceClient(model="meta-llama/Llama-3.1-8B-Instruct", provider="hf-inference")

stream = client.chat_completion(
    messages=[{"role": "user", "content": "Count from 1 to 5 slowly."}],
    max_tokens=80,
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)
print()

Tool call loop·python

from huggingface_hub import InferenceClient
import json

client = InferenceClient(model="meta-llama/Llama-3.1-70B-Instruct", provider="together")

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

messages = [{"role": "user", "content": "What's the weather in Seoul?"}]
resp = client.chat_completion(messages=messages, tools=tools, max_tokens=120)
choice = resp.choices[0]

if choice.message.tool_calls:
    call = choice.message.tool_calls[0]
    args = json.loads(call.function.arguments)
    # Pretend we ran the tool
    tool_result = {"city": args["city"], "temp_c": 22, "conditions": "clear"}
    messages.append(choice.message)
    messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(tool_result)})
    final = client.chat_completion(messages=messages, max_tokens=120)
    print(final.choices[0].message.content)

External links

Exercise

Wire up a streaming chat loop that prints chunks as they arrive. Then add a single tool (get_time returning a fixed string). Verify the model decides whether to call it. Note: how the streaming behavior changes when the response is a tool call vs a plain message.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousProvider Routing and the Pricing Surface Next →OpenAI-Compatible Endpoints

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.