Function Call Streaming

~22 min · streaming, function-calls

Level 0Tokenizer

0 XP0/54 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

When the model decides to call tools during streaming, you receive tool call fragments via delta.tool_calls. These must be accumulated by index before execution.

Key insight: tool call arguments arrive as string fragments. You must buffer them by index and parse the complete JSON only after finish_reason indicates "tool_calls".

Why fragments and not whole arguments

The model generates the JSON one token at a time. Each token becomes a chunk. So you'll see arguments arrive as {"loc, then ation":", then Seoul"} — three fragments that together form valid JSON. json.loads on any single fragment crashes; json.loads on the concatenation succeeds.

The accumulator pattern is: keep an arguments_str per tool_call_id, append every fragment to it, and only parse when the stream signals tool-call completion (or at the end of the stream). The SDK abstracts this; in raw httpx you write the accumulator yourself.

Code

Reassembling streamed tool arguments·python

import json

stream = client.chat.completions.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather",
            "parameters": {
                "type": "object",
                "properties": {"location": {"type": "string"}},
                "required": ["location"],
            }
        }
    }],
    stream=True,
)

tool_call_accumulator = {}
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.tool_calls:
        for tc in delta.tool_calls:
            idx = tc.index
            if idx not in tool_call_accumulator:
                tool_call_accumulator[idx] = {"id": "", "name": "", "arguments": ""}
            if tc.id:
                tool_call_accumulator[idx]["id"] = tc.id
            if tc.function.name:
                tool_call_accumulator[idx]["name"] = tc.function.name
            if tc.function.arguments:
                tool_call_accumulator[idx]["arguments"] += tc.function.arguments

# Process accumulated tool calls
for tc in tool_call_accumulator.values():
    args = json.loads(tc["arguments"])
    print(f"Tool: {tc['name']}({args})")

External links

Function calling streaming

Exercise

Run a tool-using stream and log every chunk's tool_calls fragment. Concatenate them to reproduce the full arguments JSON. Verify json.loads succeeds only at the end.