Streaming Output — Partial JSON and the Pipeline

~14 min · outputs, streaming

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

Streaming changes the parsing problem

When you stream a structured output, you get partial JSON: {"verdict": "app… and so on. A naïve json.loads() fails on every chunk. The pipeline has to either buffer until done or parse incrementally with a streaming JSON parser.

Why stream

UX — show the user something within 200ms instead of after 8s.
Pipelining — start downstream work on the first complete field instead of after the last.
Cancellation — kill the request as soon as the user navigates away or the verifier rejects.

Failure modes specific to streaming

Truncation halfway through a JSON output (max_tokens, network drop).
Tool calls that arrive in pieces (start, deltas, stop).
Reasoning tokens streaming separately from output tokens.
Cancellation races — if the consumer disconnects, your code keeps running unless you wire cancellation through.

Code

Streaming JSON safely·python

import json_stream

buffer = []
with client.messages.stream(
    model="claude-opus-4-7",
    messages=[...],
    response_format={"type": "json_object"},
) as stream:
    for delta in stream.text_stream:
        buffer.append(delta)
        # incremental: try to parse, ignore JSONDecodeError
        try:
            partial = json_stream.loads("".join(buffer))
            on_partial(partial)  # update UI as fields arrive
        except Exception:
            pass
final = json.loads("".join(buffer))

External links

Exercise

Convert a non-streaming structured-output endpoint to streaming. Make the UI display fields as they arrive. Test cancellation: confirm the request stops billing tokens within 200ms.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousTool Errors — Letting the Model Recover Next →Output Repair Loops

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.