SDK Quirks — Retries, Streaming, Cancellation

The SDK is part of the prompt

Prompts behave differently when wrapped by different SDKs because the SDKs handle retries, streaming, cancellation, and tool loops in non-uniform ways. Surprises here are the most common production bugs in cross-provider code.

Retry behavior

Most SDKs retry on transient errors with exponential backoff. Defaults vary.
Retries on 429 are usually fine; retries on 5xx during a tool call may double-execute side-effects.
Disable SDK retries for non-idempotent operations; handle retry in your application code with idempotency keys.

Streaming

Event shapes differ — Anthropic's content_block_delta is not OpenAI's choices[0].delta.
Tool-call streaming emits arguments piecewise; you have to accumulate.
Errors mid-stream require careful handling — don't trust 'connection closed' as success.

Cancellation

Closing the stream stops billing on most providers — but only if you actually close it.
asyncio cancellation through async generators is a known landmine in Python; await on each chunk, don't asyncio.wait_for the iterator.
For agent loops with tool calls, cancellation should propagate into the active tool, or you'll keep paying for cleanup.

Code

Streaming with explicit cancellation (Python)·python

async def stream_response(req):
    async with client.messages.stream(**req) as stream:
        try:
            async for event in stream:
                yield event
        except asyncio.CancelledError:
            await stream.close()  # explicit close → server stops billing
            raise

SDK Quirks — Retries, Streaming, Cancellation

The SDK is part of the prompt

Retry behavior

Streaming

Cancellation

Code

External links

Exercise

Progress

Comments 0