Switching Providers — What Breaks and How to Plan

~14 min · providers, migration

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

Things that break on a swap

Tool-call format — argument parsing, role names, parallel-call shape.
JSON enforcement mode — strict schemas vs JSON mode vs tool-as-schema.
System prompt placement — top-level vs message vs system_instruction.
Token counting — counts off; cost forecasts wrong.
Refusal calibration — same input refused on one, accepted on another.
Reasoning interface — extended thinking vs reasoning_effort vs thinking_budget.
Multimodal input format — base64 vs URL vs file id.

Pre-migration checklist

Run your golden set on both providers.
Compare cost and latency, not just quality.
Identify any prompt-side changes needed (tag style, response_format).
Test the eval suite end-to-end on the new provider.
Plan for parallel runs during cutover; don't switch atomically on day one.

Code

Provider comparison harness·python

for provider in ["anthropic", "openai", "gemini"]:
    results[provider] = run_eval(
        cases=GOLDEN_SET,
        runner=PROVIDER_RUNNERS[provider],
        prompt_path=f"prompts/support/v8.{provider}.md",
    )

print_pivot(results)  # accuracy, cost, latency by provider

External links

Anthropic vs OpenAI vs Gemini comparison guides

Exercise

Run your golden set on three providers using your existing prompt. Identify the most provider-specific cost/quality differences. Decide where each provider is the best fit.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousOpen-Source Models — Llama, Qwen, Mistral Next →Modality Differences — Vision, Audio, Files

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.