Fallback Chains — Multiple Models, One Promise

~14 min · production, fallback

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

Single-model dependence is fragile

The model provider has outages. Rate limits hit. Capacity tiers fluctuate. A serious system has a fallback chain so the user keeps getting answers even when the primary path is unavailable.

The chain shape

Primary — best quality, your default.
Secondary — different provider, similar capability. Different infrastructure = different outage profile.
Tertiary — cheaper or local model, lower quality but available. Last-resort.
Cached / static fallback — for some prompts, a cached "sorry, try again" with a useful pointer is better than a blocked request.

Wiring

Detect failure: 5xx, timeout, content-policy refusal on a benign request, validator-fail-and-retry-failed.
Step the chain on failure; log every step.
Set hop budget: 3 hops max; after that, fail clean to the user with a structured error.
Track per-step quality — if your tertiary is regularly serving traffic, your eval should include it.

Code

Fallback chain·python

CHAIN = [
    {"provider": "anthropic", "model": "claude-opus-4-7"},
    {"provider": "openai",    "model": "gpt-5.5"},
    {"provider": "local",     "model": "llama4"},
]

async def call_with_fallback(req):
    last_err = None
    for step in CHAIN:
        try:
            return await call(step, req)
        except (TimeoutError, RateLimited, RefusalOnBenign) as e:
            last_err = e
            log.warning("fallback step failed", extra={"step": step, "err": str(e)})
    raise ServiceUnavailable from last_err

External links

Exercise

Build a 2-step fallback chain across two providers. Force the primary to fail (network rule, kill switch). Verify the secondary serves the request. Verify the fallback is logged and metric-tagged.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousTelemetry — What to Measure in Prod Next →Rate Limiting and Backpressure

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.