Fallback chain — multiple model, one promise

~14 min · production, fallback

Level 0수련생

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

Single-model 의존이 fragile

Model provider가 outage 가져. Rate limit hit. Capacity tier fluctuate. Serious system이 fallback chain 가져서 primary path unavailable해도 user가 답 keep getting.

Chain shape

Primary — best quality, 너의 default.
Secondary — different provider, similar capability. Different infrastructure = different outage profile.
Tertiary — cheaper나 local model, 낮은 quality인데 available. Last-resort.
Cached / static fallback — 일부 prompt에 cached "sorry, try again" with useful pointer가 blocked request보다 나을 수 있어.

Wiring

Failure detect: 5xx, timeout, benign request에 content-policy refusal, validator-fail-and-retry-failed.
Failure에 chain step; 모든 step log.
Hop budget set: 최대 3 hop; 그 후 user한테 structured error로 fail clean.
Per-step quality track — tertiary가 정기적으로 traffic serve하면 너의 eval에 포함.

Code

Fallback chain·python

CHAIN = [
    {"provider": "anthropic", "model": "claude-opus-4-7"},
    {"provider": "openai",    "model": "gpt-5.5"},
    {"provider": "local",     "model": "llama4"},
]

async def call_with_fallback(req):
    last_err = None
    for step in CHAIN:
        try:
            return await call(step, req)
        except (TimeoutError, RateLimited, RefusalOnBenign) as e:
            last_err = e
            log.warning("fallback step failed", extra={"step": step, "err": str(e)})
    raise ServiceUnavailable from last_err

External links

Exercise

두 provider 가로지르는 2-step fallback chain build. Primary force fail (network rule, kill switch). Secondary가 request serve verify. Fallback이 logged, metric-tagged 인지 verify.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousTelemetry — production에서 뭐 측정 Next →Rate limiting과 backpressure

이 페이지에서 버그를 발견하셨거나 피드백이 있으세요?문제 신고

🔔 답글 알림 (로그인 필요)

로그인 — 댓글을 남기려면 로그인해 주세요.

아직 댓글이 없어요. 첫 댓글을 남겨보세요.