Client Design — Retry, Backoff, Circuit Breakers

"A production HTTP client isn't httpx.get(url). It's httpx.get(url) wrapped in retry-with-backoff (for transient failures), respect-Retry-After (for rate limits), idempotency-aware (don't retry POSTs blindly), and circuit-breaker-protected (so a flapping upstream doesn't tank your whole service)."

The Three Layers of Production Client Resilience

1. Retry on transient failure. Network glitches, 5xx errors, 429 rate limits — all temporary. Retry with exponential backoff: 1s, 2s, 4s, 8s, capped at some maximum. Add jitter so a thundering herd of clients don't all retry in lockstep.

2. Idempotency awareness. Only retry methods that are idempotent (GET, HEAD, OPTIONS, PUT, DELETE) automatically. POST/PATCH need an Idempotency-Key header to be safe; without one, fail and bubble up. Retry decisions must reflect the method.

3. Circuit breaker for sustained failure. If an upstream service is fundamentally down (10+ consecutive 5xx, latency spike), STOP calling it for a cooldown period. "Open" the circuit; fail fast (return cached, return error) for subsequent requests; after cooldown, send one probe; if it works, "close" and resume.

The Exponential Backoff Formula

Standard exponential backoff: delay = min(base * 2^attempt, max_delay). With jitter to avoid synchronization: delay = random_uniform(0, base * 2^attempt) (full jitter) or delay = base * 2^attempt + random(0, base) (equal jitter).

# Attempt 0:  ~1s
# Attempt 1:  ~2s
# Attempt 2:  ~4s
# Attempt 3:  ~8s
# Attempt 4:  ~16s (capped at max_delay)
# Total elapsed: ~31s before final failure

Jitter is non-negotiable. Without it, if 10,000 clients all hit a 500-returning endpoint and start retrying at the same moment, they all retry in unison — guaranteeing the upstream stays overloaded. Random delay desynchronizes them.

The Retry Decision Matrix

Response	Retry?	Notes
Network error (timeout, connection refused)	Yes (if idempotent)	Transport-level; safe to retry idempotent methods
2xx	No	Success; you're done
3xx	Follow redirect	Not a retry; protocol-level navigation
4xx (except 408, 429)	No	Client error; retry won't help
408 Request Timeout	Yes (if idempotent)	Transient server-side timeout
429 Too Many Requests	Yes, after Retry-After	Respect the header
5xx	Yes, with backoff	Server issue; may resolve

Circuit Breaker States

CLOSED — normal. Requests flow; failures are counted.
OPEN — too many recent failures. Requests short-circuit without calling upstream. Return cached value, queue for later, or return a fast error.
HALF-OPEN — cooldown elapsed. Send ONE probe request. If success, transition to CLOSED. If failure, back to OPEN.

Tuning: typical thresholds are 50% failure rate over the last 20 requests opens the circuit; 30-60 second cooldown before half-open. Tune per upstream; payment APIs deserve tighter limits than image services.

Retry + backoff handles transient failure; circuit breakers handle sustained failure. Without retry, every transient blip surfaces as an error. Without circuit breakers, sustained upstream failure cascades into your service. Both are mandatory in production; libraries (tenacity, polly, resilience4j) make them one-decorator easy.

cwkPippa's Resilience Patterns

cwkPippa's brain adapters wrap upstream LLM calls (Claude, OpenAI, Gemini) with retry on 5xx and 429, respecting Retry-After. The brain fallback chain (Codex → Claude → Gemini) acts as a manual circuit breaker — when one provider's 5xx rate spikes, the heartbeat scheduler swaps to the next brain for the next round. No formal Hystrix-style breaker; the fallback chain is effectively a poor-man's version. Production at proper scale would benefit from a real circuit breaker library, but for two-user volume the manual fallback suffices.

Code

Tenacity: exponential backoff + jitter + retry on specific exception types·python

# Retry with backoff and jitter — tenacity library
from tenacity import retry, wait_exponential_jitter, stop_after_attempt, retry_if_exception_type
import httpx

# Idempotent method — safe to retry on transient failures
@retry(
    wait=wait_exponential_jitter(initial=1, max=30, jitter=1),
    stop=stop_after_attempt(5),
    retry=retry_if_exception_type((httpx.TransportError, httpx.HTTPStatusError)),
)
def fetch_user(uid: str) -> dict:
    resp = httpx.get(f'https://api.example.com/users/{uid}')
    if resp.status_code >= 500:
        resp.raise_for_status()  # triggers retry
    if resp.status_code == 429:
        # Respect Retry-After before retrying
        import time
        wait = int(resp.headers.get('Retry-After', 5))
        time.sleep(wait)
        raise httpx.HTTPStatusError('rate limited', request=resp.request, response=resp)
    resp.raise_for_status()
    return resp.json()

pybreaker: circuit-breaker around upstream HTTP calls·python

# Simple circuit breaker — pybreaker library
import pybreaker
import httpx

breaker = pybreaker.CircuitBreaker(
    fail_max=5,           # OPEN after 5 consecutive failures
    reset_timeout=60,     # 60s cooldown before HALF-OPEN probe
    exclude=[httpx.HTTPStatusError],  # don't count 4xx as failure (client issue)
)

@breaker
def call_upstream(url: str) -> dict:
    resp = httpx.get(url, timeout=10.0)
    resp.raise_for_status()
    return resp.json()

# Usage
try:
    data = call_upstream('https://api.example.com/users/42')
except pybreaker.CircuitBreakerError:
    # Circuit is OPEN — fail fast, return cached, queue for later
    data = cached_or_fallback()
except httpx.HTTPError as e:
    # Upstream error counted toward circuit threshold
    raise

Production-shape: retry + backoff + jitter + idempotency + Retry-After·python

# Idempotency-aware retry decorator (combining retry + idempotency check)
import httpx
import time, random

IDEMPOTENT_METHODS = {'GET', 'HEAD', 'OPTIONS', 'PUT', 'DELETE'}

def call_with_resilience(method: str, url: str, idempotency_key: str | None = None, **kwargs):
    """Combine retry + backoff + idempotency awareness + Retry-After respect."""
    method = method.upper()
    # POST/PATCH only retry if caller provides Idempotency-Key
    can_retry = method in IDEMPOTENT_METHODS or idempotency_key is not None
    if idempotency_key:
        kwargs.setdefault('headers', {})['Idempotency-Key'] = idempotency_key

    max_attempts = 5 if can_retry else 1
    for attempt in range(max_attempts):
        try:
            with httpx.Client(timeout=10.0) as c:
                resp = c.request(method, url, **kwargs)
            if resp.status_code == 429:
                wait = int(resp.headers.get('Retry-After', 1))
                time.sleep(wait)
                continue
            if 500 <= resp.status_code < 600:
                if not can_retry:
                    return resp
                wait = min(2 ** attempt, 30) + random.random()
                time.sleep(wait)
                continue
            return resp
        except httpx.TransportError:
            if not can_retry:
                raise
            wait = min(2 ** attempt, 30) + random.random()
            time.sleep(wait)
    return resp  # all attempts exhausted

Exercise

Wrap a real HTTP client call (any API you have) with tenacity for retry+backoff+jitter on 5xx and TransportError. Test it by deliberately pointing at a broken URL (e.g., unused localhost port) and verify exponential backoff (~1s, 2s, 4s gap between attempts). Then add pybreaker around the same call with fail_max=3, reset_timeout=10. After 3 consecutive failures, the breaker should OPEN and subsequent calls should raise CircuitBreakerError without hitting the network. Wait 10s for cooldown, fix the URL, verify one successful probe transitions to CLOSED.

Hint

Tenacity's wait_exponential_jitter handles the math; you just decorate. The backoff verification: log timestamps of each attempt and compute deltas — they should roughly double. For pybreaker, after 3 failures the next 7-10 calls (within reset_timeout) should be near-instant (CircuitBreakerError without network). The state transition CLOSED → OPEN → HALF-OPEN → CLOSED is the canonical pattern; you're building the same primitive every production system uses.

Client Design — Retry, Backoff, Circuit Breakers

The Three Layers of Production Client Resilience

The Exponential Backoff Formula

The Retry Decision Matrix

Circuit Breaker States

cwkPippa's Resilience Patterns

Code

External links

Exercise

Progress

Comments 0