Rate Limiting and Backpressure

~12 min · production, rate-limits

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

You are rate limited; so are your users

Provider rate limits sit in three layers — RPM, TPM, and concurrency. Your application has to respect them, recover gracefully when they're hit, and propagate backpressure to your users so they don't pile up requests.

Tactics

Smoothing — token bucket / leaky bucket on egress, so bursts don't 429 you.
Retries with jitter — exponential backoff with randomization, capped attempts.
Per-user limits — protect against runaway clients; surface limits to your UI.
Backpressure — when downstream is hot, return 429 to your callers; don't queue silently.
Tier-aware routing — heavy users to a higher-tier API key, light users to a shared pool.

What to expose to users

Useful 429s ("slow down — try again in 30 seconds").
Quota dashboards for paying customers.
UI affordances (disabled buttons, banners) when rate limits are near.

Code

Token bucket on egress·python

from anyio import sleep
from asyncio import Lock
from time import monotonic

class TokenBucket:
    def __init__(self, rate_per_sec: float, capacity: int):
        self.rate, self.capacity = rate_per_sec, capacity
        self.tokens = capacity
        self.last = monotonic()
        self.lock = Lock()

    async def take(self, n: int = 1):
        async with self.lock:
            while True:
                now = monotonic()
                self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
                self.last = now
                if self.tokens >= n:
                    self.tokens -= n
                    return
                await sleep((n - self.tokens) / self.rate)

External links

Exercise

Add a token-bucket smoother to one client. Stress-test by firing 10× your provider's RPM. Verify no 429s reach the provider, and clients see steady latency instead of bursts of failure.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousFallback Chains — Multiple Models, One Promise Next →Rollouts and Canaries

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.