Cost & Rate Limit Management

~22 min · cost, rate-limits, budgets

Level 0Tokenizer

0 XP0/54 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Track costs across requests and implement client-side rate limiting to stay within quotas.

The two questions you'll be asked in month two

'Why is the bill so high?' and 'Which customer is responsible?' Both have one answer: a per-tenant ledger. Persist (tenant_id, model, prompt_tokens, completion_tokens, reasoning_tokens, ts) for every call. Roll up daily by tenant. Now the bill conversation is data, not speculation.

The adaptive concurrency move is independent: read x-ratelimit-remaining-requests from every response. Below 20% remaining → halve concurrency. Above 80% → double concurrency. You stay below the wall instead of crashing into it. Both patterns are cheap to add and impossible to add later under fire.

Code

Per-tenant budget tracker·python

from dataclasses import dataclass

PRICING_PER_1M = {
    "gpt-5.4": {"prompt": 2.50, "completion": 15.00},
    "gpt-4.1": {"prompt": 2.00, "completion": 8.00},
    "gpt-4o-mini": {"prompt": 0.15, "completion": 0.60},
}

@dataclass
class CostTracker:
    total_cost_usd: float = 0.0
    total_prompt_tokens: int = 0
    total_completion_tokens: int = 0
    request_count: int = 0

    def record(self, model: str, usage: dict) -> float:
        p = usage.get("prompt_tokens", 0)
        c = usage.get("completion_tokens", 0)
        pricing = PRICING_PER_1M.get(model, {"prompt": 0, "completion": 0})
        cost = p * pricing["prompt"] / 1_000_000 + c * pricing["completion"] / 1_000_000
        self.total_cost_usd += cost
        self.request_count += 1
        return cost

Adaptive concurrency from headers·python

import asyncio, random

async def retry_with_backoff(func, max_retries=6, initial_delay=1.0):
    delay = initial_delay
    for attempt in range(max_retries + 1):
        try:
            return await func()
        except httpx.HTTPStatusError as e:
            if e.response.status_code not in (429, 500, 502, 503, 504):
                raise
            if attempt == max_retries: raise
            # Honor Retry-After header
            retry_after = e.response.headers.get("Retry-After")
            sleep_time = float(retry_after) if retry_after else delay * (0.5 + random.random())
            await asyncio.sleep(min(sleep_time, 60.0))
            delay = min(delay * 2, 60.0)

External links

Exercise

Build a CostLedger that records every API call (tenant_id, model, in_tokens, out_tokens, usd). Generate a daily-by-tenant report. Decide where you'd put a per-tenant cap and add it as a guard.