Cost, Rate Limits, and Retry Strategy

~26 min · inference, ops

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Cost is a system property, not a model property

The same model on Provider A vs Provider B vs your own TGI box has wildly different cost characteristics. The four axes:

$/1M input tokens, $/1M output tokens — provider-set, model-tier-set.
Cold start — managed providers may scale to zero. First request: 5-30s. Subsequent: ms.
Concurrency — rate limits per second, per minute, per day. Shared with everyone on free tier; isolated on paid plans; uncapped on self-hosted.
Latency — provider region, network distance, model size, batch contention.

Retry, backoff, and idempotency

Inference calls are not idempotent — sampling makes the same input produce different output. So: don't retry blindly. On 429, exponential backoff with jitter. On 5xx, retry once with the same input. On 4xx other than 429, fail loudly. The tenacity library handles the policy in 3 lines.

Code

Tenacity-based retry policy·python

from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
from huggingface_hub.errors import HfHubHTTPError

@retry(
    stop=stop_after_attempt(4),
    wait=wait_exponential(multiplier=1, min=1, max=30),
    retry=retry_if_exception_type(HfHubHTTPError),
    reraise=True,
)
def safe_chat(client, messages):
    return client.chat_completion(messages=messages, max_tokens=200)

Cost-aware token accounting·python

def cost_for(usage_input, usage_output, dollars_per_1m_input, dollars_per_1m_output):
    return (usage_input * dollars_per_1m_input + usage_output * dollars_per_1m_output) / 1_000_000

# After a chat_completion:
# resp.usage.prompt_tokens, resp.usage.completion_tokens are present in the OpenAI-shaped response

# Example: $0.20 / 1M in, $0.60 / 1M out
print("$", cost_for(1500, 600, 0.20, 0.60))

External links

Exercise

Build a small benchmark that calls the same model 100 times against your chosen provider. Record: latency per call, total tokens, total cost. Compute p50/p95/p99 latency and total $/100 calls. Add a tenacity retry on HfHubHTTPError, simulate a forced 429, verify backoff timing.