Cost, Latency, and the Cache Asymmetry

Long context is not free just because it fits

Large contexts increase prefill work: the model must process a long prefix before generating the first token. Even when a provider supports a huge window, latency can become unacceptable for interactive collaboration. Prefill compute scales close to quadratic with token count — a 1M-token request is not 5x slower than a 200K one, it is much worse.

The three cost asymmetries

1. Input vs output — output is typically 3-5x the price of input across major providers. 2. Cached vs fresh input — cached input often costs 10-90% less than fresh input (Anthropic's cached reads are ~90% off; OpenAI and Gemini have similar tiered discounts). 3. Standard vs extended context — past the long-context threshold (often 200K), per-token rates jump.

Worked example

A 50K-token codebase review producing 8K tokens of feedback at illustrative Sonnet pricing ($3/M input, $15/M output, $0.30/M cached) costs $0.27 single-shot, ~$4.70 across 10 follow-ups without caching, but only ~$1.60 with prompt caching enabled. Same conversation, 3x cheaper, just by structuring the prefix to be reusable.

Interactive vs batch

A giant context can be perfect for one-off analysis and terrible for a tight edit loop. When the human is collaborating live, responsiveness is part of quality. Batch jobs forgive latency; interactive sessions do not.

Code

Cost shape sketch·python

scenarios = {
  "single_shot_50k":          {"input": 0.150, "output": 0.120, "total": 0.27},
  "ten_turns_no_cache":       {"input": 3.50,  "output": 1.20,  "total": 4.70},
  "ten_turns_with_caching":   {"input": 0.40,  "output": 1.20,  "total": 1.60},
}
# Caching is the single biggest cost lever after prompt design.

Identify stable prefix vs variable tail·yaml

stable_prefix:
  - system rules
  - tool schemas
  - reusable examples
  - corpus snippets you re-read each turn
variable_tail:
  - this turn's user question
  - latest diff
  - latest tool result
  - timestamp (if truly needed)

Exercise

Take a repeated workflow, such as reviewing the same repo across many questions. Identify the stable prefix and the variable tail. Estimate cost with and without caching across 20 turns.

Hint

Stable prefix: rules, tool schemas, corpus. Variable tail: this turn's question, latest diff, latest command output.