Cost Engineering — Where the Money Goes

The bill has predictable shape

For most prompt-using systems in 2026, the cost is dominated by input tokens (because context is large), then reasoning tokens (when extended thinking is on), then output tokens, then everything else. Caching, model tier choice, and reasoning routing are the three biggest levers.

Levers in order of impact

Prompt caching — 5–10× reduction on stable prefixes.
Model-tier routing — send routine requests to mid-tier; reserve top-tier for hard ones.
Reasoning budget routing — only enable reasoning when needed.
Context trimming — RAG top-K tighter, document compression, dropping turns from history.
Output discipline — schema-bound outputs are smaller than free-form ones.
Provider competition — periodic comparison; switch where appropriate.

What not to optimize for

Don't trade quality for cost without measuring the quality drop.
Don't optimize average cost when tail cost is the issue (one slow long-context call drowns 1,000 cheap ones).
Don't optimize cost without optimizing operational complexity.

Code

Cost attribution per request·python

@dataclass
class CostBreakdown:
    input_tokens: int
    cache_read_tokens: int
    cache_write_tokens: int
    output_tokens: int
    thinking_tokens: int

    def total_usd(self, rates) -> float:
        return (
            self.input_tokens     * rates.input_per_tok
          + self.cache_read_tokens * rates.cache_read_per_tok
          + self.cache_write_tokens * rates.cache_write_per_tok
          + self.output_tokens    * rates.output_per_tok
          + self.thinking_tokens  * rates.thinking_per_tok
        )

Cost Engineering — Where the Money Goes

The bill has predictable shape

Levers in order of impact

What not to optimize for

Code

External links

Exercise

Progress

Comments 0