Context Caching — How and Where to Use It

The largest production cost lever

Modern APIs let you mark parts of the prompt as cached — the system prompt, a long document, a tool catalog. Cache hits are billed at a fraction of the input cost (Anthropic: 10% read, 25% write, with TTLs in the minutes-to-hours range). For a system that calls the same prompt 1M times a day, caching is the difference between a $60K/month bill and a $6K/month bill.

What to cache

System prompt — almost always, unless it's tiny.
Long documents that change rarely — manuals, reference data.
Tool definitions — schemas and descriptions, especially for many-tool agents.
Few-shot examples — if the example block is large and stable.

What not to cache

The user's current message.
Any per-request data.
RAG results (different per query).

The order trap

Cache breakpoints are positional. Anything before the breakpoint is cached; anything after is fresh. If you put a tiny system prompt and then a huge document, set the breakpoint after the document. If you reorder the prompt, the cache is invalidated. Decide cache layout once, then keep it stable.

Code

Anthropic prompt caching·python

client.messages.create(
    model="claude-opus-4-7",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LARGE_REFERENCE_DOC,  # 50k tokens, rarely changes
            "cache_control": {"type": "ephemeral"}
        },
        {
            "type": "text",
            "text": "You are a contract analyst. ..."
        }
    ],
    messages=[{"role": "user", "content": user_question}]
)

Context Caching — How and Where to Use It

The largest production cost lever

What to cache

What not to cache

The order trap

Code

External links

Exercise

Progress

Comments 0