Caching Strategies — What's Worth Caching

Three caches, three uses

Prompt cache (provider-side) — system prompts, large stable docs, tool definitions. Reduces input cost dramatically.
Response cache (your side) — for deterministic prompts on identical inputs (rare in chat, common in extraction / classification).
Semantic cache — match incoming queries to recent similar queries; serve cached response if similarity is high. Most useful in FAQ-shaped traffic.

Tradeoffs

Prompt cache: cheap, almost free, no quality risk. Default on for stable prefixes.
Response cache: needs determinism (temperature 0) and stable input hashing.
Semantic cache: powerful but risky — wrong matches return wrong answers confidently. Set a high similarity threshold and have a fallback.

Invalidation

Caches need a story for invalidation. Prompt cache TTLs are provider-managed (minutes to hours). Response cache invalidates on data updates. Semantic cache invalidates on policy changes. Put invalidation in the same diff as the change that requires it.

Code

Semantic cache with similarity threshold·python

def semantic_cache_lookup(query: str, threshold: float = 0.95):
    emb = embed(query)
    hit = vector_db.search(emb, k=1)
    if hit and hit[0].score >= threshold:
        return hit[0].cached_response
    return None

result = semantic_cache_lookup(query)
if result is None:
    result = call_model(query)
    vector_db.upsert(embed(query), cached_response=result)

Caching Strategies — What's Worth Caching

Three caches, three uses

Tradeoffs

Invalidation

Code

External links

Exercise

Progress

Comments 0