Probing Cache Hits and Hidden State

Cache state is operational state

If your provider exposes cache hit metrics, treat them like first-class telemetry. A workflow you thought was cache-friendly may be silently rebuilding the prefix every turn — the cost graph will tell you, but only if you log it. Cache hit ratio is a leading indicator for both cost and behavioral consistency.

Probe the cache deliberately

Send the same prefix twice in quick succession; check whether the second request reports a cache hit. Send the prefix with one byte changed; check whether the cache misses. These are five-minute tests that reveal whether your caching strategy actually works as intended.

Hidden state lives in caches and replays

Some 'mysterious' model behaviors trace back to cache state — a prefix from a previous experiment is still warm and biasing answers. Cold-restart the session, force-bust the cache (change a single byte in the prefix), and see whether the behavior persists. If not, the bug was hidden cache state, not the model.

Code

Cache hit ratio in telemetry·python

def cache_hit_ratio(usage):
    fresh = usage.input_tokens - getattr(usage, "cache_read_input_tokens", 0)
    cached = getattr(usage, "cache_read_input_tokens", 0)
    total = fresh + cached
    return cached / total if total else 0.0

if cache_hit_ratio(response.usage) < 0.3 and prefix_should_have_hit:
    log_cache_miss_anomaly(response)

Probe sequence·text

1. Send prefix P + question Q1. Record cached_tokens.
2. Send prefix P + question Q2 (same P). Expect cached_tokens > 0 on second call.
3. Send prefix P' (P with one byte changed) + question Q3. Expect cached_tokens = 0.
If step 2 shows no hit, your prefix is not actually byte-stable -> hunt the cache buster.

Cache state is operational state

Probe the cache deliberately

Hidden state lives in caches and replays

Code

External links

Exercise

Progress

Comments 0