Observability: Watching for Silent Decay

~22 min · ops, monitoring

Level 0Scout

0 XP0/41 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

What can degrade silently

Cache hit rate drops — pipeline regression, cache key change, cache wiped
Top-k score distribution shifts — corpus drift, embedding model regression
Empty result rate climbs — filter too aggressive, score floor too high, or the corpus has gaps
Latency creeps up — index getting bigger, autovacuum behind, GPU contention
LLM 'I don't know' rate climbs — retrieval quality decay reaches the LLM

The minimum dashboard

Six counters, plotted over time, will catch most silent failures before users do:

Queries per minute
Median + p95 retrieval latency (per stage)
Median + p95 top-1 score
Empty-result rate (no candidate above the floor)
Embedding cache hit rate
LLM 'I don't know' rate

Code

Emit retrieval metrics with prometheus-client·python

from prometheus_client import Histogram, Counter

QUERY_LATENCY = Histogram('retrieval_latency_seconds', 'End-to-end retrieval latency', ['stage'])
TOP1_SCORE    = Histogram('retrieval_top1_score', 'Top-1 similarity score')
EMPTY_RESULT  = Counter('retrieval_empty_total', 'Queries that returned no candidates above floor')

def instrumented_retrieve(question: str, k: int = 5, min_score: float = 0.3):
    with QUERY_LATENCY.labels(stage='embed').time():
        qvec = embed(question)
    with QUERY_LATENCY.labels(stage='search').time():
        results = query(NEW, qvec, k)
    above = [r for r in results if (1 - r['distance']) >= min_score]
    if not above:
        EMPTY_RESULT.inc()
    elif above:
        TOP1_SCORE.observe(1 - above[0]['distance'])
    return above

External links

Exercise

Wire the four most important counters (latency, top-1 score, empty-result rate, cache hit rate) into your retriever. Run the system for a day. Look at the chart. Pick the one number you would page yourself on at 3 a.m. — that is your SLO.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousVersioning Embeddings Without Breaking Production Next →Multi-Tenant Isolation

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.