Telemetry — What to Measure in Prod

~12 min · production, telemetry

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

Three dashboards every LLM endpoint needs

Health — error rate, refusal rate, p50/p95 latency, token usage, cost per call. Alerted on.
Quality — eval score, judge-flagged outputs, complaint rate, regeneration rate. Reviewed weekly.
Routing — which prompt version / model / tool path served which traffic. Debugging surface for incidents.

Specific metrics to watch

Output validation pass rate (schema match).
Token cost per resolved request (not just per call).
Tool-call retry rate.
Reasoning-token consumption distribution.
p95 latency by prompt version.

Alerts vs dashboards

Alert on the things that need a human now (cost spike, error rate, validation failure rate). Dashboard everything else. Wake people up for emergencies, not metrics.

Code

Per-request metric record·python

metrics.timing("llm.latency_ms", elapsed_ms, tags=[f"prompt:{ver}", f"model:{model}"])
metrics.incr("llm.requests", tags=[f"prompt:{ver}", f"outcome:{outcome}"])
metrics.gauge("llm.input_tokens", usage.input_tokens, tags=[f"prompt:{ver}"])
metrics.gauge("llm.output_tokens", usage.output_tokens)
metrics.gauge("llm.thinking_tokens", usage.thinking_tokens or 0)

External links

Exercise

Add prompt-version-tagged metrics for one endpoint. Build a dashboard showing pass rate, latency, and cost by version. Verify the dashboard updates when you deploy v_new.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousLogging Prompts and Responses Without Leaking Next →Fallback Chains — Multiple Models, One Promise

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.