Three numbers, one decision
For any prompt change you can move along three axes: cost (tokens, $), quality (golden-set score), and latency (p95 ms). Improving all three at once is rare; usually you trade one against another. Make the trade explicit.
Useful comparisons
- v_new improves quality by 4% but increases p95 latency by 800ms — worth it for batch jobs, not for chat.
- v_new reduces cost by 40% but loses 1% on the golden set — usually worth it; verify the lost cases are not the high-stakes ones.
- v_new is faster (smaller model) and ties on quality on routine inputs but degrades on hard ones — route hard inputs to the bigger model.
The dashboard
Eval runs should produce one chart per prompt: cost vs quality, latency vs quality. Decisions become visible. Stakeholders can see why you chose what you chose.