The trace tells you where it went wrong
When a reasoning model produces a wrong answer, the trace is the evidence. Most teams never read the trace — they tweak the prompt by feel, ship a fix that may or may not work, and move on. Reading the trace is slower up front and faster across a quarter.
What to look for
- Wrong premise — model anchored on a phrase from the input that wasn't load-bearing.
- Skipped step — model jumped to a conclusion without checking a constraint.
- Repeated step — model re-stated the same sub-result without progress.
- Verifier rejection — model proposed a plan that the verifier killed; the next branch wasn't materially different.
- Token-budget exhaustion — trace ran out of room before the answer.
Trace as eval input
Once you have a few annotated bad traces, the failure modes become test cases. Add them to your eval set. The next prompt change that fixes them counts as progress; one that doesn't, doesn't.