Reading Tool-Call Traces for Context Clues

~23 min · tools, traces, diagnosis

Level 0Window Watcher

0 XP0/50 lessons0/13 achievements

0/100 XP to next level100 XP to go0% complete

Tool calls are diagnostic gold

When behavior is mysterious, the tool-call trace often explains it. The model called read("file_X") three times this session — that means file_X was the working set. The model never called read("file_Y") — that means file_Y is not in the working context, regardless of whether you mentioned it. Trace patterns expose what the model actually paid attention to, not what you hoped it would.

Look for missing calls

The most useful trace observation is often a tool that was not called. If the rule says 'always validate before commit' and there is no run_tests call before the commit, the rule failed silently. The presence of a call is normal; the absence is the bug.

Trace patterns to watch

Repeated reads of the same file (working set is too small or eviction too aggressive). Cascading retrieval calls (the routing logic is failing). No tool calls at all on a complex task (the model is hallucinating from priors). Each pattern points at a different fix.

Code

Trace summary shape·json

{
  "session_id": "...",
  "tool_calls": [
    {"tool": "read", "args": {"path": "backend/auth.py"}, "turn": 3},
    {"tool": "read", "args": {"path": "backend/auth.py"}, "turn": 7},
    {"tool": "read", "args": {"path": "backend/auth.py"}, "turn": 14}
  ],
  "never_called": ["run_tests", "git_status"]
}

Trace audit prompt·text

Looking only at the tool-call trace from this session:
  - Which files were loaded most often?
  - Which expected calls never happened?
  - Which calls cascaded together (pattern)?
Use these observations to diagnose where context attention concentrated and where it failed.

External links

OpenAI Responses API reference

Exercise

Pick one of your AI sessions with tool calls. Summarize the trace: most-called tools, never-called tools, cascading patterns. Diagnose one behavior you noticed using only the trace.

Hint

The 'never-called' column is where the silent failures live.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousWhen the Model Disagrees with Itself Next →Probing Cache Hits and Hidden State

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.