An LLM agent makes decisions in a loop: pick a tool, execute, observe the result, decide the next step. The final answer matters, but so does how it got there. Did the agent use the right tool? Did it loop unnecessarily? Did it succeed at acceptable cost?
Agent-specific metrics
Task success — did the agent achieve the goal? (binary or graded)
Tool-call accuracy — were the right tools chosen, with the right arguments?
Trajectory efficiency — number of steps to completion vs optimal.
Cost — total tokens, total tool calls, total wall-clock time.
Recovery — when an early step fails, does the agent recover?
Goal alignment — does the agent pursue the user's actual goal, or drift?
The trajectory eval pattern
Capture every step the agent takes: thought, tool call, tool result, next thought. Score (a) the final outcome and (b) the trajectory quality. A "perfect answer reached after 47 unnecessary tool calls" should fail your eval, not pass.
Principle: Agent evaluation is trace-aware by necessity. If you only score final outputs, you miss the half of the work where agents fail invisibly — by being slow, expensive, or confused mid-trajectory.
Sandbox for safe execution
Agents that run code, browse, or modify state need a sandbox. Inspect AI's Docker sandbox is one option; SWE-bench-style ephemeral repo clones are another. Never run an unsandboxed eval on a system that touches real infrastructure.
Code
Trajectory capture and step-level scoring·python
from dataclasses import dataclass, field
@dataclass
class AgentStep:
thought: str
tool: str
args: dict
result: str
cost: float # tokens or wall-time
@dataclass
class AgentTrajectory:
task: str
steps: list[AgentStep] = field(default_factory=list)
final_output: str = ""
success: bool = False
def trajectory_metrics(traj: AgentTrajectory, optimal_steps: int):
return {
"success": traj.success,
"n_steps": len(traj.steps),
"efficiency": optimal_steps / max(len(traj.steps), 1),
"total_cost": sum(s.cost for s in traj.steps),
"unique_tools": len({s.tool for s in traj.steps}),
}
Tool-call accuracy with DeepEval·python
from deepeval.metrics import ToolCallAccuracyMetric, AgentGoalAccuracyMetric
from deepeval.test_case import LLMTestCase, ToolCall
case = LLMTestCase(
input="What is the weather in Tokyo right now?",
actual_output="It is 18°C and partly cloudy in Tokyo.",
tools_called=[
ToolCall(name="get_weather", arguments={"city": "Tokyo"}),
],
expected_tools=[
ToolCall(name="get_weather", arguments={"city": "Tokyo"}),
],
)
assert_test(case, [ToolCallAccuracyMetric(), AgentGoalAccuracyMetric()])
Inspect AI agent eval with sandbox·python
from inspect_ai import Task, task
from inspect_ai.solver import use_tools, generate
from inspect_ai.tool import bash, python
from inspect_ai.scorer import includes
@task
def coding_agent_task():
return Task(
dataset=[Sample(
input="Find the line count of all .py files under src/ and report the total.",
target="42",
)],
solver=[
use_tools([bash(), python()]),
generate(),
],
scorer=includes(),
sandbox="docker", # tools execute in isolated container
message_limit=20, # caps trajectory length
)
Pick one agent task in your product. Capture the trajectory of 20 runs (thought / tool / args / result). Score success, step count, and cost separately. The cases with success=True but high step count are your next prompt-engineering targets.
Progress
Progress is local-only — sign in to sync across devices.