Testing Agent Behavior

What you can unit-test

You cannot deterministically unit-test 'the model picks the right tool', but you can test: tool definitions are valid JSON Schema, hooks return the expected verdict for each pattern, the permission handler approves/denies correctly, and the audit logger emits the right shape. The deterministic seams around the model are testable normally.

What needs integration tests

End-to-end behavior — does the agent solve the task, does it stay within sandbox, does it produce correct citations — needs integration tests against a date-pinned model. Write them as scenarios with expected behavioral assertions ('produces a one-line summary', 'never calls Write', 'cites at least one source').

Regression tests for safety

The most important tests are the ones that prove your defenses fire. Inject a known prompt-injection string in a fixture; assert the agent surfaces it. Provide a deny-listed Bash command in a prompt; assert the hook refuses. These tests catch the day someone removes a safeguard 'because it was failing'.

Principle: Test the model behavior coarsely; test the safeguards rigorously. The model can drift; your invariants must not.

Code

Hook unit test·python

import pytest
from my_agent.hooks import block_destructive_bash

def bash_call(command: str) -> dict:
    return {"hook_event_name": "PreToolUse", "session_id": "s1", "transcript_path": "", "cwd": "/tmp",
            "tool_name": "Bash", "tool_input": {"command": command}, "tool_use_id": "t1"}

@pytest.mark.asyncio
@pytest.mark.parametrize("cmd,denied", [
    ("ls -la", False),
    ("rm -rf /tmp/foo", True),
    ("dd if=/dev/zero of=/tmp/zero", True),
    ("echo hello", False),
])
async def test_block_destructive_bash(cmd, denied):
    out = await block_destructive_bash(bash_call(cmd), "t1", {"signal": None})  # the real callback
    decision = out.get("hookSpecificOutput", {}).get("permissionDecision")
    assert (decision == "deny") == denied

End-to-end behavior assertion·python

import pytest
from claude_agent_sdk import AssistantMessage, ClaudeAgentOptions, TextBlock, ToolUseBlock, query

@pytest.mark.asyncio
async def test_summarizer_stays_in_sandbox(tmp_path):
    (tmp_path / "input.md").write_text("hello world\n")
    options = ClaudeAgentOptions(
        cwd=str(tmp_path),
        tools=["Read"],          # Write and Bash do not exist in this run
        allowed_tools=["Read"],  # and Read needs no prompt
        model="claude-haiku-4-5-20251001",
        # hermetic: no user or project settings, no auto-memory, and no MCP server
        # you did not name here leak into the run (a big connector list alone can
        # overflow a small model's window before the test says a word)
        setting_sources=[],
        env={"CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1"},
        strict_mcp_config=True,
    )
    text, used = [], []
    async for message in query(prompt="Summarize input.md", options=options):
        if isinstance(message, AssistantMessage):  # text lives in content blocks, not message.text
            for block in message.content:
                if isinstance(block, TextBlock):
                    text.append(block.text)
                elif isinstance(block, ToolUseBlock):
                    used.append(block.name)
    assert "hello" in "".join(text).lower()
    assert set(used) <= {"Read"}  # the transcript itself shows no Write or Bash call

What you can unit-test

What needs integration tests

Regression tests for safety

Code

External links

Exercise

Progress

Comments 0