Testing Agent Behavior

What you can unit-test

You cannot deterministically unit-test 'the model picks the right tool', but you can test: tool definitions are valid JSON Schema, hooks return the expected verdict for each pattern, the permission handler approves/denies correctly, and the audit logger emits the right shape. The deterministic seams around the model are testable normally.

What needs integration tests

End-to-end behavior — does the agent solve the task, does it stay within sandbox, does it produce correct citations — needs integration tests against a date-pinned model. Write them as scenarios with expected behavioral assertions ('produces a one-line summary', 'never calls Write', 'cites at least one source').

Regression tests for safety

The most important tests are the ones that prove your defenses fire. Inject a known prompt-injection string in a fixture; assert the agent surfaces it. Provide a deny-listed Bash command in a prompt; assert the hook refuses. These tests catch the day someone removes a safeguard 'because it was failing'.

Principle: Test the model behavior coarsely; test the safeguards rigorously. The model can drift; your invariants must not.

Code

Hook unit test·python

import pytest
from my_agent.hooks import block_destructive_bash

@pytest.mark.asyncio
@pytest.mark.parametrize("cmd,allowed", [
    ("ls -la", True),
    ("rm -rf /tmp/foo", False),
    ("dd if=/dev/zero of=/tmp/zero", False),
    ("echo hello", True),
])
async def test_block_destructive_bash(cmd, allowed):
    ctx = make_context(tool_name="Bash", tool_input={"command": cmd})
    out = await block_destructive_bash(ctx)
    assert out.allow == allowed

End-to-end behavior assertion·python

@pytest.mark.asyncio
async def test_summarizer_stays_in_sandbox(tmp_path):
    (tmp_path / "input.md").write_text("hello world\n")
    options = ClaudeAgentOptions(
        cwd=str(tmp_path),
        allowed_tools=["Read"],
        permission_mode="acceptEdits",
        model="claude-haiku-4-5-20251001",
    )
    text = "".join([e.text async for e in query(prompt="Summarize input.md", options=options) if hasattr(e, "text")])
    assert "hello" in text.lower()
    # Confirm no Write or Bash event hit the audit log
    assert all(line["tool"] != "Write" for line in audit_lines())

What you can unit-test

What needs integration tests

Regression tests for safety

Code

External links

Exercise

Progress

Comments 0