Tool Contract Tests

Tools change. Schemas drift. Models update. Without a test that fails when the contract drifts, your agent quietly degrades for weeks before someone notices. Tool contract tests are the cheapest insurance you can buy.

Start with three test shapes:

Schema validity: every tool definition is a valid JSON Schema and round-trips through your provider's SDK without warnings. This is one fast unit test.
Selection accuracy: a fixture file pairs prompts with the tool that should be called. Run a small model (cheap) over each prompt, check the tool name, and report a pass/fail rate. You are not measuring the model's intelligence; you are measuring whether your descriptions still steer correctly.
Argument correctness: for a smaller fixture set, also assert key arguments. Did the model pass customer_id="C-9" from the user's message? Did it format the date correctly? Argument tests catch description regressions.

The trap is over-testing. Do not assert exact wording of the model's prose between tool calls — that drifts every release for no good reason. Assert structure: which tool, what arguments, did the loop terminate. Save the prose assertions for end-to-end tests that you tolerate flaking on.

Run these tests in CI when tool descriptions or schemas change, and on a nightly cadence with the latest model from your provider. The first time you catch a model upgrade silently changing your selection rate, you will understand why this was always a real category of test.

Code

A minimal fixture-based test·python

import pytest, json

FIXTURES = json.load(open("tests/tool_fixtures.json"))

@pytest.mark.parametrize("prompt,expected_tool", [(f["prompt"], f["tool"]) for f in FIXTURES])
def test_tool_selection(prompt, expected_tool, agent):
    resp = agent.one_turn(prompt)
    actual = next((b.name for b in resp.content if b.type == "tool_use"), None)
    assert actual == expected_tool, f"{prompt!r}: expected {expected_tool}, got {actual}"

Fixture file shape·json

[
  {"prompt": "where is order 9912?", "tool": "search_orders"},
  {"prompt": "show me the new arrivals in shoes", "tool": "search_catalog"},
  {"prompt": "refund order 9912 — wrong size", "tool": "propose_refund"},
  {"prompt": "what's your return policy?", "tool": null}
]

Exercise

Write a 10-prompt fixture file for one of your agents. Include three prompts where the correct answer is 'no tool, just talk'. Run the suite. The percentage of correct selections is your starting baseline; document it. Now change one tool description and re-run — watch the rate move. That movement is your test working.

Code

External links

Exercise

Progress

Comments 0