Inspect AI: Agent and Safety Evaluation

The agent and safety evaluation framework from UK AISI

Inspect AI is an open-source framework from the UK AI Safety Institute, designed for evaluating LLM agents and safety properties. It is the framework most safety-focused organizations adopt because it is built around tasks that include tool use, multi-step reasoning, and adversarial scenarios.

What it does well

Agent evaluation — supports tool use, sandbox environments, and stateful task completion.
Safety evaluation — built-in patterns for capability assessment, dangerous-capability evals, and red-teaming.
Reproducibility — every eval run is fully scripted and re-runnable.
Multi-modal — supports text, image, and tool outputs.
Provider-agnostic — OpenAI, Anthropic, Google, local models.

Where it shines

If you are evaluating an agent that uses tools, browses, executes code, or plans across multiple steps, Inspect AI is built for that workload. Other frameworks bolt agent support on top; Inspect AI assumes it.

Where it doesn't fit

For simple prompt-and-response evals, the framework is overkill. promptfoo or DeepEval are lighter for that use case.

Principle: If your eval involves tool use, code execution, or multi-step planning, reach for Inspect AI. If it's prompt-and-response, reach for something lighter.

Code

Install and minimal task·python

# pip install inspect-ai
from inspect_ai import Task, eval, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.solver import generate

@task
def country_capitals():
    return Task(
        dataset=[
            Sample(input="What is the capital of France?", target="Paris"),
            Sample(input="What is the capital of Japan?", target="Tokyo"),
        ],
        solver=generate(),
        scorer=includes(),
    )

# Run from CLI: inspect eval my_eval.py --model openai/gpt-4o-mini

Agent task with tool use·python

from inspect_ai.solver import use_tools, generate
from inspect_ai.tool import bash, python

@task
def code_task():
    return Task(
        dataset=[Sample(
            input="Find all .py files modified in the last 7 days, return count.",
            target="42",
        )],
        solver=[
            use_tools([bash(), python()]),
            generate(),
        ],
        scorer=includes(),
        sandbox="docker",  # tools run in a sandbox
    )

Inspect AI: Agent and Safety Evaluation

The agent and safety evaluation framework from UK AISI

What it does well

Where it shines

Where it doesn't fit

Code

External links

Exercise

Progress

Comments 0