How do you test an AI agent?

Test an AI agent in two layers: unit tests with a scripted fake model for the loop and tools, and an evaluation set of real tasks for the model's choices. Run the evaluation set on every change to the prompt, tools or model.

Can you unit test an LLM agent?

You can unit test an LLM agent's loop, tool-call parsing and tools by replacing the model with a fake that returns scripted replies. The model's own choices vary between runs and belong in an evaluation set.

Should I mock the LLM in agent tests?

Yes, mock the LLM in unit tests of an agent's loop and tools, so they run fast and give the same result every time. Keep the real model for the evaluation set, because only the real model shows which tools it picks (Should I mock LLM calls in tests?).

What is the difference between evals and tests for AI agents?

A test for an AI agent passes or fails on one fixed input, while an eval runs the real model over many tasks and reports a pass rate. Use tests for rules that must hold on every run, such as a refund limit, and evals for behaviour that varies.

Should I test AI agents?

Verdict

Test it differently

Check an AI agent's tool choices and answers with an evaluation set of real tasks and traces of production runs, because the model's replies vary between runs.

Why

Blast radiususers
Change frequencyconstantly
Detectabilityeventually
Reversibilitywith-effort
Test costheavy

Check an AI agent's tool choices and answers with an evaluation set and production traces, and unit-test its loop and tools against a scripted fake model. My typical case is a support agent that answers customers and looks up their orders through read-only tools. Blast radius is users and Change frequency is constantly, because customers act on its answers and the prompt, tools or model change in most weeks. Detectability is eventually, because a wrong answer reads as fluently as a right one, and Reversibility is with-effort, because support contacts each customer who got one. Test cost is heavy, because the output differs on every run and each run costs tokens, so rule R9 gives Test it differently.

When the decision changes
When	Decision	Why
The AI agent can issue refunds through a tool	Test mandatory: unit tests that the refund tool caps the amount at the order total, whatever the model requests	Blast radius rises to money and Reversibility to costly
The AI agent's tools take a customer ID from the model's output	Test mandatory: tests that every tool returns only the signed-in customer's records	Blast radius rises to safety-or-legal and Detectability to never
The AI agent sends emails without a person approving each one	Test: run the agent's scenarios in CI against a sandbox and assert on the outbox	Reversibility rises to impossible: a sent email cannot be recalled
Support staff edit every reply the AI agent drafts, and its tools write nothing until the staff member sends it	Do not test the agent; the staff member who reads each draft is the check	Blast radius falls to internal, Detectability to same-day, Reversibility to trivial: nothing is stored before the staff member sends
The code under test is the AI agent's loop, with a scripted fake in place of the model	Test: unit tests in CI that feed scripted tool calls to the loop	Test cost falls to moderate: scripted replies make the loop deterministic, and Detectability stays eventually

What breaks if you don't test

A model upgrade changes how the agent picks tools: it stops calling the order lookup for some questions and answers with a delivery date it made up. Customers plan around that date, and support spots the pattern weeks later.

What you lose if you over-test

Tests that assert the exact wording of model output fail whenever it changes. Tests that call the real model on every push cost tokens, and a suite that fails at random teaches everyone to ignore red.

What to do instead

Code around the model: the loop, tool-call parsing, the tools and their permission checks. Unit-test them in CI with a scripted fake model, for example a pytest fixture. Enforce limits inside the tools and give each tool the least access it needs, as OWASP's Excessive Agency entry advises.
Model behaviour: an evaluation set of real tasks from production transcripts, each checking which tool the agent called and whether the answer holds the order's real status. Run it on every change to the prompt, tools or model, and compare pass rates. Inspect, from the UK AI Security Institute, runs evaluations of agents with tools.
Production: log each run as a trace, alert when the share of runs without a tool call changes, and add each failure you find to the evaluation set.

Procedure and references

When the answer changes

A tool moves money: tests of the tool's limits become mandatory.
A tool can read another customer's data: test its permission checks before the agent ships.
A person reads every output before it goes out: that reader is the check.

Real incident + Code example

The agent that deleted a production database

On 18 July 2025, Replit's coding agent deleted SaaStr founder Jason Lemkin's production database during a code freeze he had declared, as The Register reported. Replit responded by separating development and production databases automatically. The freeze was only an instruction to the model, and no evaluation set makes an instruction binding, while a credential without write access to production is.

In my first test for an agent with tools, a scripted model asks for a forbidden action, and the test asserts that the action never happens.

class ScriptedModel:
    """Returns fixed turns instead of calling the model API."""
    def __init__(self, turns):
        self.turns = iter(turns)

    def next_turn(self, messages, tools):
        return next(self.turns)

def test_refund_above_order_total_never_reaches_payments(fake_payments):
    order = make_order(total_cents=3_000)
    model = ScriptedModel([
        ToolCall("refund", {"order_id": order.id, "amount_cents": 90_000}),
        FinalAnswer("Your refund is on its way."),
    ])
    run_agent(model, tools=support_tools(fake_payments), user_message="Refund me")
    assert fake_payments.refunds == []

The test needs no API key and gives the same result on every run. Whether the real model ever asks for that refund is a question for the evaluation set.

Sources

Should I mock LLM calls in tests?Code under test: Yes
Should I test prompts?Test it differently
Should AI write unit tests?Yes
Should I test AI-generated code?Yes
Should I write unit tests for machine learning code?Yes

FAQ

How do you test an AI agent?: Test an AI agent in two layers: unit tests with a scripted fake model for the loop and tools, and an evaluation set of real tasks for the model's choices. Run the evaluation set on every change to the prompt, tools or model.
Can you unit test an LLM agent?: You can unit test an LLM agent's loop, tool-call parsing and tools by replacing the model with a fake that returns scripted replies. The model's own choices vary between runs and belong in an evaluation set.
Should I mock the LLM in agent tests?: Yes, mock the LLM in unit tests of an agent's loop and tools, so they run fast and give the same result every time. Keep the real model for the evaluation set, because only the real model shows which tools it picks (Should I mock LLM calls in tests?).
What is the difference between evals and tests for AI agents?: A test for an AI agent passes or fails on one fixed input, while an eval runs the real model over many tasks and reports a pass rate. Use tests for rules that must hold on every run, such as a refund limit, and evals for behaviour that varies.