Should I test function calling?

Yes, test the code that runs function calls by feeding each handler scripted calls with valid and invalid arguments. Check whether the model picks the right function in a separate evaluation set.

Does strict mode make tool-call tests unnecessary?

Strict mode does not make tool-call tests unnecessary, because it guarantees only that a call matches your schema. A date in the right format can still be the wrong date, and a valid ID can belong to another customer.

How do I test that the model picks the right tool?

Test the model's choice of tool with an evaluation set of real user requests, each labelled with the expected tool and key arguments. Run it against the real model on every prompt, tool description or model change.

Should I test LLM tool calls?

Verdict

Yes

Test the code that runs LLM tool calls: in CI, feed each tool handler scripted calls with valid, malformed and out-of-range arguments, and assert what it stores and which error it returns to the model.

Why

Blast radiususers
Change frequencyregularly
Detectabilityeventually
Reversibilitywith-effort
Test costmoderate

Test the handlers that run LLM tool calls, with scripted calls in CI. My typical case is a feature where a hosted model calls your own functions, such as moving a booking to another date. Blast radius is users, because a wrong call changes a customer's booking. Change frequency is regularly, because schemas and handlers change about once a month. Detectability is eventually, because a handler that accepts a loose argument stores a plausible wrong value without an error, and Reversibility is with-effort, because support repairs each record. Test cost is moderate, because a scripted call is a fixed object that a test compares exactly, so rule R11 gives Test.

When the decision changes
When	Decision	Why
The failure to catch is the model picking the wrong tool or argument value after a prompt, tool description or model change	Test it differently: an evaluation set of logged requests that checks the tool name and key arguments, run on every such change	Test cost rises to heavy, because the real model's calls differ between runs
A tool issues a refund with an amount taken from the model's arguments	Test mandatory: scripted calls with an amount above the order total, a negative amount and a repeated call	Blast radius rises to money and Reversibility to costly, because an overpaid refund needs a claw-back
A tool reads records by a customer ID taken from the model's arguments	Test mandatory: scripted calls with another customer's ID that assert the tool returns no data	Blast radius rises to safety-or-legal and Detectability to never, because a leaked record reads like a normal answer
Every tool only reads public catalogue data for a live chat reply, and a rejected call shows an error that users report	Test minimally: one scripted call per tool, plus a regression test for each reported bad call	Detectability moves to same-day and Reversibility to trivial, because users see the error and nothing is saved
Only staff use the tool-calling assistant, and the tool writes each change only after the staff member confirms it in an internal screen	Do not write tool-call tests; the staff member who confirms each change is the check	Blast radius falls to internal, Detectability to same-day and Reversibility to trivial, because nothing is stored before the staff member confirms it
The tools run in a prototype on your own machine against your own test account	Do not test; try the calls by hand, and write handler tests before a customer uses the feature	Blast radius falls to none, because only you run the prototype

What breaks if you don't test

The model sends an argument your handler did not expect: a number as a string, a missing optional field, a date in the customer's own format. The handler guesses, the call succeeds, and the model tells the customer the change is done. Nobody sees the wrong value until the customer acts on it.

What you lose if you over-test

Handler tests that call the real model fail at random and cost tokens on every push. Tests that assert the exact JSON the model produced break on every prompt edit. With strict mode on, the provider constrains each call to your schema, so a test that the call is valid JSON checks the provider instead of your code.

How to test

Build each tool call by hand as the object your SDK passes to the handler.
For each tool, write one valid call, one call per argument the handler must reject, and boundary values; pytest parametrize keeps the bad arguments in one test.
Assert what the handler stores and which error it returns. Anthropic's guide to handling tool calls returns errors with is_error and a message the model can act on.
Turn on strict tool use or OpenAI's strict function calling, so wrong types never reach the handler.

Procedure and references

When the answer changes

A tool moves money or reads records by an ID the model supplies.
You change the model or a tool description and need to know it still picks the right tool.
A staff member confirms each change before the tool stores it.

Real incident + Code example

The visits that moved a month early

On a field-service booking app I worked on, the reschedule_visit tool took its date as a plain string, and the handler passed it to a lenient parser. UK customers typed "04/05" for 4 May, the model copied that format into the call, and the parser, reading the month first, stored 5 April. The model confirmed the date in words, so customers saw nothing wrong. Twelve days later a dispatcher asked why technicians kept finding nobody home. By then 41 visits had moved, and support phoned every customer to ask which date they meant.

We made the handler reject anything but YYYY-MM-DD and added these tests:

from datetime import date
import pytest
from booking.tools import handle_tool_call

def reschedule(visit_date):
    return {"id": "t1", "name": "reschedule_visit",
            "input": {"visit_id": 7, "date": visit_date}}

def test_iso_date_moves_the_visit(customer_visits):
    result = handle_tool_call(reschedule("2026-05-04"), customer_visits)
    assert not result.get("is_error")
    assert customer_visits.get(7).date == date(2026, 5, 4)

@pytest.mark.parametrize("bad", ["04/05/2026", "next Monday", ""])
def test_other_formats_go_back_to_the_model(customer_visits, bad):
    result = handle_tool_call(reschedule(bad), customer_visits)
    assert result["is_error"] is True
    assert "YYYY-MM-DD" in result["content"]
    assert customer_visits.get(7).date == date(2026, 5, 20)

Sources

Should I test AI agents?Test it differently
Should I mock LLM calls in tests?Code under test: Yes
Do I need evals for my LLM app?Test it differently
Should I test prompts?Test it differently
Should I test input validation?Yes

FAQ

Should I test function calling?: Yes, test the code that runs function calls by feeding each handler scripted calls with valid and invalid arguments. Check whether the model picks the right function in a separate evaluation set.
Does strict mode make tool-call tests unnecessary?: Strict mode does not make tool-call tests unnecessary, because it guarantees only that a call matches your schema. A date in the right format can still be the wrong date, and a valid ID can belong to another customer.
How do I test that the model picks the right tool?: Test the model's choice of tool with an evaluation set of real user requests, each labelled with the expected tool and key arguments. Run it against the real model on every prompt, tool description or model change.