Should tests call the real LLM API?

Unit tests should not call the real LLM API; they should use a fake client that returns recorded replies. Call the real model only in an evaluation set, so a provider outage does not fail the build.

Should I mock the OpenAI API in tests?

Yes, mock the OpenAI API in unit tests at the client boundary, with replies recorded from your production model. Include a refusal, a cut-off reply and a timeout.

How do I check reply quality if my tests mock the model?

Check LLM reply quality with an evaluation set of real inputs that calls the real model, separate from the mocked unit tests. Run it on every prompt or model change.

Should I mock LLM calls in tests?

Answer

Yes, mock LLM calls in tests of your own code: a fake model client that returns replies recorded from the real model, including malformed, refused and cut-off replies, tests prompt building, parsing and error handling in CI.

Verdict on the code under testYes

Why

Blast radiususers
Change frequencyregularly
Detectabilityeventually
Reversibilitywith-effort
Test costmoderate

Test your code around the model with a fake client, and keep real LLM calls out of the unit test build. My typical case is a backend feature that sends a ticket to a hosted model and stores the tags it returns. Blast radius is users, because a mistagged ticket waits in the wrong queue. Change frequency is regularly, because the prompt template, the reply schema or the parser changes about once a month. Detectability is eventually, because a parser that falls back to an empty tag list raises no error, and Reversibility is with-effort, because the stored tags need a script to fix. Test cost is moderate, because a fake client returns a fixed reply that a test compares exactly, so rule R11 gives Test.

When the decision changes
When	Decision	Why
The failure to catch is a worse reply from the model itself, after a prompt edit or a model version change	Test reply quality differently: run an evaluation set against the real model on every prompt or model change, outside the unit test build	Test cost rises to heavy, because a real reply differs on every run and needs graded inputs
The model's reply goes to customers as an email that no person reads first	Test: run the evaluation set against the real model as a release gate that blocks on a failed case	Reversibility rises to impossible, because a sent email cannot be recalled, even with Test cost heavy for reply quality
The model's reply sets a refund amount that code pays out	Test mandatory: fake-model tests that return an amount above the order total and assert that code caps the refund	Blast radius rises to money and Reversibility to costly, because an overpaid refund needs a claw-back
The model picks a customer ID that a tool uses to read records	Test mandatory: fake-model tests that request another customer's ID and assert the tool returns nothing	Blast radius rises to safety-or-legal and Detectability to never, because a leaked record reads like a normal answer
The reply appears live in a chat window, stores nothing, and a failed parse shows an error that users report	Test minimally: one fake-model test of the main path, plus a regression test for each reported reply shape	Detectability moves to same-day and Reversibility to trivial, because users see the error and nothing is saved
Staff read and edit every model draft in an internal tool before anyone else sees it	Do not write fake-model tests; the staff member who reads each draft is the check	Blast radius falls to internal, Detectability to same-day and Reversibility to trivial

What breaks if you don't test

A model wraps JSON in a code fence, refuses the request, or stops at the token limit. Without a test for each shape, the parser takes the fallback branch, every request succeeds, and wrong data piles up until someone reads the records.

What you lose if you over-test

Real calls make the build depend on the provider's uptime, rate limits and your API budget. With 120 tests at 1.5 seconds per sequential call, each run waits 3 minutes. A real reply differs between runs, so exact assertions fail at random.

Mocks that assert the exact prompt text break on every wording edit. The Lincoln Loop team replaced mocks with a fake LLM backend so a refactor did not touch hundreds of tests.

How to test

Replace the model at the client boundary, not inside your own wrapper. Pydantic AI's TestModel and FunctionModel do this for agents; for a plain SDK, use create_autospec from unittest.mock so the fake keeps the client's signature.
Record replies from the real model with VCR.py or by hand, and strip the API key from the recording. Keep a fenced reply, a refusal, a cut-off reply and a timeout.
Assert on what your code stores for each recorded reply.
Check reply quality in a separate evaluation set against the real model, as Anthropic's guide to building evaluations describes.

Procedure and references

When the answer changes

A model reply moves money or picks whose records a tool reads.
You need to know whether the model still answers well after a prompt edit or a model change.
Staff read every reply before it reaches anyone else.

Real incident + Code example

The fake reply nobody had seen from the model

On a helpdesk product I worked on, the unit tests for ticket tagging mocked the model client with a reply a developer had typed by hand: plain JSON, nothing around it. After we moved to a newer model version, some replies came wrapped in a Markdown code fence. json.loads failed, the fallback stored an empty tag list, and those tickets went to the general queue. A team lead noticed the empty billing queue nine days later, and a script tagged 2,300 tickets again. We switched to recorded replies:

import pytest
from tagging import tag_ticket

FENCE = "`" * 3
RECORDED = {
    "plain": '{"tags": ["billing"]}',
    "fenced": FENCE + 'json\n{"tags": ["billing"]}\n' + FENCE,
    "refusal": "I can't help with that request.",
}

class FakeClient:
    def __init__(self, reply):
        self.reply = reply
    def complete(self, prompt):
        return self.reply

@pytest.mark.parametrize("shape", ["plain", "fenced"])
def test_tags_parsed_from_every_json_shape(shape):
    assert tag_ticket("Card charged twice", FakeClient(RECORDED[shape])) == ["billing"]

def test_refusal_goes_to_review_not_empty_tags():
    assert tag_ticket("Card charged twice", FakeClient(RECORDED["refusal"])) == ["needs-review"]

Sources

Do I need evals for my LLM app?Test it differently
Should unit tests make real API calls?Test it differently
Should I test AI agents?Test it differently
Should I retest my app after an LLM model upgrade?Code under test: Test it differently
Should I test prompts?Test it differently

FAQ

Should tests call the real LLM API?: Unit tests should not call the real LLM API; they should use a fake client that returns recorded replies. Call the real model only in an evaluation set, so a provider outage does not fail the build.
Should I mock the OpenAI API in tests?: Yes, mock the OpenAI API in unit tests at the client boundary, with replies recorded from your production model. Include a refusal, a cut-off reply and a timeout.
How do I check reply quality if my tests mock the model?: Check LLM reply quality with an evaluation set of real inputs that calls the real model, separate from the mocked unit tests. Run it on every prompt or model change.