Should I check that RAG answers stay grounded in their sources?

Yes, check that RAG answers stay grounded with an evaluation set of real questions, run on every change to the prompt, retrieval, documents or model. Verify in code that each quote the model gives exists in the retrieved document.

Can a unit test catch LLM hallucinations?

A unit test catches LLM hallucinations only in structured output: it can assert that a quoted passage exists in its document or that an extracted field matches a labelled value. Free-text claims need an evaluation set with a rubric grader, because the wording changes on every run.

Can a language model judge whether an answer is hallucinated?

Yes, a second language model can compare each claim in an answer with the source documents and mark unsupported claims. Before you trust its grades, compare them with a person's grades on 30 answers.

How do I stop a chatbot from making up policies?

To stop a chatbot from making up policies, require a quote from retrieved policy text for each rule it states, and tell it to decline when no passage answers the question. Add 10 questions your policies do not cover to the evaluation set, and fail any answer that does not decline.

Should I test for hallucinations?

Verdict

Test it differently

Test for hallucinations differently: check answers to real questions against your own documents in an evaluation set on every prompt, retrieval or model change, instead of unit tests that assert reply text.

Why

Blast radiususers
Change frequencyregularly
Detectabilityeventually
Reversibilitywith-effort
Test costheavy

Test for hallucinations with an evaluation set that checks answers against your own documents. My typical case is a help assistant that answers customers from the company's documentation. Blast radius is users, because customers act on the answer. Change frequency is regularly: the prompt, the documents or the model version change about once a month. Detectability is eventually, because an invented fact reads as fluently as a true one and raises no error, and Reversibility is with-effort: support corrects the answer after the customer complains. Test cost is heavy, because the reply differs on every run, so rule R9 gives Test it differently.

When the decision changes
When	Decision	Why
You ask a chat model questions for your own work and use its answers yourself	Do not test for hallucinations; check each fact you use against its source	Blast radius falls to none: a made-up fact reaches only you
Support staff read and edit every drafted answer before a customer sees it	Do not build a hallucination check; the staff member who edits each draft is the check	Blast radius falls to internal, Detectability to same-day, Reversibility to trivial
The model copies order numbers and delivery addresses from support emails into fields	Test: assert each field against labelled emails in CI, and fail when a value does not appear in its email	Test cost falls to moderate: an extracted field compares exactly with a labelled value; Detectability stays eventually
Answers go to customers by email with no person reading them first	Test: run the grounding evaluation set in CI, and block the release on a failed case	Reversibility rises to impossible: a sent email cannot be recalled
The assistant answers questions about refunds, fares or cancellation fees	Test mandatory: evaluation cases for every refund and fee rule gate each release	Blast radius rises to money and Reversibility to costly: a tribunal can hold the company to a refund rule its assistant invented
The assistant answers questions about medication doses	Test mandatory: evaluation cases for every dose question gate each release, and the safety rules of your domain come first	Blast radius rises to safety-or-legal and Reversibility to impossible: a made-up dose can harm a person

What breaks if you don't test

A hallucination is an answer that states something your documents do not say. The retrieval step returns the wrong page, or no page, and the model fills the gap with a likely answer: steps through a settings menu that the product does not have. Nothing fails in the logs, and support learns about the invented menu from complaints weeks later.

What you lose if you over-test

A grader model that scores every claim varies between runs, so a few cases fail at random and the team stops reading the report. Unit tests that assert the exact answer text break with each model version.

What to do instead

Collect 50 real questions from your logs, with the document that answers each one, plus 10 questions your documents do not answer.
Ask the model to quote the passage behind each claim, as Anthropic's guide to reducing hallucinations recommends, and check the quotes in code.
Require a refusal on the unanswerable questions.
Grade the remaining claims with a second model against a rubric.
Run the set on every change to the prompt, retrieval code, documents or model version, and read a weekly sample of production answers.

Procedure and references

When the answer changes

The answer states refunds, prices or doses: the evaluation set becomes mandatory.
Answers reach customers before any person reads them: the set becomes a release gate.
The output is a value copied from a document into a field: exact assertions work as ordinary tests.

Real incident + Code example

The refund rule a chatbot invented

In November 2022, Air Canada's website chatbot told Jake Moffatt that he could claim a bereavement fare within 90 days after his flight, while the bereavement page it linked to ruled out claims after travel. Air Canada argued that the chatbot was a separate legal entity. In Moffatt v. Air Canada, 2024 BCCRT 149, the tribunal rejected that argument and ordered the airline to pay $812.02 in damages, interest and fees. A check that each stated rule quotes the linked page fails on that answer:

# load_cases() is the project's own function.
# A reply: {"text": str, "declined": bool, "quotes": [{"doc_id": str, "quote": str}]}
CASES = load_cases("evals/policy_questions.jsonl")

def check(case, reply, docs):
    problems = []
    for q in reply["quotes"]:
        if q["quote"] not in docs[q["doc_id"]]:
            problems.append(f"quote not found in {q['doc_id']}")
    if case["answerable"] and not reply["quotes"]:
        problems.append("answer with no supporting quote")
    if not case["answerable"] and not reply["declined"]:
        problems.append("answered a question the documents do not cover")
    for fact in case.get("must_contain", []):
        if fact not in reply["text"]:
            problems.append(f"missing fact: {fact}")
    return problems

A claim written outside the quotes still needs the rubric grader.

Sources

Do I need evals for my LLM app?Test it differently
Should I evaluate RAG retrieval separately?Yes
Should I test prompts?Test it differently
Should I retest my app after an LLM model upgrade?Code under test: Test it differently
Should I mock LLM calls in tests?Code under test: Yes

FAQ

Should I check that RAG answers stay grounded in their sources?: Yes, check that RAG answers stay grounded with an evaluation set of real questions, run on every change to the prompt, retrieval, documents or model. Verify in code that each quote the model gives exists in the retrieved document.
Can a unit test catch LLM hallucinations?: A unit test catches LLM hallucinations only in structured output: it can assert that a quoted passage exists in its document or that an extracted field matches a labelled value. Free-text claims need an evaluation set with a rubric grader, because the wording changes on every run.
Can a language model judge whether an answer is hallucinated?: Yes, a second language model can compare each claim in an answer with the source documents and mark unsupported claims. Before you trust its grades, compare them with a person's grades on 30 answers.
How do I stop a chatbot from making up policies?: To stop a chatbot from making up policies, require a quote from retrieved policy text for each rule it states, and tell it to decline when no passage answers the question. Add 10 questions your policies do not cover to the evaluation set, and fail any answer that does not decline.