Should I test RAG retrieval?

Yes, test RAG retrieval with labelled questions when users read the answers and the retrieval setup changes about once a month. Retrieval returns document IDs, so a CI test can check recall at 5 exactly.

Do I need retrieval metrics if the answers look right?

Yes, a RAG system needs retrieval metrics even when its answers look right, because a language model writes a fluent answer from incomplete context. Recall at 5 shows a missing document on the pull request.

Which metric should I use for RAG retrieval?

Start RAG retrieval evaluation with recall at k, where k is the number of chunks you pass to the model. Ranking metrics such as nDCG matter when the prompt keeps only the first few results.

Should I evaluate RAG retrieval separately?

Verdict

Yes

Evaluate RAG retrieval separately: label 50 real questions with the documents that answer them, and check recall at 5 in CI on every chunking, embedding or document change.

Why

Blast radiususers
Change frequencyregularly
Detectabilityeventually
Reversibilitywith-effort
Test costmoderate

Evaluate retrieval on its own, with a labelled question set in CI. My typical case is a support assistant that answers from a few thousand help-centre articles. Blast radius is users, because customers read the answers, and Change frequency is regularly, because chunking, the embedding model or the articles change about once a month. Detectability is eventually: without the right chunk, the model still writes a fluent answer. Reversibility is with-effort, because a revert means a reindex and customers have already followed the wrong steps. Test cost is moderate, because retrieval returns document IDs from your own corpus, which a test compares exactly, so rule R11 gives Test.

When the decision changes
When	Decision	Why
Only you query the RAG index, over your own notes	Do not evaluate RAG retrieval separately; open the sources the tool cites	Blast radius falls to none: a missed document reaches only you
Staff read every RAG answer and open its cited sources before replying to a customer	Do not evaluate RAG retrieval separately; the staff member who checks the sources is the check	Blast radius falls to internal, Detectability to same-day, Reversibility to trivial
The documents and retrieval settings change a few times a year, as in a fixed archive of product manuals	Test RAG retrieval minimally: 20 labelled questions with a recall check, run when the index is rebuilt	Change frequency falls to rarely
The RAG index holds documents of many customers, and each customer may see only their own	Test mandatory: a test that one customer's query never returns another customer's documents	Blast radius rises to safety-or-legal and Detectability to never: a leaked document reads like a normal answer
The RAG assistant quotes prices and refund terms from the retrieved documents	Test mandatory: labelled questions for every price and refund document, each with recall at 5 of 1.0	Blast radius rises to money and Reversibility to costly
Users ask open research questions for which no fixed set of documents is correct	Test RAG retrieval differently: a person rates a weekly sample of production retrievals, with an alert when the rating falls	Test cost rises to heavy, because relevance needs a person's judgment; Detectability stays eventually

What breaks if you don't test

If you judge only final answers, a retrieval regression hides behind the model. With the right article missing, the model fills the gap from a neighbouring article, and a grader model scores the reply as fluent. When an answer eval fails, you cannot tell whether retrieval or generation broke. Customers notice first, in tickets about wrong instructions.

What you lose if you over-test

Precision, recall, MRR and nDCG at three values of k give twelve numbers per run, and a drop in one of them does not say what to fix. Labels tied to chunk IDs go stale at every re-chunking, so label source documents. A floor at today's exact score fails on a harmless reordering, and the team learns to rerun the job. A retrieval score says nothing about a prompt that ignores good context, so keep the answer evals.

How to test

Take 50 real questions from your logs. A person who knows the content records the source documents that answer each one.
Compute recall at 5 for each question: the share of labelled documents in the top five results.
Set the floor 0.05 below the current mean, and fail when any question loses every relevant document.
Run the test in CI on changes to chunking, the embedding model, top-k, filters or the document set.
Log retrieved IDs in production, and add every question with a missed document to the set.

The Stanford chapter on evaluation in information retrieval defines the metrics, and Ragas context recall scores retrieval against reference answers instead of document labels.

Procedure and references

When the answer changes

A person reads every answer and its cited sources before a customer sees it: that reader is the check.
The index holds documents of several customers: tests that retrieval filters by customer become mandatory.

Real incident + Code example

The re-chunking that dropped step four

On a router maker's support assistant I worked on, we cut the chunk size to 300 tokens, down from 800. The firmware update guide split into four chunks, and step four, a factory reset, fell out of the top five for most update questions. Our answer evals passed, because the grader model read three steps as a complete procedure. Three weeks later support tied boot-loop tickets to the assistant. This retrieval test would have failed on the pull request:

import json
from app.search import retrieve  # query -> ranked hits with doc_id

CASES = [json.loads(line) for line in open("tests/retrieval_cases.jsonl")]

def recall_at_5(case):
    found = {hit.doc_id for hit in retrieve(case["query"], top_k=5)}
    relevant = set(case["relevant_doc_ids"])
    return len(found & relevant) / len(relevant)

def test_mean_recall_stays_above_floor():
    mean = sum(recall_at_5(c) for c in CASES) / len(CASES)
    assert mean >= 0.85, f"recall@5 fell to {mean:.2f}"

def test_no_question_loses_every_relevant_document():
    lost = [c["query"] for c in CASES if recall_at_5(c) == 0]
    assert not lost, f"nothing relevant retrieved for: {lost}"

Our mean before the change was 0.90, so the floor is 0.85.

Should I evaluate an embedding model on my own data?Yes
Do I need evals for my LLM app?Test it differently
Should I test for hallucinations?Test it differently
Should I test prompts?Test it differently
Should I mock LLM calls in tests?Code under test: Yes

FAQ

Should I test RAG retrieval?: Yes, test RAG retrieval with labelled questions when users read the answers and the retrieval setup changes about once a month. Retrieval returns document IDs, so a CI test can check recall at 5 exactly.
Do I need retrieval metrics if the answers look right?: Yes, a RAG system needs retrieval metrics even when its answers look right, because a language model writes a fluent answer from incomplete context. Recall at 5 shows a missing document on the pull request.
Which metric should I use for RAG retrieval?: Start RAG retrieval evaluation with recall at k, where k is the number of chunks you pass to the model. Ranking metrics such as nDCG matter when the prompt keeps only the first few results.