Should I rerun retrieval evals after changing chunk size?

Yes, rerun the labelled retrieval questions after every change to chunk size, overlap or splitting rules. A new chunk size moves passages across boundaries, and recall at 5 shows which questions lost their documents.

How do I unit test a RAG chunker?

Unit test a RAG chunker on real fixture documents with exact invariants: no source line is lost, no chunk exceeds the token limit, and tables and code blocks stay whole.

Should I test a library text splitter?

Do not test the internals of a library text splitter, because another team owns that code. Test how your settings split your own documents.

Is a retrieval eval enough to catch chunking bugs?

No, a retrieval eval catches a chunking bug only when one of its questions asks about the damaged passage. Chunker unit tests check every line of every fixture document.

Should I test chunking for RAG?

Verdict

Yes

Test RAG chunking: unit tests that no source text is lost, no chunk exceeds the embedding model's token limit and tables stay whole, plus the labelled retrieval questions on every change to chunk size, overlap or splitting rules.

Why

Blast radiususers
Change frequencyregularly
Detectabilityeventually
Reversibilitywith-effort
Test costmoderate

Test the chunker as code, and rerun the retrieval questions whenever chunking changes. My typical case is a support assistant over a few thousand help pages. Blast radius is users, because customers act on the answers. Change frequency is regularly: chunk size, overlap or the splitter changes about once a month. Detectability is eventually, because a lost paragraph leaves the model enough context to write a fluent, incomplete answer. Reversibility is with-effort: a fix means a re-index and replies to tickets. Test cost is moderate: a chunker is a function from text to chunks, so a test compares its output exactly, and rule R11 gives Test.

When the decision changes
When	Decision	Why
Only you query the RAG index, over your own notes	Do not test chunking; open the sources the tool cites when an answer looks wrong	Blast radius falls to none: a lost or split passage reaches only you
Staff read every RAG answer and open its cited sources before replying to a customer	Do not test chunking; the staff member who checks the sources catches a broken chunk	Blast radius falls to internal, Detectability to same-day, Reversibility to trivial
The ingestion job that splits documents and writes the chunks to the index changes a few times a year	Test chunking minimally: one test that runs the ingestion job on sample documents, reads the stored chunks back and checks that no text is lost and every chunk fits the token limit	Change frequency falls to rarely
One index holds the documents of several customers, and each chunk must carry its customer's tenant ID for the query filter	Test mandatory: a test that every chunk the ingestion writes carries the tenant ID of its source document	Blast radius rises to safety-or-legal and Detectability to never: a chunk from another customer reads like a normal answer
Chunks hold the price tables that the assistant quotes to customers	Test mandatory: tests that no price table splits across chunks and every price row stays with its plan name	Blast radius rises to money and Reversibility to costly: a quoted wrong price is honoured or refunded
A language model rewrites each document into short statements before indexing, and its output differs on every run	Test chunking differently: run the labelled retrieval questions after each re-index and have a person compare 20 random chunks with their source pages	Test cost rises to heavy, because no exact expected chunk exists; Detectability stays eventually

What breaks if you don't test

A splitter change drops or cuts text, and nothing errors: ingestion succeeds and the model answers from what remains. A table header lands in one chunk and its rows in the next, so the model reads values without column names. A retrieval eval catches a lost passage only when one of its questions asks about that passage.

What you lose if you over-test

Snapshot tests of every chunk boundary break on each chunk size change, and the team approves the new snapshots without reading them. Re-embedding the corpus on every pull request adds embedding spend for changes that never touch ingestion; run the retrieval questions when ingestion files change.

How to test

Pick ten real documents for fixtures, including a table, a code block, a file with no trailing newline and your longest page.
Write unit tests of three invariants: every non-empty source line lands in a chunk, no chunk exceeds the embedding model's token limit, and a table stays in one chunk.
Assert that each chunk carries its document ID, because the assistant cites sources by that ID.
Run the labelled retrieval questions from the RAG retrieval answer on every change to chunk size, overlap or splitting rules.

The LangChain text splitter reference documents chunk_size and chunk_overlap, and the LlamaIndex node parser guide shows how to run a parser on its own, outside the index.

Procedure and references

When the answer changes

A person reads the cited sources of every answer before a customer sees it: that reader catches a broken chunk.
One index serves several customers: a test that every chunk carries its tenant ID becomes mandatory.

Real incident + Code example

The splitter that lost the last section

On a documentation assistant I worked on, we replaced a character splitter with our own Markdown heading splitter. It dropped the final section of any file without a trailing newline, which was 140 of 410 API reference pages. On most of them that section was "Rate limits". The assistant told developers that endpoints had no rate limit. Our 50 retrieval questions passed, because none asked about limits. A customer pasted the assistant's reply into a ticket 17 days later. These tests would have failed on the pull request:

from pathlib import Path
import pytest
from app.ingest import split_document, count_tokens

DOCS = sorted(Path("tests/fixtures/docs").glob("*.md"))
LIMIT = 512

@pytest.mark.parametrize("path", DOCS, ids=lambda p: p.name)
def test_every_line_lands_in_a_chunk(path):
    text = path.read_text()
    joined = "\n".join(c.text for c in split_document(text, LIMIT))
    lost = [line for line in text.splitlines() if line.strip() and line not in joined]
    assert not lost, f"lost lines: {lost[:3]}"

@pytest.mark.parametrize("path", DOCS, ids=lambda p: p.name)
def test_no_chunk_exceeds_the_limit(path):
    assert all(count_tokens(c.text) <= LIMIT for c in split_document(path.read_text(), LIMIT))

def test_a_table_stays_in_one_chunk():
    text = Path("tests/fixtures/docs/rate-limits.md").read_text()
    rows = [line for line in text.splitlines() if line.startswith("|")]
    assert any(all(r in c.text for r in rows) for c in split_document(text, LIMIT))

Should I evaluate RAG retrieval separately?Yes
Should I evaluate an embedding model on my own data?Yes
Should tests use a real vector database?Yes
Do I need evals for my LLM app?Test it differently
Should I test for hallucinations?Test it differently

FAQ

Should I rerun retrieval evals after changing chunk size?: Yes, rerun the labelled retrieval questions after every change to chunk size, overlap or splitting rules. A new chunk size moves passages across boundaries, and recall at 5 shows which questions lost their documents.
How do I unit test a RAG chunker?: Unit test a RAG chunker on real fixture documents with exact invariants: no source line is lost, no chunk exceeds the token limit, and tables and code blocks stay whole.
Should I test a library text splitter?: Do not test the internals of a library text splitter, because another team owns that code. Test how your settings split your own documents.
Is a retrieval eval enough to catch chunking bugs?: No, a retrieval eval catches a chunking bug only when one of its questions asks about the damaged passage. Chunker unit tests check every line of every fixture document.