Should I mock the vector store in tests?

Mock the vector store only in tests of code that uses hits the store has already returned. Code that writes, filters or deletes chunks needs the real vector database, because a mock accepts any filter.

Should vector store tests call the embedding model?

No, vector store tests should use small hand-written vectors instead of calling the embedding model. Hand-written vectors give exact results at no API cost.

Can tests use Chroma in memory when production runs another vector database?

No, tests of filter and delete code need the vector database production runs, because each store filters and indexes in its own way. Chroma in memory fits only when production runs Chroma.

How many records does a vector database test need?

A test of a filtered query on an approximate index needs a few thousand records, with index use forced on. Tests of writes and deletes need a handful of records.

Should tests use a real vector database?

Verdict

Yes

Yes, tests of code that writes, filters and deletes chunks in a vector store should run against the same vector database as production, started in a container or its local emulator, with the production index type and distance metric and small hand-written vectors.

Why

Blast radiususers
Change frequencyregularly
Detectabilityeventually
Reversibilitywith-effort
Test costmoderate

Yes, run tests of your retrieval code against the vector database production runs. My typical case is a support assistant whose code writes, deletes and filters chunks in pgvector, Qdrant or Pinecone. Blast radius is users, because customers read answers built from the retrieved chunks. Change frequency is regularly: filters, metadata fields and index settings change about once a month. Detectability is eventually, because a filter that matches too little still returns some chunks and a fluent answer. Reversibility is with-effort, since a fix can mean a re-index, and Test cost is moderate: the store runs in a container, and hand-written vectors give exact results, so rule R11 gives Test.

When the decision changes
When	Decision	Why
One vector collection holds the documents of several customers, and every query must filter by tenant ID	Test mandatory: tests with chunks of two tenants and near-identical vectors, asserting that each query returns one tenant's chunks	Blast radius rises to safety-or-legal and Detectability to never, because another customer's chunk reads like a normal result
Production runs a managed vector service with no local edition or emulator, and a filtered query can return fewer chunks without an error	Test it differently: a scheduled job runs 20 known filtered queries on the production index and alerts when one returns fewer chunks	Test cost rises to heavy, because every test needs a shared cloud index, while Detectability stays eventually and Reversibility with-effort
The retrieval code and index settings change a few times a year, as in a fixed archive of product manuals	Test minimally: one container test that writes chunks with metadata and checks that a filtered query returns them	Change frequency falls to rarely
A one-time script copies every chunk from one vector database to another	Test it differently: rehearse the copy from a snapshot, compare record counts, and compare the top 10 chunk IDs of 50 logged queries on both stores	Change frequency falls to once while Blast radius stays users
Only you query the vector store, in a side project over your own notes	Do not run a vector database in tests; open the cited sources when an answer looks wrong	Blast radius falls to none, because a missed chunk reaches only you

What breaks if you don't test

A mocked store accepts any filter and returns whatever the test put in. The pgvector documentation states that with approximate indexes filtering is applied after the index is scanned: with HNSW and the default hnsw.ef_search of 40, a condition that matches 10% of rows returns about four rows. A mock passes, and production returns fewer chunks for small categories with no error.

What you lose if you over-test

Tests that embed real text through the model API cost money on every run. Assertions on exact similarity scores break when the store changes its index code, because HNSW search is approximate. Answer quality belongs in a retrieval evaluation set.

How to test

Start the production store in a container once per test run: the pgvector/pgvector image, Qdrant's image, or Pinecone Local.
Create the collection with the production index type and distance metric, using three-dimensional vectors you write by hand.
Test each filter, a delete, and an update of existing chunks, asserting on chunk IDs.
For a filtered query, load a few thousand rows and force index use, because a small table gets a sequential scan.

Testcontainers for Python starts the container, and the pgvector README documents index options.

Procedure and references

When the answer changes

One collection holds several customers' documents: tenant filter tests become mandatory.
The store has no local edition: watch filtered queries in production.

Real incident + Code example

The product line that returned one chunk

On a lab equipment maker's support assistant I worked on, 60,000 chunks lived in pgvector behind an HNSW index, and our tests mocked the store. The smallest product line held 3% of the chunks, so its queries got one or two chunks from the 40 index candidates, and the assistant often said it had no information. Support noticed four weeks later. We upgraded to pgvector 0.8.0, set hnsw.iterative_scan = relaxed_order for iterative index scans inside search, and added this test:

import pytest
from testcontainers.postgres import PostgresContainer
from app.retrieval import connect, create_schema, add_chunk, search

@pytest.fixture(scope="session")
def db():
    with PostgresContainer("pgvector/pgvector:pg17", driver=None) as pg:
        conn = connect(pg.get_connection_url())
        create_schema(conn, dimensions=3)  # production HNSW index, cosine ops
        yield conn

def test_filtered_search_fills_top_k_through_hnsw_index(db):
    for i in range(2000):  # 2% of chunks belong to the "centrifuge" line
        line = "centrifuge" if i % 50 == 0 else "pipette"
        add_chunk(db, f"chunk-{i}", line, [1.0, i / 2000, 0.0])
    db.execute("SET enable_seqscan = off")  # use the index, as production does
    hits = search(db, [1.0, 0.0, 0.0], product_line="centrifuge", top_k=5)
    assert len(hits) == 5
    assert all(hit.product_line == "centrifuge" for hit in hits)

Sources

Should I evaluate RAG retrieval separately?Yes
Should I evaluate an embedding model on my own data?Yes
Should tests use a real database?Yes
Are Testcontainers worth it?Yes
Should integration tests use mocks?Code under test: Yes

FAQ

Should I mock the vector store in tests?: Mock the vector store only in tests of code that uses hits the store has already returned. Code that writes, filters or deletes chunks needs the real vector database, because a mock accepts any filter.
Should vector store tests call the embedding model?: No, vector store tests should use small hand-written vectors instead of calling the embedding model. Hand-written vectors give exact results at no API cost.
Can tests use Chroma in memory when production runs another vector database?: No, tests of filter and delete code need the vector database production runs, because each store filters and indexes in its own way. Chroma in memory fits only when production runs Chroma.
How many records does a vector database test need?: A test of a filtered query on an approximate index needs a few thousand records, with index use forced on. Tests of writes and deletes need a handful of records.