Should I test embeddings?

Do not unit test embedding vectors, because their exact values change with every model version. Test retrieval instead with an evaluation set of real queries and their answering documents, scored by recall@10.

Should I retest retrieval after changing the embedding model?

Yes, run the retrieval evaluation set on the old and the new embedding model before the switch ships. Re-embed the whole corpus, because vectors from two models in one index return plausible wrong documents.

How many queries does an embedding evaluation set need?

Start an embedding evaluation set with 100 real queries and their labelled answering documents. Add every query that users report as failing.

Is a public leaderboard score enough to choose an embedding model?

No, a public leaderboard score only narrows the shortlist of embedding models, because the MTEB authors found that no method wins on every task. Choose between the shortlisted models with recall@10 on your own queries.

Should I evaluate an embedding model on my own data?

Verdict

Yes

Yes, test minimally: before each model switch, run one recall@10 comparison of the current and the candidate model on 100 real queries labelled with the documents that answer them, instead of choosing by a public leaderboard score or writing unit tests of the vectors.

Why

Blast radiususers
Change frequencyrarely
Detectabilityeventually
Reversibilitywith-effort
Test costmoderate

Test minimally: gate each model switch on one recall@10 comparison over your own labelled queries. My typical case is a help-centre search whose team replaces the model when a better one appears. Blast radius is users, because customers read what the search returns. Change frequency is rarely: the model changes a few times a year. Detectability is eventually, because a weaker model still returns plausible documents, and Reversibility is with-effort, because a revert means re-embedding the whole corpus. Test cost is moderate: 100 real queries labelled with the documents that answer them take a day or two, and recall@10 compares document IDs exactly, so rule R12 gives Test minimally.

When the decision changes
When	Decision	Why
Only you use the semantic search, over your own notes	Do not build an evaluation set; try your own queries by eye and note each failure	Blast radius falls to none: poor results reach only you
A script clusters last year's support tickets once, for an internal report	Do not evaluate the embedding model; read sample tickets from each cluster before sharing the report	Change frequency falls to once and Blast radius to internal: only staff read a one-time result
Queries and stored vectors can come from different embedding model versions, for example after a partial re-index	Test: a unit test that fails when the model name used for queries differs from the model name stored with the index	Test cost falls to trivial: model names compare exactly; Detectability stays eventually, as mixed vectors return plausible wrong documents
Retrieved documents supply the prices or refund terms that an assistant quotes to customers	Test mandatory: before any model switch, evaluation queries must return the current version of every price and refund document in the top 3	Blast radius rises to money and Reversibility to costly: a quoted wrong price is honoured or refunded
One vector index holds the documents of several customers, separated by a tenant ID	Test mandatory: tests that every query after a re-index filters by the signed-in customer's tenant ID, plus the evaluation set	Blast radius rises to safety-or-legal and Detectability to never: another customer's document reads like a normal result
Users ask open research questions, for which no fixed set of documents is correct	Test it differently: a person rates a weekly sample of production search results, with an alert when the rating falls	Test cost rises to heavy, because relevance needs a person's judgment; Detectability stays eventually

What breaks if you don't test

The team switches to a model with a higher leaderboard average, and every query still returns ten results. The MTEB authors found that no embedding method dominates across all tasks, so a model that wins on public web text can rank your product codes or jargon lower. The assistant answers from near-miss documents, and support hears about it weeks later.

What you lose if you over-test

Unit tests that assert exact vector values break on every model version and say nothing about ranking. A labelled set of 2,000 queries takes weeks to build, and nobody re-checks that many labels. On 100 queries, one query moves recall@10 by up to 0.01, so a 0.01 gain is noise.

How to test

Take 100 real queries from search logs.
Record the IDs of the documents that answer each query.
Before a switch, run one evaluation script that embeds the corpus with both models and compares recall@10. Block the switch when the candidate scores lower.
Add every query that users report as failing to the set.

The Sentence Transformers evaluation reference documents InformationRetrievalEvaluator, and the MTEB repository runs a model on your own retrieval task.

Procedure and references

When the answer changes

The results feed prices or refunds: the evaluation set becomes a mandatory release gate.
Users ask open questions with no fixed correct documents: a person rates a weekly sample of results instead.
Only you read the results: look at them yourself and skip the set.

Real incident + Code example

The leaderboard winner that lost our part numbers

On a help centre for an industrial parts supplier I worked on, we switched to an embedding model with a higher public retrieval average. Three weeks later support noticed that searches such as "HX-2040 seal kit" returned articles for neighbouring part numbers. We labelled 120 logged queries in two days: recall@10 was 0.84 for the old model, 0.71 for the new. On queries with a part number, the new model scored 0.52 against 0.90. We re-embedded 38,000 articles with the old model overnight and kept the 120 queries as the gate:

import json
from sentence_transformers import SentenceTransformer
from sentence_transformers.evaluation import InformationRetrievalEvaluator

def load(path):
    with open(path) as f:
        return json.load(f)

queries = load("eval/queries.json")    # {query_id: text}
corpus = load("eval/corpus.json")      # {doc_id: text}
relevant = {q: set(ids) for q, ids in load("eval/relevant.json").items()}

evaluator = InformationRetrievalEvaluator(
    queries, corpus, relevant, precision_recall_at_k=[10], name="own"
)

def recall_at_10(model_name):
    return evaluator(SentenceTransformer(model_name))["own_cosine_recall@10"]

current = recall_at_10("models/current")
candidate = recall_at_10("models/candidate")
assert candidate >= current, f"candidate {candidate:.2f} < current {current:.2f}"

Sources

arxiv.org/abs/2210.07316

Should I evaluate RAG retrieval separately?Yes
Do I need evals for my LLM app?Test it differently
Should I retest my app after an LLM model upgrade?Code under test: Test it differently
Should I write unit tests for machine learning code?Yes
Should I test prompts?Test it differently

FAQ

Should I test embeddings?: Do not unit test embedding vectors, because their exact values change with every model version. Test retrieval instead with an evaluation set of real queries and their answering documents, scored by recall@10.
Should I retest retrieval after changing the embedding model?: Yes, run the retrieval evaluation set on the old and the new embedding model before the switch ships. Re-embed the whole corpus, because vectors from two models in one index return plausible wrong documents.
How many queries does an embedding evaluation set need?: Start an embedding evaluation set with 100 real queries and their labelled answering documents. Add every query that users report as failing.
Is a public leaderboard score enough to choose an embedding model?: No, a public leaderboard score only narrows the shortlist of embedding models, because the MTEB authors found that no method wins on every task. Choose between the shortlisted models with recall@10 on your own queries.