Should I rerun evals before switching LLM models?

Yes, rerun the full evaluation set on the new model version before you switch production to it. Block the switch on any case that passed on the old version and fails on the new one.

Should I pin the LLM model version?

Yes, pin a dated model version in configuration, so a model upgrade happens only when you choose it and your evaluation set runs first. A model alias can move to a new version without a deploy on your side.

Do my unit tests catch problems from a model upgrade?

No, unit tests of the code around a language model pass unchanged after a model upgrade, because they do not grade the replies. An evaluation set of real inputs with a check for each case catches a reply that got worse.

Can I switch to a cheaper LLM without retesting?

No, run your evaluation set on a cheaper LLM before the switch, because a smaller model can follow the same prompt less exactly. Switch only when no case that passed on the current model fails.

Should I retest my app after an LLM model upgrade?

Answer

Yes, retest before the switch, with an evaluation set instead of unit tests: run 50 real inputs on the old and the new model version, and switch only when no case that passed on the old version fails on the new one.

Verdict on the code under testTest it differently

Why

Retest a model upgrade with an evaluation set of real inputs before the switch, for my typical case: a support assistant on a hosted model whose provider retires the pinned version and names a replacement. Blast radius is users, and Change frequency is rarely, because a team moves to a new model version a few times a year. Detectability is eventually, because a new model that follows one instruction differently still writes fluent replies and raises no error. Reversibility is with-effort: you can move back while the old version is still served, but users have already acted on the bad replies. Test cost is heavy, because each reply differs from run to run and needs graded cases, so rule R9 gives Test it differently.

When the decision changes
When	Decision	Why
Only you use the LLM app, and you read each reply before you act on it	Do not retest the model upgrade; switch, and read the replies as you use them	Blast radius falls to none: a worse reply reaches only you
Staff read and edit every reply of the LLM app before a customer sees it	Do not build a retest for the model upgrade; the staff member who edits each reply is the check	Blast radius falls to internal, Detectability to same-day, Reversibility to trivial
The model returns one label from a fixed list, such as a support queue	Test minimally: one CI test that runs labelled examples on the new model version and fails below the old version's pass rate	Test cost falls to moderate, because a label compares exactly; with Change frequency at rarely, one test pays off
The LLM app's replies quote prices or grant refunds	Test mandatory: evaluation cases for every pricing and refund rule gate the model switch, and a unit test covers an amount cap in code	Blast radius rises to money and Reversibility to costly
The LLM app answers questions from customer records	Test mandatory: evaluation cases that try to pull another customer's data run on the new model version, and tests check that retrieval filters by the signed-in customer	Blast radius rises to safety-or-legal and Detectability to never: a leaked record reads like a normal answer

What breaks if you don't test

Across a model upgrade the prompt, the code and every unit test stay the same, so CI passes. The new version reads the same prompt differently: it drops a field the old version always filled, or answers in another language. Nothing throws, and customers or support staff notice weeks later.

What you lose if you over-test

Unit tests that assert exact reply text fail on the new version, and someone rewrites the expected strings to match output nobody graded. A human review of 300 cases at 3 minutes each takes 15 hours, a figure from a published article on model swaps, which can push the switch past a provider's retirement date.

What to do instead

Pin a dated model version in configuration, so an upgrade happens only when you change it.
Keep an evaluation set of 50 real inputs from your logs, plus every input that produced a bad reply.
Run the set on the old and the new version. A case that passed before and fails now blocks the switch.
Keep unit tests for the code around the model: the template, the parser and the schema check.
After the switch, read a daily sample of replies for a week.

Anthropic's guide to building evaluations covers grading. The deprecation pages of Anthropic and OpenAI list retirement dates, which set your deadline.

Procedure and references

When the answer changes

A person reads every reply before a customer does: that reader is the check.
The output is a label from a fixed list: one exact test replaces graded cases.
Replies quote prices or read customer records: the evaluation set becomes a mandatory gate.

Real incident + Code example

The upgrade that answered in English

On a travel booking site I worked on, the help widget ran on a model version the provider was retiring. We ran our 60 evaluation cases on the replacement, saw no failures, and switched. Three weeks later an agent asked why satisfaction scores from German customers had dropped. The new version answered in English whenever a customer pasted an English booking confirmation into a German message. None of our cases mixed two languages. We added 12 such cases, fixed the prompt with one line, and now run this comparison before every version change:

# CASES and run_case are the project's own evaluation harness;
# case.problems(reply) returns a list of problems, empty when the reply passes.
OLD = "support-model-2025-10-01"  # pinned version IDs
NEW = "support-model-2026-06-15"

def regressions(cases):
    lost = []
    for case in cases:
        passed_old = not case.problems(run_case(case, OLD))
        passed_new = not case.problems(run_case(case, NEW))
        if passed_old and not passed_new:
            lost.append(case.id)
    return lost

if lost := regressions(CASES):
    raise SystemExit(f"new version fails cases the old one passed: {lost}")

Sources

dev.to/tokenhub_dev/build-the-eval-set-before-you-swap-the-model-1dlc

Do I need evals for my LLM app?Test it differently
Should I test prompts?Test it differently
Should I mock LLM calls in tests?Code under test: Yes
Should I test for hallucinations?Test it differently
Should I test AI agents?Test it differently

FAQ

Should I rerun evals before switching LLM models?: Yes, rerun the full evaluation set on the new model version before you switch production to it. Block the switch on any case that passed on the old version and fails on the new one.
Should I pin the LLM model version?: Yes, pin a dated model version in configuration, so a model upgrade happens only when you choose it and your evaluation set runs first. A model alias can move to a new version without a deploy on your side.
Do my unit tests catch problems from a model upgrade?: No, unit tests of the code around a language model pass unchanged after a model upgrade, because they do not grade the replies. An evaluation set of real inputs with a check for each case catches a reply that got worse.
Can I switch to a cheaper LLM without retesting?: No, run your evaluation set on a cheaper LLM before the switch, because a smaller model can follow the same prompt less exactly. Switch only when no case that passed on the current model fails.