Verdict
Test it differently
An LLM app that users see needs evals: an evaluation set of 50 real inputs with a check for each, run on every prompt or model change, in place of unit tests that assert the reply text.
Why
- Blast radiususers
- Change frequencyregularly
- Detectabilityeventually
- Reversibilitywith-effort
- Test costheavy
Build evals for an LLM app that users see, and skip unit tests of the reply text. My typical case is a support assistant built on a hosted model. Blast radius is users, and Change frequency is regularly, because the prompt, retrieval or model version changes about once a month. Detectability is eventually, because a worse reply is still fluent and raises no error, and Reversibility is with-effort, because users act on bad replies before the revert. Test cost is heavy, because the reply differs on every run, so rule R9 gives Test it differently, with an evaluation set as the control.
| When | Decision | Why |
|---|---|---|
| Only you use the LLM app, to try an idea | Do not build evals for the LLM app yet; save each bad output as a future eval case | Blast radius falls to none: a bad output reaches only you |
| Staff edit every output of the LLM app before a customer sees it | Do not build evals for the LLM app; the staff member who edits each output is the check | Blast radius falls to internal, Detectability to same-day, Reversibility to trivial |
| The LLM app returns one label from a fixed list | Test: assert the expected label for labelled examples in CI, with a minimum pass rate | Test cost falls to moderate: a label compares exactly; Detectability stays eventually |
| The LLM app quotes prices or grants refunds | Test mandatory: eval cases for each pricing and refund rule, plus unit tests of an amount cap in code | Blast radius rises to money and Reversibility to costly |
| The LLM app answers from customer records | Test mandatory: eval cases that try to pull another customer's data, plus tests that retrieval filters by the signed-in customer | Blast radius rises to safety-or-legal and Detectability to never: a leaked record reads like a normal answer |
What breaks if you don't test
Without evals, a prompt edit or model upgrade is checked on a few inputs the developer tries. A change that fixes those inputs can break one nobody tried: the reply drops a fact or refuses a common request. A replacement for a retired model version can read the same prompt differently, and support hears about it from users weeks later.
What you lose if you over-test
An eval set that nobody prunes can grow to 2,000 generated cases, and at 1.2 seconds per sequential call a run takes 40 minutes. A grader model that scores one reply differently across runs fails a few cases each time, and the team stops reading the report. Unit tests of the exact reply text break with each model version.
What to do instead
- Collect 50 real inputs from your logs, plus every input that produced a bad reply.
- Check in code where you can: an exact match for a label, a "contains" check for a fact such as an order number, a JSON schema for structured output. A person or a second model grades tone against a rubric.
- Run the set on every change to the prompt, retrieval or model version.
- Log the prompt and model version with each reply, read a weekly sample, and add bad replies to the set.
Anthropic's guide to building evaluations and OpenAI's evals guide describe grading methods, and Inspect runs eval sets from Python.
When the answer changes
- A person reads every output before a customer does: that reader is the check.
- The output is a label from a fixed list: exact assertions work as ordinary tests.
- The output touches money or personal data: the eval set becomes mandatory.
Cost estimate + Real incident
What an eval set costs against one missed regression
On a support tool I worked on, a shortened summary prompt stopped asking for the order number, and support agents looked up orders by hand for two weeks. Both sides, priced with rates I assumed:
| Item | Assumption | Cost |
|---|---|---|
| Build the set | 50 cases, 16 developer hours at $80 | $1,280 |
| Keep it working | 2 hours a month for a year at $80 | $1,920 |
| Run it | 100 runs of 100 model calls (reply and grader) at $0.01 | $100 |
| The regression | 10 agents lose 1 minute on each of 20 daily tickets for 10 days, at $35 an hour | $1,170 |
| Trace the cause | 8 developer hours at $80 | $640 |
The set costs $3,300 in its first year and the regression costs $1,810, so the set pays off at two caught regressions a year out of 12 monthly changes. Replies that customers read raise the price of each miss.
Related questions
- Should I use an LLM as a judge?Code under test: Test it differently
- Should LLM evals run in CI?Code under test: Test it differently
- Should I test prompts?Test it differently
- Should I test AI agents?Test it differently
- Should AI write unit tests?Yes
FAQ
- Are LLM evals worth it?
LLM evals are worth it when users read the output and the prompt or model changes about once a month. By my estimate, a 50-case set costs $3,300 in its first year and pays off by catching two regressions that each cost $1,810 of staff time.
- How many cases does an LLM eval set need?
Start an LLM eval set with 50 real inputs from your logs, plus every input that has produced a bad reply. Add each production failure as a new case.
- Can a language model grade my evals?
Yes, a second language model can grade replies against a written rubric, for qualities such as tone that code cannot check. Before you trust the grader, compare its scores with a person's on 30 replies.
- Should LLM evals run in CI?
Yes, run LLM evals in a CI job that starts when the prompt, retrieval code or model version changes. Block the release on a failed case when replies reach customers unread.