Should evals block a deploy?

LLM evals should block a deploy on a code-checked case that regressed, such as a missing order number or invalid JSON. Graded scores block the deploy only when replies reach customers unread, or when replies touch money or customer records.

Should LLM evals run on every pull request?

No, LLM evals should run only on pull requests that change the prompt, retrieval code or model version, selected with a path filter. Unit tests with a fake model client cover every other change.

What should fail an eval job in CI?

An eval job in CI should fail when a case checked by code passes on the main branch and fails on the pull request. Scores from a grader model vary between runs, so the job reports them for a reviewer instead of failing on them.

Should LLM evals run in CI?

Answer

Yes, run LLM evals in CI, as a separate job that starts when the prompt, retrieval code or model version changes: it fails when a code-checked case passes on the main branch and fails on the pull request, and it reports graded scores for the reviewer to read before the merge.

Verdict on the code under testTest it differently

Why

Blast radiususers
Change frequencyregularly
Detectabilityeventually
Reversibilitywith-effort
Test costheavy

Run the evaluation set in its own CI job, apart from the unit tests. My typical case is a support assistant on a hosted model with an evaluation set of 100 cases. Blast radius is users, and Change frequency is regularly, because the prompt, retrieval or model version changes about once a month. Detectability is eventually, because a worse reply is still fluent and raises no error, and Reversibility is with-effort, because users act on bad replies before the revert. Test cost is heavy, because each reply differs from run to run, so rule R9 gives Test it differently, with the evaluation set as the control.

When the decision changes
When	Decision	Why
The LLM app sends its replies to customers as emails that no person reads first	Test: the LLM eval job in CI blocks the merge and the deploy on any failed case, graded cases included	Reversibility rises to impossible, because a sent email cannot be recalled
The LLM app returns one label from a fixed list, such as a support queue	Test: run the labelled examples as an ordinary CI test on every pull request, with a minimum pass rate	Test cost falls to moderate, because a label compares exactly
Staff read and edit every output of the LLM app before a customer sees it	Do not add an eval job to CI; the staff member who edits each output is the check	Blast radius falls to internal, Detectability to same-day, Reversibility to trivial
The LLM app quotes prices or grants refunds	Test mandatory: eval cases for every pricing and refund rule block the merge, and a unit test covers an amount cap in code	Blast radius rises to money and Reversibility to costly
The LLM app answers questions from customer records	Test mandatory: eval cases that try to pull another customer's data block the merge, and tests check that retrieval filters by the signed-in customer	Blast radius rises to safety-or-legal and Detectability to never: a leaked record reads like a normal answer

What breaks if you don't test

An evaluation set that runs only from a developer's laptop gets skipped on the small prompt edit that looks harmless. The unit tests pass, because a fake model client returns the same reply as before. The real model drops a fact or refuses a common request, and users report it weeks later.

What you lose if you over-test

An eval job on every pull request pays for model calls on changes that never touch the model: $750 a month for 250 pull requests at $3 per run, against $48 for a path-filtered job plus a weekly run. A job that blocks the merge on a grader model's score fails a few cases on each run by chance, and the team reruns it until it passes.

What to do instead

Keep fake-model unit tests of the prompt template, parser and error handling in the regular CI job.
Add a separate eval job with a path filter on the prompt files, the retrieval code and the model version setting. GitHub Actions documents path filters under on.pull_request.paths.
Fail the job on a code-checked case (an exact label, a required order number, a JSON schema) that passes on the main branch and fails on the pull request.
Post graded scores next to the main branch's scores, and let the reviewer read every case whose score dropped.
Run the full set weekly on a schedule too, because help articles and model aliases change without a pull request.

Anthropic's guide to building evaluations covers grading methods, and Inspect runs an eval set from the command line.

Procedure and references

When the answer changes

Replies reach customers unread: every failed case blocks the merge.
The output is a label from a fixed list: the cases become an ordinary test.
A person reads every output first: that reader is the check.

Code example + Cost estimate

A path-filtered eval job and its monthly bill

The workflow I use, where evals/run.py is the project's own runner:

name: evals
on:
  pull_request:
    paths: ["prompts/**", "app/retrieval/**", "config/model.yaml"]
  schedule:
    - cron: "0 6 * * 1" # Mondays: catches help article and alias changes
jobs:
  evals:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      # fails on code-checked regressions, posts graded scores as a comment
      - run: python evals/run.py --baseline main --fail-on code-checks
        env:
          MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}

The monthly cost, with assumptions I chose for a team of five developers:

Item	Assumption	Cost
One model call	3,000 input tokens at $3 per million, 400 output tokens at $15 per million (assumed prices)	$0.015
One run	100 cases, each a reply and a grader call, 10 in parallel at 8 seconds: 80 seconds	$3
Eval job on every pull request	250 pull requests a month	$750 and 5.6 CI hours
Path-filtered job plus weekly run	12 pull requests touch the prompt, retrieval or model; 4 scheduled runs	$48

The path-filtered job costs $576 a year, less than one missed regression that takes a week of one support agent's time at $35 an hour ($1,400).

Sources

jimbobbennett.dev/blogs/evals-in-your-ci-cd-pipeline

Do I need evals for my LLM app?Test it differently
Should I test prompts?Test it differently
Should I mock LLM calls in tests?Code under test: Yes
Should I retest my app after an LLM model upgrade?Code under test: Test it differently
Should I use an LLM as a judge?Code under test: Test it differently

FAQ

Should evals block a deploy?: LLM evals should block a deploy on a code-checked case that regressed, such as a missing order number or invalid JSON. Graded scores block the deploy only when replies reach customers unread, or when replies touch money or customer records.
Should LLM evals run on every pull request?: No, LLM evals should run only on pull requests that change the prompt, retrieval code or model version, selected with a path filter. Unit tests with a fake model client cover every other change.
What should fail an eval job in CI?: An eval job in CI should fail when a case checked by code passes on the main branch and fails on the pull request. Scores from a grader model vary between runs, so the job reports them for a reviewer instead of failing on them.