Can automated evals replace human evaluation?

Automated evals replace human evaluation for grading each change, but not for checking the evals themselves. A person still grades a sample of outputs against the grader's calls, because a grader passes any reply that matches a wrong reference.

How many LLM outputs should a person review?

For a support assistant, a person should grade about 30 random production outputs a week against the eval rubric. Every failure found becomes a new eval case.

Can I trust an LLM judge without human grades?

No, trust an LLM judge only after comparing its calls with a person's grades on the same cases, such as 50 hand-graded replies. Repeat the comparison on the weekly sample, because agreement can drop when the prompt, documents or model change.

Should a person approve every LLM output before it ships?

A person should approve every LLM output only when staff can read and edit each draft before sending it. At 2,000 replies a day, run the eval set in CI and grade a weekly sample instead.

Do I still need human review of LLM outputs?

Answer

Yes, keep human review of LLM outputs as a weekly sample: a person grades 30 production outputs against the eval rubric and checks the automated grader against those grades, instead of approving every output before users see it.

Verdict on the code under testTest it differently

Why

Blast radiususers
Change frequencyregularly
Detectabilityeventually
Reversibilitywith-effort
Test costheavy

Keep human review of LLM outputs as a sample that grades both the outputs and the automated grader. My typical case is a support assistant with automated evals. Blast radius is users, because customers act on the replies. Change frequency is regularly: the prompt, documents or model version change about once a month. Detectability is eventually, because a wrong reply reads as fluently as a right one, and Reversibility is with-effort: support corrects the customer after a complaint. Test cost is heavy, because no assertion decides whether free text is correct, so rule R9 gives Test it differently.

When the decision changes
When	Decision	Why
Only you read the LLM outputs, for your own work	Do not set up human review of LLM outputs; you judge each output as you use it	Blast radius falls to none: a bad output reaches only you
Staff read and edit every LLM output before a customer sees it	Do not add a sampled human review; the staff member who edits each output already reviews it	Blast radius falls to internal, Detectability to same-day, Reversibility to trivial
The LLM returns one label from a fixed list, such as a ticket category	Test: people label 200 examples once, and CI asserts those labels with a minimum pass rate	Test cost falls to moderate: a label compares exactly with a human label; Detectability stays eventually
LLM outputs go to customers by email with no person reading them first	Test: run the eval set as a release gate, and have a person grade the judge's pass and fail calls on 30 cases before each release	Reversibility rises to impossible: a sent email cannot be recalled
The LLM output states refund amounts or fees	Test mandatory: eval cases for every refund and fee rule gate each release	Blast radius rises to money and Reversibility to costly: a paid refund needs a claw-back
The LLM output gives medical or legal guidance	Test mandatory: a qualified person grades the eval set before each release, and the rules of your domain come first	Blast radius rises to safety-or-legal and Reversibility to impossible: wrong guidance can harm a person

What breaks if you don't test

A grader model checks replies against what you gave it: a rubric, reference answers, retrieved documents. When the reference is wrong, the grader passes wrong replies with high scores. A failure the rubric does not name, such as a reply that asks for a password, fails no check. Without a person reading replies, complaints reveal both weeks later.

What you lose if you over-test

Approving every output does not scale: 2,000 replies a day at 1 minute each is 33 hours of reading a day, and customers wait for approval. Reviewers who approve hundreds of fluent replies in a row start to skim. A human-only eval of 300 cases at 3 minutes a case takes 15 hours, too slow to repeat on each prompt change.

What to do instead

Grade 50 eval cases by hand, noting why each reply passes or fails.
Run the grader on the same 50 cases. Keep it only for criteria where it agrees with the person on at least 45 cases; OpenAI's evaluation guide names skipped calibration against human evals as an anti-pattern.
Run code checks and the grader in CI on every prompt, retrieval or model change, as Anthropic's guide to building evaluations describes.
Each week, a person who knows the product grades 30 random production replies and records where the grader disagreed. Each failure becomes an eval case.

Procedure and references

When the answer changes

A person edits every output before a customer sees it: that editor is the review.
Outputs go to customers by email with no person reading them first: human-checked evals become a release gate.
The output is a label: people label examples once, and exact assertions replace ongoing review.

Real incident

The grader that trusted old documentation

On a support assistant I worked on for a project management tool, we replaced the weekly human read with a grader model after it matched our grades on 50 cases. Four months later the export settings moved from the Account menu to a new Workspace page, and the documentation lagged two sprints. The assistant kept sending customers to the Account menu, and the grader scored those replies 5 of 5, because they matched the retrieved article. A support lead spotted it three weeks later, after 400 affected conversations. We brought back the weekly sample, graded by a support agent who uses the product daily.

Should I use an LLM as a judge?Code under test: Test it differently
Do I need evals for my LLM app?Test it differently
Should LLM evals run in CI?Code under test: Test it differently
Should I test for hallucinations?Test it differently
Should I retest my app after an LLM model upgrade?Code under test: Test it differently

FAQ

Can automated evals replace human evaluation?: Automated evals replace human evaluation for grading each change, but not for checking the evals themselves. A person still grades a sample of outputs against the grader's calls, because a grader passes any reply that matches a wrong reference.
How many LLM outputs should a person review?: For a support assistant, a person should grade about 30 random production outputs a week against the eval rubric. Every failure found becomes a new eval case.
Can I trust an LLM judge without human grades?: No, trust an LLM judge only after comparing its calls with a person's grades on the same cases, such as 50 hand-graded replies. Repeat the comparison on the weekly sample, because agreement can drop when the prompt, documents or model change.
Should a person approve every LLM output before it ships?: A person should approve every LLM output only when staff can read and edit each draft before sending it. At 2,000 replies a day, run the eval set in CI and grade a weekly sample instead.