Is LLM-as-a-judge reliable?

An LLM judge is reliable for your rubric only after you measure it against a person's grades on that rubric. In the MT-Bench study, GPT-4 as a judge agreed with human preferences more than 80% of the time on pairwise comparisons of chat answers, with position, verbosity and self-enhancement biases.

Can you trust LLM-as-a-judge evals?

Trust LLM-as-a-judge evals when the judge fails at least 80% of the replies a person failed and passes at least 90% of the replies a person passed, on 30 or more labelled replies. Those thresholds are mine; raise them when a missed bad reply costs more.

Should the judge be a different model from the one it grades?

Yes, use a different model as the judge than the model that wrote the reply. The MT-Bench study found that a judge can favour answers from its own model.

How many human labels do I need to check an LLM judge?

Check an LLM judge on at least 30 replies graded by a person, with at least 10 failures among them. Add every production reply where a person later disagrees with the judge.

Should I use an LLM as a judge?

Answer

Use an LLM as a judge in your evaluation set only for qualities code cannot check, such as tone or completeness, and trust its grades only after they match a person's pass and fail grades on 30 labelled replies; check facts and formats in code.

Verdict on the code under testTest it differently

Why

Blast radiususers
Change frequencyregularly
Detectabilityeventually
Reversibilitywith-effort
Test costheavy

Use an LLM as a judge for what code cannot check, and grade the judge against a person first. My typical case is a support assistant whose free-text replies customers read. Blast radius is users, because a judge that passes a bad reply lets it reach customers. Change frequency is regularly: the prompt, rubric or judge model changes about once a month. Detectability is eventually, because a lenient judge still prints a green report, and Reversibility is with-effort: support corrects the replies after complaints. Test cost is heavy, because replies differ on every run and only a person can say what a good one is, so rule R9 gives Test it differently.

When the decision changes
When	Decision	Why
Only you read the replies of the LLM app, to try an idea	Do not set up an LLM judge; read the replies yourself and save each bad one as a future evaluation case	Blast radius falls to none: a bad reply reaches only you
Support staff read and edit every drafted reply before a customer sees it	Do not set up an LLM judge; the staff member who edits each draft is the check	Blast radius falls to internal, Detectability to same-day, Reversibility to trivial
The LLM app returns one label from a fixed list, such as a ticket category	Test without an LLM judge: assert the expected label for labelled examples in CI, with a minimum pass rate	Test cost falls to moderate: a label compares exactly; Detectability stays eventually
Replies go to customers by email with no person reading them first	Test: the judged evaluation set blocks the release on a failed case	Reversibility rises to impossible: a sent email cannot be recalled
The assistant states refund amounts or cancellation fees	Test mandatory: evaluation cases for every refund and fee rule, with each stated amount checked in code; the LLM judge grades only the wording	Blast radius rises to money and Reversibility to costly: a judge can pass a fluent reply with a wrong amount
The assistant answers from customer records	Test mandatory: evaluation cases that try to pull another customer's data, with a code check for other customers' identifiers in each reply	Blast radius rises to safety-or-legal and Detectability to never: a leaked record reads like a helpful answer to a judge

What breaks if you don't test

Without a judge, tone and completeness go unchecked, because nobody reads 200 replies after each prompt edit. An uncalibrated judge is worse: on a set where 3 of 30 replies are bad, a judge that passes all 30 agrees with the person on 90% of replies and catches none. The team ships a prompt change on that report, and customers find the replies that skip a step.

What you lose if you over-test

A set of 200 cases graded on 5 criteria makes 1,000 judge calls per run. A judge asked for a numeric score can give the same reply a 7, then an 8, and a threshold on that score fails cases at random. A judge that checks an order number or a label adds error to a check that an equality assertion does exactly.

What to do instead

Grade 30 replies from your logs as pass or fail against a written rubric, with at least 10 failures among them.
Write the judge prompt with one criterion per call, a short reason, then a pass or fail answer. Use a different model from the one that wrote the reply, as Anthropic's guide to building evaluations advises.
Measure two rates on the labelled replies: failed replies the judge fails, and passed replies it passes.
When the judge compares two replies, run each pair in both orders and count a win only when both orders agree.
Rerun the calibration when the rubric or judge model changes.

Inspect scorers include model-graded scoring.

Procedure and references

When the answer changes

The output is a label, a field or JSON: assert it in code instead.
A person reads every reply before a customer does: that reader is the check.
Replies state amounts or read customer records: the rules become mandatory tests in code, and the judge grades only the wording.

Code example

A judge that passes everything scores 90%

Plain agreement hides a lenient judge, because most logged replies are good. I split calibration into two rates with thresholds:

# labels.jsonl: replies graded "pass" or "fail" by a person.
# judge() is the project's own call to the grader model with the rubric.
import json

rows = [json.loads(line) for line in open("evals/labels.jsonl")]
bad = [r for r in rows if r["human"] == "fail"]
good = [r for r in rows if r["human"] == "pass"]

def grade(r):
    return judge(r["input"], r["reply"], rubric="evals/rubric.md")

caught = sum(grade(r) == "fail" for r in bad) / len(bad)
kept = sum(grade(r) == "pass" for r in good) / len(good)
print(f"bad replies failed: {caught:.0%}, good replies passed: {kept:.0%}")
assert len(bad) >= 10, "label more failed replies first"
assert caught >= 0.8 and kept >= 0.9, "rewrite the rubric, then rerun"

A judge that passes all 30 replies, 3 of them bad, catches 0% and fails the check.

Sources

arxiv.org/abs/2306.05685

Do I need evals for my LLM app?Test it differently
Do I still need human review of LLM outputs?Code under test: Test it differently
Should LLM evals run in CI?Code under test: Test it differently
Should I test for hallucinations?Test it differently
Should I test prompts?Test it differently

FAQ

Is LLM-as-a-judge reliable?: An LLM judge is reliable for your rubric only after you measure it against a person's grades on that rubric. In the MT-Bench study, GPT-4 as a judge agreed with human preferences more than 80% of the time on pairwise comparisons of chat answers, with position, verbosity and self-enhancement biases.
Can you trust LLM-as-a-judge evals?: Trust LLM-as-a-judge evals when the judge fails at least 80% of the replies a person failed and passes at least 90% of the replies a person passed, on 30 or more labelled replies. Those thresholds are mine; raise them when a missed bad reply costs more.
Should the judge be a different model from the one it grades?: Yes, use a different model as the judge than the model that wrote the reply. The MT-Bench study found that a judge can favour answers from its own model.
How many human labels do I need to check an LLM judge?: Check an LLM judge on at least 30 replies graded by a person, with at least 10 failures among them. Add every production reply where a person later disagrees with the judge.