AI engineering · 18 of 42

LLM-as-a-Judge

Use a model to grade another model's output

Scroll

Use a model to grade another model's output

Once outputs are open-ended, exact-match scoring stops working: a good answer worded differently fails, and a bad answer with the right keywords passes. So a second model reads the question, the answer and a rubric, and grades it.

It scales in a way human review does not, and it can apply criteria that no string comparison could express.

It is also a model, with everything that implies. Judges reward length and fluency, favor their own writing style, miss subtle factual errors, and give confident scores either way. Before trusting one, check its agreement against a set of human labels, and keep checking — a judge that drifts silently is worse than no judge at all.

Evaluation
A MODEL GRADING ANOTHER MODEL the question the answer given the rubric judge model grade 4 / 5 PASS Exact string matching cannot tell a good answer from a badly worded one, so at any scale this is often the only grader available. The judge has its own biases: it favors length, fluency and its own style, and a confident score is not a correct one. Check it against human labels first.
A question, an answer and a rubric going to a judge model, which returns a score and a verdict.