You have a number that says your agent works. This is how to find out whether the number means anything

Twelve steps, four parts, three systems under test — and the component whose removal improved the score in all three.

You have a number. This is how to find out whether it means anything.

Somewhere there is a percentage that says your agent works. Somebody quotes it in a review. It is high, and it keeps being high while the agent does things in production that the number said it would not do.

This volume is about the gap between those two facts. It is not about statistics, and it assumes you know nothing about them: every term it needs is explained on the page that uses it, and the definitions travel with every page.

The spine of the volume: three situations, and how much of the answer you hold in each.

What moves: the box of things you can compare an answer against shrinks panel by panel. In the third it is empty and the comparison arrow ends in nothing.

The one result worth arriving for

One word decides whether the expensive part of an eval helps or hurts, and it is not a word about the model. It is whether the thing is allowed to say I cannot tell.

Add a judge that must answer every case, and removing it improves recall in all three systems. Give the same judge one extra option and removing it makes things worse in all three. Same detection, same corpus, one word of difference.

system	a judge that must answer	the same judge, allowed to abstain
the code-fixing agent	-0.06 recall from removing it	+0.08 recall from removing it
the failure classifier	-0.20 recall from removing it	+0.00 recall from removing it
the document-QA agent	-0.41 recall from removing it	+0.00 recall from removing it

A negative number means the eval was better off without the judge. Every entry in the left column is negative and no entry in the right column is.

Coverage is not accuracy. Something that answers when it does not know is worse than nothing, because it turns a visible gap into an invisible error. The repair is not to remove it. The repair is to let it say so.

What you can do afterward

Say what your number is counting. Name the mistake your eval is structurally unable to see. Know which of six things to build next, and — measured, not asserted — when to stop building.

Part 1 — why the number you have is not evidence

01The eval that reports everything is finerecognize their own eval in the worst one measured heremeasured

02Three situations, and how much of the answer you holdsay which of the three they are in before choosing an instrumentconcept

03Names for the ways an answer can be wrongname the four outcomes, and say which one leaves no traceconcept

Part 2 — building something worth measuring with

04Why your own traffic will not contain your hard casesstop waiting for real failures to accumulatemeasured

05Breaking right answers on purposebuild a corpus where no label is anybody's opinionmeasured

06Cheapest check firstorder their checks, and avoid the bug that empties a suitemeasured

Part 3 — four results that change how a number is read

07One word decides whether the expensive part helps or hurtscheck what a verdict-giving component does when it has no opinionmeasured

08Letting the eval say it cannot tellprefer a smaller honest answer to a complete guessmeasured

09The wrong answer that passes every general checksee what their agent's design already decided about checkabilitymeasured

10Watching a live system with no answers to compare againsttell a dead signal from a dirty one from a clean onemeasured

Part 4 — what to do about it

11Why tuning the alarm is the wrong first moveask whether the signal is wrong before the threshold ismeasured

12What to build, in what order, and when to stopfollow an order that ends early, and turn the dial that ends itmeasured

Every term this page uses

recall: Of the answers that were genuinely wrong, the fraction your eval caught.
precision: Of the answers your eval flagged as wrong, the fraction that really were.
agreement: How often the eval and the answer key said the same thing.
corpus: The set of answers your eval grades. Also called an eval set.
grader: One check that looks at one answer and returns a verdict.
judge: A grader that asks a model to rule on answers the cheap checks could not decide.
abstention: The eval keeping a third answer beside right and wrong: I cannot tell.
horizon: How much of the right answer you hold. There are three, and they shrink.
detector: A check that looks for one specific kind of wrongness.
monitor: Something watched continuously in production, with no answer key to compare against.
threshold: A rule that waits for a problem to persist before raising an alarm.
label: What the answer key says about one row: right or wrong.
operator: One specific edit that breaks a right answer, and so decides its label.
stale: An answer that quotes a real source correctly and is wrong, because the figure moved.
grounding: Whether a claim is actually supported by the source the answer points at.
false negative: A wrong answer your eval passed. The only outcome that leaves no trace.

Taken as already known, and so not defined here: eval, agent, model, prompt, tool, test, harness. That list is a claim about who is reading, and it is printed so it can be argued with.