Reading an Eval Report what a result is, and the three questions to carry › Why a good measurement arrives unreadable 12 steps, no labs
01 — part 1, what a result is, and the three questions to carryexhibit

Why a good measurement arrives unreadable

Somebody is about to show you a number and ask you to decide something, and this page is the number they are going to show you.

Here is a result of the kind that actually circulates. One card, forwarded into a review, sitting above a decision somebody wants today.

Evaluation summary — reply draftingconstructed for this volume
headline94% accuracy
what was testedthe reply-drafting agent, current build
recommendationenable for all support queues
prepared bythe team that built it
date of measurementnot stated

Everything on this card was made up for this volume. Your job is to judge it, not to believe it.

Read it the way you would read it in the meeting, and notice what you concluded. Most people conclude two things. That the agent works. And that the remaining question is a business one — which queues, how fast, who tells the team.

Neither conclusion is available from this card. Not because the number is wrong — assume it was measured honestly by people who were trying — but because a share on its own cannot carry the weight of either conclusion. There is nothing here to compare it against, nothing saying how many cases it was worked out over, nothing saying what a failure looked like, and no date.

nothing goes wrong here, and the number still arrives unreadable a measurement carefully done, by people who tried it leaves the room it was made in what it was over when, and on what who graded it what it was measured against everything that made it readable stays behind the room where you decide one number, on a slide, and a question for you it is high. Do we ship?
The usual defect is not a bad measurement. It is a good measurement arriving somewhere stripped of everything needed to read it.
What moves: the number travels from where it was measured into the room where the decision happens. Four labels travel with it and fall away at the boundary, so only the bare figure crosses.

What this volume is and is not

It is not a course in statistics and it assumes you know none. There is no arithmetic in it that you cannot do in your head, and every special word it needs is defined in a panel that travels with every page.

It is a set of eight questions you can ask out loud, in a room, without having read the work. Three of them do most of the job and they are on the next page. The other five are the deeper pass. Each question comes with what a bad answer sounds like, because the answers matter more than the questions and are harder to judge.

The card above comes back at the end of the volume, unchanged. You will not read it the same way, and the difference is the whole point.

Every term this page uses

denominator
The count a rate was worked out over. Ninety-four percent of what, and of how many.
baseline
What the same measurement says about doing it the old way, or the trivial way, or not at all.
held-out
Cases kept away from whoever built the thing, so passing them means something.
contamination
The thing being tested has already seen the test cases, so the score measures memory rather than ability.
grader
Whoever or whatever looked at each answer and decided it was right or wrong.
agreement
How often two graders looking at the same answers said the same thing.
abstention
A grader allowed to return a third verdict: I cannot tell.
stale
Measured on a version that no longer exists.
exhibit
A report made up for this volume, for you to judge. Its numbers are not claims about the world.

Taken as already known, and so not defined here: eval, test, model, agent, metric. That list is a claim about who is reading, and it is printed so it can be argued with.