Reading an Eval Report Introduction 12 steps, no labs

Eight questions that tell you whether a number is evidence

Three of them you can carry in your head, each with what a bad answer sounds like — and one report here answers all eight and still misleads.

A high number is not evidence, and eight questions tell you which kind you have.

A share appears on a slide. It is high. Somebody you trust is quoting it, and there is a decision attached to it that they would like today. You are not going to read the work behind it, and you should not have to.

This volume is eight questions you can ask out loud in that room, without having read anything, and what a bad answer to each one sounds like. It is not a course in statistics and it assumes you know none. Every special word it needs is defined in a panel that travels with every page.

every result comes out of this four parts. You were shown the last one. the cases somebody chose these questions four, five, seven, eight the thing that runs them on some version, on some day question six the thing that grades them it ruled on every answer questions nine, ten one number the part you were shown question eleven the cases travel the whole length and arrive as one figure. Each question in this volume is a question about exactly one part. nothing here is about the number itself.
The spine of the volume: every result comes out of a machine with four parts, and each question is a question about one of them.
What moves: cases travel the whole length of the machine and arrive as a single figure. Arrows then point up from each question to the part it is about.

The three to carry, if you read nothing else

  1. Compared to what? A result with nothing beside it cannot be read.
  2. What is not in it? The runs that errored or came back empty, and the situations nobody wrote a case for.
  3. Show me three failures. If nobody can produce three, nobody looked.

The most frequently violated question in this volume is not one of those three — and the page that picks them says why, and prints the criteria so you can disagree with the ordering.

What you can do afterward

Ask three questions from memory in a meeting. Tell a report nobody read from one somebody read. Say plainly, without sounding difficult, that a result does not yet support the decision attached to it — and know the one kind of wrong report that none of this catches.

All eight questions, with what a bad answer to each one sounds like, are on one page you can print.

Every term this page uses

denominator
The count a rate was worked out over. Ninety-four percent of what, and of how many.
baseline
What the same measurement says about doing it the old way, or the trivial way, or not at all.
held-out
Cases kept away from whoever built the thing, so passing them means something.
contamination
The thing being tested has already seen the test cases, so the score measures memory rather than ability.
grader
Whoever or whatever looked at each answer and decided it was right or wrong.
agreement
How often two graders looking at the same answers said the same thing.
abstention
A grader allowed to return a third verdict: I cannot tell.
stale
Measured on a version that no longer exists.
exhibit
A report made up for this volume, for you to judge. Its numbers are not claims about the world.

Taken as already known, and so not defined here: eval, test, model, agent, metric. That list is a claim about who is reading, and it is printed so it can be argued with.