Reading an Eval Report whether the number can be read at all › Why a result with nothing beside it cannot be read 12 steps, no labs
04 — part 2, whether the number can be read at allreasoning

Why a result with nothing beside it cannot be read

A number on its own cannot be read, because reading it means knowing what the same measurement says about doing the job some other way.

This is the least sophisticated-sounding question in the volume and the most effective one. It is also the easiest to ask without friction, because it sounds like curiosity rather than suspicion.

A result is a position, and a position needs a reference point. Told that an agent resolved most of the cases it was given, you know nothing at all until you know what the people currently doing that work resolve, and what the crudest possible approach resolves. Those two numbers turn a figure into a claim.

the same handful of cases, measured two ways a small pile of cases the result the old way the gap both markers move, and they move together. neither number can be read on its own at this count. The distance between them holds still through all of it. so the gap is the claim, and the number is not. A result with nothing beside it gives you the jittering marker and no bracket.
Both numbers are unstable. The distance between them is not, which is why the comparison is the claim and the figure on its own is not.
What moves: two markers jitter up and down together while a dashed bracket between them keeps exactly the same height throughout.

Three reference points, in order of how often they are missing

ask forwhat it tells youwhat it sounds like when it is bad
the current process, on the same caseswhether this is an improvement at all“we did not measure the manual process” — then the comparison you are being asked to make has not been made
the obvious cheap approach, on the same caseswhether the expensive thing earned its cost“that would not be a fair comparison” — ask why not, specifically. Sometimes the answer is good
doing nothingwhether the problem was worth solvinglaughter, usually. Ask anyway: it is the reference point that most often reveals a small problem being solved expensively

On the same cases. Three words, and they carry most of this page. A result measured on one set and a reference point measured on a different set cannot be subtracted from each other, and the two sets are almost never the same unless somebody deliberately made them so.

The good answer to this question is short and boring: here is the number, here is what the old way scored on the identical cases, here is the gap. If you get that, this question is finished and you can move on. If you get a discussion, the gap does not exist yet.

Every term this page uses

denominator
The count a rate was worked out over. Ninety-four percent of what, and of how many.
baseline
What the same measurement says about doing it the old way, or the trivial way, or not at all.
held-out
Cases kept away from whoever built the thing, so passing them means something.
contamination
The thing being tested has already seen the test cases, so the score measures memory rather than ability.
grader
Whoever or whatever looked at each answer and decided it was right or wrong.
agreement
How often two graders looking at the same answers said the same thing.
abstention
A grader allowed to return a third verdict: I cannot tell.
stale
Measured on a version that no longer exists.
exhibit
A report made up for this volume, for you to judge. Its numbers are not claims about the world.

Taken as already known, and so not defined here: eval, test, model, agent, metric. That list is a claim about who is reading, and it is printed so it can be argued with.