Reading an Eval Report what a result is, and the three questions to carry › The four parts every result comes out of 12 steps, no labs
03 — part 1, what a result is, and the three questions to carryreasoning

The four parts every result comes out of

Every result comes out of a machine with four parts, and each question in this volume is a question about one of those parts.

Every result you are shown comes out of the same machine, and the machine has four parts. Naming them is worth one page, because after this the eight questions stop being a list to memorize and become a map of one object.

every result comes out of this four parts. You were shown the last one. the cases somebody chose these questions four, five, seven, eight the thing that runs them on some version, on some day question six the thing that grades them it ruled on every answer questions nine, ten one number the part you were shown question eleven the cases travel the whole length and arrive as one figure. Each question in this volume is a question about exactly one part. nothing here is about the number itself.
The four parts, and which question attaches to each. Nothing in this volume is about the number itself.
What moves: cases travel the whole length of the machine and arrive as a single figure. Arrows then point up from each question to the one part it is about.
the partwhat it iswhat can go wrong with it
the casessomebody chose a set of situations to testthe wrong ones, too few of them, or ones the thing had already seen
the runthe cases were put through some version, on some daya version that no longer exists, and runs that never finished
the gradingsomething ruled on each answernobody knows what ruled, and nothing ever checked it
the numberthe rulings were added up and one figure came outnothing, usually. This is the part that is almost always correct

The last row is the surprising one. The arithmetic is nearly always right. Almost every bad result you will ever be handed is a correct sum over the wrong cases, or a correct sum whose losses happened earlier in the machine. Doubting the number is the wrong instinct and it makes you look unserious. Ask about the other three parts.

This is also how to handle a question this volume never asked. Something about a result bothers you and you cannot name it: work out which of the four parts your unease is about, and the question usually writes itself.

→ Find the last result anyone showed you and identify its four parts. Two of them are usually missing from the page entirely, and noticing which two is this whole volume in miniature.

Every term this page uses

denominator
The count a rate was worked out over. Ninety-four percent of what, and of how many.
baseline
What the same measurement says about doing it the old way, or the trivial way, or not at all.
held-out
Cases kept away from whoever built the thing, so passing them means something.
contamination
The thing being tested has already seen the test cases, so the score measures memory rather than ability.
grader
Whoever or whatever looked at each answer and decided it was right or wrong.
agreement
How often two graders looking at the same answers said the same thing.
abstention
A grader allowed to return a third verdict: I cannot tell.
stale
Measured on a version that no longer exists.
exhibit
A report made up for this volume, for you to judge. Its numbers are not claims about the world.

Taken as already known, and so not defined here: eval, test, model, agent, metric. That list is a claim about who is reading, and it is printed so it can be argued with.