Reading an Eval Report whether the number can be read at all › How a true number can describe something that no longer exists 12 steps, no labs
06 — part 2, whether the number can be read at allreasoning

How a true number can describe something that no longer exists

A number measured on a version that no longer exists is a true fact about something that no longer exists.

This is the cheapest question in the volume. Two words — when, and what version — and it is skipped more often than any other, because the answer feels like bookkeeping rather than evidence.

the number is still true. It is true about something else now the prompt changed a tool was added the model moved a threshold was tuned the cases were edited someone fixed a bug it changed again the measurement taken here, once and quoted about all of this two words fix it: ask for the date and the version. It is the cheapest question in this volume and the one most often skipped.
The measurement never moves and never becomes wrong. What grows is the distance between it and the thing it is being quoted about.
What moves: a marker travels rightward past seven changes while the pinned measurement stays where it was taken, and a bar showing the gap lengthens.

Notice what the drawing does not say. It does not say the measurement was sloppy or that anybody acted in bad faith. The number was true when it was taken and it is still true about the thing it was taken on. The thing it was taken on no longer exists.

This matters more here than it used to, and for a specific reason: the things being measured now change in ways that leave no trace a manager would notice. A prompt is edited. A tool is added. A threshold moves. None of those look like a release, none of them appear in a change log you read, and any of them can move a result more than a month of engineering would.

How to tell without reading a change log

askthe answer that is finethe answer that is not
when was this run?a date, immediately, without checking“recently”, or a date that arrives after a search
what exactly was running?something you could point at again and re-runa description of the system rather than an identifier of one
has anything changed since?a short specific list, or a confident no“nothing significant” — which is a judgment, made by somebody with an interest, about the thing in question

A result nobody can date is not a stale result. It is a result that cannot be placed at all, which is worse, because a stale number can at least be re-run on purpose.

→ Ask for the date and the version on the next result you are shown. Two words, and it is the cheapest of the eight questions to say out loud — nobody has ever taken offense at being asked when something was measured.

Every term this page uses

denominator
The count a rate was worked out over. Ninety-four percent of what, and of how many.
baseline
What the same measurement says about doing it the old way, or the trivial way, or not at all.
held-out
Cases kept away from whoever built the thing, so passing them means something.
contamination
The thing being tested has already seen the test cases, so the score measures memory rather than ability.
grader
Whoever or whatever looked at each answer and decided it was right or wrong.
agreement
How often two graders looking at the same answers said the same thing.
abstention
A grader allowed to return a third verdict: I cannot tell.
stale
Measured on a version that no longer exists.
exhibit
A report made up for this volume, for you to judge. Its numbers are not claims about the world.

Taken as already known, and so not defined here: eval, test, model, agent, metric. That list is a claim about who is reading, and it is printed so it can be argued with.