Reading an Eval Report whether the number can be read at all › What a share hides when its count is missing 12 steps, no labs
05 — part 2, whether the number can be read at allreasoning

What a share hides when its count is missing

A share is two numbers and you were shown one of them, so the useful move is to ask for the other one.

Every share is two numbers. Somebody worked out how many cases came back right, and divided it by how many cases there were. You were shown the result of the division and not the number underneath it, and the number underneath it decides how much the share is worth.

the same share, twice. Only the count is different over a handful of cases how wide the truth could be over a great many how wide the truth could be same marker different band
The same share over a handful of cases and over a great many. The marker is at an identical height in both panels; the band around it is not.
What moves: cases accumulate in both panels and the dashed band around the marker narrows on the right while the marker itself does not move.

Here is the whole of the arithmetic, and it is arithmetic you can do in a meeting. Fifteen right out of sixteen cases and a hundred and fifty right out of a hundred and sixty cases are the same share. The first could comfortably be hiding a true ability of anywhere from about two thirds to nearly everythingmodeled: fifteen right out of sixteen cases, and nothing else assumed. The second could not.

You do not need to know how that range is worked out. You need to know that it exists, that it is wide when the count is small, and that the count is the thing left off the slide.

Why this did not make the three

It is the most frequently violated question in this volume. Almost every result that circulates is a share with no count attached. And it still came fifth, for a reason worth stating plainly rather than hiding.

The answer is a number, and a number on its own rarely changes what you do. Told the count was small, you conclude the result is thin — which you could have guessed. Told it was large, you conclude very little, because a large count over the wrong cases, or with the failures quietly removed, is worth no more than a small one. The question in the next step is the deeper version of this one, and it is the one that made the list.

The bad answer to watch for is a big count that arrived suspiciously fast. A large set of cases is expensive to build when somebody has to decide what the right answer is for each one. A very large set that appeared quickly usually means the right answers were generated rather than determined, and then you are measuring agreement with a guess.

Every term this page uses

denominator
The count a rate was worked out over. Ninety-four percent of what, and of how many.
baseline
What the same measurement says about doing it the old way, or the trivial way, or not at all.
held-out
Cases kept away from whoever built the thing, so passing them means something.
contamination
The thing being tested has already seen the test cases, so the score measures memory rather than ability.
grader
Whoever or whatever looked at each answer and decided it was right or wrong.
agreement
How often two graders looking at the same answers said the same thing.
abstention
A grader allowed to return a third verdict: I cannot tell.
stale
Measured on a version that no longer exists.
exhibit
A report made up for this volume, for you to judge. Its numbers are not claims about the world.

Taken as already known, and so not defined here: eval, test, model, agent, metric. That list is a claim about who is reading, and it is printed so it can be argued with.