The Judgment Gap how the gap between producing and checking opens › The ratio that tells you how wide your gap is 12 steps, no labs
03 — part 1, how the gap between producing and checking opensreasoning

The ratio that tells you how wide your gap is

One number tells you where you are, you can work it out from records you already keep, and the hard part is deciding what counts.

One number. Work produced in a period, divided by the work somebody actually looked at. Compute it before reading the rest, because every page after this one is easier to read against your own figure than against an argument.

work produced work examined the division is easy. The denominator is the whole problem counts as examined? somebody read it who could have caught a problem yes it passed through a review step no a check ran on it no nobody complained no produced, over examined one number, from records you keep the comfortable reading of the easy reading of examined number that means nothing.
The division is easy. The denominator is the whole problem, because the comfortable reading of *examined* gives you a number that means nothing.
What moves: the two pipes feed a filter labeled counts as examined, which accepts one candidate test and rejects three.

What counts as examined

This is the only difficult part and the whole value is in getting it right. Examined means somebody looked at it who had enough context to have caught a problem. It does not mean a review step was traversed, a check ran, or nobody objected.

does it count?why
somebody read it who could have caught a problemthis is the definition. Note the could have — not did
it passed through a review stepa step is a place, not an event. Ask whether anybody looked
an automated check ran on ituseful, and it is not judgment. A check tests what somebody thought of before
nobody complainedstep four is entirely about why this is not evidence

Take the strict reading. You will get an uncomfortable number, and an uncomfortable number you can act on is worth more than a comfortable one that is measuring review steps traversed.

With a few hundred pieces of work in a quarter and a strict count of what was genuinely read, most teams find something like three or four produced for every one examinedmodeled: an illustration only — substitute your own two counts. The number itself matters less than whether it is moving, which is why the instruction is to compute it and keep it.

→ Compute yours, for one quarter, before reading on. Two counts and a division, from records you already keep. Write it down with the date, because the second reading is worth more than the first and you will want to have taken the first.

Every term this page uses

produced-to-examined
How much work your team produced, divided by how much of it somebody actually looked at.
examined
Looked at by somebody with enough context to have caught a problem. Not the same as having passed through a review step.
remainder
Work that was produced and not examined. It does not go anywhere — it merges into what everything after it is built on.
judgment
Deciding whether a piece of work is good, which needs knowing the domain well enough to recognize a wrong answer.
inspection
Checking finished work at the end, rather than changing how the work is made.
checkable
Able to be judged without understanding the whole of it. The property that decides how much needs judgment at all.
constraint
The thing that limits output. Adding more of anything else does not help until it moves.
rework
Work that had to be revisited after somebody called it done.
vigilance
Staying alert for a rare problem across a long stretch of time.

Taken as already known, and so not defined here: agent, model, review, throughput, headcount. That list is a claim about who is reading, and it is printed so it can be argued with.