The Judgment Gap why the reporting improves while it happens › How the bill arrives, and why nobody can attribute it 12 steps, no labs
06 — part 2, why the reporting improves while it happensreasoning

How the bill arrives, and why nobody can attribute it

When it finally shows up there is no single change to point at, and that follows from how it accumulated.

It arrives as a class of problems rather than an incident, and the thing that makes it recognizable is what is missing from the investigation: there is no single change to point at.

the remainder assembled out of a great many correct pieces and then something goes wrong no single change to point at no decision anybody made nobody to ask what it was meant to do the question looking for a cause passes the whole thing and leaves which follows from how it accumulated: the remainder merged, so by now there is nothing left that is singly to blame.
Assembled out of a great many correct pieces. The question looking for a cause passes the whole thing and leaves.
What moves: the reservoir is high, an incident box names what is missing, and a query travels the full width of the frame and passes out of it.

That follows from the mechanism rather than being a separate observation. The remainder merged into what came after it, so by the time something fails, the failure rests on a long series of pieces each of which was individually plausible. There is no faulty change because there was no change — there was an accumulation.

what an investigation expectswhat it finds here
a change that introduced itmany changes, none of them wrong on its own
somebody who decided this behaviornobody. The behavior was inherited from something nobody examined
a record of what it was supposed to donothing, because nobody specified it — it emerged
a fix that prevents recurrencea fix for this instance, and no way to know how many others exist

The last row is why this is expensive rather than merely annoying. You can fix the instance. You have no way to estimate how many more there are, because the quantity you would need for that estimate is the remainder, and nothing measures it.

Do not expect this to be written up as what it is. A review that finds many small plausible contributions and no single cause will conclude that the system is complex and recommend more care. That is a reasonable conclusion from the evidence in front of it, and it is the wrong one, and nobody in the room has the number that would show why.

→ Name which of your currently-improving reported numbers you can no longer read at face value. The list is short, specific and yours — and having it written down is what stops you quoting one of them in a review next month without the caveat.

Every term this page uses

produced-to-examined
How much work your team produced, divided by how much of it somebody actually looked at.
examined
Looked at by somebody with enough context to have caught a problem. Not the same as having passed through a review step.
remainder
Work that was produced and not examined. It does not go anywhere — it merges into what everything after it is built on.
judgment
Deciding whether a piece of work is good, which needs knowing the domain well enough to recognize a wrong answer.
inspection
Checking finished work at the end, rather than changing how the work is made.
checkable
Able to be judged without understanding the whole of it. The property that decides how much needs judgment at all.
constraint
The thing that limits output. Adding more of anything else does not help until it moves.
rework
Work that had to be revisited after somebody called it done.
vigilance
Staying alert for a rare problem across a long stretch of time.

Taken as already known, and so not defined here: agent, model, review, throughput, headcount. That list is a claim about who is reading, and it is printed so it can be argued with.