Which of your numbers stopped meaning what they meant
Four metrics examined, four different ways they broke, what replaces each one — and the one judgment no number left can support.
Your metrics did not get worse. They started measuring something else.
Every number you report went up. None of them is computed wrongly. And most of them were never measurements of anything you cared about directly — they were things that were easy to count and that used to move when the thing you cared about moved.
That connection is what broke, and a broken connection does not make any gauge read zero. This volume names which of your numbers are affected, in which of four ways, and what to put in their place.
Four metrics, four different failures
| the metric | how it broke |
|---|---|
| counting output | the stand-in stopped standing in for effort or for value |
| cycle time | the stretch being timed moved, and the queue moved outside it |
| individual records | who did the work stopped being a decidable question |
| coverage and its relatives | it was only ever meaningful because it was expensive |
The four were chosen because each breaks differently, so the set generalizes. By the third step you should be able to apply them to a number this volume never mentions.
Every condemned number here comes with a replacement, or an explicit admission that there is not one. There is exactly one of the latter. It is about judging individual people, and it is the most important page in the volume.
What you can do afterward
Mark every line on your own dashboard as readable, readable-with-a-caveat, or retired. Compute the one number most teams do not, from records you already keep. Walk into a calibration meeting knowing which number cannot help you. And retire a metric upward without it sounding like an excuse.
The four failures, the replacements and the retirement script are on one page you can print.
Every term this page uses
- proxy
- Something easy to count that stands in for something you actually care about and cannot count directly.
- denominator
- The count a rate was worked out over. How many, not what share.
- attribution
- Deciding which person or which change produced a result.
- cycle time
- How long one piece of work takes from being started to being shipped.
- rework
- Work that had to be revisited after somebody called it done.
- outcome measure
- Something measured about the result rather than about the effort spent.
- gameable
- Cheap to satisfy without doing the thing the measure was meant to reward.
- lag
- The delay between something happening and the number moving in response.
- target
- A measure somebody is expected to move, rather than one you only watch.
Taken as already known, and so not defined here: agent, metric, dashboard, pull request, coverage. That list is a claim about who is reading, and it is printed so it can be argued with.