Authoring was always the most automatable part
Writing tests was always the most automatable thing in the discipline. Everything in this article follows from that one fact, and none of it is a claim about how good the generated tests are.
Work automates well when it is bounded, repetitive, low-stakes and checkable. A test case is all four: it has clear edges, you write a great many that rhyme, a bad one is survivable, and it either compiles and runs or it does not. There is almost nothing in an engineering organization that scores better on those four properties.
So authoring went first, and it went fast. That is not a prediction; it is the thing that already happened, and it is worth separating from the argument people usually have about it. The interesting question is not whether generated tests are any good. It is what becomes scarce once producing them stops being the constraint.
Four things move, and they move in different directions. One becomes a hard ceiling where it used to be a soft cost. One becomes the center of the job. One changes shape rather than degree. And one leaves a central organization needing to say out loud what it is for.
Device hours became the constraint
Authoring was a cost. Execution is a wall. When the cost collapsed, the wall became the thing that decides.
Running a test consumes device time, build time and a slot in a cycle. In an organization with a shared physical fleet and dated release trains, that capacity is finite in a way a budget is not: you cannot answer a tenfold increase in tests by spending tenfold, because there is no supplier to spend it with. The fleet is the fleet, and it is already contended.
The consequence is easy to state and uncomfortable to act on. If a suite grows by a large multiple against a fixed fleet, exactly one of two things happens: the cycle gets proportionally longer, or a subset runs. Nobody accepts the first, so the second happens — and for most organizations it happens implicitly, decided by whatever timed out, whatever somebody disabled last quarter, and whatever the scheduler reached first.
That used to be fine, because the selection was made upstream and invisibly: tests existed only if somebody had time to write them, so scarcity of authoring effort was the selection mechanism. Remove it and nothing takes its place automatically.
Which makes selection the job it never was
Test selection has always existed as a topic and almost never as somebody's responsibility. It is about to be the most consequential recurring decision in a quality organization, because it decides which risks are covered this cycle and which are not.
| what changes | from | to |
|---|---|---|
| what limits the suite | how much somebody could write | how much the fleet can run |
| how selection happens | implicitly, via authoring effort | explicitly, or badly |
| what a new test costs | the time to write it | a slice of a capacity somebody else also wanted |
| what the scarce skill is | writing a good test | knowing which test earns its minutes |
A rough sense of the shape: if a cycle can run a fixed number of device hours and a generated suite would need several times that, the share you can actually run is a minority of it, and a shrinking oneillustration: a fixed fleet against a suite growing faster than the fleet does. It does not matter what those numbers are on your fleet. What matters is that the fraction is now something you choose rather than something that happens.
What to do about it
- Give selection an owner and a budget in device hours. Not a policy — a named person and a number, reviewed on a cadence, the way you would treat any other contended resource.
- Make the cost of a new test visible at the moment it is added. If adding a test is free at the point of decision, the capacity gets spent by whoever is least aware of it.
- Write down the current selection rule, whatever it is. Most organizations have one and cannot state it, and an unstated rule cannot be argued with or improved.
Attribution is the job now
Failure volume scales with tests multiplied by builds multiplied by trains. Deciding what a failure means does not scale at all, because it needs the system in somebody's head.
Every red test is one of four things: a product regression, a bug in the test, an infrastructure problem, or a difference in the environment it ran in. Choosing between those four is the work. It requires knowing what the system is supposed to do, which is exactly the knowledge that does not arrive with more tests.
This is the part of the discipline that gets larger rather than smaller, and it gets larger in the worst way: the volume grows with the suite while the capacity to do it grows with headcount. A team that doubles its tests has roughly doubled its attribution load and has not doubled anything else.
The reframe for a release-train organization
Where quality is accounted for in escapes against dated releases, the escape review has always asked one question: why did we not have a test for this? That question is becoming the wrong one, because increasingly a test did exist. Three better ones replace it.
| ask instead | what a bad answer reveals |
|---|---|
| did a test for this exist, and did it run? | if it existed and did not run, this is a selection failure, not a coverage gap — and it belongs to whoever owns the previous section |
| if it ran and failed, who saw the failure? | if the answer is that it was in a batch nobody triaged, this is an attribution capacity failure |
| if somebody saw it, why was it not believed? | usually because the suite's history made it look like noise, which is the next section |
Those three sort escapes into three different remedies owned by three different people, which the old question could not do. And they are answerable from records most organizations already keep.
The measurement nobody keeps. Pass rate isreported everywhere; time from failure to attribution is reported almost nowhere. It is the number that tells you whether your triage capacity is keeping up, it is the one that moves first when it stops keeping up, and it is available from the same records as everything else.
What to do about it
- Measure time from failure to attribution, and watch its trend rather than its level. The level depends on your system; the trend tells you whether the load is winning.
- Treat attribution as a staffed function, not an interrupt. It is now a large share of the work and it is still, in most places, something people do around their real objectives.
- Change the escape question to the three above. One line in a review template, and it routes each escape to a remedy that can actually be funded.
Flake arithmetic changes shape, not just degree
A flake rate that was an annoyance becomes a signal-destroying problem at the same rate, because the rate was never the thing that mattered. The product was.
Hold the flake rate constant and multiply the suite. The number of false failures multiplies with it, while the number of real ones does not — real failures track defects, not test count. So the ratio of noise to signal degrades as the suite grows, at an unchanged per-test quality.
That is why this is a change of shape rather than of degree. A flake rate that was perfectly tolerable is arithmetically intolerable at ten times the suite size, and no individual test has become any worse. Teams experience this as the suite mysteriously becoming untrustworthy and look for a cause among the new tests, where there is not one.
And the real cost is not the rerun
The instinctive cost model for a flaky test is the wasted execution. That is the cheap part, and the fleet is already being discussed in the first section. The expensive part is that every false failure consumes attribution time from the person in the previous section — the scarcest capacity in the discipline — and does so indistinguishably from a real one, because if you could tell them apart cheaply it would not be flake.
| what people budget | what it actually costs |
|---|---|
| the wasted device minutes | a slice of the one capacity you cannot buy |
| nothing | the attribution time, which is the binding constraint |
| nothing | the credibility of every other failure in the same batch |
Which means the flake target has to tighten as the suite grows. A fixed acceptable rate is a rate that gets worse in effect every time the suite expands. Expressing the budget as an absolute number of tolerated false failures per cycle, rather than as a percentage, makes that automatic — and makes the trade against suite growth explicit rather than accidental.
What to do about it
- Budget false failures in absolute terms per cycle, not as a rate. Then growing the suite has to be paid for in flake reduction, which is the correct coupling.
- Make a new test's flakiness a blocking property at the point of admission. A generated test is cheap to produce and not cheap to tolerate; the gate belongs where the supply is.
- Count what flake costs in attribution hours, once, and circulate it. The device minutes argument has never persuaded anybody. The triage hours argument does, because the people it persuades are the ones spending them.
What a central quality organization is for now
Part of what a central quality organization was for was writing the tests other teams did not. That capability is now available to everybody, which does not make the organization unnecessary — it makes its mandate something that has to be said out loud.
The old arrangement rarely needed stating. A central group held scarce expertise in producing test coverage, product teams wanted coverage, and the exchange was obvious enough that nobody had to write it down. When the scarce thing stops being scarce, an unstated mandate becomes an absence rather than an understanding.
The strongest available replacement is not a different kind of production. It is ownership of whether the signal can be trusted, which is three concrete things and no others.
| what a central organization owns | what that means in practice |
|---|---|
| what runs | selection against a capacity budget, as the first section describes. Somebody has to hold this and no product team can |
| what a pass means | the flake budget, the coverage claims that are not evidence, and the semantics of a green cycle |
| who answers when it says fine | and the release is not. Escape review, and the three questions from the attribution section |
The factor that inverts the usual advice
The generic version of this argument ends with centralization, because agents diffuse knowledge and a central group can apply it everywhere. In an organization where information deliberately does not cross team boundaries, that mechanism is unavailable: an agent cannot learn from a team it has no access to, so it cannot carry expertise between them either.
Which means local, embedded expertise becomes more valuable rather than less — the opposite of the usual conclusion. The shape that follows is embedded people who know one system deeply, and a central function that owns the three rows above rather than owning production. A central group that responds to this by trying to own generation will be competing with every product team for a thing that is no longer scarce.
What to do about it
- Write the mandate as one sentence and circulate it. If nobody disagrees, it was not specific enough. Disagreement is the point of the exercise.
- Stop competing on generation. Anything a product team can now do for itself is not a reason for a central group to exist, and defending it costs the credibility needed for the three rows that are.
- Put the embedded expertise where the systems are, and the signal ownership in the center. Those are different jobs with different ladders, and conflating them is how both get done badly.
The one thing that does not change
Through all four of these, one capability stays scarce and gets more valuable: somebody who can look at a failure and say, with reasons, whether it is real.
That is the common thread. Selection needs it, because choosing which tests earn their minutes is a judgment about which risks are real. Attribution is it, exactly. The flake argument is about protecting the capacity to exercise it. And a central mandate built on signal trustworthiness is an organization arranged around it.
So the discipline is not shrinking. Its center of gravity is moving from producing signal to judging it, and the roles, the levels and the hiring bar should follow — which is a larger change than it sounds, because producing was legible and judging is not. A test written is visible in a diff. An attribution made correctly in twenty minutes instead of four hours leaves no artifact at all.
And the honest gap this article does not solve. Writing tests was also how people learned the system. It was bounded, it was corrected, and it sat next to somebody doing the unbounded version — which is the standard shape of apprenticeship. Automating it removes the rung that produced the very judgment everything above depends on, and none of the four sections above addresses that. It is a real problem, it arrives on a delay of years, and it needs deliberate spending rather than a practice.
If there is one thing to take from all four: the scarce thing used to be the ability to produce coverage, and it is now the ability to decide what a result means. Staff, measure and promote for the second, and the first will take care of itself.