AI engineering · 40 of 42

Drift

Nothing changed, and it got worse

Scroll

Nothing changed, and it got worse

Your prompt is the same, the model version is pinned, the code has not moved — and quality is falling. What changed is everything around it: new products, new phrasing from users, new failure modes, documents rewritten under the index.

It is gradual, which is what makes it dangerous. There is no incident, no alert and no commit to blame, so it gets noticed as a vague sense that things used to be better.

The defenses are unglamorous: run the golden set on a schedule rather than only on change, watch the score as a trend, sample real traffic periodically, and keep adding cases. A metric you only look at during deploys cannot show you a slope.

Operations
NOTHING CHANGED, AND IT GOT WORSE good bad month 1 month 12 The prompt is the same. The model version is pinned. The eval set has not moved. What changed is the world: new products, new phrasing, new failure modes, documents that were rewritten under the index. Which is why the golden set in concept 35 has to keep growing, and the score has to be watched.
A quality score sloping steadily downward over twelve months with nothing in the system having changed.