AI engineering · 35 of 42

Golden Dataset

The cases you refuse to break

Scroll

The cases you refuse to break

A golden dataset is a small set of inputs with known good outcomes, run on every change. It is the regression suite for a system that has no deterministic output to assert on.

Twenty cases you genuinely care about beat two thousand scraped ones, because at twenty every failure gets read by a person rather than becoming a number that drifted.

The set should be deliberately uncomfortable: the thing that broke last month, the ambiguous phrasing, the request that should be refused, the empty input. And it should grow — every production incident becomes a case, which is how it stays worth running.

Evaluation
THE CASES YOU REFUSE TO BREAK the set, written by hand · a normal question · the one that broke last month · an ambiguous phrasing · something it should refuse · an empty / hostile input every change runs it 18 / 20 pass 2 regressions named, and blocking Twenty cases you actually care about beat two thousand scraped ones, because every failure in a small set gets read by a person. Add a case every time something breaks in production. That is how the set earns its name.
A hand-written set of cases run on every change, producing passes and named regressions.