AI engineering · 17 of 42

Recall@k

Did the needed evidence appear in the first k?

Scroll

Did the needed evidence appear in the first k?

Recall@k asks one question: was the evidence you needed anywhere in the top k results? If the right document ranks third and k is three, recall@3 is 1. If it ranks fourth, it is 0.

It deliberately ignores where inside k the document landed, which makes it the right metric for the retrieval stage specifically — ranking quality is what reranking and other measures are for.

It also says nothing about whether the final answer was correct. That separation is the point: measure retrieval on its own, and you can tell a bad answer caused by missing evidence from a bad answer caused by a model that had everything it needed.

Evaluation
DID THE ANSWER MAKE IT INTO THE SHORTLIST? why did the deploy fail? query retriever 1 release notes 2 build log 3 the actual cause 4 status page 5 runbook top k = 3, and the evidence is inside → recall@3 = 1 had it ranked fourth, recall@3 would be 0 It asks one question only: was the needed evidence anywhere in the first k? It does not care where inside k, and it says nothing about whether the final answer was right. Test retrieval with it, then test the rest separately.
Five ranked results with the correct evidence third, inside a top-k window of three.