How to Prove Your Search Is the Problem, Not the Model

When your agent returns a wrong answer to a customer, find out whether the right answer is anywhere in the passages that the model receives, before you swap that model over again.

In your product, a search step retrieves passages from your own documents, from which a language model then generates the answer to the user's question. Since those two steps fail in different ways and you repair each of them differently, your first job is to tell the two failures apart. To do that, you need a set of real questions on which you mark the right answers, along with a few numbers that you can read correctly. In the previous article in this study, I argue that the search step usually breaks first, while in this one I will show you how you can prove that on your own system.

You have to take more care to read those numbers correctly than to collect them, because you can't act on a retrieval score on its own.

Take one recorded run as the example. In Narsil's benchmarks, I score search engines on public test collections from BEIR, where each collection consists of a corpus, a question set, and a set of marks for the documents that hold the answer to each question. In my run of 28 September 2026, one engine in one fixed configuration, BM25 with k1 at 0.9 and b at 0.4, scores 0.6814 on a collection called SciFact and 0.3278 on one called NFCorpus. The metric, the engine, the settings, and the machine are identical across both of them.

The same engine scores 0.68 and 0.33 in one run

Neither of those numbers is a measure of the engine on its own, because each of them is a measure of a combination of five things: this engine, this corpus, this question set, this configuration, and this metric. If you change any one of the five, the score changes with it.

The two scores differ because the two corpora differ. In SciFact, each question is a scientific claim whose right document is the paper with the evidence for or against it. The judges there mark 339 relevant documents across 300 questions, which comes to roughly one right answer per question. In NFCorpus, each question is a plain-language question about health, whose answers are in a collection of medical documents. The NFCorpus authors build 12,334 relevance marks across 323 questions automatically from the links on the NutritionFacts.org website, which comes to roughly 38 marked documents per question. Finding that single paper is a different job from finding all 38 marked documents, which means that one score is no guide to which of the two jobs the engine does well.

Take that as the first rule for reading anybody's benchmark, including mine. From a published score, you can learn how an engine compares with the other engines in the same run, on the same kind of machine. You only learn what an engine scores on your own corpus when you measure it yourself.

How to read each of the four numbers

The authors of the retrieval papers that I cite in this article report the same four numbers, each of which is a measure of a different property of a result list.

Recall@k is the share of the documents that you mark as right for a question that the engine returns inside the top k results. Under recall, each of those documents counts equally wherever it appears in the list, which means that a system that ranks every correct passage around position ninety still scores well at a k of 100.

Precision@k is the share of the right documents among the k documents that the engine returns. Precision depends only on what the engine returns, which means that a system that returns three correct passages out of a possible forty can score perfectly at a k of 3.

For MRR, which stands for mean reciprocal rank, you take the position at which the first correct document appears and average one over that position across all of your questions. The calculation stops at that first hit, which leaves everything below it unscored.

nDCG@k, which stands for normalised discounted cumulative gain, is a score for the whole ranked list. A correct document near the top counts for more than the same document further down, after which the formula divides the total by the score of a perfect ordering. To get a high score, the engine therefore has to find the right documents and also order them well, which is why I lead the benchmark tables in this article with it.

In the keyword figures from the same run, you can see how far apart these numbers can fall.

On SciFact, nDCG@10 and MRR come out close to each other, at 0.6814 and 0.6494, because with about one right answer per question, ranking that answer well and finding it first amount to nearly the same job. On NFCorpus, however, the two numbers separate, since MRR stays at 0.5305 while nDCG@10 falls to 0.3278. In this run, the first correct document appears at position one or two on most of the questions, which is enough to lift MRR to 0.5305. The low nDCG is accurate as well, because with 38 marked documents per question, finding one of them is easy while finding them all is hard. You can read one of those facts from each of the two numbers.

Fix recall first

When a language model works from the passages that your search returns, recall at the depth that you pass forward is the number to fix first. Look at the NFCorpus row again, where a recall@100 of 0.2489 means that, on an average question, three-quarters of the marked documents appear nowhere in the top 100.

No later stage in the pipeline receives those documents. The obvious candidate is a reranker, which is a second scoring model that reorders the candidates from the first search; however, it can only score what that search returns. A larger language model works over the passages already in its prompt, and it therefore adds none of its own either. When the search never returns the passage with the answer, the retrieval step fails before anybody puts the question to the model.

Hailey Joren and colleagues study that boundary in a November 2024 paper, Sufficient Context. The authors classify each question by whether the retrieved context contains enough information to answer it, so that they can tell a retrieval miss apart from a model that receives the answer and still produces a wrong one. They also report that the larger models that they test, including Gemini 1.5 Pro, GPT-4o, and Claude 3.5, tend to produce an incorrect answer once the context falls short, where they should return no answer.

Both ranking formulas model a person who reads down a page

Both nDCG and MRR predate retrieval-augmented generation, which is the pattern in which a model generates its answers out of retrieved passages. Each of the two formulas models a person who reads a result list from the top downwards and pays less attention at every position further down, which is why both of them discount the lower ranks.

Giovanni Trappolini and colleagues identify the two places where that model of a reader is wrong, in Redefining Retrieval Evaluation in the Era of LLMs (October 2025). The first is that a language model processes all k of the retrieved documents at once, which means that the positional discount fits no reader who exists. The second is that the model generates a worse answer when a related but irrelevant passage is in its prompt, while under the classical measures that same passage counts as neutral. The authors propose a replacement that they score against end-to-end answer accuracy, for which they report correlations up to 36 per cent better than the traditional measures across five datasets and six models.

Keep nDCG in your reporting all the same. Use it when you compare one retriever against another, because the authors of the published benchmarks report it, which means that your own numbers stay comparable with everyone else's. Treat it as a weak predictor of the answers that your users read, and measure those answers separately.

How big an improvement looks depends on which number you report

In the same run, Narsil's benchmark suite also measures hybrid retrieval against keyword search on its own. In hybrid retrieval, the engine fuses a keyword ranking and a vector ranking into one list, which it does by position alone under the fusion method that I use, reciprocal rank fusion.

On NFCorpus, nDCG@10 rises by 8.6 per cent with the fused ranking, while recall@100 rises by 30 per cent. On the first of those figures the change looks marginal, while on the second it is the largest improvement in the table, even though both figures come out of the same pair of result lists.

Both ways of reporting it are honest, which is why you have to guard against reporting only the one that suits your change. To report the result fairly, state the metric, the depth, and the corpus in the same sentence, as I do in this sentence: with hybrid retrieval, recall@100 on NFCorpus rises from 0.2489 to 0.3239.

Building a judged set for your own corpus

Every number above depends on a set of marks for the documents that hold the answer to each question. The marks come with the public benchmarks; however, you have to mark your own corpus yourself, in six steps.

1. 1

   Take the questions that people ask you

   Pull them from your search logs, your support inbox, and the people who ask you out loud. Questions that you invent at your desk tend to match what your engine already handles, because you build the engine and write the questions with the same assumptions.

2. 2

   Pool the candidates from every configuration

   Put every question to every retrieval configuration that you are considering, take the top results from each, and merge them into one candidate list per question. That way, you only have to judge a fraction of the corpus, while the pool still holds everything that any of your configurations can return to a user.

3. 3

   Set the pool depth question by question

   Debasis Ganguly and Emine Yilmaz argue in a 2023 study of pooling depth that the number of relevant documents varies widely from question to question, and that if you vary the judging depth to match, you can still rank the systems correctly for substantially less effort. Go deeper on the broad questions and shallower on the narrow ones.

4. 4

   Write the grading rule before you grade anything

   Define what counts as a 2, a 1, and a 0, in one sentence each, and tie each sentence to the question. If you write the rule afterwards, you can only describe your earlier grades, which is why you rarely grade the same way with it in a second session.

5. 5

   Judge without knowing which engine returns what

   Sort each pooled list by document id and hide the retriever that produces each candidate, because a judge who knows the source tends to favour their own system.

6. 6

   Record what stays unjudged

   You can judge only what some retriever returns, which means that a document that no configuration returns stays unjudged, and trec_eval then counts it as irrelevant. When you measure recall this way, you get an upper bound on your true recall, which also means that if you add a retriever later, you have to judge its new candidates before you can compare it fairly.

In the demo below, you apply that method to a single question. You grade the five pooled documents yourself and, in the last step, see the scores that the demo computes for all three configurations from your marks.

You can work through a first version of these steps without writing any code, because the Narsil demo loads the whole engine inside your browser. Once you upload your own documents as a JSON or CSV file and add the questions from your logs, you can mark the documents with the answer to each of them. The demo then computes nDCG@10, precision@10, and MRR from your marks, which stay on your own machine. The method in the demo is shorter than the full one in two ways that matter. It builds its candidate pool from the single configuration that you load, so the demo has no other retrievers to merge and no source to hide. Its marks are also yes or no in place of the graded 2, 1, and 0 from the fourth step. Since the demo computes no recall and searches with keywords alone, what it measures is the precision and the ranking of your keyword search. Treat a demo run as your first judged set, and build the full one from it.

If your corpus consists of code or technical documentation, read FreshStack (April 2025), in which Nandan Thakur and colleagues automate this recipe. They collect the corpus, break community questions and their answers into individual factual nuggets, and then use a fusion of retrieval methods to find the documents that support each nugget. They also report a result to read before you buy a reranker, since on two of their five topics the reranking step scores no higher than the first-stage retrieval that it reorders.

Letting a model do the judging, and where it goes wrong

Of everything above, you spend the most time judging a pool by hand, so the next question is whether a language model can label the pool for you. The researchers who measure it give a two-part answer.

The authors of UMBRELA (June 2024) reproduce Bing's language-model relevance assessor in the open and score it against five years of human judgements from the TREC Deep Learning track. That track is an annual retrieval benchmark that NIST, the American standards institute, organises. On single documents, the agreement is weak, since Cohen's kappa on the four-point relevance scale falls between 0.3081 and 0.3730 across the 2019 to 2023 tracks. On the ordering of systems, the agreement is strong, since Kendall's tau between the system rankings from the human labels and from the model labels falls between 0.8728 and 0.9435 over the same years. The model's labels differ from the assessors' grades on individual documents, but the systems come out in nearly the same order under both sets of labels.

Use model labels when you rank your own configurations against one another, and keep them away from a verdict on any single document. The authors of two papers describe the limits of that advice. David Otero and colleagues find, in a 2024 analysis of automatic assessment, that the top-performing systems come out in an unfair order under model-generated labels, and that significance tests on those labels produce an exceedingly high rate of false positives. Charles Clarke and Laura Dietz describe a second failure in a December 2024 paper, where a language model reranks your results and another language model assigns the relevance labels to them. Both stages then work from the same notion of relevance, which is how your score can climb while the retrieval underneath it stays the same.

The workable compromise is to let a model label the pool, grade a random sample of a few dozen documents yourself, and then compare the two orderings. Where your configurations come out in the same order under your grades and under the model's, you can use the model's labels for the rest of the judging.

Two ways in which the measurement itself goes wrong

trec_eval breaks ties in an order that your engine never produces. The tool discards the rank column in your run file and re-sorts every result by its score. It then breaks equal scores by document id in reverse lexical order, which means that it scores forty equal-scoring documents in an order that the engine never returns. Narsil's benchmark suite corrects for this by rewriting the scores of each query so that they decrease strictly in the order in which the engine returns them. Without that correction, part of what you measure is how your document ids happen to sort.

A small difference can be noise. When you compare more than two configurations at once, you raise the chance that one of them looks better by accident. David Otero and colleagues test the standard corrections for this on simulated and real TREC data, in a 2025 study of significance testing, and find that the Wilcoxon test with a Benjamini-Hochberg correction holds the error rate at the level that you ask for while it keeps the most statistical power. Apply that pair before you keep a change worth two points in the third decimal place.

For the scoring itself, trec_eval from NIST is the reference implementation, while ir-measures is a single Python interface over it and several other scorers. Ragas computes context precision, context recall, and faithfulness for retrieval-augmented systems. It computes no nDCG, however, so it measures something different from what I measure in this article.

Where to look when you cannot move the number with any setting

Once you work through this, you have a repeatable measurement of what your search step returns, along with a way of telling whether a change improves it. Your recall can still come back low, and you can find that no keyword setting, hybrid weight, or ranking depth moves it by more than a point or two. That pattern has a cause upstream of everything that I measure here, since the engine can only rank what is in the index, which in turn depends on how you split your documents into pieces before you index them. In the next article in this study, I cut three public collections into pieces of several sizes and put the same judged questions to every index, and I will show you how you can tell whether cutting raises the score on your own documents.