How to Prove Your Search Is the Problem, Not the Model

When your agent gives a customer a wrong answer, find out whether the model ever had the right answer in front of it, before you swap that model over again.

Your product turns a user's question into an answer drawn out of your own documents, where a search step retrieves the passages and a language model then writes the answer from them. Those two steps fail in different ways and call for different repairs, which is why telling them apart is the first job in front of you. To do that you would need a set of real questions with the right answers marked against them, along with a few numbers that you can read correctly. In the previous article in this study I argued that the search step usually breaks first, and this article is how you would prove that on your own system.

Reading those numbers correctly is the hard part of the job, because a retrieval score on its own will support no conclusion you could act on.

A recorded run will show you what I mean. Narsil's benchmarks score search engines on public test collections from BEIR, and each of those collections holds a corpus, a question set, and human judgements marking which documents answer which question. In the run recorded on 4 August 2026, one engine in one fixed configuration, BM25 with k1 at 0.9 and b at 0.4, scored 0.6814 on a collection called SciFact and 0.3278 on one called NFCorpus. The metric, the engine, the settings, and the machine were identical across both of them.

The same engine scores 0.68 and 0.33 in one run

Neither of those numbers describes the engine on its own, because each of them describes a pairing of five things: this engine, this corpus, this question set, this configuration, and this metric. If you change any one of the five, the score changes along with it.

The gap between the two scores comes out of the two corpora. In SciFact, each question is a scientific claim and the right document is the paper that supports or refutes it, and the judges there marked 339 relevant documents across 300 questions, which comes to roughly one right answer per question. In NFCorpus, each question is a plain-language medical query over nutrition research articles, and its judges marked 12,334 relevant documents across 323 questions, which comes to roughly 38 of them per question. Finding that single paper is a different job from covering 38 relevant articles, and no one score can tell you which of the two jobs the engine did well.

Take that as the first rule for reading anybody's benchmark, including mine. A published score shows you how an engine compares with the engines measured beside it, in the same run and on the same hardware. What your own corpus would score stays an open question until you measure it yourself.

Four numbers, and what each one leaves out

The retrieval papers cited in this article report the same four numbers, and each of them measures a different property of a result list.

Recall@k counts how many of the documents that you marked as right for a question came back inside the top k results. It scores each of those documents equally wherever it found them, so a system that ranks every correct passage around position ninety would still score well at a k of 100.

Precision@k counts how many of the k documents that came back were the right ones. It scores only what the engine showed you, so a system returning three correct passages out of a possible forty can score perfectly at a k of 3.

MRR, which stands for mean reciprocal rank, records the position at which the first correct document appeared, and then averages one over that position across all of your questions. The calculation stops at that first hit, so everything below it goes unscored.

nDCG@k, which stands for normalised discounted cumulative gain, scores the whole ranked list. A correct document near the top adds more to the score than the same document further down, and the formula then divides the total by what a perfect ordering would have produced. A high score therefore asks for both halves of the job, which are finding the right documents and ordering them well, and that is why the benchmark tables in this article lead with it.

The keyword figures from that same run on 4 August show how far apart these numbers can fall.

On SciFact, nDCG@10 and MRR come out close to each other, at 0.6814 and 0.6494, because with about one right answer per question, ranking that answer well and finding it first amount to nearly the same job. On NFCorpus, however, the two numbers separate, since MRR stays at 0.5305 while nDCG@10 falls to 0.3278. An MRR of 0.5305 is what a system scores when the first correct article appears at position one or two on most of the questions, and that is a fair description of what happened. The low nDCG is a fair description as well, because with 38 relevant articles per question, finding one of them is easy while covering them all is hard. Each of the two measures captures one of those facts.

Start with the number that limits every answer

When your retrieval supplies the passages that a language model works from, recall at the depth you pass forward is the number you should fix first. Look at the NFCorpus row again, where a recall@100 of 0.2489 means that three-quarters of the documents the judges marked relevant appeared nowhere in the top 100.

No later stage can recover those documents for you. The obvious candidate is a reranker, which is a second scoring model that reorders the candidates the first search returned, and it can only work on the candidates it was handed. A larger language model works over the passages already in its prompt, so it adds none of its own either. When the passage that answers the question never came back, the retrieval step had already failed before anybody put the question to the model.

Hailey Joren and colleagues gave that boundary a name in a November 2024 paper, Sufficient Context. The authors classify each question by whether the retrieved context contains enough information to answer it, which separates a retrieval miss from a model that was handed the answer and still produced a wrong one. They also report that the larger models they tested, including Gemini 1.5 Pro, GPT-4o, and Claude 3.5, often produced an incorrect answer once the context fell short, when they should have abstained.

The ranking metrics model a person reading down a page

nDCG and MRR both rest on an assumption that predates retrieval-augmented generation, which is the pattern from this article's opening, where a model writes its answers out of retrieved passages. Each of the two formulas models a person who reads a result list from the top downwards and pays less attention at every position further down, which is why both of them discount the lower ranks.

Giovanni Trappolini and colleagues identify the two places where that assumption breaks down, in Redefining Retrieval Evaluation in the Era of LLMs (October 2025). The first is that a language model processes all k of the retrieved documents at once, so the positional discount matches no reader who exists. The second is that a related but irrelevant passage degrades the answer the model writes, while the classical measures count that same passage as neutral. The authors propose a replacement scored against end-to-end answer accuracy, and they report correlations up to 36 per cent better than the traditional measures across five datasets and six models.

Keep nDCG in your reporting all the same. Use it when you compare one retriever against another, since the published results include it and your own numbers then stay comparable with everyone else's. Treat it as a weak predictor of the answers your users will read, and measure those answers separately.

Which number you report changes how big an improvement looks

That same run also measured hybrid retrieval against keyword search on its own. Hybrid retrieval fuses a keyword ranking and a vector ranking into one list, and the fusion method used here, reciprocal rank fusion, merges the two lists by position alone.

On NFCorpus, the fused ranking raised nDCG@10 by 8.6 per cent and recall@100 by 30 per cent. The first of those figures makes the change look marginal, while the second makes it the largest improvement in the table, and both of them come out of the same pair of result lists.

Neither way of reporting it would be dishonest, which is what makes this worth guarding against. To report the result honestly, name the metric, the depth, and the corpus in the same sentence, in the way this one does: hybrid retrieval raised recall@100 on NFCorpus from 0.2489 to 0.3239.

Building a judged set for your own corpus

Every number above rests on somebody having marked which documents answer which question. The public benchmarks arrive with those marks already made, whereas your own corpus arrives without them, so you would have to make them yourself in six steps.

1. 1

   Take the questions people asked you

   Pull them from your search logs, your support inbox, and the people who ask you out loud. Questions that you invent at your desk tend to match what your engine already handles, because the same assumptions shaped both.

2. 2

   Pool the candidates from every configuration

   Run every retrieval configuration that you are considering against every question, take the top results from each, and merge them into one candidate list per question. Judging that pool costs a fraction of judging the whole corpus, and the pool still covers everything that any of your candidates would have shown a user.

3. 3

   Set the pool depth per question, not once for everything

   Debasis Ganguly and Emine Yilmaz argue in a 2023 study of pooling depth that the number of relevant documents varies widely from question to question, and that varying the judging depth to match reproduces the same ranking of systems for substantially less effort. Go deeper on the broad questions and shallower on the narrow ones.

4. 4

   Write the grading rule before you grade anything

   Define what counts as a 2, a 1, and a 0, in one sentence each, tied to what the question calls for. A rule written afterwards records what you did, so it governs nothing, and it rarely holds up in the second judging session.

5. 5

   Judge without knowing which engine returned what

   Sort each pooled list by document id and hide the retriever that produced each candidate, because a judge who knows the source tends to favour their own system.

6. 6

   Record what stays unjudged

   You judge only what some retriever returned, so a document that no configuration found stays unjudged, and trec_eval counts an unjudged document as irrelevant. Recall measured this way is an upper bound on your true recall, which also means that a new retriever added later needs its new candidates judged before you can compare it fairly.

The demo below runs that method on a single question, where you grade the five pooled documents yourself and the last step scores all three configurations from the marks you gave.

You can run a first version of these steps without writing any code, because the Narsil demo runs the whole engine inside your browser. You upload your own documents as JSON or CSV, add the questions from your logs, mark the documents that answer each of them, and read nDCG@10, precision@10, and MRR computed from your marks, which stay on your own machine. The demo does trim the method in two ways that matter. It builds its candidate pool from the single configuration you loaded, so nothing is merged across retrievers and no source needs hiding, and its marks run yes or no in place of the graded 2, 1, and 0 from the fourth step. Treat a demo run as your first judged set, and build the full one from it.

If your corpus holds code or technical documentation, Nandan Thakur and colleagues published an automated version of this recipe as FreshStack (April 2025), where they collect the corpus, break community questions and their answers into individual factual nuggets, and then find the documents that support each nugget through a fusion of retrieval methods. They also report a result you should read before you buy a reranker, since on two of their five topics the reranking step scored no higher than the first-stage retrieval it was reordering.

Letting a model do the judging, and where that breaks

Judging a pool by hand is the slow part of everything above, so the next question is whether a language model can do the grading instead. The measured answer comes in two halves.

UMBRELA (June 2024) reproduces Bing's language-model relevance assessor in the open and scores it against five years of human judgements from the TREC Deep Learning track, an annual retrieval benchmark run by the American standards institute NIST. On single documents the agreement is weak, since Cohen's kappa on the four-point relevance scale runs between 0.3081 and 0.3730 across the 2019 to 2023 tracks. On the ordering of systems the agreement is strong, since Kendall's tau between the system rankings built from the human labels and from the model labels runs between 0.8728 and 0.9435 over the same years. The model's grades differ from the assessors' grades on individual documents, but the two sets of labels still put the systems in nearly the same order.

That result splits the use cases in two. Use model labels when you rank your own configurations against one another, and keep them away from a verdict on any single document. Two papers mark out the edges of that advice. David Otero and colleagues find, in a 2024 analysis of automatic assessment, that model-generated labels rank the top-performing systems unfairly, and that they produce an exceedingly high rate of false positives once they feed significance tests. Charles Clarke and Laura Dietz describe a second failure in a December 2024 paper, where a language model reranks your results and another language model grades them, so both stages work from the same notion of relevance and your score climbs while the retrieval underneath it has stayed where it was.

The workable compromise is to let a model grade the pool, grade a random sample of a few dozen documents yourself, and then compare the two orderings. Where your grades and the model's grades rank your configurations the same way, the model's labels can do the rest of the judging for you.

Two ways the measurement itself goes wrong

trec_eval breaks ties in an order that your engine never produced. The tool discards the rank column in your run file and re-sorts every result by its score. It then breaks equal scores by document id in reverse lexical order, so it would grade forty equal-scoring documents in an order that never came out of the engine. Narsil's harness corrects for this by rewriting the scores of each query so that they decrease strictly in the order the engine returned them. Without that correction, part of what you measure is how your document ids happen to sort.

A small difference can be noise. Comparing more than two configurations at once inflates the chance that one of them looks better by accident. David Otero and colleagues tested the standard corrections for this on simulated and real TREC data, in a 2025 study of significance testing, and found that the Wilcoxon test with a Benjamini-Hochberg correction holds the error rate at the level you asked for while it keeps the most statistical power. Run that pair before you keep a change worth two points in the third decimal place.

For the scoring itself, trec_eval from NIST is the reference implementation, and ir-measures puts a single Python interface over it and several other scorers. Ragas computes context precision, context recall, and faithfulness for retrieval-augmented systems, and it includes no nDCG, so it measures something different from what this article has been measuring.

Where to look when no setting moves the number

Once you have run this through, you will have a repeatable measurement of what your search step returns, along with a way of telling whether any change improved it. Your recall may still come back low, with no keyword setting, hybrid weight, or ranking depth moving it by more than a point or two. That pattern has a cause upstream of everything measured here, since ranking can only order what the index already contains, and what the index contains follows from how you split your documents into pieces before you indexed them. In the next article in this study I take that apart, and I report what happened when I cut two public corpora five ways each and put the same judged questions to all ten indexes.