Teams that build an AI feature tend to hit the same problem again and again. The demo goes well, because the agent returns correct answers to the team's questions. Once the real users arrive, however, the agent's answers turn unreliable. An answer can mention a policy that the company does not have, give last year's pricing, or describe a feature that does not exist. Once the users stop trusting those answers, the team usually shelves the feature.
The team's usual explanation is that the model is hallucinating, which is why the team moves to a larger model and rewrites the prompt once again. When neither change improves the answers, the team drops the to zero. The answers do improve on the questions that the team tests, although they get worse again as soon as a user asks something slightly different.
The cause is usually one step earlier. Your system passes the wrong context to the model, which then generates its answer from exactly the context that you give it. The failure is therefore a search problem, which means that you can measure it.
How the system produces a wrong answer
A retrieval-augmented generation system, or RAG for short, works in two stages. In the first stage it retrieves, which means that it searches your documents with the question that a user asks and returns the handful of passages that it scores as the most relevant. In the second stage it generates, which means that it puts those passages into the prompt so that the model can generate its answer from them.
People watch the second stage, because it is the part that they can read; however, the answer can never be better than the passages that the first stage returns. If the passage with the answer never appears among the top results, then the model never receives it, whichever model you use. The model can then either return no answer or generate something plausible in its place. That second behaviour is the one that the team calls hallucination, although the cause is usually a miss at the retrieval step.
The generation step can fail as well. Even with the right passage in the prompt, a model sometimes generates its answer from its training data in place of the passage. That failure is the harder of the two to diagnose. A retrieval miss, however, is the cheaper one to catch, which is why the retrieval step is the place to start.
The short version
A language model can only work from the context that you give it. When retrieval is weak, therefore, the model can produce fluent and certain-sounding answers that happen to be wrong. When you repair the retrieval step, those answers change, whereas a different model still receives the same passages.
Why the team blames the model
Your system logs every retrieval as a success, whatever the search returns. When the right passage is missing from the results, the system still logs a completed query, while the model generates a fluent paragraph that sounds right and happens to be wrong. Since the failure shows up in the text that the model generates, the team blames the model.
In the example below, you can look at one such wrong answer from the chat window and then from the search results behind it.
How much time off do I get a year?
You get 15 days of time off a year.
Keyword search has its own version of the problem. Suppose that your search engine matches on exact terms and that a user asks about 'time off', while the words in your handbook are 'paid leave' throughout. None of the terms will match, which means that the right passage will score close to nothing. Pure vector search has the opposite weakness, since it matches on meaning and can return the wrong passage for an exact token such as a product code, an error number, or a name. In both cases, the answer that the user needs is in your data the whole time, yet the search step never returns it.
Measure retrieval like the search problem that it is
You can measure the retrieval step precisely. The evaluation toolkits report a standard set of quality measures, which you can borrow for your own feature.
The method is plain enough that it is worth doing by hand the first time. Collect between thirty and a hundred of the questions that your users ask you, and mark, for each of them, the document or passage with the correct answer in it. That labelled set is the ground truth that you score everything against. Send each of those questions through your search step on its own and record the passages that it returns, with no model in the loop, so that you are testing the retriever by itself. Then score the passages that it returns against the ones that you mark as correct, so that in place of a vague sense that the feature is off, you have a number that you can act on.
You need three measures here, each of which has a standard name. The first, precision, is the share of the returned passages that are about the question. When your precision is low, you are handing the model a pile of noise and hoping that the model uses the right line in it.
The second, recall, is the share of the passages in your data with an answer to the question that the search returns at all. When your recall is low, the passage that matters is never in the prompt, which means that the model cannot give that answer whatever you change in the prompt. Both of them usually appear as precision@k and recall@k, where k is the number of top results that you choose to score.
The third measure is a score for the order, since it matters whether the best passage appears near the top, where it has the most effect on the answer, or down at position nine. MRR and nDCG are two ranking scores for exactly that. Your recall can fall when you raise your precision, while your precision can fall when you raise your recall, which is why teams also track the F1 score. That score is the harmonic mean of precision and recall, and it therefore falls whenever either of the two falls.
The search returns these 10 passages for one question, from a corpus that holds 5 relevant passages.
Precision@10
0.40
4 relevant / 10 retrieved
This is the share of the returned passages that are genuinely about the question.
Recall@10
0.80
4 found / 5 in corpus
This is the share of all the relevant passages in your data that the search returns.
F1 score
0.53
2 x (P x R) / (P + R)
This is the harmonic mean of precision and recall, and it drops when either one drops.
What MRR and nDCG mean
MRR stands for mean reciprocal rank. For each question, the score is one over the position of the first correct passage, so the top position scores 1, the second scores 0.5, and the fifth scores 0.2. The final number is the average of those scores across your test set. nDCG stands for normalised discounted cumulative gain. It is a score for the whole ranked list, in which a relevant passage near the top counts for more than one further down. The formula then divides that score by the score of a perfect ordering, which means that 1.0 is a perfect ranking. The @10 after either measure means that only the top ten results count towards the score.
When you score your whole test set on those three measures, which the evaluation toolkits can compute for you, you turn a hunch into a measurement that you can track over time. The next time that a feature returns wrong answers and somebody blames the model, you can therefore check in that measurement whether the right passage is in the prompt at all.
You can carry out the method in this section without writing any code in the Narsil demo, which loads the engine into your browser and computes nDCG@10, precision@10, and MRR from your marks. In the next article in this study, I will show you how you can build a judged set of your own, both in the demo and in full.
How to raise the numbers
Once you can measure the retrieval step, you can prove that a fix improves your numbers before you decide to keep it.
Use keyword and vector search together and blend the two sets of scores. Keyword search matches the exact terms, codes, and names, while vector search matches paraphrase and intent, which means that a hybrid search can return both kinds of answer. On both in the recorded benchmark behind this study, a hybrid retriever scores above either approach on its own.
Your search can only return the chunks that you split your documents into. In a chunk that is too large, the sentence that you need is among unrelated text, while in a chunk that is too small, the context around that sentence is missing. In the article on chunking in this study, I measure how far the scores on three public collections of short and long documents change with that one choice.
Retrieve a generous set of candidates first, and then pass them through a second and more precise model that reorders them before you put the top few into the prompt. A re-ranker can raise your precision at the top of the list, where the passages have the most effect on the answer, for little extra cost at the generation step.
Reshape the raw question before you search with it, whether that means expanding an abbreviation, splitting a compound question into two, or swapping in the synonym that appears in your corpus. A search with the user's exact wording seldom returns the best passages.
The discipline matters more here than any one of those techniques. Change a single setting, score your labelled set again, and keep the change only where the numbers improve. That loop is how you turn an unreliable feature into a dependable one, so repeat it for every change that you're considering.
A worked example
I spend a good deal of my own time on this problem, which is why I build Narsil, an open-source distributed search engine. With Narsil you get full-text, vector, and hybrid search inside one engine, which turns the choice between keyword and semantic retrieval into a single setting that you tune.
The claim, measured
In Narsil's public benchmarks, I follow the same loop that I describe in this article, with a fixed set of questions, a marked correct answer for each of them, and a score for every retrieval method. On SciFact, a public test set of scientific claims, keyword search on its own scores 0.68 nDCG@10 while hybrid search scores 0.70, where 1.0 is a perfect ranking. The scores for Elasticsearch and OpenSearch rise by a similar margin, from 0.68 to 0.71. On pure vector search at the same recall, Narsil processes the most queries a second on SciFact and on NFCorpus, a public set of plain-language questions over medical documents. On the DBpedia set of 995,000 documents, however, OpenSearch processes more, at 869 queries a second against Narsil's 670.
Read the full benchmarksWhichever tool you use, you can put the quality of your retrieval on a dashboard as a number and improve it, as you already do with or an error rate. Once you treat it that way, a hallucination problem usually turns out to have a specific cause that you can find.
Where to start
If an AI feature of yours is unreliable in production, measure the search step before you touch the model. Build a small labelled set of the real questions that your users ask, and record what your search step returns for each of them. When the right passage is missing from those results, the model can't generate a correct answer, so repair the search step. Once you measure your retrieval, you can tell which step to repair.
Updates
In the version of 21 July 2026, I write that Narsil is weaker on pure vector search, at less than half the speed of the fastest engines. In the benchmark run of 28 September 2026, Narsil processes the most vector queries a second on SciFact and NFCorpus, while OpenSearch processes more on DBpedia, at 869 against 670. I also now call the correct answers marked in place of human-labelled, and I give the hybrid gain for Elasticsearch and OpenSearch alone.
Go to this section: A worked example