Most AI Hallucinations Are Retrieval Failures in Disguise

Teams that build an AI feature often run into the same problem. The demo goes well, because you ask the agent a question and it answers correctly. Once the real users arrive, however, the answers turn unreliable. The agent will cite a policy that your company never wrote, quote last year's pricing, or explain a feature that nobody built. Once the users stop trusting what it tells them, the team usually shelves the feature.

The usual explanation is that the model is hallucinating, so the team moves to a larger model, rewrites the prompt once again, and, when neither of those helps, drops the temperature to zero. The answers do improve on the questions the team tested, although they break again as soon as a user asks something slightly different.

The cause usually lies one step earlier. The model receives the wrong context and then does exactly what you asked of it, which is to answer from what it was given. That makes it a search problem, and a search problem leaves evidence you can measure.

How a wrong answer is produced

A retrieval-augmented generation system, which most people shorten to RAG, works in two stages. In the first stage it retrieves, which means that it takes the question a user asked, searches your documents, and returns the handful of passages it scores as the most relevant. In the second stage it generates, which means that it puts those passages into the prompt and asks the model to answer out of them.

The second stage draws all of the attention, because it is the part you can read; the first stage, however, sets the limit on how good the answer can ever be. If the passage that answers the question never appears among the top results, then no model can recover it, so the model will either decline to answer or fill the gap with something that sounds plausible. That second behaviour is what people call hallucination, and it is often a miss at the retrieval step instead.

The retrieval step is not the only place where an answer can break down. Even with the right passage in the prompt, a model will sometimes set aside what it was given and answer out of its training instead. That failure is the harder of the two to pin down, and a retrieval miss is the cheaper one to catch, so the retrieval step is where you should start.

Why the model takes the blame

The retrieval step fails quietly, because it reports success whatever it returns. When the right passage is missing from the results, your system logs a completed query and returns a fluent paragraph that reads well and happens to be wrong. Since the failure surfaces in the model's own words, the model is what people blame.

Keyword search adds a version of the problem of its own. If your retrieval matches on exact terms, and a user asks about 'time off' while your handbook says 'paid leave' throughout, then none of the terms will match and the right passage scores close to nothing. Pure vector search has the opposite weakness, since it captures meaning well while it misses the exact tokens such as product codes, error numbers, and names. In both cases the answer the user needed was in your data the whole time, and the search step passed over it.

Measure retrieval like the search problem it is

You can measure the retrieval step precisely. The evaluation toolkits report a standard set of quality measures, and you can borrow the same ones for your own feature.

The method is plain enough that it is worth doing by hand the first time. Collect between thirty and a hundred of the questions your users ask you, and mark, for each of them, the document or passage that holds the correct answer, since that labelled set becomes the ground truth you will measure everything against. Run each of those questions through your search step on its own and record the passages it returns, with no model in the loop, so that you are testing the retriever by itself. Then score the passages it returned against the ones you marked as correct, and a vague sense that the feature is off turns into a number you can act on.

Three checks do most of the work here, and each of them has a standard name. The first counts how many of the returned passages were genuinely about the question, and it is called precision. When your precision is low, you are handing the model a pile of noise and hoping that it picks the right line out of it.

The second counts how many of the places in your data that could have answered the question came back at all, and it is called recall. When your recall is low, the passage that mattered never reached the prompt, so no amount of prompt engineering would rescue that answer. You will see both of them written as precision@k and recall@k, where k is the number of top results you have chosen to score.

The third check asks about order, since it matters whether the best passage appeared near the top, where the model weighs it most heavily, or down at position nine. The ranking scores called MRR and nDCG measure exactly that. Improving your precision will often lower your recall, and the reverse holds as well, which is why teams also track the F1 score. That score is the harmonic mean of precision and recall, so it falls whenever either of the two falls.

Running those three checks across your whole test set turns a hunch into a measurement that you can track over time, and the evaluation toolkits will compute the scores for you. So the next time a feature misbehaves and somebody blames the model, that measurement will tell you whether the right passage ever reached the prompt.

What moves the numbers

Once you can measure the retrieval step, the fixes in front of you become concrete, because you can prove that each of them improved your numbers before you decide to keep it.

The discipline matters more here than any one of those techniques. Change a single setting, run your labelled set through again, and keep the change only where the numbers have improved. That loop is how an unreliable feature turns into a dependable one, and you can run it again for every change you are considering.

A worked example

I spend a good deal of my own time on this problem, which is why I built Narsil, an open-source distributed search engine. It runs full-text, vector, and hybrid search inside one engine, so the choice between keyword and semantic retrieval becomes a single setting you tune.

What the benchmark demonstrates matters more than the tool that produced it, because the quality of your retrieval is a number you can put on a dashboard and improve, in the way that you already treat latency or an error rate. Once you treat it that way, a hallucination problem usually turns out to have a specific cause that you can find.

Where to start

If an AI feature of yours is unreliable in production, measure the search step before you touch the model. Build a small labelled set of the real questions your users ask, and record what your search step returns for each of them. When the right passage is missing from those results, the model never had a chance of answering correctly, so the search step is what you would repair. Once you have measured your retrieval, you can name the step that needs the work.