Every team building an AI feature meets the same wall. The demo is flawless. You ask the assistant a question, it answers perfectly, and everyone in the room nods. Then real users arrive, and the answers turn unreliable. The assistant cites a policy that does not exist, quotes last year's pricing, or explains a feature you never shipped. Trust drains within a week, and the feature is quietly shelved.
Almost everyone reaches for the same explanation: the model is hallucinating. So they move to a larger model and rewrite the prompt yet again, and when that fails they drop the temperature to zero. The answers improve for a day, then break the moment a user asks something slightly different.
The model is usually not the culprit. In most retrieval-augmented systems, the model is handed the wrong context and does exactly what you asked of it; it answers from what it was given. That is a search problem, and search problems leave evidence you can measure.
How a wrong answer is produced
A retrieval-augmented generation (RAG) system works in two stages. First it retrieves: it takes the user's question, searches your documents, and returns the handful of passages it judges relevant. Then it generates: it places those passages in the prompt and asks the model to answer from them.
The second stage draws all the attention, because it is the part you can read. The first stage decides whether the answer can be correct at all. If the passage that holds the answer never reaches the top results, no model on the market can recover it. The model will either decline or fill the gap with something plausible, and that second behaviour is what people call hallucination. More often than not, it is a retrieval miss in disguise.
Retrieval is not the only place an answer can break. Even with the right passage in front of it, a model will sometimes ignore what it was handed and answer from memory instead, and stronger models have not made that go away. But it is the smaller share of what goes wrong, and the harder part to pin down. The retrieval miss is the bigger cause and the cheaper one to catch, so that is where to start.
The one-line version
A language model can only reason over the context you give it. Weak retrieval produces confident, fluent answers that happen to be wrong. Strong retrieval is the difference between a demo and a product.
Why the model takes the blame
Retrieval fails quietly. When the right passage is missing, you get no error and no stack trace; you get a fluent paragraph that reads well and happens to be wrong. The failure looks like the model talking, so the model takes the blame.
Keyword search makes this worse than people expect. If your retrieval matches on exact terms and a user asks about 'time off' while the handbook says 'paid leave', the words never meet and the right passage scores close to nothing. Pure vector search has the opposite weakness: it captures meaning well, yet it slips on exact tokens such as product codes, error numbers, and names. Either way, the answer the user needed sat in your data the whole time, and the search step walked past it.
Measure retrieval like the search problem it is
You cannot fix what you refuse to measure, and retrieval can be measured precisely. Information retrieval has had rigorous quality metrics for decades, long before language models existed, so borrow them.
The method is plain and worth doing by hand. Collect thirty to a hundred questions your users ask, and for each one mark the document or passage that holds the correct answer; that labelled set becomes the ground truth you measure against. Run each question through your search step on its own and record the passages it returns, before the model ever sees them, so you are testing the retriever by itself and not the whole pipeline. Then score those passages against the ones you marked as correct. The vague sense that 'it feels off' becomes something you can measure.
What you are checking has standard names, and three checks carry most of the weight. Of the passages the search brought back, how many were genuinely about the question? That is precision, and when it is low you are handing the model a pile of noise and hoping it picks out the right line. Of all the places in your data that could have answered the question, how many did the search return? That is recall, and when it is low the passage that mattered never reached the prompt, so no amount of prompting will rescue the answer. You will see these two written as precision@k and recall@k, where k is how many of the top results you choose to score. The third check is about order: did the best passage sit near the top, where the model pays attention, or down at position nine where it may as well be missing? That is what ranking scores such as MRR and nDCG capture.
Counting the answers to those three checks across your test set turns a hunch into a measurement you can track, and any search library will compute these scores for you. When a feature misbehaves and the model takes the blame, you will often find the retriever was returning the right passage only about half the time.
What moves the numbers
Once you can measure retrieval, the fixes are concrete, and you can prove that each one earned its place.
Run keyword and vector search together and blend the scores. Keyword search nails exact terms, codes, and names; vector search catches paraphrase and intent. Together they cover each other's blind spots, and on most real corpora a hybrid retriever beats either approach on its own.
How you split your documents decides what can be retrieved at all. Chunks that run too large bury the relevant sentence in a wall of unrelated text; chunks that run too small lose the context that gave the sentence meaning. This single choice often moves recall further than swapping the model ever would.
Retrieve a generous set of candidates, then pass them through a second, more precise model that reorders them before the top few reach the prompt. It costs little next to generation, and it lifts precision where it counts most, at the very top of the list.
Reshape the user's raw question before you search: expand an abbreviation, split a compound question in two, or swap in the synonym your corpus uses. The question a user types is seldom the question that retrieves best.
The discipline matters more than any single technique. Change one thing, run your labelled set again, and keep the change only when the numbers improve. That loop is how an unreliable feature becomes a dependable one, and it is repeatable rather than lucky.
A worked example
I spend a good deal of my time on this problem, which is why I built Narsil, an open-source distributed search engine. It runs full-text, vector, and hybrid search in a single engine, so the choice between keyword and semantic retrieval becomes one tunable setting instead of two systems to connect.
The numbers held up
Measured on the public BEIR information-retrieval datasets, Narsil's BM25 reproduces the published Anserini baseline within 0.005 nDCG@10, which is the calibration that makes the rest of the comparison trustworthy. Served over HTTP, it ranks level with the Lucene engines on SciFact; embedded in one process, it takes the top nDCG@10 against Orama and MiniSearch. The full quality, throughput, and memory tables sit in the repository along with the method and reproduction steps, so you can rerun the comparison yourself.
What the benchmark demonstrates matters more than the tool itself: retrieval quality is a number you can put on a board and improve, the same way you treat latency or an error rate. Treat it that way, and most 'hallucination' problems turn out to have an address.
Where to start
If an AI feature of yours is unreliable in production, resist the urge to touch the model first. Build a small labelled set of real questions, measure what your retrieval returns, and read the answers the model never had a chance to get right. The fix usually sits upstream of the model, and once you measure it, you can finally see it.
