Your AI Agent Is Not Hallucinating. Your Search Is Broken, and You Can Prove It.

Your agent sounds certain and gets it wrong

Picture your AI agent answering a customer's question wrongly, in wording that sounds certain of itself. By now you may have swapped the model over, rewritten the prompt, and dropped the temperature. Since the wrong answers keep arriving through all of that, the model is unlikely to be what causes them.

The failure happens one step earlier, before your agent writes a word. A search over your own documents picks out the handful of passages that it will work from, so your agent can only answer out of the passages that search returned.

The step that sets the limit on every answer

That step is called retrieval, and it sets the limit on everything your agent can say. When it returns the passage that holds the answer, your agent has what it needs; when it returns four passages that merely resemble the question, your agent writes its answer out of those four.

Suppose a new parent asks your agent about time off for a new baby, while your staff handbook says 'parental leave' throughout. The search returns whichever pages share the most words with that question, and the rule appears on none of them, so your agent answers out of the closest page it was handed. Your monitoring meanwhile records a successful query and a completed response, which means that your customer reads a wrong answer as a correct one. You would catch a failure of that kind only by reading the answer yourself.

Two reasons your search misses the passage

The first is the cut you make when you index, since you have to split a long document into indexable pieces before your search can return one of them, and that splitting is what people call chunking. When one of those cuts falls through the middle of an answer, the first piece keeps the topic while the second keeps the rule, so neither of the two pieces answers what your user asked on its own.

The second reason is vocabulary. Keyword ranking looks for the words that your documents use, while your users ask in the words they reach for themselves, and those two sets of words can be entirely different. Reordering the results afterwards works only on the passages that the first search already returned, so you would have to repair the search step itself.

Embeddings close that vocabulary gap, although they add compute over every document you index and every query you answer. Which of the two reasons applies to you is a question about your own documents and your own users, and no published benchmark can settle it on your behalf.

Only your own documents can settle which one

Getting to that answer takes two things: the questions your users already bring you, with the right answers marked against them, and a small set of measures to score what comes back.

The hard part is reading those measures correctly, because the same search step, in one run over one set of documents, can score well on one measure and badly on another. Both of those scores would be honest, and picking the wrong one would leave you improving a number your users never notice.

This study runs to five articles because you would have to rule out each of the two reasons separately, and because a measurement you can defend to a doubter is the only kind worth acting on. Skipping ahead to the fix is how you would end up paying for embeddings that your corpus never needed.

What you will be able to do by the end

By the end of these five articles you will be able to measure what your search step returns, in figures that you can defend to somebody who doubts them. You will also be able to name which of the two reasons applies to your corpus, prove that a change repaired it and that the problem did not move elsewhere, and decide whether embeddings earn their price on the documents you have.

All of that runs on the questions your users already ask you, together with a record of which documents answer them, so you would need no labelled research dataset and no rewrite of your product.