Your agent's answers sound certain and are wrong
Picture your AI agent providing a wrong answer to a customer's question, in wording that sounds certain. If your agent still provides wrong answers after you swap the model over, rewrite the prompt, and drop the temperature, then the model is unlikely to be the cause.
The failure happens one step earlier, before the model generates a word. Because a search over your own documents returns the handful of passages from which your agent works, your agent can only generate its answer from those passages.
Your agent's answer can only come from what the search step returns
That step is called retrieval. When retrieval returns the passage with the answer in it, your agent can provide a correct answer; when it returns four passages that merely resemble the question, your agent generates its answer from those four.
Your documents
Handbooks, policies, tickets, notes
Cut into pieces
This cut sets what the search can find at all.
Indexed
The index stores each piece as one unit.
Ranked for a question
Ranking reorders the pieces and nothing more.
Top results to the model
Everything else stays out of the prompt.
If a passage is missing from the last stage, the model never receives it. A reranker reorders the candidates that the first search returns, while a larger model processes only the passages already in its prompt.
Suppose that a new parent asks your agent about time off for a new baby, while the words in your staff handbook are 'parental leave' throughout. The search returns whichever pages share the most words with that question, although the rule appears on none of them. Your agent therefore generates its answer from the closest page that the search returns. Your monitoring meanwhile records a successful query and a completed response, while your customer takes a wrong answer for a correct one. You catch a failure of that kind only when you read the answer yourself.
Two reasons why your search never returns the passage
The first reason is the cut that you make when you index, because you have to split a long document into indexable pieces before your search can return one of them. That splitting is called chunking. When one of those cuts falls through the middle of an answer, the topic is in the first piece while the rule is in the second, so the answer to what your user asks is in neither piece on its own.
The second reason is vocabulary. Keyword ranking matches the words in your documents, while your users ask in the words that they reach for themselves, which can be an entirely different set. Because a reranker can only reorder the passages that the first search already returns, you have to repair the search step itself.
You can close that vocabulary gap with embeddings, although you then need compute to embed every document that you index and every query that you answer. Which of the two reasons applies to you is a question about your own documents and your own users, which you cannot settle from a published benchmark.
Settle it on your own documents
To get to that answer, you need two inputs. The first is the questions that your users already bring you, with a note beside each one of the document where its answer is, and the second is a small set of measures to score what the search returns.
Collecting those measures is arithmetic over what the search returns, whereas you have to read them with care, because in a single pass over one set of documents the same search step can score well on one measure and badly on another. Both of those scores are honest, which means that if you pick the wrong one, you can end up improving a number that your users never notice.
I split this study into five articles because you have to rule out each of the two reasons separately, and because a measurement that you can defend to a doubter is the only kind that is worth acting on. If you skip ahead to the fix, you can end up paying for embeddings that you do not need for your corpus.
What you can do by the end
By the end of these five articles, you can measure what your search step returns, in figures that you can defend to somebody who doubts them. You can also tell which of the two reasons applies to your corpus, prove that the fault is gone after your change without the problem moving elsewhere, and decide whether embeddings are worth their cost on the documents that you have.
Because you do all of that with the questions that your users already ask you, together with a note of the document where each answer is, you need no labelled research dataset and no rewrite of your product.