A developer queries an internal AI assistant. The assistant looks up the documentation, finds a relevant troubleshooting document, and provides an answer.

You get an incorrect answer.

This can seem like a problem with retrieval. Maybe the embedding model didn't find the correct document, or the index needs rebuilding.

But when you look at the retrieved documents, the correct document is already there.

This means you're potentially dealing with a reliability issue. The RAG system has found the relevant information, but it isn't giving you a reliable answer. Improving the search alone won't necessarily fix it.

The issue can occur after retrieval, when the application decides which information to trust, what it means, and whether there's enough evidence to answer the question.

You have multiple stages in a RAG system.

A RAG system has several stages.

The application takes in a question, finds relevant information from documents, and passes it to a language model to generate an answer based on the context provided.

Every part of this process can affect the outcome.

Say you have an internal troubleshooting assistant with documentation for three environments: Development, Staging, and Production.

A user asks what to do when their tests fail in Staging.

The search returns a troubleshooting document with the correct information for fixing issues in Staging. It also returns a runbook with similar instructions for Production.

Both documents are relevant to the general topic, but only one applies to the user's situation.

If the assistant combines them, the answer may contain technically correct information but give the wrong advice.

The issue here isn't necessarily with the search engine. It's with the application not maintaining enough context to know which information to use.

Irrelevant documents can be returned.

Vector searches are designed to return semantically relevant information.

However, they won't necessarily return the most up-to-date, authoritative, or environment-specific information.

Say you have three documents in your knowledge base. All describe how to restart a service.

One is an up-to-date runbook. Another is an old troubleshooting note, and the third is an incident report.

A similarity search may return all three because they discuss the same service and symptoms.

The assistant then needs to decide which document should be treated as the source of truth.

For a well-designed RAG system, you'll need metadata such as environment, system, version, publication date, and document type.

This metadata should be used during retrieval wherever possible to narrow the results to information that applies to the user's request.

If someone asks about Staging, we wouldn't want to recommend Production instructions just because the error messages are similar.

This can't always be solved by writing a more persuasive prompt. Some restrictions need to be enforced in the retrieval and application layers.

Chunking can cause information loss.

Chunking is another common reason for misleading answers.

Long documents are often broken into smaller chunks. This makes them easier to embed and search, but important conditions can become separated from the instructions they apply to.

Say you have a runbook containing this instruction:

Restart the worker if the queue is stuck for more than 10 minutes.

Later in the document, it says this should only be done in a non-production environment and only after confirming that no jobs are running.

If the retrieved chunk doesn't include those conditions, the model may recommend an action that isn't appropriate.

The wording in the chunk may be correct, but the information is incomplete.

Teams should test whether important conditions are preserved throughout document processing and retrieval.

For procedural documents, keeping headings, warnings, prerequisites and related steps together may be more useful than creating equal-sized chunks.

Where possible, chunk boundaries should follow the structure of the original document.

Overconfident Answers

Say someone asks whether their integration tests are failing because of a recent deployment.

The assistant finds a deployment note and a few troubleshooting documents. None of them include the actual test execution logs.

The model may produce a plausible explanation based on similar issues, but there isn't enough evidence to confirm the cause.

A reliable assistant should distinguish between what is known and what remains uncertain.

It could respond with:

The deployment documentation describes a failure pattern similar to the issue you're facing. However, without the execution logs, I couldn't confirm the cause.

This answer is less certain, but it's more useful.

It tells the user what information is missing and what to investigate next.

RAG systems should be tested on whether they recognise when there isn't enough evidence to provide a reliable answer.

Check the citations.

Adding citations can make generated answers more transparent.

However, citations aren't useful if the referenced passages don't support the claims.

The assistant may cite a document discussing the correct service even though that document doesn't support its conclusion.

For example, a troubleshooting document may mention a timeout as one possible symptom.

The assistant might cite that document and confidently state that the timeout was caused by a recent deployment.

The citation looks legitimate, but it doesn't establish the cause.

Your evaluation should check the relationship between each claim and its cited passage.

In practice, you'll want to know whether the source supports the claim, whether it applies to the requested environment, and whether the assistant ignored contradictory evidence.

The presence of citations and the correctness of those citations should be measured separately.

Know where your issue is.

If your RAG assistant provides incorrect answers, you'll want to identify where the problem occurs.

This can be difficult if you're only testing the final response.

A useful approach is to break the evaluation into three areas:

• Retrieval quality: Did the search return the correct documents needed to answer the question?
• Context suitability: Were the documents current, applicable, and appropriate for the user's request?
• Answer grounding: Was the final answer supported by the retrieved evidence?

Say you have a test question with a known answer and a verified supporting source.

If the source never appears in the search results, retrieval needs investigation.

If the source appears but gets ignored during context selection, the issue is in the intermediate processing.

If the source reaches the model but the answer contradicts it, the problem is with answer generation or how the model uses the evidence.

Each of these issues requires