Evaluate RAG Retrieval and Answer Quality

A polished answer is a poor diagnostic. When a retrieval-augmented generation service gets a question wrong, the source might be absent, the search might miss it, or the model might mishandle evidence it received. Each failure needs a different fix.

Keep two connected scorecards: one for the evidence retrieved and one for the answer produced. Microsoft's RAG evaluator documentation separates process measures from system measures. The RAGAs research paper also treats retrieval, faithful use of context and generation quality as distinct dimensions. A small team can apply that separation without building a research platform.

Build cases that name the evidence

For each test question, record the approved source document and the passage or fact that should support the answer. Record the user's access scope, the applicable date and any acceptable alternative source. Keep a short expected answer, but do not require identical wording.

Include three types of question. Some have one clear source. Some need two sources together, such as a standard rule and its exception. Some cannot be answered from the approved collection. That last group checks whether the service admits a gap.

An illustrative case might ask whether a particular product includes on-site installation. The expected evidence is the package description plus a regional exclusion. Retrieving only the description should fail evidence completeness even if the final answer happens to guess the exclusion.

Use a development set for tuning and a separate held-out set for release decisions. Once the team has repeatedly adjusted a prompt around a question, that question is no longer a strong independent check of general behaviour.

Run retrieval without the answer model

Save the returned document IDs, chunk IDs, ranks and filter settings. Inspect whether the necessary evidence appears within the context budget actually sent downstream. A relevant result at rank 40 is no help if the application sends only the first five chunks.

For a small labelled set, three measures are enough to start:

• the share of answerable questions with all required evidence available;
• the number of irrelevant or duplicate chunks occupying the context budget; and
• the cases returning material outside the permitted date or access scope.

Use standard retrieval metrics where they help compare many runs, but define your labels first. “Relevant” can mean vaguely related or sufficient to decide the question. Those are different targets.

Group failures by cause. Missing source content belongs with the document owner. Poor extraction belongs with ingestion. A filtered-out approved document belongs with metadata or access configuration. A relevant chunk ranked too low belongs with search design. This keeps a model upgrade from becoming the default answer to every defect.

Score the answer against the context it received

Now run generation with the saved retrieval output. Check whether each material claim is supported, whether the answer includes the necessary exception and whether its citations point to the supporting passages.

Groundedness and usefulness can diverge. An answer that repeats one correct sentence while ignoring half the question may be grounded but incomplete. An answer based faithfully on an expired policy may still be wrong for today's decision. Microsoft's end-to-end evaluation guidance discusses several measures because one number cannot express those trade-offs.

If using an LLM judge, save its model, rubric and version. Have a person review disputed cases and a sample of passes. The judge is another fallible component. Do not make a convincing explanation from the judge the sole evidence that the answer is correct.

Use controlled comparisons to locate the fault

Two additional runs are particularly useful. Give the answer model the known correct evidence: if it still fails, retrieval is not the immediate cause. Then keep the evidence fixed while changing the generation configuration: this isolates the prompt or model change from the search change.

Change one component per comparison where practical. If chunking, embedding model, ranking and generation all change together, an improved score tells you little about which change helped or which one to reverse.

Record latency, failed requests and cost alongside quality. A search configuration that gains a little coverage while making the service unusably slow may be the wrong operational choice.

Make the release decision explicit

Set thresholds by consequence before seeing the results. Any unauthorised retrieval should block a release. A missed optional explanatory detail may be acceptable. Publish the categories and exceptions in the test report rather than hiding failures inside an average.

Keep the old configuration available during a limited rollout and record the failing case that would trigger a return to it. The codebase evaluation note makes the same case for task-specific tests; here, the extra requirement is a trace from the question through retrieved evidence to the final claim.

References

• Microsoft, Retrieval-Augmented Generation evaluators. Accessed 17 October 2026.
• Es, James, Espinosa Anke and Schockaert, RAGAs: Automated Evaluation of Retrieval Augmented Generation. EACL System Demonstrations, March 2024, pp. 150–158; abstract and publication record read. Accessed 17 October 2026.
• Microsoft, Large language model end-to-end evaluation. Accessed 17 October 2026.