RAG Evaluation: Score Retrieval and Answers Separately

What is RAG evaluation, in one answer?

RAG evaluation checks two things separately: whether search pulled back the passages the question needed, and whether the answer stayed inside what search pulled back. A single "was the answer right" score hides which of the two failed. Split them from day one, and every wrong answer points you at the part of the system that needs fixing.

Retrieval-augmented generation, or RAG, is the pattern behind most "chat with your documents" systems. A question comes in, a retriever searches your content for relevant passages, and a model writes an answer from those passages. It is the most common way businesses put their own knowledge in front of a language model, and it is the place we see teams grade themselves the least usefully.

The usual first eval is a spreadsheet of questions and a column that says right or wrong. That is a fine start for a demo. It stops being useful the first week you try to improve the system, because it cannot tell you what to change.

Why does one accuracy score mislead you?

Because a wrong answer has two very different causes, and they have different owners.

The first cause is a retrieval miss. The passage that answers the question exists in your content, but search never brought it back. The model did its best with what it got, which was the wrong material. The fix lives in the index: how documents are chunked, how they are embedded, what metadata is attached, how many results come back, whether a reranker sits on top.

The second cause is a generation failure. Search brought back exactly the right passage, and the model wrote past it. It blended in something from its training data, overstated a number, or answered a slightly different question than the one asked. The fix lives in the prompt, the grounding instructions, the model choice, or the output format.

A single accuracy score lumps both together. Worse, it hides a third case that looks fine on paper: an answer that is perfectly faithful to the retrieved text, where the retrieved text was the wrong document. The model did its job. The answer is still wrong. If you only grade the answer, you will spend a week tuning prompts for a problem that lives in search.

This is the core difference from ordinary software testing, which we wrote about in why evals are not software tests. A RAG system has two probabilistic stages chained together, and each needs its own measurement.

What should you measure on the retrieval side?

Two questions, and they pull in opposite directions.

Did the needed passages come back at all? This is context recall. The Ragas documentation defines it as measuring "how many of the relevant documents (or pieces of information) were successfully retrieved" and notes that calculating it "always requires a reference to compare against" (docs.ragas.io, fetched 2026-10-26). That second line is the one teams skip. You cannot measure recall without someone writing down, per question, which source should have come back.

How much of what came back was noise? This is context precision. Retrieval that returns ten passages to answer a question that needs one is not free. Every extra passage costs tokens on every call, and every irrelevant passage is a chance for the model to anchor on the wrong thing.

The trade-off is real. Pulling back more results tends to raise recall and lower precision. Pulling back fewer does the opposite. Measuring both is the only way to tune the number of results, the chunk size or a reranker on evidence instead of instinct.

If your documents are messy, retrieval scores will tell you fast. Duplicate versions of the same policy, scanned PDFs with no text layer and pages that mean nothing out of context all show up as low recall. Our notes on getting your documents ready for AI cover what to clean before you blame the retriever.

What should you measure on the answer side?

Again two questions: is the answer grounded, and is it on point?

Faithfulness asks whether every claim in the answer is supported by the retrieved context. Ragas defines it as how factually consistent a response is with the retrieved context, scored from 0 to 1 as the number of claims in the response supported by the retrieved context divided by the total number of claims (docs.ragas.io, fetched 2026-10-26). An answer with five claims, four of them traceable to the passages, scores 0.8. The fifth claim came from somewhere else, and that is the one that will end up in a screenshot.

Relevance asks whether the answer addresses the question that was actually asked. A faithful answer can still dodge the question by summarizing the retrieved passage instead of answering from it. Ragas lists this as response relevancy alongside faithfulness in its retrieval-augmented generation metrics, next to context precision, context recall, context entities recall and noise sensitivity on the retrieval side (docs.ragas.io, fetched 2026-10-26).

Both answer-side checks usually need a grader, and the grader is usually another model. That works, with care: graders drift, prefer longer answers and need their own spot checks against human judgment. We covered how to build and calibrate one in using a model as the grader.

How do you read the two scores together?

The value of splitting the eval shows up when you read the scores side by side:

None of this needs a platform to start. It needs a reference set and the discipline to score both halves every time.

How do you build the first reference set?

This is the part no tool does for you, and it is where most of the value sits.

Our recommendation is to start small: 30 to 50 real questions, collected from the people who will actually use the system, not invented by the team building it. That number is our starting point, not a benchmark. It is enough to show patterns and small enough that a domain expert can write the references in an afternoon or two.

For each question, a person who knows the material writes down two things: the source passage or document that should come back, and the key facts a correct answer must contain. The first feeds context recall. The second gives your faithfulness and relevance graders something to check against.

A few habits make the set last:

• Include questions with no answer in the content. The right behavior there is to say so, and you want to know if the system invents one instead.
• Include near-duplicates. Two policies with similar names, an old version and a new one. These are where retrieval quietly fails.
• Keep it in version control. The reference set is part of the system, and it changes when the content changes.
• Rerun it on every change. A new chunk size, a new embedding model, a reworded prompt or a fresh document load can all move scores. We wire this into CI the same way we described in our post on evals as quality gates, so a regression blocks the merge instead of reaching users.

This is the "Figure it out" step of how we test before we ship: agree on what right looks like, in writing, before anyone tunes anything.

When is a platform worth it?

Once the reference set exists and real traffic is flowing. Libraries such as Ragas give you the metric definitions and scoring code. Platforms package the same idea with dashboards, tracing and production monitoring; Galileo, for example, lists RAG Evals as its own category next to Agent, Safety, Security and Custom evals on its home page (galileo.ai, fetched 2026-10-26).

Either can save real time. Neither writes your reference answers, and neither knows which document in your shared drive is the current policy. Pick the tool after you know what you are measuring, not before. This is a big part of the AI integration for business work we do: the eval comes first, the tooling second.

FAQ

Do I need a reference answer for every question?

For context recall, yes: you need the expected source per question. Faithfulness can be scored against the retrieved context alone, but you still want reference facts to catch answers that are grounded and wrong.

Can a model grade faithfulness?

Yes, and most teams do. Spot-check the grader against human judgment on a sample, and recheck it when you change the grading model.

How often should we rerun RAG evals?

On every change to the index, chunking, embeddings, retriever, prompt or model, and on a schedule as new documents land.

What if retrieval scores are fine but users still complain?

Look at the questions users actually ask versus the ones in your reference set. The gap is usually there, and the fix is to add their questions to the set.

Build it with the eval in place

If your team is putting company knowledge in front of a model and wants it measured from the first commit, not after the first complaint, tell us what you are building.