Our AI answers counted wrong answers as successes. Why end-to-end checks hide that, how to score each stage of a RAG system, and why tests must start clean.
Our AI answers had a quality problem we couldn’t see. The system counted an answer as a success even when it was wrong.
That’s the first thing to fix when you test a RAG system (retrieval-augmented generation: search your documents, then have a model answer from what it found, explained in what RAG is, in plain words). Getting an answer back proves the pipeline ran. It doesn’t prove the answer is right.
Why a fluent answer hides the failure
A RAG answer passes through several stages before anyone reads it. Any of them can fail quietly. Search can miss the right document. The step that picks what to read can drop it. The model can misread it. And the final text still reads well, because a model writes fluently from whatever it was given.
Judged only at the end, all of those look the same: a confident answer. Without knowing what the right answer was, you can’t tell right from wrong. Without knowing where it went wrong, you can’t tell which stage to fix.
The principle: write down the right document first
Every test question needs two things written down before you run anything: the document that holds the answer, and the correct answer. With those, every stage can be scored on its own, and the first stage that fails is the one to fix.
| Stage | What it does | What to check |
|---|---|---|
| Search | finds candidate documents | Is the right document in the results? |
| Pick | keeps the few worth reading | Did the right document survive the cut? |
| Read | the model reads the picked documents | Did it pull out the part that answers the question? |
| Answer | writes the reply | Is it correct, wrong or unanswered? |
If search never found the document, a smarter model won’t help. If search found it and pick dropped it, look at how results are scored and cut. If the model had the right document and still got it wrong, look at the reader model or the prompt.
The first two checks are automatic once you know the right document. Read and answer usually need a person grading, at least until you trust an automatic grader.
The second hidden cause: state
At one point the same code scored 6/22 or 17/22, depending only on the cache. Nothing in the code changed. A cache carries results from earlier runs into this one, so a run with a warm cache and a run with a cold one are testing different things. A test that can swing that far on hidden state isn’t measuring the code.
So I built an evaluation runner that runs the real production path in a disposable, read-only pod. A pod is the smallest thing Kubernetes runs: one or more containers that start and stop together. Each part of that does a job:
- The real production path. The same code and settings users get, not a script that imitates them. A copy drifts from the real thing, and then you’re testing the copy.
- Disposable. A fresh pod for each run, thrown away after, so no run inherits a warm cache from the one before.
- Read-only. A test run can’t change anything it touches.
Randomness is the other source of swing. Models don’t always give the same output for the same input, so a single good score can be luck. When I compared models for each role, the one that did best was also the only one that reproduced its own result across runs. A model that scores differently every time is not one you can tune against.
Start small and varied, then grow the set
We don’t start an experiment on thousands of questions. Every test starts small, and grows only once the idea has earned it:
- A small, varied first set. A few dozen test questions, each with its right document written down, picked to cover different kinds of question: names and codes, broad topics, answers in tables, questions the documents can’t answer. At this stage variety matters more than size. A big set of similar questions hides exactly the cases that break.
- Check every result by hand. At this size you can read every answer, so you learn why something failed, not just that it did.
- Stop weak ideas early. If an idea doesn’t help on the small set, it ends there, while stopping is still cheap.
- Grow the set to validate. An idea that survives moves to a bigger set, and then to a much larger one. Small sets are fast but noisy, so a result has to hold as the set grows before we believe it.
- Ship in stages. Even a validated change goes to dev, then staging, then production, so a surprise reaches as few people as possible.
This keeps each experiment short and the risk small. Most ideas are settled on the small set, and only the few that look real pay for the bigger runs. Our embedding model test sat at the big end of that ladder: 935 test questions, judged blind.
How to build your own eval set
- Write realistic test questions, worded the way people really ask, and write down the right document and the correct answer for each. Vague wording and half-remembered names are what break search, so put them in on purpose. A few dozen is enough to start.
- Include the hard kinds on purpose: questions built around names and codes, answers that live in a table, and questions your documents can’t answer, so you can check the system says so.
- Score search, pick, read and answer separately, and note the first stage that failed.
- Grade every answer correct, wrong or unanswered. An unanswered question is honest. A wrong answer is confident and misleading, and it’s the one that costs a user’s trust. Fewer unanswered questions only count as progress if wrong answers don’t climb to fill the gap.
- Run each test at least twice, on the real path, from a clean state that can’t write to real data. If results swing, find the hidden state before you trust any number.
- Record the cost per question with every run. A change that adds two correct answers and doubles the cost is a decision, not a win.
- Re-run after every change, including the ones you’re sure about, and keep re-running after launch.
What it changed
On a hand-graded test set of 47 SharePoint questions, before and after the changes in the case study:
| Answer mode | Correct | Unanswered |
|---|---|---|
| Smart | 19 → 24 | 12 → 6 |
| Thinking | 26 → 28 | 6 → 1 |
Smart and Thinking are two of our answer modes. Thinking is the deeper one.
Stage scores also showed things a final score would have buried:
- Keyword search earns its place. On totals it looked weaker than vector search, but it rescued 5 of 14 name-heavy questions. That’s why I changed how the two get merged, in why rank fusion flattened our scores.
- The cheaper reader made fewer mistakes: 2 wrong answers out of 87 against 6–7 for the pricier one. With its own score, the read stage showed the swap as its own change.
- Some popular ideas made answers worse, and the scores said so plainly. Those are in AI experiments we didn’t ship.
Measuring after a change ships matters too. Re-measuring after our move to LanceDB is how I found that compressed vector indexes were losing recall on large tenants, written up in the recall you lose to compressed vector indexes.
The lesson
An answer is not a correct answer. Once I scored the stages separately, most of the wins were obvious, and so were the experiments not worth shipping. The numbers are in the case study.
Quick answers
- How do you evaluate a RAG system?
- Give every test question a known right document and a correct answer, then score each stage on its own: did search find the document, was it picked, did the model read it correctly, and is the final answer correct, wrong or unanswered.
- Why score retrieval separately from the final answer?
- Because a fluent answer can rest on the wrong document. Scoring retrieval on its own tells you whether a bad answer came from search missing the document or from the model misreading it, and those need different fixes.
- What is recall@k?
- Recall@k is the share of the relevant items that show up in the top k results. With one right document per question, it simply asks whether that document is in the top k.
- Why do RAG evaluation results change between runs?
- Hidden state such as caches, plus randomness in model output, can move scores without any code change. Run each test more than once, in a fresh environment, so the score reflects the code.