Compressed vector indexes are fast but can miss the right results. Our recall@10 fell to 0.29 on large tenants, and a refine step brought it to 0.83.
After we moved every production tenant to LanceDB, vector search was fast. Then I measured recall on our large tenants. With a compressed index and no refine step, recall@10 was 0.29.
Fast, in other words, did not mean right. Here is what recall@10 measures, why compression costs you some of it, how to check your own, and the refine step that brought ours back to 0.83.
Recall@10 asks: did we find the right ten?
A vector search returns the chunks whose embeddings sit nearest to the question’s embedding. Exact search, also called flat or brute-force search, compares the question with every vector. It always finds the true nearest ten, and it gets slow on big tables.
An approximate index is a shortcut. It finds near neighbours fast by looking at only part of the data, and accepts that it will sometimes miss.
Recall@10 measures how much it misses. Take the true ten nearest from exact search. Count how many of them the index returned in its top ten. Divide by ten. A score of 1.0 means it found all ten. A score of 0.29 means that, on average, fewer than three of the ten results were the right ones.
In RAG this matters directly. RAG, retrieval-augmented generation, means finding the right passages first and then having the model answer from them. The model can only answer from the chunks it’s given. If search misses the chunk that holds the answer, the model answers without it.
Why compression costs recall
Indexes for large collections often compress the vectors, so more of them fit in memory and each comparison is cheap. A common way to do it is product quantisation, PQ for short.
PQ cuts each vector into short pieces. For each position, it learns a small set of typical pieces, called a codebook. Then it stores each piece as the number of its closest codebook entry, instead of the real values.
Think of describing every colour by the nearest paint chip in a catalogue. “Chip 41” takes far less space than the exact colour, and colours that are close in real life usually get close chips. But two blues that are almost the same can land on the same chip, and then you can’t tell which one is closer to the blue you’re looking for.
LanceDB’s IVF_PQ index is one example. IVF groups the vectors into partitions around centre points and only searches the partitions nearest the question. PQ then compresses the vectors, which LanceDB’s docs put at usually 1/64 to 1/16 of their raw size. The search ranks by these compressed distances, and those are estimates. When many candidates sit close together, the estimates can put the wrong ones on top.
That’s one common reason recall drops first on the biggest tables. The more vectors there are, the closer together the nearest ones sit, so the same small errors in estimated distance swap more of them in and out of the top ten. The biggest tenants were exactly where we found the gap.
The fix: re-check the shortlist with the full vectors
Compression is good at finding the neighbourhood and bad at ordering the houses in it. So use it for the first part only. Ask the index for more candidates than you need, re-score that longer shortlist with the full, uncompressed vectors, and keep the best.
In LanceDB this is the refine_factor setting. With a limit of 10 and a refine factor of 10, LanceDB takes the 100 best candidates by compressed distance, reads their full vectors, computes exact distances, and returns the true top 10 of those 100. LanceDB’s docs suggest trying refine factors between 5 and 50 and measuring latency as you go.
A second setting, nprobes, is the number of partitions searched. Searching more partitions also raises recall, at some cost in speed.
How to check your own recall
- Take a sample of real queries. From production, not made up.
- Get the true answers. Run each query as an exact search. In LanceDB,
bypass_vector_index()skips the index and does a flat scan. - Run the same queries through your index with your production settings.
- Compute recall@10 per tenant or table size. An average across everyone can hide the big ones, and the big ones are where ours dropped.
- Try refine and nprobes. Measure recall and latency together, and pick the cheapest setting that gets you the recall you need.
- Run it again when tables grow a lot, or when you rebuild or change the index.
What refine changed for us
On our large tenants, adding refine brought recall@10 back from 0.29 to 0.83, for about 8 ms. At the time of writing, that fix is documented and next on our list.
The reason it went unseen until then is worth stating. The move from Postgres to LanceDB was measured as it went, and the load test said LanceDB was fast: 5.3 ms at 100k rows, 6.2 ms at 500k. But speed doesn’t tell you whether the results are the right ones. A compressed index can be fast and wrong at the same time, and no latency chart will show it.
The lesson
Measure recall against exact search, not just speed, and keep measuring after the switch. That’s how we found this gap.
All our numbers from this migration are in the case study. For the wider question of testing retrieval before it reaches users, see testing a RAG system.
Quick answers
- What is recall@10 in vector search?
- It is the share of the true 10 nearest neighbours, found by exact search, that also appear in the 10 results an approximate index returns. A recall@10 of 1.0 means the index found all ten.
- Does product quantization reduce recall?
- It can. Product quantization stores each vector in a compressed, approximate form, so search ranks by estimated distances, and close results can be put in the wrong order or missed.
- What does refine_factor do in LanceDB?
- It makes LanceDB fetch more candidates than you asked for, the limit times the refine factor, using compressed distances, then re-rank them with the full vectors and return the top results.
- How much latency does a refine step add?
- LanceDB's docs describe the cost as typically a few milliseconds and suggest refine factors between 5 and 50. On our large tenants, refine took recall@10 from 0.29 to 0.83 for about 8 ms.
- How do I measure the recall of a vector index?
- Run a sample of real queries as an exact search to get the true nearest results, run the same queries through the index, and count how many of the true results the index returned.