Why keyword search still earns its place next to vector search, why rank-only fusion flattened every score to about 0.031, and what we replaced it with.
Our AI search runs two searches on every question, one by meaning and one by keyword, and merges the results. After the merge, nearly every result had the same score: about 0.031. A score that’s the same for everything can’t tell a strong match from a weak one. This is why that happened, what it was hiding, and what I replaced it with.
Two searches, each blind where the other sees
Both searches look through the same chunks, the short passages cut from a company’s documents. If that’s new, what RAG is, in plain words walks through the whole flow.
Vector search works by meaning. An embedding model turns the question and every chunk into an embedding, a list of numbers that stands for what the text means, and search returns the chunks closest to the question. It finds “terminate the subscription” when you asked how to cancel your plan.
Keyword search works by the words themselves. Many engines, Elasticsearch among them, score with a formula called BM25, which rewards chunks that contain the question’s words, especially rare ones. It finds INV-2041 when you type INV-2041.
Names and codes are where the two part ways. An embedding squeezes a passage into a fixed list of numbers about its meaning, and a name or a code carries little meaning to squeeze. Two invoice numbers one digit apart can look almost the same to an embedding. To keyword search, that rare string is the most useful thing in the question.
How we found keyword search was earning its place
We test ideas in stages, and this one started where every experiment does: small and varied. The first set was 35 test questions, each with a known right document, picked to cover different kinds of question rather than many of one kind. That’s small enough to check every result by hand, and varied enough to show where each search breaks. I scored each search on its own.
Vector search found the right document for 25. Keyword search found it for 23.
On totals, keyword search looks like the one to drop. The breakdown by question type said otherwise. On name-heavy questions, the ones built around a person, a product, a project or a file, keyword search rescued 5 of 14. Those are the questions where one search knows the answer and the other doesn’t, so the way the two lists get merged decides whether the right document survives.
At this size, a gap of two between the searches is noise. The breakdown is the signal. A small set can’t settle which search is better overall, but it can show a pattern worth testing further, quickly and cheaply. That’s its job. An idea that survives it moves on to bigger sets, and it ships only if the result holds as the set grows. How we test in stages explains the whole approach.
Why rank-only fusion flattened every score
The two searches score on different scales. A vector similarity and a BM25 score can’t simply be added.
The popular way round this is reciprocal rank fusion (RRF). It ignores the scores and uses only positions. Each list gives a document 1 / (k + rank) points, and documents are sorted by their total. k is a constant. The 2009 paper that introduced RRF set it to 60, and Elasticsearch uses 60 by default.
Do the arithmetic with k = 60 and the problem shows:
| Where the document ranked | Fused score |
|---|---|
| 1st in both lists | 2/61 = 0.0328 |
| 5th in both | 2/65 = 0.0308 |
| 10th in both | 2/70 = 0.0286 |
| 1st in keyword search, missing from vector search | 1/61 = 0.0164 |
The constant is large next to the ranks that matter, so anything near the top of both lists lands between about 0.029 and 0.033. Our scores sat in that band, at about 0.031. Two things get lost.
Confidence. A near-perfect match and a loose one can sit one place apart and end up with almost the same score. The score stops saying how good a match is, so nothing downstream can use it to decide how many results to keep, or whether anything relevant came back at all.
The rescue. Look at the last row. When keyword search ranks the right document first and vector search misses it, RRF gives it 0.0164. A so-so document that both lists placed fifth scores 0.0308 and beats it. That is the name-heavy case from the experiment, and RRF puts it below anything both searches placed in their top ten.
The principle: merge scores, not just positions
Keep how strong each match was, and let the kind of question decide which search to trust more. A 2022 study of hybrid retrieval by Bruch, Gai and Ingber points the same way: a weighted sum of normalised scores beat RRF, and its one weight could be tuned from a small set of labelled queries.
The fix, step by step
I replaced rank-only fusion with score fusion weighted by query type. To do the same:
- Put both scores on one scale. Rescale each search’s scores to run from 0 to 1.
- Add them with a weight. Fused score = w × keyword + (1 − w) × vector.
- Choose the weight by query type. A question about a name or a code leans on keyword search. A descriptive question leans on vector search.
- Tune the weights on labelled questions, the same kind of set that showed you the problem in the first place.
One caution on step 1. Scaling each list by its own lowest and highest score always gives the top result a 1.0, even when it’s poor. If you want the fused score to mean confidence, scale by the score’s known range instead. Cosine similarity, for example, always runs from −1 to 1, so a weak best match stays weak.
What it changed
The fused score carries the strength of each match again. A strong keyword hit on a name can now outrank a document both searches only half liked, which is the case rank fusion scored lowest. And because the score means something, it can drive a cut-off.
RRF still has its place. It’s a sensible first merge when you have no labelled questions yet. Just don’t read its score as a confidence, and move to weighted scores once you have a small labelled set or anything downstream reads the score.
The lesson
Score each search on its own. The total made keyword search look like the weaker one, and only the breakdown showed what it was saving. The method is in how to test a RAG system, and the numbers behind this series are in the AI answers case study.
Quick answers
- What is hybrid search?
- Hybrid search runs vector search and keyword search on the same question and merges the two result lists. You get matches by meaning from one and exact matches on names, codes and IDs from the other.
- Is keyword search still needed if you have vector search?
- Often, yes. Vector search matches meaning but can miss exact names, product codes and IDs, which keyword search finds directly. In our test set, keyword search rescued 5 of 14 name-heavy questions.
- What is reciprocal rank fusion (RRF)?
- Reciprocal rank fusion merges ranked lists using only positions. Each list gives a document 1 / (k + rank) points, with k usually set to 60, and documents are sorted by their total.
- Why are RRF scores so close together?
- Because the constant k is large next to the ranks that matter. With k = 60, any document near the top of both lists scores between about 0.029 and 0.033, so the score says little about how good a match is.
- Is RRF or weighted score fusion better for hybrid search?
- RRF needs no tuning but throws away how strong each match was. A weighted sum of normalised scores keeps that, and research has found it can beat RRF once its weight is tuned on a few labelled queries.