Why a leaderboard can't tell you how a model will do on your data, and how to design a fair test: realistic questions, one change, a blind judge.
Changing the embedding model can make every answer a little better or a little worse, and nobody will notice by reading a few. The leaderboard score that put the model on your list won’t tell you either. Here is why, and how to design a test you can trust before a model reaches customers.
An embedding is meaning written as numbers
An embedding model turns a piece of text into a list of numbers called a vector. Think of the numbers as coordinates on a map of meaning. Texts that mean similar things land close together, even when they share no words. “How do I reset my password” and “I’m locked out of my account” end up near each other.
Our AI search works on those coordinates. Every chunk of every document gets a vector. A question gets one too, and we return the chunks that sit nearest to it. If the model puts the wrong things near each other, no amount of clever search will fix it.
A model’s dimensions are how many numbers it gives each text. Fewer numbers mean less storage and faster comparisons, as long as quality holds.
One thing to know before changing models: vectors from two different models can’t be compared. Switching means embedding every stored chunk again. So a model change is a migration, not a config flag. (The search side of that is in why we moved AI search from Postgres to LanceDB.)
Why a leaderboard can’t choose for you
The usual starting point is a public leaderboard such as MTEB, the Massive Text Embedding Benchmark. It scores models on dozens of public datasets across several kinds of task, from retrieval to classification and clustering.
That makes it a good shortlist and a poor decision. A benchmark measures a model against someone else’s documents, someone else’s questions and someone else’s idea of a right answer. Each of those can differ from yours:
- Your documents aren’t the benchmark’s documents. Ours are a company’s own files and the content of its connected tools. A benchmark can’t tell you how a model handles that.
- Your questions aren’t the benchmark’s questions. Benchmark queries are written for the benchmark. Real ones come from real people with real habits: half-sentences, internal jargon, a product name spelled three ways.
- An average hides the task you care about. A high overall score can come from tasks that have nothing to do with search.
- Public datasets leak. A model trained on data close to the benchmark scores well there and not necessarily anywhere else. The user guide to the leaderboard, published on Hugging Face, warns about data leakage and overfitting, and tells readers to test their top few choices on their own data.
What you need to know is narrower than anything a leaderboard measures. For the questions your users ask, over the documents your customers hold, does this model put the right chunk near the question? Only your own data can answer that.
The principle: run it like an experiment
A fair test has three properties. Only one thing changes between the two sides. Nothing about the setup can favour either side. And there are enough questions that luck can’t explain the result. Every step below serves one of those.
How to design a fair test
- Shortlist from the leaderboard, then put it away. Look at the retrieval scores, and at the datasets closest to your kind of documents, not the overall average.
- Write realistic questions. Hundreds of them, worded the way people really ask, with the vague phrasing and half-remembered names that break search. Tidy questions test what you thought of, not what people ask.
- Change one thing. Same chunks, same vector store, same reranker. Only the model differs. If two things change, you can’t tell which one moved the result.
- Blind the judge. For each question, a judge sees both results side by side and picks the better one, without knowing which model produced which. A judge that knows can favour the one it expects to win.
- Shuffle the order for every question. Judges, and LLM judges in particular, are known to prefer an answer partly because of where it sits. If one model is always shown first, that bias becomes part of its score. Shuffling spreads it evenly, so it can’t push the result either way.
- Use enough questions, and say how sure you are. Report a win rate with a p-value. The p-value answers one question: if the two models were really equally good, how often would a test like this show a gap at least this big by luck alone? It depends heavily on the number of questions. On a few dozen, a modest lead can easily be chance. On hundreds, the same lead can be hard to explain any other way. A p-value says a lead is probably real, not that it’s big.
- Add a labelled set if you can. For questions with one known right document, measure Acc@1 (accuracy at 1): how often the right result comes back in first place. It’s a second opinion that doesn’t depend on a judge.
- Test with and without the reranker. A reranker is a second model that takes the top results from search and re-scores each one against the question, so the best match moves to the top. It can close a gap between two models, or hide one.
- Price the whole thing. Count dimensions (storage and search time), memory, latency and per-token fees, not only quality. A model that ties on quality and costs less is a win.
- Roll out like a migration. A new model means new vectors everywhere. Move one environment at a time, and keep the old vectors until the new ones have held.
What our tests showed
A blind judge on 935 test questions. To pick our latest model, I ran a head-to-head on 935 test questions with a blind, position-randomised judge. The new model’s win rate was 54.6%, with p = 0.008.
54.6% isn’t a landslide, and it doesn’t need to be. It says the new model gave the better result more often than not, across a large test set. A p-value of 0.008 means a gap this big would turn up by luck less than 1 time in 100 if the models were equal, so luck is a poor explanation. The model then rolled out to dev, staging and production in turn.
A small in-process model against a paid API. A separate test asked whether we needed to pay for embeddings at all. We had been calling a paid embedding API. I compared it with a local 256-dimension model that runs in-process, meaning inside our own AI service instead of over the network. They matched: both scored 91.4% Acc@1, and 97.1% with the reranker.
A tie meant we could stop paying a per-token fee. The cost moved somewhere else: memory per AI pod went from 1.5 to 2 GiB, because the model now lives inside the pod. That was the right trade. The rest of that story is in the server costs case study.
The second test is the one people skip. A fair test can tell you to buy less, not only to buy better.
The lesson
Use the leaderboard to choose what to test, and your own questions to choose what to ship. If a model only wins on the leaderboard, don’t ship it.
Quick answers
- What is an embedding model?
- An embedding model turns text into a list of numbers, called a vector, so that texts with similar meanings get vectors that sit close together. Search then returns the stored vectors closest to the question's vector.
- Is the MTEB leaderboard enough to choose an embedding model?
- No. It is a good way to build a shortlist, but it scores public datasets that may look nothing like your documents or your users' questions. Test the shortlist on questions written the way your users really ask.
- How do I compare two embedding models fairly?
- Run both on the same test questions with everything else unchanged, hide which model produced which result, shuffle the order for every question, and use enough questions that luck can't explain the result.
- What is position bias in an LLM judge?
- It is a judge's tendency to prefer an answer because of where it appears, first or second, rather than what it says. Randomising the order for each question spreads that bias evenly, so it can't favour either system.
- Do I need to re-embed my documents when I change embedding models?
- Yes. Vectors from different models can't be compared with each other, so every stored chunk has to be embedded again with the new model, and questions must be embedded with the same model.