Why an AI answer cache keyed on question text misses repeats, what our cache's records showed, and how to key on the retrieved documents instead.
An answer cache sounds simple. If someone asked this before, return the stored answer and skip the model. The hard part is deciding what “asked this before” means, and the obvious answer, the same question text, is the wrong one.
Why a question-text cache misses
A cache is a lookup table, and the key decides what counts as a repeat. Keyed on question text, a repeat only counts when the words match. People rarely ask the same thing the same way:
- “What’s our leave policy?”
- “how many days of leave do I get”
- “leave policy for 2026?”
Three keys, one answer. Lowercasing and stripping punctuation catches a few more. It doesn’t catch rewording.
A semantic cache is the usual next step. It turns each question into an embedding, a list of numbers that stands for its meaning, and returns a stored answer when a new question is close enough. That catches rewording, but “close enough” is a threshold you have to guess, and it fails both ways. “Leave policy for interns” and “leave policy for managers” sit very close as embeddings and want different answers.
Both keys are built from the question. Neither looks at what the answer is actually made from.
What the cache’s records showed
The cache’s own records pointed the way. Of the questions that missed, 63% had retrieved the same documents as one already answered. The wording was new. The evidence wasn’t.
That follows from how RAG works. The model doesn’t answer from memory. Retrieval finds the documents first, and the model writes its answer from them. (If that’s new, start with what RAG is.) Two questions that pull up the same documents, in the same answer mode, from the same model, are mostly asking for the same answer, whatever words they use.
The principle: key on what shapes the answer
An answer is shaped by its evidence, the mode, the model and the prompt, not by the exact words of the question. Key the cache on those, and rewordings become hits.
How to build a document-set key
- Measure before you build. Log each question’s company and sorted document IDs for a week or two, without serving anything. Count how often a new question’s set matches an earlier one. That’s roughly the hit rate to expect. Then read a sample of the would-be hits next to the answers they’d have got.
- Run retrieval as usual. You need its result to build the key, so a hit still pays for the search. That’s fine. Search is the cheap part. A hit skips the expensive part, the model writing the answer.
- Take the documents retrieval picked, after removing anything this user isn’t allowed to see. Sort their IDs so the order doesn’t matter, and include each document’s version.
- Add whatever else shapes the answer: the company, the answer mode, the model and the prompt version. Hash it all into one key.
- Let feedback reach the cache at once. A cache makes a good answer cheap. It also makes a bad answer common, because every hit serves it again. In ours, a thumbs-down retires a cached answer immediately, so the next matching question goes back to the model.
What to watch for
- Stale documents. With each document’s version in the key, an edit makes a new key and the old entry never matches again. Add a time limit as a backstop.
- Who can see what. Never share entries across companies. Inside one, build the key from the documents left after access filtering, so a user only matches answers built from documents they can open.
- Same documents, different question. “When does the new travel policy start?” and “Who approves travel exceptions?” may both land on the same policy and want different answers. A document-set key alone can’t tell them apart. Compare the new question with the one that produced the cached answer too, and track thumbs-downs on cached answers apart from fresh ones. If cached answers collect more, tighten the key.
- Your tests. A warm cache changes what a test measures. Turn it off, or start it empty, whenever you measure quality. Testing a RAG system has more on that.
What it changed
Caching by document set hits 33–36% of the time. Each hit skips the model writing an answer, which is where most of the cost of a RAG answer sits, as what an AI answer costs shows. And a flagged answer leaves the cache the moment someone flags it, so a mistake doesn’t keep spreading.
The lesson
Cache on what shapes the answer, not on how the question was worded. In RAG, that’s the documents. The rest of the numbers are in the case study.
Quick answers
- Can you cache LLM responses?
- Yes. Store each answer under a key that captures everything that shapes it, such as the documents used, the prompt version and the model, and remove entries when sources change or users flag them.
- Why does my LLM cache have a low hit rate?
- Usually because it's keyed on the question text, and people word the same request in different ways. Keying on what shapes the answer, such as the retrieved documents, catches repeats that exact matching misses.
- What is semantic caching for LLMs?
- Semantic caching stores answers keyed by the meaning of a question, usually an embedding, and returns a stored answer when a new question is close enough. It catches rewording but can match questions that look alike and mean different things.
- How do you keep cached AI answers from going stale?
- Put each document's version in the cache key so an edit creates a new key, add a time limit as a backstop, and remove an entry as soon as a user reports a bad answer.
- What hit rate can an AI answer cache get?
- It depends on your traffic. At BHyve, caching answers by the set of retrieved documents hit 33–36% of the time.