Reordering prompts for caching and adding few-shot examples both made our AI answers give up more often. Why scoring each stage made both calls obvious.
Two well-known prompt changes made our AI answers give up more often. Each came with a good reason to try it, and neither shipped. Neither call was hard, either, because our tests score each stage of an answer on its own.
What a give-up is
Our answer engine searches a company’s knowledge base and connected tools like SharePoint, Gmail and Outlook, then writes an answer. A give-up is when it replies that it couldn’t find the answer instead of answering.
Sometimes that’s the honest reply. But on a test set where every question has a known right document, a give-up is a miss.
Why each idea looked good
Reorder the prompt for caching. Providers like OpenAI and Google cache prompts automatically. If a request starts with exactly the same text as a recent one, they reuse the work for that shared start, called the prefix, and bill those tokens for less. So the standard advice is to put what never changes first, like the instructions, and what always changes last, like the question and its documents. It’s a cheap change, and it saves money.
Add few-shot examples. Few-shot prompting means putting a handful of worked examples in the prompt, a sample question with its ideal answer, so the model follows the pattern. It’s one of the most common prompting tips.
What the scores showed
The reorder was worth about ₹20 a month. It pushed give-ups from 50% to 56%. Few-shot examples pushed them to 84%. Both stayed off.
Why scoring each stage made the call obvious
Each test question has a known right document, and a run scores four stages against it:
- Search: did the right document come back?
- Pick: was it among the documents sent to the model?
- Read: did the model pull the right part out of it?
- Answer: was the final answer right, or did it give up?
A change to the prompt the model answers from comes after search and pick have done their work. So if the right document was still found and still picked, and the model gave up anyway, the cause has nowhere else to be. Without stage scores, a rise in give-ups could be blamed on search, on the data or on luck, and the argument goes on.
The other half is putting cost next to quality. About ₹20 a month on one side, more unanswered questions on the other. Once both are on the page, the call makes itself.
As for why a reorder can hurt: the order of a prompt is part of what the model reads. Research on long prompts has found that models use information best near the start or the end and worst in the middle, so moving the question and documents around changes how well the model uses them. Treat a reorder as a quality change with a billing side effect, and test it like one.
Few-shot examples have a cost side too. They add tokens to every call, so even a neutral result would have meant paying more for nothing.
Stage scores catch good surprises as well as bad ones. A cheaper reader model turned out to make fewer mistakes, not more: 2 wrong answers out of 87 against 6–7 for the pricier one. Measuring the stages separately is what made that visible.
Caching itself wasn’t the problem. Explicit context caching, set up on purpose rather than chased by reordering, did cut the cost of our deepest mode. That’s in what an AI answer costs.
Make sure the score is real first
Two checks before believing any result. Run it more than once, because models don’t always give the same output for the same input. And start every run from the same clean state, because a warm cache can move a score with no change to the code. Testing a RAG system covers both.
Write the negative result down
A negative result that only lives in someone’s head gets tried again. Someone reads the same caching guide or prompting tip, makes the same change and finds out all over again.
A written one does three jobs:
- It stops the repeat. The next person finds the note before they find the tip.
- It records the conditions. “Few-shot examples raised give-ups on this model, with this prompt, on this test set” is a different claim from “few-shot doesn’t work”. When the model changes, you know what to rerun.
- It shows your tests can say no. If an evaluation never rejects anything, you can’t trust it when it says yes.
A few lines are enough: the change, why you tried it, how you measured it, quality and cost before and after, the decision, and what would make it worth trying again.
The lesson
A change that didn’t ship is still a result. Score the stages, put cost next to quality, and write down what you kept off. All the numbers here come from the case study.
Quick answers
- What is few-shot prompting?
- Few-shot prompting puts a handful of worked examples, such as a question and its ideal answer, into the prompt so the model follows the same pattern. Measure it on your own task before shipping it.
- How does prompt caching work?
- Providers such as OpenAI and Google reuse the work for a prompt prefix they have seen recently and bill those tokens at a lower rate. It only applies when the start of the prompt matches exactly, so stable content should come first.
- Does the order of a prompt change the answer?
- It can. Reordering changes the input the model sees, and models use information differently depending on where it sits in a long prompt. Treat a reorder as a quality change and test it.
- How do you know if a prompt change made things worse?
- Score each stage of the answer against a known right document. If search and pick are unchanged and the final answers get worse, the prompt is the cause.