I own the engine behind BHyve's AI answers, which searches a company's knowledge base and connected tools like SharePoint, Gmail and Outlook. This is how I made it cheaper per question, measurably more accurate, and configurable per customer without a fork.
Rank-only fusion flattened every score to about 0.031, the deepest mode cost more than it billed, and 6,043 token records had no question attached.
Score fusion tuned per query type, every token metered per question, models picked per role on measured quality, and three answer modes sold as credits.
The problem
Two things were invisible. Quality: the system counted an answer as success even when it was wrong. Cost: Thinking mode re-sent its whole transcript every round, so four rounds sent about 90,000 input tokens instead of 33,000, around 140 times the cost of a fast question for 4 credits.
And every customer wanted something different: their own AI provider and key, their own extraction depth, their own SharePoint folders in or out.
The idea
Make quality and cost measurable per question first, then tune. Score each stage (search, pick, read, answer) against a known right document, price every token correctly, and turn every customer difference into a setting instead of a branch.
How I did it
- Ran a selection experiment on 35 real questions. Vector search found the right document for 25 and keyword search for 23, but keyword search rescued 5 of 14 name-heavy questions. I replaced rank-only fusion with score fusion weighted by query type.
- Picked models per role on measured results. On 7 ground-truth questions, the default mode answered 1; a deeper re-fetch plus a different model answered 6. That model was also the only one that reproduced its own result across runs, 11 of 12 twice.
- Fixed the deepest mode: capped its loop at 4 rounds (₹4.15 down to ₹2.06 a question), added explicit context caching (a measured turn went from $0.0025 to $0.00025), and moved it onto the same model as the default.
- Built an answer cache from production traffic: 63% of misses had retrieved the same documents, so caching by document set hits 33–36% of the time. A thumbs-down retires a cached answer immediately.
- Tested what not to ship. Reordering prompts for provider caching was worth about ₹20 a month and pushed give-ups from 50% to 56%; few-shot examples pushed them to 84%. Both stayed off.
- Built an evaluation harness that runs the real production path in a disposable read-only pod, after finding the same code scored 6/22 or 17/22 depending only on the cache.
- Made it configurable per company: vector store, answer modes and credits, bring-your-own OpenAI or Gemini key, extraction tier, connector access rules, and a hot cache of up to 20,000 files.
The impact
On 47 hand-graded questions from a customer’s SharePoint, Smart mode went from 19 to 24 correct answers and halved unanswered questions, 12 to 6. Thinking mode went from 26 to 28 correct and from 6 unanswered to 1. Cost is measured on every run: about $0.0012 for a knowledge-base answer, $0.0025 for a SharePoint answer in Smart mode and $0.0048 in Thinking mode.
A cheaper reader model turned out to make fewer mistakes, not more: 2 wrong answers out of 87 against 6–7 for the pricier one. Measuring the stages separately is what made that visible.
An answer is not a correct answer. Once I scored the stages separately, most of the wins were obvious, and so were the experiments not worth shipping.