Our deepest AI answer mode cost more than it billed. Why an agent loop costs more every round, how we found it, and how a cap and a cache cut the bill.
Our deepest answer mode, Thinking, cost more than it billed. One Thinking answer cost around 140 times a fast question, and we charged 4 credits for it. The model’s price wasn’t the problem. The shape of the loop was.
First, find the cost per question
A monthly bill tells you how much you spent, not what spent it. When I started, many of our token records had no question attached. We could see the spend, but not which question, mode or step it belonged to. (A token is the unit a model reads and bills by, often part of a word.)
So metering came before any tuning. Every token is now recorded against the question that spent it, and priced correctly. Keep input, cached input and output apart when you do this, because each has its own price. Once cost is counted per question, an outlier like Thinking mode has nowhere to hide.
Why a loop costs more every round
Thinking mode works in rounds. It reads what it has, decides it needs more, asks for another search or document, and goes again until it can answer. That’s what people mean by an agent loop.
A model doesn’t remember the previous call. Each round is a new request, so it has to carry everything again: the instructions, the question, every document read so far and every earlier step. You pay for all of it, every round.
A toy example, not our numbers. Say each round adds one block of text:
| Round | Blocks sent this round | Blocks sent in total |
|---|---|---|
| 1 | 1 | 1 |
| 2 | 2 | 3 |
| 3 | 3 | 6 |
| 4 | 4 | 10 |
Four rounds of work, ten blocks of input. The total grows roughly with the square of the number of rounds, and the last rounds cost the most.
In a RAG system this lands on the expensive side of the bill. RAG means the system finds documents first and sends them to the model with the question (what RAG is explains it). Output usually costs more per token than input, but a short answer can sit on top of thousands of tokens of documents, so input is usually where the money goes.
That was Thinking mode. It re-sent its whole transcript every round, so four rounds sent about 90,000 input tokens instead of 33,000.
The principle: limit the repeats, then make them cheap
There are two levers. Cap how many rounds a loop can run, so the repeated text can’t keep growing. Then cache the part that repeats, so you stop paying full price to send it again. The cap limits how much repeats. The cache makes what repeats cheap.
The process
- Meter per question. Without it you can’t see which mode or step is costing you, or check that a fix worked.
- Cap the rounds. The late rounds cost the most, so a ceiling removes the worst of it. I capped Thinking mode at 4.
- Cache what repeats. In a loop, most of each request is the same as the last: the same instructions, the same documents, plus a little new text. Taking Google’s Gemini models as the example, there are two kinds of caching. Implicit caching is automatic: if a request starts with the same text as a recent one, those tokens can be billed for less, but the saving isn’t guaranteed. Explicit context caching is something you set up: you send the repeated content once to create a cache, and later requests point at it. Cached tokens cost less, and you pay a storage fee for as long as the cache lives. An agent loop is a good fit, because its instructions and the documents it has read come back every round. I added explicit context caching to Thinking mode.
- Check quality on a graded set. Cheaper only counts if the answers hold up. Grade them against known answers, not by feel.
- Keep each mode under its price. We sell answer modes as credits, so a customer sees a simple price per answer instead of a token count. That only works if each mode’s measured cost stays under what it bills, and per-question metering makes that a number you can watch.
What it changed
- Capping the loop took Thinking mode from ₹4.15 to ₹2.06 a question, half what it was.
- With explicit context caching, a measured turn went from $0.0025 to $0.00025.
- Quality held. The cap wasn’t the only change to Thinking mode (it also moved onto the same model as the default mode), so this belongs to the whole set. On a hand-graded test set of 47 SharePoint questions, Thinking mode went from 26 to 28 correct answers, and from 6 unanswered to 1.
Cost is now measured on every run. A change that adds correct answers but doubles the cost shows up as a trade-off to decide on, not a surprise on the next bill.
The cheapest answer is one the model never has to write. That’s the idea behind our answer cache. One caching idea didn’t make it: reordering prompts to chase implicit caching, written up in AI experiments we didn’t ship.
The lesson
In a loop, the last rounds cost the most. Cap them, cache what repeats, and count cost per question so you can see it happen. The full story is in the case study.
Quick answers
- What is a token in an LLM?
- A token is the unit a language model reads and writes, often a piece of a word. In English one token is about four characters on average, and providers bill per million tokens.
- Why are AI agents expensive to run?
- A model doesn't remember earlier calls, so each round of an agent loop re-sends the whole conversation so far. Input grows every round, and four rounds can cost far more than four times one round.
- How do you reduce the cost of an AI agent loop?
- Record tokens per question so you can see which step costs most, put a hard cap on the number of rounds, and cache the content that repeats from one round to the next.
- What is explicit context caching?
- Explicit context caching lets you store a large, repeated part of a prompt with the provider once and point later requests at it. Cached tokens are billed at a reduced rate, plus a storage fee for as long as the cache lives.
- How much does one AI answer cost?
- It depends on the model and how much text you send. On BHyve's platform a knowledge-base answer measured about $0.0012, and a SharePoint answer in the deepest mode $0.0048.