Our platform was sized for thousands of users and served 30 to 100 a day. Why infrastructure cost is part of the product, and how to find what you overpay for.
Our platform was built for thousands of users. It was serving 30 to 100 a day. Nothing was broken. It was just expensive, because every service reserved capacity for a load we didn’t have yet, and reserved capacity costs the same whether anyone uses it or not.
At an early-stage SaaS company, I think that’s a product problem, not just an ops one. Here’s why, how it happens, and the questions that find it.
Cost belongs next to price
Our target is for infrastructure to cost no more than about a fifth of what a user pays.
That one sentence ties together decisions that usually live in different meetings. The price is a product decision. Which features run heavy AI work is a product decision. How the services are built decides what each of those costs. When infrastructure takes more than its share, something else has to give: the price goes up, the margin disappears, or a feature gets cut.
So what it costs to serve a user is a property of the product, like how fast it is. Engineers are the ones who can see that number, which makes it our job to say it out loud.
Overspending happens one sensible decision at a time
Nobody chooses to overpay. Each service gets sized for the growth everyone hopes for. Each environment gets its own full copy, to be safe. Heavy work runs next to live traffic, because that’s simplest. Every one of those choices is reasonable on its own. None of them gets revisited, because an over-sized system never breaks.
We had all three. The AI service ran as three full deployments, one each for dev, staging and production, each autoscaling on its own and each paying for its own embedding API. And heavy indexing shared pods with the service answering live questions, so capacity for the biggest jobs had to be there all day.
Measure the headroom before cutting anything
The first step wasn’t a cut. It was a load test: about 2,000 simulated users, no errors, and every pod under 10% CPU. A load test sends a large amount of simulated traffic to a system to see how it copes.
That showed how much room we really had, and we turned it into a capacity and pricing model. A capacity model says how much traffic a given setup can carry. Put prices next to it and you can see what serving each user costs against what they pay.
Cutting without that is guessing. Cut too little and you keep paying for air. Cut too much and your users find the limit before you do.
Ask when a workload needs to exist
Most of the savings came from asking when a workload actually needs to exist, not from squeezing it. That question has three versions.
Does each environment need its own copy? Not if the service is stateless. A stateless service keeps nothing between requests, so it doesn’t matter which environment a request comes from. We made the AI service compute-only, so it extracts, chunks and embeds while the platform stores the vectors and runs search. Then one service could serve all three environments. The cutover had strict ordering, a rollback plan and a canary, a first small release of the new version, so a mistake shows up while it can only hurt a little.
The fair objection is that a bad release now reaches dev and production alike. Separate copies would contain it. But that safety was costing three deployments, three sets of secrets and three paid embedding APIs, for a platform serving 30 to 100 users a day. A canary and a rollback plan buy back most of it for much less. A different company at a different size could reasonably decide otherwise.
Does this work need to run now? A bulk import has to be searchable by morning, not in seconds. Work that can wait goes on a queue that drains at night, on pods scaled with KEDA that don’t exist by day. The same reasoning put a GPU on that queue for the night only. A GPU only at night explains why.
Should this be a fee or a fixed cost? A paid embedding API charges for every token, forever, and the bill grows with every document a customer adds. A model running inside the service needs more memory per pod, a fixed cost you can see in advance. We only switched after an A/B test showed the local model matched the paid API on accuracy, at 91.4% Acc@1 for both. Acc@1 is how often the right passage comes back first.
Notice what happened in that last one. Memory per AI pod went up, and that was the right trade. Cost work that only tries to make every number smaller misses trades like this. Sometimes the cheaper system uses more of one thing.
What it changed
- Three AI deployments became one, with 75% fewer idle AI pods.
- For 56% of every day, the night workers cost nothing, at zero pods.
- Live answers stayed fast, because heavy work no longer shares a pod with them.
Smaller images and a CPU-throttling fix came out of the same effort. Shrinking Docker images and Why CPU throttling hides cover those, and the server costs case study has the rest.
Questions I’d ask about any platform
- What does serving one user cost, against what they pay? Write down a target.
- How much headroom do you really have? Load-test before you cut.
- Which services are copies of each other, and could one stateless service serve them all?
- Which work has to happen now, and which can wait for the night?
- Which per-use fees could become a fixed cost you control?
- Which number should go up so the total comes down?
The lesson
The cheapest compute is the compute that isn’t running. Ask when each workload needs to exist before you ask how to make it smaller.
Quick answers
- How much of revenue should cloud costs be for a SaaS startup?
- There's no universal rule. At BHyve the target is for infrastructure to cost no more than about a fifth of what a user pays, which leaves room for everything else the business has to pay for.
- Why do startups overspend on cloud infrastructure?
- Usually because services are sized for the growth everyone hopes for, and nothing pushes the sizes back down. An over-sized system never breaks, so nobody looks at it again.
- How do you reduce Kubernetes costs at a startup?
- Load-test to learn your real headroom, size requests from what you measured, merge copies that don't need to be separate, and move work that can wait to workers that scale to zero.
- Can one service serve dev, staging and production?
- It can if the service is stateless, keeping no data between requests, so one environment's data never mixes with another's. A bad release then reaches every environment, so it needs a careful rollout with a canary and a rollback plan.