Our platform was built for thousands of users and was serving 30 to 100 a day. I went through it service by service and cut what it reserved, while keeping live answers fast.
Three copies of the AI service (dev, staging, prod), 4 to 11 pods, each with its own secrets and its own paid embedding API.
One stateless AI service on a single cluster serving all three environments, with embeddings computed in-process and no per-token fee.
The problem
Cloud spend at an early-stage SaaS is a product problem, not just an ops one. Our target is for infrastructure to cost no more than about a fifth of what a user pays, and every service had been sized for a load we didn’t have yet.
The AI service was the clearest case. It ran as three full deployments, one per environment, each autoscaling on its own and each calling a paid embedding API. A legacy extraction service still carried PyTorch, a headless browser and ten routes nobody called.
The idea
Stop paying for capacity we only need occasionally. Make the AI service stateless so one instance can safely serve every environment, move heavy indexing to the hours when nobody is asking questions, and shrink everything that ships to a node.
How I did it
- Built a capacity and pricing model with a load test: about 2,000 simulated users at p95 601 ms with 0% errors, and every pod under 10% CPU. That told me exactly how much room we had.
- Right-sized the extraction service twice (Jan and Jun 2026): cut its requests, removed PyTorch, transformers, Playwright and Chromium from the image, deleted ten unused routes and 825 lines.
- Made the AI service compute-only (extract, chunk, embed), moved vector writes and search into the platform, and wrote a cutover runbook with strict ordering, rollback and a canary.
- Moved embeddings in-process after an A/B showed a local 256-dimension model matched the paid API on accuracy: 91.4% Acc@1 for both, 97.1% with the reranker.
- Parked bulk imports and files over 8 MiB on a 22:00–07:00 queue, drained by KEDA-scaled night pods that don’t exist by day. Measured first: scaling out to 2-core pods beat a 16-core pod, which cost 5.9× more.
- Added GPU embedding on Modal for the night only: 4,018 chunks a minute against 147 on CPU, on one card instead of four, with a hard monthly budget of $10–30 and automatic fallback to CPU.
- Fixed a hidden CPU-throttling bug (63% of CPU periods throttled, down to 0.4%), and rebuilt the Docker images after a node ran out of disk.
The impact
The idle AI footprint went from 4 pods and 1.4 vCPU to 1 pod and 0.5 vCPU, and its worst-case burst from 8 vCPU to 2. The extraction service now requests 750m CPU and 1.5 GiB across three environments instead of 2.2 vCPU and 8.8 GiB. A five-app deploy pulls about 0.55 GB of images instead of 6 GB.
Live answers stayed fast, because heavy work no longer shares a pod with them. Processing a document of about 60 chunks went from 192 seconds to about 14. Memory per AI pod went up, from 1.5 to 2 GiB, because the embedding model now lives inside it, and that was the right trade.
The cheapest compute is the compute that isn’t running. Most of the savings came from asking when a workload actually needs to exist, not from squeezing it.