A GPU rented all day sits idle, and live questions barely use one. Why queued night work suits a per-second GPU, with a hard budget and a CPU fallback.
On CPU, our embedding runs at 147 chunks a minute. A chunk is a short passage cut from a document, and every one needs an embedding before it can be searched. That rate is fine for one document. For a bulk import of thousands, it’s a long wait.
The obvious answer is a GPU. The obvious problem is that a GPU rented all day costs the same whether it’s working or not, and big batches only come now and then. What made a GPU worth it wasn’t the GPU. It was moving the heavy work onto a queue first.
Why embedding is slow on a CPU
An embedding is a list of numbers that captures what a piece of text means, so passages about similar things get similar numbers. It comes from a neural network, and running one is mostly a single kind of arithmetic: multiplying large grids of numbers together, over and over.
A CPU has a small number of powerful cores, built to handle varied work quickly, one step after another. A GPU has thousands of simpler cores, built to do the same arithmetic on many numbers at the same time. Multiplying grids of numbers is exactly that kind of work.
The gap grows with batching. Instead of one chunk at a time, you send many together and the GPU works through them in one pass. The more chunks in a batch, the more of those cores do useful work.
Why this work fits a queue, and a queue fits a GPU
That last point decides where a GPU belongs.
Live questions are the wrong shape. A single short question gives a GPU very little to do in parallel. Sending it to a GPU in another cloud adds a network round trip that can cost more time than the GPU saves. So live embedding stays on CPU, inside our AI service.
Bulk work can wait. Bulk imports and large files need to be searchable by morning, not in seconds. So they wait on a night queue, drained by pods that KEDA, a Kubernetes autoscaler, brings up at night and takes back to zero by day. What is KEDA? explains how.
A queue turns a trickle into a batch. Once heavy work waits on a queue, it arrives at the GPU as thousands of chunks at once, none of them in a hurry. That’s close to ideal for a GPU.
A burst should be paid for as a burst. A GPU rented by the hour costs the same whether it’s working or not. A serverless GPU turns that around: you hand the provider a function and its inputs, it starts a container on a GPU, runs the work, bills you for the time it ran and shuts the container down. Modal, which we use, is one such platform. You write ordinary Python functions, say what hardware each one needs, and pay by the second.
The principle: decide when work runs before deciding how fast
The order matters. First move the work that can wait out of the day, so nothing has to stay running for it. Then make the night shorter. A GPU on a queue is a cheap speed-up. A GPU kept around for live traffic that barely uses it is a standing cost.
And keep it a speed-up. The work has to finish without it.
How to set it up
- Queue the work that can wait first. The GPU only pays off once heavy work arrives in batches.
- Keep a CPU path that works on its own. The GPU is optional. If it can’t run, for whatever reason, the work still finishes on CPU, just more slowly.
- Use the same embedding model on both paths. Vectors from two different models don’t live in the same space, and comparing them returns nonsense.
- Send large batches, and load the model once per container. Loading the model for every batch wastes the time you came for.
- Put a hard cap on spend, and decide what happens at the cap before you hit it. Modal lets you set a workspace budget. You can also count spend in your own code and stop sending work at a limit.
- Fall back to CPU automatically. Nobody should have to wake up because the GPU wasn’t there.
- Check cold starts and idle time. A new container loads the model before it does anything useful, and by default stays up for a short while after its last input. Find out how your provider bills both. They’re small next to a long batch, and large next to a trickle of tiny jobs.
What it changed
On a GPU, embedding runs at 4,018 chunks a minute, against 147 on CPU, on one card instead of four. Its spend has a hard monthly budget of $10–30, and if the GPU isn’t available the work falls back to CPU and still finishes.
When a night-only GPU doesn’t make sense
- Each request is small and someone is waiting for the answer.
- The GPU would be busy most of the day. Per-second billing is a good deal for bursts. For steady load all day, a GPU you reserve may well cost less.
- Getting a speed-up would mean changing your embedding model.
The lesson
A GPU is cheapest when the work comes to it in batches. Put the work on a queue first, and the GPU only has to exist while the queue does.
The server costs case study has the rest of what we cut, and Cloud cost is a product problem is why we bothered.
Quick answers
- Why are embeddings faster on a GPU?
- An embedding model is a neural network, and running one is mostly large matrix multiplications. A GPU has thousands of cores that do that arithmetic in parallel, especially when many chunks are sent in one batch.
- Do you need a GPU to compute embeddings?
- No. Embedding models can run on a CPU, just more slowly. A GPU pays off when there's a large batch of text to process.
- What is a serverless GPU?
- A GPU you use per job instead of renting one full time. The provider starts a container on a GPU when work arrives, bills for the time it runs, and shuts it down when there's nothing left to do.
- What is Modal?
- Modal is a serverless cloud platform for running Python code, including on GPUs. It runs your functions in containers that scale with demand and down to zero, and bills compute by the second.
- When is a serverless GPU cheaper than a reserved one?
- When the work comes in bursts. Per-second billing suits a batch that runs and then stops. For load that keeps a GPU busy most of the day, a reserved GPU often costs less.