Why CPU throttling hides behind a calm CPU graph

A CPU limit is enforced every 100 ms, but graphs show averages. Why throttling hides, how to measure it, and how fixing it made our extraction 14× faster.

Our AI extraction was slow. A document of about 60 chunks took 192 seconds to process. A chunk is a slice of a document small enough to embed and search on its own. Nothing was failing. It was just slow.

The cause was CPU throttling. In 63% of the kernel’s CPU periods, the container had already used up its allowance and was made to wait. Throttling doesn’t crash anything or log an error, and it barely shows on a CPU graph. Here’s why, how to measure it and how to fix it.

A CPU limit is a budget that resets every 100 ms

When you give a container a CPU limit in Kubernetes, the Linux kernel enforces it with CFS bandwidth control, the part of the kernel that caps how much CPU time a group of processes can use. It splits time into periods, 100 ms each by default, and gives the container a quota of CPU time for each one.

The quota is the limit times the period. A limit of 1 CPU means 100 ms of CPU time in every 100 ms. A limit of 500m means 50 ms.

The catch is that every thread in the container draws from the same quota. Say the limit is 1 CPU and the work runs on four threads across four cores. Together they spend the 100 ms quota in the first 25 ms of the period. Then the kernel stops all of them for the remaining 75 ms. That stop is throttling, and the period counts as a throttled period.

Think of a shared data plan that resets every tenth of a second. Four people on one plan use it up four times as fast, then everyone waits for the reset.

Why it hides behind a low average

Three things keep throttling out of sight.

The graph averages, the kernel doesn’t. CPU graphs show use averaged over seconds or minutes. The limit is enforced every 100 ms. A container can burn through its quota early in almost every period and still never go over its limit on average.

Waiting isn’t use. A paused thread uses no CPU, so the time spent throttled never appears on a usage graph at all. The graph shows the work. It can’t show the waiting. kubectl top has the same blind spot, because it reports usage, not throttling.

Slow looks like something else. Nothing errors. The work just takes longer, and slow work is easy to blame on the model, the network or the size of the document.

So the number that shows throttling isn’t CPU use. It’s the throttled share: out of all the periods, how many ended with the container waiting. CPU use tells you how much a container worked. The throttled share tells you how often it wanted more and was told no. Ours was 63%.

Why the limit and the work stop matching

A CPU limit isn’t wrong in itself. Throttling starts when the limit no longer fits how the work runs. The usual causes:

• Thread pools sized from the node, not the container. Many runtimes and libraries count the cores they can see on the machine and start that many threads, without looking at the container’s limit. Python’s os.cpu_count() reports the node’s cores, and Go only started setting GOMAXPROCS from the container’s CPU limit in version 1.25. Eight threads on a 2-CPU limit spend the quota in a quarter of each period.
• Work that is parallel by design. Batch processing, like embedding many chunks at once, is built to use every core it can get. A small limit cuts it off in every period.
• Code that changed after the limit was set. A limit that fit last quarter’s code may not fit this quarter’s.

How to check yours

1. List the containers that have a CPU limit. Only those can be throttled.

2. Graph the throttled share next to CPU use. If you run Prometheus with the standard container metrics from cAdvisor, the monitoring built into the kubelet, this query gives the share per container, from 0 to 1:

   sum by (namespace, pod, container) (
     rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])
   )
   /
   sum by (namespace, pod, container) (
     rate(container_cpu_cfs_periods_total{container!=""}[5m])
   )

   There’s no official threshold. Published alert rules vary, commonly from 10% to 25%. Without Prometheus, on a node with cgroup v2, the files cpu.max and cpu.stat under /sys/fs/cgroup inside the container hold the quota, the period and the same two counters.

3. Compare the work’s thread count with its limit. More busy threads than cores in the limit is the usual sign.

4. Time a real unit of work, like one document, not only CPU percent. That’s the number your users feel.

How to fix it

Each cause has a matching fix:

• Match the threads to the limit. Set the thread count explicitly so it fits the CPU you’ve given the container.
• Raise the limit to fit the work. If the work is parallel by design, give it the CPU it needs. Then compare the extra cost with the time you save.
• Drop the CPU limit and keep the request. With no limit there’s no quota, so there’s no throttling. The request still sets each container’s share of CPU when the node is busy. The trade-off is that a busy container can take spare CPU on the node, which makes performance less predictable for its neighbours. Memory limits are a different matter, and I’d keep those. Kubernetes in plain words explains the difference.

After any of them, check the throttled share again, and check it after every resize. The fix that fits today can stop fitting after the next code change.

What it changed

The throttled share fell from 63% of periods to 0.4%. The same document of about 60 chunks went from 192 seconds to about 14, which is 14× faster. A document that took over three minutes now takes seconds.

The lesson

CPU use shows how hard a container worked, not how long it waited. On any container with a CPU limit, watch the throttled share too.

It was one of the fixes in bringing our server costs down, service by service.

Quick answers

What is CPU throttling in Kubernetes?

When a container uses up its CPU limit within a scheduling period, 100 ms by default, the Linux kernel pauses it until the next period starts. That pause is CPU throttling.

Why is my pod throttled when its CPU usage looks low?

CPU graphs show averages over seconds or minutes, but the limit is enforced every 100 ms. Several busy threads can spend a period's whole quota early and wait out the rest, while the average still looks modest.

How do I check if a Kubernetes pod is CPU throttled?

In Prometheus, divide the rate of container_cpu_cfs_throttled_periods_total by the rate of container_cpu_cfs_periods_total for the container. The result is the share of periods in which it was throttled.

Should I remove CPU limits in Kubernetes?

It depends. Without a CPU limit there is no throttling, and CPU requests still set each container's share when the node is busy, but a busy container can then take spare CPU and make performance less predictable.

What is the default CFS period in Kubernetes?

100 ms. A CPU limit becomes a quota of CPU time per period, so a limit of 500m allows 50 ms of CPU time in every 100 ms.