Kubernetes in plain words: nodes, pods and how big to make them

What a cluster, node, pod and deployment are, what requests and limits really do, and why to size pods from measured load instead of expected load.

What is Kubernetes, what is a pod, and how big should one be? Here are the pieces in plain words, then the reasoning we use at BHyve to size them: measure the real load before you pick a number, and try many small pods before one big one.

Kubernetes exists because containers need a manager

A container is your program packed with everything it needs to run, so it behaves the same on a laptop and on a server. Running one container on one server is easy. Running dozens across several servers is not. Something has to decide where each one runs, restart the ones that crash, add copies when traffic grows and send every request to a copy that’s healthy.

That something is Kubernetes. You tell it what you want, for example “three copies of the API, each with half a CPU core and 512 MiB of memory”. It keeps comparing what’s running with what you asked for, and fixes the difference. If a server dies, it starts the lost copies somewhere else.

The pieces, from biggest to smallest

Cluster. The whole system: a control plane that makes the decisions, and a set of machines that do the work.

Node. One machine in the cluster. In the cloud it’s almost always a virtual machine, so if you’re wondering where the VM is in Kubernetes, this is it. Every node has a fixed amount of CPU and memory.

Pod. The smallest thing Kubernetes runs: one or more containers that share a network address and start and stop together. Most pods hold a single container. When people say “we run four pods of the API”, they mean four copies of it.

Container. Your program, built from an image. It always lives inside a pod.

Three more pieces show up in almost every config file:

Deployment. A standing order: “keep N identical pods of this running”. Kubernetes replaces pods that die, and swaps old pods for new ones gradually when you ship a new version.

Service. A stable name and address in front of a group of pods. Pods come and go, and their addresses change. The service stays put and spreads traffic across the pods that are ready.

Namespace. A named section of a cluster, used to keep groups of things apart, such as one per team or per environment. Names only have to be unique inside a namespace.

If an office building helps, here’s the whole set in one table:

Requests and limits do two different jobs

Every container can declare two numbers for CPU and for memory. They sound alike and do very different things.

A request is a reservation. The scheduler, the part of Kubernetes that picks a node for each pod, only places a pod where there’s enough unreserved CPU and memory to cover its requests. It goes by reservations, not by what’s in use. A node whose pods have reserved 3.5 of its 4 cores won’t take a pod that asks for 1 core, even if the node is mostly idle.

A limit is a ceiling that applies while the container runs. CPU and memory hit it differently:

• Over the CPU limit, the container is throttled. The kernel pauses it until its next slice of CPU time. It stays alive and gets slower. Nothing crashes, so it’s easy to miss. Why CPU throttling hides shows how to spot it.
• Over the memory limit, the container can be killed. The kernel’s out-of-memory killer stops it (an OOM kill), and Kubernetes restarts it.

Going over the memory request, as opposed to the limit, is allowed. But if the node runs short of memory, that pod is among the likely ones to be evicted. And if you set a limit without a request, Kubernetes copies the limit and uses it as the request.

The units: CPU is counted in cores. 1 is one core, which on a VM means one vCPU, and 500m (500 millicores) is half of one. Memory uses Mi and Gi, so 512Mi is 512 mebibytes.

A resources block looks like this. The values are only an example:

resources:
  requests:
    cpu: "500m"       # half a core, reserved when the pod is placed
    memory: "512Mi"   # reserved when the pod is placed
  limits:
    cpu: "1"          # above one core, the container is throttled
    memory: "1Gi"     # above 1 GiB, the container can be OOM-killed

Here’s why this matters for the bill. On most setups you pay for nodes, and nodes fill up by requests, not by use. A pod that requests 2 cores and uses 0.2 still takes 2 cores off its node. Oversized requests are seats you’ve paid for that nobody sits in.

Autoscaling adds pods when they’re busy

The Horizontal Pod Autoscaler (HPA) is built into Kubernetes. Every 15 seconds by default, it checks a metric for a deployment and changes how many pods it runs. The usual metric is CPU, measured as a percentage of each pod’s CPU request. So a pod with no CPU request gives it nothing to work with. When load drops, it waits out a five-minute window by default before removing pods, so a short lull doesn’t cause churn.

A separate cluster autoscaler can add nodes when pods don’t fit, and remove nodes that sit mostly empty.

The HPA is good at “more of the same when it’s busy”. On CPU alone it never goes below one pod, because CPU can only be measured on a pod that’s running. For “nothing at all until work arrives” or “only at night” there’s KEDA, Kubernetes Event-driven Autoscaling. It scales on things like queue length or the clock, down to zero pods. I explain it in What is KEDA?

How big should a pod be?

Over-sizing is the default, because it never breaks

Our platform was built for thousands of users and was serving 30 to 100 a day. Nothing was broken. It was just expensive, because every service reserved capacity for a load we didn’t have yet.

That’s how it usually happens. Requests get set once, early, by someone estimating future traffic. Estimates lean generous, because running out feels worse than overpaying. Then nothing ever pushes them back down. An over-sized pod works perfectly, so nobody looks at it again. The gap between what’s reserved and what’s used grows quietly, and because nodes fill up by requests, you pay for all of it.

Measure before you pick a number

The only way to close that gap is to find out what the pods really use. So before changing a single request, we load-tested the platform and built a capacity and pricing model from the result. A load test sends a large amount of simulated traffic and watches how the system copes.

About 2,000 simulated users ran against it with no errors, and every pod stayed under 10% CPU. That answered the question every request depends on: how much room do we really have? Even at that load, the pods were close to idle, so the room was far bigger than our requests assumed.

Without a measurement like that, every request in your config is a guess, and guesses tend to be generous.

Set requests from what you measured

1. Load-test the service, or watch it through a normal busy week.
2. Note each container’s typical and peak CPU and memory.
3. Set the CPU request a little above typical use. Give memory more headroom above peak, because running out of memory kills a container and running out of CPU only slows it down.
4. Check again after any big change to the code or the traffic.

That last step matters more than it looks, because requests should follow the code. A service that stops loading something heavy needs less set aside for it. When we cleared out the libraries our extraction service no longer used, we could cut what it requests by 67% for CPU and 83% for memory. Shrinking Docker images covers how we found them.

Right-sizing doesn’t only mean cutting. Our AI pods got more memory, because the embedding model now runs inside them instead of behind a paid API. That was the right trade. The goal is a request that matches what the pod needs, in either direction.

Many small pods can beat one big pod

The night workers, which process bulk imports and large files, raised a different sizing question: one big pod or several small ones? We measured both. Scaling out to 2-core pods beat a single 16-core pod, which cost 5.9× more.

Bigger isn’t automatically faster. The usual reasons small pods win:

• Many programs can’t keep 16 cores busy. A single-threaded step, or time spent waiting on the network or disk, leaves most of the cores idle while you pay for all of them.
• A 16-core pod needs a node with 16 free cores. Small pods fit into the gaps on nodes you already have.
• Small pods let you add and remove capacity in small steps, so you pay for roughly what the work needs right now.
• When one small pod crashes, you lose a small share of the work.

Big pods still make sense when one task truly needs a lot of memory or cores at once, such as a model that has to load whole. The only way to know which side your workload is on is to test both, and compare cost as well as speed.

A sizing checklist

• Load-test before you change a single number.
• Set CPU and memory requests on every container. The scheduler and the HPA both depend on them.
• Give memory headroom. Set CPU limits with care, and check for throttling if you do.
• Try several small pods before one big one, and compare cost as well as speed.
• Ask whether the workload needs to exist all day. The cheapest compute is the compute that isn’t running.
• Measure again after every big change.

The server costs case study has the rest of how we applied this.

Quick answers

What is a node in Kubernetes?

A node is one machine in a Kubernetes cluster, usually a virtual machine from a cloud provider. Kubernetes places pods on nodes and tracks how much CPU and memory each node has left to reserve.

What is the difference between a pod and a container?

A container is one packaged program. A pod is the smallest thing Kubernetes runs: one or more containers that share a network address and start and stop together.

What is the difference between requests and limits in Kubernetes?

A request is a reservation the scheduler uses to pick a node for the pod. A limit is a ceiling while the pod runs: a container over its CPU limit is throttled, and one over its memory limit can be OOM-killed.

How do I choose CPU and memory requests for a pod?

Measure first. Load-test the service or watch a normal busy week, set requests a little above typical use with extra headroom for memory, and check again after any big change.

Is it better to run many small pods or one big pod?

It depends on the work, so test both. Small pods often win, because many programs can't keep a lot of cores busy at once, and small pods fit into spare room on nodes you already have.