Running a database on Kubernetes: what changes when a pod holds data

Kubernetes' defaults assume any pod can be thrown away. Why that breaks for a database, how we ran LanceDB with that in mind, and a checklist for your first.

Until LanceDB, nothing we ran on our cluster had to remember anything. Any pod could die and Kubernetes would start a fresh one somewhere else, and nobody noticed. A vector store is different. If its disk goes, the search index goes with it.

LanceDB became the platform’s first stateful workload when we moved AI search out of Postgres (the full story is here). The hard part wasn’t LanceDB itself. It was that the Kubernetes defaults we’d relied on so far all assume the opposite of what a database needs.

Kubernetes is built around throwing pods away

A pod is the smallest thing Kubernetes runs: one or more containers that start and stop together. Kubernetes treats pods as disposable. It stops them during deploys, moves them when a node is drained, and replaces them when they crash. If that’s new, Kubernetes in plain words covers the basics.

That works when a service is stateless. A stateless service keeps nothing between requests. Everything it needs arrives with the request or lives in a database somewhere else. Kill it, start another, and nobody notices.

A stateful service keeps data that has to survive it. A database is the obvious case. Its files are the whole point. For a stateful service, every one of those routine stops, moves and replacements is a moment where data can be lost, or written by two things at once.

So the question changes. It’s no longer “is a pod running?” It’s “is the data still there, and is only one thing writing to it?”

Four ways the usual setup puts data at risk

The container’s own disk dies with it

A container’s filesystem disappears with the container. So Kubernetes keeps storage in separate objects, with lifetimes of their own:

• A PersistentVolume (PV) is a piece of real storage, such as a cloud disk, that the cluster knows about.
• A PersistentVolumeClaim (PVC) is a request for storage: “I need this much space, writable from one node.” Kubernetes binds each claim to a volume.
• A StorageClass describes a kind of storage the cluster can create on demand: which driver makes the disk, what type it is, and what happens to it later.

The pod mounts the claim. The claim is bound to the volume. The volume is the disk. Delete the pod and the claim and volume stay where they are. Kubernetes even holds back deleting a claim while a pod is still using it.

Deleting the claim deletes the disk, by default

The real risk to your data isn’t the pod. It’s the claim. When a claim is deleted, its volume’s reclaim policy decides what happens to the disk underneath.

Volumes created through a StorageClass inherit its reclaim policy, and a StorageClass with no policy set defaults to Delete. If nobody thought about it, your database disk goes when its claim goes. That default makes sense for the stateless world it was built for, and it’s exactly wrong for a database.

Retain has a running cost too. A retained volume isn’t reused automatically. It stays Released because the old data is still on it, and you keep paying for it until someone clears it out.

A normal deploy runs two copies for a moment

A Deployment’s default rolling update starts the new pod before it stops the old one. For a stateless service that’s what keeps it available. For a service that must have exactly one writer, it means two writers for a moment, or a new pod stuck waiting for a disk the old one still holds.

The access mode doesn’t save you either. ReadWriteOnce, the most common one, means one node, not one pod. Two pods on that node can both mount the volume read-write. ReadWriteOncePod, stable since Kubernetes v1.29, is the mode that means one pod.

Keeping the disk doesn’t protect what’s on it

A disk you keep protects you from deleting it. It does nothing when the data on it goes bad: a bad write, a buggy migration, a corrupted file. The disk faithfully keeps whatever you put on it. Only a copy from before the damage helps.

The principle: decide what the data must survive, then make Kubernetes enforce it

Each of those risks has an answer, but you have to choose it. The defaults choose for a stateless service.

So start with a list of what the data has to survive: a restart, a deleted pod, a deleted claim, a lost node, a bad write. Then give each one an answer that Kubernetes enforces, not one that lives in a README. A rule only written down gets broken on a busy day.

For LanceDB we made three choices:

• One writer, by design. One process does all the writing, so there’s never a second writer to coordinate with or race on the same files.
• A disk that outlives the pod. The storage class keeps the disk if the pod is deleted.
• Snapshots as the backup. The service runs on a managed disk with daily snapshots, kept for 14 days. A snapshot is a point-in-time copy of a disk that you can restore into a new one.

A checklist for your first stateful workload

1. Write down what the data has to survive. Each item needs a different answer.
2. Set the reclaim policy on purpose. Use reclaimPolicy: Retain in the StorageClass for anything you can’t rebuild. The default is Delete.
3. Create the disk where the pod can run. volumeBindingMode: WaitForFirstConsumer waits until a pod that uses the claim is scheduled before making the disk, so the two end up in the same place.
4. Allow the disk to grow. allowVolumeExpansion: true lets you grow a volume later by editing the claim. You can grow a volume this way, but never shrink it.
5. Make Kubernetes enforce the number of writers. A StatefulSet, ReadWriteOncePod, or a Deployment with the Recreate strategy, which stops the old pod before starting the new one. A StatefulSet also promises that at most one pod with a given identity runs at a time.
6. Take snapshots on a schedule and restore one before you need it. Kubernetes has a VolumeSnapshot API that takes a snapshot when asked. A daily schedule and an expiry have to come from your cloud’s snapshot policy or a backup tool. Until you’ve restored one, you don’t really know it works.
7. Load-test at realistic sizes and find out which number grows: memory, disk or query time. That’s the number to plan around.
8. Alert on disk usage well before the disk is full.
9. Clear out Released volumes from time to time. With Retain, nobody else will.

What it changed

We load-tested it at realistic sizes before trusting it with production. Vector search took 5.3 ms at 100k rows and 6.2 ms at 500k. Memory stayed flat. Only the disk grew.

That told us where to look. If memory doesn’t grow with the data, the pod’s memory request can stay where it is, and disk is the one number to plan around. At today’s row size the production disk has room for roughly 13–20 million chunks, the short passages of documents that search works on. That’s a known point to plan the next resize around rather than a surprise.

Every production tenant now runs AI search on LanceDB, and query time no longer depends on how many documents a company has. How we moved AI search from Postgres to LanceDB covers the migration.

The lesson

Running a database on Kubernetes isn’t hard. It means undoing defaults you never had to think about while every pod was disposable.

Quick answers

Can you run a database on Kubernetes?

Yes, if the data lives on a PersistentVolume that outlives the pod, the reclaim policy is chosen on purpose, the number of writers is enforced, and there are snapshots you have tested restoring.

Does deleting a pod delete its PersistentVolume?

No. Deleting a pod leaves its PersistentVolumeClaim and the volume behind it in place. The volume's reclaim policy only applies when the claim itself is deleted.

What is the difference between the Retain and Delete reclaim policies in Kubernetes?

With Delete, removing the claim also deletes the volume and the real disk. With Retain, the volume and its data are kept, marked Released, and someone has to clean it up or reuse it by hand.

What is the default reclaim policy for a Kubernetes StorageClass?

Delete. If a StorageClass doesn't set reclaimPolicy, volumes it creates are deleted along with their claim.

What does ReadWriteOnce mean in Kubernetes?

The volume can be mounted read-write by a single node. Several pods on that same node can still use it. ReadWriteOncePod, stable since Kubernetes v1.29, limits it to a single pod.