Why a read-only container turned an attack into a patch

Attackers aimed a critical Next.js flaw at our website. A read-only, non-root container stopped the payload, so the fix could be a small, careful patch.

Attackers were using a critical Next.js flaw against our production website at BHyve to try to install a crypto-miner. The payload didn’t land. The container the site ran in had a read-only filesystem and ran as a non-root user, and that had already stopped it. So the job was a patch, not a breach.

The problem: a flaw being used against us

The flaw was in React Server Components, the part of React that renders on the server. It was rated CVSS 10, the top of the scale used to score how bad a vulnerability is, and it was being actively exploited. A flaw like this lets a crafted request run the attacker’s code on the server.

The attempts were trying to install a crypto-miner: a program that uses someone else’s machine to mine cryptocurrency for the attacker.

Why the payload didn’t land

Two settings explain it.

A container is an app packed with the files it needs, running as an isolated process on a shared machine. Its root filesystem is those files: the operating system libraries, the runtime and your code. A read-only root filesystem means the container can read its files but can’t change them or add new ones. If the app needs scratch space, you give it one small writable folder and nothing else.

Non-root means the process runs as an ordinary user, not as root, the all-powerful admin account. This matters more than people expect. By default, root inside a container has the same user ID as root on the host machine, with only the container’s isolation in between.

A typical “install a miner” payload has to do a few things in order: download a program, save it to disk, make it executable and start it. A read-only filesystem breaks the “save it” step. There’s nowhere to put the file. Running as non-root closes the usual workarounds, like installing packages or writing into system folders. Each step the attacker needs is a step that fails.

That’s also the limit of it. A read-only container doesn’t fix the flaw. Code can still run in memory, and it can still reach whatever the app itself can reach. What the container takes away is the easy next step. That buys time, and the patch still has to happen.

The principle: plan for the outer layer to fail

You can’t predict which flaw comes next. You can decide, ahead of time, what an attacker’s code is allowed to do once it runs. That’s defence in depth: layers, so that when one fails the next still holds. Here the outer layer, the framework, had a hole. The layer underneath, the container, held.

The second half of the principle is about the fix. When a layer below has bought you time, use it to make the smallest change that closes the hole, not the biggest one.

Patch inside the major version

Version numbers usually have three parts: major, minor and patch. A patch release fixes bugs without changing how things work. A major release is allowed to break things.

Because the payload had already been stopped, I could choose the safest fix rather than the fastest-looking one. I patched Next.js and React within the same major version, to avoid a risky upgrade mid-incident. A major upgrade means new behaviour, changed APIs and a round of testing, which is a lot to take on with exploit attempts arriving. The patch release carried the fix and very little else.

Major upgrades are their own project, best done when nothing is on fire. There’s more on that in upgrading Next.js for security.

How to harden a container

1. Run as non-root, and make the platform enforce it rather than trusting the image.
2. Make the root filesystem read-only, with a few small, specific writable folders.
3. Drop capabilities and block privilege escalation, so a process can’t gain powers its parent didn’t have.
4. Filter system calls with the runtime’s default seccomp profile.
5. When a flaw is being exploited, ship the smallest patch that fixes it. Plan the big upgrade after.

In Kubernetes, the system that runs and restarts containers across a group of machines, the first four live in a container’s securityContext. If Kubernetes is new to you, start with Kubernetes in plain words. A minimal general example:

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault

runAsNonRoot makes Kubernetes refuse to start the container if it would run as root. Dropping ALL capabilities removes the special Linux powers a process can otherwise hold. For a folder the app must write to, such as /tmp, mount an emptyDir volume: an empty scratch folder that lasts as long as the pod. Give it a size limit, and keep mounts like that few.

Kubernetes has a built-in strict profile called Restricted. It requires non-root, no privilege escalation, dropped capabilities and a seccomp profile. It doesn’t require a read-only filesystem. That one you have to ask for.

What it changed

The production website was patched while the exploit attempts were still coming in. The payload didn’t land, so the incident stayed a patch instead of a breach, and the fix could be a small one chosen for safety instead of a rushed upgrade.

The lesson

Defence in depth is boring until the day it isn’t. Do the boring part before you need it.

The rest of the security work is in the security case study.

Quick answers

What is a read-only root filesystem in a container?

It means the container's own files, from the operating system libraries to the app code, can't be changed or added to while it runs. Anything that must write gets a small, separate volume.

Why run containers as non-root?

A process running as an ordinary user can't install software or change system files. If an attacker gets code running inside the container, they inherit those limits.

How do you make a Kubernetes container read-only?

Set readOnlyRootFilesystem: true in the container's securityContext, and mount an emptyDir volume for any folder the app must write to, such as /tmp.

Does the Kubernetes Restricted pod security profile require a read-only filesystem?

No. Restricted requires non-root, no privilege escalation, dropped capabilities and a seccomp profile. readOnlyRootFilesystem is a setting you have to add yourself.

Does a read-only container replace patching?

No. It limits what an attacker can do once their code runs, but the flaw is still there. It buys time to patch carefully instead of in a panic.