Release by tag: ship the exact image that passed staging

Two builds of the same commit can differ. Why rebuilding for production ships something untested, and how promoting the staging image by tag avoids it.

If your pipeline builds the app again for production, production runs something nobody tested. So at BHyve, production is a tag, not a rebuild. The image that passed staging is the image that ships.

The problem: what you tested isn’t always what you ship

Production used to have its own branch and a manual promotion step. A setup like that leaves a gap between what staging tested and what production runs. A merge into a production branch can carry changes that never ran in staging. And if that merge starts a new build, production gets an image that has never run anywhere.

When production then breaks, “it passed staging” stops being evidence. You can’t be sure the thing that broke was ever in front of a test.

Why two builds of the same commit can differ

It feels as though building the same commit twice should give the same result. Often it doesn’t, because a build depends on more than your code.

• The base image. A tag like node:22 is a moving pointer. It’s updated when a new patch release comes out, so a later build can start from different files.
• System packages. A lockfile pins your app’s dependencies, but a package installed in the Dockerfile with something like apt-get install resolves to whatever is current that day.
• Build settings. Some values are baked in at build time. Next.js, for example, writes NEXT_PUBLIC_ variables into the bundle when it builds. Two pipelines with different settings make two different apps.
• The build itself. Many builds aren’t exactly reproducible. Timestamps, file ordering, or a cache hit on one run and a miss on the next can change the output.

Each of these is small. Together they mean the thing you tested and the thing you shipped can be different files. A rebuild also pays for the same CI minutes twice, which mattered to us because cutting CI minutes was the other half of the same redesign: how we cut CI minutes by 71%.

The principle: build once, then move the same bytes

The fix is to treat the image as the thing you test and the thing you release. Build it once, then promote it from one environment to the next without touching it.

That relies on how registries name images. A container image is your app packaged with everything it needs to run. A registry stores each one under a digest: a fingerprint of its exact contents, like sha256:4f1c.... Tags such as staging or v1.4.0 are names that point at a digest, and names can move. A digest can’t.

So promotion means giving the staging image a production name, or telling production to run that digest. Nothing is built. Most registries and tools can add a tag to an image that’s already stored without pulling or rebuilding it. Docker’s buildx imagetools create command is one example.

The release itself is a git tag: a name pinned to one exact commit, like v1.4.0. A signed tag also carries a cryptographic signature, so anyone can check who created it. The tag says which commit is being released and who released it. The digest says exactly what ships.

How it works for us

Each environment has a single trigger.

Everything up to staging builds and tests as normal. Production never builds anything. It only runs an image that has already run in staging.

Where GitOps fits

ArgoCD is part of our release stack, so here’s GitOps, the idea behind it, in plain words.

GitOps means the description of what should be running lives in a git repository: which apps, which image, how many copies. A tool inside the cluster keeps the cluster matching that description. Change the file in git and the cluster follows. Change the cluster by hand and the tool notices the drift and can put it back.

ArgoCD is one of those tools, built for Kubernetes, the system that runs and restarts containers across a group of machines. (New to it? Start with Kubernetes in plain words.)

Release by tag and GitOps fit together. The tag names the release. Git keeps the record of what was deployed, when and by whom. In a typical setup, promoting a release ends with a change in git that points production at the new image, and the GitOps tool rolls it out.

How to set it up

1. Build each commit once. Push the image tagged with its commit SHA, so every image traces back to exactly one commit.
2. Deploy dev and staging from those images. Branches pick an image, they don’t make one.
3. Promote by digest on a release tag. Find the image staging ran for that commit and point production at the same digest.
4. Sign release tags, and limit who can create them. A release should always have a name and a person behind it.
5. Keep environment settings out of the image. Read them at runtime, or one image can’t serve every environment.

What it changed

We’ve shipped 256 production releases since the first tag, about 1.7 a day. Each one is a signed tag, and what it ships is an image that already ran in staging.

That answers two questions about every production change without anyone having to dig: who released it, and what exactly shipped. It also changes what a rollback means. Going back is running the previous tag’s image, not rebuilding an old commit and hoping it comes out the same. And because every change to the main branch needs approval from someone other than its author, a release traces back to reviewed changes. More on that in why every change needs a second reader.

When release by tag fits

It fits when:

• You have a staging environment and want production to match it exactly.
• You want a clear record of what’s in production and who released it.
• You want a rollback to be the previous image, not a fresh build.

Think twice when:

• You practise continuous deployment and every merge goes straight to production. A tag adds a step you chose not to have.
• Your staging doesn’t resemble production. Promoting from it gives false confidence.
• Your build bakes in per-environment settings. Move those to runtime config first.

The lesson

Ship the thing you tested. A fresh build for production is a new, untested thing.

The CI/CD case study has the rest of the pipeline behind it.

Quick answers

What does build once, deploy many mean?

You build an artifact such as a container image once, test it, and deploy that same artifact to each environment instead of rebuilding it for staging or production.

Why shouldn't you rebuild a Docker image for production?

A rebuild can pick up a newer base image, different system packages or different build settings, so production would run an image that was never tested in staging.

What is the difference between a Docker image tag and a digest?

A tag is a name that can be moved to point at a different image. A digest is a sha256 fingerprint of the image's contents, so it always refers to exactly the same image.

What is GitOps?

GitOps keeps the desired state of your deployments in a git repository and uses a tool in the cluster, such as Argo CD, to keep what's running in line with what git says.

What is a signed git tag?

A tag created with git tag -s carries a cryptographic signature from the person who made it, so anyone can verify who created the release.