What is CI/CD? Choosing pipelines that don't waste time

CI, continuous delivery and continuous deployment in plain words, how to shape a pipeline, and why most pipeline time goes on moving data, not compiling.

Every change to a product raises the same question: is this safe to put in front of users? CI/CD is the habit of letting a machine answer most of that question, every time, in the same way.

Here is what the terms mean, how to pick the shape of your pipeline, and where pipelines quietly waste time and money. The examples come from BHyve, where we build and ship a monorepo of 14 apps.

CI, continuous delivery and continuous deployment are three different promises

Continuous integration (CI) means everyone merges small changes into the shared code often, and every change is built and tested automatically. The promise: the main branch always works.

Continuous delivery goes one step further. Every change that passes is packaged and ready to release. A person still decides when it goes to production. The promise: you could release right now.

Continuous deployment removes that last decision. Every change that passes goes to production on its own. The promise: if it’s merged and green, it’s live.

A restaurant kitchen is a fair picture. CI is tasting each part of the dish as it’s made. Continuous delivery is every plate finished and waiting at the pass for the head chef to send it out. Continuous deployment is plates leaving the moment they’re ready.

People say “CI/CD” for the whole thing. The CD half can mean delivery or deployment, so ask which one someone means.

A pipeline is four stages

A pipeline is the list of automated steps a change goes through between a commit and running software. Most have the same four stages.

1. Build. Turn source code into something that runs: install dependencies, compile, bundle.
2. Test. Run the checks: unit tests, type checks, linting, maybe a few end-to-end tests. If anything fails, the pipeline stops here.
3. Package. Wrap the result into one artifact, the thing you will deploy. Today that’s usually a container image: your app plus everything it needs to run, so it behaves the same on any machine. Images live in a registry, a server that stores them by name and version.
4. Deploy. Put that artifact into an environment and start it.

The steps run in jobs. A job is a group of steps that runs on one runner, the machine that does the work. On GitHub Actions’ hosted runners, each job gets a fresh virtual machine. That keeps every run clean, but it also means every job starts from nothing.

Dev, staging and production each do one job

An environment is one running copy of your product, with its own servers, settings and data. Most teams have three.

• Dev is where work in progress lands. It breaks sometimes, and that’s fine.
• Staging is as close to production as you can afford. It’s the last check before real users see a change.
• Production is what customers use.

When you need staging

You need staging when a mistake in production is expensive: paying customers, data migrations you can’t undo, integrations you can’t fake on a laptop, or a customer who asks how changes are tested before release. You can often skip it for a side project, or an internal tool with a handful of users and a rollback you’ve actually practised.

Three choices shape your pipeline

One repo per app, or one monorepo

A monorepo is one repository that holds many apps and the code they share.

With one repo per app, each pipeline is simple because each repo builds one thing. The cost shows up elsewhere. A feature that touches three services means three pull requests and three deploys, and shared code has to be published and versioned like a package.

That’s where BHyve started: one frontend talking to twelve backend services, each in its own repo with its own deploy. Every feature touched three or four repos, so we moved everything into one Nx monorepo. Nx is a build tool for monorepos that knows which project depends on which. The Platform v2 case study has the details.

A monorepo moves the hard part into CI. The pipeline has to know which apps a change touches, or every change rebuilds everything. Ours didn’t at first: one library change used to rebuild six apps.

A rule of thumb: separate repos are fine when apps rarely change together. A monorepo pays off when they often do, as long as CI builds only what changed.

Branch-based or tag-based releases

In a branch-based setup, merging into a branch deploys to the matching environment. It’s easy to follow. The trouble starts at production. A separate production branch means extra merges, and a merge can carry changes that never ran in staging.

A tag is a name pinned to one exact commit, like v1.4.0. In a tag-based setup, branches still deploy to dev and staging, and creating a tag is what releases to production.

At BHyve, a merge deploys to dev, the main branch goes to staging, and a signed tag promotes the exact image staging ran. Production used to have its own branch and a manual promotion. Why we switched is in release by tag.

Build once, then promote

Whichever you pick, build each version once. The image that passed staging should be the image that runs in production. If you rebuild for production, you ship something nobody tested, because a dependency or the base image may have moved since the staging build.

Promotion is cheap. A registry identifies every image by a digest, a fingerprint of its exact contents. Promoting means pointing production at the same digest, not building again.

Most pipeline time goes on moving data, not compiling

Pipeline time costs you twice: once on the bill, and again while engineers wait for feedback.

GitHub Actions bills hosted runners per minute, at a rate that depends on the runner, and rounds each job up to the next whole minute. Standard runners on public repositories are free. Two things follow. Ten short jobs cost more than one job doing the same work. And anything every job repeats, like checking out code, installing dependencies or pulling an image, is paid for again in each one.

Measure before you optimise

You’d expect compiling to be the slow part. When I measured our CI run by run, before changing anything, compiling was a small part of it. Most of the time went on moving data: saving and restoring caches, pulling a heavy base image, and starting a fresh machine for every app.

Every fix below follows from that. Before you change anything in your own pipeline, read the step timings on a few real runs and see which pile is bigger: the steps that compute, or the steps that only move files around.

Cache things by how often they change

A cache keeps files from one run, like installed dependencies or built code, so the next run can skip the work that made them. Each entry is saved under a key. If the next run finds the same key, it restores the files. If the key has changed, it does the work again and saves a new entry. On GitHub Actions an entry can’t be changed once it’s saved, so new contents always need a new key.

A good key changes exactly when the contents would. Dependency caches are usually keyed on a hash of the lockfile, the file that pins every package to an exact version. The key then changes only when someone changes a dependency, and most runs get a hit.

The trap is putting things that change at different speeds into one cache. Ours held third-party packages and our own built code together. Packages change rarely, and our code changes on every commit. One cache has one key, so the key changed on every commit and the whole cache went stale each time, packages included. Every run uploaded a fresh full-sized entry, and the cache kept overflowing its limit. Each GitHub repository gets 10 GB of Actions cache by default, and above that GitHub evicts the least recently used entries. The ones being thrown away were the ones we wanted to reuse.

The fix is to split the cache by how often things change. Dependencies get a cache of their own, refreshed only when dependencies change. The shared code most apps use is built once, cached and reused by every app instead of being compiled again for each one. The full reasoning is in how we cut CI minutes by 71%.

Build only what changed

Monorepo tools can work out which projects a change affects: the projects whose files changed, plus every project that depends on them. Nx calls these “affected” projects. Build those and leave the rest alone.

In our pipeline, one planning step works out which apps a change touches and starts only as many build machines as that needs. Zero is a real answer: if no app changed, nothing builds.

Use fewer runners

One runner per app sounds tidy. It also means each app pays for its own fresh machine, its own install and its own image pull, and each rounds up to its own whole minute. A machine per app was one of the places our time actually went. Giving a few machines a list of apps each pays the setup a few times instead of once per app.

Cancel runs nobody will read

If you push twice in a minute, the first run is already out of date. Unless you say otherwise, both run to the end and both are billed. Our pipeline now cancels any run that a newer push replaces.

GitHub Actions does this with a concurrency group. A minimal example:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Runs of the same workflow on the same branch share a group, and a new run cancels the one in progress. For deploy jobs you may want to leave cancel-in-progress off, so a deploy is never stopped halfway.

Keep images small

A base image is the image your app’s image is built on top of. Every fresh runner downloads it before it can build, and every server that runs a new version of an app pulls its image before the pod can start. So image size is paid on every build and on every deploy.

The quickest way to shrink an image is to remove what nothing uses. A dependency that no code imports still gets installed, cached and copied into the image. Removing ours made our base image 25% smaller. Smaller images pull faster, so pods come up sooner. Shrinking Docker images has the method.

What it added up to

At BHyve, these changes took a one-app deploy from 21 billed minutes to 6, and pushing all ten apps from 69 to between 21 and 28. A one-app deploy now takes about 5 minutes end to end, so the team hears back about a change in minutes.

A checklist for your pipeline

• Decide what CD means for you: a person releases, or the pipeline does.
• Build once. Promote the same image from staging to production.
• Keep a staging environment if a bad release would hurt a customer.
• Measure step timings on real runs before you optimise anything.
• Cache things by how often they change. Key dependency caches on the lockfile, and keep built code out of them.
• In a monorepo, build only the affected projects, and build shared code once.
• Put work on fewer runners. Every job repeats its setup and rounds up.
• Cancel superseded runs with a concurrency group.
• Remove dependencies nothing uses. Every runner and every server pulls them.

Quick answers

What is the difference between continuous delivery and continuous deployment?

With continuous delivery, every change that passes the pipeline is ready to release, but a person decides when it goes to production. With continuous deployment, every passing change goes to production automatically.

What are the stages of a CI/CD pipeline?

Most pipelines have four: build the code, test it, package it into an artifact such as a container image, and deploy that artifact to an environment like staging or production.

How are GitHub Actions minutes billed?

GitHub bills hosted runners per minute, at a rate that depends on the runner type, and rounds each job's time up to the nearest whole minute. Standard hosted runners are free for public repositories.

How do I cancel outdated GitHub Actions runs?

Add a concurrency group to the workflow, for example keyed on the workflow name and branch, and set cancel-in-progress to true. A new run in the same group then cancels the one still running.

Do I need a staging environment?

You need one when a bad release is expensive, for example with paying customers, data migrations or integrations you can't test locally. Small internal tools with a fast, practised rollback can often skip it.