How we cut CI minutes by 71%: cache by how often things change

Our CI time went on moving data, not compiling. Why a cache that mixes packages with built code goes stale on every commit, and how we fixed it.

A one-app deploy at BHyve used to bill 21 minutes of CI. It now bills 6. Almost none of that came from compiling faster, because compiling was never where the time went. It went on moving data around, and the biggest part of that traced back to how we cached.

The problem: every push rebuilt everything

Our monorepo had grown to 14 apps, and CI treated every push the same way: rebuild, re-cache and redeploy as if everything had changed. A small change to one shared library rebuilt six apps. Builds were slow and expensive. (Why everything lives in one repo is its own post.)

We measured before changing anything

The natural guess is that a slow pipeline means a slow compiler. Before changing anything, I measured where the minutes actually went, run by run. Every GitHub Actions run keeps the timing of each job and each step, and every number in the study was tied to a run ID, so anyone could open the run and check it.

Compiling turned out to be a small part of it. Most of the time went on moving data:

• saving and restoring caches,
• pulling a heavy base image,
• starting a fresh machine for every app.

None of this shows up if you only time the build command. On GitHub’s hosted runners, every job starts on a fresh virtual machine with nothing on it. Before it can build, it checks out the code, restores caches and downloads images. After it builds, it uploads the cache again. GitHub bills by the minute and rounds every job up to the next whole minute, so each extra machine pays for all of that again.

If you want to check your own pipeline the same way:

1. Pick a handful of recent, typical runs and note their run IDs.
2. Split each job’s time by step: setup, install, cache restore, build, test, cache save, image pull and push.
3. Add up the steps that compute something and the steps that only move data. If the second pile is bigger, the compiler isn’t your problem.

The cause: one cache for two things that change at different speeds

The cache was the clearest problem. To see why, it helps to know how a CI cache decides what to reuse.

A cache entry is a bundle of files saved under a key. At the start of a run, CI looks for an entry with the right key. If it finds one, it restores the files and skips the work that made them. If not, it does the work and saves a new entry at the end. On GitHub Actions an entry can’t be changed once it’s saved, so new contents always mean a new key.

So everything depends on the key. A good key changes exactly when the cached files would change, and not otherwise.

Dependencies are the easy case. A dependency cache is usually keyed on a hash of the lockfile, the file that records the exact version of every installed package, such as package-lock.json. The lockfile only changes when someone adds, upgrades or removes a package. So the key can stay the same for weeks, and almost every run restores the cache untouched.

Our own built code is the opposite. It changes on every commit, so any key for it has to change on every commit too.

Our cache held both, as one thing. One entry has one key, and that key has to change when either part changes, or a run would restore stale built code. So it changed on every commit. Every commit made the whole cache stale, packages included, even though the packages hadn’t moved. Each run uploaded and downloaded far more than it needed.

It got worse from there. Every commit added another full-sized entry. Each repository gets 10 GB of cache by default, and above that GitHub evicts the entries used least recently. Our cache kept overflowing its limit and throwing away the very entries we wanted to reuse. Runs were paying to save a cache that the store was busy deleting.

The principle: cache things by how often they change

A cache is only as stable as the fastest-changing thing in it. So split it by how often things change, and give each part a key built only from its own inputs. In a monorepo that usually means three groups:

• Third-party packages. They change when the lockfile changes, which is rarely.
• Shared code. The libraries most apps depend on. They need building again only when they change, not every time an app does.
• The app you’re working on. It changes all the time, and only it needs building fresh.

A well-ordered Dockerfile works on the same idea. Docker caches each step as a layer, and when one layer changes, every layer after it is rebuilt. That’s why Dockerfiles copy the lockfile and install packages first, then copy the source. A code change rebuilds only the last layers, and the install stays cached.

The fix, step by step

1. Give dependencies their own cache. Packages got a cache of their own, refreshed only when the dependencies themselves change. Most runs restore it untouched.
2. Build the shared code once and reuse it. I worked out which code most apps have in common, the shared libraries, and cached its build separately. Every app reuses that build instead of compiling the libraries again. Monorepo tools such as Nx do this by hashing a project’s files and the files of everything it depends on, then reusing the stored output when the hash matches.
3. Build only what changed. Monorepo tools call the projects a change touches “affected”: the ones whose files changed between two commits, plus every project that depends on them. A change to one app affects that app. A change to a shared library affects every app that uses it. One planning step now works out which apps a change really touches and starts only as many machines as that needs, sometimes none. Apps that need building share those machines, so the setup is paid a few times instead of once per app.
4. Stop paying for stale work. Push twice in a minute and the first run is out of date before it finishes, but it still runs to the end and still bills. Now a newer push cancels the run it replaces. In GitHub Actions that’s a concurrency group, shown in what is CI/CD.
5. Remove what nothing uses. A dependency that no code imports still gets installed, cached and copied into images on every run. I removed ours. How to find yours is in shrinking Docker images.

The knock-on effect: smaller images, faster pods

Removing unused dependencies didn’t only speed up CI. It made the base image smaller. A base image is the image every app’s image is built on top of, so the saving repeats in two places.

Every fresh CI machine downloads the base image before it can build, so a lighter one shortens every run. And every server that runs a new version of an app has to pull its image before the pod can start. A pod is the unit Kubernetes runs: one or more containers started together. Smaller images pull faster, so pods come up sooner, rollouts finish quicker, and new copies are ready sooner when traffic rises.

What it changed

• A one-app deploy bills 6 CI minutes instead of 21, 71% fewer, and takes about 5 minutes end to end.
• Pushing all ten apps bills 21 to 28 minutes instead of 69.
• The base image is 25% smaller, and build time halved, from 542 s to 274 s.
• At current usage that’s roughly $40 to $45 a month off the Actions bill. It’s an estimate.

Billed minutes can be higher than the time you wait, because jobs that run side by side are each billed and each one rounds up.

The money is modest. The bigger gain is time: the team gets feedback on a change sooner, and every deploy has less to pull.

The lesson

A cache is only as stable as the fastest-changing thing in it. Measure first, then cache things by how often they change.

The other half of the redesign, shipping the exact image that staging tested, is in release by tag. The CI/CD case study has the whole story.

Quick answers

Why is my GitHub Actions workflow so slow?

Often it isn't compiling. Read each step's timing on a few real runs: installs, cache save and restore, image pulls and job setup usually take more time than the build itself.

What should a GitHub Actions cache key be based on?

On the inputs that decide what's in the cache, and nothing else. A dependency cache is usually keyed on a hash of the lockfile, so the key only changes when dependencies do.

Why does my CI cache miss on every run?

Usually because its key changes on every commit, often because one cache holds something that changes every commit, like built code. Split things that change at different speeds into separate caches.

What happens when the GitHub Actions cache goes over 10 GB?

Each repository gets 10 GB by default. Above the limit, GitHub evicts the entries used least recently, and it removes any entry not used for seven days. Admins can raise the limit and pay for the extra.

What are affected projects in an Nx monorepo?

The projects whose files changed between two commits, plus every project that depends on them. Building only those skips everything the change can't have touched.