How we sped up our local CI without dropping a check

Coding agents write and merge most AnyRouter changes. An agent pushes, waits for CI, fixes what failed, and pushes again. When writing code gets fast and CI does not, CI is the thing everyone waits on.

Our CI does not run on GitHub. We turned GitHub Actions CI off to save money, and every check runs as pnpm local-ci on one machine with 16 cores and 15 GB of memory. Many agents share that machine. The first post in this series describes that loop.

This post started with Owen Gretzinger's How I made our CI 20x faster. In one line, Owen audited where CI time went and fixed each cause in turn, such as slow runners, a slow linter, cache misses and test isolation. Some of the post's ideas fit our setup and some did not. We say which below.

Before changing anything, we read the 50 most recent local CI logs, from 23 to 26 October. We split each run into three parts.

• Queue. Time spent waiting for the machine or a lock. The mean was 0 seconds.
• Setup. Time to prepare a clean copy of the repo and its dependencies. The mean was 52 seconds.
• Run. Time inside the checks. The mean was 326 seconds, the median was 20 seconds, and the longest run took about 2,900 seconds.

The gap between the mean and the median was the most useful number. Half of the runs finished their checks in 20 seconds. A few runs that typechecked about 20 apps one after another pulled the mean up. So the problem was not every run. It was the heavy tail, and the heavy tail was typecheck.

Other things we expected to be slow were not. Biome lint took about 1 second. Only 1 of the 50 runs ran the full test suite, because local CI runs only the tests related to the changed files. Waiting for the machine cost nothing. The limit was memory, not cores.

The logs also showed some smaller costs.

• The image build step missed its cache on 19 of 21 runs. Each run deletes untracked files to start clean, and that deleted about 1,200 generated social card images. So every run drew them again.
• A change to the lockfile creates a new working copy for CI with no TypeScript build state. The logs had 11 lockfile keys in 6 days, and a cold start for every typecheck program took about 5 minutes.
• The test setup file loaded Testing Library and React DOM into every test file, even the ones that do not use a DOM. That was 38 to 42 percent of the time for a small batch of node tests.

Then we found the biggest problem, and it came from a fix. On 1 November we changed the root pnpm typecheck so that it checks every package and app. Before that, type errors in some test files could reach main. The change was right. But local CI still ran its own typecheck for the API, the server core, the UI and each app after the root step. That made 29 TypeScript programs per run, about 26 of them checked twice, all one after another. A warm run took 162 seconds.

Nothing failed, so nothing told us. A duplicated check is green and slow. Now, when local CI plans the root typecheck, it plans exactly the set that pnpm typecheck covers and runs each program once. It reads that set from each package's package.json, so a new app is picked up without a hand-written list.

The first post in this series said local CI ran its checks one at a time because the machine has 15 GB of memory and cannot take parallel builds. That was true for a fixed number of jobs. On this machine, three cold TypeScript programs started together were killed by the out-of-memory handler at about 1.5 GB free.

So we did not pick a number of jobs. A new script starts the largest program first. It starts another program only when that program's peak memory fits in the free memory, after it subtracts the growth still ahead of the running programs and keeps 2 GB in reserve. It learns each program's peak from past runs, and assumes 2.8 GB for a program it has not seen.

A program that is killed with no type error reruns once, alone. If it is killed again, the run is reported as out of memory, not as a failed check. An agent that sees a failed typecheck goes looking for a type error. An agent that sees out of memory knows to wait and retry.

Warm runs kept at least 2.5 GB free. In the cold parallel run, no program was killed.

• The 15 cheap guards, such as schema drift and app import boundaries, now run 4 at a time. They only read files, so they are safe to run side by side. Heavy checks and test runs still take their own lock.
• One guard parsed every TypeScript file to find issue references. It now skips files that contain no # followed by digits.
• A new CI working copy starts with the TypeScript build state and caches of the most recently used copy. TypeScript rechecks any file that changed, and we confirmed that an injected type error still fails on the copied state.
• Rendered social cards are stored in a cache outside the files that the clean step deletes, keyed by path and source. Cards unused for 14 days are removed.
• Testing Library loads only in tests that have a DOM.
• A pre-push hook runs Biome on the changed files and reports errors only. An agent hears about a lint error before it pushes, not after CI runs.

These are the numbers, each measured before and after on the same commit. Each row says how many runs it took.

Running each program once, still one at a time, took the warm typecheck from 162 to 111 seconds. Running them side by side took it to 79. So about 60 percent of the win came from removing the duplicate, and the rest came from parallel work.

These are per-check numbers. We have not yet measured the mean of a whole local CI run after the change. The next 50 logs will show that.

One trap is worth its own section. Local CI runs only the tests related to the changed files. A script often builds that list from a git diff, like T=$(git diff --name-only main | grep test), then runs vitest run $T.

If the diff comes back empty, for example because a git pull failed and HEAD did not move, the command becomes vitest run with no files. To Vitest, that means run the whole suite. On our machine the whole suite risks the out-of-memory handler. We stopped one such run after 600 seconds.

The guard is one line. Check the list before you run it, with [ -n "$T" ] && vitest run $T. Owen makes a similar point about Nx. A rule that skips work needs its own test, because a skip that fails open looks green or runs everything.

Owen's post lists nine changes. Here is how each one mapped to our setup.

Owen's team paid for speed. Owen's post reports a CI bill that went from $230 to $4,300 a month and calls it worth it. Our constraint is different. CI runs on a machine we already have, so we looked for time we were wasting before time we could buy.

You can give this prompt to your own coding agent. It asks for the same audit and the same rules we followed.

Audit this repo's CI and make it faster. Never drop or weaken a check.
1. Pull the last 50 CI runs. Split each job's time into queue, setup and run.
2. Report mean, median and max for each part. If the mean is far above the median, find the slow runs behind it.
3. List the 5 slowest jobs and what each one spends its time on.
4. Check the usual causes: slow runner queue, slow linter, slow typechecker, too few cores for the tests, a fresh app load per test file, cache misses or caches a clean step deletes, serial work that could run side by side, checks on code the change cannot reach, and checks that never run before push.
5. Look for the same check running twice, and for an empty file list that means "run everything".
6. Run work in parallel within a memory budget. Report a memory kill as out of memory, not a failed check.
7. Measure each change before and after on the same commit, median of 3. Never skip tests or lower thresholds to improve a number.
8. Leave out changes that cost money (paid runners, bigger machines, more shards). List them for a human with time saved and cost.
9. Open one PR with a before and after table and the commands to reproduce it.