Get the Compiler Off the Agent's Machine

When people talk about scaling coding agents, the bottlenecks they name are the model and the token budget. The first one we hit was the computer the agents run on.

Until recently we lived by a rule nobody had written down: on one machine, stop at five or six agents. Start another and the whole machine slows to a crawl, the agents included.

The machine is not underpowered. Our agent host is a 14-core Mac mini (M4 Pro) with 64 GB of memory, and the repository is a Bazel monorepo of roughly five million lines. The problem sits deeper: our build system was designed for people, and its users are now agents.

The fix fits in one sentence: the agent's machine should only coordinate and verify. An agent's reasoning already runs on a remote model, and the last heavy job left on the local machine was compilation. We handed compilation to a remote execution pool built from machines we already owned for CI, and kept only the agent processes, build analysis and the tests that need local devices on the host. Trunk-based development, gated by impact analysis over the build graph, lets many agents keep merging into one trunk without turning it red.

Figure 1: Before, every worktree's Bazel treats the whole machine as its own. After, action execution moves to a pool shared by every client, and the host keeps only agents, build analysis and tests.

Why can five or six agents bring down a 14-core machine?

People write code serially: change a few lines, build, wait, stare at the error, change again. One build at a time, with long stretches of thinking in between. A computer that runs one build smoothly is a good development machine.

Agents work differently. Edit, build, test, read the result, edit again: that is the inner loop. The thinking happens on a remote model and costs the local CPU nothing, so all the local machine sees is one build request after another, and agents do not get tired. Nor do we run just one. Each task gets its own agent, its own git worktree and its own branch, and several tasks move forward at once.

With Bazel and worktrees, every one of those loops pays three costs:

• A Bazel server. Bazel gives each workspace path its own server process, a resident JVM that takes about 1.5 GB of memory on our host.
• An output base. Also keyed by path, it holds the worktree's execution root and every intermediate output, anywhere from a few gigabytes to a few dozen.
• A set of local actions. Compiling, linking and code generation all run locally. Our local default is --jobs=HOST_CPUS, so every Bazel server schedules as if all 14 cores were its own.

That last cost is what freezes the machine. Bazel's local resource accounting is per server, and the servers cannot see each other. Five worktrees building cold means five schedulers each packing 14 compile processes onto the same 14 cores: seventy Swift, Rust and clang processes competing for 14 cores and 64 GB of memory, none of them aware of the other four. The oversubscription is structural. Tuning --jobs only lets you pick between one slow build and several builds that freeze together.

The agents need resources too. While this post was being written, the host was running 13 Claude Code processes using about 5.7 GB of memory, before counting their MCP servers, browsers and iOS simulators. Agent tool calls are sensitive to latency, and once compile processes starve them, the throughput of the whole system collapses with them.

For scale, here is what the numbers in the rest of this post refer to:

Line counts are physical lines, blank lines and comments included, in git-tracked source files; JSON, YAML, Markdown, lockfiles and vendored third-party code are excluded.

The machine should only coordinate and verify

Break down the computation one agent uses and you find five loads with very different properties:

The heaviest load, inference, was remote from the start. Of the remaining four, only action execution is both heavy and indifferent to where it runs, and it happens to be the bulk of the local load. Move it out and the host is left with coordination, meaning agent processes and build analysis, and verification, meaning the tests that must run on local devices.

What makes it movable is Bazel's action model. Every action declares its inputs, outputs, command line and environment, and its cache key is the hash of all of them. Two properties follow. As long as the platform matches, an action produces the same result on any machine; as long as the key matches, anyone can reuse its result. The first makes remote execution work, and the second makes a remote cache work.

So the architecture has five parts, and it does not stand without any one of them:

• Bazel: hermetic, content-addressed actions are the precondition for moving work elsewhere and sharing it at all.
• Monorepo: every product lives in one graph and one cache keyspace, so whatever one agent built, the next one hits, and impact analysis is only precise on a complete graph.
• Worktrees: each agent gets its own working copy, branch and merge request, and nobody steps on anybody else.
• RBE and RBC: remote build execution hands actions to a cluster, and a remote build cache lets results flow between every worktree and every machine. Our development machines and agents use RBE; CI uses a separate remote cache.
• Trunk-based development plus impact analysis: branches stay short-lived and merge often, each merge request runs only what it affects, and trunk stays green.

Worktrees are cheap isolation only if the right things are shared

In our workflow, one agent means one worktree, one branch and one merge request. Worktrees share a single git object store, so creating one only checks files out instead of cloning again. But cheap for git is not cheap for the build. Output bases are assigned by workspace path, so to Bazel, a new worktree is a repository it has never seen: no analysis cache and no outputs. If nothing were shared, every new agent would begin with a full cold build.

The principle at this layer: working copies and execution state are private, and everything that can be addressed by content is shared globally.

The private part has to reclaim itself, or 94 worktrees fill the machine quickly:

• An idle Bazel server exits after 15 minutes (--max_idle_secs=900). With 94 worktrees on the host, only 2 Bazel servers were resident at the time of writing.
• A sweeper reclaims output bases every 15 minutes, tiered by the state of the worktree. A deleted worktree's base goes immediately, one whose commits have landed on trunk goes after 24 quiet hours, one still carrying unmerged commits goes after 72, and under disk pressure every threshold drops to 30 minutes. It only ever deletes build caches, never a worktree, so the worst outcome of a wrong call is a single rebuild. Even so, Bazel storage on this host lives on its own 4 TB volume, of which 1.5 TB is in use.

Two more rules exist only because the developers are agents.

First: one verification owner per worktree. When two Bazel commands meet on the same output base, the second one waits for the lock. For an agent in a hurry, the obvious move is to start another output base: --output_base=/tmp/build-$$. It looks like parallelism. In fact it conjures a cold output base out of nowhere, with a full rebuild, tens of gigabytes more disk, and another round of fighting every other worktree for cores.

People rarely do this. Agents do. So the rule cannot stay a best practice; it has to live where agents read and where CI checks. The repository's guide for agents says to batch targets into one Bazel invocation and let Bazel serialize the overlaps it cannot avoid. An agent-facing invariant manifest also forbids these flags in the Makefile and every script, and a violation fails CI (the background on these manifests is in Invariants Addressed to an Agent, Not to a Reviewer):

id = "build-output-ownership"
paths = ["Makefile", "**/*.mk", "**/*.sh", ...]
invariant = "Bazel output ownership is per worktree and the host policy is its single writer ..."
forbid = ["*--output_base*", "*--output_user_root*", "*--ignore_all_rc_files*"]

Second: inputs can change while a build is running. People seldom edit sources mid-build, but agents routinely keep editing while a compile is in flight. If an input changes while an action executes, the result pairs a cache key computed from the old content with outputs built from the new content, and once that lands in a host-wide disk cache or a remote cache, it poisons everyone. We run with --guard_against_concurrent_changes=full, which refuses to publish a result when any input changed during execution. Bazel 9's default, lite, only checks main-repository sources, which is not enough when dozens of worktrees share one cache.

The physical resources that really are scarce go to host-wide arbiters instead. Locally started backend services get a port offset hashed from the worktree name, so dozens of development environments run side by side without colliding. iOS simulators are admitted through a host-wide capacity lease: when too many worktrees want UI tests at once, the excess waits in a queue instead of booting more simulators.

A NativeLink pool on eight CI Mac minis we already had

Our builds are full of Apple-platform actions such as Swift compilation, resource compilation, signing and packaging. Those need Xcode, which means macOS, so the pool has to be made of Macs. We chose NativeLink: written in Rust, source-available under FSL-1.1-Apache-2.0 (each release converts to Apache-2.0 after two years), and an implementation of Bazel's Remote Execution API whose worker runs natively on macOS.

We bought no hardware. The eight Mac minis were already our GitLab macOS runners, and each now also runs a NativeLink worker as a launchd system service, lending the CI machines' idle capacity to development machines and agents.

Clients need a single opt-in Bazel config:

# rbe.bazelrc: only active with --config=rbe
common:rbe --remote_executor=grpc://<pool>:50051
# Use the executor's own CAS and Action Cache: inputs uploaded anywhere else
# would be missing where the workers read them
common:rbe --remote_cache=
common:rbe --remote_instance_name=

# Every worker advertises these; cpu_count is the share of a worker one action reserves
common:rbe --remote_default_exec_properties=OSFamily=darwin
common:rbe --remote_default_exec_properties=cpu_arch=aarch64
common:rbe --remote_default_exec_properties=cpu_count=2

# Only what the requested targets produce comes back; intermediates stay in the pool
common:rbe --remote_download_toplevel
# The pool caches what its workers built; results built locally stay local
common:rbe --remote_upload_local_results=false
common:rbe --remote_timeout=180s

# Parallelism is bounded by the pool, not by local cores; past capacity the scheduler queues
common:rbe --jobs=64

# Tests still execute locally: iOS tests need local simulators, integration tests local services
common:rbe --strategy=TestRunner=sandboxed,local

The --jobs=64 line answers the oversubscription problem from earlier: concurrency is queued by one scheduler that sees every client, instead of each worktree assuming it owns the whole machine.

A few decisions are less obvious:

1. Opt in per invocation, and probe first. Remote execution has no local fallback, so when the executor is unreachable the build simply fails. That is why it is not the default. The guide for agents says to probe the pool with nc -z first, and to drop --config=rbe and build locally when it does not answer.
2. Tests stay local. Builds go to the pool and tests run on the host, because iOS tests need its simulators and capacity lease, and integration tests need services started on it.
3. Download top-level outputs only. --remote_download_minimal downloads nothing at all and only suits checks that ask whether something compiles. The top-level outputs of 16 apps come to about 2 GB, but one product downloads in 1–3 seconds over gigabit Ethernet, so risking a later step failing on a missing file is not worth it.
4. The worker injects no environment variables into actions. The DEVELOPER_DIR and SDKROOT an Apple action needs are derived by the worker from variables the action already carries in its cache key, such as XCODE_VERSION_OVERRIDE, by the same rule Bazel applies when it executes locally. A worker without the matching Xcode fails the action rather than building it with a different Xcode and caching the result. We deleted the worker's hard-coded DEVELOPER_DIR, PATH and DYLD_LIBRARY_PATH. None of them was part of any cache key, and DYLD_LIBRARY_PATH had remote rustc loading a different libLLVM all along.
5. A worker's local cache keeps only the inputs it pulled. All eight workers' local CAS stores once sat at 97–99% of their cap. Analysis showed that 122 GB of the pool's roughly 250 GB were blobs present on a single machine, nearly half of them large outputs the workers had produced themselves, while the hot inputs all eight needed came to 1.3 GB. Now a worker's local CAS caches only the inputs it fetched, and outputs are written only to the central CAS.

The results, measured between 2026-10-23 and 11-09 ("forced" means every cache was bypassed so each action really executed on the pool):

For agents, the first row matters most: a worktree is warm the moment it exists. The local disk cache only holds what this one machine has built, while the pool's CAS holds the union of everything every machine and worktree on RBE has built.

One inode, several names

iOS and Rust moved onto the pool first. Android would not follow. Its whole JVM toolchain, from desugaring, resource compilation and merging, and R class generation to dex merging and Kotlin compilation, failed intermittently on the pool and never locally. The errors were all over the place:

FileNotFoundException: …/nativelink/work/<another operation>/…/lib/tzdb.dat
Error loading java.security file
VerifyError: Could not link verifier

The clue was in the path of the first error. The directory the JVM tried to read belonged to another action, which had usually finished already and had its directory deleted.

Root cause. When a NativeLink worker materializes inputs, it hard-links files from its local CAS into each action's working directory. No data is copied, which is fast and saves disk. But a hard link means one inode with many names: one in the CAS, plus one in every action directory that uses it.

The trouble comes from programs that find their companions by asking where they themselves are. HotSpot asks dladdr for the path of libjvm.dylib and derives java.home from it; CPython locates its standard library from where libpython3.11.dylib sits; executables find the dynamic libraries under ../lib through @executable_path. On macOS, when dyld reports where an image lives, it gives whichever name the kernel most recently recorded for that inode. That may be the CAS path, or the directory of another action that may already be gone.

So a JVM in one action would confidently go looking for its time-zone database in another action's directory that no longer existed.

Figure 2: One inode, several names. dyld reports the name the kernel recorded last, which can belong to an action that has already finished; clonefile gives each action a private inode with a single name.

We peeled the same mechanism three times.

1. Executables surfaced first. rustc reported Library not loaded: @rpath/libLLVM.dylib, and hermetic Python could not find its own libpython; that is the @executable_path route. We gave every Mach-O main executable a private inode, and in an isolated reproduction, failures went from 1,007 of 2,400 runs to zero.
2. The JVM kept failing. The java executable was fixed, but libjvm.dylib is a library, and dladdr reports the name of the library's own inode. So libjvm.dylib joined the list of files that get a private inode.
3. In the full acceptance run after the JVM fix, a Python tool failed one more time. CPython also locates its standard library from the position of libpython, and on the pool the symptom is an error such as No module named 'copyreg'. So libpython joined the list as well.

Each layer was measured before it was fixed. We wrote a probe: a throwaway package of 900 genrules, each running the JDK once and printing java.home and its working directory, forced to execute on the pool. Before the fix, 150 of the 900 (16.7%) reported a java.home inside some other operation's directory, and not one of the eight workers was clean. A local stress test at a concurrency of eight made the pattern sharper:

The fix. When the worker materializes inputs, it uses APFS clonefile(2) to give two kinds of files a private inode: every Mach-O main executable, identified by MH_EXECUTE in its header because Bazel marks every remote input executable and permission bits cannot tell them apart, and the libraries named in the private_inode_libraries setting, currently libjvm.dylib and libpython3.11.dylib. A cloned file has exactly one name, so dyld cannot report the wrong one. Every other input is still hard-linked.

Why not clone everything? For two reasons, both learned the hard way:

• Code-signature checks. The first time a new inode is executed or loaded, macOS verifies its signature again. On the SIP-enabled Mac minis that costs a fixed 180–200 ms per new inode. Cloning rustc's librustc_driver (80 MB) and libLLVM (137 MB) as well would take a minimal rustc compile at a concurrency of eight from 37 ms to 636 ms.
• Header identity. We tried cloning the entire input tree, and every header became an inode of its own. clang decides whether two headers are the same file by inode, so an Objective-C library that exposes its headers through symlinks started failing with duplicate interface definition. That approach was reverted.

So cloning covers an allowlist: images that locate themselves through dyld and have actually failed. The cost is about 0.28 seconds of first load per JVM action and about 0.2 seconds per hermetic Python action. A cold full Android build gets a few percent slower, an incremental one touches only a handful of actions so the difference vanishes, and actions with no JDK or hermetic Python among their inputs pay nothing.

rustc also uses dladdr to find librustc_driver and occasionally fails with Failed finding sysroot, the same class of problem. Its libraries stay shared for performance, the failure rate is on the order of one in ten thousand, and a rerun passes, so we have deliberately left it alone for now.

Upstream. The root cause is in how NativeLink materializes inputs on macOS. We filed TraceMachina/nativelink#3137 and submitted the fix as #3138, which is still a draft as of this writing. Until it merges, our workers run 1.8.0 plus those two commits. Spelling out private_inode_libraries in the configuration doubles as a safeguard: an unpatched NativeLink does not recognize the field and refuses to start, rather than quietly running with shared inodes.

Acceptance. After deployment, 2,700 JVM probes spread across the eight workers came back with none misplaced. The Python probes went from 458 of 2,700 misplaced before libpython joined the allowlist to zero after it. In the forced full acceptance run that followed, with 9,144 remote actions for Android and 6,950 for iOS, the only failure was one of the rustc flakes described above, which struck 2 of the pool's 20,327 rustc invocations that day. From that day on, Android builds went to the pool too.

Remote execution is a full audit of hermeticity. Every assumption about paths, environment variables and the filesystem that happens to hold on one machine fails at random, a few times in a thousand, at the scale of a pool. Guessing will not find these. Write a probe that measures the failure rate, fix the cause, and use the same probe to prove the rate has reached zero.

Trunk-based development: make CI cost follow the change

The other end of the build pressure is CI. Agents make the number of merge requests grow linearly with concurrency. Run everything on every merge request and the runners give out first; run too little and trunk turns red. A red trunk stalls every agent that rebases onto it, because each of their pipelines inherits the breakage.

Our trunk rules are simple. Every change starts as a short-lived branch off trunk and merges back once it passes automated checks in a merge request; release branches are managed by the release system, and an ordinary push to one triggers no pipeline. The hard part is deciding which checks to run, and our answer is to let Bazel decide. The first CI job runs an impact analyzer written in Rust that reads the files a merge request changed, computes the affected targets on the build graph, and generates a child pipeline containing only those targets.

Figure 3: From worktree to trunk. Each merge request runs only what it affects, product builds happen after the merge, and a nightly run covers every UI test.

Changed files are classified by how they change the build graph, and each class gets its own answer:

The last row is a principle: failing to find an impact is not the same as having none. A failed analysis may cost machine time, but it may never make a skipped test look like a passed one.

Three policies sit on top:

• Merge requests run tests, not product builds. Debug and release builds for iOS and Android run in the trunk pipeline after merge. Whichever merge broke a product build is the merge whose trunk pipeline turns red, so attribution is exact.
• UI tests are chosen by ownership plus sentinels. A merge request runs the UI tests of the products it edits, plus four sentinels: the owners whose UI-test dependency closures the change reaches most deeply. Every other UI test is left to a nightly full run, and the nightly query that lists the full set must be complete. If a package fails to load or no UI test turns up at all, the nightly run fails instead of quietly running fewer tests.
• A new commit cancels the previous pipeline of the same merge request. Only the latest commit's result decides whether it can merge.

None of these rules came from intuition. We measured where nine days of runner time, from 2026-10-26 to 11-03, actually went:

BUILD files follow the same idea. Editing a BUILD file used to mean the whole package, and editing a .bzl file meant every package that loaded it, so a registry.bzl loaded by every product meant that registering one new product re-ran 24 products. With rule-level hash comparison, only rules that actually changed trigger a rerun. The same class of problem, solved on the side of product declarations, is the subject of Twenty-Three Products, One Blast Radius.

The last step belongs to the agent

The merge flow we give agents is a fixed skill: commit, rebase onto the latest trunk, push, open a merge request, wait for the pipeline, and merge only after collecting evidence that everything is green.

The waiting step taught us something representative. Our pipeline has two layers: the parent runs static checks, and a generated child runs the affected tests. Once, the API call that lists the child's jobs transiently returned nothing. The script read "could not read" as "no jobs, all done", the parent happened to be green, and the merge was initiated against a pipeline that later failed.

The rule since then: green takes evidence; it is never the default. A layer that could not be read in full during a polling round is unknown rather than empty, and while the bridge job that triggers the child is still running, the child layer has not finished, whatever its job list says at that moment.

That is the same principle as in impact analysis: in an automated system, "I don't know" has to fail closed. A person who sees a strange "0/0" gets suspicious; an agent does not. The reverse holds as well, since a red should not be manufactured out of missing evidence either, which is the argument of Whoever Ran the Test Owns the Verdict.

What it costs, and what is still open

None of this comes free. These are the costs and gaps we know about:

• No local fallback. When the executor is unreachable, a --config=rbe build fails, which is why it is opt-in per invocation. The pool sits on the office LAN, and reaching it from elsewhere means going through something like Tailscale.
• Shared capacity, co-located with CI. Every development machine and agent shares the 46 slots, while CI jobs run locally on the same machines. A batch of cold builds can fill the pool for minutes to tens of minutes.
• The trust boundary is the LAN. The pool's gRPC port has no authentication, and any client that can reach it can write to the Action Cache. Keeping local results out of the pool is a client-side convention, not something the server enforces.
• Two caches that are not joined yet. CI's remote cache (bazel-remote) and NativeLink's CAS share nothing.
• Tests are still local. With compilation gone, iOS simulators are the largest load on the host. At the time of writing, the processes of three booted simulators held about 36 GB of resident memory (RSS, shared pages included).
• A few loose ends. The occasional rustc sysroot failure; one worker whose connection to the scheduler keeps dropping, still under investigation; a cold build's critical path pinned by a Rust build script that runs for about nine minutes; and the JDK, which ships with every action as an input, at the cost of the 0.28 seconds mentioned above.

Six things to take away

1. Agents are load generators, so design for N concurrent builds. A build configuration tuned for one person oversubscribes structurally under N agents. Only a scheduler that sees every client turns that oversubscription into a queue.
2. Move only what is heavy and indifferent to location. Keep agent processes, build analysis and device-bound tests local, and hand deterministic action execution to the cluster. Classifying the load by its properties does more than any amount of tuning.
3. Hermeticity is the entry ticket, and remote execution is the audit. Every environment variable outside the cache key, and every program that finds its companions by asking where it is, is a latent failure. Measure the failure rate with a probe, fix the cause, and prove the rate is zero with the same probe.
4. Invariants for agents must be machine-checkable. Agents optimize locally, for example by starting another output base to dodge a lock. Put the rule where agents read it and where CI checks it.
5. CI cost should scale with the change, not with the repository. Measure where runner time goes before deciding what to fix, then start with the largest bucket. Our two biggest kinds of waste were 17% and 29% of merge-request runner time.
6. Fail closed, because unknown is not empty. A failed impact query runs the whole graph, an incomplete nightly test list fails the night, and a pipeline that cannot be read in full is never judged green. People grow suspicious of anomalies; agents do not.

What's next

• Merge CI's remote cache with the RBE CAS, so that whatever CI built is a cache hit for agents, and the other way around;
• Add Linux workers for Go, TypeScript and part of the Rust actions (the OSFamily scheduling property must then change from priority to exact matching, or nothing stops an Apple action from landing on a Linux worker);
• Authenticate clients to the pool and isolate writes by instance;
• Move the scheduler's configuration into the repository and render it the way the workers' configuration already is;
• Preinstall the JDK on the workers the way Xcode already is, sparing every JVM action its signature check.

The build system has a new kind of user

Looking back, the biggest shift was this: the build system's user has changed. It no longer serves one person who waits, thinks and does one thing at a time, but a crowd of machine developers that run concurrently, never tire and optimize locally. In the human era, Bazel's hermeticity and content addressing were engineering virtues that made builds faster and more reliable. In the agent era, they are the infrastructure that decides whether parallel work runs at all.

Today our agent host does only what it should: coordination, and verification on local devices. Thinking goes to the model, compilation goes to the pool, and correctness goes to impact analysis on trunk. The rule about stopping at five or six agents is one we are in the middle of forgetting.

Thanks to Trace Machina and the NativeLink community.