The Dependency That Changed Its Default and Told Nobody

One line, no description

We upgraded a build rules library. The upgrade did what upgrades do.

The library ships a default_runner label — the thing that decides how an iOS test actually executes. Our version of the upgrade included this change:

Release 5.0.0 replaced the default iOS test runner with a runner driven by xcodebuild test-without-building. One line in default_runner/BUILD. Empty description. Merged 2026-06-10, the day after a major OS release.

The neighbouring commits in that repository make the direction obvious: one consolidating per-platform test runners, one exposing the simulator creator as a replaceable action, and the new runner itself still failing on the newest toolchain. But direction is not a breaking-change notice, and the commit that could have been one had nothing in it.

We found out four months later, and not from the upgrade.

What "does not read our environment variables" costs

Running an iOS test is a shared-resource problem. A simulator is not a pure function — it is a machine with a name, a lifecycle, and a lease. Our answer to that was a small simulator layer that owns three things: how a device is named, how creation and deletion are serialised, and how a device is reclaimed. Tests acquire a named simulator from it rather than letting the runner invent one.

The new runner brings its own simulator creator, and that creator does not read the environment variables we use to name and reserve devices. So:

• Devices came up as BAZEL_TEST_<device>_<os> instead of names our capacity layer could track.
• Creation and deletion stopped being serialised with each other.
• Reuse defaulted on, so a device outlived the job that created it.

1,070 of our unit-test targets left the capacity lease overnight.

And this is the part worth being precise about: nothing failed.

The symptom was not an error

A test suite that collides with another suite on the same simulator has two possible outcomes. The bad one is that the tool notices and reports a failure. The outcome we got is the other one: both suites ran, on a device someone else was also driving, and both passed.

Every green check we had was still green. The builds were correct. The assertions were correct. What had changed is that a thousand targets were sharing a fixed set of machines without any coordination, so wall-clock times drifted, ordering became load-dependent, and the results were conditional on neighbours that nothing in the build graph knew about.

This is the specific shape of dependency risk that version pins do not protect against. A version pin says "this code." It says nothing about which default that code resolves, and a default is a decision the upstream made on your behalf, in a file you do not read, describing behaviour you never called.

The failure only became visible later, and indirectly — when a different runner cut made the paywall UI tests for every product go entirely red, and the investigation led back to a runner swap months earlier.

The fix is a single owner, not a configuration

The tempting repair is to set the environment variables the new runner wants, or to patch its simulator creator. Both of those make the next upstream change a fresh incident.

What actually fixed it is that every iOS test target in the repository already reaches its runner through one of our own macros — one for unit tests, one for UI tests, both resolving to a single runner target we control. That is not a convention; it is forced. The upstream default is pinned to a label inside the upstream repository, and downstream code cannot point it at anything of ours, so the whole repository was already going through our entry point.

Which means the swap could not silently take hold, and when we pinned the old runner back, one line changed the execution environment for every test we have.

The lesson generalizes past build systems. When a default can be changed by someone else's release, the durable protection is not vigilance or a pin. It is making sure that no code of yours can reach the default directly — so that changing it is a deliberate act in your repository rather than an inherited property.

The migration we could not finish

Pinning back fixed the capacity problem and left the upstream migration unfinished, so we tried to complete it properly. It did not work.

The new runner drives tests through xcodebuild test-without-building, which is incompatible with the local StoreKit configuration our paywall tests rely on. Every paywall UI test went fully red:

All eleven products that test their paywall through that path were affected at once. So we pinned back, which is the honest outcome, and wrote down what the pinning costs: four patches to the older runner, which keep its device naming, lifecycle serialisation and boot behaviour working. Those patches are now ours to maintain.

We also wrote down what we do not know. Alongside the paywall cluster there are roughly twenty other failures in one product's editor and home screens, four keyboard-extension targets reporting that the system keyboard cannot find our extension, an account-flow failure and a smoke-test failure. Whether they share the same cause was never established by a controlled comparison, and the guide says so explicitly rather than grouping them under a cause that was only proven for a different set of failures.

That last part is the part we would defend hardest. The temptation in a rollback write-up is to explain everything that is currently red. Resisting that costs one paragraph and preserves the only thing that makes the rest of the document trustworthy.

Takeaways

Audit what a dependency's defaults do, not only what its version pins. A pin fixes the code you get. It does not fix which branch of behaviour that code resolves, and a default is a decision somebody else made about your system, in a file you did not read, on a date you were not watching.

Treat "everything is green" as a claim about the environment, not just the code. Our tests passed while a thousand of them shared machines they had no lease on. Green means the assertions held; it never meant the run was the one you thought you performed. If correctness depends on a resource being exclusively yours, that exclusivity has to be asserted somewhere, not assumed by the harness you inherited.

Route every consumer through an owner you control, so the default cannot change under you. The strongest protection here was not a version pin or a config setting — it was the fact that no code of ours could reach the upstream default directly. That is what turned a silent four-month regression into a one-line change.

And when you pin something back, write down what you gave up and what you never explained. A rollback note that explains every currently-red test is not thorough; it is a guess wearing a document's clothes. Naming the failures you have not diagnosed is what makes the ones you did explain worth reading.