Tech9/19/2026·4 min read

Green Locally, Red in CI, on a Target You Never Touched

Two Android SDK constants moved into the file that looked right — and that file is loaded by a module extension. Our lockfile went stale, and CI went red on Go linters.

The most natural-looking place

We raised the Android target SDK. Alongside that, two constants — the minimum and target SDK — were moved into versions.bzl, which is where the other Android versions already lived and which is what a reasonable person would call the right home for them.

It was the wrong place, and the reason is not about versions or about Android.

download_sdk.bzl is the implementation of the android_sdk module extension — the code that downloads and pins the SDK and the native toolchain. It loads versions.bzl. Which means everything in versions.bzl is part of that extension's transitive load closure, and adding anything to it changes the extension's digest.

Which means the lockfile is now stale. And the error names:

ERROR: MODULE.bazel.lock is no longer up-to-date because: The implementation
of the extension '…Toolchains/SDK:download_sdk.bzl%android_sdk' or one of its
transitive .bzl files has changed.

Read that message after a two-line change to a version file and you are not being pointed at your change. You are being pointed at a download toolchain.

Why it was green here and red there

The same commit was green on every developer's machine and red in CI. Not intermittently. Not sometimes. Structurally, and that difference is the whole story.

Bazel's local default is to update a stale lockfile. So locally, the stale lockfile was rewritten as a side effect of running anything, and the build proceeded. Nobody ever saw the state that CI sees, because the machine that produced it had already repaired it.

CI is configured with lockfile_mode=error. It refuses to repair, because repairing a lockfile inside CI is how you get a build that depends on when it happened to run. So CI reported the truth, and the truth was red — on lint:go, on lint:rust, and on the architecture guard.

Three red jobs, none of which has anything to do with Android. The dependency was indirect: a file that an extension loads changed, so every target that transitively uses that extension was invalidated, and that set is the entire repository.

The lesson is narrow and completely actionable:

Verify with the flag your strictest reader uses, not with your own default.

It took one command to confirm the fix and it is not the command anyone would have run otherwise:

bazel build --lockfile_mode=error //DevOps/Tooling/Guardrails/runner:guard  ✓
bazel build --lockfile_mode=error //Products/Android/riff:riff_aab        ✓
MODULE.bazel.lock unchanged; manifest still target 36 / min 24

The actual mistake

Moving the constants to a different file was the fix we shipped, but describing it as "moving them out of the module extension's load graph" undersells it. The deeper point is that the two constants should never have been together.

One pair describes what the operating system will do to your app: which behaviours are required, which are deprecated, which permissions the platform expects. They change when you decide to adopt a new platform policy.

The others in that file describe what to download: which android.jar to compile against, which emulator image to test on. They change when a new toolchain is released.

These are different facts with different reasons to change, and putting them in one file binds their change rates together. The cost is not stylistic. The cost was measured: every time we change the target SDK, the module lockfile moves — even though nothing about the toolchain download changed at all.

So the file was not wrong because it was the wrong directory. It was wrong because it merged two reasons to change into one unit. Once you can name the two reasons, the correct split is obvious, and so is the name for the new file: manifest_sdk.bzl.

This is the same lesson as splitting a string table by consumer, and the same lesson as separating a per-session surface from a per-release one. A file is a unit of change. Anything you put in one is a claim that those things will change together, and the claim is enforced by whatever tool reads the file.

The structural fix, when you can take it

There is a version of this that needs no new file at all.

Elsewhere in the build, a module extension was producing a projection of product metadata, and the fix was to move the projection out of the extension entirely — the extension does what it is uniquely supposed to do, and everything else reads a generated file. That change removed lines from the lockfile rather than updating them, which is the signature of having done it right: you cannot tell from a lockfile whether it grew because a dependency genuinely changed or because your own source file got swallowed by a load closure.

The diagnostic behind it is worth keeping:

If a file you routinely edit changes a digest you do not own, you have put an everyday file inside somebody else's closure. Move it out.

Build systems like this one treat lockfiles as the record of "what did we fetch." But they are also, unavoidably, the record of "what did we load." Those are different things, and the second one is yours to manage — and the most common way a repository's own lockfile churns is not dependency updates at all. It is someone adding a constant to a file that happens to sit inside an extension.

Takeaways

Know which of your files sit inside a dependency's load closure. A file loaded by a module extension is coupled to every hash in the lockfile, and the resulting error names the extension implementation rather than your change — which is exactly the shape of a failure nobody can attribute from the message.

Run the strictest reader's flag at least once. Local defaults repair; CI refuses. Any repository where those two differ has a class of failure that is invisible by construction, and the fix is a single command with a different flag. Green locally means "the tool fixed it," which is not the same claim as "the tool did not object."

Then look for two facts with two reasons to change. The target SDK says what the platform does to your app; the versions file says what to download. Same extension, same repository, same directory — and binding them meant every policy change moved the lockfile. When a change causes work it has no logical connection to, that is the signal, and it usually points at a file boundary rather than at a tool.

And where you can, remove your own files from the closure entirely rather than finding a better address inside it. A lockfile that shrinks is a lockfile that is telling you the truth; one that gets updated on every ordinary edit has stopped being a record of anything you control.

Keep reading