Eight gibibytes, please
The Android CI job was dying before it built anything. The error was a JVM asking the operating system for eight gigabytes and being told no:
-Xms8g os::commit_memory 8589934592 failed
Eight gigabytes is the default initial heap for the Java tools in the Android rule set. Initial is the word that matters. -Xms is a reservation the process makes at startup, not a ceiling it grows toward, so every persistent worker asked for the full eight gigabytes the moment it launched and the container did not have that much. It was not slow. It was not flaky. It was a job that could not start, on a runner whose memory was the thing under test.
There was one detail in that error that should have slowed somebody down immediately. The single-build-ourselves case passed. Nothing about the libraries was wrong; nothing about them was different. The only thing that changed was whether an android_binary was being built, and an android_binary is what pulls in dexing and desugaring. Those tools want gigabytes. The library tests never touched them.
That asymmetry was the clue, and it is a much better starting question than "why is CI red": CI is not a slower machine, it is a machine with a budget.
Attempt one: put the flag where everyone puts flags
The first fix was the standard one. Bazel has flags for Java heap size, so we set them.
build:ci --jvmopt=-Xms512m -Xmx4g
test:ci --jvmopt=-Xms512m -Xmx4g
The reasoning underneath it is worth naming because it is so widely believed: a small initial heap means no big reservation, a four gigabyte ceiling is lazy, and that is enough for dexing a small app without exhausting concurrent workers. The commit records the verification: the same configuration with a sixteen-megabyte heap fails the build, so the flag is definitely reaching something. Locally, the app built green.
CI kept dying. Same error, same eight gigabytes.
Attempt two: the same flag, one level over
Attempt two had a better diagnosis and no better outcome. The tools that were asking for eight gigabytes run in the exec configuration, not the target configuration, because they are not products of your graph; they are tools that run your graph. --jvmopt sets JVM options for the target configuration, so it was pointing at the wrong one. --host_jvmopt sets them for the exec configuration.
build:ci --host_jvmopt=-Xms512m -Xmx4g
test:ci --host_jvmopt=-Xms512m -Xmx4g
Same verification, more precise this time: sixteen megabytes and the dexer dies immediately, four gigabytes and the app builds. And again CI kept dying, in the same place.
Two attempts, two different configurations, two verifications that passed and looked convincing, and the symptom unmoved. That is the moment where the failure stops being a configuration problem and starts being a question about what a verification can actually establish.
What the probe had proved
The probe was the same in both attempts: shrink the heap to something absurd, confirm the build breaks, conclude the flag is connected. It is a good instinct. Shrinking a knob until the thing breaks is how you find out whether the knob is attached, and the instinct generalises far past heap flags.
But look at what the outcome contains. "Build failed" contains no process name, no heap value, no indication of which Java process died. It is a symptom, and a symptom is satisfied by every cause with the same shape. In a build this size there are dozens of JVMs. It is entirely consistent with the flag working perfectly on the JVM you configured and doing nothing at all to the JVM that was drowning.
And that reading was the correct one. Persistent workers do not take their heap from the command line. They take it from the tool's own declaration, and the last setting on the JVM's command line wins, so a flag that sets the launcher's heap loses to the value the tool ships with. The knob was attached. It was attached to the wrong thing.
The sharper version of the probe, in hindsight, is one that watches the failure happen in the process you care about, with the value you set, on the machine where the bug lives. Change one thing, and the failure should appear in the same place the original failure appeared. Change two things, and the result is a different experiment wearing the same coat.
Attempt three: stop configuring, start reading
The fix that held stopped using flags. It changed the declaration:
-keep ... sorry.
Concretely: a patch against the rule set, in four files, replacing every one of those eight gigabytes with a small initial heap and a four gigabyte ceiling. And one more change that is easy to skip and easy to regret. The SDK's own tools are declared by a helper file that the patcher refused again and again, so that one is a copy-and-replace, performed in a file we own:
if '"-Xms8g"' not in helper or '"-Xmx8g"' not in helper:
fail("helper no longer declares the heap literals; the copy-and-replace is now a no-op")
That guard is the whole thing. A string replacement that matches nothing does not fail. It succeeds, changes nothing, and the heap quietly goes back to eight gigabytes in some future week, with no signal to anyone. When a fix depends on a literal being present, assert the literal is present.
The verification that meant something
The verification that told the truth was not a build. A build tells you that something worked; it never tells you what worked. The verification that told the truth was reading the value where the reader reads it:
rules_android jvmopts: -Xms512m -Xmx4g
androidsdk helper.bzl: -Xms512m -Xmx4g
Two reads, both at the source, both showing the number that matters. No cleverness to it. It is just the difference between asking whether the outcome looks right and asking whether the thing you changed is the thing that was wrong. The outcome is compatible with being fixed by the wrong change. The value is not.
Two smaller lessons sat in the same afternoon. The patch had to be produced as a diff, in the format the patcher expects, and produced against a copy that was clean: a fetch that fails halfway leaves the tree patched, and every diff after that is measured against something already modified, which is how a correct fix turns into a correct fix plus a pile of nonsense. And the honest summary of both failed attempts is that local green told us nothing. The bug only lives where memory is scarce. A laptop is never going to reproduce "the container cannot grant eight gigabytes," so "it builds locally" was a statement about the laptop, filed under the heading of evidence.
Takeaways
A verification that proves something broke is only as good as the specificity of "something". Name the process, name the value, and run it where the original failure happened. A flag that reaches some JVM is not a flag that reaches the JVM that was drowning, and the difference between those two claims is the difference between a fix and two wasted afternoons.
The flag reaches an invocation. The thing you are configuring lives in a declaration. Find the reader of the setting before you reach for a flag to change it.
When the outcome looks right, do not stop there. Read the value at the source. Two reads of a number beat any number of green builds, because a green build cannot distinguish "fixed" from "fixed by something else."
And treat every string replacement as something that can silently match nothing, and every patch as something that has to be generated cleanly. Both are places where doing less than you think produces a build that looks fine and is not.