The green that was reported as red
Our CI is an orchestrator that decides what to build, what to run, and whether the result is acceptable. For a long time it decided the last part by reading the build log.
That worked until a flaky test existed. Bazel has a retry mechanism: a target that fails can be re-run, and if it passes, Bazel reports it green. From Bazel's perspective that is the correct, documented behavior — the retry is how it distinguishes an intermittent failure from a real one, and it has its own accounting for how often it happens. From ours, the log for that target contained both the failure and the pass.
So the orchestrator, reading the log, saw the failure and reported red. Not because the test had failed — because it had failed once, and then not. Every flake in the fleet was unreportable as a flake and reportable only as a failure, which meant the one signal that could tell us where our real instability lived was destroyed on arrival. Meanwhile the occasional genuinely-broken retry-and-pass case was indistinguishable from the same event, so the retry rule was tuned by vibes.
There was a second, quieter version of the same bug. A query that failed to expand was being reported as "no targets were affected by this change" — an infrastructure failure dressed up as a clean bill of health. A pipeline that says "nothing to do" and a pipeline that says "I could not find out" are the same two words, and only one of them is safe to act on.
Two questions wearing the same coat
The root cause is that two unrelated questions had been written into the same place:
- What did the test runner decide?
- What does CI want to do about it?
They are not the same question. The second is a policy choice — it changes when we decide to tolerate a flake — and it can only be answered by whoever owns the policy. The first is an observation, and re-deriving it from the runner's human-readable output is a guess dressed as a fact. Output is not a contract. It is prose that changes when someone improves the formatting.
The fix moves the observation into a place where it is genuinely observed. Bazel emits a machine-readable protocol for every build event, and the orchestrator now reads that stream: which targets were built, what status each ended in, and how many failed attempts it took to get there. Nothing downstream re-computes the pass/fail bit. The modules are separated deliberately, and the comment at the top of the policy module says why in one line — every past false red came from a place that re-derived what Bazel decided while meaning to express what CI wanted.
The change was 28 files and roughly 1,700 lines added, most of it in the event-parsing and verdict modules, because the log-scraping logic had been scattered through the orchestrator and removing it meant untangling it from four other stages that had grown around it.
Absence of evidence is not a failure
The redesign produced one decision worth isolating, because it is the one most CI systems get wrong.
If the build produces no event stream at all — the connection died, the artifact was missing, the tool halted before emitting — the verdict is unknown. Not zero failures. The pipeline returns a verdict object that reports no passes, no failures, and no skips, and lets the exit code and the retry classifier decide what happens next. Manufacturing a test failure out of missing evidence produces a red build that points at the code instead of at the infrastructure, which is exactly backwards: it sends a human to read the diff that has nothing to do with what broke.
The same logic applies one level down. When iterating targets, three cases get counted separately, and only one of them is a failure:
- ended green — passed
- ended in a genuine test verdict (an assertion failed, a target timed out) — failed
- neither — skipped
A target that never built, or whose tool halted, produced no verdict. Counting it as failed is the same error in miniature, and it is the error that made "0 tests failed, 47 targets unaccounted for" reportable as a clean run.
A retried pass is a policy, not a fact
With the observation clean, the remaining question is what we do about a target that passed only after a retry. That is a policy, and it now lives in one place — a constant, next to the sentence explaining it:
/// Whether a target Bazel retried into a pass is accepted as a pass.
const ACCEPT_RETRIED_PASS: bool = false;
false is the choice we already made: a retried pass fails the job, so a flake stays visible and gets fixed instead of being quietly absorbed into the baseline. It costs a full extra run for every flake and buys no green — a real trade, and the kind that normally gets re-litigated in whichever direction whoever is debugging last happened to be feeling. What changed is that it is now only a trade. It is one constant, in one file, with its reasoning attached. Flipping it is the entire of "let declared-flaky targets be exempt"; nothing else in the verdict path has an opinion about flakiness.
Where an exception genuinely exists, it is a named entry rather than a policy change. A target whose flake cannot be fixed from this repository — one iPad UI suite whose intermittent failures come from a system assertion inside SpringBoard's own multitasking transition, crashing the shell before the app under test ever launches — is exempt, because for it the "keep flakes visible" policy buys nothing: every merge request touching that target pays an extra full run and then gets rejected anyway. Each entry carries the evidence that the cause is external and the condition for removing it, and the match is on the exact target label. A prefix or substring match here would silently cover targets nobody granted the exemption to.
The result is a policy that is greppable, reviewable, and deletable. When the underlying cause is fixed upstream, removing the exemption is a line deletion — not an archaeology project to work out whether the suite is stable again.
The other half: nine TestMains
The same invariant had a second owner, in Go, and it had regrown nine times.
Go's TestMain runs before and after a test binary's tests. Ours used it to clean up throwaway fixtures — a Postgres container, a queue, a temporary directory. The cleanup runs after the tests, and if the fixture refuses to stop, the natural thing to write is to fail the test binary from inside cleanup, by calling os.Exit(1) or assigning a non-zero exit code. That overwrites the verdict the test binary had already decided. A run whose tests all passed would report failure because a container would not shut down in time, and — worse in a quieter direction — a run that had a real failure could have it overwritten by a fixture that failed differently.
The rule now has one owner, and the separation is the same shape as the Rust side: disposing a fixture is best-effort and reports failure without changing the verdict. Cleanup for the process's exit code and cleanup for a test-scoped assertion are two different operations with two different owners, and the package that owns fixture teardown provides both rather than letting each test file reinvent them.
That distinction is enforced, because nine copies is not an accident but a copy-paste culture:
id = "test-verdict-ownership"
risk = "high"
paths = ["Backends/**/*_test.go", "Analytics/**/*_test.go", "Foundation/Backend/**/*_test.go"]
forbid = ["*code = 1", "*os.exit(code)*"]
The first glob is end-anchored deliberately, so that a longer literal — a counter that legitimately reaches a value ending in those characters — does not trip it. The rules were measured against every Go file in those three trees: together they select the nine known copies and nothing else. The CI half of the same invariant is a separate manifest, because a single flat forbid-list applied to every listed path would run the orchestrator's vocabulary against every Go test file, and some of that vocabulary is ordinary English.
Takeaways
A verdict belongs to whoever ran the test. Any stage that re-derives pass/fail from a runner's output is guessing — output is prose, and prose changes when someone improves the formatting. Read the runner's machine-readable event protocol, and treat the difference between "reported" and "derived" as a boundary worth naming in your code.
Separate the observation from the policy. Most CI bugs live in the seam where "what happened" and "what we do about it" share a function. Two modules with one line each saying which is which costs almost nothing and makes the next bug obvious.
Distinguish three states, not two. Absent, passed, and failed are different facts, and collapsing absent into either one produces a confidently wrong answer. Absence of evidence is an infrastructure state; let the retry and exit-code machinery decide what it means.
And make tolerance a constant, not a mood. The instinct to absorb flakes is strong and mostly wrong, and the correction is not to ban it — it is to write it down in one place with its cost attached, and require every exception to name its cause and its own expiry.