The guard watches the thing it guards
Our architecture guard does something slightly unusual, which is that it watches the repository while it verifies the repository.
The reason is sound. A guard that reads files to decide whether an invariant holds is only correct if the files it read are the files that exist — and code, tests and configuration all change underneath a long-running check. So the tool records native filesystem activity while it works, using the platform's own change-notification API, and treats unexpected writes to the tree as a condition to report rather than silently verify a moving target.
The rule it enforced was the blunt version: the repository does not change while a run is in progress. Violate it and the run is not trustworthy.
The tests were flaky, and the flakiness was not the test's fault
The tests for that watcher failed intermittently. Not often, and not obviously — which is the worst shape for a failure, because every occurrence looks like a one-off and none of them accumulate into a pattern you can act on.
The cause was not in our code. On macOS, filesystem change notification is delivered by a system daemon, and delivery is bounded by that daemon's scheduling. An event is not delivered when it happens; it is delivered when the daemon gets around to it. The window is small and usually invisible, and occasionally it is not.
So we had built a test whose pass/fail depended on an interval we do not control and cannot shorten. Every mitigation we reached for first — a longer sleep, a retry, a generous timeout — treats the symptom. A longer sleep raises the probability of passing without changing the fact that the assertion is about a delivery time.
That distinction matters more than it sounds. A test that waits longer is a test that is still asserting something unobservable, only more expensively. If the delivery genuinely can be late, then at some timeout the assertion becomes a coin flip rather than a guarantee, and the number you pick is a guess dressed as engineering.
What the platform actually guarantees
The productive move was to stop asking the watcher to prove things it cannot prove, and to split the question in two.
FSEvents delivers metadata and namespace changes. It is good at those, and it is the right tool for the causal fence — the guarantee that an event about to be handled belongs to this run and not an earlier one. It is a delivery mechanism with an explicitly asynchronous contract.
kqueue answers a different question. It pins the inode of the repository root so that structural identity changes are observable synchronously, and watches individual files so that a target of an authorized exchange can be checked directly rather than inferred from a stream. A nonzero proof delta has to be matched by a rename or delete on that specific file.
Two mechanisms, two guarantees, two different failure modes — and the rule that matters is that neither of them infers causality from access time. The original bootstrap did exactly that, which is why replacing it with a causal boundary was the change that actually moved things rather than the one that added a sleep.
The other half of the native work was upstream. The notification library we pin had lifecycle problems of its own — a stream startup error that was swallowed, a shutdown that could fail to wake the run loop under load, dropping one watcher purging device-wide history it should not have, and a raw path or an unknown future event flag able to unwind through a native callback. Each of those is the kind of defect that shows up once a month, on someone else's machine, and is impossible to reproduce on demand. Taking the upstream fixes and pinning them was cheaper than diagnosing them twice.
Twelve commits, ending in a deletion
The sequence is worth listing because the ending is the part nobody plans for:
- an access-time bootstrap replaced with a causal boundary;
- vnode expectation lookup scoped to macOS, since it never worked anywhere else and had been quietly pretending to;
- the upstream lifecycle fixes pulled in, then partially backed out when they turned out to change behaviour we depended on;
- the FSEvents lifecycle test made deterministic;
- panic coverage added, because the native callback path needed to be provable;
- the tests serialized against each other;
- and finally, on the last day: remove daemon-dependent FSEvents probes.
Two days after that, the rule itself went. The guard no longer enforces "the repository does not change while a run is in progress."
Not because it stopped being true. Because we could not verify it, and we had established over two weeks that the cost of trying was real: tests that flake for a reason outside the code, that get re-run rather than read, and that train a team to treat a red guard as noise rather than as information.
An invariant you cannot observe reliably is not a safety property. It is a source of flakes with a good story attached.
What we kept
Almost none of this work was wasted, and the parts that survived are the parts that do not depend on delivery timing:
- the split between a delivery fence and structural identity, which is a better design than the single watcher we started with;
- the rule that causality is never inferred from access time;
- the pinned upstream fixes, which are now ours and no longer a monthly surprise;
- and the removed probes, which replaced silent flakiness with an honest absence.
That last one is the change I would point at first. A guard suite that is green because it no longer contains a test that cannot be made reliable is worth more than a green suite that is green by luck. The failure we accepted is a real one — an unauthorized edit during a run may no longer be caught by that specific rule — and it is a failure we chose, with a name, instead of one we accumulated by ignoring a flaky test for two weeks.
Takeaways
Before writing an assertion about a timing window, establish who owns it. If the interval is bounded by another daemon's scheduling, you are not writing a slow test; you are writing a test that will eventually disagree with the machine, and no amount of timeout tuning changes that. Assert on the guarantee, not on the delivery.
Split a composite guarantee into the parts the platform actually provides. We had one watcher asked to do two things, and the asynchronous half poisoned the synchronous one. Fence causality with the delivery mechanism and pin identity with the synchronous one, and let each fail the way it fails.
More retries are not determinism. Every mitigation that lengthens a wait is a bet that the next run gets a better machine, and the day it loses is the day you learn the test was never checking anything.
And delete the invariant you cannot enforce. It is a legitimate engineering outcome, not a failure of discipline, provided you name what you are giving up. A guard suite that is green because it stopped asserting something unknowable is more trustworthy than one that is green by luck — and the re-runs it no longer absorbs were time you were spending not reading red builds.