Cache the Failure, Never the Success

Ten tests, ten minutes, one missing element

Our UI suites assert preconditions before they do their real work: is the account signed in, is the list long enough to paginate, is the fixture data there. A precondition is checked with a generous timeout, because it depends on the simulator being slow rather than on the product being wrong.

That design is fine until the precondition is false. Then it is the worst thing in the suite, because every case independently pays the full timeout for the same verdict.

Here is a real measurement. A ten-case suite asserted that a list could paginate. It could not. Each case waited out its own sixty seconds:

• One round of the suite: 12.7 minutes
• The build system's whole-suite flaky retry: ×2
• The coverage re-run: ×2

A seventy-minute job, of which fifty minutes were spent waiting for an element that was never going to appear. The suite was not slow because of anything it was testing. It was slow because it had discovered a fact and then re-derived it ten times.

The instinct is to lower the timeout. That trades one wrong answer for another — the suite now fails faster for the right reason and, eventually, passes for the wrong one on a slow machine.

Memoise the negative, and only the negative

The fix is a shared piece of test infrastructure that remembers which preconditions this process has already found unmet:

public static func evaluate(_ name: String, verify: () -> Bool) -> Bool {
    if isKnownUnmet(name) {
        print("⏭ 前提「\(name)」已在本 suite 判负，跳过重复验证")
        return false
    }
    let met = verify()
    if !met { markUnmet(name) }
    return met
}

The first case waits for the real answer, as it must. Every later case in the same process skips straight to the verdict it already has. Ten cases, one timeout, the same outcome.

What is deliberately not cached is the positive case. When a precondition holds, every case still waits for its element for real, on its own cold launch, for the full timeout if it takes that long. That asymmetry is the entire design, and it comes down to what each verdict is worth:

• A failed precondition is usually a property of the build and its fixtures — the data is not there, the permission is not granted, the simulator booted wrong. Those do not change between two cases in one process. Caching it loses nothing.
• A passed precondition is a real measurement. It is the evidence that the element appeared within the timeout, and it is exactly the thing that tells you the suite is getting slow or starting to flake. Cache it and you have deleted the signal.

This is the opposite of the usual advice about caching, and it holds because the expensive state in CI is failure, not success. A failure is slow and redundant: N cases pay N times for one verdict. A success is slow but not redundant — each case genuinely needed its own answer.

The cache has to announce itself

A cached verdict changes what the suite does, so it has to say so. The skip prints the precondition's name and the reason. Without that line, ten cases failing instantly and ten cases passing instantly produce identical test output, and the next person to read the log has no way to tell a correctly short-circuited suite from one that quietly stopped testing anything.

That is not hypothetical politeness. Every caching mechanism in a test suite eventually gets mistaken for the thing it replaced, and the only defence is that the replacement is visible in the output.

It also matters that the memory is scoped to the process and can be cleared. Several unrelated suites can run in one process, and the mechanism's own tests need a clean slate — so the reset is part of the API rather than something each caller reaches for by re-instantiating.

What adopting it cost

The interesting number is not the runtime. It is that the change made the test code smaller:

A net reduction of 264 lines. That is not because the helper is short — it is because every suite had grown its own version of the same pattern, usually written slightly differently, and each copy carried its own timeout constant and its own idea of when to give up. Removing the idiom from every suite deleted more than the shared helper added.

There is a second reason the copies were worth removing. Once each suite owns its own retry-and-timeout logic, someone eventually tunes one of them in a hurry — shorter, because it was slow — and you get fifteen suites with fifteen subtly different definitions of "slow." A shared precondition check with one timeout is a smaller surface, and it is a surface where the number is written down once.

The general shape

The pattern here is not really about UI tests, and it is worth stating more broadly:

In any repeated verification, ask whether the verdict is a property of the world or of the attempt.

The world did not change between case three and case four. The attempt did. When a verdict is about the world — the fixture is missing, the permission was denied, the build shipped the wrong bundle — it is safe to remember it for the lifetime of the run. When it is about the attempt — did this render in time, did this request return — remembering it destroys the measurement.

Most caching advice assumes the opposite default, because in ordinary software the expensive thing is the success: you want to avoid recomputing an answer you already have. In test suites the expensive thing is the failure, because that is the state where the timeout is long and the work is guaranteed to be wasted.

The generalisation is small and, we think, transferable to anything that verifies a precondition repeatedly: orchestration systems that re-check whether a dependency is ready, retry loops that re-establish a connection before each attempt, and health checks that run on a timer. In all three, ask whether the thing you are about to re-verify could have changed since the last time, and cache the verdicts that could not.

Takeaways

When a suite is slow, measure what it is waiting for before you shorten anything. Ours was spending fifty minutes of a seventy-minute job on a fact it had already established, and no timeout tuning would have recovered it — the work was simply being repeated.

Asymmetric caching is usually right in verification code, and the asymmetry is not arbitrary. A failed verdict is a fact about the environment that will not change inside one run; a passed verdict is a measurement, and caching it deletes the signal you would use to notice the suite getting slow. Cache the negative, always re-do the positive.

Make the cache visible in the output. Ten instant failures and ten instant successes are indistinguishable unless something says which happened, and a silent cache is indistinguishable from a test that stopped testing.

And expect the migration to delete code. Replacing a per-suite retry idiom with a shared check removed more lines than it added across sixteen products, because each copy had its own timeout constant. A single shared number, written once, is worth more than fifteen locally-tuned ones — and the only way to find them is to delete the pattern rather than add a rule against it.