A crash no test could reach
We were writing a pixel test for Sigma's 3D scenes on Android, and installed the shipped product APK to compare against. It died the moment it entered SigmaActivity:
java.lang.UnsatisfiedLinkError: No implementation found for …
Every suite covering that code was green. The fix turned out to be nine lines, most of them a comment. The interesting part is why nothing caught it.
Names are the ABI
Sigma's Android layer binds Kotlin to C through plain JNI. A method declared external in Kotlin is resolved at runtime by symbol name, and the name is built from the class and the method:
JNIEXPORT jlong JNICALL
Java_com_aio_sigma_SigmaNative_sceneLoadLegacyWithKv(JNIEnv *env, jclass clazz, …)
R8, Android's shrinker and obfuscator, renames classes and methods by default. In the release build it turned SigmaNative into hr and sceneLoadLegacyWithKv into F. The runtime then went looking for an implementation of hr.F, and there has never been a symbol by that name. The native code was in the APK the whole time; it just no longer had a name anyone was asking for.
So for this kind of binding, the names are not decoration. They are the linking contract between two languages, and an obfuscator that rewrites them has changed the ABI of the app.
Three green suites, three different programs
Each layer of testing was real, and each one was green for a true reason.
- Host JVM tests compile the Kotlin and run it on a desktop JVM. The native library is never loaded there, so no JNI symbol is ever resolved.
- Native tests compile the C side on the host, against a fake JNI environment and a fake engine. They prove the bindings marshal arguments correctly. They cannot prove the JVM can find them.
- The instrumentation test ran on a real emulator, with the real native libraries. But it ran a self-instrumented APK, built from the same sources by a rule that never invokes R8. Its
SigmaNativestill had every original name.
None of these was the shipped app. Every one of them was a faithful test of a program we do not ship, and the bug lived only in the one we do.
Why the test rule chose the wrong build
The instrumentation rule can run either way: install the real product APK and test it as a black box, or build a self-instrumented APK with test and app in one process. It picks a default from the product's metadata:
if ship_artifact == None:
ship_artifact = bool(proguard_specs)
The reasoning is sound on its face. If the product declares R8 rules, test what R8 produces; if it does not, the self-instrumented APK is equivalent. The demo app declares proguard_specs = [], so it got the self-instrumented build.
When that line was written in July, it was true. android_product only ran R8 when the product passed its own rules.
In September, release orchestration changed that. Every release build now has to produce an R8 mapping, because the crash pipeline needs one to symbolicate. So android_product dropped its if proguard_specs: gate and started prepending a base rule file unconditionally. From that day the build's answer to "does this product go through R8" was always, and the test rule's answer was still only if it declares rules.
Two files were answering the same question, and they had stopped agreeing. Nothing failed when they diverged, because the test rule never asked the build anything. It asked a proxy, the product's own declared list, and the proxy had been accurate for exactly as long as nobody changed the thing it stood in for.
The fix, and where it went
The keep rule did not go into the demo app. It went into the base file that android_product prepends to every release build:
-keepclasseswithmembernames,includedescriptorclasses class * {
native <methods>;
}
Any class that declares a native method now keeps its name, and so do those methods. No product has to remember it, and a product added next year inherits it without knowing it exists.
This rule is not novel. It is part of Android's stock default configuration, which is why most Gradle projects never meet this crash: they inherit it. A Bazel build applies only the rules you hand it, and our base file had been written from scratch. It covered Kotlin metadata, annotations and generic signatures, and it did not cover this. Writing your own replacement for a platform default makes you the owner of everything the default used to do, including the parts nobody listed.
What the keep rule does not close
The rule closes this crash. It does not close the gap that hid it.
The new pixel test still self-instruments, and that is deliberate: it reads the app's own Window through PixelCopy, which a black-box test against a separately installed APK cannot do. Some tests need to be inside the process, and those tests cannot run the shipped binary.
The answer is not to give up in-process tests. It is to pair them with a black-box smoke on the shipped artifact, one that crosses each native boundary at least once. The Android tests for Rifff, whose feed items are Sigma programs, are already split that way: an R8 two-APK smoke on the release artifact, and an in-process runtime suite whose BUILD file says, in so many words, that it "does not claim to exercise R8 output". That sentence is the right instinct. It names which binary the suite is entitled to speak for.
A test makes claims about the program it ran. Before you count a green, check which program that was.
Takeaways
A passing test speaks only for the binary it built. Here, three suites were green and each ran something we do not ship: a JVM with no native library, a fake JNI on the host, and an APK that R8 never touched. Coverage of the right code in the wrong build is still coverage of the wrong build.
When two places answer the same question, make one of them ask the other. The test rule inferred "goes through R8" from a product's declared rules; a release change made R8 unconditional, and the inference went quietly stale. A proxy stays correct only until someone changes the thing it stands for, and that person is rarely looking at the proxy.
With name-based JNI, names are the native ABI. An obfuscator that renames a class with native methods has broken linking, and it shows up only after install, where host tests and unshrunk builds cannot see. Put the keep rule where every release build inherits it, not in the product that happened to crash first.
And if you replace a platform default, you own all of it. The stock configuration keeps native methods; our hand-written base file did not, because nobody wrote that line. Every rule you leave out of a file like that is a decision, made silently, until something crashes and makes it for you.