What we were trying to prove
Sigma's 3D scenes had sat behind an unchecked box in the spec for a long time. The engine side was complete, but it had never run on a real runtime, and no screenshot existed that proved anything was on screen.
Closing that box starts with a question that sounds trivial: how do you turn "it drew something" into an assertion that can fail?
"Make sure the frame is not empty" is the first answer. It is also wrong, in a specific and instructive way.
Two kinds of trivially true
A 3D scene does not own a window on Android. The host hands the engine a TextureView, and the engine paints the wgpu output onto it. Before wgpu draws its first frame, that rectangle holds nothing, and TextureView shows whatever the host page itself is painting behind it.
Which means the frame you may capture looks like this:
center = corners = rgb(78,158,160) distinct colours = 1
One flat tile. It is not empty in two different senses: it is not zero, and it is not white. It has colour.
That is exactly where verdicts go wrong. "Not empty" is true of it. "Not all-white" is true of it. They are not passing verdicts, they are verdicts that cannot fail, and a verdict that cannot fail is not information. It is a way for "we tested it" and "it works" to feel like the same sentence.
Bind the verdict to content
The fix is to make the verdict say what it expects to see, and to make that checkable. It expects a planet, centred, over a background. So:
The centre must differ from the corners. Take the mean colour of the middle 40% of the frame and of each 8% corner, and the Euclidean distance between them. A flat tile makes that zero. The rendered planet is far from the background: 135.7, measured. We set the bar at 30, which noise and compression jitter cannot reach, since those are single digits.
The corners must agree with each other. This closes the other false positive. If the whole frame were noise, the centre would also differ from the corners, but it would not be "a planet over a background". It would be noise. Four corners of one flat background should be near-identical, and we require their spread to stay under 20.
The frame must contain enough distinct colours. Quantise each pixel to five bits per channel and count the kinds. The ceiling is 32768, and jitter contributes almost nothing to it. A frame that cleared its background colour and drew no geometry at all has one or two kinds. We require 32.
Each one is a different face of "something was drawn": subject, background, detail. Together they are a statement about content, not a statistic about brightness.
Alive and empty are the same picture
There is also motion, and it is harder, because it needs two frames, and "something changed" is not the same claim as "the thing you care about changed".
The scene is otherwise static. It rotates, and the rotation is advanced by the render loop, not by engine state, which is the part that makes it a clean signal: two frames of a stopped loop are byte-identical. So we compare two frames captured 900ms apart.
But we do not settle for "the whole frame changed by at least 0.5% of pixels". Whole-frame change can come from a corner twitch, an exposure shift, a recompression. What we want to show is that the planet turns, so we compare only the centre region and require more than 1% of its pixels to differ.
Centre rather than edges, because the edges are the background, and the background does not move. The predicate is therefore not "the picture changed" but "the thing that should be moving is moving".
Taking the frame is its own set of traps
Then the test went red three times, for reasons that had nothing to do with verdicts.
Starting the scene. The helper we used to launch an activity waits, internally, for the main looper to have no pending messages. This scene posts a frame every vsync, forever. So the main looper never goes idle, and the launch never returns. Breadcrumbs stop right after the call, no timeout rescues it, and the whole instrumentation run hangs until the harness kills it. We launch directly instead and poll the lifecycle monitor. The polling happens on the test thread; the main thread is touched exactly once, to fetch the window and the rectangle. Taking pixels never touches it again.
Reading pixels back. TextureView's own bitmap accessor is a synchronous GPU read-back, and it queues behind the swap chain that the main thread is busy feeding. So the test doesn't fail, it disappears into a queue. PixelCopy goes through the compositor instead, asynchronously, and hands the frame back and lets go. That is the other reason this test took a callback on its own HandlerThread: the main thread is running the render loop, and on software Vulkan a frame takes about 90ms, so a callback queued on the main looper is starved there too. It has to be on a thread of its own.
Waiting is its own small discipline. The harness gives you a condition waiter, and its built-in assumption is that you are waiting for a condition that stays true: wait for it, then assert. That assumption is backwards here, because the thing we want is not steady state. GPU initialisation and the first texture upload happen after the surface is attached, and a frame captured before that is an empty texture. So the test polls a frame every 250ms until it has content, and the assertion runs on the frame it got. A test that fails because it grabbed too early is a flake; a test that waits is not. Each poll logs the frame's pixel summary, and when the wait times out the assertion message carries the last frame's numbers, so the failure says what it saw without anyone reproducing it by hand.
The shape
Strip out Android and the GPU and this is one idea:
An assertion either names a fact about content, or it is satisfiable by a trivial one. Decide which you are writing.
"Not empty", "something changed", "looks right" are all claims about a statistic of the whole frame, and whole-frame statistics are the easiest thing in the world to satisfy accidentally. With an empty view, with a background fill, with a surface that has not received its first frame yet, they pass. What they should say is a conjunction: the subject differs from the background, the background agrees with itself, the subject is the part that moves.
The price is that you have to know what success looks like. That is most of the work, and it is also the part that pays: this test now fails when the camera points at nothing, when the model does not upload, when the renderer clears and returns early, when someone edits the scene so that the planet fills the frame and touches the corners. Each of those used to be invisible.
Two more things earned their place the same way. The first frame is read from the host, not from the guest, because on an Apple Silicon emulator the guest-side screencap cannot read a GPU-composited frame back and returns constant pure white, byte for byte the same size every time, even for a perfectly ordinary 2D screen. A white screenshot there is a broken capture, not a broken app. And the harness runs on software Vulkan, which is slow, roughly 90ms a frame, so every timeout in this test is a multiple of that rather than a guess.
Takeaways
Before you trust a verdict, ask what would satisfy it. If the answer is "an empty view, or a background fill, or a surface that has not received its first frame", then the verdict is a description of the failure case, and it will pass through every one of them on the way to the real failure. Name the region and the relationship instead: the centre is a planet, the corners are background, and the two are 135 apart.
Wait for the condition before you assert on it, and log what each attempt saw. A test that fails because it was impatient is the worst kind of test, because it will teach people to re-run it. A test that polls until the frame has content costs one loop and never flakes.
Know where you are standing before you take a screenshot. On an emulator, the host is the only honest camera. Software rendering is slow enough that every timing constant in the test is a multiple of the frame time, not a round number someone liked. And a capture path that queues behind the thread doing the rendering does not fail loudly, it disappears, which is worse.
Write down what the test is not allowed to check. Not "not all-white", not a whole-frame statistic, not a single frame read at a fixed moment. Those are the shapes that will come back, and writing them down is cheaper than rediscovering them on a red build at midnight.