More Words Made It Worse

The benchmark we built to be argued with

Our input engine ships a lexicon, a static language model and a long-tail expert model. Before any of that went near a device, we built an evaluation harness: a fixed set of held-out cases, scored top-1, top-3, top-5 and MRR, with latency measured across a whole pinyin insertion sequence rather than a single key.

The point of building it was not to produce a number for the marketing page. It was to make the trade-offs legible, including the ones we expected to lose. So the first real run compared our shipped data against the largest lexicon we could legally evaluate — a five-times-larger open one — on identical cases and an identical scoring configuration.

Five times the entries, and it ranked worse — on every accuracy metric. It also cost nearly five times the memory. Its one advantage was latency, and it bought that with a smaller, more targeted model on our side rather than with the bigger lexicon.

The sentence the harness was built to make possible is one we would not have believed without the table:

More entries does not automatically mean better ranking.

That is not a claim about our data being good. It is a claim about the shape of the problem: a lexicon is a set of candidate generators, and every entry it adds is also a competitor for the position you were trying to win. Size is not coverage, and coverage is not precision.

The experiment that changed nothing was the useful one

Quantizing the neural model from float to packed int10 and int8 was supposed to shrink it. The evaluation measured exactly what it was supposed to:

Identical. Every accuracy figure unchanged, MRR unchanged, latency within noise.

It would have been easy to present this as "quantization is free." The honest reading is narrower and more useful: quantization was free on this model and this workload, and that is a claim you only get to make because you measured it rather than assumed it. On a different model, a different architecture, or a task sensitive to tail behaviour, the same operation could move the number. The measurement is what licenses the sentence, and it does not license any broader one.

We shipped the quantized model. That part was easy. The valuable outcome was knowing which claim we were allowed to make.

A model that cannot promote itself

The most deliberate part of the system is the part that says no.

Our current model is not a candidate. Not because it failed, and not because someone forgot — its manifest states, as fields:

production_ready = false
promotion_eligible = false
use_policy = research-only
blind_data_used = false

Nine named readiness blockers are open, including that the frozen blind evaluation has not been run, that one chat source is research-only, and that the licence provenance of a lexicon is unverified. The training data volume target has been met, and the documentation is explicit that this is not a reason to promote. Volume is not evidence.

And the mechanism is structural rather than cultural. Three properties do the work:

The report cannot declare itself eligible. Promotion is not a field the training run sets; it is a verdict the promotion stage reaches. A report that claims eligibility is not a report you promote.

Signing is performed by someone else. The promotion path builds a detached-signature supervisor as a separate artifact and depends on it explicitly, so that OpenSSH signature verification cannot silently fall back to an unbuilt or unrelated helper. The thing that says "this is good" is not the thing that produced the candidate.

The bundle is checked against a fixed inventory at the end. The final production builder compares its own runnable inventory, item by item, against the trusted candidate report. Any drift in what is actually executable rejects the promotion — including drift introduced by editing a local manifest to claim readiness.

Every one of those is a place where a tired person at 6pm on a deadline could have taken a shortcut. The design assumes they will, and removes the opportunity instead of asking them not to.

A missing measurement is not a pass

The memory gate is the sharpest example of that discipline, and it is not about the model at all.

The release profile requires that both peak resident set size and peak physical footprint stay under 10 MiB. On macOS the system tool reports both. On Linux it reports only resident set size, so footprint is unavailable.

The policy does not average those cases. A complete, passing, four-arm ablation run on Linux produces a report that is nonetheless marked rejected — not weakened, not annotated, not approved with a caveat. The documentation states the reasoning plainly: Linux can finish the run and produce an auditable report, and that report still cannot be promoted, because the evidence required to make the decision was never gathered. Promotion should be completed on the target platform.

The same rule appears in a smaller place: where footprint is measured, it must be strictly greater than zero, because a reported zero is indistinguishable from a measurement that did not happen.

This is a design choice with a real cost — the result is that a legitimate improvement validated on Linux cannot be promoted until someone reruns it on macOS. We accepted that cost deliberately. A report that says "the metric I needed was unavailable" is worth more than one that substitutes the metric it had, because the substitution is invisible to every downstream reader and indistinguishable from a pass.

What the blind set refuses to claim

One last piece, because it is the same principle applied to a dataset.

The evaluation set for a model that will not be promoted is held under a custody protocol, and the attestation it produces states in its own fields what has not been established: that the blind set has not been proven unopened, and that semantic leakage has not been proven absent. It says outright that the hash used is not tamper-proof, because anyone who can rewrite the corpus can rewrite the manifest that describes it.

Meanwhile the gates that can be checked are checked, and they are strict: minimum blind set sizes, a fixed candidate depth, minimum domain and interaction coverage, and per-cell minimums that stop a suite from passing by testing the easy cases. On one platform, the export step fails closed — the trajectories cannot be released without selectors that would themselves leak the design.

The pattern across all of it is the same, and it is the reason we can write any of this down at all:

Report what you measured, report what you could not, and never let the second category be silently converted into the first.

A benchmark that only produces favourable numbers is an advertisement. Ours produced a table where our smaller data won, a quantization result that licensed exactly one sentence, and a model that will not promote itself no matter how the gates land.

Takeaways

Build the evaluation to be argued with. The reason to write a benchmark before you need it is so that a result you dislike is still a result you have to publish. Ours showed that five times the data scored worse, which is not a flattering finding and is the entire reason the harness was worth writing.

Distinguish "the change is free" from "the change was free here." Identical metrics before and after quantization licenses one claim about one model on one workload. Without the measurement you would have had to either skip the optimization or overclaim it, and both are worse than shipping.

Make the thing that judges unable to be argued with by the thing being judged. Every shortcut we could think of taking was answered by structure rather than by policy: eligibility is a verdict reached elsewhere, the signature comes from a separately built supervisor, the final bundle is diffed against a fixed inventory, and a manifest edited by hand fails at the last step. Reliance is a design material.

And refuse to convert absence into success. If the tool could not measure the thing the decision depends on, the decision does not get made — the report says so and stops there. That is inconvenient, and it is the reason anyone can believe the reports that do say yes.