The Correct Fix That Turned the Feature Off

When context makes it worse

Our keyboard ranks candidates. A user reporting a bug had taken a photo and typed zhanshi ("display"). With no preceding context, 展示 — the word they wanted — ranked first. With the context they had typed, the same word ranked ninth, behind 战, 战术, 转世, and five others.

The ordering with context matched their screenshot exactly, which is the part that made it tractable. We could reproduce a user's mistake on a machine on our desk, which is rarer than it sounds and worth being grateful for.

A ranking that inverts when you add information is not a tuning problem. Something in the scoring treats context as a penalty. So we went looking for the root cause, and got it wrong several times in instructive ways.

What reading the code gave us

The first thing the candidate dump showed was a hard-coded branch. Single characters with no context had their language score set to 0.0, while multi-character words went through the real path and received an actual negative log-probability. Both landed in the same candidate list and were compared on the same number, so the language term had quietly become a length penalty — it only ever moved multi-character words.

That is a real defect. It is a defensible explanation for a symptom. It is also not the cause of this bug, and we found out the expensive way:

Unify the baseline — make the decoder also skip the language term without context — and the eleven failing cases did not move. Not one metric changed: top-1, top-3, top-5 and MRR all identical.

The tempting conclusion was "this defect doesn't intersect our test cases." That conclusion was wrong too, and the correction came later: the cases could not see the defect because their expected answers happened to occupy the one complete candidate each slate had. The zero change was not evidence of absence. It was evidence that the test could not observe it.

The experiment that changed everything and measured nothing

The second hypothesis was the scoring function itself. We replaced the language term with mutual information — the standard fix for exactly this shape of problem, where a common word wins purely on frequency.

Eleven cases reordered. One moved from fifth to fourth. That is a change that rewrites the entire ranking and buys nothing measurable, and it is worth naming clearly because the instinct after a null result is to try again with different constants:

A change that reorders everything and moves no metric is a finding, not a tuning miss.

What it told us was that the ordering was not what the metric was sensitive to. We were tuning a term whose contribution was already too small to matter, and every experiment in that family was going to return the same number.

Two diseases with one face

The third hypothesis was the corpus, and this one was half right — which is the most useful kind of wrong.

Our language model is trained on a large archive of an anonymous Chinese discussion forum. The users of our keyboard write engineering Chinese. "的时候" followed by 战士 really is more likely than 的时候 followed by 展示 in that corpus, and the model was not broken; it was answering a different question than the one we were asking.

We could quantify how far off we were, which is the step that made this hypothesis testable rather than merely plausible: the three failing cases needed the conditional probabilities to move by between 1.6 and 5.2 bits, and a bigram that genuinely occurs in-domain differs from one that does not by more than ten bits. So the gap was closeable, and the corpus was the right lever.

Then we tested a conversational corpus instead, and it repaired none of the three. The reason is the part worth keeping. Two of the three cases have no bigram in the corpus at all — the words are not even in the top-20,000 frequent phrases — so the model falls back to unigrams and ranks purely by frequency. Those cases need coverage. Only one case had a bigram that was estimated badly, and that one needs better estimation.

Swapping the corpus improves estimation and cannot manufacture coverage. Two diseases, one symptom, and no single fix for both — which is exactly why a plausible hypothesis produced a clean null.

The actual cause, and the one-line fix

The cause was arithmetic, and it had been sitting in plain sight the whole time.

The transition score was an unsmoothed maximum-likelihood estimate: P(word | previous) = support / count(previous). With a support of one — the transition exists exactly once in 58 million characters — that estimate is not "this pairing is rare." It is noise, and it scored 3.46 bits below the word's own unigram probability.

So context actively demoted a candidate:

The competitor with no evidence for the context fell back to its baseline and beat the one with a single sighting of evidence. The user's screenshot was this table.

The fix is one rule, in the one function that can see both quantities:

Context may refine the baseline, never worsen it. Take max(conditional, marginal).

It is not a fudge factor and it is not a weight. It is a statement about what context is for. A noisy estimate should be ignored, not believed; and "ignore it" must mean "fall back", not "subtract". Shipping it, with the shipped corpus and no data changes at all, moved 展示 from ninth back to first.

We raised the gate floors in the same change — top-1 from 0.36 to 0.545, MRR from 0.58 to 0.702 — and reverting the one line turned the gate red. That is the only way a fix like this is protected: make the floor describe the fixed world, not the broken one.

The fix that turned the feature off

Two days later, with the full mixed corpus built, the fix was still in place — and the gates read identical to the shipped model. Every number the same. Context was not doing anything at all.

The rule was still correct. It was being applied to two quantities that were not on the same scale:

marginal = log2( c(word) / c(most-frequent) )     ← denominator 35.6M
conditional = log2( c(prev, word) / c(prev) )

The marginal used the most-frequent word as its denominator while the unigram total is 752 million, which inflated every marginal by about 4.4 bits. Since c(prev, word) ≤ c(word) always, and the context here is 的 — itself the most frequent word — the conditional score could not catch up under any circumstance. So max returned the marginal every single time, and the mechanism that had just been fixed was silently inert.

Nothing crashed. The gates were green. The feature was simply off, wearing the fix's name.

The real fix put both terms on one denominator and interpolated:

unigram = log2( (1-λ) · c(w) / N )                          N = total unigram count
bigram  = log2( λ · c(prev,w)/c(prev) + (1-λ) · c(w)/N )     λ = 0.9

The interpolated form is provably ≥ the unigram form, so context can lift a candidate and can never push it below where it already stood — the original rule survives, now as a mathematical property of the scoring function rather than a defensive clamp on top of it.

Measured on the full corpus, the gains landed only where the evidence existed:

And the contextual gate, which had been reading 0.636 / 0.818 / 0.756, moved to 0.636 / 0.909 / 0.768 — the no-context half deliberately unchanged.

Takeaways

Distrust a root cause you found by reading code. Every hypothesis in this hunt was supported by the source, and the two most confident ones were wrong or unobservable. The candidate table is not the scoring function; it is a projection of it, and a defect that a table cannot show you is not thereby absent.

Read a null result carefully, in both directions. A change that reorders everything and moves no metric has told you the ordering was not what you were measuring — stop tuning that family. And a test that cannot see a defect is not evidence the defect is absent; before drawing that conclusion, ask what the test is blind to.

Separate "we estimated it badly" from "it was never there." They look identical from the outside — both surface as a wrong ranking — and they need opposite remedies. Quantify before you swap anything: measuring how far the wrong answer had to move to become the right one is what turned a plausible hypothesis into a testable one.

And check your fix on a second dataset. A rule that is correct in principle can be inert in practice because the quantities it compares are on different scales, and the symptom is that the gates pass identically forever. Verify the fix by confirming the mechanism is doing something, not by confirming the number did not move.