Building the loop
App store optimization is normally done by taste. Someone looks at what competitors rank for, writes a title and a keyword string, waits, and adjusts. It is possible to get good at this, and it is also possible to spend a month making a change that makes things worse and never finding out why.
So we built the closed loop. Each app gets its own iteration directory holding its own measurement, its own history, and its own reasoning. A shared script does the mechanical part: fetch the ranking queries we actually appear for, pull the competitor set from each storefront, and classify every term. Another script scores a candidate term for intent match — whether someone searching it wants what we make — separately from value, how much exposure it would buy. The output is meant to be actionable, not descriptive: a list of terms to keep and a list to delete.
Then we ran it across ten products at once, and it told us to delete some of the best terms we had.
All three of the reasons it was wrong are worth writing down, because they are not about ASO. They are about what happens when a classifier's input is quietly shaped by the thing it is being asked to judge.
Bias one: the category Apple gave us is not the category we compete in
The classifier's strongest feature was simple. For each ranking query, look at what dominates the results. If that dominant category is not the category Apple assigned our app, call it a mismatch — we rank for a term whose results are in a different market, which usually means the ranking is an accident.
The feature is reasonable. The premise is wrong.
Apple assigns each app one primary category. That category is chosen for store navigation, and it frequently is not the category the app actually competes in. Our measurements across ten products:
- one photo-forensics app, filed under Utilities, while 57% of its real competitors were photo and video apps — the top results for its queries were camera and evidence apps in three different languages;
- one color-reference app filed under Photo & Video, whose competitors were actually in iPad Utilities (31% games, 31% iPad, 20% utilities);
- an allergy scanner filed under Medical whose competitors were 46% sticker apps;
- a time-tracking app filed under Finance whose competitors were 53% business apps.
Five of ten. For a feature built on "does the dominant genre equal our genre", the predicate dominant_genre != our_genre was true by construction for half the portfolio — so every query in those categories came back a mismatch, no matter how well it fit.
One product was the worst case and the best illustration. Of its 214 ranked queries, 87 were judged mismatches and only 16 survived as fit. Read at face value, the tool was saying: delete three quarters of the terms you rank for, including the ones taking you to position one.
Bias two: THIN is supply, not intent
The second bias is smaller and produced a larger share of the damage.
Searches are returned with a result count, and we bucket anything under ten results as THIN. The original reasoning was that a near-empty result set is a low-value target. What we had actually written was a measurement of supply — how many apps exist in that corner of the store — and we filed it next to intent.
Then THIN, mismatch and noise were collapsed into one boolean, "this term is misfit". Which means the best possible outcome and the worst possible outcome got the same mark:
Result one of two, where the other is a direct competitor, and we are the only serious answer in that entire search.
That is a small, defensible, high-intent win — the opposite of a bad keyword. It scored identically to a page full of unrelated tower-defense games.
Measured across ten products: 249 of 648 flagged tokens, 38%, were only THIN. The extremes are the story. One product had zero genuine mismatches among its 27 flagged terms — all 27 were THIN. Following the flag would have deleted that product's highest-scoring term in its highest-value market, plus the term it ranked first for in another.
The interesting part is that the value side of the scorer already knew the difference: THIN terms were scored at 0.3× fit value and mismatches at 0.1×. Only the boolean that said "delete this" was fooled. One field was right, the other was wrong, and the wrong one drove the deletes.
Bias three: the competitor set is learned only where we already rank
The third bias is the one that makes the first two self-confirming, and it is the most generalizable thing here.
To know whether a ranking query fits, you need to know who the competitors are. We derive that set from the queries we already rank for — which is the only source available. Then we use the set to judge the queries.
So the classifier's knowledge of the competitive field is proportional to our existing success. Where we rank well, there are many seeds, the competitor set is rich, and results look like neighbours. Where we rank poorly, there are few seeds, the set is thin, and almost anything looks like it belongs to some other market.
The measurement is stark:
| Our position | Queries | Mismatch rate |
|---|---|---|
| 1–10 | 541 | 0.0% |
| 61–200 | — | 30.2% |
A 30-point spread that has nothing to do with intent. Every deep-position mismatch verdict was, in substance, a verdict about our own ranking.
This one is not fully fixable — with no seed query there is genuinely no evidence. But it does change what the number means, and that is enough: a mismatch count concentrated at deep positions is evidence of weak ranking, and reading it as evidence of bad keyword selection sends you to delete terms when the actual problem is that nobody has heard of you.
What changed
Two changes, and the second one is the more broadly useful.
Stop requiring the seeds to look like us. A competitor's top results no longer have to be in our own assigned category to count as a competitor. If we rank for the query and the front of the field is internally coherent — one category holding at least half of it — the neighbours count, whatever Apple filed them under. One product went from 29 known competitors to 108. Its mismatch count fell from 87 to 38 and its fit count rose from 16 to 77.
Treat our own top-10 placement as evidence. If we are in the first ten results for a query, that is a strong signal the query is relevant to us; requiring it to additionally pass a category comparison was the feature causing the harm in the first place.
The THIN separation came out of the same pass, along with a rule worth stating generally: before deleting any term, look at the class of its best query, not at an aggregate flag.
And the check itself is now written into the rules rather than remembered. Before changing a threshold or the category logic, reconcile per product: what share of ranked queries fall in each category, and what category actually dominates the front page of the mismatch set. A product with too few competitors to reason about is marked unreliable, and its readings do not count as evidence until it has some ranking.
Takeaways
Any classifier whose training signal comes from where you already succeed will grade you on your weakness. That is not a bug you can tune away; it is structural, and the tell is a verdict rate that varies wildly with a variable you did not put in. If your mismatch rate is zero where you rank first and thirty percent where you do not, you are measuring your rankings.
Check the labels your features depend on. "Our primary category" is a navigation decision made by a store, not a competitive fact, and building a classifier on it imports every store's filing quirk. When a feature is this strong, its premise deserves more scrutiny than its threshold ever will.
Separate what you measure from what you mean. Supply, intent and exposure are three different questions, and merging them into one "bad keyword" flag means the cheapest wins get deleted alongside the real mistakes. When one field disagrees with another about the same row, trust the one with the more specific job — and check which one is wired to the destructive action.
And keep the check in the rules. Every one of these was found by reconciling two tables against each other on real data, which is a thing a person does and a tool does not. That check is now written down, with the reasoning, in the place where the next person who wants to lower a threshold will read it.