The instinct is to stop the bleeding
Paid search has a threshold, and the threshold is the whole point of having a tool. When cost per acquisition crosses the line you set, something is wrong, money is still moving, and the correct response is obviously to act.
Ours fired on a two-install aggregate. The day's cost per acquisition was several times our ceiling. The rule in our own playbook said: find the terms responsible, stop them.
We found them. Three terms, with five, two and one clicks respectively, and zero installs between them. We stopped all three, and we wrote the disclaimer into the receipt: at this sample size, the chance of falsely condemning a good term is somewhere between a quarter and a half.
Then we cut anyway.
Two of the three were fine
The next day, with the data settled and attribution backfilled, the picture was different in a way that is worth being precise about:
- One term was the best performer in the entire campaign. It had one click. It was stopped.
- One term was healthy. It had five clicks, and residual delivery before the stop took effect showed it converting well. It was stopped.
- One term genuinely had no signal, and stopping it was correct.
- And the threshold breach resolved itself overnight, without the intervention. It had been caused by attribution arriving late, not by the terms.
So the action was wrong twice over: it cut two healthy terms, and it was not even necessary. The decision everyone talked about afterwards — "we should have held one more day" — was the decision we had actually made correctly and then talked ourselves out of.
This is the shape of a bad automation story, and it is worth naming precisely. Nothing malfunctioned. The threshold fired, the analysis was correct, the arithmetic was correct, the recommended action was defensible under the rules as written, and a human being with full information chose to override them.
What actually went wrong
Three things, and only the third one is the interesting one.
The rule was permissive in the wrong direction. Our contract said cost per acquisition must be persistently above the ceiling. One day is not persistent, and we noted that — in the report, while writing the recommendation to cut. Knowing the contract and disregarding it in the same document is a thing that happens constantly.
Slicing made the sample worse, not better. The aggregate was built from two installs. We then took that aggregate apart and made three separate judgements on five, two and one clicks each. That is not a more careful analysis of the same evidence. It is three decisions made on less evidence than the one decision we started with, all pointing the same way, which makes the aggregate feel like a consensus when it is really one small number counted three times.
And the disclaimer was in the wrong place. This is the part that generalizes. We had computed that a quarter to a half of our cuts could be wrong, and we recorded that in the document whose entire purpose was to justify a destructive action. A probability attached to a recommendation does not restrain anyone. It is one more line to read past, on the way to the decision you were already going to make.
Knowing the odds is not the same as being stopped by them.
Make the instrument unable to report without the evidence
The fix was not a better rule. We already had a rule, and it was reasonable, and it had been overridden anyway.
The fix was to change what the measurement tool is able to say.
Every cost-per-acquisition number the measurement script prints now carries its sample:
{"metric": "cpa", "value": 12.00, "sample": {"installs": 2, "active_days": 1}}
Not in a note. Not in prose underneath. In the same object, so that it cannot be quoted without being seen. The reason is not stylistic: an analyst forgets to mention sample size, because mentioning it makes the conclusion less satisfying. A print statement does not forget.
The threshold check now reads the sample before it reports a breach as an action:
- Fewer than about five installs, or at most two spending days — this is noise, not a verdict. The alarm has already escalated to a human, which is the correct and sufficient response. The instruction is to hold and keep watching, and specifically not to make per-term keep-or-kill decisions off it.
- Term level: fewer than about ten taps — cannot distinguish a bad keyword from an unlucky one. At a quarter conversion rate, five clicks with no conversion is unremarkable; it happens roughly one time in four.
There is a second-order reason the first rule is correct, and it comes from how attribution behaves. Install counts only go up, retroactively, as the network reports them. A campaign whose spend is capped will therefore tend to improve on its own. A breach measured this morning is disproportionately likely to be a measurement artifact, and waiting costs one day of budget while a wrong kill costs a healthy term permanently — you will not learn that you were wrong, because a stopped term stops producing the data that would have told you.
Two smaller disciplines, both from the same pass
Spend-rate denominators. The tool reports how fast a campaign is burning its budget. That number divides by the number of days that actually spent, not the number of days in the reporting window — because a window query pads non-spending days with zeros, and dividing by the window turns "spent a full day's budget on day one" into a spend rate near zero. A denominator error does not just make a number wrong; it inverts the conclusion the number is used for, because a healthy saturated campaign reads as starved and gets told to raise bids.
Alert thresholds are not breakeven thresholds. The runaway ceiling we alert on is deliberately far above the number at which a campaign pays for itself. Setting the alert at breakeven is the intuitive choice and it is wrong: breakeven is crossed routinely during normal operation, so the alert fires most rounds, and a signal that fires most rounds carries no information. There is a reason a runaway stop and a profitability threshold are different numbers, and writing them as the same number turns a working alarm into background noise.
Takeaways
When a threshold fires on a thin sample, the breach is a question, not a diagnosis. The instinct to distinguish those is correct and worth acting on, and the failure mode is always the same: treating "not enough data yet" as "confirmed, and expensive to wait."
Watch for the analysis that makes the sample smaller. Slicing an aggregate into per-item decisions is not more rigorous. It is the same weak evidence, cut into pieces that are individually too small to support any conclusion and collectively look like agreement.
Put the sample size inside the measurement, not beside it. A probability written into the justification for a destructive action is a note. A number that cannot be printed without its evidence attached is a gate. If the person acting can forget the caveat, the caveat is not a control.
And price the two errors against each other explicitly. Waiting costs a day of budget; a wrong cut costs a term forever, and — because a stopped term stops reporting — you will never find out you were wrong. Any threshold that ignores that asymmetry will eventually be crossed by a number that resolves itself overnight.