While building a marketing mix model for a subscription app client, we were asking: how much revenue does Google Ads actually bring in? Google’s own reports looked good, but we know not to blindly trust them. So we ran a geo holdout test.
We paused Google App Campaigns (UAC) in 28 randomly chosen US states for six weeks and kept them running in the other 23. Going in, the working belief was that most first-purchase revenue came from Google. It didn’t. Revenue in the paused states fell only slightly more than in the others, and the gap was small enough to be chance.
Here is how we ran the test, what it showed, and what we decided.
Why can’t platform ROAS tell you if Google Ads works?
Platform ROAS counts every purchase the ads touched. App Campaigns are built to find the people most likely to buy, and some of them would have bought anyway. So the better the targeting, the better the reported ROAS looks, whether or not the ads changed anything.
In other words, platform ROAS answers “which conversions did Google touch?”, not “which conversions would have disappeared without Google?” Only an incrementality test answers the second question, and that is the one budget decisions should depend on.
How we designed the geo holdout test
Google was paused in 28 states and kept running in 23. We randomized by US state, and included all 51 (the 50 states plus DC). Revenue varies a lot between states, so we used stratified random assignment: states were grouped by revenue level, then assigned at random within each group, so both sides got a similar mix of big and small markets.
Before starting, we checked that both groups had been following similar revenue trends over the five months before the test. They had, so any gap that opened up during the test could be read as the effect of the pause.
| Unit | US state, stratified random assignment |
|---|---|
| Groups | 28 states paused, 23 states running |
| Baseline | About five months before the test |
| Test | Six weeks with Google App Campaigns fully paused in the paused states |
| After | Ads back on everywhere, used as a second check |
| Outcome | Daily first-purchase revenue per state (lifetime and subscription) |
What did the holdout test show?
Before the test, we wrote down what would count as a real effect. The working assumption was that almost all first-purchase revenue came from Google, so if revenue in the paused states dropped by 50% or more, we would know Google mattered a lot.
What happened instead: both groups declined by a similar amount.
During the six weeks, revenue fell in both groups, by around a fifth. Something market-wide or seasonal pulled everything down at once, and when the test ended, both groups recovered together.
The paused states fell slightly further. That small extra drop is the estimated effect of pausing Google: roughly 3–4% of first-purchase revenue, depending on how you measure it. It was not statistically significant (p = 0.50). A gap this size would show up about half the time when you split the states at random with no ad pause at all.
The range of effects the data is consistent with (the 95% confidence interval) runs from −15% to +16%. Even the top of that range is about a third of the 50% drop we would have seen if Google were driving most of the revenue.
How do you avoid overstating a geo test result?
A geo test can be set up well and still give a misleading answer if it’s analyzed the wrong way. Here is the trap. The data had thousands of rows: 51 states, each with a revenue number for every day. Run a standard regression on all of them and you treat every state-day as an independent piece of evidence. It isn’t. A state’s revenue on Tuesday is mostly the same state behaving the same way as on Monday. Counting both as separate evidence overstates how much you know, and makes the result look far more certain than it is.
We randomized at state level, so we analyzed at state level, with a reshuffling test (randomization inference):
- Calculate the real difference-in-differences: the change in average daily revenue from before to during the test, paused states minus running states.
- Shuffle which 28 states are labeled “paused”, keeping every revenue number where it is, and recalculate. Repeat 2,000 times.
- See where the real result falls among the 2,000 made-up ones.
If the real result sits comfortably inside the range randomness produces on its own, you can’t tell it apart from noise. Ours sat right in the middle. The method makes no assumptions about how revenue is distributed.
Small methodological detail. Big impact on the conclusion.
A second check: did the paused states catch up when ads came back?
If pausing Google really caused revenue to drop, the paused states should fall behind the others while ads are off, then catch up once ads are back on. A seasonal coincidence has no reason to follow that schedule. So we ran the same reshuffling test, comparing the test period with the weeks after ads resumed. Again, the change was well within what chance alone produces.
All three agree: the difference is well within what random assignment alone produces. Significant would mean below 0.05.
What did we decide?
We cut Google spend in the US by more than half. We did not switch it off. “No detectable lift” isn’t the same as “no lift”, and the test has limits:
- First purchases only. Renewals and longer-term value weren’t measured.
- Six weeks. It shows what Google added during the test, not its long-term impact. Measuring that needs a different framework.
- One experiment. It ran once, in spring. Advertising can work differently in other seasons, and a repeat would give stronger evidence.
- 51 states. That is a small sample for this kind of test, which is why the range of possible effects is wide.
So we cut Google spend rather than turning it off. The test also measured the spend level running at the time, and money above that level usually earns less than the first dollars do. So the cut likely cost even less than the test suggests.
One more detail: the account had one of its best reported Android ROAS periods while this test ran. When we compared like for like, the test’s best estimate of what Google actually returned was about 40% lower than what attribution reported.
Platform ROAS and incrementality tell different stories, and it’s worth looking at both.
Why use a model and a test, not just attribution?
This test started inside a marketing mix model, and its result went back into it. Each way of measuring ads has its own job:
Each platform credits itself and points you its way. It assigns credit; it doesn’t measure impact.
An independent direction, read from your own data across every channel at once.
Move in one direction in some markets only, and check whether the compass was right.
Real-world impact is messy, so we keep learning in a loop: read the compass to see where money works, form a hunch, test it by pausing ads in some markets, then recalibrate the model and test the next idea. No single number is perfect, but the model and the tests check each other, and each round makes the next decision safer.
Lessons for your own incrementality test
- Platform ROAS shows what the platform touched, not what it caused. We saw the same thing when we tested brand traffic in App Campaigns.
- Decide in advance what would count as a real effect. Write the threshold down before the test, so the result can’t be reinterpreted afterwards.
- Analyze at the level you randomized. Randomized by state means analyzed by state.
- Use geo tests to calibrate everything else. They don’t depend on user-level tracking, which is why we recommend them in our 2026 app tracking guide.
- Check that the difference disappears when the ads come back on. Real effects follow the schedule. Coincidences don’t.
FAQ
What is an incrementality test?
An incrementality test measures how much of a result an ad channel actually caused, by comparing a group exposed to the ads with a group that isn’t. Platform reports count every conversion the ads touched. An incrementality test counts only the ones that would have disappeared without them. Where you can’t pause a channel, marketing mix modeling estimates the same thing from historical data.
What is a geo holdout test?
A geo holdout pauses ads in a random set of regions while they keep running elsewhere, then compares revenue between the two. Here that meant 28 of 51 US states for six weeks, against five months of baseline. It needs no user-level tracking, so ATT and cookie loss don’t affect it.
Why is platform ROAS different from incremental ROAS?
Platform ROAS credits the ads with every conversion they touched, including people who would have bought anyway. Incremental ROAS counts only the revenue the ads added. In this test, the account had one of its best reported Android ROAS periods while the holdout found no detectable lift.
What counts as a meaningful effect in an incrementality test?
It depends on what you believe going in, and it should be set before the test. Here the working belief was that Google drove most first-purchase revenue, so we set the bar at a 50% drop. The result was 3–4%, with a plausible range up to about 16%. Pick the threshold based on the decision it should inform, not after you see the numbers.
Next up
Same app. Same channel. The UK instead of the US. Very different result. We’ll write that one up next.
The team behind the test
Chinmay KulkarniData Scientist · designed the test and ran the analysis
Markus SeppamApp & Business Pro · ran the Google campaigns and the holdout