Could our geo test even see Reddit's lift? The formula said yes, at 8.5%. The simulations said not even at 25%.

A subscription app client wanted to know whether Reddit ads drive first-purchase revenue. The budget was small. So before launching, we asked a different question: if Reddit does work, could our test even see it?

We designed a geo experiment across 205 US DMAs, chose two standard estimators, and checked their power two ways: with parametric formulas and with non-parametric simulations. The formulas said we could detect a lift as small as 8.5%. The simulations said that even a 25% lift would be caught in fewer than 1 in 10 experiments.

Here is how we designed the test, why the methods disagreed, and why we sided with what the data actually supports.

Why does a small budget change the question?

A small budget buys a small lift. A small lift is hard to separate from the normal day-to-day noise in revenue.

If the test can't see an effect of the size the budget can realistically produce, the outcome is decided before launch. You spend the money, the result comes back "not significant", and you've learned nothing. Worse, "not significant" gets read as "Reddit doesn't work".

So the first job wasn't to measure Reddit. It was to find the smallest lift this test could reliably catch: the minimum detectable effect, or MDE. Only then can you size spend so the campaign has a fair chance of proving itself.

How we designed the test: DMAs, not states

We randomized at the DMA level, not the state level. More independent units means more statistical power.

A state-level test gives you 51 units at most. Splitting the US into DMAs gave us 205. Each DMA is its own small market, so the test gets four times as many independent comparisons from the same country.

What should you stratify on?

Revenue varies hugely between markets, and a pure coin flip could put most of the big DMAs on one side. Stratifying fixes that: group similar units first, then randomize within each group. The question is what to group them by.

• Only pre-treatment variables. A causal diagram (DAG) is useful here as a filter. Anything the ads could influence, like post-launch traffic or clicks, sits downstream of treatment and must stay out.
• The strongest predictors of the outcome. In a randomized test you don't need to block confounders, because randomization already handles them. Stratification is there to remove noise, so pick what best predicts the outcome.
• Usually, the outcome's own history. Here, each DMA's pre-launch and post-launch first-purchase revenue correlated at 0.94. No other covariate comes close to that.

So we grouped DMAs into 5 strata by pre-launch first-purchase revenue and randomized within each. Finer strata, down to matched pairs, can add a little more power. With about 200 units and a heavy-tailed outcome, a few wider strata are more robust.

Both choices served one goal: give a small budget the best chance of being detected, without buying that sensitivity with extra false positives.

Two ways to estimate the effect: DiD and ANCOVA

Once the ads run, you need an estimator: a rule that turns revenue data into one number for "how much did Reddit add?" Both standard choices compare treatment DMAs with control DMAs. They differ in how they use the pre-launch data, and so in what they assume.

Difference-in-differences (DiD) compares how much each group changed from before to after launch:

\text{Effect} = (\bar{Y}^{T}_{\text{post}} - \bar{Y}^{T}_{\text{pre}}) - (\bar{Y}^{C}_{\text{post}} - \bar{Y}^{C}_{\text{pre}})

It cancels out any fixed gap between the groups, such as the treatment side simply having smaller markets. Its key assumption is parallel trends: without the ads, both groups would have moved together.

ANCOVA regresses each DMA's post-launch revenue on its treatment status and its pre-launch revenue. Rather than subtracting the whole baseline, it learns how much of the past carries into the future and adjusts by exactly that much. When pre and post are strongly correlated, it is usually the more precise of the two.

We ran both because they rest on different assumptions. When they agree, that's reassuring. When they disagree, it tells you which assumption the data is breaking.

Did the two groups start out comparable?

Mostly, yes. Before launch, the treatment DMAs had about 7% less total first-purchase revenue than control. Stratification narrows gaps like this but can't remove them entirely. That's why the analysis compares each group to its own baseline rather than comparing raw totals.

What matters more is whether the groups were moving together. A difference-in-differences or ANCOVA read only works if treatment and control would have tracked each other without the ads. We checked this two ways:

• Trend divergence: were the groups already drifting apart before launch? No (p = 0.58).
• Split-half placebo: pretend launch happened halfway through the pre-period, and see if a fake "effect" appears. It didn't (p = 0.85).

Both checks passed. Any gap that opens after launch can be read as the effect of the ads.

The revenue was heavy-tailed, even after taking logs

Before launch, the largest DMA brought in about 25 times the first-purchase revenue of the median DMA. That's not a few big cities on a smooth curve. It's a distribution where a handful of markets carry a large share of the total.

To measure how extreme the tail is, we estimated the tail index (alpha) with the Hill estimator, the maximum-likelihood estimate for a Pareto-type tail. The threshold that matters is 2. At or below it, the distribution has no finite variance in theory, and averages can be swung by one or two extreme units.

Since we already knew revenue was skewed, we also ran everything on log revenue. Logs compress big markets and are the standard fix for skew. It helped, but not enough:

On both scales, the data itself rejects the bell-curve shape that most textbook power formulas assume. That one number shapes everything that follows.

Parametric vs non-parametric: what's the difference, and why it matters

Every power analysis needs an estimate of how noisy the outcome is. There are two ways to get it.

Parametric methods assume the data follows a known shape, usually a normal distribution with a stable, finite variance. From that assumption, a formula gives you the standard error, the power, and the minimum detectable effect (MDE). They are fast and precise, as long as the assumption is true.

Non-parametric methods don't assume a shape. They let the data's own distribution answer the question. A bootstrap simulation resamples real DMAs, injects a known lift, and counts how often the test catches it. A permutation test reshuffles the treatment labels thousands of times to see what pure chance looks like.

Why it matters: when the data really is close to normal, both approaches agree, and the formula just gets there faster. When it isn't, the formula doesn't warn you. It returns a precise-looking number built on an assumption the data has already broken.

One nuance: our bootstrap sits in between. It resamples the data non-parametrically, but still decides significance with a formula-based (HC3) standard error. The permutation version uses no formula at all.

What the methods said, on raw and log revenue

We simulated lifts of 5%, 10%, 15%, 20% and 25%, and measured power: the share of simulated experiments in which the test correctly detects the lift. The usual planning target is 80%. The MDE is the smallest lift that reaches it.

Read across the rows and the split is stark.

The formulas got more optimistic. After the log transform, closed-form ANCOVA says it can detect an 8.5% lift. That's small enough for a modest budget to produce, which makes it the most tempting number in the table.

The simulations didn't budge. On both scales, every non-parametric method stayed below 10% power at every lift we tested. Even a 25% lift would be caught in fewer than 1 in 10 experiments. The target is 8 in 10.

So which one do you plan around?

Checking the false-positive rate first

Before trusting any power number, check how often the test reports an effect that isn't there. We simulated the experiment 500 times with no lift injected. At the usual 5% significance level, a well-calibrated test should raise a false alarm about 5% of the time.

On raw revenue, ANCOVA raised a false alarm in almost 1 experiment in 5, more than three times the intended rate. Its standard errors were too small for data this heavy-tailed. A test that underestimates noise will look both more sensitive and more trigger-happy at the same time.

The log transform fixed the false-alarm rate: both estimators land close to 5%. This is the interesting part, because it would be easy to stop here and accept the 8.5% MDE.

But a calibrated false-positive rate is not the same as a valid power formula. The 8.5% figure still comes from a closed-form calculation that assumes finite variance. On the log scale the tail index is 1.80, below 2. The data falsifies that assumption. Using the 8.5% would mean making a strong assumption the data doesn't support.

The non-parametric methods don't need that assumption. And they agree with each other: bootstrap and permutation power stayed within a few percentage points at every lift, on both scales. Two methods built on different foundations giving the same answer is what made them worth believing.

The uncomfortable conclusion

As designed, with 205 DMAs and a 28-day window, this test cannot reliably detect any lift between 5% and 25%, on raw or log revenue. The only methods that say otherwise are parametric formulas, and the data falsifies the assumption they rest on.

Whether to launch anyway is not a statistical question. It depends on what each kind of mistake costs you:

• A false positive: concluding Reddit works when it doesn't, and continuing to spend on a channel that adds nothing.
• A false negative: concluding Reddit doesn't work when it does, and cutting a channel that was paying off.

The tempting route is parametric ANCOVA on log revenue. Its closed-form MDE is 8.5%, the lowest in the table. In simulation, its false-alarm rate was 5.8%, close to the 5% target. On raw revenue, the same estimator's simulated false-alarm rate was 18%. If you are comfortable assuming log revenue is close to normal, you could size the budget around an 8.5% lift and go ahead.

That is a legitimate choice about risk tolerance, but it doesn't change the data. The tail index on log revenue is 1.80, below 2, so the assumption behind the 8.5% doesn't hold. If the simulations are right, the bigger risk is a false negative: a lift the formula says you'd catch would most likely come back not significant.

Lessons for your own geo test

1. Run the power analysis before the budget is set. A test that can't see the lift your budget buys is decided before it starts.
2. Randomize at the finest level you can target. DMAs gave us 205 units where states would have given 51.
3. Stratify on the strongest pre-treatment predictor of the outcome. Use a DAG to rule out anything downstream of treatment; usually the outcome's own history wins.
4. Choose estimators by the assumptions you can defend. DiD needs parallel trends; ANCOVA needs well-behaved errors.
5. Measure the tails before trusting a formula. A tail index below 2 means parametric power calculations are off the table, even after a log transform.
6. Check the false-positive rate before the power. A test that underestimates noise looks more sensitive than it is.
7. Trust agreement between methods that don't share assumptions. When the formula disagrees with every simulation, believe the simulations.

FAQ

What is a minimum detectable effect (MDE)?

The smallest true lift a test will detect reliably, usually 80% of the time. A smaller lift can be real and still come back "not significant".

When should I use DiD and when ANCOVA?

DiD when you're confident the groups would have moved in parallel and want robustness to fixed gaps. ANCOVA when pre- and post-period outcomes are strongly correlated and its error assumptions hold. Running both is a useful cross-check.

Doesn't a log transform solve heavy tails?

It helps. Here it brought the false-positive rate back to about 5%. But the tail index on log revenue was still below 2, so the parametric power formula remained unsupported.

What is permutation inference?

A test that reshuffles which regions are labeled "treatment" thousands of times and recalculates the effect each time. If the real result looks like one of the shuffled ones, it can't be told apart from chance. It makes no assumptions about the shape of the data.