Before running a Reddit geo test, we checked the smallest lift it could reliably detect. A standard power formula and our simulations disagreed (22% vs 26% in one historical window). We first blamed skewed purchase data. The investigation pointed elsewhere: when ad lift lands unevenly across states, the test's uncertainty grows once the ads are running, and the formula doesn't account for that. In our stress tests the formula stayed near 18 to 20%, while the simulated threshold ranged from about 19% to over 60%. The lesson: test several plausible patterns of where lift could land before trusting a single MDE.
We were considering a Reddit geo experiment because a null result would be costly in either direction: we could cut a channel that truly worked, or keep spending because a lucky significant estimate overstated the effect. When our analytical MDE and simulated MDE disagreed, we tried to understand why. We first suspected skewed purchase data. The investigation led somewhere more useful.
Yet one randomized experiment reports “not significant.” A real effect can be missed when an experiment lacks power.
And among the noisy estimates that do cross a significance threshold, effect sizes can be exaggerated. This is a selection effect, often called the winner's curse.
This opening is hypothetical. We are not reporting the results of an actual Reddit incrementality campaign.
The moving teal dots are illustrative response pulses: they show that in a real test, response may appear unevenly across treated states. They are not observed data.
The teal pulses are purely illustrative. Think of them as small animated markers for hypothetical response hotspots after treatment, not real company data and not estimated lift for any state.
A non-significant result is not the same as no effect.
An underpowered test may fail to detect an effect that matters to the business. It can also produce unstable estimates: if only the largest noisy estimates cross a significance threshold, those selected results may overstate the true effect.
Low power does not, by itself, imply an increased false-positive rate. These are different risks. We wanted to understand the first one before committing to a Reddit experiment.
A fixed illustrative true effect of +20%; estimates are deterministic, normally distributed examples of 100 repetitions. Two-sided 5% z threshold. The selected average is not an estimate from the Reddit study.
Our two answers did not agree.
We proposed randomizing 25 of 51 US state-level geographies to treatment, using 56 days of pre-period data and 28 days of treatment-period data.
We chose ANCOVA: a regression that estimates the post-period treatment difference while adjusting for each state's pre-period purchases. It was more precise than unadjusted comparisons and difference-in-differences in our historical randomization checks.
Minimum detectable effect (MDE) is the smallest true lift that reaches a specified power, here 80%. A mathematical approximation gave one number; repeated simulated assignments gave another.
Raw-count ANCOVA, one earlier historical-window comparison. Curves are illustrative interpolations anchored to the reported MDEs, not stored historical point estimates. Analytical here means a fixed-null-HC3-SE normal approximation.
What did the power formula actually assume?
Before simulating treatment outcomes, we needed a baseline prediction: if a campaign really caused a given lift, how often should our geo experiment detect it? Our analytical benchmark uses the fitted no-treatment HC3 standard error and a normal approximation to answer that question.
For each hypothetical treatment/control assignment, regress post-period average daily purchases on treatment assignment and pre-period average daily purchases. HC3 is a leverage-adjusted, heteroskedasticity-robust estimate of the treatment coefficient's standard error.
Convert the chosen aggregate percentage lift into the expected change δ in this regression's additive treatment coefficient. Those are not automatically the same quantity, especially with unequal state sizes.
Assume the treatment estimate is approximately normal, centered at δ, with uncertainty fixed at SE₀. Reject zero if its absolute z-statistic exceeds 1.96 (two-sided 5% test).
Φ is the standard normal cumulative distribution function; δ/SE₀ is the assumed signal-to-noise ratio. We invert this predicted power curve to find the lift reaching 80% power: the analytical MDE.
Keep the assumed signal fixed and increase the uncertainty used in the formula.
The illustration uses an arbitrary fixed signal of 2.8 relative units, not company data or the observed Reddit study. Increasing SE lowers the predicted chance of crossing the significance threshold.
We did not compute every result from this formula. We compared its predicted power and MDE with empirical rejection rates from randomized synthetic experiments, where post-treatment outcomes and HC3 standard errors were recalculated. The later CI study likewise used post-treatment HC3 intervals. The fixed-null-SE confidence interval shown there is an illustrative diagnostic of the planning assumption, not standard practice for reporting experimental results.
Was the distribution of purchases to blame?
Counts were full of zeros and occasional large observations. We suspected that dispersion, skewness or kurtosis might make a normal-reference analytical power calculation unreliable.
Variance describes spread; skewness describes asymmetry; excess kurtosis describes the importance of unusually large deviations relative to a normal distribution.
But the shape of individual purchases is not the same as the sampling distribution of a treatment-effect estimator. We needed to isolate the mechanisms rather than assume one caused the other.
Wider distributions place more outcomes far from their average. Curves are illustrative probability-density shapes, not empirical state-day histograms.
Re-create the purchase process. Then turn up the noise.
We built a Gamma–Poisson simulator: historical state shares and national trends establish an expected rate; a Gamma multiplier varies that rate; a Poisson draw produces nonnegative integer purchases. The dispersion parameter controls extra variability beyond an ordinary Poisson model.
Before stress-testing it, we compared simulated and historical distributions, zero rates, temporal behavior and randomization uncertainty. The fit was useful, though imperfect: not proof that every real-world dependency was reproduced.
Stylized equal-mean count model, not measured experimental SNR. Here SNR = fixed effect / illustration-scale uncertainty. Actual ANCOVA SNR also depends on geography, regression adjustment and treatment pattern.
What connects dispersion to power?
If outcomes fluctuate more while the expected effect stays fixed, the effect becomes harder to distinguish from noise.
Signal-to-noise ratio (SNR) compares the expected treatment effect with the standard error of its estimate. Higher SNR usually means higher power.
We used five dispersion settings (0, 0.05, 0.20, 0.50, 1.00). The model's count variance is mean + dispersion × mean². Raising dispersion also tended to raise skewness, kurtosis and estimator noise, but those things moved together.
We treated the simulator as a laboratory, not a replica. It was designed to be realistic enough to stress-test the experiment, while staying simple enough to manipulate.
Higher dispersion coincided with a wider power gap.
At a simulated 20% aggregate advertising lift, empirical detection rates were lower than the analytical estimates across all five dispersion settings.
At first, that seemed consistent with our heavy-tail hypothesis. But dispersion also changed baseline noise and the signal-to-noise ratio. We had not shown that skewness or kurtosis independently explained the discrepancy.
So we changed direction: instead of asking only how outcomes vary before treatment, we asked how estimated uncertainty behaves after treatment.
Percentages aggregated across the original synthetic simulation study. Dispersion changes several properties simultaneously; this graph is not an isolated causal effect of kurtosis.
What if the standard error changes when the treatment works?
This was the turning point in the investigation. A standard error estimates how much a treatment-effect estimate would vary across repeated experiments. It is not the same as the standard deviation of individual purchases.
If the uncertainty used in the analytical power formula is too small under treatment, the experiment can look more sensitive on paper than it really is in practice. That would help explain why analytical MDE was optimistic.
The dots represent hypothetical repeated experiment estimates. The spread across repeated estimates is the sampling SD; a single experiment's HC3 SE is an estimate of uncertainty and need not equal that spread.
The difference our analytical approximation missed
We used HC3 robust standard errors in a regression that estimates one common additive treatment effect across states.
Our analytical power calculation used the standard error under no treatment, then projected power as if that uncertainty still applied under positive lift.
But when simulated advertising lift varied in absolute size across states, the treatment regression's residuals changed. So did its HC3 standard error.
This is a diagnostic decomposition, not a claim that every percentage point belongs to one cause. But in this sequence, the largest drop arrived when we updated the treatment-period standard error.
Same total lift. Different places receiving it.
Treatment-effect heterogeneity (HTE) means treatment effects differ across units. A uniform percentage increase already creates different absolute increases when states vary in baseline activity.
This is the bridge between the changing standard error in the previous section and the power results in the next section. We stress-tested five effect-allocation patterns while preserving the same expected total incremental purchases in treated states. Read the visuals left to right: where lift lands → who gets more absolute lift → what the regression sees.
Schematic tile cartogram. Darker teal means a larger expected absolute increment.
Each dot is one state: farther right means a larger baseline; higher means a larger absolute increment.
Heteroskedasticity means residual spread is not constant. If the spread widens for larger states, uncertainty can increase.
Numeric summary from the actual HTE diagnostic at dispersion 0.05. The diagrams above explain the mechanism; they do not generate these numbers.
Power changed much more than we expected.
Up to this point we had a plausible mechanism: the standard error could change when lift was distributed unevenly across states. Now we ask the planning question directly.
When planning a real Reddit test, we do not know the true HTE pattern in advance. So these scenarios should be read as stress tests. At the historically calibrated dispersion setting, the fixed-null-SE analytical approximation suggested MDEs near 18–20% under every tested effect pattern. The simulated detection thresholds were sometimes much larger.
Measured pooled 0–60% power-curve MDEs. “>60%” is censoring, not an exact threshold. These are hypothetical effect mechanisms, not real Reddit campaign outcomes.
Empirical power at a 60% aggregate lift in the dispersion 0.05 stress test, still short of the 80% target.
The analytical MDE for the same scenario was 19.62%.
This is not a claim that Reddit lift would actually be distributed this way. It is a stress test showing that an experiment can be substantially less sensitive than expected when its actual effect allocation differs from what planning assumptions capture.
Even here, the analytical comparison was given each scenario's correct expected treatment signal. So the gap reflects more than assuming the wrong average effect; treatment-dependent uncertainty appears to matter as well.
Unlike the earlier conceptual charts, the bars encode the supplied simulated MDE results. Analytical values reach 80% within the grid. Large-state-favored empirical MDE remains above 60% in all five scenarios.
The same lift can produce very different 95% confidence intervals.
After looking at power and MDE, we asked a closely related question: if the same aggregate lift were distributed differently across states, how much could the reported uncertainty change?
Nobody would normally report a final confidence interval using only pre-treatment uncertainty. We show that fixed-uncertainty interval here only as a visual reference: a way to compare the planning-style approximation with the wider HC3 uncertainty observed under different HTE patterns.
To isolate width, all intervals are centered at the same illustrative 20% effect. The widening comes directly from the measured HC3 CI inflation in the simulations; the exact endpoints are shown only to help the eye.
These are the increases in 95% CI width relative to the fixed-uncertainty reference. They do not mean the effect itself changed, only how uncertain it looked.
So the planning approximation can misrepresent uncertainty, while the actual post-treatment HC3 interval remains much closer to its intended 95% coverage in these scenarios.
Where our conclusion could break.
We think the stress tests identify a meaningful vulnerability in our chosen test and approximation. They do not establish a universal rule, and we would not want the article to be read that way.
We would welcome checks that challenge the synthetic process, treatment scenarios, estimator choice or HC3 variance behavior. A result that weakens this interpretation would be useful evidence, not a failure of the investigation.
In particular, our outcome model is additive ANCOVA, while the business asks about aggregate percentage lift. A differently weighted or multiplicative estimator might substantially change the picture.
Does Gamma–Poisson reproduce reality?
Only approximately. We calibrated marginal distributional features and a regression-adjusted randomization metric, but did not reproduce every shared national shock, state correlation, or evolving treatment environment. Such dependence may change power and calibration.
Are the HTE scenarios plausible?
They are intentionally varied stress scenarios, not observed advertising effects. Large-state-favored allocation may be extreme; actual effects might be milder, concentrated elsewhere, zero for some geographies, or negative. A stress result should not be presented as a forecast.
Is our estimator aligned with the business metric?
ANCOVA fits a common additive state-average coefficient. We normalized simulated effects to equal aggregate lift among treated geographies. These are not identical estimands; changing the estimator or weighting could alter sensitivity materially.
Does HC3 SE inflation equal true variance inflation?
Not necessarily. HC3 estimates uncertainty from regression residuals and leverage. Under HTE, that estimate can behave differently from the actual sampling SD. Our empirical power concerns the specific ANCOVA + HC3 test, not every valid inferential method.
Is the analytical method a straw man?
It is a common, but incomplete, fixed-null-standard-error planning approximation. A more suitable analytical model that incorporates treatment-dependent variance and effect heterogeneity could do better. We have not compared all available power calculations.
How precise are the simulation MDEs?
We used finite simulated panels and assignments, overlapping historical windows, a 5-percentage-point lift grid and interpolation. Monte Carlo uncertainty remains. Values exceeding 60% were censored, not estimated precisely.
What we think we learned, and what we still do not know.
We began by suspecting dispersion, skewness and kurtosis. Those features did move with power error, but they did not isolate the cause.
When we held the untreated purchase process fixed and changed only the distribution of treatment effects, the analytical and empirical power gap changed markedly. In our chosen ANCOVA + HC3 procedure, treatment-dependent uncertainty appeared to be an important part of the explanation.
If we plan a geo test under one assumption about treatment heterogeneity but reality looks different, the test may be less informative than we expect.
That matters because an underpowered experiment can lead to two different mistakes. It can miss a real effect and encourage the wrong business conclusion. Or, if a noisy estimate happens to cross the significance threshold, it can overstate the size of the effect.
We cannot know a future treatment-effect distribution in advance. That is why we think the practical lesson is to stress-test several plausible HTE patterns, not to trust one analytical MDE or one simulation as ground truth.
Methods and further reading
Historical and synthetic numerical results in the article come from the investigation described here, not from the linked papers. Concept animations are explicitly illustrative. No identifiable business-level volumes or state purchase data are published.
Gelman & Carlin (2014): Type S / Type M errors and exaggeration among significant estimates.
Lin (2013): regression adjustment and randomization inference.
Morris, White & Crowther (2019): design and interpretation of simulation studies.
Vickers & Altman (2001): ANCOVA / baseline adjustment.
More from our measurement work: turning off Google Ads in roughly half the US for six weeks, building a synthetic UK to judge Google App campaigns, and how MMM, attribution and incrementality testing differ.
The interactive figures run in your browser, without external JavaScript or data requests. All US tile positions and activity values are synthetic schematics, not geographic-level business observations.