Work & thinkingApproachMeasurement
Did this work?
Before deciding how much credit an ad gets, ask what changed because it ran. A geo holdout is one way to investigate that question.
- Question
- What would have happened without this spend?
- One approach
- Pause the advertising being tested in selected markets and compare them with markets where it continues.
- What it gives us
- An estimate of the effect, with uncertainty, to inform the next budget decision.
The work starts before the ads stop.
The markets need to support a credible comparison. Their historical behavior, other marketing activity, seasonality and local changes can all affect what we conclude. Random assignment, where feasible, helps make that comparison more reliable.
The test also needs enough data and enough time to detect a difference that would matter to the decision.
How much changed, and how sure are we?
Compare the markets’ performance while accounting for how they behaved before the test. Look at the estimated effect, the uncertainty around it and whether the comparison still holds.
An unclear result can mean the test cannot distinguish an effect from noise. It doesn’t automatically mean the advertising had no effect.
What should change in the budget?
The estimate has to be weighed against the cost of the test and the decision at stake. The useful outcome is enough evidence to make a better call about the next dollar, with a clear account of what the test can and cannot tell us.
Recommendations