Work & thinkingApproachMeasurement
Did this work?
Before deciding how much credit an ad gets, ask what changed because it ran.
- Question
- What would have happened without this spend?
- One approach
- Pause ads in selected markets and compare their results with markets where ads continue.
- What it gives us
- An estimate of the effect, with uncertainty, to inform the next decision.
The work starts before the ads stop.
Markets need to be comparable in ways relevant to the question. Historical behavior, other marketing activity, seasonality and local changes can all affect what we conclude.
The test also needs enough data and enough time to detect an effect worth acting on.
An estimate deserves an explanation.
Compare how the groups change relative to their behavior before the test. Then look at the size of the estimated effect, the uncertainty around it, and whether the comparison still holds.
Sometimes the useful answer is that the test can’t tell us yet.
What should change in the budget?
The result has to be weighed against the cost of running the test and the decision at stake. The aim is enough evidence to make a better call about the next dollar.
Recommendations