Three ways to measure incrementality — holdout, on/off, ITS
How do I measure the incremental effect of advertising?
A randomized holdout is the most accurate. Otherwise use pre/post or ITS.
Incrementality is the outcome advertising actually created. There are three main designs. A randomized holdout, a pre/post comparison around turning a campaign on or off, and interrupted time series (ITS). The accuracy ranking is clear, and so is the difficulty ranking.
Side by side
| Method | Design | Causal claim | Requirements | Caution |
|---|---|---|---|---|
| Control holdout | Split exposed and unexposed groups at random, at the same time | Supported | Platform holdout feature or geo split | Small samples give intervals too wide to conclude |
| Turn on / turn off | Compare before and after a campaign starts or stops | Limited | A clear intervention date and enough pre/post data | Seasonality and promotions are mixed in |
| ITS (interrupted time series) | Extend the pre-intervention trend to estimate the counterfactual | Limited | Many pre-period observations and a stable trend | Ignoring autocorrelation understates the interval |
Which one to use
- If your platform supports holdouts
- Use a holdout. Randomization is what allows a causal claim.
- If the campaign cannot be paused, such as brand
- Use ITS to extend the pre-period trend, with autocorrelation correction.
- If the budget is too small for a control group
- Start with pre/post, but add a control series and read it as difference-in-differences.
- If the result is non-significant
- Do not conclude no effect. Check whether power was sufficient first.
Frequently asked questions
It is not recommended. Seasonality, promotions, and competitive shifts land in the same window. Adding a control series and reading it as difference-in-differences removes much of that bias.
No. Small samples or small effects go undetected even when a real effect exists. Read the width of the confidence interval and check whether the test had enough power.
There is no fixed minimum, but you need enough to separate trend from seasonality. A short pre-period makes the counterfactual unstable and widens the interval.
Tools for this
- Incrementality AnalysisUse holdout or pre/post data to estimate the outcomes advertising added beyond what would have happened anyway.
- Brand Campaign Incrementality | ITSEstimate brand-campaign lift in brand search, direct traffic, or signups, then route to ITS or holdout analysis based on the data you have.
- A/B Experiment AnalysisPlan sample size and review lift, confidence intervals, and power to decide whether to ship, stop, or collect more data.