Three ways to measure incrementality — holdout, on/off, ITS
How do I measure the incremental effect of advertising?
A randomized holdout is the most accurate. Otherwise use pre/post or ITS.
Incrementality is the outcome advertising actually created. There are three main designs. A randomized holdout, a pre/post comparison around turning a campaign on or off, and interrupted time series (ITS). The accuracy ranking is clear, and so is the difficulty ranking.
Side by side
| Method | Design | Causal claim | Requirements | Caution |
|---|---|---|---|---|
| Control holdout | Split exposed and unexposed groups at random, at the same time | Supported | Platform holdout feature or geo split | Small samples give intervals too wide to conclude |
| Turn on / turn off | Compare before and after a campaign starts or stops | Limited | A clear intervention date and enough pre/post data | Seasonality and promotions are mixed in |
| ITS (interrupted time series) | Extend the pre-intervention trend to estimate the counterfactual | Limited | Many pre-period observations and a stable trend | Ignoring autocorrelation understates the interval |
Which one to use
- If your platform supports holdouts
- Use a holdout. Randomization is what allows a causal claim.
- If the campaign cannot be paused, such as brand
- Use ITS to extend the pre-period trend, with autocorrelation correction.
- If the budget is too small for a control group
- Start with pre/post, but add a control series and read it as difference-in-differences.
- If the result is non-significant
- Do not conclude no effect. Check whether power was sufficient first.
Frequently asked questions
Can pre/post alone prove incrementality?
It is not recommended. Seasonality, promotions, and competitive shifts land in the same window. Adding a control series and reading it as difference-in-differences removes much of that bias.
If a holdout is non-significant, does advertising not work?
No. Small samples or small effects go undetected even when a real effect exists. Read the width of the confidence interval and check whether the test had enough power.
How many pre-period points does ITS need?
There is no fixed minimum, but you need enough to separate trend from seasonality. A short pre-period makes the counterfactual unstable and widens the interval.
Tools for this
- Incrementality AnalysisUse holdout or pre/post data to estimate the outcomes advertising added beyond what would have happened anyway.
- Brand Campaign Incrementality | ITSEstimate brand-campaign lift in brand search, direct traffic, or signups, then route to ITS or holdout analysis based on the data you have.
- A/B Experiment AnalysisPlan sample size and review lift, confidence intervals, and power to decide whether to ship, stop, or collect more data.