MMM or an incrementality experiment — which comes first
Should I run MMM or an incrementality experiment first?
Run the experiment first. MMM fills in channels you cannot test.
These are not substitutes; they play different roles. An experiment establishes cause through randomization but covers one thing at a time. MMM covers every channel at once but rests on observational data, so it is not causal. If you need an order, experiments come first and MMM fills the gaps.
Side by side
| Criterion | Incrementality experiment | MMM (marketing response regression) |
|---|---|---|
| Nature of evidence | Causal when randomized | Observational association — not causal |
| Scope per run | One channel or campaign | All channels at once |
| Data required | Exposure and conversions during the test | Weekly per-channel spend and outcome panel |
| Time required | Two to four weeks depending on design | At least one year recommended |
| What it cannot do | Interactions across many channels | Claim causation, or separate collinear channels |
| What you do with it | Decide whether to scale that channel | Draft an initial budget split |
Which one to use
- When you need one channel's effect
- Run an experiment. Randomization is what supports a causal claim.
- When you need to split the whole budget
- Draft it with MMM and confirm large moves with an experiment.
- When channel spends always moved together
- MMM cannot separate them. Check multicollinearity first.
- When you have less than a year of data
- Start with experiments and variance decomposition instead of MMM.
Frequently asked questions
Can I move budget directly on MMM output?
Large moves are not recommended. MMM estimates association in observational data, not causation. Treat it as a draft and confirm sizable shifts with an experiment before committing.
A channel coefficient came out negative. Is that cannibalization?
Usually not. Sparse spend or collinearity with another channel makes coefficient signs unstable. Check multicollinearity and sample size before reading it as cannibalization.
What about channels I cannot experiment on?
For channels you cannot pause, such as brand, estimate the intervention effect with ITS or approximate it with MMM attribution. In both cases report a range rather than a point claim.
Tools for this
- MMM Contribution AnalysisUse marketing mix modeling to decompose channel, base-demand, and event contribution.
- Incrementality AnalysisUse holdout or pre/post data to estimate the outcomes advertising added beyond what would have happened anyway.
- A/B Experiment AnalysisPlan sample size and review lift, confidence intervals, and power to decide whether to ship, stop, or collect more data.
- VIF Multicollinearity CheckCheck VIF and channel-spend correlation from a CSV before attempting to separate channel contribution in MMM.