Experiment analysis
Check whether the difference is more than chance and whether the next experiment decision is justified.
End of analysisNext steps and reference material below. Not part of the analysis result.
Next step
Proof
Full journey
How do I judge whether an A/B result is significant?
Check whether the interval excludes zero, and whether the sample was sufficient.
Method and FAQOpen only when you need the methodology
See design rules and read-out criteria
When to use it
Use it to size a creative, landing, or offer test before launch, to judge a finished test, and to decide whether to keep collecting data. Exposures (or sessions) and conversions per group are enough.
What to inspect
Read the observed lift with its confidence interval and the power to detect that difference. An interval crossing zero means 'not decidable with this sample', not 'no effect'—the distinction prevents premature stops.
How to act
Ship when the result is both statistically and practically meaningful. When it is undecidable, check the additional sample and time required before choosing to continue or stop. Peeking daily and stopping at significance inflates false positives.
Frequently asked questions
Does a non-significant result mean no effect?
No. The sample may simply have been too small to detect one. Claiming 'no effect' requires separate evidence that power was sufficient.
Can I stop early when results look good?
Not advisable. Stopping the moment a test turns significant converts random variation into declared wins and raises false positives. Filling the planned sample is safer.
How is this different from incrementality?
A/B compares two variants; incrementality estimates what advertising added beyond what would have happened anyway. Different questions, so they are separate tools.