Check whether the difference is more than chance and whether the next experiment decision is justified.
Analysis and view settings
Save analysis options and view/download choices to your account. Budgets, dates, columns and data values stay in this device’s project settings. Loading a setup does not confirm study conditions.
Loading…
Reuse this analysis setup
Keep the mappings, filters and model options in your project.
Before you launch the test — calculate how many samples (people) you need to trust the result.
What do you need to check now?
Experiment settings
Test type
Data currency
How many people does the experiment need?
Enter the current conversion rate and the minimum lift (MDE) you want to detect, and the required sample size per arm is calculated.
e.g. 5% = 5
e.g. a 10% rise over baseline = 10
Required samples per arm
31,235
Combined (2 arms)
62,470
Expected conversion rate A → B
5.00% → 5.50%
z_α/2 + z_β
1.960 + 0.842
At 5,000 people/day of average traffic, you'll need about 13 days; at 10,000/day, about 7 days.
Interim-read plan · O’Brien–Fleming
Set the number of interim looks before launch. Early looks use much stricter boundaries; this table does not validate unplanned peeks.
Look
Target information
Cumulative sample
|z| boundary
Nominal p boundary
1
25%
15,618
4.049
< 0.0001
2
50%
31,235
2.863
0.0042
3
75%
46,853
2.338
0.0194
4
100%
62,470
2.024
0.0429
Boundaries are calibrated so the combined two-sided Type I error across all planned looks equals the entered α. The final nominal p boundary can therefore be below α. This does not cover adaptive stopping, continuous monitoring, or other metrics.
Budget calculator
Enter the expected CPR (Cost Per Result) to calculate the total budget needed for the sample size above. Leaving CPR_B blank assumes it equals A.
Cost per conversion
When the variant has a different CPR
Arm A budget
₩78,087,500
Arm B budget
₩78,087,500
Total budget needed
₩156,175,000
CPR A = ₩2,500 · CPR B = ₩2,500 · n per arm = 31,235
What size of difference can this test detect?
As the sample size (per arm) grows, the minimum detectable effect (MDE) shrinks. A lower baseline conversion rate needs more samples.
Purchased pass required · excludes trial
Statistical notes for experts
Binary (CVR) · z-test: z = (p̂_B - p̂_A) / √(p̄(1-p̄)(1/n_A + 1/n_B)). Reject the null hypothesis when p-value < α.
Continuous · Welch t: t = (μ̂_B - μ̂_A) / √(s_A²/n_A + s_B²/n_B). The p-value and 95% confidence interval use the Student-t distribution with Welch–Satterthwaite degrees of freedom.
Bayesian: Binary only. Beta(1+x, 1+n-x) posteriors under a Beta(1,1) prior are compared deterministically using an exact finite sum, normal approximation, or adaptive integration.
Budget calculation: Total Budget = n_per_arm × (CPR_A + CPR_B).
Power curve: Derived by inverting sample size (binary search) to find the minimum detectable MDE.