Before you launch the test — calculate how many samples (people) you need to trust the result.

§1Select mode

Test type

Currency

§2① Parameter calc before launching the test

Enter the current conversion rate and the minimum lift (MDE) you want to detect, and the required sample size per arm is calculated.

Current conversion rate (%) · baselinee.g. 5% = 5

Minimum lift to detect (%) · MDE (relative)e.g. a 10% rise over baseline = 10

Significance level α

Statistical power

Required samples per arm

31,235

Combined (2 arms)

62,470

Expected conversion rate A → B

5.00% → 5.50%

z_α/2 + z_β

1.960 + 0.842

At 5,000 people/day of average traffic, you'll need about 13 days; at 10,000/day, about 7 days.

Budget calculator

Enter the expected CPR (Cost Per Result) to calculate the total budget needed for the sample size above. Leaving CPR_B blank assumes it equals A.

Expected CPR · Arm A (₩)Cost per conversion

Expected CPR · Arm B (₩, optional)When the variant has a different CPR

Arm A budget

₩78,087,500

Arm B budget

₩78,087,500

Total budget needed

₩156,175,000

CPR A = ₩2,500 · CPR B = ₩2,500 · n_per_arm = 31,235

§3MDE vs Sample Size power curve

As the sample size (per arm) grows, the minimum detectable effect (MDE) shrinks. A lower baseline conversion rate needs more samples.

Current conversion rate (%) · baseline

Significance level α

Statistical power

§4Statistical notes

Binary (CVR) · z-test: z = (p̂_B - p̂_A) / √(p̄(1-p̄)(1/n_A + 1/n_B)). Reject the null hypothesis when p-value < α.
Binary · Sample Size: n = 2 × (z_α/2 + z_β)² × p̄(1-p̄) / δ²
Continuous · Welch: z = (μ̂_B - μ̂_A) / √(s_A²/n_A + s_B²/n_B). Accurate when n > 30 per arm.
Bayesian: Binary only. Prior Beta(1,1). Posterior Beta(1+x, 1+n-x). Monte Carlo, 10,000 sims.
Budget calculation: Total Budget = n_per_arm × (CPR_A + CPR_B).
Power curve: Derived by inverting sample size (binary search) to find the minimum detectable MDE.

Loading…

§2① Parameter calc before launching the test

Enter the current conversion rate and the minimum lift (MDE) you want to detect, and the required sample size per arm is calculated.

Current conversion rate (%) · baselinee.g. 5% = 5

Minimum lift to detect (%) · MDE (relative)e.g. a 10% rise over baseline = 10

Significance level α

Statistical power

Required samples per arm

31,235

Combined (2 arms)

62,470

Expected conversion rate A → B

5.00% → 5.50%

z_α/2 + z_β

1.960 + 0.842

At 5,000 people/day of average traffic, you'll need about 13 days; at 10,000/day, about 7 days.

Budget calculator

Enter the expected CPR (Cost Per Result) to calculate the total budget needed for the sample size above. Leaving CPR_B blank assumes it equals A.

Expected CPR · Arm A (₩)Cost per conversion

Expected CPR · Arm B (₩, optional)When the variant has a different CPR

Arm A budget

₩78,087,500

Arm B budget

₩78,087,500

Total budget needed

₩156,175,000

CPR A = ₩2,500 · CPR B = ₩2,500 · n_per_arm = 31,235

§4Statistical notes

Binary (CVR) · z-test: z = (p̂_B - p̂_A) / √(p̄(1-p̄)(1/n_A + 1/n_B)). Reject the null hypothesis when p-value < α.

Binary · Sample Size: n = 2 × (z_α/2 + z_β)² × p̄(1-p̄) / δ²

Continuous · Welch: z = (μ̂_B - μ̂_A) / √(s_A²/n_A + s_B²/n_B). Accurate when n > 30 per arm.

Bayesian: Binary only. Prior Beta(1,1). Posterior Beta(1+x, 1+n-x). Monte Carlo, 10,000 sims.

Budget calculation: Total Budget = n_per_arm × (CPR_A + CPR_B).

Power curve: Derived by inverting sample size (binary search) to find the minimum detectable MDE.