A/B Test Sample Size Calculator

How to calculate sample size for A/B tests?

This formula calculates the minimum participants per group needed to detect a statistically significant difference in an A/B test.

The main formula for sample size per group is:

n_per_group = ((z_alpha + z_beta)^2 * (p1(1-p1) + p2(1-p2))) / (d^2)

Using the A/B sample size calculator: an example

Consider an example: baseline conversion 5%, MDE 2 percentage points, 80% power, and 5% significance (alpha).

Step-by-step calculation:

1. Identify parameters: Baseline (p1) = 0.05, MDE (d) = 0.02, Alpha (α) = 0.05 (z_alpha = 1.96), Power = 0.80 (z_beta = 0.84).
2. Calculate the combined variance and Z-score terms based on these values.
3. Divide the result by the squared MDE (d^2).
4. This calculation yields approximately 14,000 users per group.

The calculated sample size aids in planning your A/B test for reliable results.

Frequently Asked Questions

How to choose an appropriate MDE?

The MDE should be the smallest practical difference meaningful for your business goals.

What if my traffic is too low for the calculated sample size?

If traffic is low, consider running the test longer, increasing MDE, or adjusting power/alpha.

How does this apply to tests with multiple variants?

For A/B/n tests (multiple variants), you need larger sample sizes and must adjust the significance level for multiple comparisons.