A/B Test Significance Calculator

How to calculate A/B test significance?

Significance is determined by comparing the conversion rates of two groups while accounting for sample size and variance.

Z = (p1 - p2) / sqrt(p * (1 - p) * (1/n1 + 1/n2))

Using the A/B test calculator: an example

For example, consider a test where the Control group has 1,000 Visitors and 50 Conversions, while the Variation group has 1,050 Visitors and 75 Conversions.

Step-by-step calculation:

• Calculate the conversion rates for both the Control and Variation groups.
• Determine the pooled proportion of the combined dataset.
• Calculate the standard error based on the pooled proportion and sample sizes.
• Find the Z-score and the corresponding P-value to determine statistical significance.

A 95% confidence level is the standard for most A/B tests because it provides a reliable threshold for determining that the results are not simply due to random noise.

Frequently Asked Questions

What is a P-value?
The probability that the observed difference happened by chance; lower is better.

Why is sample size important?
Larger samples reduce the margin of error and increase the power of your test, helping you detect smaller differences with confidence.

What does 95% confidence actually mean?
It means if you ran the test 100 times, you would expect the same result 95 times.