/ Marketing

A/B Test Significance Calculator

Real winner or just noise?

Enter the visitors and conversions for variant A and variant B, and this tool runs a two-proportion z-test to tell you whether B truly beats A. It compares the conversion rates, checks how likely the gap is pure chance, and gives you a p-value plus a plain verdict: significant winner, or keep the test running.

How significance is calculated

A two-proportion z-test asks a simple question: if both variants really converted at the same rate, how often would random chance alone produce a gap this big? First it pools both variants into one combined rate, then measures how many standard errors apart the two observed rates sit. That distance is the z-score.

The p-value turns that z-score into a probability. A p-value of 0.03 means there is a 3% chance you would see a gap this large from noise alone. When the p-value drops below your threshold (0.05 at 95% confidence), the result is significant and B is a real winner. Above it, the gap is still inside the range of luck, so keep collecting data.

Worked examples

Variant A Variant B p-value Verdict (95%)
100/1000 150/1000 < 0.001 Significant
250/5000 300/5000 0.028 Significant
200/2000 210/2000 0.602 Not yet
Variant A
Variant B
Rate A 10%
Rate B 15%
Rel. uplift +50%
p-value 0.001
Verdict B is a significant winner

Questions people ask

What does statistical significance mean?

It means the difference between your two variants is large enough that random chance is an unlikely explanation. At 95% confidence you accept a 5% risk of calling a winner that is really just luck. Significance is a threshold you pick before the test, not proof that B is better in every case.

What sample size do I need?

Enough that a real difference clears the noise. As a rough guide, small lifts on low base rates need thousands of visitors per variant, while a big swing on a healthy conversion rate can show up in a few hundred. Do not stop the moment the p-value dips below 0.05. Set a target sample size first and let the test finish, otherwise you will catch a lot of false winners.

What is a p-value?

The p-value is the probability of seeing a gap at least this large if the two variants actually performed the same. A p-value of 0.03 means there is a 3% chance the result is noise. Lower is stronger. Below your alpha (0.05 at 95% confidence) you call the result significant.