A/B Test Significance Calculator
Is your winning variant a real result, or just random noise? Enter your numbers and find out in seconds.
Variant A (control)
Conversion rate: 10.00%
Variant B (test)
Conversion rate: 12.00%
Uses a two-tailed two-proportion z-test. Statistical significance means the difference is unlikely to be random chance — it does not by itself prove the winning variant is better in every context, or account for factors like seasonality, test duration or novelty effects. Keep tests running long enough to capture a full, representative sample.
How to read the result
Significant win/loss: the p-value is below your chosen threshold (e.g. under 0.05 for 95% confidence), so the difference is unlikely to be luck. Not yet significant: the gap could still be random — usually the fix is more traffic, not a decision. The relative uplift shows how much better (or worse) variant B is than A in percentage terms, and the p-value is the probability of seeing this gap if the two were actually identical.
Frequently Asked Questions
What does "statistically significant" actually mean?
It means the difference between your two variants is large enough, given your sample size, that it's unlikely to be down to random chance. At 95% confidence, a significant result means there's roughly a 5% or lower probability you'd see a difference this big if the two variants were actually identical. It does not guarantee the winning variant is truly better \u2014 it just makes pure luck an unlikely explanation.
What is a p-value?
The p-value is the probability of seeing a difference at least as large as the one you observed, assuming the two variants perform identically. A p-value of 0.03 means about a 3% chance the result is a fluke. Lower is stronger: for 95% confidence you need p below 0.05, for 99% confidence you need p below 0.01. This calculator computes it with a two-proportion z-test.
How much traffic do I need before I can trust the result?
There's no single magic number \u2014 it depends on how big the real difference is and how confident you want to be. As a rough guide, a few hundred conversions per variant is usually the minimum before results stabilise, and small differences (a 1-2% relative uplift) can need tens of thousands of visitors to detect reliably. If this calculator says "not yet significant," the honest answer is often just to keep the test running longer rather than calling it early.
Why does the calculator sometimes say the result is not significant even though one variant is clearly ahead?
Because with a small sample, a visible-looking lead can easily be random. If variant B is converting at 12% versus 10% for A, that looks like a win \u2014 but on only 100 visitors each, that gap is well within the range of coincidence. Significance testing exists precisely to stop you shipping changes based on noise. Keep collecting data until the sample is large enough for the difference to hold up.
Is this a one-tailed or two-tailed test?
This calculator uses a two-tailed test, which checks whether the variants differ in either direction (B could be better or worse than A). Two-tailed is the safer, more conservative default for most marketing A/B tests, because you usually care about detecting a real change rather than only confirming a pre-decided winner.