A/B Test Significance Calculator
A two-proportion z-test from conversions and visitors per arm, with the p-value and the interval for the difference.
A/B Test Significance Calculator
A two-proportion z-test from conversions and visitors per arm, with the p-value and the interval for the difference.
What the page prints, and what it refuses
Every figure is printed with the rule or formula that produced it, and any numerical tolerance is stated on the result rather than implied. The page refuses an input the formula does not define and names the value it read; it prints no answer it cannot support.
The calculation runs entirely in this browser tab; nothing is uploaded.
The page tests whether two conversion rates differ. A control at 100 of 1000 and a variant at 130 of 1000 gives rates of 10 % and 13 %, a pooled standard error of 0.01427, z = 2.1027 and a two-tailed p of 0.0355.
The p-value uses the pooled variance, which is the null hypothesis that both arms share one rate, while the confidence interval uses the unpooled variance, which is the usual convention for an interval. The page prints both and names the difference.
Pooled test, unpooled interval
Under the null both arms have the same rate, so the standard error uses the pooled proportion p̂ = (x₁ + x₂)/(n₁ + n₂). That is what makes the test statistic a z with a standard normal reference.
The interval for the difference estimates the variance from each arm separately, because the interval describes the observed difference rather than a hypothesised one. Using the pooled error for both is a common but different choice, and the page does not make it silently.
How to use A/B Test Significance Calculator
Enter conversions and visitors for the control, then for the variant, and press Run.
The result prints both rates, the difference, the pooled rate, the standard error, z, the two-tailed p and the 95 % interval.
What to Expect
Best for
- Comparing two conversion rates
- Checking a dashboard's significance claim
- Reporting an interval beside a p-value
Not ideal for
- More than two arms, which needs a different test
- Paired or repeated observations
- A rate of exactly 0 % or 100 % in both arms
What this tool keeps
- All four counts exactly as entered
What may need cleanup
- Nothing: every figure is a final number
Common errors
- Conversions above visitors are refused
- A zero pooled standard error is refused with the reason
Example use cases
- Test 100/1000 against 130/1000
- See the interval include zero
- Check a marketing report
Sample input
100 of 1000 and 130 of 1000
Sample output
10 % vs 13 %, z = 2.1027, p = 0.0355, 95 % CI 0.0021 … 0.0579
Who this is for
- Growth analysts
- Product managers reading a test result
Frequently asked questions
Why is p two-tailed?
The test asks whether the rates differ in either direction, which is the usual two-arm question. The page prints the convention so a one-sided reading is not assumed.
What does the interval add?
It gives the plausible size of the difference, not just whether zero is excluded. A significant 3-point lift and a significant 0.3-point lift have very different consequences.
Can I enter rates instead of counts?
No. The test needs the counts to form the pooled rate and the standard error. Rates alone would lose the sample size, which is the whole basis of the test.
Build custom tools for your store
Join the SDK program to create useful ecommerce utilities for your store.