A/B testing statistical confidence calculator
Check whether the gap between your variants is statistically significant: p-value, confidence interval and estimated impact, in frequentist or Bayesian mode.
Calculation method
Post-Test Calculator - frequentist
Z-test comparing two proportions with pooled variance under H0: SE = √(p̄(1−p̄)(1/nA + 1/nB)), z = (pB − pA) / SE, where p̄ is the pooled rate. One-sided or two-sided p-value depending on the setting, alpha corrected by Bonferroni for the number of comparisons. The confidence interval covers the absolute difference in rates.
Limitation: this p-value is only valid for ONE analysis, on a sample size decided before launch. Checked multiple times during the test, it underestimates the risk of false positives. This is precisely what the mSPRT and GST tabs correct.
Post-Test Calculator - Bayesian
Beta-binomial model. Non-informative Beta(1, 1) prior on each rate, Beta(1 + conversions, 1 + non-conversions) posteriors. 100,000 Monte Carlo draws to estimate P(B > A) and the credible interval on the rate difference. The decision threshold is adjusted via Bonferroni correction based on the number of comparisons.
Note: the probability displayed is a posterior probability, it should not be interpreted as a p-value and does not control a frequentist error rate.
FAQ
Frequently asked questions about statistical confidence in A/B testing
- A statistical confidence calculator helps assess whether the difference observed between two variants (for example, in an A/B test) is statistically significant or simply due to chance. It helps you make data-driven decisions by identifying the best-performing version with a certain level of certainty.
Tell us more about your project
We know how to increase the performance of your digital channels.
- Experimentation
- UX & Experience
- Personalization & CRM
- Data & Analytics
- Training
- Agentic commerce
