Statistical significance test A/B test

Statistical analysis method used to determine whether the results observed in a test (such as an A/B test) are reliable enough to be attributed to a real effect rather than to chance or random fluctuation. It is a fundamental tool for validating or rejecting a hypothesis, based on quantitative data.

🎯 Objective:

Assess whether the difference between two (or more) variants is statistically significant, meaning it is unlikely to be due to chance. This enables informed decision-making and minimizes false positives (wrongly concluding that a variation is better).

🔍 How it works:

The test relies on two hypotheses:

• Null hypothesis (H₀): there is no real difference between the tested variants.
• Alternative hypothesis (H₁): there is a significant difference.

A p-value is then calculated: the probability of obtaining a gap at least as large as the one observed, if the null hypothesis were true.

→ If the p-value is below the significance threshold (usually 0.05), we reject the null hypothesis → the difference is considered statistically significant.

📊 Associated metrics:

• P-value: a measure of surprise; the lower it is, the more likely the effect is real.
• Confidence level (often 95%): the probability of not being wrong when concluding there is a difference.
• Statistical power: the test's ability to detect a real effect if one exists, often set at 80% or higher.
• Minimum Detectable Effect (MDE): the smaller the expected effect, the more traffic is needed to reach a conclusion.

🧪 Application in CRO:

In an A/B test or multivariate test, the statistical significance test is essential to:

• validate a variation as “winning” or not,
• avoid misinterpretation due to noise effects or tests stopped too early (p-hacking),
• ensure the results are generalizable, not simply tied to a specific period or segment.