p-value

Statistical measure used in the frequentist approach to indicate the probability of observing the obtained results (or even more extreme results) if the null hypothesis were true. In other words, it quantifies the degree of surprise of the observation compared to a situation where there would be no real difference between the tested variants.

In a CRO / A/B testing context:

In an A/B test, the null hypothesis states that variation B has no effect compared to A.
➡️ A low p-value (e.g., 0.03) indicates that the observed results are poorly compatible with this hypothesis, which reinforces the idea that a real effect exists.

Key takeaways:

• A p-value below 0.05 (the classic threshold) is often considered statistically significant → the null hypothesis is rejected.
• A p-value above 0.05 doesn't prove there is no effect, only that there is not enough evidence to conclude.
• The p-value does not measure the probability that the variation is better, nor the magnitude of the effect. For that, you should look at the uplift, the confidence intervals or the estimated business impact.

Common pitfalls in CRO:

• Interpreting a p-value in isolation without taking into account the data volume, the MDE or the business context.
• "P-hacking": stopping a test as soon as it reaches 0.049 can lead to false positives (misleading results).
• Forgetting that the p-value says nothing about practical significance: a minimal effect can be significant but useless.