Frequentist

Statistical approach that consists in estimating the parameters of a model (mean, conversion rate, etc.) solely on the basis of observed data, without using anya priori information. It is based on concepts such as thenull hypothesis, significance tests, p-values and confidence intervals, and is the traditional approach to statistical analysis.

A/B testing and CRO:

The frequentist approach is historically used in manyA/B testing confidence calculators to determine whether a variation produces a significant effect or not.

It seeks to answer the question:

"If the variation tested has no real effect, what is the probability of obtaining such a result by chance?"

But this approach is based on strong constraints:

• Sufficient sample size,
• Test duration respected (risks associated with early shutdown),
• Technical interpretation of results (p-value ≠ probability that B is better).

Limits in a CRO context :

• Tests often take longer because of the need to reach a predefined sample size,
• Less intuitive for non-technical teams,
• More sensitive to reading errors or risky practices such as p-hacking,
• Does not directly give the probability that a variation is better.

Current trend: shift to Bayesian approach

In CRO, the frequentist approach is increasingly being replaced by Bayesian statistics, which :

• provides faster, more usable results,
• dynamically adapts to sample size,
• allows us to directly estimate the probability that a variation will outperform,
• facilitates decision-making by product and marketing teams.

Platforms such as VWO, Convert, Optimizely (new version) or AB Tasty (Bayesian mode) now integrate Bayesian statistical engines to speed up analysis and reduce interpretation errors.

‍