Frequentist

A statistical approach that involves estimating a model's parameters (mean, conversion rate, etc.) solely from observed data, without using prior information. It relies on concepts such as the null hypothesis, significance tests, p-values and confidence intervals, and constitutes the traditional approach to statistical analysis.

In A/B testing and CRO:

The frequentist approach has historically been used in many A/B testing confidence calculators to determine whether a variation produces a significant effect or not.

It seeks to answer the question:

“If the tested variation has no real effect, what is the probability of obtaining such a result by chance?”

But this approach relies on strong constraints:

• Sufficient sample size,
• Respected test duration (risks associated with stopping early),
• Technical interpretation of results (p-value ≠ probability that B is better).

Limitations in a CRO context:

• Often longer tests due to the need to reach a predefined sample size,
• Less intuitive for non-technical teams,
• More susceptible to misreading or risky practices such as p-hacking,
• Does not directly give the probability that a variation is better.

Current trend: shift toward the Bayesian approach

In the field of CRO, the frequentist approach is increasingly being abandoned in favor of Bayesian statistics, which:

• provides actionable results faster,
• dynamically adapts to sample size,
• allows direct estimation of the probability that a variation outperforms,
• facilitates decision-making for product and marketing teams.

Platforms such as VWO, Convert, Optimizely (new version) or AB Tasty (Bayesian mode) now incorporate Bayesian statistical engines to speed up analysis and reduce interpretation errors.