Statistical confidence

Level of certainty one has in the results of a statistical test. It is generally expressed as a percentage, for example 95%, which means that if the test were repeated a large number of times under the same conditions, 95% of the tests would lead to the same conclusion. This is a central concept in frequentist analyses, used in most classic A/B testing tools.

In the frequentist approach:

• Statistical confidence is related to the probability of wrongly rejecting the null hypothesis (type I error risk).
• A 95% confidence level corresponds to a significance threshold (p-value) of 0.05.
• This indicates that one accepts at most a 5% chance that the observed difference is due to chance.

Example: If variation B has a statistical confidence of 96%, it is considered to significantly outperform version A with a margin of error below 4%.

In the Bayesian approach:

• We rather talk about the probability that one variation is better than the other (e.g.: "variation B has a 92% chance of being superior to A").
• This is not a rejection test, but a direct estimate of the probability of a real gain.
• This approach is often more intuitive for business teams, but depends more heavily on priors (initial assumptions).

In CRO:

Whichever framework is chosen, statistical confidence is essential for:

• validating the results of an A/B test,
• limiting false positives (effects due to chance),
• make informed decisions based on reliable data.

Caution: high statistical confidence does not guarantee a significant business impact. It must be cross-referenced with other indicators (uplift, sample size, incremental value) to guide an implementation decision.