Skip to content
DefinitionStatistiques

Statistical confidence

Level of certainty one has in the results of a statistical test.

Level of certainty one has in the results of a statistical test. It is generally expressed as a percentage, for example 95%, which means that if the test were repeated a large number of times under the same conditions, 95% of the tests would lead to the same conclusion. This is a central concept in frequentist analyses, used in most classic A/B testing tools.

In the frequentist approach:

  • Statistical confidence is related to the probability of wrongly rejecting the null hypothesis (type I error risk).
  • A 95% confidence level corresponds to a significance threshold (p-value) of 0.05.
  • This indicates that one accepts at most a 5% chance that the observed difference is due to chance.

Example: If variation B has a statistical confidence of 96%, it is considered to significantly outperform version A with a margin of error below 4%.

In the Bayesian approach:

  • We rather talk about the probability that one variation is better than the other (e.g.: "variation B has a 92% chance of being superior to A").
  • This is not a rejection test, but a direct estimate of the probability of a real gain.
  • This approach is often more intuitive for business teams, but depends more heavily on priors (initial assumptions).

In CRO:

Whichever framework is chosen, statistical confidence is essential for:

  • validating the results of an A/B test,
  • limiting false positives (effects due to chance),
  • make informed decisions based on reliable data.

Caution: high statistical confidence does not guarantee a significant business impact. It must be cross-referenced with other indicators (uplift, sample size, incremental value) to guide an implementation decision.

Move from definition to practice in Statistiques

From hypothesis to statistical analysis, we design and run your A/B tests end to end. Let's talk for 30 minutes about your challenges.