Statistical Power

Anticipating a test's ability to detect a real effect

📌 Definition of statistical power in A/B testing

Statistical power is the probability that an A/B test detects a real effect if it actually exists.
In other words, it evaluates the test's ability to avoid a false negative – that is, to avoid wrongly concluding that a variation is ineffective when it actually produces a genuine impact.

In a conversion rate optimization (CRO) context, sufficient power ensures that decisions made from a test are sound and actionable.

🎯 Recommended level

In most cases, a statistical power of 80% is considered the acceptable minimum.
This means that the test has an 80% chance of detecting a real effect, with only a 20% risk of missing it (Type II error).

⚖️ Factors influencing power

Power depends on three key parameters:

1. Sample size
   → The more traffic or conversions you have, the smaller the effects you can detect.
2. Minimum Detectable Effect (MDE)
   → This is the minimum effect size you want to be able to detect (e.g., +5% conversion).
   → The smaller the expected effect, the more traffic is needed to detect it with power.
3. Confidence level (e.g., 95%)
   → It determines the threshold above which a result is considered statistically significant.
   → The stricter the level, the lower the power (with a constant sample size).

❗ Why it's crucial in A/B testing

Without sufficient power:

• You risk rejecting a winning variation due to insufficient evidence,
• Your test may be labeled as “neutral” when in fact the effect exists but is masked by a lack of data,
• You waste time, traffic and optimization opportunities.

💡 Conversely, good power strengthens the reliability of learnings, even in the case of a non-significant result.

🧪 Concrete examples

SituationLow powerRiskYou test a variation with a 3% MDE but low traffic40%You have a 60% chance of missing a real effectYou target a variation at +10% on 20,000 sessions85%Good power → reliable conclusion in case of success or failure

✅ Best practices in CRO

• Calculate power before running the test, using a tool (Optimizely, AB Tasty, Evan Miller) or a custom model (BigQuery, R),
• Define a realistic MDE based on business stakes (e.g.: 2% on a homepage ≠ 10% on a product page),
• Extend the test duration or pool traffic to reach sufficient power,
• Never interpret a “neutral” test without having checked its actual power.