Statistical Power
The probability that an experiment detects an effect that really exists. It depends on sample size, the metric's variance and the smallest effect you want to see; Kohavi, Deng and Vermeer's rule of thumb is n = 16σ²/δ² users per variant (95 percent confidence, 80 percent power).
Growth ConceptsTerm 80 / 81
Formula
Users per variant ≈ 16 × σ² ÷ δ²; for conversion σ² = p(1−p), δ = p × relative effect
In practice
A low-powered experiment does not only miss effects; it also makes significant results less trustworthy. Kohavi's table: when 10 percent of experiments truly win and power drops from 80 to 20 percent, 52.9 percent of significant results are false positives. For a low-traffic product the lesson is simple: fewer, bigger experiments rather than more (our reading). The Experiment Capacity tool runs this calculation for your own traffic.