Tool · Measuring
How many reliable decisions does this traffic buy you a year?
AI made producing variants free; it did not grow your traffic. This tool takes the real traffic of a page or flow and computes how many weeks one experiment needs, how many reliable decisions a year that allows, and which effects you will never see at this traffic.
Kohavi, Deng, Vermeer (KDD 2022) formula · values are written to the address bar and can be shared
Inputs
Reliable decisions a year: 1 · Duration of one experiment: 33.4 weeks
Result
Reliable decisions a year: 1
Reliable decisions a year
1
Back to back on the same flow. In layered parallel setups each experiment still waits for the same sample.
Seeing this effect at this traffic takes longer than a quarter.
That means at most three decisions a year. Either aim for a bigger effect, measure a step higher in the funnel (more users, higher rate), or move from A/B to coarser but honest methods such as before/after comparisons and holdouts.
Year calendar
Reliable decision: 1
Duration of one experiment
33.4 weeks
4,615 users a week, 153,839 users per experiment.
Users per variant
76,920
Kohavi's shortcut n = 16σ²/δ²; this tool derives the factor from the confidence and power you chose.
False positives among significant results
22%
At a 10 percent win rate and 80 percent power. Logic of Kohavi 2022, Table 2.
What you cannot see at this traffic
If you fix the duration, this is the smallest relative effect you can detect. Gains below it are indistinguishable from noise.
in 2 weeks
40.8%
in 4 weeks
28.9%
in 13 weeks (one quarter)
16%
Signature question
If you measured the 10 percent gain you saw at this traffic again tomorrow, under the same conditions, would you see it?
The calculation assumes a binary metric such as conversion rate and equally split variants. For continuous metrics such as revenue, variance must be measured separately. The result is directional; it does not replace your experimentation platform's own power analysis. Values are written to the link in the address bar.
Method
Where does the formula come from?
The sample size follows the rule of thumb in Kohavi, Deng and Vermeer's KDD 2022 paper: n = 16σ²/δ². n is users per variant, σ² the variance of the metric, δ the smallest absolute difference you want to detect. For a conversion rate σ² = p(1−p) and δ = p × relative effect. The paper's example: at 3.7 percent conversion and a 10 percent relative effect, 41,642 users per variant. The factor 16 is the rounded value for 95 percent confidence and 80 percent power (exact value 15.7); this tool derives the factor from your settings.
The duration of one experiment is total users per experiment divided by weekly traffic, never below two weeks. Microsoft ExP's 2024 study found that 9 percent of 14-day results fall outside the 95 percent confidence interval of the 7-day result; 5 percent would be expected. Reliable decisions a year is 52 weeks divided by that duration.
The false positive share follows the logic of the same paper's section on low-powered experiments: the share of significant results that are not real depends on your win rate and your power. At a 10 percent win rate and 80 percent power, 22 percent of significant results are false positives; at 20 percent power, 52.9 percent. The inflation check computes the actual power each experiment gets at your planned pace and recomputes that share.
Do not stop at the first significant glance: Evan Miller's 2010 calculation shows that checking significance after every observation and stopping when significant turns a 5 percent target into a 26 percent real false positive rate. Sequential testing (Johari, Pekelis, Walsh) allows peeking; a fixed-sample experiment does not.
The idea behind this tool: apparent experiment speed and real learning speed are not the same thing. Experiment Inflation →
Frequently asked
With AI we ship ten variants a week. Why doesn't this tool change its answer?
Because the sample size does not depend on who wrote the variant. There is no AI in the formula: only traffic, base rate and the effect you want to see. Building became free; knowing did not. Adding variants splits traffic and lowers power per experiment.
We run experiments in parallel. Why is the yearly count computed back to back?
In a layered setup several experiments can share the same traffic, but each still waits for its own sample from that same traffic. The bottleneck is not 'how many at once' but 'how many weeks each, and at what effect size'. Overlapping experiments on the same page or flow also carry interaction risk.
The result says 'cannot be seen at this traffic'. What should I do?
Three paths: make a bigger change and look for a bigger effect; measure a step higher in the funnel that sees more users; or move from A/B to coarser but honest methods such as before/after comparisons, cohort tracking and holdouts. Deciding small improvements by principle, without measuring, is also a method; an A/B test run as a ritual is not.
My base conversion is very low, like 0.5 percent. Does the tool still work?
It works, but the required sample grows a lot; low-rate metrics need disproportionate traffic to show an effect. Often the right answer is to measure a higher intermediate metric, such as reaching the checkout page. That is the same direction as Kohavi's advice on sensitive surrogate metrics.
Why do you ask for the win rate?
Because the probability that a significant result is real depends on how many of your experiments truly work. The large-company rates Kohavi reports range from 8 to 33 percent. If you do not know yours, 10 percent is a reasonable start; an optimistic guess makes the tool optimistic too.
Can I use it for a metric like revenue or basket size?
This version assumes a binary metric such as conversion and builds variance as p(1−p). For continuous metrics you need to measure variance from your own data and put it into the same formula. Variance reduction techniques such as CUPED also lower the sample; they are not included here.
Sources
- [1]Kohavi, Deng, Vermeer: A/B Testing Intuition Busters, KDD 2022 (sample-size formula, low power and false positive table)
- [2]Microsoft ExP: External Validity of Online Experiments, 2024 (7-day results not always predicting 14 days)
- [3]Evan Miller: How Not To Run an A/B Test, 2010 (repeated significance testing)
- [4]Johari, Pekelis, Walsh: Always Valid Inference, 2015 (sequential testing)
- [5]Georgiev: What Can Be Learned From 1,001 A/B Tests?, 2022 (median effect and average effect among winners)
- [6]Statsig docs: Power Analysis (definition of minimum detectable effect)