Experiment Inflation: AI Made Building Free, Not Knowing

The story starts like this (an example). Six months ago a growth team could ship one A/B test a week. Design, code, event tracking, analysis; each took a few days. Then AI arrived. Variants are written in minutes, landing pages are ready in an afternoon, SQL writes itself. Now they run ten experiments a week. The experiment count on the dashboard went up tenfold.

The growth number stayed where it was.

I am writing this for that team's question: “We do everything faster with AI, so why are we still not growing?” The answer is not in a tool or a model. It is in which step of the growth loop got cheap, and which one did not.

Building became free, knowing did not

The classic growth loop has four steps: form a hypothesis, build the variant, measure, learn. AI pushed the second step close to zero. That is a real gain; do not belittle it. Writing variants, wiring flows, producing lifecycle messages per segment, pulling cohorts without waiting for an analyst; all of that used to take weeks and now takes hours. The long tail, the list of things that would be nice to do if anyone had time, became doable for the first time.

But the third step, measuring, did not get cheaper with AI. The raw material of measurement is not code. It is traffic. For an experiment to tell you anything, enough people have to see it. AI does not give you people. Your traffic today is whatever it was six months ago.

The bottleneck did not disappear. It moved. You used to be stuck at building; now you are stuck at knowing. And a team that is stuck at knowing and responds by running more experiments is like attaching more water mills to the same river without adding water. Each mill turns slower. And every one of them looks like it is turning.

Experiment inflation

I call this experiment inflation. In an inflationary economy nominal income rises while purchasing power does not. In experiment inflation the nominal number of tests rises while learning does not. The mechanism is simple.

Test count goes up. Traffic stays flat. Traffic per test goes down. Statistical power per test goes down. Low-powered tests produce two things: real effects you cannot see, and fake effects you think you see. The second is more dangerous, because the dashboard labels it a winner.

Let us put numbers on it. In a 2022 paper, Ron Kohavi and colleagues give a simple formula for the users needed per variant: sixteen times the variance divided by the square of the smallest effect you want to detect. For a flow converting at 2 percent, reliably seeing a 10 percent relative lift needs roughly 78 thousand users per variant. About 157 thousand for two variants.

Now put that on an example: a product with 20 thousand monthly visitors. A single experiment needs about eight months. Eight months. If you are looking for a 20 percent lift, two months. Attach ten tests a week to that traffic and each one sees a tenth of those 20 thousand, and none of them can ever tell you anything.

One section heading in Kohavi's paper reads, verbatim, that experiments with low statistical power are not trustworthy. The general guidance is equally plain: A/B tests are useful when you have at least thousands, preferably tens of thousands, of active users.

Where the fake winners come from

Low power does not only mean “you cannot see”. It means seeing wrong. Three mechanisms are at work, and all three grow with test count.

First, the false positive rate. By Kohavi's table, if 10 percent of your ideas truly work and your tests run at 80 percent power, 22 percent of your significant results are false. Drop power to 20 percent, which is exactly what experiment inflation does, and 53 percent of significant results are false. More than half of your winners are not winners.

Second, the winner's curse. The mechanism Kohavi describes is this: a significant result at low power systematically overstates the true effect. For the scale, look at Georgi Georgiev's study of 1,001 real A/B tests: the median effect across all tests was 0.08 percent, essentially zero; among tests declared winners, the average effect was 15.9 percent. Georgiev attributes the high share of winners (33.5 percent) to the relatively higher quality of the tested ideas and their implementation; combined with low power, it is the mechanism that explains why a dashboard number shrinks in production (opinion).

Third, the peeking habit. In Evan Miller's 2010 simulation, if you check after every observation and stop the moment the result is significant, the 5 percent false positive rate you designed rises to 26 percent. The more often you look, the larger it gets. A team running ten tests a week looks at ten dashboards every day. AI made dashboards cheap too.

Then there is novelty. Microsoft's experimentation platform found in a 2024 analysis that 9 percent of fourteen-day results fall outside the seven-day confidence interval; 5 percent would be expected. Tests that are run fast and closed fast measure the first week's excitement, not behavior.

None of this is AI's fault. All of it existed before. AI simply let you produce it ten times faster.

The signature question

Before starting any experiment, ask one question: “Do I have the traffic to see the effect this test is looking for? If not, why am I running it?”

The question makes half the team uncomfortable, because the answer is usually “no”. Then there are two options: look for a bigger effect, or skip the test and learn another way. Both are legitimate. What is not legitimate is running without knowing the answer.

You do not have to do the arithmetic in your head. /tools/experiment-capacity (the Experiment Capacity Calculator) asks for your monthly traffic, your base conversion rate and the smallest effect you care about; it shows how many weeks one test takes, how many reliable decisions you can make per month, and where your false winner rate ends up when power drops.

What to do at low traffic

Having no traffic does not mean abandoning experimentation. It means increasing decisions per visitor instead of tests per week. Five steps.

Step one: big bets. Drop the 3 percent button-color effects. If your traffic can only see effects of 20 percent and above, only test changes that could plausibly produce 20 percent and above: the whole onboarding flow, the structure of the pricing page, the path to the first moment of value. Do not test small bets. Decide and move on.

Step two: pre-register. Before starting, write down the effect threshold and the duration. This is exactly Kohavi's recommendation: experimentation systems should encourage users to pre-register the minimum effect of interest. A written threshold is the antidote to peeking.

Step three: at least two weeks. The rule Kohavi has recommended since 2014 for novelty effects. A test closed fast is a test that is wrong fast.

Step four: reduce variance. Instead of running longer, measure sharper. Methods that use a user's pre-experiment behavior as a covariate let you see smaller effects with the same traffic. This is the least known lever available to a low-traffic team.

Step five: a holdout group. Once personalization and agent-driven flows arrive, measuring individual variants loses meaning; you are showing everyone something different. Then keep a share of users away from new features for months and measure the system's total effect. Statsig's documentation puts a typical holdout at 1 to 2 percent; at low traffic that share is meaningless and you need 10 percent or more (opinion). It is expensive. Uncontrolled personalization is more expensive.

Three patterns

There are three common bad patterns (opinion); look for them on your own dashboard.

The experiment count metric. If the sprint report says “we ran 40 experiments this month”, the team thinks the experiment is the output. The experiment is the input. The output is a changed decision. Add a line to the report: “how many decisions did we change this month on reliable evidence”. A small number there is normal (opinion).

The winner collection. If three or four winners are declared every month and the conversion rate is unchanged at year end, the winner's curse is at work. Re-measure your winners six months later; most effects will have shrunk (opinion).

The dashboard race. If tests close before their planned duration “because they became significant”, your false positive rate is not 5 percent; in Miller's simulation it rises as high as 26. Give the authority to close a test to the calendar and to nobody else.

Roadmap

Week one: count the experiments of the last three months. For each, compute how many users per variant it saw and the smallest effect it could have seen at that traffic. Show the result to the team. Seeing together that most tests were never in a position to say anything is more convincing than this article.

Week two: add three mandatory fields to the experiment template: smallest effect sought, required sample size, planned end date. A test without all three does not start.

Month one: re-sort the experiment backlog by effect size. Remove effects your traffic cannot see; settle those without a test. Run the rest one at a time, at full traffic.

Month three: spend the time AI gave you on knowing instead of building. Fix the event schema, prepare covariates, set up the holdout. AI helps here too; have it write the analysis code and pull the cohorts. If the bottleneck moved, the investment has to move with it.

Setting expectations

Follow this path and your test count will fall. Perhaps from ten a week to three a month. The team will feel it as a step back. It is not. By Kohavi's account, most experiments at Bing fail, about a third succeed across Microsoft, and the ones that succeed improve key metrics by 0.1 to 1 percent. Even on the world's largest experimentation platforms, learning is slow and expensive. It will be in your product too. The difference is between accepting the slowness and learning reliably, or running fast and learning fiction.

AI gave you one thing: it zeroed the cost of building. If you invest the freed time in knowing, that is a large gift. If you invest it in building more tests, it is only a larger dashboard. Which one you chose will be written, six months from now, not on the dashboard but on the income statement.

Sources

Ron Kohavi, Alex Deng, Lukas Vermeer, “A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments”, KDD 2022: https://bit.ly/ABTestingIntuitionBusters

Ron Kohavi, Alex Deng, Roger Longbotham, Ya Xu, “Seven Rules of Thumb for Web Site Experimenters”, KDD 2014: https://exp-platform.com/Documents/2014%20experimentersRulesOfThumb.pdf

Georgi Georgiev, “What Can Be Learned From 1,001 A/B Tests?”, 2022: https://blog.analytics-toolkit.com/2022/what-can-be-learned-from-1001-a-b-tests/

Evan Miller, “How Not To Run an A/B Test”, 2010: https://www.evanmiller.org/how-not-to-run-an-ab-test.html

Microsoft Experimentation Platform, “External Validity of Online Experiments: Can We Predict the Future?”, 2024: https://www.microsoft.com/en-us/research/articles/external-validity-of-online-experiments-can-we-predict-the-future/

Statsig, “Holdouts”, product documentation: https://docs.statsig.com/experiments/holdouts-introduction