Alp SuleimanGrowth Partner

The Real Growth of AI: Why Perceived and Measured Speedup Diverge, and How That Shapes Growth Decisions

Developers believed AI made them 20 percent faster; in the measurement, tasks took 19 percent longer. How to read “AI sped us up” the way you would read real growth.

Growth PartnerArticle 01 / 027 min read

Published

In the first half of 2025, the research organization METR ran an experiment with 16 experienced developers who had spent years working on large open-source projects. 246 real tasks were taken from their own projects, and for each one it was randomly decided whether AI could be used. Before starting, the developers predicted AI would speed them up by 24 percent. The measurement showed the opposite: when AI was allowed, tasks took 19 percent longer.

The interesting part comes after. Once the experiment was over, the same developers who had lived through the slowdown still believed AI had made them 20 percent faster.

This is not an article about coding. It is the question this site has been asking from the start, in a new disguise: the gap between apparent growth and real growth. In an inflationary economy, revenue rising by, say, 60 percent does not mean the company grew 60 percent; first you have to take out the price effect. Speed that comes with AI works the same way. Felt speed is the nominal number. To get to the real one, you have to subtract a few things from it: the expectation, the choice of task, the rework that shows up later.

Why do perception and measurement diverge?

To be fair first: METR's finding does not say “AI doesn't work.” The researchers state it plainly: the study does not provide evidence that AI fails to speed up most developers, and they do not generalize the results beyond software development. The developers in the study were working on projects they knew very well, averaging more than 22,000 stars and over a million lines of code, on tasks of about two hours each. The tool was mainly Cursor Pro (with Claude 3.5/3.7 Sonnet). Among the possible explanations for the slowdown, METR lists the high quality standards of these projects.

At the other end is the experiment GitHub published in 2022. 95 professional developers were asked to write an HTTP server in JavaScript. The group using Copilot finished in 1 hour 11 minutes on average; the group without it took 2 hours 41 minutes: 55 percent faster. The share who completed the task was 78 percent in the Copilot group and 70 percent in the other.

The two experiments don't contradict each other; they measure different things. One is a well-defined task built from scratch, identical for everyone. The other is real work inside a large system you have maintained for years and whose rules you know by heart. AI can contribute a lot to the first and little, or even negatively, to the second. There is no single answer to “how much does AI speed us up?”; the answer depends on the type of task.

A hint about why perception drifts comes from Stack Overflow's 2025 developer survey. 84 percent of developers use or plan to use AI tools. But more of them distrust the accuracy of AI (46 percent) than trust it (33 percent). The most common complaint, chosen by 66 percent, is solutions that are “almost right, but not quite.” An almost-right output can feel fast: something has appeared on the screen, the work seems to be moving. The time spent turning it into fully right may be scattered, and may never be remembered as a separate line item.

The amplifier: it improves a good system and magnifies a bad one

Google Cloud's DORA 2025 report, drawing on nearly 5,000 technology professionals, paints a large-scale version of the same picture. 90 percent of respondents use AI at work, and more than 80 percent believe it has increased their productivity. This year the report sees a positive relationship between AI adoption and software delivery throughput. But the same report says AI adoption still has a negative relationship with delivery stability: speed goes up, and without strong automated testing, mature version control and fast feedback loops, the higher volume of change turns into instability.

The report's key insight sums it up: AI doesn't fix a team; it amplifies what's already there. Strong teams get better; struggling teams find their problems more visible and more intense.

For a growth leader, that line is a warning. AI adds speed; the system decides where that speed flows. Add speed to a system without measurement and you go in the wrong direction faster, and notice it later.

The same trap in the growth team

From here on this is an inference: the data comes from software development, and carrying it over to growth metrics is our reading. Still, what these findings are really about is measurement more than engineering. Sentences commonly heard about AI in growth teams (illustrative, not quotes from a particular team) share the same structure:

  • “Our experiment velocity doubled.” Launching more A/B tests a week does not mean learning more. Did the tests close with enough sample, or did AI only speed up the production of variants? If change volume went up and decision quality went down, that can be read as the growth-team version of DORA's stability finding.
  • “AI onboarding raised activation.” What did you measure activation with? A user getting past a screen, or experiencing the product's core value for the first time? AI can make the first step easier; and when the first step gets easier, you may have quietly loosened the definition of activation.
  • “Personalized messages improved retention.” In which cohort, in which month? A message being clicked and a customer still paying in month three are different things. A first-week lift is like nominal sales growth in an inflationary economy; it is real only if it holds in the tail of the cohort curve.
  • “The team is X percent more productive with AI.” Does that number come from a survey or from a time measurement? METR's developers also said they were faster when asked at the end of the experiment.

The pattern is the same in each: a felt or intermediate metric takes the place of the outcome metric.

Five questions for a real reading

When you hear “AI made us X percent faster” or “X percent bigger,” ask these five questions, the way you would adjust for inflation.

1. Which task? Well-defined work from scratch, or complex, context-heavy work inside an existing system? A meaningful part of the gap between GitHub and METR probably lives in the answer to this question. METR itself notes it is plausible or likely that AI tools are useful for less experienced developers, or for developers working in an unfamiliar codebase. Don't spread a gain from one type of task across the whole team or the whole funnel.

2. Who? Someone new to the area, or someone who knows it by heart? METR's experienced developers hint that for people who already know a lot, what AI can add may be limited. On your team, who gains and who loses?

3. What was measured: perception, time or outcome? “We believe,” “we feel,” “80 percent of us think we're more productive” are measurements of perception. They are valuable, but they don't replace time and outcome. The core of METR's finding is this: the same people, on their own real work, can say they got 20 percent faster while the measurement shows the work took 19 percent longer.

4. What do stability and quality show? Is there a quality metric next to the speed metric? In software that means reverted changes and incidents; in growth it can be refunds, support tickets, cancellations, the reproducibility of an experiment result. If speed rises while stability falls, then, as DORA puts it, the system is being amplified, not fixed.

5. What is the counterfactual? What would have happened without AI? A randomly assigned control group, an older cohort, a similar team over the same period. Without a comparison, the number also carries seasonality, the market's overall growth, and the team simply getting better anyway. This is not just a theoretical difficulty. In August 2025 METR started repeating the experiment with a larger group and found that some participants weren't submitting some tasks at all because they didn't want to do them without AI (based on survey responses, 30 to 50 percent of developers were not submitting some tasks). They wrote that their new results were only “very weak evidence” for the size of that increase in speedup over early 2025, and decided to change the design of the experiment. The more a tool is adopted, the harder the counterfactual becomes to build. That is why it has to be built early, before the tool is everywhere.

Closing

The point of this article is not that AI doesn't speed things up. METR's early-2026 update also says, based on conversations with participants, that developers are likely more sped up now than in early 2025. They are probably right.

The question is which number you run that speed on. Nominal speed looks good in a meeting. Real speed is the only number that can carry budget, hiring and roadmap decisions. The next time you hear “X percent with AI,” ask the five questions and subtract what needs subtracting.

Take care.

Sources