Nearly every AI coding assistant is sold on a speed promise, and most developers who use one report feeling faster. But when researchers set out to measure that speedup directly, rather than asking developers how fast they felt, the results have been far messier than either side of the AI debate suggests. The most rigorous study available, a 2025 randomized controlled trial from the AI evaluation nonprofit METR, found that experienced open-source developers were actually slower when using AI tools on their own familiar codebases, even though they believed they had sped up. Other research, including GitHub’s own studies and the 2025 DORA report from Google Cloud, points toward real gains, at least for certain tasks, tools, and organizations. This piece walks through what each credible, citable study actually measured, where its conclusions stop, and why the evidence as a whole is still unsettled rather than one-sided.
At a Glance
| Study | Who Ran It | What It Found | Key Caveat |
|---|---|---|---|
| Early-2025 RCT (“Measuring the Impact of Early-2025 AI…”) | METR (independent AI research nonprofit) | Experienced developers were 19% slower on real issues when allowed to use AI, despite forecasting a 24% speedup and later believing they were about 20% faster | Only 16 developers and 246 tasks, all on large, familiar open-source codebases, using Cursor with Claude 3.5/3.7 Sonnet in early 2025 |
| 2026 follow-up / uplift update | METR | Weak signal of a swing toward speedup (returning developers around 18% faster, new developers around 4% faster), but both confidence intervals cross zero | METR itself calls the evidence “very weak” due to selection bias: developers most convinced AI helps increasingly refused to do tasks without it |
| Copilot productivity research (2022) | GitHub (vendor-run) | 95 professional developers writing a new HTTP server from scratch were about 55% faster with Copilot (1h11m vs. 2h41m) and had a higher task-completion rate | Single, well-defined greenfield task; tested 2022 Copilot Technical Preview, not today’s agentic tools; GitHub is the product’s maker |
| 2025 DORA State of AI-Assisted Software Development | Google Cloud / DORA | 90% of developers now use AI; AI adoption is newly linked to higher team delivery throughput, reversing the prior year’s finding; over 80% self-report individual productivity gains | Based on self-reported survey data and organizational correlations, not controlled, task-level timing |
| 2025 Stack Overflow Developer Survey | Stack Overflow | 68% of professional developers use AI tools daily or weekly, yet favorable sentiment fell from over 70% in 2023–2024 to about 61%, and only roughly a third trust AI output’s accuracy | Measures usage and sentiment, not measured speed or output quality |
METR’s Randomized Trial: Slower, Not Faster, Despite Feeling Faster
The study generating the most debate is METR’s “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” published in July 2025. METR recruited 16 developers who were already experienced, long-tenured contributors to large open-source repositories, averaging more than 22,000 GitHub stars and over a million lines of code. The developers brought 246 real issues, mostly bug fixes, features, and refactors, averaging roughly two hours of expected work each. Each issue was randomly assigned to either allow or disallow AI assistance; when allowed, developers mostly used Cursor Pro with Claude 3.5 or 3.7 Sonnet. Work was screen-recorded and self-timed, and developers were paid $150 an hour.
The result ran against expectations on all sides: when AI was allowed, developers took about 19% longer to complete their issues, with a 95% confidence interval spanning roughly 2% to 39% slower. Just as notable was the gap between belief and reality. Before starting, developers forecast that AI would speed them up by about 24%. Afterward, having just experienced the slowdown, they still estimated that AI had made them roughly 20% faster. METR itself is careful about the limits of this finding: 16 developers is a small sample, participants may differ systematically from developers who declined to take part, and the results describe experienced contributors working in large, familiar codebases with high internal standards for documentation and testing. METR also notes it cannot rule out that developers were still climbing the learning curve on Cursor, and that the results reflect tools as they existed in early 2025.
METR’s 2026 Update: A Walk-Back on Certainty, Not a Reversal
METR did not stop there. In a February 2026 update, the organization described a second, larger study begun in August 2025: 57 developers (10 returning from the original study, 47 new), 143 repositories, and more than 800 tasks, this time paid $50 an hour. The raw numbers moved toward speedup: returning developers were estimated to be about 18% faster, and new developers about 4% faster. But both confidence intervals cross zero, meaning the data cannot statistically distinguish those numbers from no effect at all.
More important than the point estimates is what METR says drove them. Between roughly 30% and 50% of surveyed developers reported skipping tasks in the study because they didn’t want to do them without AI, which likely strips out the tasks where AI would have helped the most and biases the measured result toward looking slower than reality. Lower pay than the original study may have compounded the selection problem. METR’s own conclusion is blunt: developers are probably more sped up by AI in early 2026 than the original study found for early 2025, but the organization calls its evidence for the actual size of that improvement “very weak,” and it is redesigning the experiment rather than standing behind a new number. For anyone tracking how autonomous coding agents like Claude Code and Codex differ from older autocomplete-style tools, this matters: the tools used in the second study were materially more advanced than Cursor with Sonnet 3.5, yet the research still couldn’t pin down a confident number.
GitHub’s Own Research: A Large Speedup on a Narrow Task
GitHub’s most frequently cited productivity research dates to 2022, published on its own research blog while Copilot was still in technical preview. It combines two studies. The first was a survey of more than 2,000 developers who had signed up for the Copilot preview, built around the SPACE framework for developer productivity; over 90% agreed they completed repetitive tasks faster, but this is self-reported perception from an early-adopter population, not a measurement.
The second part is a genuine controlled experiment: 95 professional developers were randomly split into two groups and asked to write an HTTP server in JavaScript from scratch, with correctness and completeness scored automatically by a test suite. The group with Copilot access finished in an average of 1 hour 11 minutes versus 2 hours 41 minutes without it, a 55% speed gain that GitHub reports as statistically significant (P=.0017, 95% CI of 21% to 89%), and also had a higher completion rate (78% vs. 70%). The gap with METR’s finding is explained largely by what was being measured: a short, well-specified, greenfield task with no existing codebase conventions to learn or preserve, versus real maintenance work inside large, idiosyncratic repositories. It’s also a vendor-run study of a now-outdated tool version, which is worth weighing alongside the result itself.
DORA 2025: Near-Universal Adoption, and a Flip at the Organizational Level
The DORA (DevOps Research and Assessment) team at Google Cloud takes a different approach entirely: a large annual survey of software professionals correlating AI adoption with team-level delivery outcomes rather than timing individual tasks. The 2025 report found that AI adoption among software professionals had reached about 90%, up 14 points from the prior year, with professionals spending a median of roughly two hours a day working with AI tools. More than 80% of respondents said AI had improved their individual productivity, and 59% reported a positive effect on code quality, both self-reported.
The headline organizational finding is a reversal: in DORA’s 2024 report, higher AI adoption had been associated with lower software delivery throughput and stability; in 2025, adoption is now linked to higher throughput. DORA frames this cautiously, describing AI as a “mirror and a multiplier” that amplifies the strengths of cohesive, well-run engineering organizations and exposes the weaknesses of fragmented ones, rather than a tool with a fixed, universal productivity effect. Trust remains mixed: only about 24% of respondents report a great deal or a lot of trust in AI output. None of this is a controlled measurement of how much faster any individual developer becomes; it is a correlational, self-reported picture of how AI use tracks with outcomes across many organizations.
Stack Overflow 2025: Heavy Use, Cooling Trust
The 2025 Stack Overflow Developer Survey adds a sentiment and adoption layer rather than a productivity measurement. Among professional developers, 50.6% reported using AI tools daily and another 17.4% weekly, for a combined 68%. But favorable sentiment toward AI tools, which had been above 70% in both 2023 and 2024, fell to about 61% among professionals in 2025. Trust in the accuracy of AI output is notably thin: roughly a third of professional respondents say they trust it, while close to half say they distrust it. Over 40% rated AI as bad or very poor at handling complex tasks specifically. Read together with the benchmark scores covered in our explainer on SWE-bench, HumanEval, and what AI coding benchmarks actually measure, this survey is a useful reminder that benchmark performance, daily usage, and developer confidence can all move independently of each other.
Why the Findings Diverge
These studies aren’t simply contradicting each other; they’re measuring different things under different conditions, and the gaps line up with a few recurring variables.
- Task familiarity and codebase complexity. METR’s slower result came from maintenance work inside large, idiosyncratic, long-lived codebases where developers already had deep tacit knowledge; GitHub’s faster result came from a short, self-contained, greenfield task with no existing conventions to navigate. AI assistance appears to help more when there’s little context to absorb and less when a developer already knows the codebase better than the model does.
- Experience level and expectations. METR’s participants were highly experienced and had high internal quality bars (tests, documentation, linting), which may make AI-generated suggestions a net cost to review and correct. Less experienced developers, or those working outside their specialty, may see larger relative gains, though METR’s data doesn’t directly test that.
- Measurement method. Self-reported speed (GitHub’s 90%+ “faster” figure, DORA’s 80%+ productivity figure, developers’ own post-task estimates in METR’s study) consistently reads more positive than directly timed, randomized measurement. METR’s own developers believed they were faster even while a stopwatch showed otherwise, which is a strong argument for discounting pure self-report.
- Tool and model generation. GitHub’s 2022 result used an early Copilot preview; METR’s 2025 result used Cursor with Claude 3.5/3.7 Sonnet; METR’s 2026 follow-up involved more capable agentic tools like Claude Code and Codex. Capability has moved quickly, and a result anchored to one tool generation may not hold for the next.
- Selection bias in who gets studied. METR explicitly flagged that developers most convinced of AI’s value increasingly refused to complete tasks without it, skewing later data. The direction of that bias is toward understating AI’s benefit in the newer study, which is part of why METR won’t commit to a number yet.
Bottom Line
There is no single honest answer to “do AI coding tools make developers faster,” because the best available evidence doesn’t converge on one. The most methodologically rigorous study of experienced developers working in familiar, complex codebases found them slower with AI, not faster, despite feeling faster, a finding worth taking seriously precisely because it cuts against both marketing claims and developer intuition. At the same time, a controlled study of a short, well-defined task found a large, statistically significant speedup, and broad industry survey data shows adoption climbing toward near-universal along with self-reported productivity and, newly, a positive link to team-level delivery throughput. What’s genuinely unsettled is magnitude and conditions, not existence: AI coding tools can speed some work up and slow other work down, and the honest position in late 2026 is that task type, codebase familiarity, developer experience, and tool generation all swing the outcome, with METR’s own follow-up research still unable to put a reliable number on where things stand today.
FAQ
Did the METR study prove AI coding tools make developers slower?
It proved that, for 16 experienced developers doing real maintenance work in large, familiar open-source codebases with early-2025 tools, measured time was about 19% longer with AI allowed. It did not establish that this holds for all developers, all tasks, or current tools; METR’s own 2026 follow-up suggests the picture may now be more favorable to AI, while cautioning that its new evidence is weak.
Why did developers in the METR study believe they were faster when they weren’t?
METR doesn’t fully explain the mechanism, but notes that developers forecast a 24% speedup beforehand and still estimated a roughly 20% speedup afterward despite the opposite result. This mirrors a broader pattern across the research covered here: self-reported speed consistently runs more positive than directly timed results.
Is GitHub’s 55%-faster Copilot statistic still relevant?
It’s a real, statistically significant result from a controlled study, but it dates to a 2022 Copilot preview and a single greenfield task (writing a new HTTP server), and it was run by the tool’s own maker. It says more about short, well-defined, from-scratch tasks than about day-to-day maintenance in large codebases.
Does the DORA report contradict METR’s findings?
Not directly; they measure different things. DORA tracks self-reported individual productivity and organization-level delivery throughput across a large survey population, while METR directly times a controlled sample of experienced developers on real tasks. DORA’s 2025 finding that AI adoption now correlates with higher throughput is an organizational, correlational result, not a controlled measurement of individual task speed.
What should a team take away from this research?
Treat vendor-reported and self-reported speedups skeptically, expect AI assistance to help more on unfamiliar, well-specified, or boilerplate-heavy work than on deep maintenance of complex systems a developer already knows well, and watch for the gap between how fast AI use feels and how fast it actually is.
Sources
- METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” (July 2025)
- METR study paper on arXiv (2507.09089)
- METR, “An update on our uplift study” (February 2026)
- GitHub, “Research: quantifying GitHub Copilot’s impact on developer productivity and happiness” (2022)
- Google Cloud / DORA, 2025 State of AI-Assisted Software Development Report
- Google, “How are developers using AI? Inside our 2025 DORA report”
- Stack Overflow Developer Survey 2025: AI section
Discover more from Software Online Review
Subscribe to get the latest posts sent to your email.
