The cleanest outcome within the METR trial can also be the one most definitely to be misinterpret.
In a randomized managed trial posted in July 2025, METR reported that skilled open-source builders took 19% longer to finish actual coding duties once they had been allowed to make use of AI instruments. After the research, those self same builders estimated, on common, that AI had made them 20% sooner.
That is one research, not settled consensus. It was run in a specific setting: 16 skilled open-source builders, 246 duties, mature repositories, and AI instruments accessible between February and June 2025. METR itself now labels the outcome as historic, saying it now not essentially displays present AI instruments or present developer workflows.
That caveat issues. The discovering just isn’t that AI can’t assist programmers. It’s that, on this trial, self-reported pace and measured completion time pointed in reverse instructions.
What METR truly examined
METR, quick for Mannequin Analysis and Menace Analysis, is a nonprofit analysis group that evaluates AI methods. The research was launched because the paper Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, by Joel Becker, Nate Rush, Elizabeth Barnes and David Rein. METR additionally printed a plain-language write-up of the outcome.
The research design was unusually near actual work. The builders weren’t requested to unravel toy issues or timed coding puzzles. They labored on points from open-source repositories they already knew nicely. On common, they’d about 5 years of prior expertise with the initiatives concerned.
Every job was randomly assigned to one among two circumstances: AI allowed or AI disallowed. When AI was allowed, builders might use the instruments they most popular. In apply, METR says they primarily used Cursor Professional with Claude 3.5 or Claude 3.7 Sonnet, together with chat, agent mode and autocomplete options.
The duties had been actual bug fixes, options and refactors. They averaged about two hours every. Builders recorded their screens and reported the implementation time required.
That design makes the outcome extra helpful than a broad survey asking whether or not AI feels useful. It additionally retains the outcome slim. The trial studied skilled contributors in mature, high-quality open-source codebases. It doesn’t mechanically describe junior programmers, greenfield initiatives, enterprise groups, one-off scripts, low-stakes prototypes, or later generations of instruments.
The measured slowdown
Earlier than beginning their duties, builders anticipated AI entry to scale back completion time by 24%. After the research, they nonetheless estimated that AI had diminished completion time by 20%.
The measured outcome went the opposite approach. METR discovered that permitting AI elevated completion time by 19%.
The pattern was small in individuals and bigger in duties: 16 builders and 246 accomplished points. As a result of every developer labored throughout each circumstances, the researchers might evaluate AI-allowed and AI-disallowed work inside the identical pool of individuals moderately than evaluating two fully separate teams.
METR’s write-up says the slowdown endured throughout a variety of checks and various analyses. The paper additionally examined doable explanations, together with experimental artifacts, undertaking high quality requirements, developer expertise with AI, and the type of duties being tried.
The outcome ought to nonetheless be dealt with fastidiously. A 19% slowdown on this setting just isn’t a common estimate of AI’s impact on coding. It’s an estimate from this trial, utilizing these builders, these duties, these repositories and early-2025 instruments.
Why notion and time might diverge
The hole between the builders’ estimates and the measured result’s the half that issues past coding.
There are a number of believable causes AI might really feel sooner whereas producing slower job completion. A software could make particular person moments really feel simpler whereas including overhead throughout the entire job. It may well draft code shortly, however the developer could then spend time studying, checking, correcting, adapting and integrating that output. It may well scale back the unpleasantness of beginning a job whereas quietly rising assessment time later.
In skilled codebases, a lot of the work just isn’t typing. It’s understanding implicit constraints, preserving type, noticing edge instances, respecting checks, sustaining abstractions, and making adjustments that can survive assessment. A mannequin might be helpful and nonetheless miss a few of that context. The price of discovering and repairing these misses could fall on the human developer.
There may be additionally a psychological downside. If AI produces one thing shortly, the seen first step seems accelerated. The later checking could really feel like regular engineering work, not like AI overhead. A developer can subsequently keep in mind the help extra clearly than the time spent validating it.
That doesn’t make the builders silly. It makes the research a warning about measurement. Human impressions of productiveness are noisy, particularly when a software adjustments the feel of labor moderately than merely making the identical work shorter.
What the research doesn’t say
METR was specific about limits which can be straightforward to lose in retelling.
The research doesn’t present that AI instruments fail to hurry up most builders. It doesn’t present that AI instruments fail in different domains. It doesn’t present that future AI methods will fail in the identical setting. It doesn’t rule out higher workflows, higher prompting, higher scaffolding, repository-specific fine-tuning, or use instances the place AI is extra useful.
It additionally doesn’t imply builders ought to ignore AI in the event that they discover it helpful. Productiveness is just one dimension of software use. A developer may use AI as a result of it makes work extra nice, helps with exploration, lowers friction, teaches unfamiliar APIs, or makes some duties much less tedious. These advantages could matter even when a slim timing measure doesn’t enhance.
However the trial does problem a standard assumption: that perceived speedup is a dependable substitute for measured speedup. On this case, it was not. The builders believed the software had made them sooner after finishing the research, regardless that the measured job instances confirmed the other.
The result’s already growing old
AI coding instruments modified shortly after the research interval. METR’s trial coated instruments accessible from February to June 2025. By early 2026, the group had began a follow-up experiment after which introduced it was altering the research design as a result of the brand new knowledge had change into onerous to interpret.
In that February 2026 update, METR stated later outcomes confirmed some proof of speedup, however choice results made the central estimate unreliable. Extra builders had been reluctant to take part if they may need to work with out AI, and a few averted submitting duties they particularly needed AI for. Which means the later experiment could have missed the individuals and duties the place AI was anticipated to assist most.
This is a vital correction to the straightforward story. The 2025 trial is powerful proof about one second and one setting. It’s weaker proof about AI coding instruments now, and weaker nonetheless about what they’ll change into.
The extra sturdy lesson is methodological. If a software is vital sufficient to reshape work, it is necessary sufficient to measure with one thing higher than vibes. Surveys can inform us what employees really feel. Benchmarks can inform us what fashions can do beneath simplified circumstances. Discipline trials can present what occurs when actual individuals use actual instruments on actual duties. None is full by itself.
A productiveness declare ought to survive contact with time
The METR result’s uncomfortable as a result of it cuts in opposition to a well-recognized know-how story. A software might be succesful, spectacular and extensively appreciated whereas nonetheless failing to shorten a particular type of work in a particular setting.
That’s not a contradiction. It’s a reminder that software program growth just isn’t solely code era. The work contains judgment, reminiscence, assessment, coordination and accountability for penalties. AI could assist some components of that course of whereas slowing others.
The placing a part of the research is not only that skilled builders had been slower with AI. It’s that they thought they’d been sooner. For anybody attempting to grasp AI’s impact on work, that hole will be the extra vital discovering. Productiveness claims mustn’t finish on the feeling of acceleration. They need to survive contact with the clock.