Evidence review: how much does AI really save in coding?

In short A single percentage is no basis for planning - what matters is what was measured.

What it is about

On a single task, AI saves around 21 to 55.8% of the time in experiments. Across a whole project much less remains: converted to project level, field experiments land at around 5 to 14%.

Why the numbers are so far apart

• One task in the lab: The 55.8% (Peng et al., 2023) comes from a controlled experiment with recruited developers and a single task written from scratch (an HTTP server in JavaScript). Authors at Microsoft Research and GitHub.
• One task in enterprise code: Google researchers measured roughly 21% less time in a randomised trial with 96 Google engineers (2024) - one task, one company, a wide confidence interval. A Google-internal study.
• Real work at three companies: In three field experiments with 4,867 developers, completed tasks rose by 26.08%, with a standard error of 10.3% (Cui, Demirer et al., 2025). This counts volume, not outcome. Co-authors at Microsoft.
• What comes after: At a B2B software company with a “double the output” mandate, throughput per developer reached 2.09x - and load per reviewer roughly doubled as well. A longitudinal study of 802 developers and 196,212 pull requests from January 2024 to April 2026, without randomised adoption (He, Denisov-Blanch et al., 2026, preprint, no vendor involvement).

Three of the four studies involved the vendors. They are controlled experiments with a disclosed method, but not independent - which is why the affiliation is noted for each study.

From task to project

The studies measure at different levels. Lab values such as the 55.8% measure only the efficiency lever on an isolated task. Field experiments already include the context of real code bases; only the share of working time the lever acts on is missing. Normalised through the same formula, the field experiments land at around 5 to 14% in the project. The full calculation is in 85% on the slide, 36% in the project.

A special case

In early 2025, 16 experienced open-source developers worked on 246 real tasks in their own large repositories, randomly with or without AI. With AI they took 19% longer - and afterwards still believed they had been 20% faster (METR, 2025). This is no value for the effect; sample and setting are too specific. But it shows how far perception and measurement can diverge.

A side note: according to METR, the repeat from August 2025 produced no reliable result because many developers no longer wanted to work without AI - 30 to 50% did not even submit some tasks (METR, 2026).

What we make of it

The saving on a task is real, but it is not the saving in a project. In between lie context readiness, share of activity and the effort for review. For offers this means a range with stated conditions instead of a single percentage - measured in the actual project.