Measure effects instead of estimating them - which is what we do in our projects.
What it is about
METR had experienced open-source developers work on real tasks in their own projects - randomly with or without AI tools. With AI, they took around 19% longer on average.
What surprised us
The participants themselves estimated they had been about 20% faster with AI. Perceived and measured effect were far apart.
What we take from it
The study reflects an early state of the tools and experienced people in code they know very well. Still, the lesson is clear: measure effects, don’t estimate them. That is why our references state how results were measured - see From ticket to pull request and 85% on the slide, 36% in the project.
