Yes, but their newer write-up discusses that (and shows that the self-reported numbers have gone up radically, in a way that suggests that even if there is some inflation, the numbers are almost certainly positive if you deflate).
They also have an update -- linked from the original study! -- explaining that it's out of date and no longer reliable, and explaining why they had to cancel a follow-up study because it was understating productivity gains (but also was showing wins for the people who carried over from their previous study): https://metr.org/blog/2026-02-24-uplift-update/
The authors of this paper decided to ignore all of METR's follow-up data and discussion, and to report only the ancient number from early 2025 (a time when Windsurf was state of the art). And then, rather than apologizing for it, and caveating it as a number not to be taken seriously, they described it as a study done "recently."
That's either shockingly dishonest or incredibly out-of-touch.
> shows that the self-reported numbers have gone up radically, in a way that suggests that even if there is some inflation, the numbers are almost certainly positive if you deflate
Or the hype machine is even stronger, making people inflate those numbers even more.
If I wanted to be uncharitable, the question of "how much slower you'd be without AI" compares against people's AI-less productivity from 2026, which could have become much less than 2025 AI-less productivity, if hypothetically AI use makes people helpless and deskilled.
They also have an update -- linked from the original study! -- explaining that it's out of date and no longer reliable, and explaining why they had to cancel a follow-up study because it was understating productivity gains (but also was showing wins for the people who carried over from their previous study): https://metr.org/blog/2026-02-24-uplift-update/
The authors of this paper decided to ignore all of METR's follow-up data and discussion, and to report only the ancient number from early 2025 (a time when Windsurf was state of the art). And then, rather than apologizing for it, and caveating it as a number not to be taken seriously, they described it as a study done "recently."
That's either shockingly dishonest or incredibly out-of-touch.