> METR also conducted an in-depth study that asked experienced developers to code with a popular AI assistant. After they finished their tasks, the developers claimed that using the AI had made them 20 percent more productive. But independent evaluators in the study actually concluded that using AI did the opposite: it increased task completion time by about 20 percent.
This somewhat reflects my experience... I can totally see the back-and-forth dance taking longer in some cases.
But I also think there is more than efficiency being unlocked here. Sure, a developer might have cranked out a rough MVP in less time, but with this they're also often continuously updating a README, tests and other infrastructure as they go.
One could argue about whether that's its own footgun - relying on tests that don't really test what they should, and let critical bugs through later.
This somewhat reflects my experience... I can totally see the back-and-forth dance taking longer in some cases.
But I also think there is more than efficiency being unlocked here. Sure, a developer might have cranked out a rough MVP in less time, but with this they're also often continuously updating a README, tests and other infrastructure as they go.
One could argue about whether that's its own footgun - relying on tests that don't really test what they should, and let critical bugs through later.