I'm surprised (but not completely) to see most of the comments pushing back at the results.

I agree that having more background data will be necessary or that the state of AI-assisted tooling in 2024 was way different than it is today (BTW Sergey, GitHub Copilot was launched in 2021, Cursor AI was first launched in 2023, etc), but what serious study or more than just anecdotal evidence is anyone putting forward as a counterargument?

If your pushbacks are based on your perceptions, I have bad news: those have been demonstrated in other studies, such as the METR study, to be completely unreliable.

Either most people have literally fallen for the marketing propaganda of the companies that wish you to believe their tools are much better than they really are in practice, or somehow there is so much desire to believe in a 10x improvement that anything pointing at it not being possible is just discarded as "bad propaganda" or a "junk study".

To all of you sharing this sentiment: please share studies conducted by independent entities proving this one (or all the other ones) wrong. Let's be scientific about the issue and forget about whether we like team A or team B.

And to be clear, I'm in no way affiliated with Engineering Enablement or GetDX.