AI coding tools may not speed up every developer, study shows

A new study by AI research group METR challenges the widely assumed productivity gains of AI coding tools for experienced developers, reporting that their use *increased* task completion time by 19% against a forecasted 24% reduction. This randomized controlled trial, involving seasoned open-source developers on complex codebases, suggests current 'vibe coders' may introduce inefficiencies like increased prompting time and struggle with large systems. The findings question the immediate ROI and universal workflow acceleration narrative for investors regarding AI software engineering tools, despite researchers' caveats about AI's rapid evolution and other studies showing positive impacts.

Analysis

A recent study by the non-profit research group METR introduces a significant counterpoint to the prevailing narrative that AI coding tools universally enhance developer productivity. The randomized controlled trial, involving 16 experienced open-source developers working on large code repositories, found that access to state-of-the-art AI tools increased task completion time by 19%, directly contradicting the participants' forecast of a 24% time reduction. Researchers theorize this inefficiency may stem from time lost to prompting the AI and the models' struggles with complex codebases. However, the findings are heavily caveated; the study's authors acknowledge other, larger studies have shown productivity gains and emphasize the rapid pace of AI model improvement could render these results obsolete within months. Additionally, while most participants had experience with LLMs, 44% were first-time users of Cursor, the primary tool provided, which could have impacted performance despite training. This report, alongside mentions of other studies flagging AI-introduced code errors and security vulnerabilities, suggests the path to realizing productivity gains is not linear and that the practical application of AI in specialized, high-stakes environments like enterprise software development remains a challenge.

Key Decisions for Investors

• Investors should temper near-term expectations for productivity-driven growth in the AI developer tool sector, as this study suggests the return on investment for experienced developers in complex environments may be delayed or lower than hyped.
• Monitor specific adoption metrics beyond headline user growth, such as user retention and reported efficiency gains on complex, real-world projects, to better gauge the actual value proposition and product-market fit of these tools.
• Differentiate between investments in foundational model providers like Google and application-layer toolmakers, as the study's findings reflect challenges in application and integration rather than a fundamental flaw in the core AI models.
• Consider the secondary implications, such as the rising need for AI code auditing and security solutions, which could present opportunities in the cybersecurity sector as enterprise adoption of AI coding assistants continues.