‘A complete accuracy collapse’: Apple throws cold water on the potential of AI reasoning – and it's a huge blow for the likes of OpenAI, Google, and Anthropic

Apple's research indicates that AI reasoning models from companies like OpenAI, Google, and Anthropic face significant limitations in solving complex logic problems, experiencing a complete accuracy collapse beyond a certain level of difficulty. The study challenges the prevailing narrative that these models are highly effective for tasks requiring human-like reasoning, suggesting current benchmarks are flawed and susceptible to data contamination. This casts doubt on the generalizability and real-world applicability of large reasoning models (LRMs) and may lead customers to question the capabilities of AI 'thinking' models.

Analysis

Apple's recent research paper, "The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity," presents a significant challenge to the prevailing optimism surrounding the capabilities of advanced AI reasoning models. The study reveals that leading Large Reasoning Models (LRMs) from prominent developers, including OpenAI’s o1 and o3, DeepSeek R1, Anthropic’s Claude 3.7 Sonnet, and Google’s latest Gemini, experience a "complete accuracy collapse" when confronted with logic puzzles exceeding a certain complexity threshold. This occurs despite the models efficiently handling low-complexity tasks. Apple's researchers contend that current LRM benchmarks, often centered on coding and mathematical problems, are flawed and susceptible to data contamination, failing to provide adequate control over variables. To counter this, Apple devised new puzzles emphasizing logical reasoning, which demonstrated that beyond a specific complexity, model accuracy plummeted to zero, even when the solution algorithm was provided in the prompt, suggesting an inherent limitation. Furthermore, models exhibited 'overthinking' on simpler problems, wasting computational resources after finding a solution. These findings cast serious doubt on the generalizability of LRMs and the claims by developers that these models can 'think' or 'reason' like humans on complex tasks, representing a substantial setback for the narrative that these AIs are ready for sophisticated enterprise applications. The research suggests a potential fundamental flaw in current LRM algorithms, causing them to lose focus or cease effective computation when challenges intensify, which could compel developers to re-evaluate their training methodologies and architectures, and may lead customers to question the true capabilities of these 'thinking' AI systems.

Key Decisions for Investors

• Investors should critically re-evaluate the current capabilities and near-term enterprise adoption potential of advanced AI reasoning models, particularly from companies like Google (GOOGL, GOOG), given Apple's findings of a 'complete accuracy collapse' in complex problem-solving.
• Consider the potential for a recalibration of valuations and growth expectations for companies heavily reliant on the unproven advanced reasoning capabilities of current AI, as these limitations could delay commercialization or require significant further R&D investment.
• Monitor for developments in AI benchmarking standards and new architectural approaches that directly address the identified flaws, as breakthroughs in overcoming these reasoning limitations will be critical for the next wave of AI innovation and market leadership.