Some signs of AI model collapse begin to reveal themselves

A recent article suggests that AI models are beginning to exhibit signs of "model collapse," where accuracy and reliability degrade as AI systems are increasingly trained on their own outputs. This phenomenon, driven by error accumulation, loss of tail data, and reinforcing feedback loops, leads to AI generating questionable or inaccurate results, particularly when searching for specific data like market share statistics. Bloomberg Research found that Retrieval-Augmented Generation (RAG), while reducing AI hallucinations, can also increase the risk of leaking private data and producing biased investment advice, raising concerns about the responsible use of AI and its potential to become less valuable over time.

Analysis

The article posits that Artificial Intelligence (AI) systems are showing initial signs of 'model collapse,' a degradation in performance as they are increasingly trained on their own, potentially flawed, outputs. This phenomenon is characterized by error accumulation, loss of tail data, and reinforcing feedback loops, leading AI to generate less accurate and reliable results, as evidenced by the author's experiences with AI search tools, including Perplexity, providing 'questionable' financial data from non-authoritative sources instead of SEC 10-K filings. This issue isn't isolated, with similar observations across other major AI search bots. A Bloomberg Research study involving 11 leading Large Language Models (LLMs) such as GPT-4o and Claude-3.5-Sonnet found that Retrieval-Augmented Generation (RAG), while reducing hallucinations, paradoxically heightens risks like private data leakage, misleading market analyses, and biased investment advice. The piece expresses significant skepticism regarding the mitigation of model collapse, citing the rapid proliferation of AI-generated content—OpenAI reportedly generates 100 billion words daily—and the tendency for users and businesses to prioritize easily produced 'AI slop' over quality, human-generated content, potentially impacting the long-term value proposition of AI technologies and the companies heavily invested in them, such as Google (GOOG/GOOGL) and Microsoft (MSFT).

Key Decisions for Investors

• Investors should exercise caution regarding the long-term reliability and returns of AI-centric investments, given the articulated risks of 'model collapse' and the potential for deteriorating output quality.
• Scrutinize companies heavily reliant on or developing AI, including Alphabet (GOOG/GOOGL) and Microsoft (MSFT), for transparent strategies addressing data integrity, model degradation, and the sourcing of high-quality training data.
• Apply heightened due diligence to AI-generated financial analysis and investment recommendations, considering the reported increased risk of misleading or biased outputs even from advanced systems utilizing RAG.
• Monitor the evolving landscape of AI-generated content and its potential to pollute data ecosystems, which could impact sectors reliant on authentic and verifiable information.