Researchers give doomsday warning about building AI too fast

A new book by Eliezer Yudkowsky and Nate Soares, "If Anyone Builds it, Everyone Dies," warns of an imminent existential threat from AI, arguing that current large language models are uncontrollable 'alchemy' destined to break free, pursue unpredictable agendas, and lead to human extinction, advocating for an immediate halt to all AI development. While this extreme view faces significant skepticism, the article underscores a more immediate concern for institutional investors: the 'black box problem' of frontier AI models, whose inherent unpredictability and lack of full understanding limit their practical utility and could cause investment in these technologies to slow or dry up without significant breakthroughs.

Analysis

The discourse surrounding AI development is increasingly polarized, as highlighted by the new book from researchers Eliezer Yudkowsky and Nate Soares, which posits an inevitable, near-term existential threat from uncontrollable AI. This extreme 'doomer' perspective, which advocates for a complete and immediate halt to AI development, contrasts sharply with broader expert opinion, where a 2023 survey indicated only 5% of AI researchers believe the technology will lead to human extinction. The more pressing and immediate concern for investors, as framed in the article, is the 'black box problem.' Frontier AI models are described as being 'grown more than they are built,' leading to inherent unpredictability that currently limits their application for critical, high-stakes tasks. This operational unreliability suggests that without significant breakthroughs in model interpretability and control, the utility of these large models may be confined to supplementary features rather than core business functions. Consequently, the current high levels of capital investment in frontier AI face a tangible risk of diminishing returns, potentially slowing or drying up if the technology fails to bridge the gap between its current state and reliable, scalable commercial applications.

Key Decisions for Investors

• Investors should critically evaluate the commercialization timelines for companies heavily invested in frontier AI models, focusing on evidence of solutions to the 'black box problem' that would enable use in mission-critical applications.
• Monitor for technical breakthroughs related to model alignment and interpretability, as these would serve as key de-risking events and potential catalysts for the next phase of AI-driven growth.
• Portfolio construction should account for the high degree of uncertainty; while existential risks are a low-probability tail event, the more immediate risk of a capital investment cycle bust due to stalled technological progress warrants a cautious and diversified approach to the AI sector.