As AI companies get closer to ‘recursive self-improvement,’ this physics professor calls full autonomy ‘the worst idea in the history of humanity’

Anthropic said Claude now contributes to 26% of its model R&D, completing many tasks end-to-end from high-level prompts under human supervision, signaling progress toward recursive self-improvement (RSI). OpenAI has launched an automated research intern and targets an automated AI researcher by March 2028, while Elon Musk has suggested xAI could reach fully automated model improvement by year-end or no later than 2027. The advances raise material AI safety and governance concerns, with Anthropic favoring a verifiable coordinated slowdown and OpenAI acknowledging that alignment progress may not keep pace with increasingly capable systems.

Analysis

For MSFT, the investable implication is less near-term revenue and more a potential shift in the AI risk premium. A deliberately constrained deployment posture can strengthen enterprise procurement confidence and reduce downside from a high-profile safety failure, supporting Azure and Copilot adoption in regulated verticals; however, it risks ceding the “frontier capability” narrative to faster-moving private labs and compressing the multiple premium if customers perceive Microsoft’s models as less capable.

The second-order effect is likely an escalation in governance requirements rather than an immediate development halt. Larger incumbents with cloud distribution, audit tooling, identity controls and enterprise contracts—MSFT, AMZN and GOOGL—should be relatively advantaged if customers or regulators demand provenance, monitoring and human-approval layers. Pure model vendors face a more difficult trade-off: pushing autonomy raises regulatory and reputational tail risk, while slowing raises capital-intensity and competitive risks because compute commitments remain largely fixed.

Over the next days, this is unlikely to alter earnings estimates absent a concrete regulatory action or disclosed capex/guidance change. Over 1-3 months, watch whether enterprise AI buyers explicitly prioritize controllability over benchmark performance; that would favor Microsoft’s distribution-led model. Over 6-18 months, the key variable is whether AI-assisted R&D materially lowers model-development costs: if so, hyperscaler returns on AI capex could improve faster than consensus, but only if incremental demand—not merely cheaper model iteration—absorbs the resulting capacity.

Consensus may overvalue dramatic autonomy claims as a direct catalyst. The more probable commercial outcome is that autonomy becomes embedded in narrow workflows with costly human oversight, limiting immediate margin expansion while increasing demand for security, governance and inference infrastructure. This thesis is falsified if enterprise buyers demonstrably migrate workloads to less-controlled platforms because capability gaps outweigh compliance concerns, or if MSFT signals reduced AI infrastructure spending before utilization improves.

Key Decisions for Investors

• Maintain MSFT as a core AI exposure rather than add on this news alone; reassess after the next Azure growth and AI-capex disclosure. Add only if Azure growth reaccelerates while capex-to-revenue intensity stabilizes, which would validate that governance-led enterprise adoption is monetizing.
• Consider a 3-6 month long MSFT / short ARKK pair for a regulated-enterprise AI adoption scenario: MSFT has distribution and compliance monetization, while ARKK is more exposed to long-duration, sentiment-driven autonomy narratives. Exit if MSFT underperforms ARKK by 10% after a confirmed acceleration in frontier-model capability or Azure guidance weakens.
• Create an alert for concrete U.S. or EU rules requiring AI audit trails, human review, or incident reporting. On verified rulemaking—not rhetoric—evaluate long MSFT versus a basket of smaller AI software names, as compliance costs should favor platform incumbents; no position is warranted until scope and enforcement timing are known.
• Do not position for a near-term AI-development slowdown. A coordinated, verifiable pause remains a low-probability outcome; if one emerges, reduce high-capex hyperscaler exposure initially because utilization and model-release cadence could lag committed infrastructure spend.