Cloudflare says Perplexity’s AI bots are ‘stealth crawling’ blocked sites

Cloudflare alleges that AI search startup Perplexity is deliberately circumventing website content restrictions, including robots.txt and WAF rules, by impersonating standard browsers and using rotating IP addresses to mask its AI crawlers. This activity, observed across tens of thousands of domains and millions of daily requests, exacerbates prior concerns about Perplexity's content acquisition practices, despite the company's dismissal of the report as a 'publicity stunt.' Cloudflare has since delisted Perplexity as a verified bot and deployed new blocking methods, underscoring the escalating conflict between AI models and content publishers over data scraping and intellectual property rights.

Analysis

A report from Cloudflare (NET) alleges that AI search startup Perplexity is actively circumventing website restrictions to scrape content, escalating a significant conflict over data rights in the AI industry. Cloudflare provides specific evidence of this activity, claiming Perplexity's crawlers, when blocked, will conceal their identity by impersonating a standard user agent, such as "Google Chrome on macOS," and utilize rotating IP addresses and autonomous system networks (ASNs) not declared on its public lists. The scale of this alleged activity is substantial, impacting "tens of thousands of domains" with "millions of requests per day." This incident reinforces prior concerns about Perplexity's data acquisition methods, including previous instances of bypassing paywalls and ignoring robots.txt files. For Cloudflare, this event is strategically significant; by delisting Perplexity as a verified bot and rolling out new blocking tools, NET positions itself as a critical partner for publishers seeking to protect their intellectual property from unauthorized AI training, aligning with its CEO's public stance and potentially strengthening its service offerings.

Key Decisions for Investors

• Investors should view this development as a positive catalyst for Cloudflare (NET), as it reinforces the company's value proposition in cybersecurity and content protection, and they should monitor the adoption of its new AI-blocking features as a potential growth driver.
• The allegations against Perplexity serve as a proxy for a systemic risk across the AI industry; therefore, it is critical to scrutinize the data acquisition ethics and legal exposure of any investment in AI model developers, as regulatory and litigative headwinds are likely to intensify.
• For those invested in digital media and content-owning companies, this event underscores the urgent need to assess their intellectual property protection strategies against unauthorized AI scraping and to explore potential revenue streams from data licensing agreements.