Perplexity accused of scraping websites that explicitly blocked AI scraping

Cloudflare has accused AI startup Perplexity of deliberately circumventing website scraping blocks, including Robots.txt and user agent identification, to acquire data from sites that explicitly denied access. Cloudflare's research, based on observations across tens of thousands of domains and millions of daily requests, indicates Perplexity obscured its identity to bypass these controls. Perplexity denies the allegations, calling them a "sales pitch" and disputing the bot's ownership. This highlights the escalating tension between AI companies' vast data requirements and content owners' efforts to protect their intellectual property, with Cloudflare actively developing tools to manage such AI scraping.

Analysis

Cloudflare (NET) has publicly accused AI startup Perplexity of engaging in systematic and deceptive web scraping, creating a significant reputational and operational challenge for the AI firm. According to Cloudflare's research, Perplexity actively circumvents explicit publisher prohibitions, such as Robots.txt files, by masking its identity through generic browser user-agents and altered Autonomous System Networks (ASNs). The scale of this alleged activity, observed across tens of thousands of domains with millions of daily requests, underscores the aggressive data acquisition tactics employed by some AI developers. While Perplexity has dismissed the claims as a "sales pitch" and denied ownership of the bots, Cloudflare's detailed accusations and subsequent de-listing of Perplexity's bots from its verified list lend credibility to the findings. This conflict is strategically advantageous for Cloudflare, as it directly validates the market need for its recently launched anti-scraping tools and publisher monetization marketplace. The incident highlights the escalating tension between the data ingestion requirements of AI models and the intellectual property rights of content creators, signaling a critical flashpoint for potential future regulation and litigation within the tech sector.

Key Decisions for Investors

• For investors in Cloudflare (NET), this incident should be viewed as a positive catalyst that reinforces the company's strategic positioning and the value proposition of its new AI-focused security and monetization products.
• Investors with exposure to the AI sector, particularly startups reliant on web-scraped data, must increase scrutiny of data acquisition ethics and practices, as this dispute signals rising legal, reputational, and operational risks.
• The conflict between Perplexity and Cloudflare indicates a growing market for data governance and access control solutions, presenting a potential investment opportunity in companies that build the infrastructure to manage or monetize AI's access to proprietary content.