Cloudflare Blocks Perplexity AI Over Deceptive Crawling Practices

Cloudflare has removed Perplexity from its Verified Bots program and implemented blocking measures after discovering the AI company was using deceptive tactics to crawl websites while bypassing established protocols and restrictions.

The decision comes amid growing tensions between AI companies and web infrastructure providers over proper data collection practices. This move by Cloudflare, a major internet security company, signals increasing scrutiny of AI firms' web crawling behaviors.

Investigation Reveals Stealth Tactics

Cloudflare's investigation uncovered multiple concerning practices by Perplexity, including:

• Rotating IP addresses and changing Autonomous System Numbers (ASNs) to evade blocks
• Spoofing user agents to appear as regular Chrome browsers rather than bots
• Ignoring robots.txt protocols that specify crawling permissions
• Using undeclared IP addresses outside their official crawler network

"The Internet as we have known it for the past three decades is rapidly changing, but one thing remains constant: it is built on trust," Cloudflare stated in their announcement. The company emphasized that crawlers must be transparent and follow website directives.

Competing Claims Over AI Assistants

Perplexity has challenged Cloudflare's characterization of their activities, arguing that their AI assistants are legitimate tools serving user requests, not malicious bots. In their rebuttal, Perplexity stated: "When companies like Cloudflare mischaracterize user-driven AI assistants as malicious bots, they're arguing that any automated tool serving users should be suspect."

Impact on Web Infrastructure

The dispute highlights growing conflicts between AI companies' data needs and website owners' rights to control access to their content. Website administrators using Cloudflare services may need to review their settings if they want to allow Perplexity's crawlers.

Additional Security Considerations

Website owners should implement robust security measures to protect their content while maintaining accessibility. This includes:

• Regular monitoring of crawler activity
• Implementation of rate limiting
• Maintenance of updated robots.txt files
• Documentation of authorized crawler access

The situation continues to evolve as both parties maintain their positions, reflecting broader industry tensions over AI data collection practices and website autonomy.