ChatGPT's Fetch Bot May Ignore Your Robots.txt — And OpenAI Says That's By Design

New data shows ChatGPT's page-fetching bot reached disallowed pages on more sites than any other AI crawler tracked in 2026 — and OpenAI's own documentation explains why your block may not hold.

If you've carefully configured your robots.txt file to keep AI crawlers off your site, you may want to check your server logs. TollBit's latest State of the Bots report for the first half of 2026 reveals a growing gap between what website operators ask for and what AI-driven bots actually do — and the explanation comes straight from OpenAI itself.

---

What the Data Actually Shows

TollBit's report focuses on bot behavior across European and North American websites and the findings are striking. Among all AI page-fetching agents tracked, ChatGPT-User — the bot ChatGPT deploys when a user requests a page — was disallowed by more sites than any other agent of its kind. Yet it also reached disallowed pages on more sites than any other tracked bot.

In European markets specifically, approximately 15% of identified AI page-fetchers accessed URLs that sites had explicitly marked as off-limits. That figure doesn't sound alarming on its own until you look at which agents are responsible. ChatGPT-User, Bytespider, and Youbot each accessed disallowed pages on nearly half of the European sites that had explicitly listed them in robots.txt. Among this group, ChatGPT-User reached the highest number of sites.

The disallow rates themselves tell their own story:

• Around 26% of North American websites have blocked ChatGPT-User
• Only 13% of European sites have done the same for Perplexity-User
• Newer agents like Claude-User sit at just 9% disallow rates in Europe
• Most of the newest AI agents have single-digit block rates across the continent

ChatGPT-User remains a clear outlier in terms of both how often it's blocked and how often it shows up anyway. For website operators already navigating the broader challenges of website security and unauthorised access, this pattern adds a complicated layer — one where the intruder may be acting with its developer's implicit blessing.

Why the Numbers Matter Beyond the Headline

The gap between disallow rates and actual compliance isn't just a technical anomaly — it signals a fundamental tension between the established web standards that site operators rely on and the evolving interpretations that AI companies are applying to those standards. When nearly half of sites explicitly blocking a bot still see it arrive, the robots.txt file begins to function less like a rule and more like a suggestion.

---

Why OpenAI Says Robots.txt May Not Apply

The core of this issue lies in how OpenAI has documented ChatGPT-User's behavior. According to OpenAI's official crawler documentation, the bot visits a page when a ChatGPT user asks a question that requires fetching that specific URL. Because the action is initiated by a human user rather than an autonomous crawl, OpenAI's position is that robots.txt rules may not apply in those cases.

Perplexity takes a similar stance and states that Perplexity-User generally ignores the robots.txt file for the same user-initiated reason. Anthropic draws a sharper line and states that all three of its bots respect the file — a position Search Engine Journal reported on in February of this year.

TollBit's methodology doesn't split hairs on intent. The report treats any bot request to a disallowed URL as a bypass regardless of what the operator claims motivated it. That approach puts the compliance question in plain numerical terms rather than philosophical ones.

The User-Initiated Loophole Explained

This distinction matters more than it might first appear. The argument OpenAI and Perplexity are making rests on a meaningful technical and legal difference: a crawler autonomously indexing your site is doing something categorically different, they contend, from a bot retrieving a page because a human explicitly asked for it. Whether that distinction holds up under broader legal or regulatory scrutiny is an open question — particularly as data governance and compliance requirements in Europe continue to tighten around automated data collection.

A disallow entry for ChatGPT-User is effectively a request — one that OpenAI's own documentation acknowledges may carry a built-in exception. As the report makes clear, the file only shows what you asked for while your server logs show what actually arrived.

---

The Visibility Trade-Off Website Operators May Be Missing

There's an additional layer to this situation that website operators should understand before reaching for a blanket block. OpenAI uses a separate bot called OAI-SearchBot to determine whether a site appears in ChatGPT's search results. ChatGPT-User is the fetching agent — the one that retrieves content when a user specifically requests it.

Sites that block both agents in an attempt to shut out all AI traffic have effectively eliminated their visibility in ChatGPT search while retaining a fetching restriction that OpenAI's documentation already qualifies with a carve-out. In practical terms, that's a trade where the certain loss is visibility and the uncertain gain is fetch control.

This situation asks site owners to weigh discoverability against control — without a clean answer available yet. It's a tension that mirrors broader risks and strategic challenges that businesses face when integrating AI into their operations — where the benefits and the risks frequently arrive together, and opting out carries its own costs.

Network-Level Enforcement Is Becoming the More Decisive Battleground

Looking ahead, the network layer may prove more decisive than the file layer. Cloudflare announced updates to its crawler management controls that shift enforcement away from the robots.txt file and place it at the network level. For bots Cloudflare recognises, compliance will no longer depend on whether the crawler chooses to honour the file.

Beginning September 15, new domains added to Cloudflare will have Training and Agent crawlers blocked by default on pages carrying ads, while Search crawlers remain permitted.

What This Means for Your Strategy

Whether the user-initiated loophole survives broader scrutiny remains an open question. The argument rests on the distinction between a crawler autonomously taking a page and a user asking for one — but as all major AI assistants now fetch pages in real time, that line is becoming harder to draw.

In the meantime, the most practical steps available to site operators are concrete ones:

• Audit your server logs and CDN records rather than relying solely on robots.txt to understand which bots are actually reaching your site and which pages they're accessing.
• Review your bot-blocking strategy with the OAI-SearchBot and ChatGPT-User distinction in mind — blocking both may cost you ChatGPT search visibility without guaranteeing fetch control.
• Consider network-level enforcement tools like Cloudflare's updated crawler controls if you want compliance that doesn't depend on whether an AI agent chooses to honour your instructions.