Skip to content
Outrich Data

Block GPTBot and other AI crawlers: robots.txt generator

To block GPTBot, OpenAI's training crawler, add two lines to robots.txt: "User-agent: GPTBot" and, on the next line, "Disallow: /". This generator writes that rule and the matching rules for the other AI crawlers. Choose allow, block or leave out for each one and copy the result. Each crawler is listed with its operator, its purpose (training, search index or user-triggered fetch) and a link to the operator's documentation. robots.txt is advisory: it asks a crawler to stay out and does not enforce it. Nothing is sent anywhere.

Working with gigabytes, whole folders or a live system? The free outrich command line tool does this on your own machine at any size.

How to use it

  1. 01Start from a preset, such as blocking training crawlers while allowing AI search, or set each crawler yourself.
  2. 02Read the purpose column before you block. Blocking a search crawler can remove your pages from that assistant's answers and citations.
  3. 03Copy the block and paste it into the robots.txt file at the root of your site, alongside the rules you already have.
  4. 04Paste the finished file into the AI crawler robots.txt checker to confirm it does what you meant.

What it does not do

  • robots.txt is a request, not a lock. RFC 9309 states that its rules are not a form of access authorization. Crawlers that ignore the standard need a firewall or bot management rule instead.
  • Some user-triggered fetchers are documented by their operators as not always following robots.txt. The description beside each crawler says so where the operator does.
  • A group that names a crawler replaces your general rules for that crawler. If your * group disallows /admin/, a crawler you explicitly allow here no longer inherits that rule. Add the same Disallow lines under it if you need them.
  • The rules apply site-wide. For per-folder rules, edit the Allow and Disallow paths by hand.
  • The list covers crawlers whose operators publish a robots.txt token. It was checked on the date shown in each documentation link and operators add or rename tokens over time.

Questions

About the ai crawler robots.txt generator.

How do I block GPTBot in robots.txt?
Add a group of two lines to the robots.txt file at the root of your site. Line one: User-agent: GPTBot. Line two: Disallow: /. OpenAI's crawler documentation, read 2026-11-17, says that disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models. The rule covers GPTBot only. OAI-SearchBot and ChatGPT-User have their own tokens, and OpenAI says robots.txt rules may not apply to ChatGPT-User because a user starts the request.
Which AI crawlers collect training data, and which are search or user-triggered?
Training crawlers: GPTBot, ClaudeBot, Applebot-Extended, MistralAI-Training, AI2Bot. Search crawlers: OAI-SearchBot, Claude-SearchBot, Google-CloudVertexBot, PerplexityBot, meta-webindexer, Amzn-SearchBot, MistralAI-Index, DuckAssistBot, Diffbot, PetalBot. User-triggered fetchers: ChatGPT-User, OAI-AdsBot, Claude-User, Google-GeminiNotebook, Google-Agent, Perplexity-User, meta-externalfetcher, Amzn-User, MistralAI-User, Diffbot-User. Mixed use or control tokens: Google-Extended, meta-externalagent, Amazonbot, CCBot. Each is classified from its operator's own documentation, checked 2026-11-17. The table in the tool above shows what each operator says. Blocking a training crawler does not block the same operator's search or user-triggered agent.
Does this tool fetch my site or send anything anywhere?
No. It runs entirely in your browser and makes no network requests. The checker reads only the text you paste, and the generator builds its output on your device. You can confirm it by turning off your internet connection after the page loads: the tool keeps working.
Should I block AI crawlers?
It depends on what you want from them. Blocking training crawlers keeps new pages out of future training sets where the operator honors robots.txt. Blocking search and user-fetch crawlers can also remove you from AI answers that would have linked to you. Many sites block training and allow search, which is the default preset here.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
They are three separate OpenAI agents with separate robots.txt tokens, so each can be allowed or blocked on its own. The table shows what OpenAI's documentation says each one is for. Anthropic and Perplexity split their crawlers in a similar way.
Does blocking AI crawlers get me paid for my content?
No. A robots.txt rule only asks a crawler not to fetch pages. It creates no license and no payment. Licensing is a separate step: a signed agreement that grants defined rights to a defined dataset. For companies, the data AI labs pay for is mostly private work records such as tickets, calls and documents, not public web pages. Outrich Data buys that kind of data directly: see selling company data.
Does blocking remove my content from models that already trained on it?
No. A robots.txt rule affects future fetches only. It does not delete copies already collected or change models already trained.
Is it really free?
Yes. There is no account, no limit on the number of uses and nothing to install.

Cleaned it up? Sell it to Outrich.

Outrich Data buys company data directly. Check in five minutes whether yours qualifies, with no call.

esc