Block GPTBot and other AI crawlers: robots.txt generator
To block GPTBot, OpenAI's training crawler, add two lines to robots.txt: "User-agent: GPTBot" and, on the next line, "Disallow: /". This generator writes that rule and the matching rules for the other AI crawlers. Choose allow, block or leave out for each one and copy the result. Each crawler is listed with its operator, its purpose (training, search index or user-triggered fetch) and a link to the operator's documentation. robots.txt is advisory: it asks a crawler to stay out and does not enforce it. Nothing is sent anywhere.
Working with gigabytes, whole folders or a live system? The free outrich command line tool does this on your own machine at any size.
How to use it
- 01Start from a preset, such as blocking training crawlers while allowing AI search, or set each crawler yourself.
- 02Read the purpose column before you block. Blocking a search crawler can remove your pages from that assistant's answers and citations.
- 03Copy the block and paste it into the robots.txt file at the root of your site, alongside the rules you already have.
- 04Paste the finished file into the AI crawler robots.txt checker to confirm it does what you meant.
What it does not do
- robots.txt is a request, not a lock. RFC 9309 states that its rules are not a form of access authorization. Crawlers that ignore the standard need a firewall or bot management rule instead.
- Some user-triggered fetchers are documented by their operators as not always following robots.txt. The description beside each crawler says so where the operator does.
- A group that names a crawler replaces your general rules for that crawler. If your * group disallows /admin/, a crawler you explicitly allow here no longer inherits that rule. Add the same Disallow lines under it if you need them.
- The rules apply site-wide. For per-folder rules, edit the Allow and Disallow paths by hand.
- The list covers crawlers whose operators publish a robots.txt token. It was checked on the date shown in each documentation link and operators add or rename tokens over time.
Questions
About the ai crawler robots.txt generator.
How do I block GPTBot in robots.txt?
Which AI crawlers collect training data, and which are search or user-triggered?
Does this tool fetch my site or send anything anywhere?
Should I block AI crawlers?
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
Does blocking AI crawlers get me paid for my content?
Does blocking remove my content from models that already trained on it?
Is it really free?
Related tools · all 29
AI crawler robots.txt checker
Paste a robots.txt and see which AI crawlers it allows, blocks or never mentions.
Personal data scrubber
Strip names, emails, phones, cards, keys and addresses from text and files.
Dataset profiler
Rows, columns, tokens, date span, duplicates and personal-data columns in one report.
Column hasher and remover
Drop sensitive columns or replace them with salted hashes that still join.
Cleaned it up? Sell it to Outrich.
Outrich Data buys company data directly. Check in five minutes whether yours qualifies, with no call.