AI crawler robots.txt checker
Paste the contents of a robots.txt file and enter a page path. For each AI crawler, the checker shows whether that page is allowed, blocked or unspecified, and points to the exact rule and line that decided it. It applies the standard matching rules: the crawler's own group first, then the * group, with the longest matching rule winning.
Working with gigabytes, whole folders or a live system? The free outrich command line tool does this on your own machine at any size.
How to use it
- 01Open yoursite.com/robots.txt in a browser tab, copy everything and paste it in.
- 02Enter the page to test: a path such as /blog/ or a full URL. The default is the home page.
- 03Read the result for each crawler and the rule that decided it. Unspecified means no group applies, so crawling is allowed by default.
- 04To change an answer, build the rules in the AI crawler robots.txt generator and test the edited file here.
What it does not do
- It checks what the file says, not what a crawler does. A crawler that ignores robots.txt is not stopped by it.
- It does not fetch your site, so it cannot see a robots.txt that differs by subdomain, a file that returns an error, or blocking done by a firewall or CDN.
- User-agent names are matched exactly, ignoring letter case, against each crawler's documented token. Paths are compared as typed, without percent-encoding normalization.
- Meta robots tags and X-Robots-Tag headers are separate controls and are not checked here.
Questions
About the ai crawler robots.txt checker.
Does this tool fetch my site or send anything anywhere?
What does unspecified mean?
Why is a crawler allowed when my * group blocks the page?
Which rule wins when Allow and Disallow both match?
Does blocking AI crawlers get me paid for my content?
Is it really free?
Related tools · all 29
AI crawler robots.txt generator
Choose which AI crawlers to allow or block and copy the robots.txt rules.
Personal data scrubber
Strip names, emails, phones, cards, keys and addresses from text and files.
Dataset profiler
Rows, columns, tokens, date span, duplicates and personal-data columns in one report.
Column hasher and remover
Drop sensitive columns or replace them with salted hashes that still join.
Cleaned it up? Sell it to Outrich.
Outrich Data buys company data directly. Check in five minutes whether yours qualifies, with no call.