Skip to content
Outrich Data

AI crawler robots.txt checker

Paste the contents of a robots.txt file and enter a page path. For each AI crawler, the checker shows whether that page is allowed, blocked or unspecified, and points to the exact rule and line that decided it. It applies the standard matching rules: the crawler's own group first, then the * group, with the longest matching rule winning.

Working with gigabytes, whole folders or a live system? The free outrich command line tool does this on your own machine at any size.

How to use it

  1. 01Open yoursite.com/robots.txt in a browser tab, copy everything and paste it in.
  2. 02Enter the page to test: a path such as /blog/ or a full URL. The default is the home page.
  3. 03Read the result for each crawler and the rule that decided it. Unspecified means no group applies, so crawling is allowed by default.
  4. 04To change an answer, build the rules in the AI crawler robots.txt generator and test the edited file here.

What it does not do

  • It checks what the file says, not what a crawler does. A crawler that ignores robots.txt is not stopped by it.
  • It does not fetch your site, so it cannot see a robots.txt that differs by subdomain, a file that returns an error, or blocking done by a firewall or CDN.
  • User-agent names are matched exactly, ignoring letter case, against each crawler's documented token. Paths are compared as typed, without percent-encoding normalization.
  • Meta robots tags and X-Robots-Tag headers are separate controls and are not checked here.

Questions

About the ai crawler robots.txt checker.

Does this tool fetch my site or send anything anywhere?
No. It runs entirely in your browser and makes no network requests. The checker reads only the text you paste, and the generator builds its output on your device. You can confirm it by turning off your internet connection after the page loads: the tool keeps working.
What does unspecified mean?
No group in the file names that crawler and there is no * group, so no rules apply to it. Under RFC 9309 that means the crawler may fetch the page. If you want a different outcome, add a group for it.
Why is a crawler allowed when my * group blocks the page?
Because the file also has a group that names the crawler. A crawler uses the group that names it and ignores the * group entirely. The two are never combined, so rules you wrote under * do not carry over.
Which rule wins when Allow and Disallow both match?
The longest matching path wins. If an Allow and a Disallow rule are the same length, Allow wins. That is the behavior RFC 9309 describes and Google documents.
Does blocking AI crawlers get me paid for my content?
No. A robots.txt rule only asks a crawler not to fetch pages. It creates no license and no payment. Licensing is a separate step: a signed agreement that grants defined rights to a defined dataset. For companies, the data AI labs pay for is mostly private work records such as tickets, calls and documents, not public web pages. Outrich Data buys that kind of data directly: see selling company data.
Is it really free?
Yes. There is no account, no limit on the number of uses and nothing to install.

Cleaned it up? Sell it to Outrich.

Outrich Data buys company data directly. Check in five minutes whether yours qualifies, with no call.

esc