robots.txt tester: see whether AI crawlers are actually blocked

Enter a domain and get a per-crawler verdict for GPTBot, ClaudeBot, Google-Extended and the rest, read straight out of that site's robots.txt.

Writing a block into robots.txt is easy. Knowing whether it does anything is not, because overlapping rules resolve in ways the file does not show you. This page fetches the file, applies RFC 9309 exactly, and names the line that decided each verdict. No login, nothing to install.

Enter the address only. Any path is dropped; robots.txt and llms.txt are read from the site root.

To test one article, paste its path. For example /posts/2026/ai

Enter a domain and press Check.

What this page decides

The verdicts describe what robots.txt says. They do not prove what a crawler does. robots.txt is a request, not a wall, and a crawler that ignores it has to be stopped at your server or CDN instead.

Why a block you wrote may not block

When rules overlap, the longest one wins. Disallow: / with Allow: /blog/ below it leaves the blog open. User-agent names ignore case, but paths do not. And a single byte-order mark at the top of the file can hide the first User-agent line entirely.

Sources

How to block AI crawlers with robots.txt

Made alongside

Other services by the same maker.