Blocking AI crawlers with robots.txt, and why it so often fails

1. What robots.txt is

robots.txt is a plain text file at the root of a site. It records which paths a crawler may fetch, and a crawler reads it before it reads anything else on the site.

It began as a mailing list convention in 1994 and became an IETF standard, RFC 9309, in 2022. That document is the rule set the large crawlers say they follow, and every verdict on this page comes from it.

One thing about it matters more than the syntax. robots.txt is a request, not a wall. Crawlers that say they honour it do; the rest simply take what they want, and stopping those means blocking at your server or CDN.

2. The basic rules: groups and longest match

The file is read in groups. One or more User-agent lines followed by Allow and Disallow lines form a group. A User-agent line appearing after a rule line starts a new group.

A crawler reads exactly one group: the one naming it. Failing that, it reads the User-agent: * group, and failing that it concludes the file has no rules for it. Name comparison ignores case (RFC 9309, section 2.2.1).

Within a group, when several rules match a path, the longest one wins. The specification calls this the match with the most octets. If an Allow and a Disallow tie on length, Allow wins (section 2.2.2).

Paths accept two wildcards. An asterisk stands for any run of characters, and a dollar sign anchors the end of the path. With neither, matching is a prefix comparison, which is why Disallow: /admin also covers /administrator (section 2.2.3).

3. The rules inside this page

Enter a domain and the server fetches two files from it: robots.txt and llms.txt. Nothing else is opened. Any path you type in the address is dropped, because robots.txt only ever lives at the root.

Neither file is stored. Each is passed back once in the response, and the verdicts are computed in your browser, so changing the test path does not call the server again.

Four guards apply. Internal addresses and localhost are refused, the fetch is cut off after 8 seconds, at most 500 KiB is read, and at most five redirects are followed, each one re-checked so a redirect cannot lead inward.

A crawler appears in the table only if its name is confirmed in its operator's own documentation. Names we could not confirm are left out, and each row carries a source link and the date we checked it.

Name matching is exact. A Googlebot group is not applied to Googlebot-Image automatically, because that kind of inheritance is defined by the operator rather than by the specification. Where an operator documents it, the basis column says so.

What this page decides is what robots.txt says. Whether a given crawler honoured it is not something we can observe, so we do not claim it.

4. Where the blocking lines go

The lines themselves are short. Put the crawler name on a User-agent line and Disallow: / underneath, and that crawler is told to take nothing. Give each crawler its own group, or stack several User-agent lines above one set of rules.

WordPress serves a virtual robots.txt even when no file exists. Upload a real robots.txt to the web root over FTP, or edit it through your SEO plugin's robots.txt screen; a real file takes precedence.

On hosted blogging platforms the file is often owned by the service. If your settings offer no robots.txt entry, that is the case, and this page is then useful for seeing what the service currently blocks on your behalf.

On a site you build yourself, drop robots.txt into the public root. Next.js generates it from a robots file in the app directory; Hugo and Jekyll take it from the static or root folder.

After you add the lines, open the domain followed by /robots.txt in a browser and confirm the lines are actually served. Caches and plugins swallow edits more often than you would expect. Then come back and run the check.

5. How llms.txt differs

llms.txt is a markdown file at the site root. It lists the shape of a site and links to its key documents, written for a language model to read rather than a person.

Its job is the opposite of robots.txt. One says what not to take, the other says what to look at, so neither substitutes for the other.

Being honest about its status: it is not a standard. It is a format one person proposed that spread, and we have not confirmed a public statement from any major AI company that it reads the file. So this page shows whether it exists and what it contains, and does not grade it.

6. Where people get stuck

If you wrote a block and this page still says allowed, there is usually a longer Allow below it. Disallow: / together with Allow: /blog/ leaves the blog open, because the longest rule wins.

Case in the user-agent name does not matter: gptbot matches GPTBot. Case in the path does: Disallow: /Admin leaves /admin open.

An empty value after Disallow blocks nothing. Plenty of files carry that line where a real block was intended. Blocking everything takes a single slash as the value.

A byte-order mark written by an editor hides the first User-agent line. This page tells you when it finds one. Save as UTF-8 without BOM.

A robots.txt that 404s counts as no restrictions at all, so deleting the file opens the site rather than closing it. A 5xx counts as a full block.

Finally, a scraper that ignores robots.txt is not stopped by any of this. That one is a server or CDN job, and this page does not judge it.