How we measured AI-crawler blocking

What was measured

One GET /robots.txt per domain across the top 25,000 domains of the Tranco 38LNL list, on 26 August 2026. For each of 18 crawler tokens we record whether the effective rules permit fetching the site root /, and whether the file names that token in a User-agent line.

Nothing else is fetched. No pages are crawled, no rendering is performed, no login, paywall or challenge is bypassed. The scanner identifies itself as DriftwitnessResearch/1.0 with a link to a page explaining what it is.

Sample

This is a census of the head of the list, not a random sample: ranks 1 to 25,000, every one attempted, none excluded by judgment. That means results describe the most-visited part of the web and generalise to nothing wider. A site at rank 400,000 is not represented and no claim here covers it.

Verdicts, and the words we use for them

Denominators

Of 25,000 domains attempted: 17,945 responded, of which 13,035 served a parseable robots.txt, 3,772 returned 4xx (which RFC 9309 treats as no restrictions) and 1,138 served HTML where robots.txt should be. 6,875 were unreachable.

Every headline percentage is over the 17,945 responding domains, not over the 13,035 with a robots.txt and not over the 25,000 attempted. A domain that responded without restrictions is a real observation of "does not block", so it belongs in the denominator. Unreachable domains are not an observation of anything and are excluded. If you would have chosen differently, the dataset carries the raw counts.

Parser

Rules are evaluated with Protego, an RFC 9309 implementation: case-insensitive exact product-token matching, wildcard * and end-anchor $ in path patterns, longest-match wins, Allow wins ties, and groups sharing a user-agent combined.

Token matching is exact, not prefix. Applebot and Applebot-Extended are different tokens governing different things — search versus AI training — and letting one stand in for the other would misreport a site's policy. A token with no group of its own falls through to *, which is what a well-behaved crawler does.

Correction log

27 August 2026 — parser replaced, all figures recomputed

The original scan used our own robots.txt matcher. It did not implement the * and $ path wildcards that RFC 9309 gives meaning to, and it treated a rule as governing the site root only when the path was literally /. Two consequences:

• Disallow: * was not read as blocking the root. Undercount.
• Allow: /$ was ignored, so a site saying "block everything except the home page" was counted as blocking the crawler outright. Overcount, and the larger effect. Domains in our own sample doing exactly this include chatgpt.com, vimeo.com and soundcloud.com.

Every verdict was recomputed from the archived robots.txt corpus using Protego. No re-crawling, no change to the sample, the denominators or any definition — only the parser. 763 of 13,035 domains changed on at least one crawler.

Bingbot is the only figure that moved by more than a rounding step. The training-versus-search carve-out went from 1,669 domains to 1,659, and its reverse from 11 to 15. The finding did not change: training crawlers are still blocked around five times as often as AI-search crawlers, and 93.8% of the sites blocking GPTBot by name have still never written a rule for OpenAI's search bot. The published dataset has been regenerated.

The uncomfortable part is that our own pre-registered protocol had already required an RFC 9309 parser, for exactly this reason, before the scan ran. The requirement was written down and the code did not meet it, and nothing checked. The scanner now uses Protego so the rescan cannot repeat it.

Limitations

1. robots.txt is a request, not enforcement. A disallow proves declared policy, not that a crawler is technically prevented. Everything here measures declared restrictions.
2. Root-path verdicts. A site disallowing only /articles/ for a bot is not counted as blocking. Deliberately conservative, and it undercounts partial restrictions.
3. One host variant. https://apex, falling back to http://apex. No www fallback, so some of the 27.5% unreachable figure is likely apexes that do not serve while www does.
4. One attempt. 6s connect, 10s read, no retry beyond the scheme fallback. Transient failures land in "unreachable".
5. Point in time. One day, one vantage point, one datacenter IP. Directives change often.
6. Tranco is popularity-biased. It measures the popular web, not the web.
7. Token classification is a judgment. Sorting 18 tokens into training, AI-search and user-triggered rests on vendor documentation of uneven quality. Bytespider's classification is the weakest of them.
8. 18 tokens is not every crawler. A token's absence from our list says nothing about a site's posture toward it.

Items 3 and 4 are known gaps against our own pre-registered protocol, which requires both a www fallback and retries. They are listed here rather than quietly fixed because the published numbers were produced without them.

The rescan

A rescan of the identical domain set under the identical protocol is what turns a snapshot into a measurement of change, and change is the thing worth knowing. Its protocol is pre-registered before the data exists, and it carries a binding rule: if the rescan does not land in its window, the next drop is titled a baseline prevalence study and may make no change-over-time claim at all. Not waivable by editorial judgment.

Data

The full per-crawler dataset is published under CC BY 4.0: ai-crawler-blocking-2026-08.json. Use it for anything, including disagreeing with us.

Found a problem with any of this? [email protected]. We amend in place and log every change here, dated.