The web blocked AI training. It never decided about AI search.
We fetched robots.txt from the 25,000 most-visited domains on the web and read the rules for eighteen crawlers. Sites block GPTBot, which trains models, about 5.2 times as often as they block OpenAI's search crawler, which sends readers. Look closer and most of that gap is not a choice anyone made.
18.8% of responding domains now name at least one AI crawler in robots.txt. 11.5% explicitly block GPTBot, against 2.2% for OpenAI's search crawler and 0.1% for Googlebot.
1,669 domains block GPTBot but allow OAI-SearchBot. Only 11 do the reverse. Yet 93.8% of the sites blocking GPTBot by name have never written a rule mentioning OAI-SearchBot at all.
Share of the 17,945 responding domains whose robots.txt names a crawler and blocks it from the site root. Domains with no robots.txt count as not blocking, which is what the standard says they are.
What we measured
On August 26, 2026 we requested /robots.txt once from each of the top
25,000 domains on the Tranco 38LNL list,[1]
then evaluated the rules the way a crawler is supposed to: most specific user-agent
group wins, longest matching path wins, and a site with no robots.txt is
unrestricted.[2] 17,945 domains answered.
13,035 of those returned a robots.txt we could parse; the rest either had none
or served a web page in its place.
For each domain we recorded, per crawler, two different things: whether the rules leave the site root reachable, and whether the site names that crawler anywhere. The second question turns out to matter more than the first.
Training crawlers get blocked. Search crawlers mostly do not.
The eighteen crawlers sort into four tiers, and the gaps between them are large. Classic search sits at the bottom: 0.1% of responding domains name Googlebot and shut it out of the site, 0.5% do it to Bingbot. Add the sites that block every crawler at once with a blanket wildcard and Googlebot's figure rises to 2.2%, which is a different thing: those sites are not thinking about Google, they are closed to everyone. Model-training crawlers sit at the top: 11.6% block Common Crawl's CCBot, 11.5% block GPTBot, 11.1% block ByteDance's Bytespider, 10.4% block ClaudeBot.
In between are the crawlers that fetch pages so an AI product can cite them. OAI-SearchBot is blocked by 2.2%, Claude-SearchBot by 1.8%, PerplexityBot by 4.0%. Same companies as the training crawlers, a fraction of the blocking.
Read as policy, that is coherent. Do not train a model on my work; do send me readers. 12.1% of responding domains hold exactly that position against OpenAI: Googlebot welcome, GPTBot refused.
Domains blocking GPTBot while allowing OpenAI's search crawler, against those doing the opposite. The web is close to unanimous about which of the two it minds.
Most of that gap was never decided
Here is where the tidy story falls apart. If those 1,669 sites had really weighed training against search, you would expect their robots.txt to mention the search crawler, even if only to permit it. Most of them never mention it.
Of the 1,655 domains that block GPTBot by name and leave OAI-SearchBot reachable, only 103 name OAI-SearchBot anywhere in the file. The other 1552, or 93.8%, have written no rule about it in any form.
OpenAI documents the two crawlers as separate tools for separate purposes, and says so in the plainest possible terms:
"Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot."
OpenAI, Overview of OpenAI crawlers[3]
That is a description of a choice. For 1552 sites in this sample it is instead a description of an accident. GPTBot has been documented since 2023; OAI-SearchBot arrived later, and a robots.txt written when there was only one bot to think about still says exactly what it said then. The permission is not a decision. It is a file nobody has opened since.
The pattern repeats with Anthropic's crawlers, which are documented the same way:[4] of the 1,538 domains that block ClaudeBot by name while leaving Claude-SearchBot reachable, only 64 have written a Claude-SearchBot rule of any kind.
The sites that did make the call are visible in the data because they wrote it down. Yahoo, Forbes, eBay, TED, Snapchat and Tripadvisor all block GPTBot at the root while carrying explicit OAI-SearchBot rules that leave the root open. Amazon, The New York Times, the BBC and CNN went the other way and blocked both crawlers by name. Ten companies, two opposite conclusions, one thing in common: somebody sat down and decided.
Naming Googlebot means tuning. Naming GPTBot means banning.
Sites mention Googlebot 1,048 times in our sample, and block it at the root in 2.5% of those cases. The rest are path rules: keep out of the cart, out of search results, out of the admin area. Naming Googlebot is how a site does housekeeping.
GPTBot is named 2,727 times, and 75.7% of those mentions block it from the whole site. ClaudeBot: 77.4%. Nobody is fine-tuning which sections a training crawler may read. The rule is a door, and it is either open or shut.
Blocking barely varies by rank
The intuition is that big publishers block AI and everyone else has not thought about it. The training numbers do not support that. GPTBot blocking is close to flat from the top of the list down to rank 25,000.
| Rank band | Domains | Names an AI bot | Blocks GPTBot | Blocks ClaudeBot | Blocks PerplexityBot | Blocks OAI-SearchBot |
|---|---|---|---|---|---|---|
| 1–1,000 | 669 | 24.7% | 12.3% | 12.1% | 9.6% | 4.8% |
| 1,001–5,000 | 2,801 | 20.2% | 11.9% | 10.6% | 6.0% | 3.0% |
| 5,001–10,000 | 3,746 | 19.3% | 11.9% | 10.9% | 4.9% | 3.0% |
| 10,001–25,000 | 10,729 | 17.9% | 11.2% | 10.1% | 2.9% | 1.6% |
Where rank does show up is in the newer decisions. PerplexityBot blocking runs 9.6% in the top thousand and 2.9% in the bottom band of our sample. The largest sites are not blocking more in general. They are the ones keeping up.
It is mostly all or nothing
3,125 domains block at least one of the nine training crawlers we tested. Among them, 59.9% block eight or nine of the nine, and 23.5% block every single one. That looks less like a curated list of acceptable AI companies than a blocklist copied from somewhere, which would also explain the misses: they are the crawlers that were announced after the copy.
Three things worth checking on your own site
- Whether your AI rules still say what you meant. If your robots.txt blocks GPTBot and says nothing about OAI-SearchBot, you are currently allowing OpenAI's search crawler. That may be what you want. It should be on purpose.
- Whether a wildcard is doing more than you think.
476 of the domains blocking training crawlers do it only through a
blanket
User-agent: *rule that also catches everything else. - Whether anyone would notice a change. robots.txt is one line in one file, edited by whoever last touched the deploy, and it silently governs who may read your entire site.
Questions people actually ask
What percentage of websites block AI crawlers?
In this scan, 11.5% of responding domains explicitly blocked GPTBot and 11.6% blocked CCBot, while 18.8% named at least one AI crawler in some form. AI-search crawlers were blocked much less: 2.2% for OAI-SearchBot, 4.0% for PerplexityBot.
Does blocking GPTBot stop ChatGPT from citing my site?
Not by itself. OpenAI documents GPTBot and OAI-SearchBot as separate crawlers with separate purposes,[3] so a rule naming only GPTBot leaves the search crawler unaffected. That is the exact gap 1552 sites in this sample are sitting in, most likely without knowing it.
Do bigger sites block AI crawlers more often?
For training crawlers, hardly at all: the rate is close to flat across every rank band we measured. For AI-search crawlers, yes, and the likeliest reason is that larger sites revise robots.txt more often rather than that they hold different opinions.
Methodology
Sample. The top 25,000 domains of the Tranco list 38LNL (generated August 24, 2026), a research ranking that combines several traffic sources and is designed to resist manipulation.[1]
Collection. One HTTPS request to /robots.txt per
domain on August 26, 2026, falling back to HTTP once if HTTPS failed, following
redirects, reading at most 512 KB. Our crawler identifies itself as
DriftwitnessResearch/1.0 and links to
a page explaining how to block it. We fetched no page
other than robots.txt and made no repeat requests.
Evaluation. Rules were grouped by user-agent and matched per RFC
9309: a crawler takes the group whose product token matches its name exactly, and
otherwise the * group; within a group the longest matching path wins and
ties resolve to allow.[2] We do not treat
Applebot rules as governing Applebot-Extended, or
Googlebot-News rules as governing Googlebot, because those
are separate product tokens controlling separate things. Allowing that substitution
changed the verdict on 5% of domains when we tested it, most visibly by doubling
Googlebot's apparent block rate. "Blocked" means the rules that apply to that
crawler deny the site root. "Explicit" means the crawler is named in a user-agent
line rather than caught by a wildcard. A 4xx response means no robots.txt, which the
standard treats as unrestricted access, so those domains count as not blocking.
Denominator. Percentages are shares of the 17,945 domains that gave us a usable answer, meaning a robots.txt we could read or a definitive "there isn't one", not of the 13,035 that returned a parseable file. Using the smaller denominator would raise every figure here by roughly a third. We publish the conservative version. A further 180 domains returned server errors or empty responses and are counted in neither.
Validation. Before the run, the parser was checked against four sites with independently known policies (nytimes.com, reddit.com, en.wikipedia.org, bbc.com). Every domain named in this article was then re-fetched individually and its rules read by hand.
What this data cannot tell you
- 27.5% of the domains we tried did not respond to us at all. Many are CDN or infrastructure hostnames that never serve a site; others block datacenter addresses. If unreachable domains block AI crawlers at a different rate than reachable ones, every figure here shifts.
- robots.txt is a request, not a fence. This measures what sites ask crawlers to do, not what crawlers do. Sites also enforce through firewalls and bot management that we cannot see from the outside.
- A single scan cannot separate a stale rule from a deliberate one with certainty. That is why we report naming and blocking as two different numbers instead of guessing at intent.
- One snapshot shows no direction. We are repeating this scan monthly against the same list and will publish what moves.
Cite this
Driftwitness Research (2026). "Who blocks AI crawlers? A scan of the top 25,000 sites." driftwitness.com/research/who-blocks-ai-crawlers — published August 26, 2026. Aggregated data: JSON, CC BY 4.0.
Sources and tools
- Tranco list 38LNL, generated August 24, 2026 — tranco-list.eu. Method: Le Pochat, Van Goethem, Tajalizadehkhoob, Korczyński & Joosen, "Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation," NDSS 2019.
- RFC 9309, "Robots Exclusion Protocol" (Koster, Illyes, Zeller, Sassman, 2022) — rfc-editor.org/rfc/rfc9309.html
- OpenAI, "Overview of OpenAI crawlers" (GPTBot, OAI-SearchBot, ChatGPT-User) — developers.openai.com/api/docs/bots
- Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?" (ClaudeBot, Claude-SearchBot, Claude-User) — support.claude.com
- Google, "Google crawlers and fetchers" (Googlebot, Google-Extended) — developers.google.com
- Perplexity, "PerplexityBot and Perplexity-User" — docs.perplexity.ai/docs/resources/perplexity-crawlers
- Common Crawl, "CCBot" — commoncrawl.org/ccbot
Corrections: none yet. If your site is described wrongly here, or you changed a rule after August 26, 2026, tell us at [email protected]; we amend in place and log every change here, dated.
Driftwitness Research is the research arm of Driftwitness, an SEO change-monitoring service for agencies, currently in early access. This scan runs monthly; the next one publishes in September.