We reread 25,000 robots.txt files after 72 hours. Almost nothing changed.

What we did

On August 26 we fetched robots.txt for the top 25,000 domains of the Tranco 38LNL list and recorded, for 18 search, AI-search, user-triggered and model-training crawlers, whether each was allowed to read the site root. That scan became our first data drop. On August 29 we ran it again against the same list, the same ranks, the same user agent and the same parser.

13,035 domains returned a parseable robots.txt the first time and 13,227 the second. 12,849 did both, and that is the only population any change rate here describes. A site that answered on the 26th and timed out on the 29th did not change its AI policy; it changed its availability. Counting that as policy movement is the easiest way to manufacture a number in a study like this, and it turns out to be the largest effect in the data.

Change is rare

415 of 12,849 files differed by even one byte, which is 3.23%. 46 files, 0.36%, changed a verdict for at least one of the 18 crawlers.

It is worth being precise about what makes a small number useful. This is not a statement that the web has settled its mind about AI. It is a measured floor. If you watch robots.txt across a portfolio of sites and your tooling reports change more often than this, the excess is coming from somewhere other than the sites. Anyone building or buying that kind of monitoring has had no published figure to check their alerting against.

Most robots.txt editing has nothing to do with AI

Of the 415 files that changed, 29 changed only in ways that carry no instruction to anybody: 17 differed in whitespace alone and 12 in comments alone. wordpress.com and wp.com differ only in a # This file was generated on … line. etsy.com rotates a joke in a comment.

That leaves 386 files whose actual directives changed. Of those, 340 — 88% — moved no AI-crawler verdict at all. They added query-string disallows, reordered rules, excluded a staging path, tightened crawl budget. The file gets edited reasonably often. It is mostly not edited about AI.

This is the number we would most like other people to reuse, because it reframes a question that is usually asked backwards. “Did the robots.txt change?” is close to useless on its own; nine times in ten the answer is yes and irrelevant. The question worth answering is whether it changed in a way that changes who may read the site. Those are different measurements, and only one of them is worth an alert.

Availability churn dwarfs policy churn

Between the two scans, 141 domains went from answering to unreachable and 328 went the other way. 469 domains moved in or out of reach, against 46 that changed a verdict — a ratio of about 10.2 to one.

This is a trap rather than a finding. A study treating “we could not read it this time” as a change would have reported roughly an order of magnitude more movement than happened, all of it its own timeouts. Our denominator rule, domains that answered in both scans, was written down before the second scan ran for exactly this reason.

Which crawlers moved

These counts are shown so nobody has to take the summary on trust. No direction is claimed. Across the 46 domains, 14 became more restrictive toward AI crawlers, 10 became less restrictive, 2 moved both ways inside one file, and 20 changed only which crawlers they name without changing who may read. With 28 domains flipping a block across 18 crawlers, there is no trend here to report.

Five changes, read by hand

Aggregates can hide a broken parser, so we opened both versions of several changed files and read them. All were genuine edits. They also show a range the counts cannot.

• auto-motor-und-sport.de added a 28-line block covering GPTBot, ChatGPT-User, OAI-SearchBot, the Claude crawlers, Google-Extended, Applebot-Extended and CCBot, carrying an internal ticket number and a German comment noting that Google-Extended and Applebot-Extended exclude model training only, leaving Googlebot and Applebot untouched. That is precisely the distinction our first drop found most sites never make, and it is a fair corrective to that finding: some publishers are reading these tokens very carefully.
• autohero.com added OAI-SearchBot: Allow and OAI-AdsBot: Allow. An explicit opt-in, not a block.
• baqend.com gained a block allowing GPTBot, ClaudeBot, PerplexityBot and Google-Extended, wrapped in # START nuxt-robots markers. The edit was made by a framework module rather than by a person deciding anything, which is the same mechanism-not-opinion pattern we measured for Cloudflare's Content-Signal in the second drop.
• blitz.gg added meta-externalagent and two sibling Meta crawlers under a Disallow: /, and separately corrected a User-Agent: to User-agent: elsewhere in the file.
• alo.bg reordered existing rules and added several query-string disallows. No AI crawler appears anywhere in the change.

The mistake that would have made this article wrong

This is the part we would want to read in someone else's write-up, so it belongs in ours.

Both scans store a per-crawler record. The obvious way to find changes is to compare those records and count the ones that differ. Done that way, 4,633 of 5,350 domains “changed”, or 86.6%. That article writes itself: the web rewrote its AI rules in three days. It would have been entirely false.

Two fields in the record are not verdicts. One stores which user-agent token matched, and it differed only in letter case between the two runs — GPTBot against gptbot. The other is a convenience flag whose definition we had changed ourselves between scans. On the two fields that actually say whether a crawler may read the site, the two runs agreed on all 192,600 cells.

We caught it because the comparison tool was checked before it was believed. Run it on two identical inputs and it must report nothing; run it on a change whose size is already known and it must reproduce that. It reproduced our earlier parser correction exactly, at 763 domains of 13,035. A tool that cannot reproduce a number you have already published has no business producing a new one.

What this cannot tell you

• How often robots.txt changes in general. This is one 72-hour window. A rate measured over three days cannot be multiplied out to a month or a year, and that will still be true when the next scan lands: a second window gives two observations, not a trend.
• Which way the web is moving on AI. 28 domains flipped a block. That is too few to have a direction.
• Why anyone changed anything. We read files, not intentions. The five cases above are described by what they contain, not by what anyone meant by them.
• Anything about sites outside this frame. The population is the top 25,000 domains of one Tranco list that answered twice. Smaller sites, and sites that refuse our crawler, are not represented.