Most sites blocking AI training never chose to

What a content signal is

In October 2025 Cloudflare published the Content Signals Policy¹, an extension to robots.txt. Alongside the familiar User-agent and Disallow lines, a site can add a directive stating what it permits its content to be used for, rather than which crawler may fetch it:

Content-Signal: search=yes, ai-input=yes, ai-train=no

search covers building a search index and showing snippets. ai-input covers feeding content into a generated answer. ai-train covers training or fine-tuning a model. The signals are advisory. Nothing enforces them, and a crawler that ignores them breaks no technical rule. They are a statement of terms, which is why Cloudflare's version ships with a preamble invoking Article 4 of EU Directive 2019/790, the text that makes a rights reservation legally legible in the EU.²

Cloudflare also wired the policy into its managed robots.txt feature³. Turn that on and Cloudflare writes the policy block and the signal line into your robots.txt for you. It is opt-in, it is available on every plan including the free tier, and it ships with a preference already chosen: search yes, AI training no.

How you can tell a default from a decision

This is the part that makes the measurement possible, so it is worth being precise about it before any number appears.

Cloudflare's inserted line is one exact string, with no spaces after the commas:

search=yes,ai-train=no,use=reference

People do not write like that. A person writing the line by hand puts spaces after commas and orders the keys however they think about them. In our data the author-written lines appear in 10 distinct spellings of what are only a handful of actual positions, including ai-train=no, search=yes, ai-input=yes and search=yes, ai-input=yes, ai-train=no, which say exactly the same thing in a different order.

We used two independent markers rather than the string alone. First, the presence of Cloudflare's policy preamble, which is long, distinctive and quotes the EU directive verbatim. No site author writes that by accident. Second, exact-string equality with the shipped default. A file carrying the preamble is machine-assisted; a file whose signal is byte-identical to the default was almost certainly not typed by a human.

What this cannot tell you. A site owner who turned the feature on, read the default and agreed with it has made a real decision that looks identical in the file to one who clicked a toggle without reading. We can measure whether a line was machine-written. We cannot measure whether anyone meant it. Every claim below is about provenance, not intent.

The split

Of 13,035 domains serving a parseable robots.txt, 1,104 carry a content signal. That is 8.5% of the sample, for a standard that is roughly ten months old, which is fast adoption by robots.txt standards.

It is also, overwhelmingly, one company's default. 1,059 files carry the Cloudflare policy preamble. 912 of those pair it with a signal line, and 870 of those are the byte-identical shipped string. That is 95.4% of the machine-assisted files and 78.8% of every content signal on the web's most-visited domains.

The contrast is the finding. Among files carrying the CDN's boilerplate, 910 of 912 say ai-train=no: 99.8%. Among the 192 files whose signal has no boilerplate anywhere near it, 82 say ai-train=yes: 42.7%.

Put plainly, when a machine fills in the answer it says no to AI training almost every time. When a person fills it in, it is close to a coin flip, tipping slightly toward no. Those two populations are not answering the same question. One is a policy shipped by a vendor; the other is a view held by a publisher.

Sites displaying a rights reservation that reserves nothing

147 domains carry Cloudflare's policy preamble, including the language invoking the EU directive, but have no Content-Signal line at all.

The effect is a robots.txt that reads like a legal notice and declares no preference. Cloudflare documents this: on free-plan domains without a managed robots.txt, the policy definition is displayed without any active preference.³ It is working as designed. It is also, from the outside, indistinguishable from a site that meant to say something and did not, and we would expect very few of those 147 owners to know the text is on their site.

Adoption is higher further down the list

Content signals are not concentrated at the top of the web. They are more common as you go down it.

That gradient is what you would expect if adoption is driven by a CDN toggle rather than by editorial policy. The largest sites tend to hand-manage their robots.txt and to sit behind their own infrastructure. Smaller sites are likelier to be on a managed platform where a single switch writes the file for them.

Some sites that did decide

Every domain named here was re-fetched by hand on August 26, 2026, separately from the bulk scan, and served the value shown.

Cloudflare's own site is worth pausing on, and not as a gotcha. It declares ai-train=yes while the default it ships to customers declares ai-train=no. That is not a contradiction. It is precisely what a preferences standard is for: a publisher weighing its own position rather than inheriting one. It is also a neat illustration of the gap this piece is about, since the company that wrote the standard is in the small minority that hand-wrote its answer.

Why we think this matters

The number that will get quoted from a dataset like this is "most sites block AI training." Our data does support a version of that sentence, and we think the sentence is misleading.

What we can actually see is that a large and rapidly growing share of the web now carries a machine-written statement about AI training, and that where humans write the statement themselves they disagree with the default about 42.7% of the time. Anyone reasoning about publisher sentiment from robots.txt aggregates is, for the most part, measuring one vendor's shipped setting.

This is the second time our own data has pointed the same way. In the previous drop we found that 93.8% of sites blocking GPTBot had never written any rule about OpenAI's separate search crawler, meaning they were allowing it by default rather than on purpose. Different mechanism, same shape: the web's AI permissions are mostly inherited, not authored.

Method

We fetched /robots.txt from the top 25,000 domains of the Tranco 38LNL list on August 24 to 26, 2026, using the declared user agent DriftwitnessResearch/1.0, which links to an opt-out page. 17,945 domains responded. 13,035 of those served a parseable robots.txt, and that is the denominator for every percentage above. A domain with no robots.txt cannot carry a content signal, so including those would deflate every rate reported here.

This analysis re-reads bytes archived during that scan. Nobody was fetched a second time to produce it, apart from the 10 named domains in the table above, which were verified individually.

Limitations. The Tranco list skews toward large and Western sites, so this is not a random sample of the web. Boilerplate detection is a heuristic: a site that copied Cloudflare's preamble by hand would be counted as machine-assisted, and one that enabled the feature and then rewrote the line would count as author-written. Both directions exist and we cannot size them. We read the first Content-Signal line in each file and do not model per-user-agent signal scoping. Adoption is a single point in time; nothing here measures change, which is what the next drop will do.

Corrections: none yet. If your site is described wrongly here, or you changed a signal after August 26, 2026, tell us at [email protected]; we amend in place and log every change here, dated.

Driftwitness Research is the research arm of Driftwitness, an SEO change-monitoring service for agencies, currently in early access. The next drop re-reads these same domains to measure what changed.