About the DriftwitnessResearch crawler
If you followed a link here from your server logs: this page explains who we are, what our crawler collects, how it behaves, and how to exclude your site in under a minute.
What we're studying
Driftwitness runs an open study of SEO drift — how often the
SEO-critical elements of public web pages change or regress over time (for example,
title tags being rewritten, noindex appearing on pages that were
indexable, canonical tags repointing, or robots.txt newly disallowing a section).
We publish the aggregate findings openly. The goal is to document how common these
silent regressions really are.
How our crawler identifies itself
Every study fetch is sent with this exact, honest User-Agent string:
We do not disguise our crawler as a browser or as another company's bot for the purpose of collecting your page data.
What it collects — and what it doesn't
- Only public pages. We never access login areas, paywalled content, or anything behind an access control.
- Only the SEO-critical elements of a page: title, meta robots / canonical tags, headings, structured-data (JSON-LD) presence, robots.txt directives, and redirect behaviour — plus HTTP status.
- We do not harvest personal data, email addresses, or full-page content for republication, and we do not build profiles of individuals.
- Visits are infrequent (typically once, occasionally re-checked weeks later) and rate-limited — a small number of requests per site, never a hammering crawl.
It respects robots.txt
Our crawler obeys robots.txt. To keep DriftwitnessResearch out of your
site entirely, add this to your robots.txt:
User-agent: DriftwitnessResearch
Disallow: /We honour that on the next visit. We never circumvent a block: if a request is challenged or refused, we record that outcome and move on — we do not solve CAPTCHAs, rotate IPs to evade blocks, or retry to get around it.
Opt out directly
Email [email protected] with your domain and we'll exclude it from every future wave and from any per-site published data — no questions asked. Excluded domains are reported only as an anonymous count.
A note on crawler-access probes
Part of the study measures which public crawler policies sites apply. To do that, a very small number of requests (homepage only, a handful per site) are sent presenting the documented User-Agent strings of well-known crawlers, to observe whether access is allowed or refused. These probes are minimal, are never used to collect or retain your page content, and never bypass any control — they only record the access outcome.
What we've published
Everything this crawler collects ends up in public findings, with the method and sample disclosed. You can read the studies at driftwitness.com/research, including the August 2026 scan of AI-crawler rules across the web's most-visited domains, which is the run most likely to have put our User-Agent in your logs.
Questions
Anything else, or a concern about a specific request in your logs? Write to [email protected] and a human will reply.