--- title: "AI Crawler List 2026: Every User-Agent, robots.txt per Bot" description: "44 AI user-agent tokens with owners and IP lists, which ones ignore robots.txt, three copy-paste robots.txt presets, and a scan of 53 live robots files." date: 2026-08-27 tags: [ai-search, robots-txt, aeo, measurement] --- # AI Crawler List 2026: Every User-Agent, robots.txt per Bot Forty-four AI-related user-agent tokens are worth knowing as of 26 August 2026, and four vendors state in their own documentation that some of their bots ignore robots.txt. Thirty-seven of the forty-four have a first-party vendor page; the remaining seven, two of them retired by Anthropic, rest on the community list. The full table is below, sorted by vendor, with the IP list you need to verify each one. Before it: what the categories mean, because "block AI" is three different decisions. ## TL;DR - AI bots come in three kinds: training crawlers, search indexers, and user-triggered fetchers. Each vendor documents them separately and lets you allow or block each one independently. - User-triggered fetchers generally do not honour robots.txt. OpenAI, Perplexity, Google and Meta say so in their own docs. Blocking them is a firewall job, not a robots.txt job. - Most major vendors now publish an IP list in JSON; Meta gives an AS number instead, and Mistral covers only two of its three bots. Verify by IP, never by User-Agent string. - In my scan of 53 robots.txt files (26 August 2026) only 6 sites fully block any AI bot, 5 publish a `Content-Signal` line, and 2 still carry Anthropic's retired `anthropic-ai` token alongside a current `ClaudeBot` group. - Three copy-paste presets at the end: allow everything, block training only, block all AI. ## What counts as an AI crawler An AI crawler is any automated client that fetches web pages on behalf of a large language model product. The name covers three jobs that vendors run as separate bots with separate rules. A training crawler collects text to fit the next model. A search indexer builds the retrieval index that grounds answers with citations. A user-triggered fetcher opens one URL because a person asked the assistant to read it. The distinction matters because the vendors' own policies differ by job. OpenAI's bots page states that each of its three robots.txt settings "is independent of the others" ([OpenAI, Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)). Google's documentation splits crawlers into "common crawlers", which "always respect robots.txt rules for automatic crawls", and "user-triggered fetchers", which "generally ignore robots.txt rules" ([Google Search Central, user-triggered fetchers](https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers)). Allowing a bot is only the first of four preconditions, and this article covers the first one exhaustively. If you have already allowed everything and still see nothing, the fault is usually further down: [why your site doesn't show in ChatGPT and Perplexity](/blog/why-your-site-doesnt-show-in-chatgpt-and-perplexity) walks the failure modes in order, and [the JavaScript trap](/blog/the-javascript-trap-why-ai-bots-see-empty-pages) covers the most common one, a page that is allowed, fetched, and empty because the bot does not run your JavaScript. ## The complete AI crawler list, from vendor docs Categories: **T** = model training, **S** = search index or retrieval, **U** = user-triggered fetch, **O** = other. "Respects robots.txt" repeats the vendor's own statement; "no doc" means I could not find a first-party page and the entry rests on the community list at [ai-robots-txt/ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt). | User-agent token | Owner | Type | Respects robots.txt | IP list / verification | |---|---|---|---|---| | `GPTBot` | OpenAI | T | Yes | [openai.com/gptbot.json](https://openai.com/gptbot.json) | | `OAI-SearchBot` | OpenAI | S | Yes | [openai.com/searchbot.json](https://openai.com/searchbot.json) | | `ChatGPT-User` | OpenAI | U | No: "robots.txt rules may not apply" | [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json) | | `OAI-AdsBot` | OpenAI | O (ad landing pages) | Not stated | [openai.com/adsbot.json](https://openai.com/adsbot.json) | | `ClaudeBot` | Anthropic | T | Yes, plus `Crawl-delay` | [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) | | `Claude-User` | Anthropic | U | Yes (vendor statement covers all three) | same file | | `Claude-SearchBot` | Anthropic | S | Yes | same file | | `anthropic-ai`, `Claude-Web` | Anthropic | retired | Not on Anthropic's current page | none | | `PerplexityBot` | Perplexity | S | Yes | [perplexity.com/perplexitybot.json](https://www.perplexity.com/perplexitybot.json) | | `Perplexity-User` | Perplexity | U | No: "generally ignores robots.txt rules" | [perplexity.com/perplexity-user.json](https://www.perplexity.com/perplexity-user.json) | | `Googlebot` | Google | S | Yes | [common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json), rDNS `*.googlebot.com` | | `Google-Extended` | Google | T (control token only) | Yes | no own UA; crawls as Googlebot | | `GoogleOther`, `GoogleOther-Image`, `GoogleOther-Video` | Google | O (R&D fetches) | Yes | common-crawlers.json | | `Google-CloudVertexBot` | Google | S (Vertex AI agents) | Yes | common-crawlers.json | | `Google-Agent` | Google | U | No: "generally ignore robots.txt" | [user-triggered-agents.json](https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json) | | `Google-GeminiNotebook` | Google | U (NotebookLM) | No | [user-triggered-fetchers.json](https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json) | | `bingbot` | Microsoft | S (also feeds ChatGPT search) | Yes per Bing help | bingbot.json on bing.com; Bing help pages did not load for me on 26 Aug 2026 | | `Applebot` | Apple | S + T | Yes, plus `noindex`, `nosnippet`, `X-Robots-Tag: applebot:` | [applebot.json](https://search.developer.apple.com/applebot.json), rDNS `*.applebot.apple.com` | | `Applebot-Extended` | Apple | T (control token) | Yes; "does not crawl webpages" | no own UA | | `meta-externalagent` | Meta | T + S | Yes | `whois -h whois.radb.net -- '-i origin AS32934'` | | `meta-externalfetcher` | Meta | U | No: "May bypass robots.txt rules" | AS32934 | | `meta-webindexer` | Meta | S (Meta AI search) | Yes | AS32934 | | `facebookexternalhit` | Meta | O (link previews) | Partial | AS32934 | | `Amazonbot` | Amazon | T + S | Yes, plus `noindex`, `noarchive` | [amazonbot/ip-addresses](https://developer.amazon.com/amazonbot/ip-addresses/) | | `Amzn-SearchBot` | Amazon | S (no training) | Yes | [searchbot-ip-addresses](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/) | | `Amzn-User` | Amazon | U (Alexa) | Listed under REP, not caveated | [live-ip-addresses](https://developer.amazon.com/amazonbot/live-ip-addresses/) | | `MistralAI-Training` | Mistral | T | Yes | none: Mistral publishes IP lists for Index and User only | | `MistralAI-Index` | Mistral | S (no training) | Implied | [mistralai-index-ips.json](https://mistral.ai/mistralai-index-ips.json) | | `MistralAI-User` | Mistral | U | Not stated | [mistralai-user-ips.json](https://mistral.ai/mistralai-user-ips.json) | | `DuckAssistBot` | DuckDuckGo | S/U (no training) | Yes, applied within 72 hours | [duckassistbot.json](https://duckduckgo.com/duckassistbot.json) | | `CCBot` | Common Crawl | T (open corpus) | Yes | [ccbot.json](https://index.commoncrawl.org/ccbot.json), rDNS `*.crawl.commoncrawl.org` | | `Bytespider` | ByteDance | T | No doc; community list says no | none | | `Diffbot`, `Diffbot-User` | Diffbot | O + U | Yes by default | none | | `omgili` / `omgilibot` | Webz.io | O (resold data) | Yes | none | | `ImagesiftBot` | Hive | O (images) | Yes, falls back to Googlebot rules | none | | `YouBot` | You.com | S | No doc | none | | `cohere-ai`, `cohere-training-data-crawler` | Cohere | U / T | No doc | none | | `Timpibot` | Timpi | T | No doc | none | Two numbers give the table scale. The community-maintained `robots.json` at ai-robots-txt listed 166 tokens on 26 August 2026, most of them small or unverified operators. Cloudflare's AI Crawl Control reference lists 20 bots it can verify cryptographically or by IP, grouped as AI Crawler, AI Search, AI Assistant and Search Engine ([Cloudflare, AI Crawl Control bots](https://developers.cloudflare.com/ai-crawl-control/reference/bots/)). ## Which AI bots ignore robots.txt? Four bots say in their own documentation that robots.txt may not apply to them, and all four are user-triggered fetchers. OpenAI: for ChatGPT-User, "robots.txt rules may not apply" because actions are user-initiated. Perplexity: Perplexity-User "generally ignores robots.txt rules" ([Perplexity, Crawlers](https://docs.perplexity.ai/guides/bots)). Google: user-triggered fetchers "generally ignore robots.txt rules" because "the fetch was requested by a user". Meta: meta-externalfetcher "May bypass robots.txt rules" ([Meta, Web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)). The logic is consistent across vendors: a person pasted your URL into the assistant, so the fetch is treated like a browser visit. If you need to stop those, robots.txt is the wrong tool. Use a WAF rule or Cloudflare's "Block AI bots" setting, which since July 2026 targets three behaviours, Search, Agent and Training, and applies to "Verified bots classified with that behavior, plus additional unverified bots" ([Cloudflare, Block AI bots](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/)). Anthropic is the exception among the big vendors: its page states that "Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt" and names all three bots, Claude-User included ([Anthropic, Does Anthropic crawl data from the web](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)). ## Does `noindex` mean anything to AI crawlers? For OpenAI and Anthropic, no. Neither bots page mentions meta robots tags or `X-Robots-Tag`; both describe robots.txt as the only control. I wrote the opposite in [an earlier article on this site](/blog/the-3-word-tag-that-hides-your-best-pages-from-ai) and have corrected it there; the honest position is that `noindex` is documented by exactly two AI-relevant vendors. Apple states that Applebot honours `noindex`, `nosnippet` ("blocks description generation and AI model usage for context"), `nofollow`, `none` and a scoped `X-Robots-Tag: applebot: nosnippet` ([Apple, About Applebot](https://support.apple.com/en-us/119829)). Amazon states that Amazonbot honours `noarchive`, `noindex` and `none` ([Amazon, Amazonbot](https://developer.amazon.com/amazonbot)). There is one more subtlety. A page blocked in robots.txt is never fetched, so a `noindex` on it is never seen by anyone. If your goal is "fetch, but don't quote", robots.txt cannot express it; Apple's `nosnippet` and the `Content-Signal` line below are the closest thing to that request. ## How to verify a crawler by IP A User-Agent string is a free-text header. Anyone scraping your site can send `Mozilla/5.0 (compatible; GPTBot/1.1)` and inherit whatever allowance you gave OpenAI. Verification means checking the source IP against the vendor's published list, which most major vendors now serve as JSON. The gaps are worth knowing before you write the rule: Meta publishes an AS number rather than a file, and Mistral covers `MistralAI-Index` and `MistralAI-User` but not `MistralAI-Training`. ```bash # scripts/verify-ai-bot.sh # Usage: ./scripts/verify-ai-bot.sh # vendor: openai | anthropic | perplexity | google | apple | commoncrawl | mistral | duckduckgo set -euo pipefail ip="$1"; vendor="$2" case "$vendor" in openai) urls="https://openai.com/gptbot.json https://openai.com/searchbot.json https://openai.com/chatgpt-user.json" ;; anthropic) urls="https://claude.com/crawling/bots.json" ;; perplexity) urls="https://www.perplexity.com/perplexitybot.json https://www.perplexity.com/perplexity-user.json" ;; google) urls="https://developers.google.com/static/crawling/ipranges/common-crawlers.json https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json" ;; apple) urls="https://search.developer.apple.com/applebot.json" ;; commoncrawl) urls="https://index.commoncrawl.org/ccbot.json" ;; mistral) urls="https://mistral.ai/mistralai-index-ips.json https://mistral.ai/mistralai-user-ips.json" ;; duckduckgo) urls="https://duckduckgo.com/duckassistbot.json" ;; *) echo "unknown vendor"; exit 2 ;; esac python3 - "$ip" $urls <<'PY' import ipaddress, json, sys, urllib.request ip = ipaddress.ip_address(sys.argv[1]) for url in sys.argv[2:]: data = json.load(urllib.request.urlopen(url, timeout=15)) prefixes = [] def walk(node): if isinstance(node, dict): for key, value in node.items(): if key in ("ipv4Prefix", "ipv6Prefix", "prefix", "cidr") and isinstance(value, str): prefixes.append(value) else: walk(value) elif isinstance(node, list): for item in node: walk(item) elif isinstance(node, str) and "/" in node: prefixes.append(node) walk(data) for prefix in prefixes: try: if ip in ipaddress.ip_network(prefix, strict=False): print(f"MATCH {ip} in {prefix} ({url})") sys.exit(0) except ValueError: continue print(f"NO MATCH for {ip}") sys.exit(1) PY ``` The script walks any of the vendors' JSON shapes (OpenAI and Google use `prefixes[].ipv4Prefix`, Anthropic and Perplexity use flat lists) and exits 0 on a match. Meta publishes no JSON; use `whois -h whois.radb.net -- '-i origin AS32934' | grep ^route` and match against those routes. Google and Apple additionally support reverse DNS: a genuine Googlebot resolves to `*.googlebot.com` or `*.geo.googlebot.com`, a genuine Applebot to `*.applebot.apple.com`. Wherever you run, the cheapest place to log candidate requests is the edge or middleware layer your framework already has — `middleware.ts` on a Next.js app, whatever runs before the route handler elsewhere: match the User-Agent against the tokens above, write `ua`, `ip` and `path` to the request log, and run the script against the IPs weekly. That log is also the raw material for measuring AI visibility rather than assuming it: [how to audit AI visibility](/blog/how-to-audit-ai-visibility) sets out the checks and the numbers this site produced when it ran them on itself. ## What 53 sites actually do: an August 2026 scan I fetched `robots.txt` from 65 domains on 26 August 2026: the AI-visibility tool blogs and technical SEO sites that compete with this one, the frameworks and hosts a developer reads (Vercel, Next.js, MDN, Cloudflare, Kinsta, WP Engine), a set of publishers, and 12 Ukrainian agencies, media and marketplaces. 53 files were readable; 12 returned an HTML shell, a 403 or a bot challenge and were excluded. A "block" below means a group for that exact token with `Disallow: /`. | Finding | Count | Share of 53 | |---|---|---| | Fully block at least one AI-related token | 6 | 11.3% | | Block `GPTBot` | 3 (medium.com, forbes.com, techcrunch.com) | 5.7% | | Block `ClaudeBot` | 3 (same three) | 5.7% | | Block `Bytespider` | 4 | 7.5% | | Block `Amazonbot` | 4 (medium.com, forbes.com, notion.so, dou.ua) | 7.5% | | Block the retired `anthropic-ai` or `Claude-Web` | 2 (forbes.com, techcrunch.com) | 3.8% | | Publish a `Content-Signal:` line | 5 (vercel.com, kinsta.com, cloudflare.com, supabase.com, agent-ready.dev) | 9.4% | | Block any user-triggered fetcher (`ChatGPT-User`, `Perplexity-User`, `Claude-User`) | 1 (techcrunch.com blocks `ChatGPT-User`) | 1.9% | | Ukrainian domains (12 readable) blocking any AI token | 1 (dou.ua, `Amazonbot` only) | 8.3% of 12 | Three things stand out. First, the blocking is concentrated in publishers: medium.com blocks six tokens, forbes.com nine, techcrunch.com nine. Every SEO tool blog, every framework site and every Ukrainian agency in the sample allows everything; the only developer-tool site that blocks anything is agent-ready.dev, and it blocks `CCBot` and `Bytespider` while explicitly allowing search bots. Second, `Content-Signal` adoption is an infrastructure story, not an editorial one: four of the five sites publishing it (Vercel, Kinsta, Cloudflare, Supabase) are hosting or CDN companies whose managed robots.txt adds the line. Third, techcrunch.com blocks `ChatGPT-User` and `Claude-Web`, one of which ignores robots.txt by OpenAI's own account and the other of which no longer exists. Method and limits: single fetch per domain with a plain HTTP client; sites that serve different robots.txt to different user-agents were not tested; 12 unreadable files skew the sample away from large publishers, who are the most likely blockers. Raw CSV is linked at the end. ## Content-Signal and the IETF draft: preferences per use, not per bot The per-bot model does not scale: 166 tokens in the community list, and every new product adds one. Two efforts express the preference by use case instead. Cloudflare's managed robots.txt emits a line such as `Content-Signal: search=yes, ai-train=no, use=reference`. The vocabulary has three keys: `search` for "building a search index and providing search results", `ai-input` for "inputting content into one or more AI models (e.g., retrieval augmented generation, grounding)", and `ai-train` for "training or fine-tuning AI models". Compliance is voluntary ([Cloudflare, Managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/)). The IETF AI Preferences working group is standardising the same idea. `draft-ietf-aipref-vocab-07` (19 August 2026) defines `train-ai` and `search` with values `y`, `n` or unknown, and `draft-ietf-aipref-attach-05` defines a `Content-Usage` HTTP header and a robots.txt rule, for example `Content-Usage: train-ai=n` for the whole site or `Content-Usage: /ai-ok/ train-ai=y` for a path ([IETF, draft-ietf-aipref-attach](https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/)). The draft states plainly that "preferences are not a security mechanism". Nothing in either document is enforced by any crawler today, but it costs one line, and the five hosting companies in my scan already ship it. ## Three robots.txt presets Each preset keeps `Sitemap:` and the parameter rules this site already uses; add your own `Disallow` lines for admin paths under the `*` group. ### Preset 1: allow everything, state your preference Use this if you want AI citations and accept training. This is what alexturik.com runs. ```txt # public/robots.txt User-agent: * Allow: / Content-Signal: search=yes, ai-input=yes, ai-train=yes Sitemap: https://example.com/sitemap.xml ``` ### Preset 2: block training, keep search and assistants Use this if you want to be cited by ChatGPT search, Perplexity, Claude and Google AI features but not to be in the next training set. The tokens below are the training-only bots and control tokens; search and user-triggered bots stay allowed by the `*` group. ```txt # public/robots.txt User-agent: * Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Amazonbot Disallow: / User-agent: MistralAI-Training Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: cohere-training-data-crawler Disallow: / User-agent: Timpibot Disallow: / Content-Signal: search=yes, ai-input=yes, ai-train=no Sitemap: https://example.com/sitemap.xml ``` Two caveats. `meta-externalagent` and `Amazonbot` both train and index, so blocking them also removes you from Meta AI search and Alexa answers; Amazon's `Amzn-SearchBot` stays allowed and does not train. `Applebot-Extended` blocks training while leaving Siri and Spotlight search on `Applebot`. ### Preset 3: block all documented AI bots Use this only if you have decided AI traffic has no value for you. Remember that the four user-triggered fetchers will still open a page when a person asks; stopping those requires a firewall rule. ```txt # public/robots.txt User-agent: * Allow: / User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: OAI-AdsBot User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Google-CloudVertexBot User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: meta-externalfetcher User-agent: meta-webindexer User-agent: Amazonbot User-agent: Amzn-SearchBot User-agent: Amzn-User User-agent: MistralAI-Training User-agent: MistralAI-Index User-agent: MistralAI-User User-agent: DuckAssistBot User-agent: CCBot User-agent: Bytespider User-agent: Diffbot User-agent: omgili User-agent: ImagesiftBot User-agent: YouBot User-agent: cohere-ai User-agent: cohere-training-data-crawler User-agent: Timpibot Disallow: / Content-Signal: search=no, ai-input=no, ai-train=no Sitemap: https://example.com/sitemap.xml ``` Stacked `User-agent` lines in one group are valid under RFC 9309, section 2.2.1, and every parser listed above supports them ([RFC 9309, Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)). `Googlebot`, `bingbot` and `Applebot` are deliberately absent from Preset 3: blocking them removes you from classic search, which is not what "block AI" usually means. ## Checking the result After deploying, confirm the file parses the way you think it does. Google's robots.txt report in Search Console shows the fetched file and any parse errors. For a specific bot, the quickest check is the group lookup with curl and awk: ```bash # scripts/robots-check.sh # Usage: ./scripts/robots-check.sh https://example.com GPTBot set -euo pipefail site="$1"; bot="$(echo "$2" | tr 'A-Z' 'a-z')" curl -sL "$site/robots.txt" | awk -v bot="$bot" ' BEGIN { ing = 0 } /^[ \t]*[Uu]ser-[Aa]gent:/ { ua = tolower($0); sub(/^[ \t]*user-agent:[ \t]*/, "", ua); gsub(/[ \t\r]+$/, "", ua) if (ua == bot) { ing = 1; print "group: " $0 } else if (!prev) { ing = 0 } prev = 1; next } { prev = 0 } ing && /^[ \t]*([Dd]isallow|[Aa]llow|[Cc]rawl-delay):/ { print " " $0 } ' ``` Run it against your production URL for each token you added. If it prints nothing, the group is missing or misspelled; the most common misspellings in the files I scanned were `GPTbot`, `Claude-bot` and `Perplexity Bot` with a space. For a browser-side version of the same question, the [AEO Checker](/extensions/aeo-checker) extension audits the open tab's robots.txt as one of its nine checks, which is the faster way to look at a site that is not yours. Its scope is deliberately narrower than this article: ten tokens, four real-time and six training, collapsed into one pass/warn/fail rather than printed per group. It also still carries `anthropic-ai` in that list, which the table above marks retired. That is a stale entry I owe the extension, and a fair illustration of how fast a hard-coded bot list ages. ## Changelog - 2026-08-27: first version. 44 tokens across 38 table rows, 53-site scan, Content-Signal and IETF `draft-ietf-aipref-attach-05`. ## FAQ ### Which AI crawlers ignore robots.txt? By their own documentation: ChatGPT-User (OpenAI), Perplexity-User (Perplexity), Google-Agent and other Google user-triggered fetchers, and meta-externalfetcher (Meta). All four are user-triggered fetchers: a person asked the assistant to open your URL, so the vendor treats the request as the user's, not the bot's. Training and search crawlers from the same vendors do honour robots.txt. ### Does blocking GPTBot remove my site from ChatGPT search? No. OpenAI runs three separate bots with independent rules. GPTBot governs training, OAI-SearchBot governs ChatGPT search answers, and ChatGPT-User fetches pages on a user's request. Blocking GPTBot alone leaves you eligible for search citations; blocking OAI-SearchBot removes you from search answers but not from navigational links. ### How do I verify that a request really came from GPTBot or ClaudeBot? Compare the request IP with the vendor's published ranges. OpenAI serves openai.com/gptbot.json, Anthropic serves claude.com/crawling/bots.json, Perplexity serves perplexity.com/perplexitybot.json, Google publishes JSON files under developers.google.com/static/crawling/ipranges/. A User-Agent string alone proves nothing: any script can send it. ### Is Google-Extended a crawler? No. Google-Extended has no user-agent string of its own. It is a robots.txt control token: crawling still happens as Googlebot, and a Disallow for Google-Extended tells Google not to use the fetched content to train future Gemini models or for grounding in Gemini Apps and Vertex AI. It does not affect Google Search inclusion. ### What is Content-Signal in robots.txt? A line Cloudflare's managed robots.txt adds, such as `Content-Signal: search=yes, ai-train=no, use=reference`. It expresses a preference per use case rather than per bot. Compliance is voluntary, and the IETF is standardising a related vocabulary (`train-ai`, `search`) in the AI Preferences working group. In my August 2026 scan 5 of 53 sites published one. ### How often should I update the AI crawler list in robots.txt? Check quarterly. Between 2024 and 2026 Anthropic retired anthropic-ai and Claude-Web and added Claude-User and Claude-SearchBot; OpenAI added OAI-SearchBot and OAI-AdsBot; Mistral added three bots. Two of the 53 sites I scanned still carry the retired `anthropic-ai` token; both block `ClaudeBot` as well, so the stale line only adds noise — but a file listing `anthropic-ai` alone would block nothing Anthropic runs today. ## Sources - [OpenAI, Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) — the three bots and the statement that each robots.txt setting is independent - [Anthropic, Does Anthropic crawl data from the web](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — the current three tokens, and the commitment that covers all of them - [Perplexity, Crawlers](https://docs.perplexity.ai/guides/bots) — PerplexityBot versus Perplexity-User - [Google Search Central, Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) - [Google Search Central, User-triggered fetchers](https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers) — where "generally ignore robots.txt rules" comes from - [Google Search Central, Verifying Googlebot](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) — the reverse DNS and IP-range method - [Apple, About Applebot](https://support.apple.com/en-us/119829) — the one vendor page that documents `noindex` and `nosnippet` for an AI crawler - [Meta, Web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) - [Amazon, Amazonbot](https://developer.amazon.com/amazonbot) — the other one - [Mistral, Robots](https://docs.mistral.ai/robots/) - [DuckDuckGo, DuckAssistBot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot/) - [Common Crawl, CCBot](https://commoncrawl.org/ccbot) - [Diffbot, robots.txt FAQ](https://www.diffbot.com/docs/crawl/faq/robots-txt) - [Cloudflare, AI Crawl Control bots reference](https://developers.cloudflare.com/ai-crawl-control/reference/bots/) — the 20 bots Cloudflare can verify - [Cloudflare, Block AI bots](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) — the firewall answer to user-triggered fetchers - [Cloudflare, Managed robots.txt and Content Signals](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) — the `Content-Signal` vocabulary - [IETF, draft-ietf-aipref-vocab](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/) - [IETF, draft-ietf-aipref-attach](https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/) — the `Content-Usage` header and robots.txt rule - [IETF, RFC 9309 Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) — section 2.2.1 on stacked `User-agent` lines - [ai-robots-txt community list](https://github.com/ai-robots-txt/ai.robots.txt) — the 166-token list, and the only source for the seven tokens with no vendor page - [Raw scan data](/aeo/robots-scan-2026-08.csv) — all 65 domains fetched on 26 August 2026, one row each Part of the [Answer Engine Optimization](/aeo) cluster — the full reading order in dependency sequence, plus a [glossary](/aeo/glossary) of every term used across these articles.