AI Crawler List 2026: Every User-Agent, robots.txt per Bot

Forty-four AI-related user-agent tokens are worth knowing as of 26 August 2026, and four vendors state in their own documentation that some of their bots ignore robots.txt. Thirty-seven of the forty-four have a first-party vendor page; the remaining seven, two of them retired by Anthropic, rest on the community list. The full table is below, sorted by vendor, with the IP list you need to verify each one. Before it: what the categories mean, because "block AI" is three different decisions.

TL;DR#

• AI bots come in three kinds: training crawlers, search indexers, and user-triggered fetchers. Each vendor documents them separately and lets you allow or block each one independently.
• User-triggered fetchers generally do not honour robots.txt. OpenAI, Perplexity, Google and Meta say so in their own docs. Blocking them is a firewall job, not a robots.txt job.
• Most major vendors now publish an IP list in JSON; Meta gives an AS number instead, and Mistral covers only two of its three bots. Verify by IP, never by User-Agent string.
• In my scan of 53 robots.txt files (26 August 2026) only 6 sites fully block any AI bot, 5 publish a Content-Signal line, and 2 still carry Anthropic's retired anthropic-ai token alongside a current ClaudeBot group.
• Three copy-paste presets at the end: allow everything, block training only, block all AI.

What counts as an AI crawler#

An AI crawler is any automated client that fetches web pages on behalf of a large language model product. The name covers three jobs that vendors run as separate bots with separate rules. A training crawler collects text to fit the next model. A search indexer builds the retrieval index that grounds answers with citations. A user-triggered fetcher opens one URL because a person asked the assistant to read it.

The distinction matters because the vendors' own policies differ by job. OpenAI's bots page states that each of its three robots.txt settings "is independent of the others" (OpenAI, Overview of OpenAI crawlers). Google's documentation splits crawlers into "common crawlers", which "always respect robots.txt rules for automatic crawls", and "user-triggered fetchers", which "generally ignore robots.txt rules" (Google Search Central, user-triggered fetchers).

Allowing a bot is only the first of four preconditions, and this article covers the first one exhaustively. If you have already allowed everything and still see nothing, the fault is usually further down: why your site doesn't show in ChatGPT and Perplexity walks the failure modes in order, and the JavaScript trap covers the most common one, a page that is allowed, fetched, and empty because the bot does not run your JavaScript.

The complete AI crawler list, from vendor docs#

Categories: T = model training, S = search index or retrieval, U = user-triggered fetch, O = other. "Respects robots.txt" repeats the vendor's own statement; "no doc" means I could not find a first-party page and the entry rests on the community list at ai-robots-txt/ai.robots.txt.

Two numbers give the table scale. The community-maintained robots.json at ai-robots-txt listed 166 tokens on 26 August 2026, most of them small or unverified operators. Cloudflare's AI Crawl Control reference lists 20 bots it can verify cryptographically or by IP, grouped as AI Crawler, AI Search, AI Assistant and Search Engine (Cloudflare, AI Crawl Control bots).

Which AI bots ignore robots.txt?#

Four bots say in their own documentation that robots.txt may not apply to them, and all four are user-triggered fetchers. OpenAI: for ChatGPT-User, "robots.txt rules may not apply" because actions are user-initiated. Perplexity: Perplexity-User "generally ignores robots.txt rules" (Perplexity, Crawlers). Google: user-triggered fetchers "generally ignore robots.txt rules" because "the fetch was requested by a user". Meta: meta-externalfetcher "May bypass robots.txt rules" (Meta, Web crawlers).

The logic is consistent across vendors: a person pasted your URL into the assistant, so the fetch is treated like a browser visit. If you need to stop those, robots.txt is the wrong tool. Use a WAF rule or Cloudflare's "Block AI bots" setting, which since July 2026 targets three behaviours, Search, Agent and Training, and applies to "Verified bots classified with that behavior, plus additional unverified bots" (Cloudflare, Block AI bots).

Anthropic is the exception among the big vendors: its page states that "Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt" and names all three bots, Claude-User included (Anthropic, Does Anthropic crawl data from the web).

Does noindex mean anything to AI crawlers?#

For OpenAI and Anthropic, no. Neither bots page mentions meta robots tags or X-Robots-Tag; both describe robots.txt as the only control. I wrote the opposite in an earlier article on this site and have corrected it there; the honest position is that noindex is documented by exactly two AI-relevant vendors. Apple states that Applebot honours noindex, nosnippet ("blocks description generation and AI model usage for context"), nofollow, none and a scoped X-Robots-Tag: applebot: nosnippet (Apple, About Applebot). Amazon states that Amazonbot honours noarchive, noindex and none (Amazon, Amazonbot).

There is one more subtlety. A page blocked in robots.txt is never fetched, so a noindex on it is never seen by anyone. If your goal is "fetch, but don't quote", robots.txt cannot express it; Apple's nosnippet and the Content-Signal line below are the closest thing to that request.

How to verify a crawler by IP#

A User-Agent string is a free-text header. Anyone scraping your site can send Mozilla/5.0 (compatible; GPTBot/1.1) and inherit whatever allowance you gave OpenAI. Verification means checking the source IP against the vendor's published list, which most major vendors now serve as JSON. The gaps are worth knowing before you write the rule: Meta publishes an AS number rather than a file, and Mistral covers MistralAI-Index and MistralAI-User but not MistralAI-Training.

# scripts/verify-ai-bot.sh
# Usage: ./scripts/verify-ai-bot.sh <ip> <vendor>
# vendor: openai | anthropic | perplexity | google | apple | commoncrawl | mistral | duckduckgo
set -euo pipefail
ip="$1"; vendor="$2"
case "$vendor" in
  openai)      urls="https://openai.com/gptbot.json https://openai.com/searchbot.json https://openai.com/chatgpt-user.json" ;;
  anthropic)   urls="https://claude.com/crawling/bots.json" ;;
  perplexity)  urls="https://www.perplexity.com/perplexitybot.json https://www.perplexity.com/perplexity-user.json" ;;
  google)      urls="https://developers.google.com/static/crawling/ipranges/common-crawlers.json https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json" ;;
  apple)       urls="https://search.developer.apple.com/applebot.json" ;;
  commoncrawl) urls="https://index.commoncrawl.org/ccbot.json" ;;
  mistral)     urls="https://mistral.ai/mistralai-index-ips.json https://mistral.ai/mistralai-user-ips.json" ;;
  duckduckgo)  urls="https://duckduckgo.com/duckassistbot.json" ;;
  *) echo "unknown vendor"; exit 2 ;;
esac
python3 - "$ip" $urls <<'PY'
import ipaddress, json, sys, urllib.request
ip = ipaddress.ip_address(sys.argv[1])
for url in sys.argv[2:]:
    data = json.load(urllib.request.urlopen(url, timeout=15))
    prefixes = []
    def walk(node):
        if isinstance(node, dict):
            for key, value in node.items():
                if key in ("ipv4Prefix", "ipv6Prefix", "prefix", "cidr") and isinstance(value, str):
                    prefixes.append(value)
                else:
                    walk(value)
        elif isinstance(node, list):
            for item in node:
                walk(item)
        elif isinstance(node, str) and "/" in node:
            prefixes.append(node)
    walk(data)
    for prefix in prefixes:
        try:
            if ip in ipaddress.ip_network(prefix, strict=False):
                print(f"MATCH {ip} in {prefix} ({url})")
                sys.exit(0)
        except ValueError:
            continue
print(f"NO MATCH for {ip}")
sys.exit(1)
PY

The script walks any of the vendors' JSON shapes (OpenAI and Google use prefixes[].ipv4Prefix, Anthropic and Perplexity use flat lists) and exits 0 on a match. Meta publishes no JSON; use whois -h whois.radb.net -- '-i origin AS32934' | grep ^route and match against those routes. Google and Apple additionally support reverse DNS: a genuine Googlebot resolves to *.googlebot.com or *.geo.googlebot.com, a genuine Applebot to *.applebot.apple.com.

Wherever you run, the cheapest place to log candidate requests is the edge or middleware layer your framework already has — middleware.ts on a Next.js app, whatever runs before the route handler elsewhere: match the User-Agent against the tokens above, write ua, ip and path to the request log, and run the script against the IPs weekly. That log is also the raw material for measuring AI visibility rather than assuming it: how to audit AI visibility sets out the checks and the numbers this site produced when it ran them on itself.

What 53 sites actually do: an August 2026 scan#

I fetched robots.txt from 65 domains on 26 August 2026: the AI-visibility tool blogs and technical SEO sites that compete with this one, the frameworks and hosts a developer reads (Vercel, Next.js, MDN, Cloudflare, Kinsta, WP Engine), a set of publishers, and 12 Ukrainian agencies, media and marketplaces. 53 files were readable; 12 returned an HTML shell, a 403 or a bot challenge and were excluded. A "block" below means a group for that exact token with Disallow: /.

Three things stand out. First, the blocking is concentrated in publishers: medium.com blocks six tokens, forbes.com nine, techcrunch.com nine. Every SEO tool blog, every framework site and every Ukrainian agency in the sample allows everything; the only developer-tool site that blocks anything is agent-ready.dev, and it blocks CCBot and Bytespider while explicitly allowing search bots. Second, Content-Signal adoption is an infrastructure story, not an editorial one: four of the five sites publishing it (Vercel, Kinsta, Cloudflare, Supabase) are hosting or CDN companies whose managed robots.txt adds the line. Third, techcrunch.com blocks ChatGPT-User and Claude-Web, one of which ignores robots.txt by OpenAI's own account and the other of which no longer exists.

Method and limits: single fetch per domain with a plain HTTP client; sites that serve different robots.txt to different user-agents were not tested; 12 unreadable files skew the sample away from large publishers, who are the most likely blockers. Raw CSV is linked at the end.

Content-Signal and the IETF draft: preferences per use, not per bot#

The per-bot model does not scale: 166 tokens in the community list, and every new product adds one. Two efforts express the preference by use case instead.

Cloudflare's managed robots.txt emits a line such as Content-Signal: search=yes, ai-train=no, use=reference. The vocabulary has three keys: search for "building a search index and providing search results", ai-input for "inputting content into one or more AI models (e.g., retrieval augmented generation, grounding)", and ai-train for "training or fine-tuning AI models". Compliance is voluntary (Cloudflare, Managed robots.txt).

The IETF AI Preferences working group is standardising the same idea. draft-ietf-aipref-vocab-07 (19 August 2026) defines train-ai and search with values y, n or unknown, and draft-ietf-aipref-attach-05 defines a Content-Usage HTTP header and a robots.txt rule, for example Content-Usage: train-ai=n for the whole site or Content-Usage: /ai-ok/ train-ai=y for a path (IETF, draft-ietf-aipref-attach). The draft states plainly that "preferences are not a security mechanism". Nothing in either document is enforced by any crawler today, but it costs one line, and the five hosting companies in my scan already ship it.

Three robots.txt presets#

Each preset keeps Sitemap: and the parameter rules this site already uses; add your own Disallow lines for admin paths under the * group.

Preset 1: allow everything, state your preference#

Use this if you want AI citations and accept training. This is what alexturik.com runs.

# public/robots.txt
User-agent: *
Allow: /
 
Content-Signal: search=yes, ai-input=yes, ai-train=yes
 
Sitemap: https://example.com/sitemap.xml

Preset 2: block training, keep search and assistants#

Use this if you want to be cited by ChatGPT search, Perplexity, Claude and Google AI features but not to be in the next training set. The tokens below are the training-only bots and control tokens; search and user-triggered bots stay allowed by the * group.

# public/robots.txt
User-agent: *
Allow: /
 
User-agent: GPTBot
Disallow: /
 
User-agent: ClaudeBot
Disallow: /
 
User-agent: Google-Extended
Disallow: /
 
User-agent: Applebot-Extended
Disallow: /
 
User-agent: meta-externalagent
Disallow: /
 
User-agent: Amazonbot
Disallow: /
 
User-agent: MistralAI-Training
Disallow: /
 
User-agent: CCBot
Disallow: /
 
User-agent: Bytespider
Disallow: /
 
User-agent: cohere-training-data-crawler
Disallow: /
 
User-agent: Timpibot
Disallow: /
 
Content-Signal: search=yes, ai-input=yes, ai-train=no
 
Sitemap: https://example.com/sitemap.xml

Two caveats. meta-externalagent and Amazonbot both train and index, so blocking them also removes you from Meta AI search and Alexa answers; Amazon's Amzn-SearchBot stays allowed and does not train. Applebot-Extended blocks training while leaving Siri and Spotlight search on Applebot.

Preset 3: block all documented AI bots#

Use this only if you have decided AI traffic has no value for you. Remember that the four user-triggered fetchers will still open a page when a person asks; stopping those requires a firewall rule.

# public/robots.txt
User-agent: *
Allow: /
 
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: meta-webindexer
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-Training
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: CCBot
User-agent: Bytespider
User-agent: Diffbot
User-agent: omgili
User-agent: ImagesiftBot
User-agent: YouBot
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Timpibot
Disallow: /
 
Content-Signal: search=no, ai-input=no, ai-train=no
 
Sitemap: https://example.com/sitemap.xml

Stacked User-agent lines in one group are valid under RFC 9309, section 2.2.1, and every parser listed above supports them (RFC 9309, Robots Exclusion Protocol). Googlebot, bingbot and Applebot are deliberately absent from Preset 3: blocking them removes you from classic search, which is not what "block AI" usually means.

Checking the result#

After deploying, confirm the file parses the way you think it does. Google's robots.txt report in Search Console shows the fetched file and any parse errors. For a specific bot, the quickest check is the group lookup with curl and awk:

# scripts/robots-check.sh
# Usage: ./scripts/robots-check.sh https://example.com GPTBot
set -euo pipefail
site="$1"; bot="$(echo "$2" | tr 'A-Z' 'a-z')"
curl -sL "$site/robots.txt" | awk -v bot="$bot" '
  BEGIN { ing = 0 }
  /^[ \t]*[Uu]ser-[Aa]gent:/ {
    ua = tolower($0); sub(/^[ \t]*user-agent:[ \t]*/, "", ua); gsub(/[ \t\r]+$/, "", ua)
    if (ua == bot) { ing = 1; print "group: " $0 } else if (!prev) { ing = 0 }
    prev = 1; next
  }
  { prev = 0 }
  ing && /^[ \t]*([Dd]isallow|[Aa]llow|[Cc]rawl-delay):/ { print "  " $0 }
'

Run it against your production URL for each token you added. If it prints nothing, the group is missing or misspelled; the most common misspellings in the files I scanned were GPTbot, Claude-bot and Perplexity Bot with a space.

For a browser-side version of the same question, the AEO Checker extension audits the open tab's robots.txt as one of its nine checks, which is the faster way to look at a site that is not yours. Its scope is deliberately narrower than this article: ten tokens, four real-time and six training, collapsed into one pass/warn/fail rather than printed per group. It also still carries anthropic-ai in that list, which the table above marks retired. That is a stale entry I owe the extension, and a fair illustration of how fast a hard-coded bot list ages.

Changelog#

• 2026-08-27: first version. 44 tokens across 38 table rows, 53-site scan, Content-Signal and IETF draft-ietf-aipref-attach-05.

FAQ#

Which AI crawlers ignore robots.txt?#

By their own documentation: ChatGPT-User (OpenAI), Perplexity-User (Perplexity), Google-Agent and other Google user-triggered fetchers, and meta-externalfetcher (Meta). All four are user-triggered fetchers: a person asked the assistant to open your URL, so the vendor treats the request as the user's, not the bot's. Training and search crawlers from the same vendors do honour robots.txt.

Does blocking GPTBot remove my site from ChatGPT search?#

No. OpenAI runs three separate bots with independent rules. GPTBot governs training, OAI-SearchBot governs ChatGPT search answers, and ChatGPT-User fetches pages on a user's request. Blocking GPTBot alone leaves you eligible for search citations; blocking OAI-SearchBot removes you from search answers but not from navigational links.

How do I verify that a request really came from GPTBot or ClaudeBot?#

Compare the request IP with the vendor's published ranges. OpenAI serves openai.com/gptbot.json, Anthropic serves claude.com/crawling/bots.json, Perplexity serves perplexity.com/perplexitybot.json, Google publishes JSON files under developers.google.com/static/crawling/ipranges/. A User-Agent string alone proves nothing: any script can send it.

Is Google-Extended a crawler?#

No. Google-Extended has no user-agent string of its own. It is a robots.txt control token: crawling still happens as Googlebot, and a Disallow for Google-Extended tells Google not to use the fetched content to train future Gemini models or for grounding in Gemini Apps and Vertex AI. It does not affect Google Search inclusion.

What is Content-Signal in robots.txt?#

A line Cloudflare's managed robots.txt adds, such as Content-Signal: search=yes, ai-train=no, use=reference. It expresses a preference per use case rather than per bot. Compliance is voluntary, and the IETF is standardising a related vocabulary (train-ai, search) in the AI Preferences working group. In my August 2026 scan 5 of 53 sites published one.

How often should I update the AI crawler list in robots.txt?#

Check quarterly. Between 2024 and 2026 Anthropic retired anthropic-ai and Claude-Web and added Claude-User and Claude-SearchBot; OpenAI added OAI-SearchBot and OAI-AdsBot; Mistral added three bots. Two of the 53 sites I scanned still carry the retired anthropic-ai token; both block ClaudeBot as well, so the stale line only adds noise — but a file listing anthropic-ai alone would block nothing Anthropic runs today.

Sources#

• OpenAI, Overview of OpenAI crawlers — the three bots and the statement that each robots.txt setting is independent
• Anthropic, Does Anthropic crawl data from the web — the current three tokens, and the commitment that covers all of them
• Perplexity, Crawlers — PerplexityBot versus Perplexity-User
• Google Search Central, Google common crawlers
• Google Search Central, User-triggered fetchers — where "generally ignore robots.txt rules" comes from
• Google Search Central, Verifying Googlebot — the reverse DNS and IP-range method
• Apple, About Applebot — the one vendor page that documents noindex and nosnippet for an AI crawler
• Meta, Web crawlers
• Amazon, Amazonbot — the other one
• Mistral, Robots
• DuckDuckGo, DuckAssistBot
• Common Crawl, CCBot
• Diffbot, robots.txt FAQ
• Cloudflare, AI Crawl Control bots reference — the 20 bots Cloudflare can verify
• Cloudflare, Block AI bots — the firewall answer to user-triggered fetchers
• Cloudflare, Managed robots.txt and Content Signals — the Content-Signal vocabulary
• IETF, draft-ietf-aipref-vocab
• IETF, draft-ietf-aipref-attach — the Content-Usage header and robots.txt rule
• IETF, RFC 9309 Robots Exclusion Protocol — section 2.2.1 on stacked User-agent lines
• ai-robots-txt community list — the 166-token list, and the only source for the seven tokens with no vendor page
• Raw scan data — all 65 domains fetched on 26 August 2026, one row each

Part of the Answer Engine Optimization cluster — the full reading order in dependency sequence, plus a glossary of every term used across these articles.