AI Crawler List 2026: Every User-Agent, robots.txt per Bot

Last checked 9 September 2026, two weeks after the 26 August 2026 scan. Same 12 tokens, same 50 domains, zero changes.

Most competing lists put a date in the title (Search Engine Journal, HumanSecurity and Momentic all do), and that date says when the list was compiled. The line above marks something narrower: when it was last checked, with the checked result published rather than assumed.

Forty-four AI-related user-agent tokens are worth knowing as of 26 August 2026, and four vendors state in their own documentation that some of their bots ignore robots.txt. Thirty-eight of the forty-four have a first-party vendor page; the rest sit on the community list. The full table, sorted by vendor with the IP list to verify each one, is below.

TL;DR#

• AI bots come in three kinds: training crawlers, search indexers, and user-triggered fetchers. Each vendor documents them separately and lets you allow or block each one independently.
• User-triggered fetchers generally do not honour robots.txt. OpenAI, Perplexity, Google and Meta say so in their own docs. Blocking them is a firewall job, not a robots.txt job.
• Most major vendors now publish an IP list in JSON; Meta gives an AS (Autonomous System) number instead, and Mistral covers only two of its three bots. Verify by IP, never by User-Agent string.
• In my scan of 53 robots.txt files (26 August 2026), only 6 sites fully block any AI bot and 5 publish a Content-Signal line. Two still carry Anthropic's retired anthropic-ai token alongside a current ClaudeBot group.
• A follow-up check on 9 September 2026 found zero change across the same 50 domains and 12 tokens tracked in August. A wider 32-token pass found extra blocks the narrower August check never tested for, which is a deeper look, not evidence of new blocking.
• Three copy-paste presets at the end: allow everything, block training only, block all AI.

What counts as an AI crawler#

An AI crawler is any automated client that fetches web pages on behalf of a large language model product. The name covers three jobs that vendors run as separate bots with separate rules. A training crawler collects text to fit the next model. A search indexer builds the retrieval index that grounds answers with citations. A user-triggered fetcher opens one URL because a person asked the assistant to read it.

The distinction matters because the vendors' own policies differ by job. OpenAI's bots page states that each of its three robots.txt settings "is independent of the others" (OpenAI, Overview of OpenAI crawlers). Google's documentation splits crawlers into "common crawlers", which "always obey robots.txt rules when crawling automatically" (Google Search Central, Google common crawlers), and "user-triggered fetchers", which "generally ignore robots.txt rules" (Google Search Central, user-triggered fetchers).

Allowing a bot is only the first of four preconditions, and this article covers the first one exhaustively. If you have already allowed everything and still see nothing, the fault is usually further down. Why your site doesn't show in ChatGPT and Perplexity walks the failure modes in order. The JavaScript trap covers the most common one: a page that is allowed, fetched, and empty because the bot does not run your JavaScript.

The complete AI crawler list, from vendor docs#

Categories: T = model training, S = search index or retrieval, U = user-triggered fetch, O = other. "Respects robots.txt" repeats the vendor's own statement; "no doc" means I could not find a first-party page and the entry rests on the community list at ai-robots-txt/ai.robots.txt.

The table above is not the ceiling. The community-maintained robots.json at ai-robots-txt already tracks several times as many entries, most of them small or unverified operators, and the count moves with nearly every commit to the repository. Cloudflare's AI Crawl Control reference lists 20 bots it can verify cryptographically or by IP, grouped as AI Crawler, AI Search, AI Assistant and Search Engine (Cloudflare, AI Crawl Control bots).

Which AI bots ignore robots.txt?#

Four bots say in their own documentation that robots.txt may not apply to them, and all four are user-triggered fetchers. OpenAI: for ChatGPT-User, "robots.txt rules may not apply" because actions are user-initiated. Perplexity: Perplexity-User "generally ignores robots.txt rules" (Perplexity, Crawlers). Google: user-triggered fetchers "generally ignore robots.txt rules" because "the fetch was requested by a user". Meta: meta-externalfetcher "May bypass robots.txt rules" (Meta, Web crawlers).

The logic is consistent across vendors: a person pasted your URL into the assistant, so the fetch is treated like a browser visit. If you need to stop those, robots.txt is the wrong tool. Use a WAF rule or Cloudflare's "Block AI bots" setting, which since July 2026 targets three behaviours: Search, Agent and Training. It applies to "Verified bots classified with that behavior, plus additional unverified bots" (Cloudflare, Block AI bots).

Anthropic is the exception among the big vendors. Its page states that "Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt" and names all three bots, Claude-User included (Anthropic, Does Anthropic crawl data from the web).

Is robots.txt enforceable? Not in a technical sense. RFC 9309, the IETF standard behind the Robots Exclusion Protocol, specifies in its introduction that crawlers "are requested to honor" its rules. The same section states plainly that "these rules are not a form of access authorization" (IETF, RFC 9309 Robots Exclusion Protocol, section 1). That also answers whether robots.txt can be ignored: yes, by design, because the standard grants no access control for anyone to enforce.

Does robots.txt still work? For every crawler in the table above whose vendor has published a compliance statement and kept it, yes: compliance there is a policy choice, not a technical constraint the file imposes. For fetchers that say they generally ignore robots.txt, or any crawler that breaks its own policy, the enforceable layer sits at the firewall. Cloudflare's Block AI bots setting or an equivalent WAF rule does that job, not the robots.txt file itself.

Does noindex mean anything to AI crawlers?#

For OpenAI and Anthropic, no. Neither bots page mentions meta robots tags or X-Robots-Tag; both describe robots.txt as the only control. I wrote the opposite in an earlier article on this site and have corrected it there; the honest position is that noindex is documented by exactly two AI-relevant vendors. Apple states that Applebot honours noindex, nosnippet ("blocks description generation and AI model usage for context"), nofollow, none and a scoped X-Robots-Tag: applebot: nosnippet (Apple, About Applebot). Amazon states that Amazonbot honours noarchive, noindex and none (Amazon, Amazonbot).

There is one more subtlety. A page blocked in robots.txt is never fetched, so a noindex on it is never seen by anyone. If your goal is "fetch, but don't quote", robots.txt cannot express it; Apple's nosnippet and the Content-Signal line below are the closest thing to that request.

How to verify a crawler by IP#

A User-Agent string is a free-text header. Anyone scraping your site can send Mozilla/5.0 (compatible; GPTBot/1.1) and inherit whatever allowance you gave OpenAI. Verification means checking the source IP against the vendor's published list, which most major vendors now serve as JSON. The gaps are worth knowing before you write the rule: Meta publishes an AS number rather than a file, and Mistral covers MistralAI-Index and MistralAI-User but not MistralAI-Training.

# scripts/verify-ai-bot.sh
# Usage: ./scripts/verify-ai-bot.sh <ip> <vendor>
# vendor: openai | anthropic | perplexity | google | apple | commoncrawl | mistral | duckduckgo
set -euo pipefail
ip="$1"; vendor="$2"
case "$vendor" in
  openai)      urls="https://openai.com/gptbot.json https://openai.com/searchbot.json https://openai.com/chatgpt-user.json" ;;
  anthropic)   urls="https://claude.com/crawling/bots.json" ;;
  perplexity)  urls="https://www.perplexity.com/perplexitybot.json https://www.perplexity.com/perplexity-user.json" ;;
  google)      urls="https://developers.google.com/static/crawling/ipranges/common-crawlers.json https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json" ;;
  apple)       urls="https://search.developer.apple.com/applebot.json" ;;
  commoncrawl) urls="https://index.commoncrawl.org/ccbot.json" ;;
  mistral)     urls="https://mistral.ai/mistralai-index-ips.json https://mistral.ai/mistralai-user-ips.json" ;;
  duckduckgo)  urls="https://duckduckgo.com/duckassistbot.json" ;;
  *) echo "unknown vendor"; exit 2 ;;
esac
python3 - "$ip" $urls <<'PY'
import ipaddress, json, sys, urllib.request
ip = ipaddress.ip_address(sys.argv[1])
for url in sys.argv[2:]:
    data = json.load(urllib.request.urlopen(url, timeout=15))
    prefixes = []
    def walk(node):
        if isinstance(node, dict):
            for key, value in node.items():
                if key in ("ipv4Prefix", "ipv6Prefix", "prefix", "cidr") and isinstance(value, str):
                    prefixes.append(value)
                else:
                    walk(value)
        elif isinstance(node, list):
            for item in node:
                walk(item)
        elif isinstance(node, str) and "/" in node:
            prefixes.append(node)
    walk(data)
    for prefix in prefixes:
        try:
            if ip in ipaddress.ip_network(prefix, strict=False):
                print(f"MATCH {ip} in {prefix} ({url})")
                sys.exit(0)
        except ValueError:
            continue
print(f"NO MATCH for {ip}")
sys.exit(1)
PY

The script walks any of the vendors' JSON shapes: OpenAI, Anthropic, Perplexity and Google all publish the same nested prefixes[].ipv4Prefix structure, and it exits 0 on a match. Meta publishes no JSON; use whois -h whois.radb.net -- '-i origin AS32934' | grep ^route and match against those routes. Google and Apple additionally support reverse DNS: a genuine Googlebot resolves to *.googlebot.com or *.geo.googlebot.com, a genuine Applebot to *.applebot.apple.com.

Wherever you run, the cheapest place to log candidate requests is the edge or middleware layer your framework already has: middleware.ts on a Next.js app, whatever runs before the route handler elsewhere. Match the User-Agent against the tokens above, write ua, ip and path to the request log, and run the script against the IPs weekly. That log is also the raw material for measuring AI visibility rather than assuming it. How to audit AI visibility sets out the checks and the numbers this site produced when it ran them on itself.

What sites actually do: an August 2026 scan, rechecked in September#

I fetched robots.txt from 65 domains on 26 August 2026. The sample spans four groups: the AI-visibility tool blogs and technical SEO sites that compete with this one, and the frameworks and hosts a developer reads (Vercel, Next.js, MDN, Cloudflare, Kinsta, WP Engine). It also includes a set of publishers and 12 Ukrainian agencies, media and marketplaces. 53 files were readable; 12 returned an HTML shell, a 403 or a bot challenge and were excluded. A "block" below means a group for that exact token with Disallow: /.

Three things stand out. First, the blocking is concentrated in publishers: medium.com blocks six tokens, forbes.com nine, techcrunch.com nine. Every SEO tool blog, every framework site and every Ukrainian agency in the sample allows everything. The only developer-tool site that blocks anything is agent-ready.dev, and it blocks CCBot and Bytespider while explicitly allowing search bots. Second, Content-Signal adoption is an infrastructure story, not an editorial one. Four of the five sites publishing it (Vercel, Kinsta, Cloudflare, Supabase) are hosting or CDN (content delivery network) companies whose managed robots.txt adds the line. Third, techcrunch.com blocks ChatGPT-User and Claude-Web, one of which ignores robots.txt by OpenAI's own account and the other of which no longer exists.

Method and limits: a single fetch per domain with a plain HTTP client. Sites that serve different robots.txt to different user-agents were not tested. The 12 unreadable files skew the sample away from large publishers, who are the most likely blockers. Raw CSV (comma-separated values) is linked at the end.

Two weeks later: what the re-check found#

A second fetch on 9 September 2026 re-read robots.txt on the same domain list with the same method. A plain GET request was parsed into User-agent groups, counting a block only when the exact token had its own group with a Disallow: / line. Of 65 domains attempted, 60 were readable this time against 53 in August, and 50 were readable in both passes; the comparison below covers those 50.

The August scan tracked twelve tokens: Amazonbot, Applebot-Extended, Bytespider, CCBot, ChatGPT-User, Claude-Web, ClaudeBot, GPTBot, Google-Extended, PerplexityBot, anthropic-ai and meta-externalagent. None of the 50 comparable domains changed on any of those twelve between the two scans, and the same five sites (vercel.com, kinsta.com, cloudflare.com, supabase.com and agent-ready.dev) still publish Content-Signal. This is not the only dataset this site publishes end to end. The AEO Adoption Scan does the same for 93 pages ranking on AEO and GEO queries, with its own method and limits stated up front.

Four limits matter for reading that "zero" correctly. Two weeks is too short to call this a trend; zero change over such a short window is the expected result, not a discovery. The wider September sweep (32 tokens instead of 12) is a deeper look with the same method, not proof of new blocking. A token found in September could have been in the August file all along, simply untested. User-agent group counts were logged in both runs but are not compared, because two parses can split one file into a different number of groups for reasons that have nothing to do with edits. The gap in readable domains (60 in September, 53 in August) comes from the User-Agent header the scan itself sent, not from anything the sites changed.

The wider September pass, which checked 32 tokens instead of 12, found blocks at three domains that were not part of the August comparison:

None of these are new blocks. The August comparison tested only twelve tokens, and FacebookBot, GoogleOther, Meta-ExternalAgent, Diffbot, cohere-ai and omgili were not among them. These files could have carried the same rules in August; September is simply the first time this scan checked.

Content-Signal and the IETF draft: preferences per use, not per bot#

The per-bot model does not scale: the community list above already tracks several times as many tokens as this table, and every new product adds one. Two efforts express the preference by use case instead.

Cloudflare's managed robots.txt emits a line such as Content-Signal: search=yes, ai-train=no, use=reference, and the vocabulary behind it has three keys. search covers "building a search index and providing search results". ai-input covers "inputting content into one or more AI models (e.g., retrieval augmented generation, grounding)". ai-train covers "training or fine-tuning AI models". Compliance is voluntary (Cloudflare, Managed robots.txt).

The IETF AI Preferences working group is standardising the same idea. draft-ietf-aipref-vocab-07 (19 August 2026) defines train-ai and search with values y, n or unknown (IETF, draft-ietf-aipref-vocab); its Security Considerations section states plainly that "preferences are not a security mechanism". draft-ietf-aipref-attach-05 defines a Content-Usage HTTP header and a robots.txt rule, for example Content-Usage: train-ai=n for the whole site or Content-Usage: /ai-ok/ train-ai=y for a path (IETF, draft-ietf-aipref-attach). Nothing in either document is enforced by any crawler today, but it costs one line, and the five hosting companies in my scan already ship it.

Three robots.txt presets#

Each preset keeps Sitemap: and the parameter rules this site already uses; add your own Disallow lines for admin paths under the * group.

Preset 1: allow everything, state your preference#

Use this if you want AI citations and accept training. This is what alexturik.com runs.

# public/robots.txt
User-agent: *
Allow: /
 
Content-Signal: search=yes, ai-input=yes, ai-train=yes
 
Sitemap: https://example.com/sitemap.xml

Preset 2: block training, keep search and assistants#

Use this if you want to be cited by ChatGPT search, Perplexity, Claude and Google AI features but not to be in the next training set. The tokens below are the training-only bots and control tokens; search and user-triggered bots stay allowed by the * group.

# public/robots.txt
User-agent: *
Allow: /
 
User-agent: GPTBot
Disallow: /
 
User-agent: ClaudeBot
Disallow: /
 
User-agent: Google-Extended
Disallow: /
 
User-agent: Applebot-Extended
Disallow: /
 
User-agent: meta-externalagent
Disallow: /
 
User-agent: Amazonbot
Disallow: /
 
User-agent: MistralAI-Training
Disallow: /
 
User-agent: CCBot
Disallow: /
 
User-agent: Bytespider
Disallow: /
 
User-agent: cohere-training-data-crawler
Disallow: /
 
User-agent: Timpibot
Disallow: /
 
Content-Signal: search=yes, ai-input=yes, ai-train=no
 
Sitemap: https://example.com/sitemap.xml

Two caveats. meta-externalagent and Amazonbot both train and index, so blocking them also removes you from Meta AI search and Alexa answers; Amazon's Amzn-SearchBot stays allowed and does not train. Applebot-Extended blocks training while leaving Siri and Spotlight search on Applebot.

Preset 3: block all documented AI bots#

Use this only if you have decided AI traffic has no value for you. Remember that the four user-triggered fetchers will still open a page when a person asks; stopping those requires a firewall rule.

# public/robots.txt
User-agent: *
Allow: /
 
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: meta-webindexer
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-Training
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: CCBot
User-agent: Bytespider
User-agent: Diffbot
User-agent: omgili
User-agent: ImagesiftBot
User-agent: YouBot
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Timpibot
Disallow: /
 
Content-Signal: search=no, ai-input=no, ai-train=no
 
Sitemap: https://example.com/sitemap.xml

Stacked User-agent lines in one group are valid under RFC 9309, section 2.2, and every parser listed above supports them (RFC 9309, Robots Exclusion Protocol). Googlebot, bingbot and Applebot are deliberately absent from Preset 3: blocking them removes you from classic search, which is not what "block AI" usually means.

Checking the result#

After deploying, confirm the file parses the way you think it does. Google's robots.txt report in Search Console shows the fetched file and any parse errors. For a specific bot, the quickest check is the group lookup with curl and awk:

# scripts/robots-check.sh
# Usage: ./scripts/robots-check.sh https://example.com GPTBot
set -euo pipefail
site="$1"; bot="$(echo "$2" | tr 'A-Z' 'a-z')"
curl -sL "$site/robots.txt" | awk -v bot="$bot" '
  BEGIN { ing = 0 }
  /^[ \t]*[Uu]ser-[Aa]gent:/ {
    ua = tolower($0); sub(/^[ \t]*user-agent:[ \t]*/, "", ua); gsub(/[ \t\r]+$/, "", ua)
    if (ua == bot) { ing = 1; print "group: " $0 } else if (!prev) { ing = 0 }
    prev = 1; next
  }
  { prev = 0 }
  ing && /^[ \t]*([Dd]isallow|[Aa]llow|[Cc]rawl-delay):/ { print "  " $0 }
'

Run it against your production URL for each token you added. If it prints nothing, the group is missing or misspelled; the most common misspellings in the files I scanned were GPTbot, Claude-bot and Perplexity Bot with a space.

For a browser-side version of the same question, the AEO Checker extension audits the open tab's robots.txt as one of its nine checks. That is the faster way to look at a site that is not yours. Its scope is deliberately narrower than this article: ten tokens, four real-time and six training, collapsed into one pass/warn/fail rather than printed per group. It also still carries anthropic-ai in that list, which the table above marks retired. That is a stale entry I owe the extension, and a fair illustration of how fast a hard-coded bot list ages.

Changelog#

• 2026-08-27: first version. 44 tokens across 38 table rows, 53-site scan, Content-Signal and IETF draft-ietf-aipref-attach-05.
• 2026-09-09: re-scan. Zero change on the 50-domain, 12-token comparable basis (same 5 Content-Signal sites); a wider 32-token pass found untested-in-August blocks at medium.com, techcrunch.com and searchenginejournal.com. Table, IP verification, presets and FAQ core unchanged.

FAQ#

Which AI crawlers ignore robots.txt?#

By their own documentation: ChatGPT-User (OpenAI), Perplexity-User (Perplexity), Google-Agent and other Google user-triggered fetchers, and meta-externalfetcher (Meta). All four are user-triggered fetchers: a person asked the assistant to open your URL, so the vendor treats the request as the user's, not the bot's. Training and search crawlers from the same vendors do honour robots.txt.

Does blocking GPTBot remove my site from ChatGPT search?#

No. OpenAI runs three separate bots with independent rules. GPTBot governs training, OAI-SearchBot governs ChatGPT search answers, and ChatGPT-User fetches pages on a user's request. Blocking GPTBot alone leaves you eligible for search citations; blocking OAI-SearchBot removes you from search answers but not from navigational links.

How do I verify that a request really came from GPTBot or ClaudeBot?#

Compare the request IP with the vendor's published ranges. OpenAI serves openai.com/gptbot.json, Anthropic serves claude.com/crawling/bots.json, Perplexity serves perplexity.com/perplexitybot.json, Google publishes JSON files under developers.google.com/static/crawling/ipranges/. A User-Agent string alone proves nothing: any script can send it.

Is Google-Extended a crawler?#

No. Google-Extended has no user-agent string of its own. It is a robots.txt control token: crawling still happens as Googlebot. A Disallow for Google-Extended tells Google not to use the fetched content to train future Gemini models or for grounding in Gemini Apps and Vertex AI. It does not affect Google Search inclusion.

What is Content-Signal in robots.txt?#

A line Cloudflare's managed robots.txt adds, such as Content-Signal: search=yes, ai-train=no, use=reference. It expresses a preference per use case rather than per bot. Compliance is voluntary, and the IETF is standardising a related vocabulary (train-ai, search) in the AI Preferences working group. In my August 2026 scan 5 of 53 sites published one.

How often should I update the AI crawler list in robots.txt?#

Check quarterly. Between 2024 and 2026 Anthropic retired anthropic-ai and Claude-Web and added Claude-User and Claude-SearchBot; OpenAI added OAI-SearchBot and OAI-AdsBot; Mistral added three bots. Two of the 53 sites I scanned still carry the retired anthropic-ai token. Both block ClaudeBot as well, so the stale line only adds noise, but a file listing anthropic-ai alone would block nothing Anthropic runs today.

Does robots.txt stop AI crawlers?#

It stops the crawlers whose vendor has committed to honoring it, which is most of the training and search bots in the AI crawler user-agent list above. It does not technically stop anything else: RFC 9309 itself says its rules are not a form of access authorization. A crawler that breaks its own policy needs a firewall rule instead.

Can you block AI crawlers?#

Yes, but robots.txt alone will not do it. State a preference with robots.txt and Content-Signal for crawlers that read and honor them, verify traffic by IP, and block anything that fails verification or ignores your rules at the firewall. Cloudflare's Block AI bots setting or an equivalent WAF rule enforces what robots.txt can only request.

Can Cloudflare block AI web crawlers?#

Yes. Cloudflare's Block AI bots setting, live since July 2026, targets three behaviors: Search, Agent and Training. It applies to bots Cloudflare has verified under that behavior, plus additional unverified bots the setting also covers. This works at the firewall layer, so it can stop a crawler even if that crawler ignores robots.txt.

Sources#

• OpenAI, Overview of OpenAI crawlers — the three bots and the statement that each robots.txt setting is independent
• Anthropic, Does Anthropic crawl data from the web — the current three tokens, and the commitment that covers all of them
• Perplexity, Crawlers — PerplexityBot versus Perplexity-User
• Google Search Central, Google common crawlers
• Google Search Central, User-triggered fetchers — where "generally ignore robots.txt rules" comes from
• Google Search Central, Verifying Googlebot — the reverse DNS and IP-range method
• Apple, About Applebot — the one vendor page that documents noindex and nosnippet for an AI crawler
• Meta, Web crawlers
• Amazon, Amazonbot — the other one
• Mistral, Robots
• DuckDuckGo, DuckAssistBot
• Common Crawl, CCBot
• Diffbot, robots.txt FAQ
• You.com, YouBot — robots.txt and crawl-delay compliance, plus the published IP range
• Cloudflare, AI Crawl Control bots reference — the 20 bots Cloudflare can verify
• Cloudflare, Block AI bots — the firewall answer to user-triggered fetchers
• Cloudflare, Managed robots.txt and Content Signals — the Content-Signal vocabulary
• IETF, draft-ietf-aipref-vocab
• IETF, draft-ietf-aipref-attach — the Content-Usage header and robots.txt rule
• IETF, RFC 9309 Robots Exclusion Protocol — section 2.2 on stacked User-agent lines; section 1 (Introduction) on the protocol being a request, not access authorization
• ai-robots-txt community list — a fast-growing, unpinned list, and the only source for the six tokens with no vendor page
• Raw scan data, August baseline — all 65 domains fetched on 26 August 2026, one row each
• Raw scan data, September re-check — the same 65 domains re-fetched on 9 September 2026. Two columns do not compare directly against the August file: blocked_tokens, because September swept 32 tokens against August's 12, so searchenginejournal.com moving from none to omgili is a wider sweep rather than a new block; and status, because August collapsed every failure into FETCH_FAIL while September keeps the cause. The zero-drift finding above is computed on the 12 shared tokens across the 50 domains readable in both runs

Part of the Answer Engine Optimization cluster — the full reading order in dependency sequence, plus a glossary of every term used across these articles.