Posts tagged “robots.txt and robots directives”
Crawler control for AI bots: robots.txt groups, meta robots and X-Robots-Tag, and what a noindex directive actually means to an answer engine.
robots.txt, meta robots and X-Robots-Tag are the three places a site says what a crawler may do, and AI bots read all three differently from search bots. Some honour the file, some ignore it outright, and several fetch under one user agent while the content they collect trains something registered under another. Blocking the wrong token costs you citations; blocking nothing at all costs you the choice.
The posts here start from the list rather than from advice: 44 AI user-agent tokens with their owners and IP ranges, re-scanned in September 2026 with the drift reported either way, and three copy-paste presets for the common positions. The rest cover the directives that remove a page from answer engines while leaving it in Google, and the two configuration mistakes that do it by accident.
3 articles
AI Crawler List 2026: Every User-Agent, robots.txt per Bot
44 AI user-agent tokens with owners and IP lists, which bots ignore robots.txt, three copy-paste presets, and a re-scan showing zero drift in September 2026.
· 24 min read
The 3-Word Tag That Hides Your Best Pages from AI
Three words in `<head>` can hide your best article from ChatGPT, Claude, and Perplexity — even if Google still shows it. The silent AEO killer.
· 10 min read
Why Your Site Doesn't Show Up in ChatGPT (or Perplexity)
Three robots.txt strategies for AI crawlers, and the two configuration mistakes that quietly cut you out of ChatGPT, Claude, and Perplexity answers.
· 12 min read