Posts tagged “robots.txt and robots directives”
Crawler control for AI bots: robots.txt groups, meta robots, X-Robots-Tag, and what noindex means to an answer engine.
3 articles
AI Crawler List 2026: Every User-Agent, robots.txt per Bot
44 AI user-agent tokens with owners and IP lists, which ones ignore robots.txt, three copy-paste robots.txt presets, and a scan of 53 live robots files.
· 19 min read
The 3-Word Tag That Hides Your Best Pages from AI
Three words in `<head>` can hide your best article from ChatGPT, Claude, and Perplexity — even if Google still shows it. The silent AEO killer.
· 10 min read
Why Your Site Doesn't Show Up in ChatGPT (or Perplexity)
Three robots.txt strategies for AI crawlers, and the two configuration mistakes that quietly cut you out of ChatGPT, Claude, and Perplexity answers.
· 12 min read