Posts tagged “robots.txt and robots directives”
Crawler control for AI bots: robots.txt groups, meta robots, X-Robots-Tag, and what noindex means to an answer engine.
3 articles
AI Crawler List 2026: Every User-Agent, robots.txt per Bot
44 AI user-agent tokens with owners and IP lists, which bots ignore robots.txt, three copy-paste presets, and a re-scan showing zero drift in September 2026.
· 24 min read
The 3-Word Tag That Hides Your Best Pages from AI
Three words in `<head>` can hide your best article from ChatGPT, Claude, and Perplexity — even if Google still shows it. The silent AEO killer.
· 10 min read
Why Your Site Doesn't Show Up in ChatGPT (or Perplexity)
Three robots.txt strategies for AI crawlers, and the two configuration mistakes that quietly cut you out of ChatGPT, Claude, and Perplexity answers.
· 12 min read