# Posts tagged “robots.txt and robots directives” > Crawler control for AI bots: robots.txt groups, meta robots, X-Robots-Tag, and what noindex means to an answer engine. 3 articles. ## Articles ### AI Crawler List 2026: Every User-Agent, robots.txt per Bot - **URL:** https://alexturik.com/blog/ai-crawler-list-robots-txt-2026 - **Markdown:** https://alexturik.com/blog/ai-crawler-list-robots-txt-2026.md - **Published:** 2026-08-27 44 AI user-agent tokens with owners and IP lists, which ones ignore robots.txt, three copy-paste robots.txt presets, and a scan of 53 live robots files. ### The 3-Word Tag That Hides Your Best Pages from AI - **URL:** https://alexturik.com/blog/the-3-word-tag-that-hides-your-best-pages-from-ai - **Markdown:** https://alexturik.com/blog/the-3-word-tag-that-hides-your-best-pages-from-ai.md - **Published:** 2026-05-08 · **Updated:** 2026-08-30 Three words in `` can hide your best article from ChatGPT, Claude, and Perplexity — even if Google still shows it. The silent AEO killer. ### Why Your Site Doesn't Show Up in ChatGPT (or Perplexity) - **URL:** https://alexturik.com/blog/why-your-site-doesnt-show-in-chatgpt-and-perplexity - **Markdown:** https://alexturik.com/blog/why-your-site-doesnt-show-in-chatgpt-and-perplexity.md - **Published:** 2026-05-02 · **Updated:** 2026-08-29 Three robots.txt strategies for AI crawlers, and the two configuration mistakes that quietly cut you out of ChatGPT, Claude, and Perplexity answers. All topics: https://alexturik.com/blog/tag