# Oleksii Turovskyi > Full-stack developer from Ukraine specializing in WordPress, Next.js, React, Node.js, and Fastify. Builds rapid MVPs for startups and maintains high-load systems, blending development with marketing insights to ship products that solve real business challenges. ## About - **Name:** Oleksii Turovskyi - **Role:** Full-stack developer (WordPress, Next.js, React, Node.js, Fastify) - **Location:** Ukraine (Kyiv) - **Email:** alexturik@gmail.com - **Website:** https://alexturik.com ## Core expertise - WordPress (custom themes, plugins, headless with WPGraphQL/ACF/Gutenberg, WordPress VIP) - Next.js 16 (App Router, RSC, Server Actions) and React 19 - Node.js, Fastify, REST APIs, GraphQL - TypeScript, JavaScript, PHP 8.3 - WooCommerce (plugins, payment integrations: Dintero, Nets, Klarna) - Marketing/analytics integration: GA4 (with BigQuery sync), GTM (client + server-side), Marketo, HubSpot, Klaviyo - Performance: Core Web Vitals, JSON-LD, SEO - Build/infrastructure: Vite, Docker, CI/CD, Vercel, Google Cloud ## Pages - [Oleksii Turovskyi — Full-stack Developer](https://alexturik.com): Portfolio, intro, projects, experience, contact. Markdown: https://alexturik.com/index.md - [Oleksii Turovskyi](https://alexturik.com/about): Who Oleksii Turovskyi is, what he builds and how he works — the author behind every article on this site. Markdown: https://alexturik.com/about.md - [Writing](https://alexturik.com/blog): Writing on full-stack development, AEO and Core Web Vitals. Markdown: https://alexturik.com/blog.md - [Topics](https://alexturik.com/blog/tag): Index of blog topics with post counts. Markdown: https://alexturik.com/blog/tag.md - [WordPress Plugins](https://alexturik.com/plugins): WordPress plugins shipped by Oleksii Turovskyi (case studies + downloads). Markdown: https://alexturik.com/plugins.md - [Chrome Extensions](https://alexturik.com/extensions): Browser extensions on the Chrome Web Store. Markdown: https://alexturik.com/extensions.md - [Tools](https://alexturik.com/tools): Free browser-only tools — nothing is uploaded, no account needed. Markdown: https://alexturik.com/tools.md - [Embedding Playground](https://alexturik.com/tools/embedding-playground): Compare up to five texts by meaning; a sentence-transformer model runs locally in the browser and returns the full similarity matrix as JSON. Markdown: https://alexturik.com/tools/embedding-playground.md - [Answer Engine Optimization](https://alexturik.com/aeo): Pillar page for the AEO/GEO cluster — the full reading order with each article summarised. Markdown: https://alexturik.com/aeo.md - [AEO Adoption Scan, 2026](https://alexturik.com/aeo/research-2026): The 2026 AEO adoption scan in full — method, signal definitions, results for 93 ranking pages, limits, and how to cite it. Markdown: https://alexturik.com/aeo/research-2026.md - [AEO & GEO Glossary](https://alexturik.com/aeo/glossary): Definitions of the AEO/GEO vocabulary — llms.txt, JSON-LD, embeddings, chunking, grounding, RAG. Markdown: https://alexturik.com/aeo/glossary.md - [Privacy Policy — alexturik AEO Checker](https://alexturik.com/aeo/privacy): Privacy policy for the AEO Checker Chrome extension (no data collected). Markdown: https://alexturik.com/aeo/privacy.md - [Newsletter](https://alexturik.com/newsletter): Subscribe to new articles by email — double opt-in, roughly one article every two weeks. Markdown: https://alexturik.com/newsletter.md - [Profile (Markdown)](https://alexturik.com/profile.md): Full résumé in plain Markdown — preferred for AI ingestion - [CV (PDF)](https://alexturik.com/cv_Oleksii_Turovskyi.pdf): Downloadable CV - [Resume (JSON Resume schema)](https://alexturik.com/resume.json): Machine-readable résumé - [Blog RSS](https://alexturik.com/blog/feed.xml): Atom/RSS feed for posts - [Full text of every article and page](https://alexturik.com/llms-full.txt): This index plus the complete Markdown, in one request - [Sitemap](https://alexturik.com/sitemap.xml) ## Agent tools (WebMCP) In-browser agents (Chrome 149+, WebMCP origin trial) can call structured tools exposed via `document.modelContext` (`navigator.modelContext` is a deprecated shim in current Chromium and warns on access). No tool creates, modifies, or deletes site data: the read tools change nothing at all, and the state-changing ones only navigate the browser or start a file download (non-destructive, safe to repeat). The runtime API does not enumerate its tools to a page scan, so the full descriptor list — names, input and output schemas, safety annotations — is served as JSON at [https://alexturik.com/api/mcp/catalog](https://alexturik.com/api/mcp/catalog) under `tools`. That same catalog lists every navigable page under `pages`, which is exactly what `navigate_to` accepts. Read-only tools (`readOnlyHint: true`): - `list_plugins` — List catalog summaries of ALL of Oleksii Turovskyi's WordPress plugins. Summaries only — to read one plugin's full page, pass its slug to get_plugin. Output: JSON { plugins: [{ slug, title, tagline, description, category, status, version, url, markdownUrl, downloadable }] }. - `list_extensions` — List catalog summaries of ALL of Oleksii Turovskyi's Chrome extensions. Summaries only — to read one extension's full page, pass its slug to get_extension. Output: JSON { extensions: [{ slug, title, tagline, description, category, status, version, webStoreUrl, url, markdownUrl }] }. - `list_blog_posts` — List catalog summaries of ALL blog posts on Oleksii Turovskyi's site. Summaries only — to read a full article, pass its slug to get_blog_post. Output: JSON { posts: [{ slug, title, description, date, tags, url, markdownUrl }] }. - `get_plugin` — Get the full Markdown page (case study + install guide + FAQ) of ONE WordPress plugin, addressed by slug. To browse all plugins or discover slugs, use list_plugins instead. Output: the page's Markdown text. - `get_extension` — Get the full Markdown page of ONE Chrome extension, addressed by slug. To browse all extensions or discover slugs, use list_extensions instead. Output: the page's Markdown text. - `get_blog_post` — Get the full Markdown article of ONE blog post, addressed by slug. To browse all posts or discover slugs, use list_blog_posts instead. Output: the article's Markdown text. - `get_contact_info` — Get contact details and social profile links for Oleksii Turovskyi. Output: JSON { name, email, website, github, linkedin, x }. - `get_resume` — Get Oleksii Turovskyi's résumé. Output: JSON { name, email, telephone, summary, experience[], education[], skills[], url, sameAs[] } — the same document as /resume.json. State-changing tools (browser-side effects only — never site data; non-destructive and idempotent): - `navigate_to` — Navigate the browser to a page on this site. Accepts an internal path like "/plugins/wdpa", "/uk/ai", "/blog/", or "/". Only known internal paths are allowed — the allowlist is the `pages` array plus every `url` in this catalog, so anything listed here is reachable. Changes only the current page — never any data — and is safe to repeat. Output: JSON { ok, path, message }; unknown paths are refused with ok: false. - `download_plugin` (only on `/plugins/`) — Download the WordPress plugin shown on the current page as a .zip. Registered only while a /plugins/ page is open, and takes no input because the plugin is the one being viewed. Triggers a token-gated download in the browser; modifies no site data and is safe to repeat. Output: JSON { ok, path, message }. Declarative tools (visible to a static crawler, no WebMCP runtime needed): - `find_content` — Find a plugin, extension, or blog post by name or slug and navigate the browser to its page; falls back to the /plugins index when nothing matches. Input { q }. Returns no data payload — it is a `
` in the site footer, the one tool a static crawler can see without running the WebMCP API. ## Ukrainian pages - [Answer Engine Optimization](https://alexturik.com/uk/aeo): Ukrainian pillar page for the AEO/GEO cluster — the four preconditions and the reading order. Ukrainian version of https://alexturik.com/aeo. Markdown: https://alexturik.com/uk/aeo.md - [Глосарій AEO та GEO](https://alexturik.com/uk/aeo/glossary): Ukrainian glossary of AEO/GEO vocabulary — llms.txt, JSON-LD, embeddings, chunking, grounding, RAG. Ukrainian version of https://alexturik.com/aeo/glossary. Markdown: https://alexturik.com/uk/aeo/glossary.md - [Олексій Туровський](https://alexturik.com/uk/about): Ukrainian author page — who Oleksii Turovskyi is and how he works. Ukrainian version of https://alexturik.com/about. Markdown: https://alexturik.com/uk/about.md - [Впровадження ШІ в малий і середній бізнес - без агенції і без AI-трансформації](https://alexturik.com/uk/ai): AI integration consulting for Ukrainian small and medium business — what gets automated, how a pilot runs, what is out of scope. Written in Ukrainian, no English equivalent. Markdown: https://alexturik.com/uk/ai.md - [Заявка на консультацію](https://alexturik.com/uk/ai/contact): Consulting enquiry form for the Ukrainian AI-integration section — what to put in it, what happens after, and the plain-email alternative. Written in Ukrainian, no English equivalent. Markdown: https://alexturik.com/uk/ai/contact.md - [Калькулятор вартості автоматизації](https://alexturik.com/uk/ai/calculator): Browser-only calculator for the Ukrainian AI-integration section — monthly token cost, hours saved and net saving for a process, across five models. Written in Ukrainian, no English equivalent. Markdown: https://alexturik.com/uk/ai/calculator.md - [Політика конфіденційності](https://alexturik.com/uk/privacy): Privacy policy for the site — what the newsletter, the contact form, GA4 and the plugin download ledger each collect, and how to have it deleted. Written in Ukrainian, no English equivalent. Markdown: https://alexturik.com/uk/privacy.md - [Список AI-краулерів 2026: user-agent і robots.txt по ботах](https://alexturik.com/uk/aeo/spysok-ai-kraulertiv-2026): 44 токени AI user-agent із власниками й списками IP, які з них ігнорують robots.txt, три готові robots.txt і скан 53 живих файлів за серпень 2026. Ukrainian version of https://alexturik.com/blog/ai-crawler-list-robots-txt-2026. Markdown: https://alexturik.com/uk/aeo/spysok-ai-kraulertiv-2026.md - [Аудит AI-видимості вручну: вісім перевірок без сервісів](https://alexturik.com/uk/aeo/audyt-ai-vydymosti-vruchnu): Ті самі вісім перевірок, що й у платних сервісів AI-видимості, тільки безкоштовно: llms.txt, robots.txt, JSON-LD і реальні цифри ринку за 2026 рік. Ukrainian version of https://alexturik.com/blog/how-to-audit-ai-visibility. Markdown: https://alexturik.com/uk/aeo/audyt-ai-vydymosti-vruchnu.md - [llms.txt для Next.js: як додати його за 15 хвилин](https://alexturik.com/uk/aeo/llms-txt-za-15-hvylyn): Що таке llms.txt, чому цей файл віддають 52,7% сторінок у видачі за AEO-запитами і два способи додати його на Next.js-блог за пʼятнадцять хвилин. Ukrainian version of https://alexturik.com/blog/how-i-added-llms-txt-to-my-nextjs-blog-in-15-minutes. Markdown: https://alexturik.com/uk/aeo/llms-txt-za-15-hvylyn.md - [Три слова в head, які ховають ваші сторінки від AI](https://alexturik.com/uk/aeo/noindex-tag-shcho-hovaie-storinky): Три слова в head здатні сховати найкращу статтю від ChatGPT, Claude і Perplexity, навіть коли Google її ще показує. Тихий убивця AEO і як його знайти. Ukrainian version of https://alexturik.com/blog/the-3-word-tag-that-hides-your-best-pages-from-ai. Markdown: https://alexturik.com/uk/aeo/noindex-tag-shcho-hovaie-storinky.md - [JSON-LD: як пояснити AI, про що ваша сторінка](https://alexturik.com/uk/aeo/json-ld-dlia-ai): Моделі спираються на структуровані дані. З розміткою schema.org модель бачить типізований обʼєкт: автора, дату, заголовок. Канонічне налаштування й перевірка. Ukrainian version of https://alexturik.com/blog/help-ai-understand-your-content-with-json-ld. Markdown: https://alexturik.com/uk/aeo/json-ld-dlia-ai.md ## Writing - [AI Crawler List 2026: Every User-Agent, robots.txt per Bot](https://alexturik.com/blog/ai-crawler-list-robots-txt-2026) ([Markdown](https://alexturik.com/blog/ai-crawler-list-robots-txt-2026.md)) — 44 AI user-agent tokens with owners and IP lists, which ones ignore robots.txt, three copy-paste robots.txt presets, and a scan of 53 live robots files. - [Skip the AI Visibility Checker: Audit Your Site by Hand](https://alexturik.com/blog/how-to-audit-ai-visibility) ([Markdown](https://alexturik.com/blog/how-to-audit-ai-visibility.md)) — Run the same eight checks a paid AI visibility checker runs, for free: llms.txt, robots.txt, JSON-LD, and real 2026 baselines to compare your site against. - [GEO vs SEO vs AEO — What Actually Changes in Your Workflow](https://alexturik.com/blog/geo-vs-seo-vs-aeo) ([Markdown](https://alexturik.com/blog/geo-vs-seo-vs-aeo.md)) — I scanned 93 pages ranking for AEO and GEO queries to see whether the people explaining the difference follow their own advice. Only 25.8% did. - [Embeddings: How AI Search Turns Your Page Into 384 Numbers](https://alexturik.com/blog/embeddings-how-ai-search-turns-your-page-into-numbers) ([Markdown](https://alexturik.com/blog/embeddings-how-ai-search-turns-your-page-into-numbers.md)) — AI search doesn't match keywords - it compares vectors. Turn your text into an embedding, run cosine similarity live, and watch stuffing win the wrong race. - [The Temporal API: Fixing 30 Years of JavaScript Dates](https://alexturik.com/blog/the-temporal-api-fixing-javascript-dates) ([Markdown](https://alexturik.com/blog/the-temporal-api-fixing-javascript-dates.md)) — JavaScript's Date has shipped the same bugs since 1995 — mutation, zero-based months, no real time zones. Temporal fixes them, run every example live. - [How I Added llms.txt to My Next.js Blog in 15 Minutes](https://alexturik.com/blog/how-i-added-llms-txt-to-my-nextjs-blog-in-15-minutes) ([Markdown](https://alexturik.com/blog/how-i-added-llms-txt-to-my-nextjs-blog-in-15-minutes.md)) — How llms.txt works, why 52.7% of pages ranking for AEO and GEO queries now serve one, and two ways to ship it on a Next.js blog in 15 minutes. - [The 3-Word Tag That Hides Your Best Pages from AI](https://alexturik.com/blog/the-3-word-tag-that-hides-your-best-pages-from-ai) ([Markdown](https://alexturik.com/blog/the-3-word-tag-that-hides-your-best-pages-from-ai.md)) — Three words in `` can hide your best article from ChatGPT, Claude, and Perplexity — even if Google still shows it. The silent AEO killer. - [Recency and Authority — Two Fields AI Uses to Trust You](https://alexturik.com/blog/recency-and-authority-two-schema-fields-that-decide-ai-trust) ([Markdown](https://alexturik.com/blog/recency-and-authority-two-schema-fields-that-decide-ai-trust.md)) — Before quoting your article, AI runs two checks — is it fresh, and who wrote it. Both ride on machine-readable signals. Here's the canonical setup. - [How to Help AI Understand Your Content with JSON-LD](https://alexturik.com/blog/help-ai-understand-your-content-with-json-ld) ([Markdown](https://alexturik.com/blog/help-ai-understand-your-content-with-json-ld.md)) — LLMs ground answers in structured data. With schema.org markup, the model sees a typed object — author, date, headline. Here's the canonical setup. - [The Invisible Tags Shaping Your AI Citations](https://alexturik.com/blog/the-invisible-tags-shaping-your-ai-citations) ([Markdown](https://alexturik.com/blog/the-invisible-tags-shaping-your-ai-citations.md)) — title, meta description, og:image, canonical — four tags decide if ChatGPT and Perplexity render your brand on the citation card. The bundle. - [Semantic HTML — Skipping Headings Breaks AI Optimization](https://alexturik.com/blog/semantic-html-for-machines-heading-hierarchy) ([Markdown](https://alexturik.com/blog/semantic-html-for-machines-heading-hierarchy.md)) — AI crawlers parse your heading outline before anything else. Nothing repairs a skipped level — here's what really breaks, and how to keep structure clean. - [The JavaScript Trap — Why AI Bots See Empty Pages](https://alexturik.com/blog/the-javascript-trap-why-ai-bots-see-empty-pages) ([Markdown](https://alexturik.com/blog/the-javascript-trap-why-ai-bots-see-empty-pages.md)) — AI crawlers don't run JavaScript. If your raw HTML is a hollow `
`, you're invisible to ChatGPT, Claude, and Perplexity. - [Why Your Site Doesn't Show Up in ChatGPT (or Perplexity)](https://alexturik.com/blog/why-your-site-doesnt-show-in-chatgpt-and-perplexity) ([Markdown](https://alexturik.com/blog/why-your-site-doesnt-show-in-chatgpt-and-perplexity.md)) — Three robots.txt strategies for AI crawlers, and the two configuration mistakes that quietly cut you out of ChatGPT, Claude, and Perplexity answers. ## Plugins - [A/B Testing for Gutenberg](https://alexturik.com/plugins/ab-testing-gutenberg) ([Markdown](https://alexturik.com/plugins/ab-testing-gutenberg.md)) — WordPress plugin adding an A/B Test block to Gutenberg — cookieless round-robin variants, click and dwell-time tracking, per-test analytics dashboard. - [Yoast Video SEO Booster](https://alexturik.com/plugins/yoast-video-seo-booster) ([Markdown](https://alexturik.com/plugins/yoast-video-seo-booster.md)) — Fix Search Console's 'video is not the main content of the page' error on WordPress + Yoast SEO. Generates video-sitemap.xml and a dedicated watch page per video. - [WP Deep Performance Analyzer](https://alexturik.com/plugins/wdpa) ([Markdown](https://alexturik.com/plugins/wdpa.md)) — Deep WordPress performance profiling — hook analysis, autoloaded options audit, asset graph, HTTP and AJAX monitoring, object-cache diagnostics. ## Extensions - [NotebookLM Mindmap Controls](https://alexturik.com/extensions/notebooklm-mindmap-controls) ([Markdown](https://alexturik.com/extensions/notebooklm-mindmap-controls.md)) — Chrome extension that adds keyboard shortcuts, zoom, and pan controls to NotebookLM's mindmap view — turn a passive diagram into an explorable knowledge graph. - [AEO Checker](https://alexturik.com/extensions/aeo-checker) ([Markdown](https://alexturik.com/extensions/aeo-checker.md)) — Chrome extension that audits the current page for AI-readiness — checks description and title length, robots meta, JSON-LD, llms.txt and surfaces a 0–100 AEO Score. ## Optional - [GitHub](https://github.com/turovskiy) - [LinkedIn](https://linkedin.com/in/alexturik) - [X / Twitter](https://x.com/alexturik) - [CodePen](https://codepen.io/turovskiy) --- # Full text: pages --- # Oleksii Turovskyi > Full-stack developer working across WordPress and Next.js. What I build, how I decide, and why this site is the reference implementation for what I write. **Canonical:** https://alexturik.com/about I build web systems end to end — usually WordPress and Next.js at the same time, and usually at the seam where the two have to agree. This is what I work on and how I go about it. *Українською: [про мене](/uk/about).* ## What I do I am currently Full-Stack Developer at [Mettevo](https://mettevo.com/team/oleksii-turovskyi), working across three things that turn out to be one thing: headless architecture, AI integration into business infrastructure, and agentic SEO. In practice that means enterprise integrations, custom REST APIs and payment gateways on one side, and autonomous agents, custom AI skills and MCP implementations on the other — with the same question underneath both. What has to be true of a system before something else can safely act on it? Two kinds of work keep coming back. The first is getting a product in front of users quickly, where most of the skill is deciding what not to build. The second is the opposite: a system that already carries real traffic and has grown expensive to change, where the skill is finding which coupling is doing the damage and cutting that one. The seam between the two stacks is where I spend the most time. A headless WordPress backend feeding a Next.js frontend is a contract, and contracts drift — usually silently, usually on a Friday. A lot of what I ship is the machinery that makes the drift loud instead. ## How I work I measure before I claim. Most of the writing on this site started as a thing I believed, failed to reproduce, and had to correct — the piece on embeddings changed direction entirely once the numbers came back the opposite way. If something here says a technique works, the article shows what was measured and on what. I write decisions down. This site keeps its architecture decisions in the repository, next to the code they constrain, because the expensive part of a codebase is not the code — it is reconstructing why it looks like that. The same habit shows up in the plugin pages: each one documents the trade-off, not just the feature. I prefer the smallest thing that removes a class of bugs. A build that fails on a too-long meta description is worth more than a checklist asking a reviewer to remember. A test asserting a status code is worth more than a comment explaining why the status matters. ## What this site is It is a portfolio, and it is also the reference implementation for the things I write about. Every AEO technique described in the articles is live here: the machine-readable `.md` mirror of every page, `llms.txt`, structured data, a WebMCP tool surface for in-browser agents. When I claim a technique is cheap to adopt, this site is the evidence — you can read its markdown, its structured data and its source. The [Chrome extension](/extensions/aeo-checker) I ship audits pages for exactly these signals, so the site and the tool check each other. ## Where to find me - **Writing:** the [blog](/blog), and the [AEO cluster](/aeo) if you came for AI search specifically - **Code:** [GitHub](https://github.com/turovskiy) - **Extensions:** [Chrome Web Store](/extensions) - **Work:** [LinkedIn](https://linkedin.com/in/alexturik) - **Email:** [alexturik@gmail.com](mailto:alexturik@gmail.com) The full résumé lives at [/resume.json](/resume.json) as JSON Resume and at [/profile.md](/profile.md) as markdown, both for machines that would rather not parse a page. --- # Writing > Notes on full-stack development — WordPress, Next.js, Core Web Vitals, and how AI search decides which pages it is willing to cite. **Canonical:** https://alexturik.com/blog Long-form notes on things I had to work out in production: how AI crawlers read a page, what Core Web Vitals actually punish, and where WordPress and Next.js meet. ## Posts ### AI Crawler List 2026: Every User-Agent, robots.txt per Bot - **URL:** https://alexturik.com/blog/ai-crawler-list-robots-txt-2026 - **Markdown:** https://alexturik.com/blog/ai-crawler-list-robots-txt-2026.md - **Published:** 2026-08-27 - **Tags:** ai-search, robots-txt, aeo, measurement 44 AI user-agent tokens with owners and IP lists, which ones ignore robots.txt, three copy-paste robots.txt presets, and a scan of 53 live robots files. ### Skip the AI Visibility Checker: Audit Your Site by Hand - **URL:** https://alexturik.com/blog/how-to-audit-ai-visibility - **Markdown:** https://alexturik.com/blog/how-to-audit-ai-visibility.md - **Published:** 2026-08-09 · **Updated:** 2026-08-29 - **Tags:** aeo, ai-search, llms-txt, structured-data Run the same eight checks a paid AI visibility checker runs, for free: llms.txt, robots.txt, JSON-LD, and real 2026 baselines to compare your site against. ### GEO vs SEO vs AEO — What Actually Changes in Your Workflow - **URL:** https://alexturik.com/blog/geo-vs-seo-vs-aeo - **Markdown:** https://alexturik.com/blog/geo-vs-seo-vs-aeo.md - **Published:** 2026-08-09 · **Updated:** 2026-08-29 - **Tags:** aeo, ai-search, seo, structured-data I scanned 93 pages ranking for AEO and GEO queries to see whether the people explaining the difference follow their own advice. Only 25.8% did. ### Embeddings: How AI Search Turns Your Page Into 384 Numbers - **URL:** https://alexturik.com/blog/embeddings-how-ai-search-turns-your-page-into-numbers - **Markdown:** https://alexturik.com/blog/embeddings-how-ai-search-turns-your-page-into-numbers.md - **Published:** 2026-07-27 · **Updated:** 2026-08-29 - **Tags:** embeddings, ai-search, aeo, web-platform AI search doesn't match keywords - it compares vectors. Turn your text into an embedding, run cosine similarity live, and watch stuffing win the wrong race. ### The Temporal API: Fixing 30 Years of JavaScript Dates - **URL:** https://alexturik.com/blog/the-temporal-api-fixing-javascript-dates - **Markdown:** https://alexturik.com/blog/the-temporal-api-fixing-javascript-dates.md - **Published:** 2026-07-01 · **Updated:** 2026-08-29 - **Tags:** javascript, web-platform JavaScript's Date has shipped the same bugs since 1995 — mutation, zero-based months, no real time zones. Temporal fixes them, run every example live. ### How I Added llms.txt to My Next.js Blog in 15 Minutes - **URL:** https://alexturik.com/blog/how-i-added-llms-txt-to-my-nextjs-blog-in-15-minutes - **Markdown:** https://alexturik.com/blog/how-i-added-llms-txt-to-my-nextjs-blog-in-15-minutes.md - **Published:** 2026-05-09 · **Updated:** 2026-08-29 - **Tags:** llms-txt, aeo, nextjs, ai-search, web-platform How llms.txt works, why 52.7% of pages ranking for AEO and GEO queries now serve one, and two ways to ship it on a Next.js blog in 15 minutes. ### The 3-Word Tag That Hides Your Best Pages from AI - **URL:** https://alexturik.com/blog/the-3-word-tag-that-hides-your-best-pages-from-ai - **Markdown:** https://alexturik.com/blog/the-3-word-tag-that-hides-your-best-pages-from-ai.md - **Published:** 2026-05-08 · **Updated:** 2026-08-30 - **Tags:** aeo, ai-search, robots-txt, html, nextjs Three words in `` can hide your best article from ChatGPT, Claude, and Perplexity — even if Google still shows it. The silent AEO killer. ### Recency and Authority — Two Fields AI Uses to Trust You - **URL:** https://alexturik.com/blog/recency-and-authority-two-schema-fields-that-decide-ai-trust - **Markdown:** https://alexturik.com/blog/recency-and-authority-two-schema-fields-that-decide-ai-trust.md - **Published:** 2026-05-07 · **Updated:** 2026-08-29 - **Tags:** aeo, ai-search, structured-data, nextjs Before quoting your article, AI runs two checks — is it fresh, and who wrote it. Both ride on machine-readable signals. Here's the canonical setup. ### How to Help AI Understand Your Content with JSON-LD - **URL:** https://alexturik.com/blog/help-ai-understand-your-content-with-json-ld - **Markdown:** https://alexturik.com/blog/help-ai-understand-your-content-with-json-ld.md - **Published:** 2026-05-06 · **Updated:** 2026-08-29 - **Tags:** aeo, ai-search, structured-data, nextjs LLMs ground answers in structured data. With schema.org markup, the model sees a typed object — author, date, headline. Here's the canonical setup. ### The Invisible Tags Shaping Your AI Citations - **URL:** https://alexturik.com/blog/the-invisible-tags-shaping-your-ai-citations - **Markdown:** https://alexturik.com/blog/the-invisible-tags-shaping-your-ai-citations.md - **Published:** 2026-05-05 · **Updated:** 2026-08-29 - **Tags:** aeo, seo, metadata, nextjs title, meta description, og:image, canonical — four tags decide if ChatGPT and Perplexity render your brand on the citation card. The bundle. ### Semantic HTML — Skipping Headings Breaks AI Optimization - **URL:** https://alexturik.com/blog/semantic-html-for-machines-heading-hierarchy - **Markdown:** https://alexturik.com/blog/semantic-html-for-machines-heading-hierarchy.md - **Published:** 2026-05-04 · **Updated:** 2026-08-29 - **Tags:** aeo, ai-search, html AI crawlers parse your heading outline before anything else. Nothing repairs a skipped level — here's what really breaks, and how to keep structure clean. ### The JavaScript Trap — Why AI Bots See Empty Pages - **URL:** https://alexturik.com/blog/the-javascript-trap-why-ai-bots-see-empty-pages - **Markdown:** https://alexturik.com/blog/the-javascript-trap-why-ai-bots-see-empty-pages.md - **Published:** 2026-05-03 · **Updated:** 2026-08-29 - **Tags:** aeo, ai-search, rendering, nextjs AI crawlers don't run JavaScript. If your raw HTML is a hollow `
`, you're invisible to ChatGPT, Claude, and Perplexity. ### Why Your Site Doesn't Show Up in ChatGPT (or Perplexity) - **URL:** https://alexturik.com/blog/why-your-site-doesnt-show-in-chatgpt-and-perplexity - **Markdown:** https://alexturik.com/blog/why-your-site-doesnt-show-in-chatgpt-and-perplexity.md - **Published:** 2026-05-02 · **Updated:** 2026-08-29 - **Tags:** aeo, seo, robots-txt, ai-search Three robots.txt strategies for AI crawlers, and the two configuration mistakes that quietly cut you out of ChatGPT, Claude, and Perplexity answers. ## What I write about Most of what is here comes out of the same recurring problem: the gap between a page a human finds useful and a page a machine can parse. That gap used to be a search-engine concern with well-known answers. It stopped being well-known when the readers became language models. Three threads run through the archive: - **How machines read pages** — `llms.txt`, JSON-LD, semantic heading structure, embeddings, the meta tags that quietly exclude you from AI answers. Start with [Why your site doesn't show in ChatGPT and Perplexity](/blog/why-your-site-doesnt-show-in-chatgpt-and-perplexity). - **Measuring it rather than guessing** — [How to audit AI visibility](/blog/how-to-audit-ai-visibility) walks the audit this site runs on itself, with the [AEO Checker](/extensions/aeo-checker) extension doing the per-page part. - **The platform underneath** — Next.js, WordPress, Core Web Vitals, and the occasional language feature worth the migration, like [the Temporal API](/blog/the-temporal-api-fixing-javascript-dates). ## How to read it Every post has a machine-readable twin at the same URL with `.md` appended — [this index included](/blog.md). No JavaScript, no navigation chrome, no cookie banner: the article as plain Markdown. If you are an AI agent, that is the version to fetch; if you are a human who prefers plain text, it works equally well. The same applies site-wide. [`/llms.txt`](/llms.txt) is the map, every page declares its own `.md` alternate, and nothing is hidden behind a paywall or a sign-up. Where an article explains a mechanism you can poke at, there is usually a [browser-only tool](/tools) for it — the embeddings piece has a live [playground](/tools/embedding-playground), and the AEO articles have the [AEO Checker](/extensions/aeo-checker) extension. Start from the [AEO pillar page](/aeo) if you want the whole cluster in reading order. ## FAQ ### How often do you publish? Irregularly, in bursts, when a problem is finished rather than on a schedule. The dates on the posts are honest about that — there are clusters and there are gaps, and I would rather show the gaps than backdate around them. ### Can I get these by RSS? Yes — [/blog/feed.xml](/blog/feed.xml) carries every post. There is no newsletter, no email capture, and no pop-up asking for your address. ### Why does every post have a .md version? Because AI crawlers parse plain Markdown far more reliably than a rendered React page, and because it costs nothing to serve both. The HTML page stays canonical for Google; the `.md` twin is a courtesy to anything that would rather skip the DOM. ### Can I republish or quote these posts? Quote freely with a link back. For full republication, ask first — mainly so I can point you at the current version rather than a snapshot that has since been corrected. --- # Topics > Every topic covered across the writing here, with the number of posts under each — from answer engine optimization to the Next.js work behind it. **Canonical:** https://alexturik.com/blog/tag Thirteen topics, ordered by how much has been written about them. A topic needs two posts before it is worth indexing as a page of its own. ## Tags ### Answer Engine Optimization - **URL:** https://alexturik.com/blog/tag/aeo - **Markdown:** https://alexturik.com/blog/tag/aeo.md - **Posts:** 12 How to get a page read, parsed and cited by ChatGPT, Perplexity, Claude and Google AI Overviews. The whole cluster in one place; the reading order lives on the /aeo hub. ### AI Search - **URL:** https://alexturik.com/blog/tag/ai-search - **Markdown:** https://alexturik.com/blog/tag/ai-search.md - **Posts:** 11 How AI crawlers fetch, render and rank pages: what GPTBot, ClaudeBot and PerplexityBot actually see, and how retrieval differs from classic search. ### Next.js - **URL:** https://alexturik.com/blog/tag/nextjs - **Markdown:** https://alexturik.com/blog/tag/nextjs.md - **Posts:** 6 App Router implementations behind every AEO technique on this site: route handlers, metadata API, markdown mirrors, llms.txt. ### Structured Data - **URL:** https://alexturik.com/blog/tag/structured-data - **Markdown:** https://alexturik.com/blog/tag/structured-data.md - **Posts:** 4 JSON-LD for machines: Article, FAQPage, Person, dates and authorship fields, and which of them AI systems read. ### robots.txt and robots directives - **URL:** https://alexturik.com/blog/tag/robots-txt - **Markdown:** https://alexturik.com/blog/tag/robots-txt.md - **Posts:** 3 Crawler control for AI bots: robots.txt groups, meta robots, X-Robots-Tag, and what noindex means to an answer engine. ### SEO - **URL:** https://alexturik.com/blog/tag/seo - **Markdown:** https://alexturik.com/blog/tag/seo.md - **Posts:** 3 Where classic search optimization overlaps with AI search, and where it stops. ### Web Platform - **URL:** https://alexturik.com/blog/tag/web-platform - **Markdown:** https://alexturik.com/blog/tag/web-platform.md - **Posts:** 3 Browser and JavaScript platform features used in the site's live demos: Temporal, in-browser ML, modern HTML. ### Semantic HTML - **URL:** https://alexturik.com/blog/tag/html - **Markdown:** https://alexturik.com/blog/tag/html.md - **Posts:** 2 Heading hierarchy, landmarks and accessible markup as a machine-readability signal. ### llms.txt - **URL:** https://alexturik.com/blog/tag/llms-txt - **Markdown:** https://alexturik.com/blog/tag/llms-txt.md - **Posts:** 2 The llms.txt convention: what it is, who serves one, and how to ship it on Next.js and WordPress. ### Embeddings - **URL:** https://alexturik.com/blog/tag/embeddings - **Markdown:** https://alexturik.com/blog/tag/embeddings.md - **Posts:** 1 · noindex (needs 2) Vector representations of text, cosine similarity, and why keyword stuffing loses in AI retrieval. Companion to the Embedding Playground. ### JavaScript - **URL:** https://alexturik.com/blog/tag/javascript - **Markdown:** https://alexturik.com/blog/tag/javascript.md - **Posts:** 1 · noindex (needs 2) Language-level posts: Temporal API, dates, ES2026. ### Measuring AI Visibility - **URL:** https://alexturik.com/blog/tag/measurement - **Markdown:** https://alexturik.com/blog/tag/measurement.md - **Posts:** 1 · noindex (needs 2) Search Console, server logs and referrers from ChatGPT, Perplexity and Copilot: how to know whether AI actually cites you. ### Metadata and Open Graph - **URL:** https://alexturik.com/blog/tag/metadata - **Markdown:** https://alexturik.com/blog/tag/metadata.md - **Posts:** 1 · noindex (needs 2) Title, description, canonical and og:* tags as AI citation signals. ### Rendering and Performance - **URL:** https://alexturik.com/blog/tag/rendering - **Markdown:** https://alexturik.com/blog/tag/rendering.md - **Posts:** 1 · noindex (needs 2) Server-side vs client-side vs edge rendering for crawlers that do not execute JavaScript; Core Web Vitals. --- # WordPress Plugins > WordPress plugins by Oleksii Turovskyi — performance profiling, Gutenberg A/B testing, Yoast video SEO. Case study and source notes on every page. **Canonical:** https://alexturik.com/plugins Plugins I've built and ship under GPL-2.0-or-later. Each page documents the problem solved, the key architectural choices, and the code highlights that matter. ## Plugins ### A/B Testing for Gutenberg - **URL:** https://alexturik.com/plugins/ab-testing-gutenberg - **Markdown:** https://alexturik.com/plugins/ab-testing-gutenberg.md - **Category:** Editor · **Status:** released · **Version:** 4.0.1 - **Requires:** WordPress 6.7+, PHP 8.1+ - **License:** GPL-2.0-or-later WordPress plugin adding an A/B Test block to Gutenberg — cookieless round-robin variants, click and dwell-time tracking, per-test analytics dashboard. ### Yoast Video SEO Booster - **URL:** https://alexturik.com/plugins/yoast-video-seo-booster - **Markdown:** https://alexturik.com/plugins/yoast-video-seo-booster.md - **Category:** SEO · **Status:** wip · **Version:** 3.0.3 - **Requires:** WordPress 6.0+, PHP 8.0+ - **License:** GPL-2.0-or-later Fix Search Console's 'video is not the main content of the page' error on WordPress + Yoast SEO. Generates video-sitemap.xml and a dedicated watch page per video. ### WP Deep Performance Analyzer - **URL:** https://alexturik.com/plugins/wdpa - **Markdown:** https://alexturik.com/plugins/wdpa.md - **Category:** Performance · **Status:** wip · **Version:** 3.0.4 - **Requires:** WordPress 6.0+, PHP 8.0+ - **License:** GPL-2.0-or-later Deep WordPress performance profiling — hook analysis, autoloaded options audit, asset graph, HTTP and AJAX monitoring, object-cache diagnostics. ## What these plugins have in common All three started as a problem on a client site that no existing plugin solved without a compromise I was not willing to make. That is the bar: I write a plugin when the alternative is a workaround I would have to explain to the next developer. The result is that none of them are general-purpose. Each one does a narrow thing thoroughly, on the assumption that you already know why you need it. ## What each one is for | Plugin | Solves | Status | Requires | |---|---|---|---| | [A/B Testing for Gutenberg](/plugins/ab-testing-gutenberg) | Split-testing blocks without cookies or a cache bypass | Released | WP 6.7+, PHP 8.1+ | | [WP Deep Performance Analyzer](/plugins/wdpa) | Finding *what* is slow, not just *that* it is slow | WIP | WP 6.0+, PHP 8.0+ | | [Yoast Video SEO Booster](/plugins/yoast-video-seo-booster) | Embedded video that Search Console refuses to index | WIP | WP 6.0+, PHP 8.0+, Yoast SEO | **A/B Testing for Gutenberg** adds a real A/B Test block to the block editor. The hard part is not the split — it is splitting behind full-page cache. Most A/B plugins set a cookie, which makes every response uncacheable and quietly undoes the performance work the site paid for. This one assigns variants round-robin at render with no cookie and no session, so WP Rocket, LiteSpeed and an object cache all keep working. **WP Deep Performance Analyzer** is twelve profiling tabs: PHP lifecycle checkpoints with time and memory deltas, the top-20 autoloaded options and total autoload weight, a `$wp_filter` dump grouped by core / plugin / theme / mu-plugin, and a log of every outbound HTTP and AJAX call. It answers the question Query Monitor leaves you with — *which* of these 90 plugins is actually costing the 400ms. **Yoast Video SEO Booster** gets embedded YouTube and Vimeo out of Search Console's "video is not the main content of the page" bucket. It registers a `/video-sitemap.xml` as a Yoast Sitemap Provider, emits `VideoObject` JSON-LD, and generates a dedicated watch page per video so the URL Google indexes has the video as its only content. ## Why token-gated download instead of the WordPress.org repository These are distributed as signed, short-lived download links from this site rather than through the plugin directory. Two reasons, and neither is secrecy — they are GPL-2.0-or-later, and the source is available on request. The first is that the directory's review queue and guidelines are tuned for plugins aimed at end users, and these are aimed at developers. The second is that updates ship through [plugin-update-checker](https://github.com/YahnisElsts/plugin-update-checker) against my own update server, which means I can push a fix the same day rather than waiting on a review. ## What each page contains Every plugin page is a case study rather than a feature list: the problem as it appeared on a real site, the architectural choice that solved it, the trade-off that choice cost, and the code that matters. If you are evaluating whether to use one, the page is written to let you decide without installing it. ## Related If the problem is smaller than a plugin, it may already be a [browser-only tool](/tools) or a [Chrome extension](/extensions) — the same checks, without installing anything into a WordPress site. ## FAQ ### Are these plugins free? Yes, all three ship under GPL-2.0-or-later at no cost. The download is token-gated to keep the update channel honest, not to charge for it — you get a signed link on the plugin's page without an account. ### Why are two of them marked WIP? WIP means the plugin runs in production on sites I maintain but the public documentation, upgrade path, or edge-case handling is not finished. Released means I would hand it to another developer without a caveat. The status on each card is the honest one, not a marketing label. ### How do updates reach an installed plugin? Through plugin-update-checker, which polls my update server rather than the WordPress.org directory. The plugin's `Update URI` header points there, so `wp-admin → Plugins` shows updates the same way it does for directory plugins. ### Can I see the source before installing? Yes — ask, and I will send the repository. The plugins are GPL-licensed, so the source travels with the zip in any case; the request just saves you the download. --- # Chrome Extensions > Chrome extensions by Oleksii Turovskyi — AEO Checker for AI-readiness audits and NotebookLM Mindmap Controls. Free, one click from the Web Store. **Canonical:** https://alexturik.com/extensions Browser extensions I've shipped to the Chrome Web Store. Free, one-click install — each page is a quick rundown of what the extension does and how to use it. ## Extensions ### NotebookLM Mindmap Controls - **URL:** https://alexturik.com/extensions/notebooklm-mindmap-controls - **Markdown:** https://alexturik.com/extensions/notebooklm-mindmap-controls.md - **Chrome Web Store:** https://chromewebstore.google.com/detail/notebooklm-mindmap-contro/naaecldchbmeohkglalmdhbaglahogdm - **Category:** Productivity · **Status:** released · **Version:** 4.0.1 Chrome extension that adds keyboard shortcuts, zoom, and pan controls to NotebookLM's mindmap view — turn a passive diagram into an explorable knowledge graph. ### AEO Checker - **URL:** https://alexturik.com/extensions/aeo-checker - **Markdown:** https://alexturik.com/extensions/aeo-checker.md - **Chrome Web Store:** https://chromewebstore.google.com/detail/alexturik-aeo-checker/ecnldkocbbhcclhgaenaciioihogbebe - **Category:** SEO · **Status:** released · **Version:** 0.2.2 Chrome extension that audits the current page for AI-readiness — checks description and title length, robots meta, JSON-LD, llms.txt and surfaces a 0–100 AEO Score. ## What these extensions have in common Both extensions here started the same way: a check I was doing by hand often enough that doing it by hand became the bottleneck. Neither is a product with a roadmap. Each one automates a specific inspection I already knew how to do manually, on the page I happen to be looking at. That origin shows up in the design. There is no dashboard, no account, and no sync. You click the icon, you get the answer for the current tab, you close the popup. Nothing persists, because nothing needs to. ## What each one is for | Extension | Answers the question | Runs on | |---|---|---| | [AEO Checker](/extensions/aeo-checker) | Will an AI crawler understand this page? | Any page you open it on | | [NotebookLM Mindmap Controls](/extensions/notebooklm-mindmap-controls) | How do I navigate this mindmap without fighting it? | notebooklm.google.com | **AEO Checker** reads the active tab's HTML and headers and scores the signals that decide whether ChatGPT, Perplexity, Claude and Google's AI Overviews can parse and cite the page — title and description length, robots directives, JSON-LD, `llms.txt`, heading structure. It is the tool version of the audit described in [How to audit AI visibility](/blog/how-to-audit-ai-visibility), and this site is its reference implementation. **NotebookLM Mindmap Controls** adds the zoom, pan and expand/collapse controls that Google's mindmap view does not ship with. Narrow by design: it does one thing to one product's one view. ## Why Chrome Web Store and not a download Every extension here is published on the Chrome Web Store rather than offered as an unpacked zip. That is a deliberate trade. The Store review process is slow and occasionally arbitrary, but it gives you three things a direct download cannot: a signed package, automatic updates, and a permissions list Google has actually looked at. The [plugins](/plugins) section works the opposite way — token-gated direct download — because WordPress plugins have no equivalent review gate, and the audience there is developers who expect a zip. If you want the same kind of check without installing anything, the [tools](/tools) run entirely in a tab. ## Permissions, briefly Neither extension asks for broad host permissions. AEO Checker uses `activeTab` and `scripting`, which means it can only read a page after you explicitly click its icon on that page — it cannot watch your browsing. Nothing is transmitted; the full detail is in the [AEO Checker privacy policy](/aeo/privacy). ## FAQ ### Are these extensions free? Yes, both are free on the Chrome Web Store, with no paid tier and no in-app purchase. They exist because I needed them, and publishing costs me nothing beyond the one-time developer registration. ### Do they work in Edge or Brave? Yes. Both are Chromium browsers and install Chrome Web Store extensions directly. Firefox is not supported — it uses a different extension packaging format, and neither extension has enough Firefox demand to justify a port. ### Does AEO Checker send the pages I visit anywhere? No. It reads the active tab's HTML in your browser, computes the score locally, shows it in the popup, and discards it when the popup closes. There is no remote server, no analytics, and no account. See the [privacy policy](/aeo/privacy) for the exact detail. ### How often are they updated? When something breaks or a signal changes — AEO Checker tracks a moving target, since crawler behaviour and Google's guidance both shift. Each extension page lists its full release history with dates, so you can see the real cadence rather than a claim about it. --- # Tools > Small, free, browser-only tools by Oleksii Turovskyi. No sign-up, no upload, no API key — everything runs locally in the tab you opened it in. **Canonical:** https://alexturik.com/tools Small things that run entirely in your browser. Nothing is uploaded and nothing needs an account. ## Tools ### Embedding Playground - **URL:** https://alexturik.com/tools/embedding-playground - **Markdown:** https://alexturik.com/tools/embedding-playground.md Compare up to five texts by meaning. A sentence transformer runs locally in your browser and gives you the full similarity matrix as JSON. ## What counts as a tool here A tool on this site has to clear one bar: it has to be useful without an account, without an upload, and without me running a server for it. If a thing needs a backend to work, it belongs in a plugin or an extension instead — those sections exist precisely because some problems cannot be solved inside a tab. That constraint is not modesty about scale. It is a privacy guarantee I can actually keep. A tool that never sends your text anywhere cannot leak it, cannot retain it, and cannot quietly start training on it after a terms-of-service update. The only way to make that claim credible is to have no server in the first place. ## Why browser-only, and what it costs Running everything client-side buys three things and costs one. | | Browser-only | Server-backed | |---|---|---| | Your text leaves the machine | Never | Every request | | Works offline after first load | Yes | No | | Cost to run at scale | Zero | Per-request | | First use | Slow — model downloads once | Fast | The cost is that first load. A sentence-transformer model is tens of megabytes, and no amount of engineering makes that instant on a cold cache. So the tools here download on a click, never on page load — you should not pay for a model you did not ask for. After that first download the browser caches it and every subsequent run is local and immediate. ## How these relate to the plugins and extensions The three sections answer three different shapes of problem: - **Tools** — you have a question right now, about text you have right now. Open a tab, get an answer, close the tab. - **[Chrome Extensions](/extensions)** — you want the answer about *the page you are looking at*, repeatedly, without leaving it. - **[WordPress Plugins](/plugins)** — you want the answer built into a site you operate, running on a schedule rather than on demand. The [AEO Checker](/extensions/aeo-checker) extension and the tools here share a lot of thinking about how machines read a page; they differ mainly in whether the input is *your* text or *the current tab*. ## FAQ ### Do these tools upload my text anywhere? No. Every tool listed on this page runs entirely in your browser tab. There is no API call carrying your text, no analytics event containing it, and no server that could store it — the site has no backend endpoint for tool input at all. ### Do I need an account or an API key? No. There is no sign-up, no key, and no rate limit, because there is nothing to meter. The tools cost me nothing to run, which is the whole reason they can be unconditionally free. ### Why is the first run slow? The first run downloads the machine-learning model into your browser's cache — that is the one-time cost of not having a server. Every run after that is local and takes milliseconds. The download starts when you click, never on page load. ### Will there be more tools? Yes, as specific problems earn one. The index stays deliberately small: a tool gets built when I hit the same question enough times that a tab is faster than a script, not to fill out a catalog. --- # Embedding Playground > Compare up to five texts by meaning. A sentence transformer runs in your browser — no upload, no API key — and returns the full similarity matrix. **Canonical:** https://alexturik.com/tools/embedding-playground Put in up to five texts and get the similarity between every pair. A sentence transformer runs in this tab and turns each one into 384 numbers; comparing meaning is then just comparing directions. Nothing is uploaded — the model downloads once and everything after that happens on your machine. ## How to read the similarity matrix Every pair of texts gets a number between roughly -1 and 1. In practice, with this model, you will see almost everything land between 0 and 1, because the model was trained on natural language and natural language sentences are rarely true opposites. | Score | What it usually means | |---|---| | 0.85 – 1.00 | Near-paraphrase — same claim, different words | | 0.60 – 0.85 | Same topic, different point | | 0.35 – 0.60 | Related field, genuinely different subject | | below 0.35 | Unrelated | Those bands are rules of thumb for *this* model, not universal constants. The useful move is comparative, not absolute: run two candidate headings against the question you want to rank for and see which scores higher. The gap between them is trustworthy even when the absolute number is not. ## What is actually happening The model is `all-MiniLM-L6-v2`, a sentence transformer that maps any text to 384 numbers — a point in 384-dimensional space. Texts that mean similar things land in similar directions. Comparing meaning then reduces to comparing the angle between two vectors, which is what cosine similarity measures. Length barely matters, because the vectors are normalised before comparison. A three-word phrase and a three-sentence paragraph making the same point will score high against each other. That is the property that makes embeddings useful for retrieval and the reason AI search does not need your exact keyword to find you. The full walkthrough, with the maths and the counter-intuitive results, is in [How AI search turns your page into 384 numbers](/blog/embeddings-how-ai-search-turns-your-page-into-numbers). ## What to use it for - **Testing a heading against a real query.** Paste the question you want to rank for and two or three candidate headings. Pick the one that scores highest — that is roughly the signal a retrieval system sees. - **Finding cannibalisation.** Paste the intros of two of your own pages. Above ~0.9 they are competing for the same intent and should probably be one page. - **Checking a summary.** Paste an article's opening paragraph and your meta description. A low score means the description promises something the page does not deliver. - **Watching keyword stuffing fail.** Add the same keyword five more times and re-run. The result is not what most SEO advice predicts, which is the point of the explainer. ## Limits worth knowing This is a small, fast, English-first model — the trade for running it in a browser tab. It handles other languages, but noticeably worse. It has no knowledge of your domain, so jargon-heavy text scores less reliably than plain prose. And it is not the model any specific search engine uses; it is a well-behaved stand-in that demonstrates the same mechanism. For a five-text comparison that is fine. For production retrieval, use a current hosted embedding model and evaluate it on your own data. ## FAQ ### Is my text uploaded anywhere? No. The model downloads to your browser once, then every comparison runs locally in the tab. Your text never leaves the machine — there is no API call carrying it and no server that could log it. ### Why does the first run take a while? It is downloading the model, about 25 MB, into your browser cache. That happens on your first click, never on page load, so you do not pay for it unless you use the tool. Every run after that is local and near-instant. ### Which model does this use? `all-MiniLM-L6-v2`, a 384-dimension sentence transformer, running through Transformers.js and WebAssembly. It is small enough to ship to a browser and good enough to make the mechanism visible, which is the goal here. ### Is cosine similarity the same as relevance? No, and conflating them is the common mistake. Cosine similarity measures whether two texts are *about* the same thing. A real ranking system also weighs authority, freshness, intent and dozens of other signals. Similarity is one input, not the verdict. ### Can I compare more than five texts? Not in this tool. Five texts already produce ten pairs, which is about the limit of what a matrix stays readable at. For larger batches, run the same model locally with Transformers.js — the explainer post has the code. --- # Answer Engine Optimization > How to get a page read, understood and cited by ChatGPT, Perplexity, Claude and AI Overviews — the whole AEO and GEO cluster in reading order. **Canonical:** https://alexturik.com/aeo Eleven articles on one question: what has to be true of a page before an AI answer engine will quote it. This is the reading order, and what each part is for. ## Every article in the cluster ### AI Crawler List 2026: Every User-Agent, robots.txt per Bot - **URL:** https://alexturik.com/blog/ai-crawler-list-robots-txt-2026 - **Markdown:** https://alexturik.com/blog/ai-crawler-list-robots-txt-2026.md - **Published:** 2026-08-27 44 AI user-agent tokens with owners and IP lists, which ones ignore robots.txt, three copy-paste robots.txt presets, and a scan of 53 live robots files. ### Skip the AI Visibility Checker: Audit Your Site by Hand - **URL:** https://alexturik.com/blog/how-to-audit-ai-visibility - **Markdown:** https://alexturik.com/blog/how-to-audit-ai-visibility.md - **Published:** 2026-08-09 · **Updated:** 2026-08-29 Run the same eight checks a paid AI visibility checker runs, for free: llms.txt, robots.txt, JSON-LD, and real 2026 baselines to compare your site against. ### GEO vs SEO vs AEO — What Actually Changes in Your Workflow - **URL:** https://alexturik.com/blog/geo-vs-seo-vs-aeo - **Markdown:** https://alexturik.com/blog/geo-vs-seo-vs-aeo.md - **Published:** 2026-08-09 · **Updated:** 2026-08-29 I scanned 93 pages ranking for AEO and GEO queries to see whether the people explaining the difference follow their own advice. Only 25.8% did. ### Embeddings: How AI Search Turns Your Page Into 384 Numbers - **URL:** https://alexturik.com/blog/embeddings-how-ai-search-turns-your-page-into-numbers - **Markdown:** https://alexturik.com/blog/embeddings-how-ai-search-turns-your-page-into-numbers.md - **Published:** 2026-07-27 · **Updated:** 2026-08-29 AI search doesn't match keywords - it compares vectors. Turn your text into an embedding, run cosine similarity live, and watch stuffing win the wrong race. ### How I Added llms.txt to My Next.js Blog in 15 Minutes - **URL:** https://alexturik.com/blog/how-i-added-llms-txt-to-my-nextjs-blog-in-15-minutes - **Markdown:** https://alexturik.com/blog/how-i-added-llms-txt-to-my-nextjs-blog-in-15-minutes.md - **Published:** 2026-05-09 · **Updated:** 2026-08-29 How llms.txt works, why 52.7% of pages ranking for AEO and GEO queries now serve one, and two ways to ship it on a Next.js blog in 15 minutes. ### The 3-Word Tag That Hides Your Best Pages from AI - **URL:** https://alexturik.com/blog/the-3-word-tag-that-hides-your-best-pages-from-ai - **Markdown:** https://alexturik.com/blog/the-3-word-tag-that-hides-your-best-pages-from-ai.md - **Published:** 2026-05-08 · **Updated:** 2026-08-30 Three words in `` can hide your best article from ChatGPT, Claude, and Perplexity — even if Google still shows it. The silent AEO killer. ### Recency and Authority — Two Fields AI Uses to Trust You - **URL:** https://alexturik.com/blog/recency-and-authority-two-schema-fields-that-decide-ai-trust - **Markdown:** https://alexturik.com/blog/recency-and-authority-two-schema-fields-that-decide-ai-trust.md - **Published:** 2026-05-07 · **Updated:** 2026-08-29 Before quoting your article, AI runs two checks — is it fresh, and who wrote it. Both ride on machine-readable signals. Here's the canonical setup. ### How to Help AI Understand Your Content with JSON-LD - **URL:** https://alexturik.com/blog/help-ai-understand-your-content-with-json-ld - **Markdown:** https://alexturik.com/blog/help-ai-understand-your-content-with-json-ld.md - **Published:** 2026-05-06 · **Updated:** 2026-08-29 LLMs ground answers in structured data. With schema.org markup, the model sees a typed object — author, date, headline. Here's the canonical setup. ### The Invisible Tags Shaping Your AI Citations - **URL:** https://alexturik.com/blog/the-invisible-tags-shaping-your-ai-citations - **Markdown:** https://alexturik.com/blog/the-invisible-tags-shaping-your-ai-citations.md - **Published:** 2026-05-05 · **Updated:** 2026-08-29 title, meta description, og:image, canonical — four tags decide if ChatGPT and Perplexity render your brand on the citation card. The bundle. ### Semantic HTML — Skipping Headings Breaks AI Optimization - **URL:** https://alexturik.com/blog/semantic-html-for-machines-heading-hierarchy - **Markdown:** https://alexturik.com/blog/semantic-html-for-machines-heading-hierarchy.md - **Published:** 2026-05-04 · **Updated:** 2026-08-29 AI crawlers parse your heading outline before anything else. Nothing repairs a skipped level — here's what really breaks, and how to keep structure clean. ### The JavaScript Trap — Why AI Bots See Empty Pages - **URL:** https://alexturik.com/blog/the-javascript-trap-why-ai-bots-see-empty-pages - **Markdown:** https://alexturik.com/blog/the-javascript-trap-why-ai-bots-see-empty-pages.md - **Published:** 2026-05-03 · **Updated:** 2026-08-29 AI crawlers don't run JavaScript. If your raw HTML is a hollow `
`, you're invisible to ChatGPT, Claude, and Perplexity. ### Why Your Site Doesn't Show Up in ChatGPT (or Perplexity) - **URL:** https://alexturik.com/blog/why-your-site-doesnt-show-in-chatgpt-and-perplexity - **Markdown:** https://alexturik.com/blog/why-your-site-doesnt-show-in-chatgpt-and-perplexity.md - **Published:** 2026-05-02 · **Updated:** 2026-08-29 Three robots.txt strategies for AI crawlers, and the two configuration mistakes that quietly cut you out of ChatGPT, Claude, and Perplexity answers. *Українською: [Answer Engine Optimization](/uk/aeo) — the four preconditions, the reading order, and what the August 2026 scan found on Ukrainian domains.* ## What AEO actually is Answer Engine Optimization is what you do so that a language model can read your page, understand what it claims, and quote it correctly in an answer it gives to someone who never sees your site. That last clause is the uncomfortable part. Classic SEO ends with a click; AEO often ends with a citation and no click at all. The work is still worth doing — being the source an answer is built from is a different and more durable position than being the tenth blue link — but it changes what you optimise for. You are no longer competing for attention. You are competing to be *quotable*. ## AEO, GEO, LLMO — is there a difference? Mostly not, and the differences that exist are about emphasis rather than technique. | Term | Stands for | Emphasis | |---|---|---| | **AEO** | Answer Engine Optimization | Being the source a direct answer is built from | | **GEO** | Generative Engine Optimization | Being represented well in generated text | | **LLMO** | Large Language Model Optimization | Model-facing framing of the same work | | **AI SEO** | — | Umbrella marketing term for all of it | Anyone selling you a methodology that only works under one of these labels is selling you a label. The full argument, with where the three genuinely diverge, is in [GEO vs SEO vs AEO](/blog/geo-vs-seo-vs-aeo). ## The four things that have to be true Every article below is one of four preconditions. They are ordered, because a failure at any level makes the ones after it irrelevant. **1 — The crawler can fetch it.** If robots.txt or a WAF blocks the agent, or the page returns a shell, nothing else matters. **2 — The crawler can read it.** HTML that arrives without JavaScript, no `noindex`, real text rather than a rendering promise. **3 — The machine can parse the structure.** A clean heading outline, JSON-LD that says who wrote this and when, metadata that is not contradicting itself. **4 — A retriever can use the chunk.** Sections that stand alone, because a section is what gets retrieved — never the whole page. Most sites fail at 2 or 4 while spending all their effort on 3. ## Reading order ### Start here — is anything reaching the crawler at all? - [Why your site doesn't show in ChatGPT and Perplexity](/blog/why-your-site-doesnt-show-in-chatgpt-and-perplexity) — which bots exist, what each is for, and the robots.txt strategies that keep you visible without donating to a training corpus. - [AI crawler list 2026: every user-agent, robots.txt per bot](/blog/ai-crawler-list-robots-txt-2026) — the reference table behind that strategy: 44 tokens with owner, job and IP list, which four vendors ignore robots.txt, and three presets you can paste. - [The JavaScript trap: why AI bots see empty pages](/blog/the-javascript-trap-why-ai-bots-see-empty-pages) — AI crawlers do not run your JavaScript. What each rendering mode actually hands them. - [The 3-word tag that hides your best pages from AI](/blog/the-3-word-tag-that-hides-your-best-pages-from-ai) — `noindex`, how it gets into production by accident, and the `X-Robots-Tag` version you cannot see in the HTML. ### Then — can the machine parse what it fetched? - [Semantic HTML for machines: heading hierarchy](/blog/semantic-html-for-machines-heading-hierarchy) — why a styled `div` is invisible to a chunker, and what each heading level claims. - [Help AI understand your content with JSON-LD](/blog/help-ai-understand-your-content-with-json-ld) — Article and FAQPage in one `@graph`, and how to verify it. - [Recency and authority: two schema fields that decide AI trust](/blog/recency-and-authority-two-schema-fields-that-decide-ai-trust) — `datePublished`, `dateModified`, and why an author string is not an author. - [The invisible tags shaping your AI citations](/blog/the-invisible-tags-shaping-your-ai-citations) — title, description, canonical, Open Graph: the four lines the citation card is built from. ### Then — will a retriever pick your chunk? - [Embeddings: how AI search turns your page into numbers](/blog/embeddings-how-ai-search-turns-your-page-into-numbers) — the mechanism underneath all of it, with a live [playground](/tools/embedding-playground) so you can watch keyword stuffing win retrieval and lose the citation. ### Then — make yourself easy to ingest - [How I added llms.txt to my Next.js blog in 15 minutes](/blog/how-i-added-llms-txt-to-my-nextjs-blog-in-15-minutes) — the machine-readable index, and the `.md` mirrors that go with it. ### Finally — measure it - [How to audit AI visibility](/blog/how-to-audit-ai-visibility) — the audit this site runs on itself, and the numbers it produced. - [GEO vs SEO vs AEO](/blog/geo-vs-seo-vs-aeo) — where the three disciplines actually diverge, and where they are the same work under different names. ## The numbers behind all of it Several articles above quote the same figures — 52.7% serving `llms.txt`, 25.8% with FAQ markup, 8.6% with a markdown twin. They come from one scan, and it has its own page rather than a method section repeated in each article. - [AEO Adoption Scan, 2026](/aeo/research-2026) — seven queries, 93 ranking pages, nine signals, measured 9 August 2026. Method, full results, the limits of what a presence check can show, and how to cite it. The raw data is served as [JSON](/aeo/research-2026.json). ## Tools that do this work for you - [AEO Checker](/extensions/aeo-checker) — a Chrome extension that scores the current tab on the parseability checks above: title and description length, robots directives, JSON-LD, `llms.txt`, heading structure. - [Embedding Playground](/tools/embedding-playground) — compare texts by meaning in your browser, to see what a retriever sees. - The [glossary](/aeo/glossary) — every term used across the cluster, defined once. ## FAQ ### What is Answer Engine Optimization? The practice of making a page readable, parseable and quotable by AI answer engines — ChatGPT, Perplexity, Claude, Google's AI Overviews — rather than merely rankable by a search engine. In practice it is four things: the crawler can fetch the page, it arrives as real HTML, its structure and metadata are machine-readable, and each section makes sense read alone. ### Is AEO different from SEO? The techniques overlap heavily; the success condition does not. SEO optimises for a ranking that produces a click. AEO optimises for being the source a generated answer is built from, which frequently produces a citation and no click. Clean HTML, fast pages and structured data serve both. Keyword density and link-building serve neither as much as their advocates claim. ### Where should I start if I only fix one thing? Check that a crawler receives your content at all. Disable JavaScript and reload your most valuable page: if the article body vanishes, nothing further down the list matters. That single check catches more real failures than every structured data refinement combined. ### How do I measure whether any of this worked? Google Search Console still shows impressions from AI surfaces, server logs show which bots actually fetch you, and per-page parseability can be checked directly with the [AEO Checker](/extensions/aeo-checker) extension. The method, and the numbers it produced for this site, are in [How to audit AI visibility](/blog/how-to-audit-ai-visibility). --- # AEO Adoption Scan, 2026 > What 93 pages ranking for AEO and GEO queries actually ship: llms.txt, JSON-LD, FAQPage, markdown variants and AI-bot blocks, measured August 2026. **Canonical:** https://alexturik.com/aeo/research-2026 Seven queries, 93 pages, nine signals, one day in August 2026. The method, the full results, and what the numbers cannot tell you — so the articles that quote them do not each carry their own copy. Three articles on this site quote numbers from one scan. This is the scan: what was collected, how each signal was defined, what came out, and what the numbers cannot tell you. If you want to cite a figure, cite this page — the articles link here rather than repeating the method. ## What was measured On 9 August 2026 I pulled the top 20 United States, English-language Google organic results for seven queries, using the DataForSEO SERP API: `llms.txt` · `answer engine optimization` · `generative engine optimization` · `how to get cited by ai` · `geo vs seo` · `aeo vs seo` · `ai visibility checker` That returned 123 URLs. 117 answered with a `200`. After dropping platform domains — YouTube, Reddit, LinkedIn, Instagram, Medium, Wikipedia and X, none of which control their own head markup in a way that means anything here — **93 pages across 78 domains** remained. Every number below is out of those 93 unless it says otherwise. These are pages that already rank for the queries where AEO is discussed. That makes them a useful sample of what the field practises, and a poor sample of the web at large. ## How each signal was defined A number is only as good as the check behind it, so here are the checks. | Signal | Counted when | |---|---| | `llms_txt` | `GET /llms.txt` returns 200 with a non-HTML content type | | `ai_bot_blocked` | robots.txt has a named group for the bot containing `Disallow: /` | | `faqpage` | `@type FAQPage` appears in any JSON-LD block on the ranking URL | | `md_variant` | `GET .md` returns 200 with a markdown content type | | `question_heading` | an `h2` or `h3` whose text ends with `?` | Everything else is a plain presence check on the fetched HTML: a JSON-LD block, a `dateModified` field, an `author` field, a ``. ## Results | Signal | Pages | Share | |---|---|---| | JSON-LD of any type | 76 / 93 | 81.7% | | `dateModified` present | 55 / 93 | 59.1% | | `author` in JSON-LD | 58 / 93 | 62.4% | | At least one question heading | 69 / 93 | 74.2% | | `llms.txt` served | 49 / 93 | 52.7% | | At least one comparison table | 34 / 93 | 36.6% | | `FAQPage` markup | 24 / 93 | 25.8% | | Markdown variant of the page | 8 / 93 | 8.6% | | Blocks at least one AI bot | 8 / 93 | 8.6% | | `HowTo` markup | 3 / 93 | 3.2% | Two figures are not page counts. Across all 2,195 headings in the sample, **410 end in a question mark — 18.7%**. And the **median page runs 2,511 words**. The shape of that table is the finding. The signals that a CMS or an SEO plugin emits without being asked — JSON-LD, `dateModified`, an author field — clear 59%. The signals that require someone to decide something — a markdown variant, `HowTo`, a comparison table — sit under 37%. `llms.txt` is the interesting middle at 52.7%, and the next section is why that number is softer than it looks. ## The llms.txt number has a footnote 41 of the 78 domains serve a working file. The sizes are not comparable to each other in any useful way: | | Bytes | Domain | |---|---|---| | Smallest | 648 | llmstxt.org — the site that publishes the specification | | Median | 8,072 | — | | Largest | 2,500,078 | coursera.org | A 2.5 MB file is not a curated index. It is an export, and an agent reading it will truncate long before the end. More to the point: **5 of the 41 domains, 12.2%, got the file from an SEO plugin update rather than a decision.** | Domain | Plugin | |---|---| | aioseo.com | All in One SEO Pro v5 | | visiblefactors.com | All in One SEO Pro v5 | | www.fasturtle.com | All in One SEO v5 | | www.boralagency.com | Yoast SEO v27 | | www.yoghurtdigital.com | Yoast SEO v27 | Those five sites did not choose a curation strategy. Their plugin shipped a feature. They are in every llms.txt adoption statistic published after that release, including this one. ## Who blocks AI bots Eight of 93 pages sit on a domain that blocks at least one AI crawler outright. Six of them are worth naming, because what they block is informative: | Domain | Ranks for | Blocks | |---|---|---| | forbes.com | answer engine optimization (#4) | GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider, Applebot-Extended, anthropic-ai, meta-externalagent | | builtin.com | generative engine optimization (#19) | GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot | | lib.guides.umd.edu | how to get cited by ai (#8) | GPTBot, ChatGPT-User, CCBot, Bytespider, Google-Extended, Applebot-Extended | | libguides.brown.edu | how to get cited by ai (#16) | GPTBot, ChatGPT-User, CCBot, Bytespider, Google-Extended, Applebot-Extended | | llmclicks.ai | ai visibility checker (#19) | CCBot, Bytespider, meta-externalagent | | coursera.org | answer engine optimization (#6) | meta-externalagent | Two university library guides ranking for "how to get cited by ai" block the crawlers that would cite them. Forbes blocks `anthropic-ai`, a token Anthropic has retired, which tells you when that file was last opened. ## What this does not show - **It measures presence, not effect.** Nothing here says a page with `llms.txt` gets cited more often. Testing that needs citation data these checks do not collect. - **The sample is self-selected.** Pages that rank for "answer engine optimization" are written by people who think about AEO. Treat every number as an upper bound for the wider web. - **One country, one language, one day.** US, English, 9 August 2026. Google results move. - **Presence is not correctness.** A `FAQPage` block counts here whether or not its answers match the visible text. So does an `llms.txt` that lists dead URLs. - **93 is small.** A five-point difference between two signals in this sample is noise. ## The baseline: this site, on the day of the scan The same run measured alexturik.com, which is how the gaps got prioritised. This is the state on 9 August 2026, before any of it was addressed: | | This site, 2026-08-09 | Sample median or share | |---|---|---| | Median words per post | 1,383 | 2,511 | | Question headings | 7 / 130 (5.4%) | 18.7% | | Pages with `FAQPage` | 3 / 22 (all under /plugins, none in the blog) | 25.8% | | Markdown variants | 15 pages | 8.6% of the sample had any | | AI bots blocked | 0 | 8.6% blocked at least one | | `llms.txt` | 10,750 bytes | median 8,072 | Those figures are deliberately frozen at the scan date and are no longer current — the blog now runs FAQ markup and question headings throughout, and every page has a markdown twin. Rerunning the scan is the only honest way to update them, which is the point of writing the method down. ## How to cite this > Turovskyi, O. (2026). *AEO Adoption Scan, 2026: 93 pages ranking for AEO and GEO queries.* alexturik.com/aeo/research-2026. Measured 9 August 2026. The raw dataset is in the repository as `docs/reports/aeo-adoption-dataset-2026-08-09.json`. If you rerun the same seven queries, I would like to see the numbers. --- # AEO & GEO Glossary > Plain definitions for the AEO and GEO terms that actually change what you build: llms.txt, JSON-LD, embeddings, chunking, grounding, RAG and more. **Canonical:** https://alexturik.com/aeo/glossary Every term here is one I had to pin down to finish a piece of work. Definitions are short, and each links to the article where the idea does something. *Українською: [глосарій AEO та GEO](/uk/aeo/glossary).* ## The disciplines ### AEO — Answer Engine Optimization Making a page readable, parseable and quotable by systems that answer a question directly instead of returning links. The success condition is being the source a generated answer is built from. See the [pillar page](/aeo) for the four preconditions. ### GEO — Generative Engine Optimization The same work, framed around how you are represented inside generated text rather than whether you are cited. In practice the techniques are identical; the distinction is emphasis. Compared properly in [GEO vs SEO vs AEO](/blog/geo-vs-seo-vs-aeo). ### LLMO — Large Language Model Optimization A third label for the same territory, framed model-first. Useful mainly as a search term. ### SEO — Search Engine Optimization Optimising for a ranking that produces a click. Shares its entire technical foundation with AEO — crawlability, speed, structure — and diverges on what counts as success. ## Machine-readable surfaces ### llms.txt A Markdown file at `/llms.txt` that gives an AI crawler a curated map of a site: what it is, which pages matter, and where the machine-readable version of each one lives. It is a proposed convention, not a standard, and it is cheap enough that the argument about adoption is beside the point. Implementation walkthrough: [How I added llms.txt in 15 minutes](/blog/how-i-added-llms-txt-to-my-nextjs-blog-in-15-minutes). ### Markdown mirror (`.md` variant) A plain-Markdown twin of an HTML page, served at the same URL with `.md` appended — [this page has one](/aeo/glossary.md). No navigation, no scripts, no cookie banner: just the content. AI crawlers parse it far more reliably than a rendered React page, and it costs nothing to serve both. The HTML stays canonical for search; the mirror is `noindex` so Google does not see a duplicate. ### AGENTS.md A file describing a repository or site to coding agents — conventions, commands, constraints. Adjacent to `llms.txt` but aimed at agents that will *write* code against you, not summarise you. ### robots.txt The fetch-time control. Tells crawlers which paths they may request. It is a request honoured by convention, not a fence — see [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html). Distinct from `noindex`, which governs *use* rather than fetching: a page blocked in robots.txt is never fetched, so its `noindex` is never seen. ### `noindex` A directive, in a `` tag or an `X-Robots-Tag` header, excluding a page from an index. For AI crawlers it reads as a full exclusion from answers and training. The header form is invisible in the HTML, which makes it the harder half of [the 3-word tag](/blog/the-3-word-tag-that-hides-your-best-pages-from-ai). ## Structure and meaning ### JSON-LD Structured data as a JSON block in ` ``` A few principles that make this block actually useful, not decorative: - `@graph` lets you place several entities in a single script and connect them via `@id`. Cleaner than two separate tags. - `@id` is not the page URL, it's a unique identifier for the entity. The hash fragment (`#article`, `#faq`) makes it stable. - `mainEntityOfPage` ties `Article` to a specific canonical URL — the same URL your [`` and `og:url`](/blog/the-invisible-tags-shaping-your-ai-citations) must name. Three places asserting the canonical is fine; three places disagreeing about it is how citation potential gets split across variants of one page. - `image` as an array with three aspect ratios (1:1, 4:3, 16:9) is a direct Google requirement for getting `Article` rich results. - `inLanguage` helps LLM platforms understand the content language and correctly cite it in language-specific queries. ### Implementation in Next.js (App Router) A working page example for Next.js 16 with Tailwind CSS. The key technical decision here is using `JSON.stringify` instead of a template literal to safely escape data. ```tsx title="app/posts/[slug]/page.tsx" import type { Metadata } from "next"; interface PageProps { params: Promise<{ slug: string }>; } export async function generateMetadata({ params }: PageProps): Promise { const { slug } = await params; return { title: `Article ${slug}`, alternates: { canonical: `https://example.com/posts/${slug}` }, }; } export default async function PostPage({ params }: PageProps) { const { slug } = await params; const canonicalUrl = `https://example.com/posts/${slug}`; const jsonLd = { "@context": "https://schema.org", "@graph": [ { "@type": "Article", "@id": `${canonicalUrl}#article`, "headline": "How to help AI understand your content with JSON-LD", "description": "A step-by-step guide on adding structured data for AI Overviews and LLMs.", "image": [ "https://example.com/images/cover-1x1.jpg", "https://example.com/images/cover-4x3.jpg", "https://example.com/images/cover-16x9.jpg", ], "datePublished": "2026-05-01T08:00:00+00:00", "dateModified": "2026-05-01T08:00:00+00:00", "inLanguage": "en-US", "author": { "@type": "Person", "name": "Olena Koval", "url": "https://example.com/authors/olena-koval", }, "publisher": { "@type": "Organization", "name": "Example Media", "logo": { "@type": "ImageObject", "url": "https://example.com/logo.png", }, }, "mainEntityOfPage": { "@type": "WebPage", "@id": canonicalUrl, }, }, { "@type": "FAQPage", "@id": `${canonicalUrl}#faq`, "mainEntity": [ { "@type": "Question", "name": "Does JSON-LD replace Microdata markup?", "acceptedAnswer": { "@type": "Answer", "text": "Yes — Google recommends JSON-LD as the primary format. Microdata is still valid, but JSON-LD is easier to maintain because it's decoupled from HTML.", }, }, { "@type": "Question", "name": "Will an LLM see my markup if it renders via JavaScript?", "acceptedAnswer": { "@type": "Answer", "text": "The reliable path is to ship JSON-LD in the initial HTML — via SSR or static generation.", }, }, ], }, ], }; return ( <> ``` Twenty words. The rendered DOM has two thousand. Ratio: 0.01. Fail. ## Where does the 0.3 threshold come from? From me. It is the cutoff [AEO Checker](/extensions/aeo-checker) uses, and I picked it because it separated shells from server-rendered pages cleanly on the sample I built the extension against. No standards body publishes it, no search engine documents it, and you will not find a vendor who will confirm that crossing it changes anything on their side. What the ratio is genuinely good for is being cheap. Raw HTML divided by what a human sees is one `curl` away, and it answers the only question that matters first: did any of your content arrive in the response at all. If raw HTML holds a meaningful share of the visible text, a bot that does not run JavaScript still gets something to read. If it holds almost none, that bot gets a stub, and no amount of ranking work downstream will rescue it. Read `0.3` as the line where I start looking, not a threshold anyone enforces. The consequences are concrete and unpleasant: - your site doesn't appear in ChatGPT, Claude, Perplexity, or Gemini answers; - your content doesn't get cited in generated overviews; - your pages don't get indexed as a knowledge source for training or RAG. This is the new SEO reality. Old Googlebot eventually learned to render JS — slowly, through two-pass indexing, but it does. The new AI crawlers do not. And they won't any time soon: the cost of executing JavaScript at LLM-training and real-time-inference scale is not economical. It's faster to skip your site and pick the competitor that returns HTML on the first byte. ## How do SSR and static generation fix it? The conclusion is obvious — render HTML on the server. Three approaches, each with its own use case: - **Server-Side Rendering (SSR)** — HTML is generated on the fly per request. Fits dynamic, personalized content: dashboards, live prices, A/B tests. - **Static Site Generation (SSG)** — HTML is generated at build time and served as a static file from a CDN. Fast, cheap, ideal for blogs, documentation, marketing pages. - **Incremental Static Regeneration (ISR)** — a compromise: static files that rebuild on a schedule or via a trigger. Combines SSG performance with SSR freshness. All three give the bot a complete HTML response from the very first byte. The ratio jumps to 0.9+, the audit goes green, the content starts showing up in AI answers. ## Why is the Next.js App Router the default I reach for? This site is built on Next.js with the App Router. Three reasons it's the right stack for AI-friendly architecture: 1. **Server Components by default.** In the App Router, every component is server-rendered until you explicitly write `'use client'`. By default you're in SSR mode without any extra configuration. Content lands in HTML automatically. 2. **Streaming HTML.** Next.js streams HTML through React Suspense — bots get the heading, metadata, and base structure instantly, then the main body streams. For crawlers with hard timeouts, this matters. 3. **Metadata as code.** The `generateMetadata` function gives you control over ``, `<meta>`, OpenGraph, and JSON-LD straight from the page component. AI bots read it from raw HTML. Here's a sample article page that humans and bots both see equally well: ```tsx title="app/blog/[slug]/page.tsx" import { notFound } from 'next/navigation'; import type { Metadata } from 'next'; interface Post { title: string; excerpt: string; contentHtml: string; publishedAt: string; author: string; } async function getPost(slug: string): Promise<Post | null> { const res = await fetch(`https://api.example.com/posts/${slug}`, { next: { revalidate: 3600 }, }); if (!res.ok) return null; return res.json(); } export async function generateMetadata( { params }: { params: Promise<{ slug: string }> } ): Promise<Metadata> { const { slug } = await params; const post = await getPost(slug); if (!post) return { title: 'Not found' }; return { title: post.title, description: post.excerpt, openGraph: { title: post.title, description: post.excerpt, type: 'article', publishedTime: post.publishedAt, authors: [post.author], }, }; } export default async function PostPage( { params }: { params: Promise<{ slug: string }> } ) { const { slug } = await params; const post = await getPost(slug); if (!post) notFound(); return ( <article> <header> <h1>{post.title}</h1> <p> <time dateTime={post.publishedAt}>{post.publishedAt}</time> {' · '} <span>{post.author}</span> </p> </header> <div dangerouslySetInnerHTML={{ __html: post.contentHtml }} /> </article> ); } ``` What's load-bearing here for AI bots: - `fetch()` runs on the server — data lands in the HTML before it's ever sent to a client; - the entire content (heading, metadata, body) appears in raw HTML; - no `'use client'` is needed because there's no interactivity, so no hydration cost; - `revalidate: 3600` provides ISR — static rebuilds once an hour without a redeploy. If you do need an interactive piece (a comment form, a like button), pull it into a separate client component: ```tsx title="app/blog/[slug]/like-button.tsx" 'use client'; import { useState } from 'react'; interface LikeButtonProps { postId: string; initialLikes: number; } export function LikeButton({ postId, initialLikes }: LikeButtonProps) { const [likes, setLikes] = useState(initialLikes); const [isPending, setIsPending] = useState(false); async function handleClick() { setIsPending(true); setLikes((n) => n + 1); try { await fetch(`/api/posts/${postId}/like`, { method: 'POST' }); } catch { setLikes((n) => n - 1); } finally { setIsPending(false); } } return ( <button onClick={handleClick} disabled={isPending}> ♥ {likes} </button> ); } ``` The server component renders the article into HTML — bots get the full text. The client component hydrates after load — users get interactivity. ## How do you validate your own rendering? Check your site in a minute: 1. Open the page, hit `Ctrl+U` (View Page Source). 2. Search for the article's main heading in the raw HTML via `Ctrl+F`. 3. Found? You're safe. Not found? You have a CSR trap. 4. Run `curl -A "GPTBot" https://your-site.com/page | wc -w` and compare to the word count in the live browser. AEO Checker's actual bands: `0.8` and above passes, `0.3` to `0.8` is a warning, below `0.3` is a fail — on the reasoning above rather than on anyone's published standard. *(Of course, if the crawler is blocked in robots.txt, it never even reaches this stage — see [my earlier post on configuring robots.txt for AI](/blog/why-your-site-doesnt-show-in-chatgpt-and-perplexity).)* ## Closing JavaScript frameworks rewrote UX forever, but the cost is invisibility to a new generation of crawlers. Move rendering to the server and your content gets a chance to be read and quoted again. Next.js App Router makes this transition almost painless — it's an architecture where the right decision is the default. ## What each rendering mode hands a JS-less crawler Every AI crawler in wide use today fetches HTML and stops. What is in that first response is the entire page as far as they are concerned. | Rendering mode | First response contains | Safe for AI crawlers | |---|---|---| | Static generation (SSG) | The full page | Yes | | Server-side rendering (SSR) | The full page | Yes | | Incremental regeneration (ISR) | The full page, possibly stale | Yes | | Streaming SSR | The shell plus content as it resolves | Yes, if content is not behind a client boundary | | Client-side rendering (CSR) | An empty `<div id="root">` | No | | Hybrid with client-only content | Layout without the article | No, for the part that matters | The last row is where most real sites sit. The page is server-rendered, passes a casual check, and the one component that holds the article body was marked client-side for an interaction that could have been progressive. ## FAQ ### Do AI crawlers execute JavaScript? As a rule, no. GPTBot, ClaudeBot and PerplexityBot fetch HTML and parse it; none of them documents running a full rendering pass the way Googlebot's second wave does. So a client-rendered page is not "slow to be indexed" for them, it is empty. ### Does Googlebot rendering JavaScript mean I am fine? No, for two reasons. Google's rendering happens in a second pass on its own schedule, so a fresh page can rank on an empty first fetch. And Google is now one consumer among several — a page that only survives because Googlebot renders it is invisible to every AI answer engine that does not. ### What is the text-to-HTML ratio, and why 0.3? It is the visible text divided by the raw HTML size. Below about 0.3 the response is mostly markup, script tags and inlined state rather than content — the signature of a page whose text arrives later. The 0.3 figure is the threshold AEO Checker uses, not an industry standard: it is a smell test that separates server-rendered pages from shells, and nothing published by a search engine or an AI vendor endorses it. ### Does streaming SSR count as server-rendered? Yes, as long as the content itself is not inside a client component that only resolves in the browser. Streaming changes when bytes arrive, not whether they exist. What breaks a crawler is content that is never in any response, not content that is in the second chunk. ### How do I check what a bot actually receives? `curl -s https://example.com/page | wc -c` against the visible text length, or simply disable JavaScript and reload. The [AEO Checker](/extensions/aeo-checker) extension reads the served HTML of the current tab, so it reports what the crawler sees rather than what the rendered DOM shows. ## Sources - [Google Search Central: JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) — the two-wave rendering model and its costs - [OpenAI: bots and crawlers](https://developers.openai.com/api/docs/bots) — what GPTBot fetches - [Perplexity: PerplexityBot](https://docs.perplexity.ai/guides/bots) — crawler behaviour and identification - [Next.js: server and client components](https://nextjs.org/docs/app/getting-started/server-and-client-components) — where the boundary actually falls Part of the [Answer Engine Optimization](/aeo) cluster — the full reading order in dependency sequence, plus a [glossary](/aeo/glossary) of every term used across these articles. --- **Has your audit revealed rendering issues on your project?** Follow me on [LinkedIn](https://linkedin.com/in/alexturik) to keep up with new AEO tools. If your site is stuck in the "CSR trap" and you need help migrating to a modern SSR architecture, [get in touch](mailto:alexturik@gmail.com) for a deep audit. --- --- title: Why Your Site Doesn't Show Up in ChatGPT (or Perplexity) description: Three robots.txt strategies for AI crawlers, and the two configuration mistakes that quietly cut you out of ChatGPT, Claude, and Perplexity answers. date: 2026-05-02 updated: 2026-08-29 tags: [aeo, seo, robots-txt, ai-search] --- # Why Your Site Doesn't Show Up in ChatGPT (or Perplexity) The audit comes back red. Three or more AI crawlers blocked. Or — worse — a global `Disallow: /` under `User-agent: *` quietly sitting in the root of your `robots.txt`, untouched since someone "ported it over" from a 2018 staging file. The result is the same regardless of cause: Perplexity won't read you, ChatGPT can't see you, Claude routes around your domain. The AI traffic that's growing fastest in 2026 walks right past your brand. Let's break the problem down — and fix it. ## How are AI bots different from Googlebot? A classic SEO crawler and an AI crawler look similar in the logs but solve very different problems. Googlebot builds a search index. AI bots build something else: training corpora for models, real-time answer indexes, or fetchers that execute one specific user query in one specific session. The big conceptual mistake is thinking *"I allowed Googlebot, so I'm fine."* Not fine. Modern AI vendors (OpenAI, Anthropic) have split their crawling infrastructure into three distinct categories, each with its own `User-agent` string: 1. **Training crawlers** — `GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended`. They collect content for model training. Blocking them prevents your content from reaching future model versions but **does not affect** real-time citation in answers. 2. **Search indexers** — `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`. These are your ticket into the answer. OpenAI explicitly tells publishers that sites blocking `OAI-SearchBot` will not appear in ChatGPT search results, even if regular navigation links are still allowed. 3. **User-triggered fetchers** — `ChatGPT-User`, `Claude-User`, `Perplexity-User`. They visit a page when a real user makes a specific request. OpenAI and Perplexity note that for user-initiated fetches, robots.txt rules may not be applied in the standard way — that's a separate discussion about server-side controls. Bottom line: *"block all AI"* is no longer a strategy. It's a matrix of decisions. ## Why is your site falling out of AI answers? ### Reason A: a global `Disallow: /` under wildcard The classic trap. Someone wrote `robots.txt` years ago, added explicit rules for Googlebot and Bingbot, and set the rest as `Disallow`: ```text title="robots.txt — DO NOT DO THIS (anti-pattern)" User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: * Disallow: / ``` Looks logical. Works catastrophically. Every AI agent not named explicitly falls under the wildcard `Disallow` — meaning ChatGPT Search, Claude, Perplexity, and Apple Intelligence are all effectively blocked. You optimized your site for two 2010s-era search engines and cut yourself off from five 2020s AI ecosystems. ### Reason B: explicitly blocking 3+ AI bots out of inertia In 2023–2024, many brands added `Disallow: /` for `GPTBot` and `CCBot` as a reaction to the scraping discourse. At the time it felt cautious. Today it's a self-inflicted wound. ```text title="robots.txt — 2023 SCANDAL REACTION (also an anti-pattern)" User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Disallow: / ``` Why is this a fail? Because `PerplexityBot` is your only path into Perplexity. `ClaudeBot` was the catch-all for both search and training until Anthropic split it; if you still have an old `Disallow: ClaudeBot` rule, that's not "opt out of training" anymore — it's loss of visibility in Claude's web tool. Anthropic has officially split crawlers into multiple bot agents. ## Which `robots.txt` strategy should you pick? Pick one. Don't mix. ### Strategy 1: maximum AI visibility Recommended for marketing sites, blogs, and documentation. ```text title="robots.txt — Strategy 1: maximum visibility" # Fits: content marketing, blog, product docs, media # --- OpenAI / ChatGPT --- User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # --- Anthropic / Claude --- User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / # --- Perplexity --- User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # --- Common Crawl (used by many LLMs) --- User-agent: CCBot Allow: / # --- Google Gemini / AI Overviews --- User-agent: Google-Extended Allow: / # --- Apple Intelligence --- User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / # --- Classic search engines --- User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / # --- Default --- User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml ``` **Pros:** maximum visibility across every AI ecosystem. Highest chance of citation. Worth pairing with an [`llms.txt`](/blog/how-i-added-llms-txt-to-my-nextjs-blog-in-15-minutes): `robots.txt` says what a crawler may fetch, that file says where the good parts are. **Cons:** your content does feed GPT and Claude training corpora. If you have regulated data or paid content, this isn't for you. ### Strategy 2: differentiated (allow search, block training) — the industry default This is the configuration most brands settle on. Logic: we want to appear in answers, but we don't want our content fed into someone else's model. ```text title="robots.txt — Strategy 2: differentiated" # Fits: B2B SaaS, edtech, publishing brands with proprietary content # --- OpenAI: allow search, block training --- User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # --- Anthropic: allow search, block training --- User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / # --- Perplexity: allow (no separate training crawler exists) --- User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # --- Common Crawl: block (indirect training pipeline) --- User-agent: CCBot Disallow: / # --- Google: block training token, keep search --- User-agent: Google-Extended Disallow: / User-agent: Googlebot Allow: / # --- Apple: block training, keep search --- User-agent: Applebot-Extended Disallow: / User-agent: Applebot Allow: / # --- Default --- User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml ``` **Pros:** you stay in ChatGPT, Claude, Perplexity, and Gemini answers. Your content does **not** go into training. Aligned with where the industry is settling. **Cons:** where both bots are allowed, OpenAI reserves the right to serve both purposes from one fetch — [its crawler documentation](https://developers.openai.com/api/docs/bots) says that "if your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling." Block `GPTBot` and you give that up: `OAI-SearchBot` then has to crawl on its own, so your effective crawl budget on OpenAI's side is lower. ### Strategy 3: hybrid with protected sections (e-commerce, portals) ```text title="robots.txt — Strategy 3: hybrid" # Fits: e-commerce, SaaS portals, sites with member areas User-agent: GPTBot Disallow: /account/ Disallow: /checkout/ Disallow: /cart/ Disallow: /api/ Disallow: /admin/ Allow: / # ... apply the same pattern to OAI-SearchBot, ClaudeBot, # Claude-SearchBot, PerplexityBot, and *. Sitemap: https://example.com/sitemap.xml ``` **Pros:** marketing, product, and content pages are visible to AI. Sensitive URLs are not. **Cons:** longer file, harder to maintain, easy to forget a rule for a new bot. ## Does "content is open" mean the model can read it? This is the crucial distinction even experienced marketers miss. Imagine `https://example.com/article-x` returns `200 OK`, renders in a browser, sits in your sitemap, is indexed by Googlebot, and scores perfectly on Lighthouse. Looks fully open. But if your `robots.txt` has `User-agent: PerplexityBot` followed by `Disallow: /`, then **PerplexityBot will physically never make an HTTP request** to that page. Not out of misplaced civility — bots that respect robots.txt filter URLs at the crawl-planning stage. The page never loads, the HTML never gets parsed, the content never gets vectorized, and your text never makes it into the model's answer. Anthropic, OpenAI, and Perplexity all state that their official bots respect robots.txt. This isn't marketing. This is the engineering reality: the bot fetches `robots.txt` first, parses it, and any URL under a `Disallow` rule is dropped from the queue. Four practical consequences people forget: 1. **OpenGraph, schema.org, hreflang, and every other on-page SEO signal are irrelevant** if the AI bot doesn't fetch the page in the first place. You're optimizing something nothing reads. 2. **The sitemap is not a backdoor.** If a URL is in the sitemap but blocked in robots.txt, robots.txt wins. 3. **HTTPS, performance, Core Web Vitals — all irrelevant** to a bot that got "forbidden" before connecting. 4. **Common Crawl archives from past years can still be valuable to a model**, but if you block `CCBot` today, future model versions will not refresh their knowledge of your site. One more uncomfortable technical detail: none of OpenAI's crawlers execute JavaScript — they fetch `.js` files but don't run them. If your content renders client-side (CSR-only React/Vue with no SSR), even a perfect `robots.txt` leaves you invisible to AI. That's not a robots.txt problem — it's a rendering architecture problem, and it has [its own article](/blog/the-javascript-trap-why-ai-bots-see-empty-pages), because the fix is architectural rather than a config line. Next.js App Router with Server Components solves it by default; a pure Vite SPA does not. ## How do you verify and deploy the change? 1. **Audit current state.** Open `https://yourdomain.com/robots.txt` directly. Scan every `User-agent` block. Look for: a global `Disallow: /`, explicit blocks for `GPTBot`/`ClaudeBot`/`PerplexityBot`/`CCBot`, deprecated identifiers (`anthropic-ai`, `claude-web` are both retired). 2. **Pick a strategy** from the three above. Don't improvise. Mixing them produces incorrect precedence rules. 3. **Deploy** the new `robots.txt` at the domain root (literally `/robots.txt`, not `/static/robots.txt` or `/public/robots.txt`). Verify with `curl -I https://yourdomain.com/robots.txt` — must return `200 OK` with `Content-Type: text/plain`. 4. **Re-indexing latency.** For search, OpenAI [documents](https://developers.openai.com/api/docs/bots) that "it can take ~24 hours from a site's robots.txt update for our systems to adjust." Don't expect an instant change — let the cycle complete. Separate check for Cloudflare/Akamai/WAF: make sure your bot-management rules don't block AI agents at the network layer. `Allow: /` in robots.txt doesn't help if the WAF returns `403` before the file is ever read. ## Who is actually knocking Blocking by reflex costs visibility, because these user agents do different jobs and only some of them relate to training. | User agent | Operator | What it is for | Blocking it costs you | |---|---|---|---| | `GPTBot` | OpenAI | Training corpus | Nothing in ChatGPT answers today | | `OAI-SearchBot` | OpenAI | ChatGPT search index | Presence in ChatGPT answers | | `ChatGPT-User` | OpenAI | Live fetch on a user's request | The user cannot open your page in-chat | | `ClaudeBot` | Anthropic | Training corpus | Nothing in Claude answers today | | `Claude-SearchBot` | Anthropic | Search index | Presence in Claude answers | | `PerplexityBot` | Perplexity | Search index | Presence in Perplexity answers | | `Google-Extended` | Google | Gemini / AI training opt-out | Nothing in Google Search rankings | The distinction that matters is training crawler versus search crawler. Blocking the first is a defensible editorial choice with no visibility cost. Blocking the second removes you from the answers — and most blanket "block the AI bots" rules do both without noticing. Seven agents is the shortlist you actually have to decide about. The full reference — 44 tokens with owner, job, whether the vendor says it honours robots.txt, and the IP list that proves one is genuine — is in [the AI crawler list for 2026](/blog/ai-crawler-list-robots-txt-2026), along with three robots.txt presets and a scan of what 53 live sites do. ## FAQ ### Will blocking GPTBot remove me from ChatGPT? No — and that is the point of separating the two. `GPTBot` collects training data; `OAI-SearchBot` builds the index ChatGPT searches when answering. Blocking the first keeps your content out of the next model without touching your presence in today's answers. Blocking the second is what makes you disappear. ### Does Google-Extended affect my Google Search rankings? No. `Google-Extended` is a separate control for Gemini and AI training, and Google states explicitly that it does not influence inclusion or ranking in Search. It is the one blanket opt-out with no measurable SEO cost. ### Is robots.txt enough to keep AI crawlers out? It is a request, not a fence — it works because the major operators choose to honour it. If you need enforcement rather than cooperation, that is a WAF or authentication problem. Conversely, `Allow: /` in robots.txt achieves nothing if your WAF returns 403 to the crawler before the file is ever read. ### Why does my site not appear even though robots.txt allows everything? Three usual causes, in order of frequency: the page is [client-rendered](/blog/the-javascript-trap-why-ai-bots-see-empty-pages) so the crawler receives an empty shell, a [`noindex` directive](/blog/the-3-word-tag-that-hides-your-best-pages-from-ai) is set at the page or header level, or bot management at the CDN blocks the user agent before it reaches your origin. All three look fine from a browser, which is why `robots.txt` gets blamed for all three. ### How do I check which of these is the problem? Fetch the page as a bot would — `curl -A GPTBot -sI https://example.com/` for the status and headers, then the body for whether the content is there at all. The [AEO Checker](/extensions/aeo-checker) extension does the page-level half of this, reading robots.txt, llms.txt and the served HTML for the tab you are on. ## Sources - [OpenAI: bots and crawlers](https://developers.openai.com/api/docs/bots) — GPTBot, OAI-SearchBot and ChatGPT-User, and the difference between them - [Anthropic: does Anthropic crawl the web?](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — ClaudeBot and how to control it - [Perplexity: PerplexityBot](https://docs.perplexity.ai/guides/bots) — identification and verification by IP - [Google Search Central: Google-Extended](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) — the AI opt-out that does not affect Search - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) — what robots.txt does and does not guarantee Part of the [Answer Engine Optimization](/aeo) cluster — the full reading order in dependency sequence, plus a [glossary](/aeo/glossary) of every term used across these articles. --- **Got questions or need help?** Follow me on [LinkedIn](https://linkedin.com/in/alexturik) for more AEO architecture write-ups. Need a deep audit of your platform or SSR configuration tuned for AI crawlers? [Get in touch](mailto:alexturik@gmail.com) and we'll work through your case.