Posts tagged “AI Search”
How AI crawlers fetch, render and rank pages: what GPTBot, ClaudeBot and PerplexityBot actually see, and how retrieval differs from classic search.
11 articles
AI Crawler List 2026: Every User-Agent, robots.txt per Bot
44 AI user-agent tokens with owners and IP lists, which ones ignore robots.txt, three copy-paste robots.txt presets, and a scan of 53 live robots files.
· 19 min read
Skip the AI Visibility Checker: Audit Your Site by Hand
Run the same eight checks a paid AI visibility checker runs, for free: llms.txt, robots.txt, JSON-LD, and real 2026 baselines to compare your site against.
· 13 min read
GEO vs SEO vs AEO — What Actually Changes in Your Workflow
I scanned 93 pages ranking for AEO and GEO queries to see whether the people explaining the difference follow their own advice. Only 25.8% did.
· 11 min read
Embeddings: How AI Search Turns Your Page Into 384 Numbers
AI search doesn't match keywords - it compares vectors. Turn your text into an embedding, run cosine similarity live, and watch stuffing win the wrong race.
· 14 min read
How I Added llms.txt to My Next.js Blog in 15 Minutes
How llms.txt works, why 52.7% of pages ranking for AEO and GEO queries now serve one, and two ways to ship it on a Next.js blog in 15 minutes.
· 13 min read
The 3-Word Tag That Hides Your Best Pages from AI
Three words in `<head>` can hide your best article from ChatGPT, Claude, and Perplexity — even if Google still shows it. The silent AEO killer.
· 10 min read
Recency and Authority — Two Fields AI Uses to Trust You
Before quoting your article, AI runs two checks — is it fresh, and who wrote it. Both ride on machine-readable signals. Here's the canonical setup.
· 8 min read
How to Help AI Understand Your Content with JSON-LD
LLMs ground answers in structured data. With schema.org markup, the model sees a typed object — author, date, headline. Here's the canonical setup.
· 11 min read
Semantic HTML — Skipping Headings Breaks AI Optimization
AI crawlers parse your heading outline before anything else. Nothing repairs a skipped level — here's what really breaks, and how to keep structure clean.
· 11 min read
The JavaScript Trap — Why AI Bots See Empty Pages
AI crawlers don't run JavaScript. If your raw HTML is a hollow `<div id="root"></div>`, you're invisible to ChatGPT, Claude, and Perplexity.
· 10 min read
Why Your Site Doesn't Show Up in ChatGPT (or Perplexity)
Three robots.txt strategies for AI crawlers, and the two configuration mistakes that quietly cut you out of ChatGPT, Claude, and Perplexity answers.
· 12 min read