← Oleksii Turovskyi

The 3-Word Tag That Hides Your Best Pages from AI

· 6 min read

Three words. Three meanings. And your single best article vanishes from ChatGPT, Claude, and Perplexity — even if Google still shows it.

I'm talking about <meta name="robots">. The tag in <head> you may never have looked at. The tag your CMS added without asking. The one that's quietly killing your AEO (AI Engine Optimization) right now.

What is this tag, anyway#

<meta name="robots" content="..."> is a directive for crawlers. It tells the machine what to do with this page, what not to do, and whether it's worth following the links on it.

The values come in pairs or alone:

  • index / noindex — index it, or don't.
  • follow / nofollow — follow internal and external links, or don't.
  • none — shorthand for noindex, nofollow.
  • noarchive — don't keep a cached copy.
  • nosnippet — don't show a text fragment in search results.

Default (no tag) = index, follow. That is exactly what you want on 99% of your public pages.

How AI agents actually read it#

GPTBot. ClaudeBot. PerplexityBot. All three have publicly committed to honoring robots-level signals. For them, noindex means much more than "don't show in search." It's a direct signal: don't use this page for model training, don't pull it into a RAG summary, fully exclude it from any generated answer.

nofollow they treat slightly weaker — more like a hint about low quality or low link trust than a hard ban. But it still gets factored in.

The conclusion is brutal: an accidental noindex on your best-of-2024 post is a concrete wall in front of the AI traffic you actually want.

Why this is a separate story from robots.txt#

  • robots.txt is the global rubber band. It covers the whole site or whole directories (wildcard patterns).
  • <meta name="robots"> is a fine-grained instrument. It works strictly at the level of an individual HTML document.

This is exactly where the most dangerous trap hides: your site can be perfectly indexable, robots.txt clean and correctly configured, the sitemap including the right URLs — and one specific article still closed off by a tag that survived from a legacy template or CMS default. The cruelest version: it can be your highest-converting post, and you don't notice anything until the AI-source traffic starts drying up.

Three scenarios for how noindex reaches production#

  1. Staging promoted to production without cleanup. Engineer set noindex on the staging domain to avoid duplicate indexing or leaking preview content into Google. Deploy ran — the tag rode along to prod, because nobody added a "strip robots-meta on production build" step in the CI/CD pipeline.

  2. CMS template inherited noindex from preview mode. Hugo, Next.js, WordPress, Webflow — framework doesn't matter. If the template has <meta name="robots" content="noindex"> under a condition like if env === 'preview' and the condition broke or the env variable didn't propagate during the build — you ship noindex to prod. Silent. No errors in the compiler console.

  3. "Coming soon" page didn't update after release. A landing was published with noindex a week before launch so nothing would index a hollow skeleton. Launch happened. Nobody removed the tag. Months later, you're wondering why your product page isn't showing up in ChatGPT's answers.

All three happen far more often than you'd think. All three slip through code review, because this tag visually blends into the rest of the "standard HTML boilerplate."

A 30-second audit#

Fastest path: open the page → View Page Source (Ctrl+U) → Ctrl+F for "robots". If you find noindex, you have a problem.

A more engineering-forward check is one DevTools Console snippet you can run on any page:

devtools-meta-robots-check.js
const robotsMeta = document.querySelector('meta[name="robots"]');
 
if (robotsMeta) {
  console.warn(`Found robots meta with content "${robotsMeta.content}"`);
  if (robotsMeta.content.includes('noindex')) {
    console.error('CRITICAL: This page is hidden from AI crawlers.');
  }
} else {
  console.log('No robots meta. Default behavior: index, follow. ✓');
}

For mass audits across the whole site, use Screaming Frog, Sitebulb, or write a small Playwright/Puppeteer script that walks your sitemap and dumps every <meta> tag in one pass.

How to fix it (Next.js example)#

Delete the tag entirely. Default behavior is index, follow. That is what you want.

Don't add <meta name="robots" content="index, follow"> explicitly. It's visual noise. The crawler does this by default — you're just rendering extra bytes for no reason.

Here's the correct shape in Next.js (App Router):

app/blog/[slug]/page.tsx
import type { Metadata } from 'next';
 
export const metadata: Metadata = {
  title: 'How meta robots actually works',
  description: 'A full breakdown of meta robots and AI crawling.',
  // DO NOT WRITE robots: { index: true, follow: true }
  // Next.js does not emit this tag by default — that is the desired behavior.
  // Use the robots object only for explicit denial:
  // robots: { index: false, follow: false }
};
 
export default function BlogPostPage() {
  return (
    <article className="max-w-3xl mx-auto py-10 px-4">
      <h1 className="text-4xl font-bold">How meta robots actually works</h1>
      <p className="mt-4">Article body…</p>
    </article>
  );
}

A clean audit treats no tag as a pass. An explicit index, follow is also a pass, but technically redundant.

Edge case — when noindex actually belongs there#

noindex makes sense on:

  • archive and tag pages that generate duplicate content.
  • Pagination pages like /blog/page/2/ — especially when rel="canonical" points back to the first page.
  • Thin-content pages: local search results, filtered listings, sort-parameter URLs.
  • Internal-only pages (/admin, /healthz, /test).

On canonical blog posts, product landing pages, the homepage — never. If your strategy reduces to "I want to be cited in Claude and ChatGPT," every noindex on a content page is a voluntary surrender of traffic.

And one more trap: the X-Robots-Tag HTTP header#

The exact same effect can be achieved at the server level via an HTTP response header:

HTTP/2 200 OK
Content-Type: text/html; charset=utf-8
X-Robots-Tag: noindex

It works identically to the meta tag, but is significantly harder to detect — it's not in the HTML at all. View Source shows nothing. You have to look at the Network tab in DevTools or check the server response from a terminal:

curl -I https://example.com/page/

If your HTML is clean and your audit shows green but the page still doesn't appear in AI answers — check the response headers. Server-level X-Robots-Tag is a category of bug that's almost impossible to spot from the front-end side.

The summary

Three words in <head> can cost you all your AI traffic. One invisible header can do the same with no trace in the HTML.

The audit takes 30 seconds. The fix is one comma and two lines of code. The risk is your highest-leverage content silently invisible to the entire generation of AI search forming right now.

Follow me on LinkedIn for more on AEO and AI-search architecture. For an audit of your site's AI readiness, get in touch.

View as Markdown