← Oleksii Turovskyi

The 3-Word Tag That Hides Your Best Pages from AI

· 10 min read · Updated

Українською: Три слова в head, які ховають ваші сторінки від AI — читати українською

Three words. Three meanings. And your single best article vanishes from ChatGPT, Claude, and Perplexity — even if Google still shows it.

I'm talking about <meta name="robots">. The tag in <head> you may never have looked at. The tag your CMS added without asking. The one that's quietly killing your AEO (Answer Engine Optimization) right now.

What is the robots meta tag?

<meta name="robots" content="..."> is a directive for crawlers. It tells the machine what to do with this page, what not to do, and whether it's worth following the links on it.

The values come in pairs or alone:

  • index / noindex — index it, or don't.
  • follow / nofollow — follow internal and external links, or don't.
  • none — shorthand for noindex, nofollow.
  • noarchive — don't keep a cached copy.
  • nosnippet — don't show a text fragment in search results.

Default (no tag) = index, follow. That is exactly what you want on 99% of your public pages.

How do AI agents read the robots tag?

GPTBot. ClaudeBot. PerplexityBot. All three publish user-agent tokens and say they honor robots.txt. What none of them publishes is a rule for the robots meta tag. OpenAI's crawler documentation covers robots.txt and says nothing about noindex at all, and I have not found a statement from OpenAI, Anthropic, or Perplexity that a noindex in your <head> keeps a page out of a training set. The version of this claim you see repeated everywhere — "noindex means don't train on me" — has no vendor behind it. Treat it as folklore.

Two vendors are the exception, and they are not the two you would guess: Apple documents that Applebot honours noindex and nosnippet, and Amazon documents the same for Amazonbot. Both are in the AI crawler list for 2026, which sets out per bot what each vendor actually commits to.

The narrower claim survives, and it is the one that costs you traffic anyway. noindex is a search-index directive, and an assistant that has to retrieve your page before it can quote it goes through a search index to find it. Block the index and you lose the citation — not because anyone honored a training opt-out, but because nothing retrieved you in the first place.

nofollow appears to be treated more loosely, closer to a quality hint than a hard ban. That one is my own reading of observed behavior, not a documented rule.

The conclusion is unchanged and still brutal: an accidental noindex on your best-of-2024 post is a concrete wall in front of the AI traffic you actually want.

How is this different from robots.txt?

  • robots.txt is the global rubber band. It covers the whole site or whole directories (wildcard patterns).
  • <meta name="robots"> is a fine-grained instrument. It works strictly at the level of an individual HTML document.

This is exactly where the most dangerous trap hides: your site can be perfectly indexable, robots.txt clean and correctly configured, the sitemap including the right URLs — and one specific article still closed off by a tag that survived from a legacy template or CMS default. The cruelest version: it can be your highest-converting post, and you don't notice anything until the AI-source traffic starts drying up.

How does noindex reach production by accident?

  1. Staging promoted to production without cleanup. Engineer set noindex on the staging domain to avoid duplicate indexing or leaking preview content into Google. Deploy ran — the tag rode along to prod, because nobody added a "strip robots-meta on production build" step in the CI/CD pipeline.

  2. CMS template inherited noindex from preview mode. Hugo, Next.js, WordPress, Webflow — framework doesn't matter. If the template has <meta name="robots" content="noindex"> under a condition like if env === 'preview' and the condition broke or the env variable didn't propagate during the build — you ship noindex to prod. Silent. No errors in the compiler console.

  3. "Coming soon" page didn't update after release. A landing was published with noindex a week before launch so nothing would index a hollow skeleton. Launch happened. Nobody removed the tag. Months later, you're wondering why your product page isn't showing up in ChatGPT's answers.

All three happen far more often than you'd think. All three slip through code review, because this tag visually blends into the rest of the "standard HTML boilerplate."

How do you audit this in 30 seconds?

Fastest path: open the page → View Page Source (Ctrl+U) → Ctrl+F for "robots". If you find noindex, you have a problem.

A more engineering-forward check is one DevTools Console snippet you can run on any page:

devtools-meta-robots-check.js
const robotsMeta = document.querySelector('meta[name="robots"]');
 
if (robotsMeta) {
  console.warn(`Found robots meta with content "${robotsMeta.content}"`);
  if (robotsMeta.content.includes('noindex')) {
    console.error('CRITICAL: This page is hidden from AI crawlers.');
  }
} else {
  console.log('No robots meta. Default behavior: index, follow. ✓');
}

For mass audits across the whole site, use Screaming Frog, Sitebulb, or write a small Playwright/Puppeteer script that walks your sitemap and dumps every <meta> tag in one pass.

How do you fix it in Next.js?

Delete the tag entirely. Default behavior is index, follow. That is what you want.

Don't add <meta name="robots" content="index, follow"> explicitly. It's visual noise. The crawler does this by default — you're just rendering extra bytes for no reason.

Here's the correct shape in Next.js (App Router):

app/blog/[slug]/page.tsx
import type { Metadata } from 'next';
 
export const metadata: Metadata = {
  title: 'How meta robots actually works',
  description: 'A full breakdown of meta robots and AI crawling.',
  // DO NOT WRITE robots: { index: true, follow: true }
  // Next.js does not emit this tag by default — that is the desired behavior.
  // Use the robots object only for explicit denial:
  // robots: { index: false, follow: false }
};
 
export default function BlogPostPage() {
  return (
    <article className="max-w-3xl mx-auto py-10 px-4">
      <h1 className="text-4xl font-bold">How meta robots actually works</h1>
      <p className="mt-4">Article body…</p>
    </article>
  );
}

A clean audit treats no tag as a pass. An explicit index, follow is also a pass, but technically redundant.

When does noindex actually belong?

noindex makes sense on:

  • archive and tag pages that generate duplicate content.
  • Pagination pages like /blog/page/2/ — especially when rel="canonical" points back to the first page.
  • Thin-content pages: local search results, filtered listings, sort-parameter URLs.
  • Internal-only pages (/admin, /healthz, /test).

On canonical blog posts, product landing pages, the homepage — never. If your strategy reduces to "I want to be cited in Claude and ChatGPT," every noindex on a content page is a voluntary surrender of traffic. The opposite move — handing AI crawlers an explicit invitation — takes about fifteen minutes via llms.txt.

What about the X-Robots-Tag header?

The exact same effect can be achieved at the server level via an HTTP response header:

HTTP/2 200 OK
Content-Type: text/html; charset=utf-8
X-Robots-Tag: noindex

It works identically to the meta tag, but is significantly harder to detect — it's not in the HTML at all. View Source shows nothing. You have to look at the Network tab in DevTools or check the server response from a terminal:

curl -I https://example.com/page/

If your HTML is clean and your audit shows green but the page still doesn't appear in AI answers — check the response headers. Server-level X-Robots-Tag is a category of bug that's almost impossible to spot from the front-end side.

Directive reference

The same word does different work depending on who is reading it.

The Google column is documented behavior, taken from Google's own robots meta reference. The AI crawlers column is not. No vendor publishes how its assistant handles the robots meta tag, so every cell in it is observed or inferred — what I see happen, not what anyone has promised. Build on the Google column; treat the other one as a working model that could be wrong.

Directive Google (documented) AI crawlers (observed / undocumented) Where it can hide
noindex Drops the URL from the index Page stops being retrievable, so it stops being cited <head>, X-Robots-Tag
nofollow Does not follow links from the page Appears to act as a weak quality hint <head>, X-Robots-Tag
none noindex, nofollow Both of the above Either
noarchive No cached copy Appears to be ignored Either
nosnippet No text fragment in results Appears to suppress direct quoting Either
absent index, follow Free to read and cite —

The bottom row is the one to aim for. An explicit <meta name="robots" content="index, follow"> is not a stronger yes than saying nothing; it is dead weight that some audits flag as a redundant directive.

FAQ

Does noindex block AI crawlers or only search engines?

It blocks the search index directly, and the AI citation as a consequence. Apple and Amazon are the only two vendors that document honouring noindex for their crawlers (Applebot and Amazonbot). OpenAI, Anthropic and Perplexity document robots.txt and say nothing about the robots meta tag, so "noindex means do not train on me" is folklore, not a vendor commitment. What still costs you the citation is narrower and certain: an assistant has to retrieve a page before it can quote it, and retrieval runs through a search index. Block the index and nothing finds you.

What is the difference between noindex and robots.txt Disallow?

Disallow stops the fetch; noindex stops the use. They are not interchangeable, and combining them backfires: a crawler blocked by robots.txt never fetches the page, so it never sees the noindex you put there. If you want a page excluded, let it be crawled and let the tag do the work.

Why does my CMS add noindex without asking?

Because staging defaults leak. The three routes are a staging environment promoted to production with its "discourage search engines" flag still on, an SEO plugin applying a rule to a whole post type, and a framework default in a shared layout that a new route inherits. All three are invisible in the editor.

Can noindex be set without appearing in the HTML?

Yes — via the X-Robots-Tag response header, which is the version that is genuinely hard to find. View Source shows nothing, the CMS shows nothing, and only the response headers give it away. Check with curl -I before concluding the HTML is clean.

How do I check a page quickly?

curl -sI https://example.com/page | grep -i x-robots-tag for the header, and a search for name="robots" in the served HTML for the tag. The AEO Checker extension checks both at once on whatever tab you are looking at.

Sources

Part of the Answer Engine Optimization cluster — the full reading order in dependency sequence, plus a glossary of every term used across these articles.

The summary

Three words in <head> can cost you all your AI traffic. One invisible header can do the same with no trace in the HTML.

The audit takes 30 seconds. The fix is one comma and two lines of code. The risk is your highest-leverage content silently invisible to the entire generation of AI search forming right now.

Follow me on LinkedIn for more on AEO and AI-search architecture. For an audit of your site's AI readiness, get in touch.

View as Markdown

Get new posts

One article every couple of weeks, on AI search and the code behind it. Confirm by email; unsubscribe in one click.

Related