How I Added llms.txt to My Next.js Blog in 15 Minutes

If ChatGPT and Perplexity can't find your best posts, you didn't give them a map. llms.txt is the map: a plain-Markdown file at your site's root, no magic and no black box, that tells the model exactly which pages matter and starts it reading there.

Key Takeaways#

• llms.txt is a curated Markdown index at a site's root that tells AI (artificial intelligence) agents which pages matter most, proposed by Jeremy Howard of Answer.AI on September 3, 2024.
• According to my August 2026 scan of 93 pages ranking for AEO (answer engine optimization) and GEO (generative engine optimization) queries, 52.7% of them, 49 of 93, sat on a domain serving a working /llms.txt file.
• File size varies from 648 bytes at llmstxt.org to 2,500,078 bytes at coursera.org, with a median of 8,072 bytes across the 41 domains that have one.
• Per the same scan, 12.2% of those domains (5 of 41) got the file automatically from an SEO plugin update, not a deliberate editorial decision.
• A Next.js route handler can generate the file at build time in about ten minutes; the failure to watch for is serving it with the wrong Content-Type.

What is llms.txt, actually?#

llms.txt is not an alternative to robots.txt, and not a replacement for sitemap.xml. Google's own crawling documentation is explicit that robots.txt exists mainly to avoid overloading a site with requests, and that it is not a mechanism for keeping a page out of Google. llms.txt does something unrelated: it's a separate genre, a curator's index for AI agents. Jeremy Howard, co-founder of Answer.AI, proposed it on September 3, 2024, and the formal spec now lives at llmstxt.org. In the sample I scanned on 2026-08-09, 52.7% of pages ranking for AEO and GEO queries now serve one. It solves a narrower problem than structuring your content with JSON-LD, the vocabulary published at schema.org: that markup describes what an individual page is, while llms.txt curates which pages matter at all.

The base logic is simple. Model context windows are too small to ingest most sites whole, and converting HTML, with its navigation, ads, and JavaScript, into clean text an LLM (Large Language Model) can read directly is expensive and unreliable. Parsing your HTML burns tokens for nothing. Instead, you hand the agent clean Markdown with a list of URLs and one-line descriptions. The agent sees the structure in two seconds. Far more importantly, it sees that structure through your eyes, not through the eyes of a sitemap generator that exports everything underneath.

Is llms.txt actually catching on?#

Yes, and it's now measurable. According to a scan I ran on August 9, 2026, using DataForSEO's live SERP API (Application Programming Interface), I pulled the top United States, English-language Google results for seven AEO- and GEO-related queries. The queries were: llms.txt, "answer engine optimization," "generative engine optimization," "how to get cited by ai," "geo vs seo," "aeo vs seo," and "ai visibility checker." After dropping platform pages like YouTube, Reddit, LinkedIn, and Wikipedia, 93 unique ranking pages remained, spread across 78 domains.

In that same scan, 49 of those 93 pages, 52.7%, sit on a domain that answers GET /llms.txt with a 200 status and a non-HTML content type, the same check the AEO Checker extension runs automatically. Those 49 page-level hits collapse to 41 unique domains, since a domain can rank for more than one query. File size varies enormously: the median is 8,072 bytes, the smallest is 648 bytes at llmstxt.org itself, the site behind the specification, and the largest is 2,500,078 bytes at coursera.org. OpenAI alone runs four separate crawlers for this kind of work, including GPTBot for model training and ChatGPT-User for live browsing, both listed in its official bots reference.

That is a real adoption rate for a file format that didn't exist before September 2024. It is not evidence that having one gets you cited more often. This scan measured presence, not influence on any AI answer; whatever pulls a page into a ChatGPT or Perplexity response is a mix of dozens of signals, several of them covered in the invisible tags shaping your AI citations.

Who's already on board?#

The honest answer splits into two groups: sites that added the file on purpose, and sites where a plugin added it for them. Both count in the 52.7% adoption number above, and the difference matters more than the headline stat does.

According to the same scan, five of the 41 domains with a working file, 12.2%, got it as a side effect of an SEO plugin update, not a deliberate decision. Two plugins account for all five:

• All in One SEO: aioseo.com and visiblefactors.com run Pro v5; fasturtle.com runs the free v5.
• Yoast SEO v27: boralagency.com and yoghurtdigital.com.

None of those five sites chose a curation strategy. Their plugin shipped a feature, auto-generated a file from existing post metadata, and the domain became a statistic in every future llms.txt survey, this one included. That is worth knowing before treating the adoption number as proof of deliberate intent.

My own file is hand-built rather than plugin-generated: 10,750 bytes, organized around a catalog of WebMCP tools alongside the usual writing and project links. You can read it directly at alexturik.com/llms.txt.

How does llms.txt differ from sitemap.xml?#

Sitemap says "here is everything I have." llms.txt says "here is what I exist for."

Structure: three sections decide everything#

# Site name
 
> One-line summary
 
## Section name
 
- [Page title](url): One-line description
- [Another page](url): Another description
 
## Optional
 
- [Less critical resource](url): Why it's optional

H1 with the name. A blockquote with one sentence that explains what the site is about. Then H2 sections with bulleted lists. The Optional section explicitly marks less-critical resources the agent can skip if context budget is tight. This is the formal part of the spec, so don't break it.

Two implementations for Next.js#

Two paths. Pick based on how often you publish: a file you maintain by hand, or a route handler that works as your own llms.txt generator.

Option A: static (5 minutes)#

Create public/llms.txt. Fill it by hand. Deploy.

# alexturik.com
 
> Practical notes on Next.js, AEO tooling, and shipping side projects fast.
 
## Featured Posts
 
- [How I Added llms.txt to My Next.js Blog in 15 Minutes](https://alexturik.com/blog/how-i-added-llms-txt-to-my-nextjs-blog-in-15-minutes): Curator's index for AI crawlers — implementation walkthrough.
- [The 3-Word Tag That Hides Your Best Pages from AI](https://alexturik.com/blog/the-3-word-tag-that-hides-your-best-pages-from-ai): How `noindex` silently kills AI traffic.
 
## About
 
- [Author bio](https://alexturik.com): Who I am and what I build.

Works. Simple. Hopelessly dead a week later, because you will guaranteed forget to update it.

Option B: dynamic, via a route handler (10 minutes)#

This is the one I run on this site. A route handler in the Next.js App Router reads frontmatter from every post and generates the markdown at build time. Write it once. It keeps itself in sync forever.

import { getAllPosts } from '@/lib/posts';
import { siteConfig } from '@/lib/site-config';
 
export const dynamic = 'force-static';
 
const HEADER = (url: string) => `# Oleksii Turovskyi
 
> Full-stack developer specializing in Next.js, React, Node.js, and WordPress. Builds rapid MVPs and maintains high-load systems.
 
## About
 
- **Name:** Oleksii Turovskyi
- **Role:** Full-stack developer
- **Email:** alexturik@gmail.com
- **Website:** ${url}
 
## Pages
 
- [Homepage](${url})
- [Blog](${url}/blog)
- [Sitemap](${url}/sitemap.xml)
`;
 
const FOOTER = `## Optional
 
- [GitHub](https://github.com/turovskiy)
- [LinkedIn](https://linkedin.com/in/alexturik)
- [X / Twitter](https://x.com/alexturik)
`;
 
export async function GET() {
  const url = siteConfig.url;
  const posts = await getAllPosts();
 
  const writingSection = posts.length === 0 ? '' :
    '\n## Writing\n\n' +
    posts
      .map((p) => {
        const html = `${url}/blog/${p.slug}`;
        const md = `${url}/blog/${p.slug}.md`;
        return `- [${p.frontmatter.title}](${html}) ([Markdown](${md})) — ${p.frontmatter.description}`;
      })
      .join('\n') +
    '\n';
 
  const body = `${HEADER(url)}${writingSection}\n${FOOTER}`;
 
  return new Response(body, {
    status: 200,
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=3600, s-maxage=86400',
    },
  });
}

The file lives at src/app/llms.txt/route.ts. Next.js automatically serves it at /llms.txt, with no rewrites and no manual config. Each blog post gets two links: the HTML version and a Markdown version (/blog/<slug>.md), so an agent that prefers raw text can grab content without parsing HTML.

That .md twin for every post is rarer than you'd think. According to the same scan, only 8.6% of the 93 ranking pages serve a raw Markdown version of themselves at a matching URL, while my own site carries 15 of them. It costs one extra line in the loader above and gives agents a version they can read without stripping a single HTML tag.

One nuance: if your post source is Contentlayer, Velite, Notion API, or Sanity, just swap out the getAllPosts() loader. The rest of the architecture doesn't change.

What goes in, what stays out#

llms.txt is not a sitemap. Duplicating it as a sitemap is a curation failure. The logic is the opposite: only what you want to see in AI citations goes here.

Worth including:

• Evergreen posts: the ones that age slowly and accumulate traffic over years.
• Tools/calculators you've built. Agents love linking to specific utilities.
• Author bio. Your personal positioning in the landscape, not a vanity entry.
• Pricing/services pages, if you run a commercial site.

Don't include:

• Legal pages (terms, privacy). Agents don't care, chatbot users don't either.
• Listing pages (/blog, /tags/foo). That's the classic sitemap's job.
• Login, account, dashboard: private space behind auth anyway.
• Category indexes without your own added value.

A short test: "Do I want this exact page cited as an authoritative source in a ChatGPT answer?" If the answer is anything other than yes, drop it.

Bonus tier: llms-full.txt#

If llms.txt is the table of contents, llms-full.txt is the entire book in a single file. Mintlify generates both formats automatically so documentation is readable by LLM models without extra fetch requests. The agent receives the whole context at once.

Should you do it? On a small archive it costs little and hands an agent the full text in one request. On a large one, weigh it against context limits: the biggest file in my scan, at coursera.org, runs 2,500,078 bytes against a median of 8,072, and a file that size gets truncated by the agent with its tail never read. I wrote one route handler that maps the same posts and concatenates the raw content from each MDX file. Same pattern as Option B above, just with body text instead of summaries.

What breaks an llms.txt file silently?#

Your server has to serve llms.txt with a Content-Type of text/plain or text/markdown. That header is used to indicate the media type of a resource, and if it says text/html instead, agents can quietly skip the file.

In the route handler above this is wired into the headers. In the static public/ variant, most CDN (Content Delivery Network) providers, including Vercel, auto-set text/plain for .txt extensions, but it's worth validating. While you're at it, double-check that none of your linked posts is silently hidden by a stray noindex directive, since llms.txt is only useful if the agent can actually fetch what you point at.

Check it with one terminal command:

curl -I https://yoursite.com/llms.txt

Look for Content-Type: text/plain or text/markdown in the response. If you see text/html, you have a config problem. Open issue.

Prefer not to run a terminal command for every deploy? The AEO Checker extension runs the same llms.txt MIME-type check automatically, alongside its robots.txt bot-allowlist audit.

This is the failure mode that's the easiest to ship and the hardest to notice. Drop a file in public/ and a CDN can still serve it as HTML, leaving it invisible to agents from day one while every browser check looks fine.

Part of the Answer Engine Optimization cluster — the full reading order in dependency sequence, plus a glossary of every term used across these articles.

Ready to check yours?

This very post is part of the llms.txt on alexturik.com, sufficient meta for you?

If you're on Next.js and haven't shipped this standard yet, the clock is running. Fifteen minutes of work. Zero downside. The reward is your best material being visible to the entire generation of AI traffic that's forming right now.

Follow me on LinkedIn. For an AEO audit or implementation help on your platform, get in touch.

FAQ#

What is an llms.txt file?#

An llms.txt file is a plain-Markdown file served at a site's root, such as yoursite.com/llms.txt, that gives AI agents a curated list of a site's most important pages, each with a one-line description. Jeremy Howard of Answer.AI proposed the format on September 3, 2024. It exists because parsing full HTML pages into clean text is slow and wastes tokens.

Is llms.txt a real thing?#

Yes. It has a public specification at llmstxt.org, and it measurably exists in the wild. In an August 2026 scan of 93 pages ranking for AEO and GEO search queries in the US, 49 of them, 52.7%, sat on a domain serving a working /llms.txt file, spread across 41 unique domains. It is not a hoax or a fringe experiment.

Are llms.txt worth it?#

Worth trying, with modest effort: the static version takes about five minutes and the dynamic Next.js route handler about ten. There is no proven citation boost; this scan measured adoption, not ranking impact. But roughly half of pages ranking for AEO and GEO queries already serve one, and a wrong MIME type will make the file invisible without raising any error.

What should be in an llms.txt file?#

Keep it short: an H1 with your site name, a one-line blockquote summary, then H2 sections listing your key pages with a one-line description each. Include evergreen posts, tools you built, and your author bio. Leave out legal pages, tag or category listings, and anything behind a login. An optional section at the end marks lower-priority links an agent can skip.

Is LLMs.txt necessary?#

No single file is mandatory. Nothing about SEO or AI visibility breaks if llms.txt is absent; it is a convenience layer on top of crawling, and no more required than robots.txt, which Google states a site is crawled fine without. But it is low-cost insurance: about fifteen minutes of setup, no server load, and it hands AI agents a reading list instead of forcing them to guess from a sitemap built for search engines, not language models.