OAI-SearchBot vs GPTBot vs ChatGPT-User: Which to Block

OpenAI's crawler documentation lists four crawlers: GPTBot for model training, OAI-SearchBot for ChatGPT search, ChatGPT-User for pages opened on a person's request, and OAI-AdsBot for ad landing pages. Only GPTBot and OAI-SearchBot are presented as robots.txt controls. For ChatGPT-User, OpenAI says robots.txt rules may not apply, and the OAI-AdsBot entry does not mention robots.txt.

OAI-SearchBot is the crawler behind ChatGPT's search features, and its robots.txt token decides whether your pages are eligible to appear in ChatGPT search answers. The full crawler list covers the other vendors.

TL;DR

• Block GPTBot to opt out of training. OpenAI says each robots.txt setting is independent, so blocking it does not affect your OAI-SearchBot setting.
• Keep OAI-SearchBot allowed to stay eligible for ChatGPT search. OpenAI recommends allowing it in robots.txt and allowing its published IP (Internet Protocol) ranges.
• In my scan of 59 readable robots.txt files on 18 October 2026, GPTBot was disallowed at the root in 6, ChatGPT-User in 5 and OAI-SearchBot in 3, and none named OAI-AdsBot.

What is OAI-SearchBot?

OAI-SearchBot is OpenAI's crawler for ChatGPT search. The documentation says it is used to surface websites in search results in ChatGPT's search features. Sites that opt out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI recommends allowing it in robots.txt, and says an update can take about 24 hours to take effect for search results.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot and OAI-SearchBot do different jobs and have separate robots.txt settings. OpenAI says GPTBot crawls content that may be used in training its foundation models, and disallowing it indicates content should not be used for training. OAI-SearchBot governs ChatGPT search. OpenAI's own example is to allow OAI-SearchBot for search results while disallowing GPTBot. If both are allowed, OpenAI may use one crawl for both purposes.

Does ChatGPT-User respect robots.txt?

OpenAI does not promise that it does. The documentation says: "Because these actions are initiated by a user, robots.txt rules may not apply." ChatGPT-User handles certain user actions in ChatGPT and Custom GPTs, and is not used to crawl the web automatically or to decide Search eligibility.

"May not apply" is neither "ignored" nor "honored", so plan for the worse reading. Disallowing ChatGPT-User has a trade-off: if OpenAI honours the line, ChatGPT will not open your page when a person asks it to, and if it does not, the line changes nothing. 5 of the 59 files in my scan disallow it at the root (reddit.com through its catch-all group). For a firm block, use authentication or a firewall rule on the IP ranges OpenAI publishes for ChatGPT-User.

What is OAI-AdsBot?

OAI-AdsBot checks the pages behind ChatGPT ads. OpenAI says it is used "to validate the safety of web pages submitted as ads on ChatGPT" and may use landing page content to decide when an ad is most relevant. The bot only visits pages submitted as ads, and its data is not used to train foundation models.

Search Engine Journal reported on 23 April 2026 that OpenAI's crawler page now listed the bot. It noted two gaps: the entry did not say how the bot treats robots.txt, and no adsbot.json IP list was linked. The second gap has closed, since the page I read on 18 October 2026 links openai.com/adsbot.json and that URL returns JSON (JavaScript Object Notation). The first gap remains: the page still says nothing about robots.txt for OAI-AdsBot.

No file in my scan names OAI-AdsBot, and the single block comes from the catch-all group on reddit.com. If you advertise on ChatGPT, do not add a rule you cannot test: the page does not say how a Disallow would affect ad review.

What do 59 real robots.txt files do with OpenAI's bots?

On 18 October 2026 I fetched robots.txt once from each of 65 domains, and 59 files were readable. The domains are the set behind the August baseline file: SEO tool vendors, publishers, developer documentation, Ukrainian agencies and e-commerce. For each token I took the group that names it, or else the * group, as RFC (Request for Comments) 9309 prescribes. A token counted as blocked when the longest matching rule for / was a Disallow. The requests identified themselves as alexturik-research/1.0 and linked to the research page.

According to this scan, sites name the training crawler 2.5 times as often as the search crawler: 15 files name GPTBot and 6 name OAI-SearchBot. Blocks are rarer but keep the ranking, with GPTBot disallowed at the root in 6 of 59 files, ChatGPT-User in 5 and OAI-SearchBot in 3. The reddit.com file names none of the four tokens and blocks all of them through its catch-all * group, so it is a blanket policy, not an OpenAI-specific choice.

Six domains treat OpenAI's bots differently from each other. Four differ across GPTBot, OAI-SearchBot and ChatGPT-User; bbc.com and nytimes.com differ only on OAI-AdsBot:

Allowlist means the group lists allowed pages and ends with Disallow: /. In the live file on 18 October 2026 figma.com uses an allowlist for OAI-SearchBot and ChatGPT-User, and the scan counts both as not blocked because Allow: /$ wins for the root. For bbc.com and nytimes.com the only difference is OAI-AdsBot: both disallow the other three tokens by name, but their * group does not block the root.

Every SEO tool vendor in the sample (ahrefs.com, semrush.com, moz.com, backlinko.com, searchengineland.com, searchenginejournal.com and yoast.com) blocks none of the four tokens at the root. The same holds for every Ukrainian agency, every readable developer-documentation site and alexturik.com.

The scan has limits you should read before quoting it:

• One snapshot, one day, one User-Agent: some sites serve a different robots.txt by User-Agent or region. Six files were unreadable, and one of them, forbes.com, blocked GPTBot in the August scan on this site, so the GPTBot count is probably an undercount.
• The 59 domains are a purposive sample around SEO, publishing and developer tooling, so the shares describe this sample only.
• A robots.txt file is a request, so the scan shows what sites ask for, not what the bots do.
• Not blocked at the root is not fully allowed: many files disallow specific paths for *, which this scan does not count.
• Do not read these counts against the earlier scans as a trend. The August scan on this site found GPTBot blocked in 3 of 53 readable files, but it counted a block only when a group named the token.

Which OpenAI bots should you block?

Decide by goal, not by bot.

Should I block GPTBot?

Block GPTBot if you do not want your content used to train OpenAI's models, and allow it if training use is acceptable. The choice does not touch your search visibility, because OpenAI says the settings are independent. Search is where the six sites in my scan that disallow GPTBot disagree. Three also disallow OAI-SearchBot: bbc.com, nytimes.com and reddit.com (through its catch-all group). Two leave it unblocked: medium.com and techcrunch.com. The sixth, figma.com, allows it only on a list of pages.

robots.txt recipes for OpenAI's crawlers

One rule decides how every recipe behaves: a crawler obeys the group that names its token and falls back to the * group only when no group does (RFC 9309, section 2.2.1). Google's robots.txt documentation states the consequence: "User agent specific groups and global groups (*) are not combined." OpenAI's page does not describe how its crawlers pick a group, so this is the standard's rule, not an OpenAI statement.

Naming a bot therefore replaces your * rules for that bot. If your * group closes /account/ and you add an OAI-SearchBot group containing only Allow: /, you have reopened /account/ for that bot.

Recipe 1: opt out of training only

# public/robots.txt
User-agent: GPTBot
Disallow: /

GPTBot now obeys this group alone, and every bot without its own group keeps following your existing * rules.

Recipe 2: open ChatGPT search when a catch-all group blocks everything

# public/robots.txt
User-agent: *
Disallow: /
 
User-agent: OAI-SearchBot
Allow: /

OAI-SearchBot follows its own group and may crawl everything, while every bot without a group follows the * group and is disallowed.

Recipe 3: ask all three tokens to stay out

# public/robots.txt
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /

Stacked User-agent lines share one group under the formal syntax in RFC 9309, section 2.2. This opts out of training and ChatGPT search, but ChatGPT-User may still open pages. OAI-AdsBot is left out on purpose, because OpenAI does not say it reads robots.txt and a site that never submits ads gets no visits from it.

Check the file before you deploy

Python's standard library can show how a parser reads the file. Save the file next to this script and run it:

# check_robots.py
from urllib.robotparser import RobotFileParser
 
parser = RobotFileParser()
with open("robots.txt", encoding="utf-8") as handle:
    parser.parse(handle.read().splitlines())
 
for token in ("GPTBot", "OAI-SearchBot", "ChatGPT-User", "OAI-AdsBot"):
    print(token, parser.can_fetch(token, "https://example.com/"))

With Recipe 1 saved as robots.txt, the script prints False for GPTBot and True for the other three tokens.

Apply a recipe in four steps:

1. Pick your goal: opt out of training, open ChatGPT search, or ask all three tokens to stay out.
2. Add the matching group to robots.txt, repeating your * path rules in any group that must keep them.
3. Run the check against the saved file.
4. Deploy, then wait a day before judging ChatGPT search.

How do you verify a request really came from OpenAI?

A user-agent string proves nothing, because any client can send it. OpenAI publishes IP ranges for each bot as JSON files: gptbot.json, searchbot.json, chatgpt-user.json and adsbot.json. Each holds a creationTime and prefixes in CIDR (Classless Inter-Domain Routing) notation. I fetched all four on 18 October 2026:

The chatgpt-user.json file has 13 times as many prefixes as gptbot.json (234 against 18) and 6 times as many as searchbot.json (234 against 39). Prefix counts are not address counts, though: in the 18 October file almost every chatgpt-user.json entry is a /28 block of 16 addresses, with one /17, while gptbot.json lists larger blocks such as 132.196.86.0/24.

The files also share prefixes. On 18 October 2026 the prefix 74.7.244.0/25 appeared in both gptbot.json and searchbot.json, so a matching address shows that traffic is OpenAI's but not which job it was doing. Comparing the full lists the same day, GPTBot and OAI-SearchBot share 6 identical prefixes, while ChatGPT-User and OAI-AdsBot share none with any other file. Read the token from the user-agent string, then check the address against the file for that token.

OpenAI's page documents these lists and describes no reverse DNS (Domain Name System) check. The files carry different creationTime values and the page I read gives no refresh interval, so fetch them on a schedule. The verification script in the crawler list takes an address and the vendor name openai, and walks all four files.

What to do next

Allowing OAI-SearchBot is one precondition of several. If you have allowed it and still do not appear, why your site does not show in ChatGPT and Perplexity walks through the others. Auditing your visibility to ChatGPT by hand shows how to check them. Tokens for the other vendors are in the full crawler list.

To check a live site, including one that is not yours, my AEO (Answer Engine Optimization) Checker extension reads the open tab's robots.txt: blocking OAI-SearchBot fails the check, blocking ChatGPT-User warns, and GPTBot is listed but costs nothing.

FAQ

What are the top 3 AI bots?

There is no single ranking of AI (artificial intelligence) bots, because the order depends on what you measure. For OpenAI, the three tokens that site owners write rules for are GPTBot, OAI-SearchBot and ChatGPT-User, named in 15, 6 and 9 of the 59 files in my scan. A cross-vendor ranking by traffic needs a dated measurement source that this scan lacks.

How do I recognize OAI-SearchBot in my logs?

Match the token OAI-SearchBot in the user-agent string. On 18 October 2026 OpenAI's page listed a string containing OAI-SearchBot/1.4 and noted that the version may change. When the bot fetches robots.txt, the string may carry an extra robots.txt marker. Then confirm the source address against searchbot.json, because anyone can send the string.

Is OAI-AdsBot a training crawler?

No. OpenAI's page says OAI-AdsBot only visits pages submitted as ads on ChatGPT and that the data it collects is not used to train its foundation models. The page does not say whether it reads robots.txt, so a Disallow line for it has no documented effect.

Sources

• OpenAI, Overview of OpenAI crawlers: the four bots, what each robots.txt setting controls, and the links to the IP lists
• OpenAI, gptbot.json: GPTBot IP ranges
• OpenAI, searchbot.json: OAI-SearchBot IP ranges
• OpenAI, chatgpt-user.json: ChatGPT-User IP ranges
• OpenAI, adsbot.json: OAI-AdsBot IP ranges
• IETF, RFC 9309 Robots Exclusion Protocol: section 2.2 on stacked User-agent lines, section 2.2.1 on group selection, and the statement that its rules are not access authorization
• Google Search Central, How Google interprets the robots.txt specification: user-agent groups and the * group are not combined
• Search Engine Journal, OpenAI's crawler docs now list OAI-AdsBot for ChatGPT ads: the 23 April 2026 report and its two gaps

Part of the Answer Engine Optimization cluster, with the full reading order in dependency sequence, plus a glossary of every term used across these articles.

Not sure which OpenAI bots your robots.txt lets in?

Follow me on LinkedIn to keep up with the technical side of optimization for AI. If your site needs a robots.txt review or an AI-visibility audit, get in touch directly.