OpenAI's Four Crawlers: GPTBot vs OAI-SearchBot vs ChatGPT-User vs OAI-AdsBot

OpenAI operates four separate bots, and the single most expensive mistake site owners make is treating them as one. Blocking the wrong one removes you from ChatGPT’s search citations; blocking the right one only opts you out of training. Each bot has its own user-agent, its own IP range file, its own robots.txt token, and — critically — its own effect on a different ChatGPT surface.

Our GPTBot deep-dive covers the training crawler on its own. This guide goes wider: all four bots, exactly what each one controls, how to verify each against OpenAI’s published IP ranges, and the misconfigurations that quietly cost visibility. OpenAI documents all four at developers.openai.com/api/docs/bots.

The four bots, side by side

Bot	Job	robots.txt token	Respects robots.txt	IP ranges
GPTBot	Crawls content for model training	`GPTBot`	Yes	gptbot.json
OAI-SearchBot	Indexes pages for ChatGPT search results and citations	`OAI-SearchBot`	Yes	searchbot.json
ChatGPT-User	Fetches a page when a user asks ChatGPT to read or act on it	`ChatGPT-User`	No — user-triggered	chatgpt-user.json
OAI-AdsBot	Validates ad landing pages	`OAI-AdsBot`	Yes	adsbot.json

The exact user-agent strings OpenAI publishes:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

Version numbers change (GPTBot and OAI-SearchBot are both on 1.4 at time of writing), so match on the token — GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot — not the full string. OAI-SearchBot is the one to watch: its string is a full Chrome UA with the token appended, so a naive “does the UA start with Mozilla/5.0 (Macintosh” filter will misclassify it as a human browser.

Which robots.txt rule affects which ChatGPT surface

This is the part worth internalizing. Each bot maps to a different place your brand can appear — or disappear — inside ChatGPT:

flowchart LR
A["GPTBot"] --> A2["Training corpus<br/>future models 'know' you"]
B["OAI-SearchBot"] --> B2["ChatGPT search index<br/>citations in answers"]
C["ChatGPT-User"] --> C2["Live fetch when a user<br/>pastes a URL / asks to browse"]
D["OAI-AdsBot"] --> D2["Ad landing-page<br/>validation"]

Each OpenAI bot controls a different ChatGPT surface

Reading it as a table of consequences:

Block GPTBot → you opt out of training crawls. Future OpenAI models are less likely to have learned about you from your own site. ChatGPT search can still find and cite you, because that’s a different bot.
Block OAI-SearchBot → you remove yourself from ChatGPT’s search index. When someone asks ChatGPT a question that triggers web search, your pages can’t be retrieved or cited. For a marketing site this is almost always the wrong move.
Block ChatGPT-User → mostly symbolic. OpenAI treats it as acting on behalf of a human, so it may bypass robots.txt; a Disallow line is a request, not a guarantee. If you truly need to stop those fetches, you need a firewall rule on the chatgpt-user.json ranges.
Block OAI-AdsBot → only relevant if you run ads whose landing pages OpenAI needs to validate.

The upshot: “block ChatGPT” is not a thing you can do with one line, and the intuitive one-liner (User-agent: GPTBot / Disallow: /) does not remove you from ChatGPT’s answers. It only touches training.

Verifying each bot against OpenAI’s IP ranges

A user-agent string is a claim anyone can copy. OpenAI publishes egress IP ranges as machine-readable JSON — one file per bot — so you can confirm each hit:

curl -s https://openai.com/gptbot.json
curl -s https://openai.com/searchbot.json
curl -s https://openai.com/chatgpt-user.json
curl -s https://openai.com/adsbot.json

The verification workflow is identical for all four:

Take the client IP from the log line.
Read the token from the user-agent (GPTBot, OAI-SearchBot, …).
Fetch the matching JSON file and test the IP against its CIDR blocks.
A token-vs-IP mismatch — say, an OAI-SearchBot UA from an IP that’s in nobody’s range — is a spoof.

The subtlety is that you must check the IP against the right file. A request wearing the GPTBot UA should match gptbot.json, not searchbot.json; OpenAI segregates the ranges by bot, and a scraper that copies one bot’s UA rarely lands inside any of them. Doing this by hand across four files, kept current, for every hit, is exactly the toil worth automating — see the Googlebot verification walkthrough for the same pattern applied to Google, and Kitbase’s crawler detection, which matches every forwarded request against all vendors’ current ranges and stores a verified-or-spoofed verdict server-side.

Common misconfigurations (and how to catch them)

These are the five failures we see most, all of them silent — nothing errors, you just quietly lose ground:

1. “We blocked GPTBot, so we’re out of ChatGPT.” You’re out of training, not ChatGPT. Search citations run through OAI-SearchBot. If your goal was privacy, fine; if your goal was to disappear from ChatGPT answers, you didn’t.

2. A blanket “block all OpenAI” rule that catches OAI-SearchBot. Rules copied from “how to block AI” listicles often disallow every OpenAI token, including the search bot. The result: you keep training (arguably the one you cared about) and delete yourself from ChatGPT search citations. Read every token in a pasted robots.txt block before you ship it — our robots.txt for AI crawlers guide breaks down each line.

3. Assuming robots.txt stops ChatGPT-User. It doesn’t reliably. Teams add a Disallow for ChatGPT-User, see continued fetches, and conclude OpenAI is “ignoring robots.txt” — when OpenAI documents exactly this behavior. Use IP-range firewall rules if you need a hard stop.

4. Matching on the full user-agent string. Pin your allow/deny or your analytics classification to GPTBot/1.4 and it silently breaks the day OpenAI ships GPTBot/1.5. Always match the token.

5. A WAF or CDN bot-protection toggle blocking all four. The most common cause of “we suddenly vanished from AI answers” is not robots.txt at all — it’s a bot-management setting at the edge that started 403-ing OpenAI’s ranges. Because these bots don’t run JavaScript, your analytics won’t show the drop; the only way to catch it is server-side crawler detection watching the crawl trend.

A sane default configuration

For most businesses that want to be discoverable in AI while opting out of training, the configuration is:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: OAI-AdsBot
Allow: /

That blocks the training crawl, keeps you fully indexed for ChatGPT search citations, and leaves ad validation alone. ChatGPT-User is deliberately omitted, because a Disallow there is unreliable anyway. If your content is the product (paywalled journalism, licensed data), you’ll want the stricter posture in should you block AI crawlers.

Whatever you choose, verify it against reality. The gap between “what our robots.txt says” and “what OpenAI’s bots actually did on our server” is where visibility leaks — and only crawl data closes it. The complementary dataset is AI Visibility, which shows whether ChatGPT’s answers actually mention and cite you once these bots have done their job.

FAQ

How many crawlers does OpenAI have? Four: GPTBot (training), OAI-SearchBot (ChatGPT search index and citations), ChatGPT-User (user-triggered page fetches), and OAI-AdsBot (ad landing-page validation). Each has its own user-agent token and published IP ranges.

Which OpenAI bot should I allow to appear in ChatGPT search? OAI-SearchBot. It builds the index ChatGPT uses when it searches the web, so blocking it removes you from those citations. GPTBot only affects training.

Does blocking GPTBot remove me from ChatGPT? No. It opts you out of training crawls. ChatGPT’s search citations come from OAI-SearchBot’s index, and content already trained on isn’t retroactively removed.

Why does OAI-SearchBot look like a normal Chrome browser? Its user-agent is a full Chrome string with OAI-SearchBot/1.4 appended. Match on the OAI-SearchBot token, and verify against searchbot.json — don’t classify it by the Chrome prefix or you’ll mistake it for a human.

Can I block ChatGPT-User with robots.txt? Not reliably. OpenAI treats it as user-initiated and may bypass robots.txt. Use a firewall rule against the published chatgpt-user.json ranges if you need a hard block.

Want to see which of OpenAI’s four bots read your site — verified, per-page, and trended? Start your free trial — 7 days, no credit card required — and catch an accidental block before it costs you citations.