---
title: "OpenAI's Four Crawlers: GPTBot vs OAI-SearchBot vs ChatGPT-User vs OAI-AdsBot | Kitbase Blog"
description: "OpenAI runs four bots, not one. Compare GPTBot, OAI-SearchBot, ChatGPT-User, and OAI-AdsBot: exact user agents, IP verification, and which robots.txt rule affects what."
canonical: https://kitbase.dev/blog/openai-crawlers-explained/
---
OpenAI operates **four separate bots**, and the single most expensive mistake site owners make is treating them as one. Blocking the wrong one removes you from ChatGPT's search citations; blocking the right one only opts you out of training. Each bot has its own user-agent, its own IP range file, its own robots.txt token, and — critically — its own effect on a *different* ChatGPT surface.
Our [GPTBot deep-dive](/blog/gptbot-explained/) covers the training crawler on its own. This guide goes wider: all four bots, exactly what each one controls, how to verify each against OpenAI's published IP ranges, and the misconfigurations that quietly cost visibility. OpenAI documents all four at [developers.openai.com/api/docs/bots](https://developers.openai.com/api/docs/bots).
## The four bots, side by side
| Bot | Job | robots.txt token | Respects robots.txt | IP ranges |
|---|---|---|---|---|
| **GPTBot** | Crawls content for **model training** | `GPTBot` | Yes | [gptbot.json](https://openai.com/gptbot.json) |
| **OAI-SearchBot** | Indexes pages for **ChatGPT search** results and citations | `OAI-SearchBot` | Yes | [searchbot.json](https://openai.com/searchbot.json) |
| **ChatGPT-User** | Fetches a page **when a user asks** ChatGPT to read or act on it | `ChatGPT-User` | No — user-triggered | [chatgpt-user.json](https://openai.com/chatgpt-user.json) |
| **OAI-AdsBot** | Validates **ad landing pages** | `OAI-AdsBot` | Yes | [adsbot.json](https://openai.com/adsbot.json) |
The exact user-agent strings OpenAI publishes:
```
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot
```
Version numbers change (GPTBot and OAI-SearchBot are both on 1.4 at time of writing), so **match on the token** — `GPTBot`, `OAI-SearchBot`, `ChatGPT-User`, `OAI-AdsBot` — not the full string. OAI-SearchBot is the one to watch: its string is a full Chrome UA with the token appended, so a naive "does the UA start with `Mozilla/5.0 (Macintosh`" filter will misclassify it as a human browser.
## Which robots.txt rule affects which ChatGPT surface
This is the part worth internalizing. Each bot maps to a different place your brand can appear — or disappear — inside ChatGPT:
**Each OpenAI bot controls a different ChatGPT surface**
```mermaid
flowchart LR
A["GPTBot"] --> A2["Training corpus
future models 'know' you"]
B["OAI-SearchBot"] --> B2["ChatGPT search index
citations in answers"]
C["ChatGPT-User"] --> C2["Live fetch when a user
pastes a URL / asks to browse"]
D["OAI-AdsBot"] --> D2["Ad landing-page
validation"]
```
Reading it as a table of consequences:
- **Block `GPTBot`** → you opt out of *training* crawls. Future OpenAI models are less likely to have learned about you from your own site. ChatGPT search can still find and cite you, because that's a different bot.
- **Block `OAI-SearchBot`** → you remove yourself from *ChatGPT's search index*. When someone asks ChatGPT a question that triggers web search, your pages can't be retrieved or cited. For a marketing site this is almost always the wrong move.
- **Block `ChatGPT-User`** → mostly symbolic. OpenAI treats it as acting on behalf of a human, so it may bypass robots.txt; a `Disallow` line is a request, not a guarantee. If you truly need to stop those fetches, you need a firewall rule on the `chatgpt-user.json` ranges.
- **Block `OAI-AdsBot`** → only relevant if you run ads whose landing pages OpenAI needs to validate.
The upshot: "block ChatGPT" is not a thing you can do with one line, and the intuitive one-liner (`User-agent: GPTBot / Disallow: /`) does *not* remove you from ChatGPT's answers. It only touches training.
## Verifying each bot against OpenAI's IP ranges
A user-agent string is a claim anyone can copy. OpenAI publishes egress IP ranges as machine-readable JSON — one file per bot — so you can confirm each hit:
```bash
curl -s https://openai.com/gptbot.json
curl -s https://openai.com/searchbot.json
curl -s https://openai.com/chatgpt-user.json
curl -s https://openai.com/adsbot.json
```
The verification workflow is identical for all four:
1. Take the client IP from the log line.
2. Read the token from the user-agent (`GPTBot`, `OAI-SearchBot`, …).
3. Fetch the matching JSON file and test the IP against its CIDR blocks.
4. A token-vs-IP mismatch — say, an `OAI-SearchBot` UA from an IP that's in nobody's range — is a spoof.
The subtlety is that you must check the IP against the **right** file. A request wearing the `GPTBot` UA should match `gptbot.json`, not `searchbot.json`; OpenAI segregates the ranges by bot, and a scraper that copies one bot's UA rarely lands inside any of them. Doing this by hand across four files, kept current, for every hit, is exactly the toil worth automating — see the [Googlebot verification walkthrough](/blog/verify-googlebot-spoofing/) for the same pattern applied to Google, and [Kitbase's crawler detection](https://docs.kitbase.dev/crawler-detection), which matches every forwarded request against all vendors' current ranges and stores a verified-or-spoofed verdict server-side.
## Common misconfigurations (and how to catch them)
These are the five failures we see most, all of them silent — nothing errors, you just quietly lose ground:
**1. "We blocked GPTBot, so we're out of ChatGPT."** You're out of *training*, not ChatGPT. Search citations run through OAI-SearchBot. If your goal was privacy, fine; if your goal was to disappear from ChatGPT answers, you didn't.
**2. A blanket "block all OpenAI" rule that catches OAI-SearchBot.** Rules copied from "how to block AI" listicles often disallow every OpenAI token, including the search bot. The result: you keep training (arguably the one you cared about) *and* delete yourself from ChatGPT search citations. Read every token in a pasted robots.txt block before you ship it — our [robots.txt for AI crawlers](/blog/robots-txt-ai-crawlers/) guide breaks down each line.
**3. Assuming robots.txt stops ChatGPT-User.** It doesn't reliably. Teams add a `Disallow` for `ChatGPT-User`, see continued fetches, and conclude OpenAI is "ignoring robots.txt" — when OpenAI documents exactly this behavior. Use IP-range firewall rules if you need a hard stop.
**4. Matching on the full user-agent string.** Pin your allow/deny or your analytics classification to `GPTBot/1.4` and it silently breaks the day OpenAI ships `GPTBot/1.5`. Always match the token.
**5. A WAF or CDN bot-protection toggle blocking all four.** The most common cause of "we suddenly vanished from AI answers" is not robots.txt at all — it's a bot-management setting at the edge that started 403-ing OpenAI's ranges. Because these bots don't run JavaScript, your analytics won't show the drop; the only way to catch it is [server-side crawler detection](https://docs.kitbase.dev/crawler-detection) watching the crawl trend.
## A sane default configuration
For most businesses that want to be discoverable in AI while opting out of training, the configuration is:
```
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: OAI-AdsBot
Allow: /
```
That blocks the training crawl, keeps you fully indexed for ChatGPT search citations, and leaves ad validation alone. ChatGPT-User is deliberately omitted, because a `Disallow` there is unreliable anyway. If your content *is* the product (paywalled journalism, licensed data), you'll want the stricter posture in [should you block AI crawlers](/blog/should-you-block-ai-crawlers/).
Whatever you choose, verify it against reality. The gap between "what our robots.txt says" and "what OpenAI's bots actually did on our server" is where visibility leaks — and only crawl data closes it. The complementary dataset is [AI Visibility](https://docs.kitbase.dev/ai-visibility), which shows whether ChatGPT's answers actually mention and cite you once these bots have done their job.
## FAQ
**How many crawlers does OpenAI have?**
Four: GPTBot (training), OAI-SearchBot (ChatGPT search index and citations), ChatGPT-User (user-triggered page fetches), and OAI-AdsBot (ad landing-page validation). Each has its own user-agent token and published IP ranges.
**Which OpenAI bot should I allow to appear in ChatGPT search?**
OAI-SearchBot. It builds the index ChatGPT uses when it searches the web, so blocking it removes you from those citations. GPTBot only affects training.
**Does blocking GPTBot remove me from ChatGPT?**
No. It opts you out of training crawls. ChatGPT's search citations come from OAI-SearchBot's index, and content already trained on isn't retroactively removed.
**Why does OAI-SearchBot look like a normal Chrome browser?**
Its user-agent is a full Chrome string with `OAI-SearchBot/1.4` appended. Match on the `OAI-SearchBot` token, and verify against [searchbot.json](https://openai.com/searchbot.json) — don't classify it by the Chrome prefix or you'll mistake it for a human.
**Can I block ChatGPT-User with robots.txt?**
Not reliably. OpenAI treats it as user-initiated and may bypass robots.txt. Use a firewall rule against the published [chatgpt-user.json](https://openai.com/chatgpt-user.json) ranges if you need a hard block.
---
*Want to see which of OpenAI's four bots read your site — verified, per-page, and trended? [Start your free trial](https://app.kitbase.dev/signup/) — 7 days, no credit card required — and catch an accidental block before it costs you citations.*