GPTBot Explained: User Agent, IP Ranges, robots.txt, and How to Track It

GPTBot is OpenAI’s web crawler for collecting training data. It crawls publicly available pages so their content “may be used in training OpenAI’s generative AI foundation models” — the models behind ChatGPT. It identifies itself honestly, respects robots.txt, and publishes its IP ranges, which makes it one of the better-behaved AI crawlers — and one of the easiest to verify.

It is also, for most sites, the most frequent AI visitor in the server logs. Here’s everything you need to know about it: how to recognize it, how to verify it isn’t being spoofed, how to control it, and whether you should.

GPTBot at a glance

The current full user-agent string:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

The version number changes over time (1.0 → 1.1 → 1.2 → 1.4), so match on the GPTBot token, not the full string.

GPTBot is one of four OpenAI crawlers

The single most common misconception about GPTBot: that blocking it removes you from ChatGPT. It doesn’t, because OpenAI operates four different bots with different jobs:

The consequences of this split:

• Blocking GPTBot opts you out of training crawls only. ChatGPT search can still index and cite your site through OAI-SearchBot.
• Blocking OAI-SearchBot removes you from ChatGPT search citations — usually the opposite of what a marketing site wants.
• You can’t fully block ChatGPT-User via robots.txt, because OpenAI treats those fetches as acting on behalf of a human, like a browser.

Each bot has its own published IP list: gptbot.json, searchbot.json, chatgpt-user.json, adsbot.json.

Finding GPTBot in your server logs

If you have access to raw access logs (nginx, Apache, CDN logs), a grep is enough to see whether GPTBot visits and what it reads:

grep -i "gptbot" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

That prints your 20 most-crawled paths. Typical patterns: GPTBot favors documentation, blog posts, and comparison pages — text-dense pages with stable URLs — and recrawls them on an irregular cadence ranging from days to weeks depending on your site’s size and change frequency.

Two things a grep can’t tell you, though:

1. Whether the traffic is really GPTBot. Anything can send that user-agent string.
2. Anything at all, if you only have JavaScript analytics. GPTBot doesn’t execute JavaScript, so it never runs your analytics snippet. Google Analytics, Plausible, and every other tag-based tool report exactly zero AI crawler traffic — not because there is none, but because they physically cannot see it.

Verifying real GPTBot (and catching spoofers)

Scrapers routinely impersonate well-known crawlers to slip past bot rules, and GPTBot is now a popular disguise. Verification is straightforward because OpenAI publishes its egress IP ranges as machine-readable JSON:

curl -s https://openai.com/gptbot.json

A request claiming to be GPTBot from an IP outside those CIDR blocks is spoofed. If you’re checking by hand: take the client IP from the log line, and test it against the published ranges.

flowchart TD
A["Incoming request<br/>UA contains 'GPTBot'"] --> B{"Web Bot Auth<br/>signature present?"}
B -->|"Yes — signature valid"| V["Verified GPTBot"]
B -->|"No signature"| C{"Source IP inside<br/>openai.com/gptbot.json ranges?"}
C -->|"Yes"| V
C -->|"No"| S["Spoofed — a scraper<br/>impersonating GPTBot"]

Doing this continuously — fetching current ranges for every AI vendor, matching every hit, flagging mismatches — is exactly the kind of job that should be automated. Kitbase’s bot & crawler detection classifies every forwarded request server-side and stores each crawler visit with a verified or spoofed verdict, using vendors’ published IP ranges and cryptographic Web Bot Auth signatures where crawlers send them. Setup is a copy-paste forwarder for Next.js, WordPress, Cloudflare Workers, Vercel, nginx, and more — human visitors are classified in memory and discarded, only bots are stored.

robots.txt recipes

GPTBot honors robots.txt, so control is one text file away. Pick your stance:

Block training crawls, stay in ChatGPT search (the most common middle ground):

User-agent: GPTBot
Disallow: /

Allow everything (the default if you say nothing — no rule needed, shown explicit for clarity):

User-agent: GPTBot
Allow: /

Block training from sensitive sections only:

User-agent: GPTBot
Disallow: /pricing-internal/
Disallow: /customers/
Allow: /

Block all OpenAI crawling including search (full opt-out — this removes you from ChatGPT search citations):

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

Three caveats worth knowing:

• robots.txt is not retroactive. Blocking GPTBot today stops future crawls; it doesn’t remove content already collected.
• robots.txt is honor-system. GPTBot complies, but spoofers ignoring your rules are precisely why verification and monitoring matter.
• ChatGPT-User is exempt. User-triggered fetches can bypass robots.txt; if you need those blocked too, you’ll need firewall rules on OpenAI’s published chatgpt-user.json ranges.

Should you block GPTBot?

It depends entirely on what your content is worth to you in each channel:

Consider blocking if your content is the product — paywalled journalism, proprietary research, licensed data. For you, training crawls extract value without attribution or traffic, and an opt-out (or a licensing negotiation) is rational.

Don’t block if you sell something and your site exists to be discovered. ChatGPT recommends products it knows about. Being present in training data and search indexes is how your brand ends up in AI answers when a buyer asks “what’s the best tool for X?” — a channel that’s growing while traditional search clicks shrink. Blocking AI crawlers here is anti-marketing: you’re removing yourself from the shelf. (This is the core insight behind Generative Engine Optimization.)

Either way, measure first. Before deciding, look at what GPTBot actually reads on your site and how often. A decision about AI crawlers made without crawler data is a guess.

What to watch once you’re tracking it

With crawler detection in place, GPTBot stops being an abstraction and becomes a dataset. The useful signals:

• Which pages get crawled most — these are the pages OpenAI’s models learn about you from. If your best comparison content isn’t on the list, it may be unreachable, noindexed, or too new.
• Crawl trends — a spike often precedes or follows model training cycles; a sudden stop can mean an accidental block (a WAF rule, a CDN bot-protection toggle) you’d otherwise never notice.
• Verified vs. spoofed ratio — a rising share of spoofed “GPTBot” traffic is scraper activity wearing a mask, which is a security signal, not an AI one.
• The other side of the funnel — crawls are the input; the output is whether AI answers actually mention and cite you, which is a separate dataset entirely: AI Visibility measures it directly by querying the engines.

FAQ

Does GPTBot respect robots.txt? Yes — OpenAI documents it and real-world behavior confirms it. Remember it’s the training opt-out only; OAI-SearchBot and ChatGPT-User are separate rules.

Does blocking GPTBot remove my site from ChatGPT? No. It stops future training crawls. ChatGPT search citations come from OAI-SearchBot’s index, and previously crawled content isn’t retroactively removed.

Does GPTBot execute JavaScript? No. It fetches server-rendered HTML. Content that only exists after client-side rendering is invisible to it — one more argument for SSR on marketing and documentation pages, and the reason JS-based analytics can’t see it.

How do I know if GPTBot traffic is real? Check the source IP against openai.com/gptbot.json. Any “GPTBot” request from outside those ranges is an impostor. Kitbase does this automatically and labels every visit verified or spoofed.

How often does GPTBot crawl a site? There’s no published schedule. Observed cadence ranges from multiple visits per day on large, frequently-updated sites to weeks between visits on small ones. Tracking it is the only way to know your number.

Want to see which AI crawlers visit your site — verified, attributed, and per-page? Start your free trial — 7 days, no credit card required — and the setup wizard will have your first crawler data flowing in minutes.