robots.txt for AI Crawlers: Copy-Paste Templates for Every Stance

Controlling AI crawlers with robots.txt comes down to writing Disallow rules against the right user-agent tokens — GPTBot for OpenAI training, ClaudeBot for Anthropic training, PerplexityBot for Perplexity, and so on. This guide gives you copy-paste templates for four common stances, a verified table of every major token, and the caveats that decide whether those rules actually do anything. The short version: robots.txt is a request, not a wall, and it only controls the well-behaved bots that choose to honor it.

The one rule that changes everything

Before any template: AI vendors run different bots for different jobs, and each has its own robots.txt token. Blocking one does not block the others. OpenAI alone runs four. Anthropic runs three. This is a feature, not a nuisance — it is exactly what lets you block training while staying citable in AI search. Get the tokens right and robots.txt becomes a scalpel; get them wrong and you either block nothing or accidentally remove yourself from AI answers.

Verified token table

These are the current, verified user-agent tokens you write rules against. “Training” bots feed model training corpora; “search/index” bots feed live AI answers and citations; “user fetch” bots retrieve a page because a human asked the assistant to.

Vendor	Token	Job	Honors robots.txt
OpenAI	`GPTBot`	Training	Yes
OpenAI	`OAI-SearchBot`	Search / index (ChatGPT search)	Yes
OpenAI	`ChatGPT-User`	User-triggered fetch	No — may bypass
Anthropic	`ClaudeBot`	Training	Yes
Anthropic	`Claude-SearchBot`	Search / index	Yes
Anthropic	`Claude-User`	User-triggered fetch	Yes
Perplexity	`PerplexityBot`	Search / index	Yes
Perplexity	`Perplexity-User`	User-triggered fetch	Often bypasses
Google	`Googlebot`	Search (also feeds AI Overviews)	Yes
Google	`Google-Extended`	AI-training opt-out token (no crawler)	Yes
Apple	`Applebot`	Search	Yes
Apple	`Applebot-Extended`	AI-training opt-out token (no crawler)	Yes
Common Crawl	`CCBot`	Training corpus (used by many AI labs)	Yes
ByteDance	`Bytespider`	Training	Inconsistent
Amazon	`Amazonbot`	Search / assistant	Yes
Meta	`Meta-ExternalAgent`	Training	Yes

Three things worth internalizing from this table:

Google-Extended and Applebot-Extended are not crawlers. They are opt-out signals. Adding a Disallow for them excludes your content from Google/Apple AI training without touching normal search crawling by Googlebot/Applebot. We unpack this in Google-Extended vs. Googlebot.
User-fetch bots are the exception to robots.txt. OpenAI’s ChatGPT-User and Perplexity’s Perplexity-User fetch a page because a human asked, and vendors often treat that like a browser request that bypasses robots.txt. Anthropic is the notable outlier — it says Claude-User does honor robots.txt.
Match on the token, not the full user-agent string. Version numbers change (GPTBot/1.4 today, something else tomorrow). The token stays stable.

Template 1: allow everything (the default)

If your robots.txt says nothing about a bot, it is allowed. You only need this template to be explicit for documentation or auditing purposes:

# Allow all crawlers, including AI
User-agent: *
Allow: /

This is the right default for most commercial sites: AI answers are a discovery channel, and being crawlable is how you end up in them. If you are unsure whether to block anything, see our decision framework — but the honest default is “allow, then measure.”

Template 2: block training, stay in AI search

The most popular middle ground. You keep your content out of model-training corpora while remaining fully citable in AI search — so you still show up in ChatGPT search, Perplexity, and Claude answers:

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

# Opt out of Google & Apple AI training (search unaffected)
User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Leave search/retrieval bots ALLOWED (no rule = allowed):
# OAI-SearchBot, PerplexityBot, Claude-SearchBot

Note what is deliberately absent: no rules for OAI-SearchBot, PerplexityBot, or Claude-SearchBot. That is intentional — those bots feed the citations that send you referral traffic. Blocking them is almost always a mistake.

Template 3: block all AI, keep normal search

For sites that want out of AI entirely — training and answers — while staying in traditional Google/Bing search:

# OpenAI — training + search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /

# Anthropic — training + search
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /

# Perplexity
User-agent: PerplexityBot
Disallow: /

# Others
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /

# Googlebot / Bingbot left allowed for normal search

Be clear-eyed about the trade-off: this removes you from AI citations and referral traffic. It makes sense when your content is the product — paywalled, licensed, or proprietary — and far less sense for a site that exists to be discovered.

Template 4: block sensitive sections only

The surgical option — welcome AI crawlers to your marketing and docs, but fence off areas that should never end up in a corpus:

User-agent: GPTBot
Disallow: /internal/
Disallow: /customers/
Disallow: /staging/
Allow: /

User-agent: ClaudeBot
Disallow: /internal/
Disallow: /customers/
Disallow: /staging/
Allow: /

The caveats that decide whether any of this works

Templates are the easy part. These four caveats determine whether your rules have any effect:

robots.txt is an honor system. Reputable bots — GPTBot, ClaudeBot, PerplexityBot — comply because their operators choose to. A scraper spoofing GPTBot reads your Disallow and ignores it. If your goal is to stop a bot rather than ask it to leave, you need firewall or edge rules on verified IP ranges, not a text file.
It is not retroactive. Blocking GPTBot today stops future training crawls; it does nothing about content already collected into models that shipped last year.
User-triggered fetchers are exempt. As the table shows, ChatGPT-User and often Perplexity-User bypass robots.txt because a human asked for the page. To block those you need IP-level rules on the vendor’s published ranges.
A syntax mistake fails silently. A typo’d token, a stray Allow: / overriding your Disallow, or a robots.txt returning a 500 — none of these throw an error. The bot just keeps crawling, and you never find out from the file itself.

That last point is the crux: robots.txt gives you no feedback. You write a rule and hope. The only way to confirm a rule is working is to watch what the crawlers actually do afterward.

How to confirm compliance

Compliance verification is an observation problem, and JavaScript analytics can’t help — AI crawlers fetch your HTML without ever running your analytics tag, so tag-based tools report zero AI crawler traffic whether your rules work or not. The signal lives in server and edge requests.

The manual approach is to grep your access logs for a token before and after a rule change and watch the count drop. The continuous approach is server-side crawler detection: forward your server or edge requests to Kitbase, and every AI crawler visit is stored with its name, vendor, the pages it hit, and a verified-or-spoofed verdict. After you add a Disallow for GPTBot, verified GPTBot visits to the blocked paths should taper to zero — and if they don’t, you’ve either got a syntax error or a spoofer, and the verdict tells you which. That is the difference between hoping a rule works and knowing it does.

For the specifics of individual bots referenced above, see GPTBot explained and the full list of AI crawlers.

FAQ

What is the robots.txt token to block GPTBot? User-agent: GPTBot followed by Disallow: /. That blocks OpenAI’s training crawler only — ChatGPT search (OAI-SearchBot) is a separate token and keeps citing you unless you block it too.

Does blocking ClaudeBot stop Claude from reading my site? It stops Anthropic’s training crawler. Claude-SearchBot (search index) and Claude-User (user-requested fetches) are separate tokens with their own rules — and all three honor robots.txt.

What’s the difference between Google-Extended and Googlebot? Googlebot is Google’s search crawler. Google-Extended is an opt-out token with no crawler behind it — it controls whether your content is used for Google’s AI training, independent of search. Blocking Google-Extended does not affect your Google ranking.

Will AI crawlers actually respect my robots.txt? The reputable ones do — it’s how they signal good faith. But it’s voluntary: spoofers ignore it, and user-triggered fetchers like ChatGPT-User may bypass it. Blocking a bot for certain requires IP-level enforcement plus monitoring.

How do I know if my robots.txt rules are working? Watch actual crawler behavior. Server-side detection shows AI crawler visits per path with a verified/spoofed label, so you can confirm a Disallow took effect — something the robots.txt file itself never tells you.

Wrote the rules — now confirm they work? Start your free trial — 7 days, no credit card required — and watch verified AI crawler visits per page, before and after every robots.txt change.