---
title: "robots.txt for AI Crawlers: Copy-Paste Templates for Every Stance | Kitbase Blog"
description: "Copy-paste robots.txt templates to allow, block, or selectively control AI crawlers like GPTBot, ClaudeBot, and PerplexityBot, plus a verified per-vendor token table."
canonical: https://kitbase.dev/blog/robots-txt-ai-crawlers/
---

**Controlling AI crawlers with robots.txt comes down to writing `Disallow` rules against the right user-agent tokens** — `GPTBot` for OpenAI training, `ClaudeBot` for Anthropic training, `PerplexityBot` for Perplexity, and so on. This guide gives you copy-paste templates for four common stances, a verified table of every major token, and the caveats that decide whether those rules actually do anything. The short version: robots.txt is a request, not a wall, and it only controls the well-behaved bots that choose to honor it.

## The one rule that changes everything

Before any template: **AI vendors run different bots for different jobs, and each has its own robots.txt token.** Blocking one does not block the others. OpenAI alone runs four. Anthropic runs three. This is a feature, not a nuisance — it is exactly what lets you block training while staying citable in AI search. Get the tokens right and robots.txt becomes a scalpel; get them wrong and you either block nothing or accidentally remove yourself from AI answers.

## Verified token table

These are the current, verified user-agent tokens you write rules against. "Training" bots feed model training corpora; "search/index" bots feed live AI answers and citations; "user fetch" bots retrieve a page because a human asked the assistant to.

| Vendor | Token | Job | Honors robots.txt |
|---|---|---|---|
| OpenAI | `GPTBot` | Training | Yes |
| OpenAI | `OAI-SearchBot` | Search / index (ChatGPT search) | Yes |
| OpenAI | `ChatGPT-User` | User-triggered fetch | No — may bypass |
| Anthropic | `ClaudeBot` | Training | Yes |
| Anthropic | `Claude-SearchBot` | Search / index | Yes |
| Anthropic | `Claude-User` | User-triggered fetch | Yes |
| Perplexity | `PerplexityBot` | Search / index | Yes |
| Perplexity | `Perplexity-User` | User-triggered fetch | Often bypasses |
| Google | `Googlebot` | Search (also feeds AI Overviews) | Yes |
| Google | `Google-Extended` | AI-training opt-out **token** (no crawler) | Yes |
| Apple | `Applebot` | Search | Yes |
| Apple | `Applebot-Extended` | AI-training opt-out **token** (no crawler) | Yes |
| Common Crawl | `CCBot` | Training corpus (used by many AI labs) | Yes |
| ByteDance | `Bytespider` | Training | Inconsistent |
| Amazon | `Amazonbot` | Search / assistant | Yes |
| Meta | `Meta-ExternalAgent` | Training | Yes |

Three things worth internalizing from this table:

- **`Google-Extended` and `Applebot-Extended` are not crawlers.** They are opt-out signals. Adding a `Disallow` for them excludes your content from Google/Apple AI training *without* touching normal search crawling by `Googlebot`/`Applebot`. We unpack this in [Google-Extended vs. Googlebot](/blog/google-extended-vs-googlebot/).
- **User-fetch bots are the exception to robots.txt.** OpenAI's `ChatGPT-User` and Perplexity's `Perplexity-User` fetch a page because a human asked, and vendors often treat that like a browser request that bypasses robots.txt. Anthropic is the notable outlier — it says `Claude-User` *does* honor robots.txt.
- **Match on the token, not the full user-agent string.** Version numbers change (`GPTBot/1.4` today, something else tomorrow). The token stays stable.

## Template 1: allow everything (the default)

If your robots.txt says nothing about a bot, it is allowed. You only need this template to be explicit for documentation or auditing purposes:

```
# Allow all crawlers, including AI
User-agent: *
Allow: /
```

This is the right default for most commercial sites: AI answers are a discovery channel, and being crawlable is how you end up in them. If you are unsure whether to block anything, see our [decision framework](/blog/should-you-block-ai-crawlers/) — but the honest default is "allow, then measure."

## Template 2: block training, stay in AI search

The most popular middle ground. You keep your content out of model-*training* corpora while remaining fully citable in AI *search* — so you still show up in ChatGPT search, Perplexity, and Claude answers:

```
# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

# Opt out of Google & Apple AI training (search unaffected)
User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Leave search/retrieval bots ALLOWED (no rule = allowed):
# OAI-SearchBot, PerplexityBot, Claude-SearchBot
```

Note what is deliberately *absent*: no rules for `OAI-SearchBot`, `PerplexityBot`, or `Claude-SearchBot`. That is intentional — those bots feed the citations that send you referral traffic. Blocking them is almost always a mistake.

## Template 3: block all AI, keep normal search

For sites that want out of AI entirely — training and answers — while staying in traditional Google/Bing search:

```
# OpenAI — training + search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /

# Anthropic — training + search
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /

# Perplexity
User-agent: PerplexityBot
Disallow: /

# Others
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /

# Googlebot / Bingbot left allowed for normal search
```

Be clear-eyed about the trade-off: this removes you from AI citations and referral traffic. It makes sense when your content is the product — paywalled, licensed, or proprietary — and far less sense for a site that exists to be discovered.

## Template 4: block sensitive sections only

The surgical option — welcome AI crawlers to your marketing and docs, but fence off areas that should never end up in a corpus:

```
User-agent: GPTBot
Disallow: /internal/
Disallow: /customers/
Disallow: /staging/
Allow: /

User-agent: ClaudeBot
Disallow: /internal/
Disallow: /customers/
Disallow: /staging/
Allow: /
```

## The caveats that decide whether any of this works

Templates are the easy part. These four caveats determine whether your rules have any effect:

1. **robots.txt is an honor system.** Reputable bots — `GPTBot`, `ClaudeBot`, `PerplexityBot` — comply because their operators choose to. A scraper spoofing `GPTBot` reads your `Disallow` and ignores it. If your goal is to *stop* a bot rather than *ask* it to leave, you need firewall or edge rules on verified IP ranges, not a text file.
2. **It is not retroactive.** Blocking `GPTBot` today stops future training crawls; it does nothing about content already collected into models that shipped last year.
3. **User-triggered fetchers are exempt.** As the table shows, `ChatGPT-User` and often `Perplexity-User` bypass robots.txt because a human asked for the page. To block those you need IP-level rules on the vendor's published ranges.
4. **A syntax mistake fails silently.** A typo'd token, a stray `Allow: /` overriding your `Disallow`, or a robots.txt returning a 500 — none of these throw an error. The bot just keeps crawling, and you never find out from the file itself.

That last point is the crux: **robots.txt gives you no feedback.** You write a rule and hope. The only way to confirm a rule is working is to watch what the crawlers actually do afterward.

## How to confirm compliance

Compliance verification is an observation problem, and JavaScript analytics can't help — AI crawlers fetch your HTML without ever running your analytics tag, so [tag-based tools report zero AI crawler traffic](/blog/why-analytics-cant-see-ai-crawlers/) whether your rules work or not. The signal lives in server and edge requests.

The manual approach is to grep your access logs for a token before and after a rule change and watch the count drop. The continuous approach is [server-side crawler detection](https://docs.kitbase.dev/crawler-detection): forward your server or edge requests to Kitbase, and every AI crawler visit is stored with its name, vendor, the pages it hit, and a **verified-or-spoofed** verdict. After you add a `Disallow` for `GPTBot`, verified GPTBot visits to the blocked paths should taper to zero — and if they don't, you've either got a syntax error or a spoofer, and the verdict tells you which. That is the difference between hoping a rule works and knowing it does.

For the specifics of individual bots referenced above, see [GPTBot explained](/blog/gptbot-explained/) and the full [list of AI crawlers](/blog/list-of-ai-crawlers/).

## FAQ

**What is the robots.txt token to block GPTBot?**
`User-agent: GPTBot` followed by `Disallow: /`. That blocks OpenAI's training crawler only — ChatGPT search (`OAI-SearchBot`) is a separate token and keeps citing you unless you block it too.

**Does blocking ClaudeBot stop Claude from reading my site?**
It stops Anthropic's *training* crawler. `Claude-SearchBot` (search index) and `Claude-User` (user-requested fetches) are separate tokens with their own rules — and all three honor robots.txt.

**What's the difference between Google-Extended and Googlebot?**
`Googlebot` is Google's search crawler. `Google-Extended` is an opt-out token with no crawler behind it — it controls whether your content is used for Google's AI training, independent of search. Blocking `Google-Extended` does not affect your Google ranking.

**Will AI crawlers actually respect my robots.txt?**
The reputable ones do — it's how they signal good faith. But it's voluntary: spoofers ignore it, and user-triggered fetchers like `ChatGPT-User` may bypass it. Blocking a bot for certain requires IP-level enforcement plus monitoring.

**How do I know if my robots.txt rules are working?**
Watch actual crawler behavior. Server-side detection shows AI crawler visits per path with a verified/spoofed label, so you can confirm a `Disallow` took effect — something the robots.txt file itself never tells you.

---

*Wrote the rules — now confirm they work? [Start your free trial](https://app.kitbase.dev/signup/) — 7 days, no credit card required — and watch verified AI crawler visits per page, before and after every robots.txt change.*