ClaudeBot Explained: Anthropic's Crawler, robots.txt, and How to Track It

ClaudeBot is Anthropic’s web crawler. It collects publicly available content to help train and improve the Claude family of models. Like OpenAI’s GPTBot, it identifies itself honestly in the user-agent, respects robots.txt, and publishes its IP ranges — which makes it one of the more transparent AI crawlers to recognize, verify, and control.

It is also, on most sites, one of the top three AI visitors in the server logs alongside GPTBot and Googlebot. Here’s how to recognize ClaudeBot, tell it apart from Anthropic’s two other Claude agents, verify it isn’t being spoofed, and decide whether to let it in.

ClaudeBot at a glance

The user-agent string ClaudeBot sends:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

The version number can change over time, so match on the ClaudeBot token, not the whole string. Note the contact address baked into the UA — claudebot@anthropic.com — which is Anthropic’s channel for crawl questions and opt-out requests.

ClaudeBot is one of three Claude agents

The most common ClaudeBot misconception is the same one people have about GPTBot: that blocking it removes you from Claude entirely. It doesn’t. Anthropic documents three separate agents, each with its own robots.txt token and its own job:

Two things make Anthropic’s setup notable:

• All three respect robots.txt. This is a real contrast with OpenAI’s ChatGPT-User and Perplexity’s Perplexity-User, both of which treat user-triggered fetches as exempt from robots.txt. Anthropic states that Claude-User honors robots.txt directives — so, unlike those two, you can control on-demand Claude fetches from a text file. (For the OpenAI comparison, see OpenAI’s four crawlers explained; for Perplexity’s, PerplexityBot explained.)
• The split mirrors the industry pattern. One bot for training, one for user-initiated fetches, one for search indexing. Blocking ClaudeBot opts you out of training crawls only; Claude-SearchBot and Claude-User are governed by separate rules.

Anthropic also supports the non-standard Crawl-delay directive, so you can throttle ClaudeBot rather than block it outright.

Finding ClaudeBot in your server logs

If you have raw access logs, a single grep tells you whether ClaudeBot visits and what it reads:

grep -i "claudebot" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

That prints your 20 most-crawled paths. As with most training crawlers, ClaudeBot gravitates toward text-dense, stable URLs — documentation, blog posts, comparison pages — and recrawls them on an irregular cadence.

Two things that grep can’t tell you:

1. Whether the traffic is really ClaudeBot. Anything can send that user-agent string, and impersonating well-known crawlers is a standard scraper tactic.
2. Anything at all, if you only run JavaScript analytics. ClaudeBot fetches HTML and never executes your page scripts, so it never fires your analytics snippet. Google Analytics, Plausible, and every tag-based tool report zero ClaudeBot traffic — not because there is none, but because they physically cannot see it.

Verifying real ClaudeBot (and catching spoofers)

Verification is straightforward because Anthropic publishes ClaudeBot’s egress IP ranges as machine-readable JSON:

curl -s https://claude.com/crawling/bots.json

A request that claims to be ClaudeBot from an IP outside those published CIDR blocks is spoofed. Anthropic is explicit that blocking by IP is not a reliable opt-out — partly because it can prevent the crawler from even reading your robots.txt — so IP ranges are best used for the opposite job: confirming that a given hit is genuine, and flagging the ones that aren’t.

flowchart TD
A["Incoming request<br/>UA contains 'ClaudeBot'"] --> B{"Source IP inside<br/>claude.com/crawling/bots.json ranges?"}
B -->|"Yes"| V["Verified ClaudeBot"]
B -->|"No"| S["Spoofed — a scraper<br/>impersonating ClaudeBot"]

Doing this by hand for one vendor is easy; doing it continuously for every AI crawler — refetching each vendor’s ranges, matching every hit, flagging every mismatch — is the part that should be automated. Kitbase’s bot and crawler detection classifies every forwarded request server-side and stores each crawler visit with a verified or spoofed verdict, using vendors’ published IP ranges and cryptographic Web Bot Auth signatures where crawlers send them. Human visitors are classified in memory and discarded; only bots are stored.

robots.txt recipes

ClaudeBot honors robots.txt, so control is one text file away. Pick your stance:

Block training crawls entirely:

User-agent: ClaudeBot
Disallow: /

Slow ClaudeBot down instead of blocking it (Anthropic supports Crawl-delay):

User-agent: ClaudeBot
Crawl-delay: 1

Block training from sensitive sections only:

User-agent: ClaudeBot
Disallow: /internal/
Disallow: /customers/
Allow: /

Block all three Claude agents (full opt-out — this also removes you from Claude’s live fetches and search-quality crawling):

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

Two caveats carry over from any robots.txt strategy:

• robots.txt is not retroactive. Blocking ClaudeBot today stops future crawls; it doesn’t remove content already collected.
• robots.txt is honor-system. ClaudeBot complies, but scrapers wearing its user-agent do not — which is exactly why verification and monitoring matter.

Should you block ClaudeBot?

The decision is the same one you face with every training crawler, and it comes down to what your content is worth in each channel. We cover the full framework in should you block AI crawlers, but in short:

Consider blocking if your content is the product — paywalled journalism, proprietary research, licensed datasets. Training crawls extract that value without attribution or traffic, and opting out (or negotiating a license) is rational.

Don’t block if you sell something and your site exists to be discovered. Claude recommends tools, products, and sources it knows about. Being present in the training corpus and reachable at answer time is how your brand ends up in Claude’s responses when a buyer asks “what’s the best tool for X?” — a channel that grows as traditional search clicks shrink. Blocking here is anti-marketing; it’s the core insight behind Generative Engine Optimization.

Either way, measure first. A decision about AI crawlers made without crawler data is a guess. Look at what ClaudeBot actually reads on your site, and how often, before you touch robots.txt.

What to watch once you’re tracking it

With crawler detection in place, ClaudeBot becomes a dataset rather than an abstraction. The useful signals:

• Which pages get crawled most — these are the pages Claude learns about you from. If your best comparison content isn’t on the list, it may be unreachable, noindexed, or too new.
• Crawl trends — a sudden stop often means an accidental block (a WAF rule, a CDN bot-protection toggle) you’d otherwise never notice.
• Verified vs. spoofed ratio — a rising share of spoofed “ClaudeBot” traffic is scraper activity in disguise, which is a security signal, not an AI one.
• The other side of the funnel — crawls are the input; the output is whether Claude’s answers actually mention and cite you. That’s a separate dataset, measured by querying the model directly with AI Visibility.

FAQ

Does ClaudeBot respect robots.txt? Yes — Anthropic documents it and supports both the standard Disallow directive and the non-standard Crawl-delay. Unusually, Anthropic states that all three Claude agents (including the user-triggered Claude-User) honor robots.txt.

Does blocking ClaudeBot remove my site from Claude? No. It opts you out of training crawls. Claude-SearchBot and Claude-User are separate tokens with separate rules, and content already collected isn’t retroactively removed.

What is the ClaudeBot user-agent string? Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com). Match on the ClaudeBot token rather than the full string, since the version can change.

How do I verify ClaudeBot traffic is real? Check the source IP against Anthropic’s published ranges at claude.com/crawling/bots.json. Any “ClaudeBot” request from outside those ranges is spoofed. Kitbase does this automatically and labels every visit verified or spoofed.

Does ClaudeBot execute JavaScript? No. It fetches server-rendered HTML, so content that only appears after client-side rendering is invisible to it — and JavaScript analytics can’t record its visits at all.

Want to see which AI crawlers visit your site — verified, attributed, and per-page? Start your free trial — 7 days, no credit card required — and the setup wizard will have your first crawler data flowing in minutes.