The Complete List of AI Crawlers in 2026 (and How to Identify Each One)

An AI crawler is a bot operated by an AI company (or a data provider that feeds them) to read the public web — for training models, for building the search indexes that AI answers cite, or for fetching a page the moment a user asks. In 2026 the ones you’re most likely to see in your logs are GPTBot, ClaudeBot, PerplexityBot, Googlebot, Bingbot, Applebot, Amazonbot, meta-externalagent, and CCBot, from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, Meta, and Common Crawl respectively.

This page is the reference. Every entry below is verified against the vendor’s own documentation, with its robots.txt token, whether it honors robots.txt, and how (or whether) you can verify it against published IP ranges. Bots we could not verify against official documentation are listed separately at the end, clearly flagged — because guessing about a crawler is worse than admitting the vendor doesn’t document it.

The four jobs an AI crawler does

Before the tables, one organizing idea. Almost every AI crawler falls into one of four jobs, and the job determines how it behaves toward robots.txt:

flowchart TD
R["AI crawler"] --> T["Training<br/>bulk corpus for models"]
R --> S["Search index<br/>pages AI answers cite"]
R --> U["User-triggered fetch<br/>one page, on demand"]
R --> C["Control token<br/>no crawling — a usage flag"]
T --> T2["Usually respects robots.txt"]
S --> S2["Usually respects robots.txt"]
U --> U2["Often bypasses robots.txt<br/>treated as acting for a human"]
C --> C2["N/A — never fetches a page"]

The pattern that matters: training and search bots generally obey robots.txt; user-triggered fetchers often don’t (vendors argue they’re acting on behalf of a person, like a browser); and control tokens — Google-Extended, Applebot-Extended — never crawl at all. They’re just flags in robots.txt that govern how already-crawled content may be used.

How to read these tables

• robots.txt token is the string you put after User-agent: in robots.txt. Always match on the token, not the full user-agent string — version numbers change.
• Respects robots.txt reflects the vendor’s stated behavior. “No” or “May bypass” means a Disallow line is a request, not a guarantee; a firewall rule on published IP ranges is the only hard control.
• Verification is how you confirm a request is genuine and not a scraper wearing the bot’s name: a published IP-range JSON file, reverse DNS (rDNS), a vendor tool, or — for a few — nothing at all.

OpenAI

OpenAI runs four bots, documented at developers.openai.com/api/docs/bots, each with its own IP-range file. The critical distinction: GPTBot is training only — blocking it does not remove you from ChatGPT search, which is OAI-SearchBot’s job.

Deep dives: GPTBot explained and OpenAI’s four crawlers compared.

Anthropic

Anthropic documents three Claude agents, all of which — unusually — respect robots.txt, including the user-triggered one. IP ranges for all three are published at claude.com/crawling/bots.json, and Anthropic supports the Crawl-delay directive.

Deep dive: ClaudeBot explained.

Google

Google’s crawlers are documented at developers.google.com, and it’s the vendor where the training/search distinction is most consequential. Googlebot is your entire SEO — never block it. Google-Extended is a control token that opts you out of Gemini training without touching Search or AI Overviews.

Deep dives: Google-Extended vs Googlebot and how to verify Googlebot.

Microsoft

Bingbot crawls for Bing Search, and because Microsoft Copilot and Bing’s AI answers draw on the Bing index, blocking Bingbot also reduces your visibility in those AI surfaces. Microsoft publishes IP ranges, offers a public Verify Bingbot tool, and legitimate Bingbot IPs reverse-resolve to *.search.msn.com.

Perplexity

Perplexity’s crawler feeds its search index, not a training corpus — so blocking it removes you from Perplexity citations. Note the 2025 Cloudflare dispute over undeclared crawling, which makes verification-by-IP especially important here.

Deep dive: PerplexityBot explained.

Apple

Apple’s Applebot powers Siri and Spotlight and may feed Apple Intelligence training; Applebot-Extended is a control token to opt out of that training without leaving Apple’s search. Applebot IPs reverse-resolve to *.applebot.apple.com, and Apple publishes IP CIDRs.

Amazon, Common Crawl, Mistral, DuckDuckGo, Meta

The remaining well-documented crawlers:

Common Crawl deserves a special mention: CCBot doesn’t power a single product, but its open corpus has trained a huge share of the models everyone else runs — so it’s often the most consequential crawler to think about, even though its per-day volume is modest.

Crawlers we could not verify against official docs

Two frequently-cited names have no official vendor documentation describing their purpose, robots.txt stance, or IP ranges. We’re listing them so the reference is honest, not to endorse the details — treat everything here as observed behavior reported by third parties, not vendor-stated fact:

• Bytespider (ByteDance) — commonly observed under a Bytespider user-agent and widely associated with training data collection. ByteDance publishes no crawler documentation, and CDN operators report inconsistent robots.txt compliance. Because there’s no published IP range, a genuine block has to happen at the WAF/edge on behavioral signals, not a Disallow line.
• xAI’s Grok crawler — xAI publishes no crawler documentation page. Cloudflare has stated that Grok’s crawler does not reliably self-identify, which makes it effectively unblockable via robots.txt and unverifiable by user-agent. Reported tokens exist but are rarely seen in practice.

The lesson these two teach applies to the whole list: a user-agent string is a claim, not proof. Which brings us to identification.

How to actually identify each one

Two failure modes make this harder than grepping for a name:

1. Spoofing. Scrapers copy famous crawler UAs to slip past bot rules. The fix is verification against published IP ranges (and, increasingly, cryptographic Web Bot Auth signatures) — which is why the “Verification” column above matters more than the token. The Googlebot verification walkthrough shows the full reverse-DNS-plus-IP method that generalizes to every vendor.
2. Invisibility to JavaScript analytics. None of these bots execute your page scripts, so Google Analytics, Plausible, and every tag-based tool report zero AI crawler traffic — not because there’s none, but because they structurally can’t see it. The only place AI crawlers appear is your server logs or a tool that reads them.

Doing verification by hand — refetching a dozen vendors’ IP files, matching every hit, keeping the list current as new bots appear — is exactly the maintenance this reference implies. Kitbase’s bot and crawler detection automates it: forward your server or edge requests, and every one is classified server-side with a vendor, a bot name, and a verified or spoofed verdict against current published ranges. Humans are classified in memory and discarded; only bots are stored. Once you can see which of these crawlers read you, the natural next question — whether AI answers actually mention and cite you — is what AI Visibility measures.

Keeping this list current

The AI crawler landscape changes monthly: new bots launch, user-agent versions bump, IP ranges rotate, and vendors add or retire control tokens. That’s the argument against hard-coding a bot list into your own infrastructure and forgetting it — and the argument for treating identification as a live, maintained function rather than a one-time robots.txt edit. Bookmark this page; we keep it verified.

FAQ

What is the most common AI crawler? On most sites, GPTBot, ClaudeBot, and Googlebot top the list, with Bingbot, PerplexityBot, and Applebot close behind. Exact ordering depends on your content and how crawlable it is.

Which AI crawlers ignore robots.txt? By design, user-triggered fetchers often do — OpenAI’s ChatGPT-User and Perplexity’s Perplexity-User are documented as bypassing it. Among undocumented crawlers, Bytespider and xAI’s Grok are reported to ignore or evade it. Most training and search crawlers (GPTBot, ClaudeBot, Googlebot, Bingbot, PerplexityBot) honor it.

How do I verify an AI crawler is genuine? Match the request’s source IP against the vendor’s published IP-range JSON, or use reverse DNS where the vendor supports it (Googlebot, Bingbot, Applebot, CCBot). A request whose IP falls outside the published ranges is spoofed.

Can Google Analytics show me AI crawlers? No. AI crawlers don’t run JavaScript, so they never trigger analytics tags. AI crawler traffic only appears in server logs or a server-side crawler-detection tool.

Should I block AI crawlers? It depends on whether your content is a product to protect or a shelf to be discovered on. See should you block AI crawlers for the decision framework and robots.txt for AI crawlers for the exact rules.

Want this list applied to your own traffic — every AI crawler that visits, verified and per-page? Start your free trial — 7 days, no credit card required — and see which bots actually read your site.