--- title: "Bot & Crawler Directory - Known Bots, Crawlers & AI Agents | Kitbase" description: "A searchable directory of known bots, crawlers, and AI agents: GPTBot, Googlebot, ClaudeBot and more. Who runs each one, and how it's verified." canonical: https://kitbase.dev/bot-directory --- # Bot & Crawler Directory 65 known bots, crawlers, and AI agents — who runs each one, how it's verified, and what it means when it shows up in your traffic. ## AI answers  (18) [ ### Amzn-User Amazon User-triggered Amazon fetches for fresh answers in products such as Alexa. AI ](/bot-directory/amzn-user)[ ### ChatGPT-User OpenAI User-triggered fetches when ChatGPT opens a page to answer a person. AI ](/bot-directory/chatgpt-user)[ ### Claude-User Anthropic User-triggered fetches when Claude opens a page to answer a person. AI ](/bot-directory/claude-user)[ ### Copilot Microsoft User-triggered Microsoft Copilot fetches for AI answers. AI ](/bot-directory/copilot)[ ### DuckAssistBot DuckDuckGo DuckDuckGo real-time crawler for AI-assisted answers with citations. AI ](/bot-directory/duckassistbot)[ ### Google-Agent Google Google user-triggered fetcher used by AI or product experiences. AI ](/bot-directory/google-agent)[ ### Google-NotebookLM Google Google fetcher associated with NotebookLM-style answer workflows. AI ](/bot-directory/google-notebooklm)[ ### Google-Read-Aloud Google Google fetcher associated with read-aloud and assistant experiences. ](/bot-directory/google-read-aloud)[ ### GoogleAgent Google Google user-triggered fetcher used by AI or product experiences. AI ](/bot-directory/googleagent)[ ### Grok-DeepSearch xAI xAI fetcher used by Grok's DeepSearch to gather sources for an answer. AI ](/bot-directory/grok-deepsearch)[ ### Kimi-User Moonshot AI User-triggered fetches when Kimi opens a page to answer a person. AI ](/bot-directory/kimi-user)[ ### meta-externalads Meta meta-externalads is an AI-related crawler operated by Meta. AI ](/bot-directory/meta-externalads)[ ### meta-externalfetcher Meta Meta fetcher used when a person requests or shares a specific URL. AI ](/bot-directory/meta-externalfetcher)[ ### MistralAI-User Mistral User-triggered fetches when Mistral's Le Chat opens a page to answer a person. AI ](/bot-directory/mistralai-user)[ ### OAI-AdsBot OpenAI OAI-AdsBot is an AI-related crawler operated by OpenAI. AI ](/bot-directory/oai-adsbot)[ ### Perplexity-User Perplexity User-triggered fetches for Perplexity answers and source citations. AI ](/bot-directory/perplexity-user)[ ### Qwen-User Alibaba User-triggered fetches when Alibaba's Qwen opens a page to answer a person. AI ](/bot-directory/qwen-user)[ ### xAI-SearchBot xAI User-triggered fetches when Grok answers a person's question. AI ](/bot-directory/xai-searchbot) ## AI indexing  (6) [ ### Claude-SearchBot Anthropic Anthropic crawler that discovers and refreshes content for Claude search. AI ](/bot-directory/claude-searchbot)[ ### Kimi-SearchBot Moonshot AI Moonshot AI crawler that builds the index behind Kimi's web answers. AI ](/bot-directory/kimi-searchbot)[ ### meta-webindexer Meta Meta crawler used to improve Meta AI search result quality. AI ](/bot-directory/meta-webindexer)[ ### MistralAI-Index Mistral Mistral crawler that builds the index behind Le Chat's web answers. AI ](/bot-directory/mistralai-index)[ ### OAI-SearchBot OpenAI OpenAI crawler that builds the search index behind ChatGPT's web results. AI ](/bot-directory/oai-searchbot)[ ### PerplexityBot Perplexity Perplexity crawler that keeps its answer and search index fresh. AI ](/bot-directory/perplexitybot) ## AI training  (25) [ ### AI2Bot Allen AI AI2Bot is an AI-related crawler operated by Allen AI. AI ](/bot-directory/ai2bot)[ ### AliyunBot Alibaba AliyunBot is an AI-related crawler operated by Alibaba. AI ](/bot-directory/aliyunbot)[ ### Applebot-Extended Apple Robots token controlling whether Apple may use crawled content for Apple Intelligence training. AI ](/bot-directory/applebot-extended)[ ### Bytespider ByteDance ByteDance crawler collecting public content used to train its AI models. AI ](/bot-directory/bytespider)[ ### ChatGLM-Spider Zhipu AI Zhipu AI crawler collecting public content for its ChatGLM models. AI ](/bot-directory/chatglm-spider)[ ### ClaudeBot Anthropic Anthropic crawler collecting public content that may be used to improve Claude. AI ](/bot-directory/claudebot)[ ### cohere-ai Cohere Cohere crawler collecting public content that may improve its models. AI ](/bot-directory/cohere-ai)[ ### cohere-training-data-crawler Cohere cohere-training-data-crawler is an AI-related crawler operated by Cohere. AI ](/bot-directory/cohere-training-data-crawler)[ ### DeepSeekBot DeepSeek DeepSeek crawler collecting public content for its models. AI ](/bot-directory/deepseekbot)[ ### Doubaobot ByteDance Doubaobot is an AI-related crawler operated by ByteDance. AI ](/bot-directory/doubaobot)[ ### ERNIEBot Baidu Baidu crawler collecting public content for its ERNIE models. AI ](/bot-directory/erniebot)[ ### Google-CloudVertexBot Google Google crawler for targeted crawls requested by site owners building Vertex AI agents. AI ](/bot-directory/google-cloudvertexbot)[ ### Google-Extended Google Robots token controlling whether Google may use crawled content for Gemini training and grounding; not a separate HTTP crawler. AI ](/bot-directory/google-extended)[ ### GoogleOther Google Generic Google crawler used by product teams for public content fetches, including research and development. AI ](/bot-directory/googleother)[ ### GPTBot OpenAI OpenAI crawler for collecting public content that may improve future models. AI ](/bot-directory/gptbot)[ ### Grok xAI Grok is an AI-related crawler operated by xAI. AI ](/bot-directory/grok)[ ### GrokBot xAI GrokBot is an AI-related crawler operated by xAI. AI ](/bot-directory/grokbot)[ ### KimiBot Moonshot AI Moonshot AI crawler collecting public content that may be used to improve Kimi. AI ](/bot-directory/kimibot)[ ### meta-externalagent Meta Meta crawler collecting public content to train its AI models and improve Meta products. AI ](/bot-directory/meta-externalagent)[ ### QwenBot Alibaba Alibaba crawler collecting public content for its Qwen models. AI ](/bot-directory/qwenbot)[ ### TongyiBot Alibaba TongyiBot is an AI-related crawler operated by Alibaba. AI ](/bot-directory/tongyibot)[ ### xAI-Bot xAI xAI-Bot is an AI-related crawler operated by xAI. AI ](/bot-directory/xai-bot)[ ### xAI-Grok xAI xAI-Grok is an AI-related crawler operated by xAI. AI ](/bot-directory/xai-grok)[ ### xAI-Web-Crawler xAI xAI-Web-Crawler is an AI-related crawler operated by xAI. AI ](/bot-directory/xai-web-crawler)[ ### YiyanBot Baidu YiyanBot is an AI-related crawler operated by Baidu. AI ](/bot-directory/yiyanbot) ## Archiver  (1) [ ### CCBot Common Crawl CCBot is an AI-related crawler operated by Common Crawl. ](/bot-directory/ccbot) ## Monitoring  (1) [ ### KitbaseAuditBot Kitbase Kitbase's own crawler, used to index and audit the sites you have connected. ](/bot-directory/kitbase-audit-bot) ## Search engine  (12) [ ### Amazonbot Amazon Amazon crawler used to improve Alexa answers and Amazon products. ](/bot-directory/amazonbot)[ ### Amzn-SearchBot Amazon Amazon crawler that makes content eligible for Amazon search experiences. ](/bot-directory/amzn-searchbot)[ ### Applebot Apple Apple crawler for Siri and Spotlight Suggestions, spanning search and training use. ](/bot-directory/applebot)[ ### Baiduspider Baidu Baidu's crawler for its search index. ](/bot-directory/baiduspider)[ ### Bingbot Microsoft Microsoft crawler for Bing Search indexing. ](/bot-directory/bingbot)[ ### DuckDuckBot DuckDuckGo DuckDuckGo's crawler for its own search results. ](/bot-directory/duckduckbot)[ ### Google-InspectionTool Google Google crawler used by Search testing tools such as Rich Results Test and URL Inspection. ](/bot-directory/google-inspectiontool)[ ### Googlebot Google Google's main crawler for Search indexing and discovery. ](/bot-directory/googlebot)[ ### msnbot Microsoft Legacy Microsoft crawler used for search indexing, now largely superseded by Bingbot. ](/bot-directory/msnbot)[ ### TikTokSpider ByteDance ByteDance crawler associated with TikTok search and discovery. ](/bot-directory/tiktokspider)[ ### YandexBot Yandex Yandex's crawler for its search index. ](/bot-directory/yandexbot)[ ### YouBot You.com You.com crawler for its search and AI answer index. ](/bot-directory/youbot) ## Social preview  (2) [ ### FacebookBot Meta FacebookBot is an AI-related crawler operated by Meta. ](/bot-directory/facebookbot)[ ### facebookexternalhit Meta facebookexternalhit is an AI-related crawler operated by Meta. ](/bot-directory/facebookexternalhit) No bots found Try a different search term or category. ## See which of these bots are hitting your site Kitbase verifies and tracks bot and AI crawler traffic on your own domain — separate from human visitors, with the same verification methods used in this directory. [Start Free Trial](https://app.kitbase.dev/signup)