Bot & Crawler Directory

User-triggered Amazon fetches for fresh answers in products such as Alexa.

User-triggered fetches when ChatGPT opens a page to answer a person.

User-triggered fetches when Claude opens a page to answer a person.

User-triggered Microsoft Copilot fetches for AI answers.

DuckDuckGo real-time crawler for AI-assisted answers with citations.

Google user-triggered fetcher used by AI or product experiences.

Google fetcher associated with NotebookLM-style answer workflows.

Google fetcher associated with read-aloud and assistant experiences.

Google user-triggered fetcher used by AI or product experiences.

xAI fetcher used by Grok's DeepSearch to gather sources for an answer.

User-triggered fetches when Kimi opens a page to answer a person.

meta-externalads is an AI-related crawler operated by Meta.

Meta fetcher used when a person requests or shares a specific URL.

User-triggered fetches when Mistral's Le Chat opens a page to answer a person.

OAI-AdsBot is an AI-related crawler operated by OpenAI.

User-triggered fetches for Perplexity answers and source citations.

User-triggered fetches when Alibaba's Qwen opens a page to answer a person.

User-triggered fetches when Grok answers a person's question.

Anthropic crawler that discovers and refreshes content for Claude search.

Moonshot AI crawler that builds the index behind Kimi's web answers.

Meta crawler used to improve Meta AI search result quality.

Mistral crawler that builds the index behind Le Chat's web answers.

OpenAI crawler that builds the search index behind ChatGPT's web results.

Perplexity crawler that keeps its answer and search index fresh.

AI2Bot is an AI-related crawler operated by Allen AI.

AliyunBot is an AI-related crawler operated by Alibaba.

Robots token controlling whether Apple may use crawled content for Apple Intelligence training.

ByteDance crawler collecting public content used to train its AI models.

Zhipu AI crawler collecting public content for its ChatGLM models.

Anthropic crawler collecting public content that may be used to improve Claude.

Cohere crawler collecting public content that may improve its models.

cohere-training-data-crawler is an AI-related crawler operated by Cohere.

DeepSeek crawler collecting public content for its models.

Doubaobot is an AI-related crawler operated by ByteDance.

Baidu crawler collecting public content for its ERNIE models.

Google crawler for targeted crawls requested by site owners building Vertex AI agents.

Robots token controlling whether Google may use crawled content for Gemini training and grounding; not a separate HTTP crawler.

Generic Google crawler used by product teams for public content fetches, including research and development.

OpenAI crawler for collecting public content that may improve future models.

Grok is an AI-related crawler operated by xAI.

GrokBot is an AI-related crawler operated by xAI.

Moonshot AI crawler collecting public content that may be used to improve Kimi.

Meta crawler collecting public content to train its AI models and improve Meta products.

Alibaba crawler collecting public content for its Qwen models.

TongyiBot is an AI-related crawler operated by Alibaba.

xAI-Bot is an AI-related crawler operated by xAI.

xAI-Grok is an AI-related crawler operated by xAI.

xAI-Web-Crawler is an AI-related crawler operated by xAI.

YiyanBot is an AI-related crawler operated by Baidu.

CCBot is an AI-related crawler operated by Common Crawl.

Kitbase's own crawler, used to index and audit the sites you have connected.

Amazon crawler used to improve Alexa answers and Amazon products.

Amazon crawler that makes content eligible for Amazon search experiences.

Apple crawler for Siri and Spotlight Suggestions, spanning search and training use.

Baidu's crawler for its search index.

Microsoft crawler for Bing Search indexing.

DuckDuckGo's crawler for its own search results.

Google crawler used by Search testing tools such as Rich Results Test and URL Inspection.

Google's main crawler for Search indexing and discovery.

Legacy Microsoft crawler used for search indexing, now largely superseded by Bingbot.

ByteDance crawler associated with TikTok search and discovery.

Yandex's crawler for its search index.

You.com crawler for its search and AI answer index.

FacebookBot is an AI-related crawler operated by Meta.

facebookexternalhit is an AI-related crawler operated by Meta.