---
title: "What is xAI-Web-Crawler? xAI's Bot Explained | Kitbase"
description: "xAI-Web-Crawler is an AI-related crawler operated by xAI. How to identify and verify it, and what its traffic means for your site."
canonical: https://kitbase.dev/bot-directory/xai-web-crawler
---

[Home](/) [Bot Directory](/bot-directory) xAI-Web-Crawler

# xAI-Web-Crawler

by xAI

AI training AI UA only (unverifiable)

Collects pages in bulk to train a model, unconnected to any individual user's question.

xAI-Web-Crawler is an AI-related crawler operated by xAI.

## What it usually means

xAI crawled this page as part of its AI crawling.

## How to identify it

User-Agent tokens

`xai-web-crawler`

robots.txt stanza

`User-agent: xAI-Web-Crawler`

Free tools

[Verify a xAI-Web-Crawler request](/tools/bot-verifier?ua=xai-web-crawler) [Check if your site allows xAI-Web-Crawler](/tools/ai-crawler-access-checker)

[xAI's page for xAI-Web-Crawler](https://x.ai/)

## Sources

-   [xAI crawler docs](https://x.ai/)

## More from xAI

[

Grok-DeepSearch

xAI fetcher used by Grok's DeepSearch to gather sources for an answer.

](/bot-directory/grok-deepsearch)[

xAI-SearchBot

User-triggered fetches when Grok answers a person's question.

](/bot-directory/xai-searchbot)[

xAI-Grok

xAI-Grok is an AI-related crawler operated by xAI.

](/bot-directory/xai-grok)[

GrokBot

GrokBot is an AI-related crawler operated by xAI.

](/bot-directory/grokbot)[

xAI-Bot

xAI-Bot is an AI-related crawler operated by xAI.

](/bot-directory/xai-bot)[

Grok

Grok is an AI-related crawler operated by xAI.

](/bot-directory/grok)

## Similar bots

[

AI2Bot

Allen AI

AI2Bot is an AI-related crawler operated by Allen AI.

](/bot-directory/ai2bot)[

AliyunBot

Alibaba

AliyunBot is an AI-related crawler operated by Alibaba.

](/bot-directory/aliyunbot)[

Applebot-Extended

Apple

Robots token controlling whether Apple may use crawled content for Apple Intelligence training.

](/bot-directory/applebot-extended)[

Bytespider

ByteDance

ByteDance crawler collecting public content used to train its AI models.

](/bot-directory/bytespider)

## See if xAI-Web-Crawler has visited your site

Kitbase verifies and tracks bot and AI crawler traffic separately from human visitors, so you know exactly who's hitting your site.

[Start Free Trial](https://app.kitbase.dev/signup)