Google-Extended

by Google

AI training AI UA only (unverifiable)

Collects pages in bulk to train a model, unconnected to any individual user's question.

Robots token controlling whether Google may use crawled content for Gemini training and grounding; not a separate HTTP crawler.

What it usually means

Google is treating this page as eligible AI training data. Blocking it removes you from Gemini training without affecting Search.

robots.txt stanza

User-agent: Google-Extended

Google crawler for targeted crawls requested by site owners building Vertex AI agents.

Google crawler used by Search testing tools such as Rich Results Test and URL Inspection.

Google fetcher associated with NotebookLM-style answer workflows.

Google fetcher associated with read-aloud and assistant experiences.

Google user-triggered fetcher used by AI or product experiences.

Google user-triggered fetcher used by AI or product experiences.

Generic Google crawler used by product teams for public content fetches, including research and development.

Google's main crawler for Search indexing and discovery.

AI2Bot is an AI-related crawler operated by Allen AI.

AliyunBot is an AI-related crawler operated by Alibaba.

Robots token controlling whether Apple may use crawled content for Apple Intelligence training.

ByteDance crawler collecting public content used to train its AI models.

Kitbase verifies and tracks bot and AI crawler traffic separately from human visitors, so you know exactly who's hitting your site.