Applebot-Extended

by Apple

AI training AI UA only (unverifiable)

Collects pages in bulk to train a model, unconnected to any individual user's question.

Robots token controlling whether Apple may use crawled content for Apple Intelligence training.

What it usually means

Nothing on its own — Applebot-Extended never fetches. It is a robots.txt token that opts pages out of Apple Intelligence training.

robots.txt stanza

User-agent: Applebot-Extended

Apple crawler for Siri and Spotlight Suggestions, spanning search and training use.

AI2Bot is an AI-related crawler operated by Allen AI.

AliyunBot is an AI-related crawler operated by Alibaba.

ByteDance crawler collecting public content used to train its AI models.

Zhipu AI crawler collecting public content for its ChatGLM models.

Kitbase verifies and tracks bot and AI crawler traffic separately from human visitors, so you know exactly who's hitting your site.