KitbaseAuditBot
The crawler behind Kitbase Site Audit. If you saw it in your logs, someone with a Kitbase account ran an AI-readiness audit of your domain — almost always the site’s own team.
Mozilla/5.0 (compatible; KitbaseAuditBot/1.0; +https://kitbase.dev/audit-bot) What it fetches
/robots.txt Checks the file is valid and whether AI crawlers are allowed /llms.txt Checks whether the site publishes an llms.txt summary /sitemap.xml Validates the sitemap (plus up to 3 child sitemaps declared in an index) Homepage Checks HTTPS/redirect health and analyzes the page markup Up to ~15 sitemap pages A small sample, analyzed for titles, structured data, Open Graph tags, canonical URLs and heading structure One audit also sends a handful of extra homepage requests using the user-agent strings of common AI crawlers (GPTBot, ClaudeBot, PerplexityBot). This tests whether a firewall or CDN blocks those crawlers at the edge — a key part of the audit — and is limited to the homepage.
How it behaves
Runs only on request
KitbaseAuditBot never crawls the web on its own. It fetches a site only when a Kitbase customer starts an audit of that domain from their dashboard.
Rate limited
At most 2 requests per second against any host, with 10-second timeouts and capped response sizes. A full audit is typically 20–40 requests over under a minute.
Read-only
GET requests only. It never submits forms, posts data, executes JavaScript, or follows links beyond the URLs listed in your sitemap.
Not for indexing or training
Fetched content is analyzed to produce an audit checklist and shown only to the customer who requested the audit. It is not published, indexed, or used to train models.
Don’t want to be audited?
Audits are meant to be run by a site’s own team. If you want
to prevent audits of your domain entirely, block the
KitbaseAuditBot
user agent at your server, firewall, or CDN — or
contact us
and we’ll exclude your domain from audits on our side.
Questions about a specific request? Include the timestamp and your domain when you reach out and we’ll trace it.