Can AI crawlers read your site?
Check whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest can reach your site. We evaluate your robots.txt for every AI crawler and probe your origin with each one's real user agent — so you catch firewall and Cloudflare blocks robots.txt can't show you.
How it works
Enter your domain
Type the domain you want to check. No sign-up, no email — the check runs immediately.
We read robots.txt and knock on your origin
We fetch your robots.txt and evaluate it for every AI crawler in the Kitbase bot registry, then request your homepage with each crawler's real user agent to see whether your CDN or firewall lets it through.
Fix what's blocked — or keep it that way
Every crawler gets a verdict: allowed, partial, blocked by robots.txt or blocked at the origin. Copy a robots.txt snippet for the ones you want to change.
robots.txt and your firewall are two different doors
robots.txt is a request. It is a plain text file that tells well-behaved crawlers which paths they may fetch. GPTBot, ClaudeBot, PerplexityBot and Googlebot all honour it — but it does nothing on its own. A crawler that ignores it will still get your pages.
A 403 from your origin is a wall. Your web server, CDN or WAF decides what actually gets served. Cloudflare's AI-crawler controls, bot-management products and hand-written firewall rules can return a block or a JavaScript challenge to a specific user agent, regardless of what robots.txt says.
The two disagree more often than you'd think. A site can say
Allow: /
for GPTBot while its CDN silently 403s every request carrying the GPTBot user agent — so the site owner
believes they are visible to ChatGPT and they aren't. The reverse happens too: a strict robots.txt with
a permissive origin means the crawler could read everything, and the polite ones simply choose
not to. This checker evaluates both doors for every crawler and tells you which one is closed.
Which crawlers we test
The list comes straight from the Kitbase bot registry — the same catalogue that powers our bot analytics — so it stays current as vendors add or rename crawlers. Every crawler with a published robots.txt token is evaluated; the ones with a known user-agent string are also probed live against your origin.
- AI training crawlers — GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, Applebot-Extended, Bytespider (ByteDance), CCBot (Common Crawl), meta-externalagent and others that collect pages to train models.
- AI search and assistant crawlers — OAI-SearchBot and ChatGPT-User (ChatGPT search and browsing), Claude-SearchBot and Claude-User, PerplexityBot, DuckAssistBot and the other fetchers that retrieve pages to answer a live question and cite the source. These are the ones that decide whether you get cited.
- Search engines, as a reference — Googlebot and Bingbot appear in their own section so you can confirm that a strict AI stance hasn't accidentally blocked traditional search.
Blocking training crawlers and allowing search crawlers is a common, defensible stance; allowing everything and blocking everything are both fine too. What matters is that the verdict matches what you intended — and that it matches at both doors.
Frequently asked questions
Does blocking GPTBot hurt my Google rankings?
No. GPTBot is OpenAI's crawler and has nothing to do with Google Search. Google uses a separate token, Google-Extended, to let you opt out of AI training without touching Googlebot — so you can block AI training crawlers and still rank normally. The reference rows for Googlebot and Bingbot in the results are there so you can confirm you haven't accidentally blocked a search engine.
Why is a crawler allowed in robots.txt but blocked at the origin?
Because they are two different doors. robots.txt is a text file that asks well-behaved crawlers to stay out of certain paths. Your origin — the web server, CDN or WAF in front of it — decides what to actually serve. Cloudflare's AI-crawler controls, bot-management products and custom firewall rules can return a 403 or a JavaScript challenge to a bot's user agent even when robots.txt says Allow. The tool probes with each crawler's real user agent so you see both answers.
What does a PARTIAL verdict mean?
The crawler is allowed to fetch your homepage, but the robots.txt group that applies to it contains at least one Disallow rule — so some paths are off-limits. That is often intentional (blocking /admin or /search, for example). We show a sample of the disallowed paths so you can confirm it's the stance you meant.
What is a challenge page?
A page a CDN or bot-management layer serves instead of your content — Cloudflare's "Just a moment…" interstitial is the best-known example. It runs JavaScript to decide whether the visitor is a browser. Crawlers cannot solve it, so a challenge is effectively a block for them. We detect the common markers and label those probes CHALLENGED.
Do you store my robots.txt or my results?
Results are cached for about ten minutes so repeated checks of the same domain don't hammer your server; after that they are gone, and we never keep a copy of your robots.txt. We do keep a usage log — the domain checked and the headline counts (how many crawlers were allowed or blocked) — so we can see how the tool is used. If you want ongoing monitoring with history and alerts, that is what the Kitbase site audit and bot analytics do.
Can I check a specific path instead of the homepage?
Not yet — this tool evaluates robots.txt for the site root (/) and probes the homepage. Path-level checks and a full crawl of your sitemap pages are part of the Kitbase site audit.
How often should I re-check?
Whenever you change robots.txt, switch CDN or firewall settings, or turn on a bot-management feature. AI crawler policies also change on the vendor side: new crawlers appear and user agents get renamed. Kitbase's daily site audit re-runs this check for you and tells you when a verdict flips.
Is this the same as the Kitbase site audit?
It's one slice of it. The site audit runs this AI-crawler access check every day alongside sitemap, llms.txt, structured-data, performance and content checks across your pages, and pairs it with bot analytics that show which crawlers actually visited. This free tool gives you the access verdict for one domain, right now.
More free tools
All tools →llms.txt Generator
Build a spec-compliant llms.txt from your sitemap in a minute.
llms.txt Checker
Validate your llms.txt against the spec and catch the usual mistakes.
AI Citation Preview
See how a page reads to Google, social cards and AI answer engines.
Bot User-Agent & IP Verifier
Paste a user agent or IP to identify the bot — and prove it's genuine.
Stop checking by hand. Kitbase watches this for you.
Bot & Crawler Analytics runs continuously on your real traffic and your real pages — with alerts when something changes, not when you remember to look.