Cloudflare's AI Crawler Controls: Block, Charge, or Measure?
Cloudflare lets you block AI crawlers, charge them with Pay Per Crawl, or audit them with AI Crawl Control. A neutral explainer — and the measurement gap it leaves.
Cloudflare gives site owners three distinct ways to handle AI crawlers: block them, charge them for access, or audit what they’re doing — and the three are easy to confuse. In July 2025 Cloudflare became the first major infrastructure provider to put AI-crawler control front and center, prompting every new domain to choose whether to allow AI crawlers at signup, launching a Pay Per Crawl beta that lets sites charge crawlers, and building the analytics feature now called AI Crawl Control (formerly AI Audit). This is a neutral walkthrough of what each one does, when it fits — and the measurement gap that opens up if you block or charge without also measuring.
The three controls, side by side
| Feature | What it does | Best for |
|---|---|---|
| AI crawler blocking | Deny AI bots at the edge, individually or in bulk | Keeping specific crawlers off your site |
| Pay Per Crawl | Return HTTP 402 Payment Required; crawlers pay to access | Monetizing content access (beta) |
| AI Crawl Control (formerly AI Audit) | Dashboard of which AI bots visit and what they read | Seeing AI crawler activity |
These solve different problems. Blocking is enforcement, Pay Per Crawl is monetization, and AI Crawl Control is measurement. Most of the confusion — and most of the mistakes — comes from reaching for one when you needed another.
Control 1: blocking AI crawlers
Since July 1, 2025, every new domain added to Cloudflare is asked at signup whether it wants to allow AI crawlers — Cloudflare’s framing is that new domains “start with the default of control,” a deliberate choice rather than a silent free-for-all. Existing sites can toggle managed rules that recognize and block known AI bots at the edge.
Edge blocking is genuinely stronger than robots.txt. A Disallow in robots.txt is an honor system — a compliant bot obeys it, a spoofer ignores it. Cloudflare sits in front of your origin and can actually refuse the request, spoofer or not, based on its own bot classification. If your goal is to stop a bot rather than politely ask it to leave, edge enforcement is the tool.
The catch is the same one that applies to any blocking decision: once you block a bot at the edge, it never reaches your origin — and it never reaches your own analytics or detection either. Cloudflare knows it blocked something, but if you rely on origin logs or a server-side detector, a blocked crawler simply disappears from your data. That’s fine if you’re certain you want it gone. It’s a problem if you’re still deciding, because you’ve removed the very evidence a good decision needs.
Control 2: Pay Per Crawl
Pay Per Crawl, launched in private beta in mid-2025, is Cloudflare’s attempt to build a marketplace between publishers and AI companies. The mechanics use HTTP’s payment status code:
- A crawler requests a page. If the publisher charges for it, Cloudflare returns
HTTP 402 Payment Requiredwith acrawler-priceheader stating the price. - The crawler can retry signaling agreement to pay, or send a
crawler-max-priceheader upfront; if the price is acceptable it getsHTTP 200 OKand acrawler-chargedheader. - Publishers set a flat, domain-wide price and choose per crawler to Allow, Charge, or Block. Cloudflare acts as the Merchant of Record, aggregating charges and distributing earnings.
Notably, Pay Per Crawl leans on the same cryptographic identity layer as Web Bot Auth: crawlers register Ed25519 keys and sign requests with HTTP Message Signatures, so Cloudflare can bill the right operator. It’s an early experiment — the broader model of compensating content owners for AI access is still evolving across the industry — but the direction is unmistakable: identity, access, and payment are converging.
Pay Per Crawl fits publishers whose content is the product. For a marketing site that wants to be crawled and cited, charging crawlers is usually counterproductive — you’d be pricing yourself out of the AI answers you’re trying to appear in. Which stance is right for you is exactly the question our decision framework walks through.
Control 3: AI Crawl Control (formerly AI Audit)
The third feature is the one people overlook, and it’s the most broadly useful: measurement. AI Crawl Control gives Cloudflare customers a dashboard of AI crawler activity — no configuration required — broken into overview, per-crawler, and metrics views. It shows activity by AI provider and bot type, which sections of your site are most popular with crawlers, and referrer patterns.
This is the honest starting point. Before you block or charge anything, you want to know which AI bots actually visit, how often, and what they read — the same “measure first” logic that should precede any blocking decision. AI Crawl Control answers those questions for sites on Cloudflare, and it’s a real strength of the platform.
The measurement gap
Here’s the tension that ties all three together. Blocking and charging both remove crawlers from your view. Measuring is the only control that adds to it. If your strategy is “block the AI bots,” you’ve also blocked your ability to observe them — and a decision you can’t observe is a decision you can’t revisit. The most defensible posture is to measure continuously, and block or charge deliberately on top of that evidence, not instead of it.
There’s also a portability point worth being honest about. Cloudflare’s AI Crawl Control is excellent — if all your traffic goes through Cloudflare. Many sites don’t: multi-CDN setups, origins behind other proxies, edge functions on Vercel or AWS, or simply not being on Cloudflare at all. Bot data that lives inside one provider’s dashboard is bot data you can’t unify with the rest of your analytics or carry with you if you switch.
This is where a provider-independent detector complements Cloudflare rather than competing with it. Kitbase’s bot & crawler detection forwards your server or edge requests to a classifier that runs the same identity checks — verified-or-spoofed verdicts against vendors’ published IP ranges and Web Bot Auth signatures when present — and stores each AI crawler visit with its name, vendor, category, and the pages it read. It works whether you’re on Cloudflare (via a Workers forwarder), Vercel, nginx, WordPress, or a mix, so your AI-crawler dataset isn’t tied to one edge provider. And it closes the loop the other way too: AI Visibility tells you whether all that crawling actually turns into your brand being cited in AI answers — the outcome no blocking or charging control measures.
Use Cloudflare’s controls for what they’re built for: block at the edge when you need enforcement, charge with Pay Per Crawl if your content is the product. Just don’t let “block” quietly become “blind.” For the full roster of bots you’ll see in any of these dashboards, see the list of AI crawlers, and for the most common one, GPTBot explained.
FAQ
Does Cloudflare block AI crawlers by default? Since July 2025, new domains are prompted at signup to choose whether to allow AI crawlers — Cloudflare frames it as starting “with the default of control.” It’s a required choice rather than a silent block, and existing sites configure it via managed rules.
What is Cloudflare Pay Per Crawl?
A beta feature that lets publishers charge AI crawlers for access using HTTP 402 Payment Required. Crawlers pay a flat, domain-wide price via signed requests, with Cloudflare acting as Merchant of Record. It suits sites whose content is the product, not marketing sites that want to be crawled.
What is Cloudflare AI Audit? AI Audit — now called AI Crawl Control — is Cloudflare’s dashboard for seeing which AI bots crawl your site, how often, and which pages they favor. It’s available to Cloudflare customers with no setup and is the “measure” option among Cloudflare’s AI controls.
Should I block AI crawlers with Cloudflare or robots.txt? Edge blocking (Cloudflare) actually enforces — it can refuse a spoofer that ignores robots.txt. robots.txt only asks compliant bots to stay away. Use edge rules when you need enforcement, robots.txt for polite signaling, and measurement underneath both.
What if my site isn’t on Cloudflare? Then Cloudflare’s AI Crawl Control isn’t available to you — but a provider-independent, server-side detector works on any stack (Vercel, nginx, WordPress, Node, and more) and keeps your AI-crawler data portable instead of locked to one edge provider.
Blocking AI crawlers is easy — knowing what you blocked is the hard part. Start your free trial — 7 days, no credit card required — and get provider-independent, spoof-checked AI crawler data on any stack.