Detecting AI Crawlers with Cloudflare Workers

A Cloudflare Worker is one of the cleanest places to detect AI crawlers: it sits in front of your origin, sees every request before your app does, and can forward each visitor’s signals to Kitbase without adding a millisecond of latency. The Worker serves the page first, then reports the request asynchronously with ctx.waitUntil so the visitor’s response is never held up by analytics. Crawlers that never run JavaScript, like GPTBot and ClaudeBot, get classified server-side and land in your dashboard, verified against spoofing.

If your site is already behind Cloudflare, this is a fifteen-line fetch handler and a single secret. This guide walks through the Worker, why it adds no latency, and how measuring crawlers relates to Cloudflare’s own tools for blocking them.

Why a Worker sees what your analytics can’t.

JavaScript analytics measures humans running a browser. AI crawlers are the opposite: they fetch your server-rendered HTML and leave, never executing the tracking snippet. That’s why tag-based tools report zero AI crawler traffic not because the bots aren’t visiting, but because a script that requires JavaScript can’t observe a client that doesn’t run any.

A Cloudflare Worker runs on the request itself, before anything renders. Every hit passes through it (human or bot, JavaScript or not) which makes it the right layer to detect crawlers. And because Cloudflare terminates the connection at its edge, the Worker has the true client IP in cf-connecting-ip, which is exactly what Kitbase needs to verify a crawler against its vendor’s published ranges.

The Worker.

Add a secret first, then deploy the handler. The Cloudflare Workers forwarder guide is the canonical reference; this mirrors it.

Prerequisite: add a Worker secret:

• KITBASE_API_KEY : your project’s secret API key (sk_kitbase_…), not the browser SDK key. The SDK key is public and won’t authenticate server-side ingestion.

The handler:

export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request); // serve the page first
    ctx.waitUntil(fetch("https://ingest.kitbase.dev/ingest/v1/server", {
      method: "POST",
      headers: { "authorization": `Bearer ${env.KITBASE_API_KEY}`, "content-type": "application/json" },
      body: JSON.stringify({ events: [{
        user_agent: request.headers.get("user-agent"),
        ip_address: request.headers.get("cf-connecting-ip"),
        method: request.method,
        host: new URL(request.url).host,
        path: new URL(request.url).pathname,
        referrer: request.headers.get("referer"),
        signature: request.headers.get("signature"),
        signature_input: request.headers.get("signature-input"),
        signature_agent: request.headers.get("signature-agent"),
      }]}),
    }));
    return response;
  },
};

Two design choices make this safe to run on production traffic:

The page is served first. const response = await fetch(request) resolves your origin’s response before anything else happens, and it’s what gets returned to the visitor. The analytics call comes after.

The report runs after the response. ctx.waitUntil hands the forwarding fetch to the Cloudflare runtime to finish after the response has been sent. The visitor never waits on it, and if the ingest call is slow or fails, their page load is unaffected. Analytics adds zero perceived latency.

flowchart LR
A["Request"] --> W["Cloudflare Worker"]
W --> O["Origin"]
O --> W
W -->|"response<br/>(visitor waits here)"| U["Visitor"]
W -.->|"ctx.waitUntil<br/>(after response)"| K["Kitbase ingest"]

The signature headers

The three signature* fields forward Web Bot Auth headers when a crawler sends them. Web Bot Auth is a cryptographic scheme that lets a bot prove its identity with a signature rather than just claiming it in the user-agent. Forwarding these lets Kitbase mark a request verified with certainty and flag anything claiming a known crawler’s name without the matching signature or IP as spoofed. It’s how you tell the real PerplexityBot from a scraper wearing its name.

What lands in your dashboard.

For every forwarded request, Kitbase decides human or bot in memory. Humans are discarded. Bots and crawlers are kept with:

• A named classification: GPTBot, ClaudeBot, PerplexityBot, Googlebot, and the rest, from the list of AI crawlers Kitbase recognizes.
• A verified-or-spoofed verdict: from Web Bot Auth signatures and vendor IP ranges.
• The path they requested: so you can see which pages the models are actually reading.
• Geography: derived from cf-connecting-ip, after which the raw IP is dropped unless IP logging is enabled.

Measuring crawlers vs. blocking them.

Here’s the part specific to Cloudflare: the platform has its own, growing set of features for blocking AI crawlers. managed bot rules, one-click “block AI bots” toggles, and crawler-control products. It’s easy to assume those replace the need for a Worker like this one. They don’t. They do a different job.

Blocking without measuring is a decision made blind. Before you toggle GPTBot off, you want to know what it reads, how often, and whether cutting it off also costs you presence in ChatGPT’s answers, because blocking a training crawler is not the same as blocking search. The Worker gives you that data; Cloudflare’s controls act on it. Run both: measure with the Worker, then use Cloudflare’s tools to enforce whatever policy the data justifies. Our guide to Cloudflare’s AI crawler blocking covers the enforcement side in depth.

Portable to other hosts.

The forward-with-waitUntil pattern isn’t Cloudflare-specific. It’s the general shape of edge crawler detection. If you run a mixed stack, the equivalents land in the same dashboard:

• Vercel: a zero-code Log Drain streams request logs, no Worker needed.
• nginx: ship the access log as JSON for self-hosted origins.

FAQ

Does the Worker slow down my site? No. It serves your origin’s response first and forwards the analytics call with ctx.waitUntil, which runs after the response is already on its way to the visitor. There’s no added latency on the critical path.

Do Cloudflare’s built-in AI bot controls make this unnecessary? No, they solve a different problem. Cloudflare’s controls block bots; the Worker measures them. You need the measurement to decide what’s worth blocking, and to catch spoofers claiming a crawler’s identity. Most teams run both.

How does Kitbase know a crawler is real and not spoofed? It checks the client IP (cf-connecting-ip) against the vendor’s published ranges and validates a Web Bot Auth signature when the crawler sends one. A request claiming to be GPTBot from outside OpenAI’s ranges, with no valid signature, is flagged spoofed.

Which API key does the Worker use? The project’s secret key (sk_kitbase_…), stored as a Worker secret. Don’t use the browser SDK key as it’s public and only works for client-side web analytics, not server-side ingestion.

What about human visitors passing through the Worker? They’re forwarded too, but Kitbase classifies them in memory and discards them immediately. Only bot and crawler requests are stored, so your visitors’ data isn’t retained by this integration.

Want to see every AI crawler behind Cloudflare — verified, attributed, and per-page — before you decide what to block? Start your free trial — 7 days, no credit card required — and deploy the Worker in minutes.