---
title: "Bot & Crawler Analytics — See Which AI Crawlers Visit You | Kitbase"
description: "Detect and attribute AI crawlers and search bots hitting your site: verified identities, per-path crawl frequency, and forwarders for Next.js and Vercel."
canonical: https://kitbase.dev/bot-analytics
---

![Server racks in a data centre](/images/product/bots-hero.jpg)

Bot & Crawler Analytics

# See who's crawling your site

AI assistants and search engines read your content around the clock, and your normal analytics deliberately hide them. Kitbase makes them visible: which crawlers arrive, which pages they fetch, and whether they are who they claim to be.

[Start free](https://app.kitbase.dev/signup) [Browse the bot directory](/bot-directory)

Verified

Checked against spoofing

Per-path

Crawl frequency by URL

Separate

Never mixed with humans

Zero-code

On Vercel and CloudFront

Top crawlers · last 7 days

GPTBot OpenAI

18.4K

ClaudeBot Anthropic

15.1K

Googlebot Google

11.8K

PerplexityBot Perplexity

7.6K

Bingbot Microsoft

6.1K

Bytespider ByteDance

4.0K

Attribution

## Every request resolved to a named crawler

A hit isn't just a user-agent string. Kitbase resolves each one to the crawler and the vendor behind it, so you can tell who is reading your content and how much of it.

-   AI crawlers named individually — GPTBot, ClaudeBot, PerplexityBot and hundreds more
-   User-agent claims are verified against the crawler's published identity
-   Per-path frequency shows what's being read and what's ignored
-   Spot a new or unusually aggressive crawler the moment it appears

Verification

## A user-agent string proves nothing

Anything can claim to be Googlebot — spoofing a header is trivial, and scrapers do it constantly to get past naive blocks. Kitbase checks each claim against the crawler's published identity, so a verified hit means the bot really is who it says.

-   Every detected request is marked verified or spoofed
-   Real crawlers are separated from things wearing their name
-   Attribution resolves the crawler and the vendor behind it

![Network cables plugged into a switch](/images/product/bots-detail.jpg)

Detection vs. filtering

## Two features, opposite goals

Kitbase does two different things with bots. It helps to know which one you're looking at — both use the same pipeline to decide what counts as a bot, they just do opposite things with the answer.

|  | 
Bot & Crawler Analytics

This page — opt-in

 | 

Bot filtering

Built into web analytics — automatic

 |
| --- | --- | --- |
| Goal | Make bots visible — see which crawlers and AI bots visit your site | Keep bots out of your human analytics |
| Where it runs | Your server or edge, forwarding requests to Kitbase | The JS SDK in the browser, plus the server-side pipeline |
| What happens to bots | Stored as their own dataset with attribution — name, vendor, verified or spoofed | Silently discarded; they never reach your dashboards or billing counts |
| Setup | Opt-in — install a forwarder for your platform | Automatic, no configuration |

Setup

## Pick your platform and paste

Detection runs at your server or edge, forwarding requests to Kitbase's ingest endpoint with your secret API key. Humans are ignored at the endpoint, so it's safe to forward everything.

[

Next.js

Edge middleware

](https://docs.kitbase.dev/crawler-detection/nextjs)[

TanStack Start

Request middleware

](https://docs.kitbase.dev/crawler-detection/tanstack-start)[

Node.js / Express

App middleware

](https://docs.kitbase.dev/crawler-detection/node-express)[

WordPress

Must-use plugin

](https://docs.kitbase.dev/crawler-detection/wordpress)[

Cloudflare Workers

fetch handler

](https://docs.kitbase.dev/crawler-detection/cloudflare-workers)[

Vercel

Log Drain · zero code

](https://docs.kitbase.dev/crawler-detection/vercel)[

AWS CloudFront

Standard logs · zero code

](https://docs.kitbase.dev/crawler-detection/cloudfront)[

nginx

Access-log shipping

](https://docs.kitbase.dev/crawler-detection/nginx)

The first time you open the Crawlers page, a wizard walks you through creating the key, installing the forwarder, and live-verifying that the first crawler arrives.

Not to be confused with

## Crawlers visiting you, not answers about you

Bot analytics tells you which AI crawlers fetch your pages. AI Visibility tells you whether those engines actually mention your brand when someone asks. They're complementary — one measures the input, the other the output.

-   Bot analytics — which crawlers arrive, how often, and on which paths
-   AI Visibility — whether ChatGPT, Gemini and Perplexity name you in answers
-   Together they connect what gets read to what gets said

[See AI Visibility](/ai-visibility)

![Source code on a screen, showing middleware setup](/images/product/bots-setup.jpg)

## Find out who's reading your content

Install a forwarder and watch the crawlers arrive — named, attributed, and verified against spoofing.

[Start 7-day free trial](https://app.kitbase.dev/signup) [Read the docs](https://docs.kitbase.dev/crawler-detection)

No credit card required