---
title: "How to Track AI Crawler Traffic on a Next.js Site | Kitbase Blog"
description: "Track GPTBot, ClaudeBot, and other AI crawlers on Next.js with Edge middleware. Why SSR analytics still misses bots, plus the exact middleware.ts to forward requests."
canonical: https://kitbase.dev/blog/track-ai-crawlers-nextjs/
---

**To track AI crawler traffic on a Next.js site, forward each request from Edge Middleware to a server-side detection endpoint that classifies the visitor — because AI crawlers never run your client-side analytics, even when your pages are server-rendered.** A `middleware.ts` file at your project root runs in the Edge runtime on every matched request, ships the visitor's signals to Kitbase without delaying the page, and lets the classification happen server-side. Human requests are ignored; only bots and crawlers are stored. This guide walks through the setup, mirrors the [official Next.js crawler-detection guide](https://docs.kitbase.dev/crawler-detection/nextjs), and explains the one thing most people get wrong about Next.js and bots.

## Why JavaScript analytics misses bots on Next.js — even with SSR

A common assumption: "My Next.js app is server-rendered, so my analytics sees everything." It doesn't, and the reason is worth understanding.

Client-side analytics — Google Analytics, Plausible, any tag-based tool — records a visit only when the browser executes its tracking script. Server-side rendering changes *how the HTML is produced*; it doesn't change *who runs the JavaScript afterward*. AI crawlers like [GPTBot](/blog/gptbot-explained/), ClaudeBot, and PerplexityBot fetch your server-rendered HTML and stop there. They don't hydrate the page, don't run your analytics snippet, and never fire a single event. So your dashboard shows zero AI crawler traffic — not because none exists, but because [tag-based tools physically can't see it](/blog/why-analytics-cant-see-ai-crawlers/).

The fix has to live where the crawler actually is: on the server, at request time. In Next.js, the cleanest place for that is [Edge Middleware](https://nextjs.org/docs/app/building-your-application/routing/middleware), which runs on every request before your page renders — for humans and bots alike.

## Prerequisites

Set a **server-only** environment variable. Do not prefix it with `NEXT_PUBLIC_` — that would expose your secret key to the browser:

- `KITBASE_API_KEY` — your project's **secret** API key (`sk_kitbase_…`), not the browser SDK key.

You'll find the secret key in your Kitbase project settings, or the guided setup wizard will hand it to you during onboarding.

## The middleware

Add a `middleware.ts` file at your project root (or in `src/`). It reads the request signals, forwards them with `event.waitUntil` so the response is never delayed, and returns immediately. Because `request.ip` was removed in Next.js 15, the client IP is read from the `x-forwarded-for` header:

```ts
// middleware.ts

export function middleware(request: NextRequest, event: NextFetchEvent) {
  const fwd = request.headers.get("x-forwarded-for");
  event.waitUntil(
    fetch("https://ingest.kitbase.dev/ingest/v1/server", {
      method: "POST",
      headers: { authorization: `Bearer ${process.env.KITBASE_API_KEY}`, "content-type": "application/json" },
      body: JSON.stringify({ events: [{
        user_agent: request.headers.get("user-agent"),
        ip_address: fwd?.split(",")[0]?.trim(),
        method: request.method,
        host: request.headers.get("host"),
        path: request.nextUrl.pathname,
        referrer: request.headers.get("referer"),
        signature: request.headers.get("signature"),
        signature_input: request.headers.get("signature-input"),
        signature_agent: request.headers.get("signature-agent"),
      }]}),
    }).catch(() => {}), // never block the response
  );

  return NextResponse.next();
}

export const config = {
  // Page requests only — skip static assets and Next internals to avoid noise.
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};
```

That's the whole integration. Deploy it, and the next crawler to hit a matched route shows up in Kitbase.

### What the pieces do

- **`event.waitUntil(...)`** lets the forwarding `fetch` finish *after* the response is sent. The page is never held up waiting on the network call — the visitor gets their HTML immediately, and the signal ships in the background.
- **`.catch(() => {})`** makes the call fire-and-forget. If Kitbase is briefly unreachable, your site doesn't error — the request is simply not recorded.
- **The `signature*` headers** are part of [Web Bot Auth](https://docs.kitbase.dev/track-events#bot-crawler-attribution). Forwarding them lets Kitbase cryptographically verify a crawler's claimed identity when the crawler signs its requests — the strongest form of verification, stronger than an IP-range check alone.
- **The `matcher`** limits forwarding to real page requests, skipping `_next/static`, `_next/image`, and `favicon.ico` so you don't fill your data with asset noise.

**How the Next.js middleware tracks crawlers without slowing the page**

```mermaid
flowchart LR
  A["Incoming request<br/>human or bot"] --> B["Edge Middleware"]
  B --> C["Page response<br/>returned immediately"]
  B -.->|"waitUntil<br/>background"| D["Kitbase classifies<br/>server-side"]
  D --> E["Bot? store with<br/>path + verdict"]
  D --> F["Human? discard<br/>in memory"]
```

## Performance and privacy notes

Two questions always come up, and the design answers both.

**Does forwarding every request slow my site?** No. The classification work happens on Kitbase's side, and the `event.waitUntil` pattern means the network call runs after your response is already on its way to the visitor. Edge Middleware is designed to run on every request; the added work here is a single non-blocking `fetch`.

**Does forwarding every request mean storing every visitor?** No. Human visitors' signals are used only to classify the request in memory and are then discarded — only bot and crawler requests are persisted. For those, the raw IP is stored only if you enable IP logging for the project; otherwise it's used to derive geolocation (country, region, city) and then dropped. It's safe to run on every request precisely because humans never end up in the dataset.

## What you see afterwards

Once data is flowing, the Kitbase dashboard turns Next.js crawler traffic into something you can actually read:

- **Which AI bots visit** — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot, and the rest, each identified rather than lumped into "bots."
- **Which pages each bot crawls most** — the per-path view that tells you [which pages AI bots read most](/blog/which-pages-ai-bots-crawl/), so you can see whether your best content is reachable or quietly being skipped.
- **Verified vs. spoofed** — every hit labelled real or impostor using published IP ranges and Web Bot Auth signatures, so a scraper wearing a `GPTBot` user agent doesn't inflate your numbers.
- **Geography and trends** — where crawlers come from and whether volume is rising, flat, or suddenly dropped (a drop to zero usually means an accidental block).

For the full request schema and every attribution field, see the [Next.js guide](https://docs.kitbase.dev/crawler-detection/nextjs) and the [API reference](https://docs.kitbase.dev/api-reference#server-side-crawler-detection). To put your Next.js coverage in context with the rest of the AI crawlers out there, the [list of AI crawlers](/blog/list-of-ai-crawlers/) is the field guide.

## Hosted on Vercel? Skip the middleware

If your Next.js app runs on Vercel, you can capture crawler requests — static assets and edge routes included — with **zero code** using a [Vercel Log Drain](https://docs.kitbase.dev/crawler-detection/vercel) instead of middleware. The log drain streams every request Vercel handles straight to Kitbase, so you get coverage even for routes middleware wouldn't match. Middleware is the right choice when you want tight control or you're hosted elsewhere; the log drain is the right choice when you want the least code on Vercel specifically.

## FAQ

**Does Next.js middleware slow down page loads?**
Not meaningfully here. The forwarding `fetch` runs inside `event.waitUntil`, which executes after the response is sent, so the page isn't held up. Middleware itself is built to run on every request in the Edge runtime.

**Why can't my Next.js analytics see GPTBot if my site is server-rendered?**
Because SSR changes how the HTML is generated, not who runs the JavaScript. GPTBot fetches the server-rendered HTML and never executes your client-side analytics snippet, so tag-based tools record nothing. You need server-side or edge-level detection, which is what the middleware provides.

**Do I need the `signature` headers in the middleware?**
They're optional but recommended. They carry [Web Bot Auth](https://docs.kitbase.dev/track-events#bot-crawler-attribution) signatures, which let Kitbase cryptographically verify a crawler's identity when it signs requests — stronger than IP-range verification alone. Forwarding them costs nothing when they're absent.

**What if the Kitbase request fails?**
It's fire-and-forget. The `.catch(() => {})` swallows any error, so a network blip or Kitbase downtime never affects your visitors — that one request just isn't recorded.

**Should I use middleware or a Vercel Log Drain?**
On Vercel, the [log drain](https://docs.kitbase.dev/crawler-detection/vercel) is zero-code and captures static assets too. Off Vercel — or when you want explicit control over what's forwarded — use the middleware shown here.

---

*Want to see every AI crawler hitting your Next.js app, verified and per-page? [Start your free trial](https://app.kitbase.dev/signup/) — 7 days, no credit card required — and the [setup guide](https://docs.kitbase.dev/crawler-detection/nextjs) has data flowing in minutes.*