How to Track AI Crawler Traffic on a Next.js Site

To track AI crawler traffic on a Next.js site, forward each request from Edge Middleware to a server-side detection endpoint that classifies the visitor — because AI crawlers never run your client-side analytics, even when your pages are server-rendered. A middleware.ts file at your project root runs in the Edge runtime on every matched request, ships the visitor’s signals to Kitbase without delaying the page, and lets the classification happen server-side. Human requests are ignored; only bots and crawlers are stored. This guide walks through the setup, mirrors the official Next.js crawler-detection guide, and explains the one thing most people get wrong about Next.js and bots.

Why JavaScript analytics misses bots on Next.js — even with SSR

A common assumption: “My Next.js app is server-rendered, so my analytics sees everything.” It doesn’t, and the reason is worth understanding.

Client-side analytics — Google Analytics, Plausible, any tag-based tool — records a visit only when the browser executes its tracking script. Server-side rendering changes how the HTML is produced; it doesn’t change who runs the JavaScript afterward. AI crawlers like GPTBot, ClaudeBot, and PerplexityBot fetch your server-rendered HTML and stop there. They don’t hydrate the page, don’t run your analytics snippet, and never fire a single event. So your dashboard shows zero AI crawler traffic — not because none exists, but because tag-based tools physically can’t see it.

The fix has to live where the crawler actually is: on the server, at request time. In Next.js, the cleanest place for that is Edge Middleware, which runs on every request before your page renders — for humans and bots alike.

Prerequisites

Set a server-only environment variable. Do not prefix it with NEXT_PUBLIC_ — that would expose your secret key to the browser:

• KITBASE_API_KEY — your project’s secret API key (sk_kitbase_…), not the browser SDK key.

You’ll find the secret key in your Kitbase project settings, or the guided setup wizard will hand it to you during onboarding.

The middleware

Add a middleware.ts file at your project root (or in src/). It reads the request signals, forwards them with event.waitUntil so the response is never delayed, and returns immediately. Because request.ip was removed in Next.js 15, the client IP is read from the x-forwarded-for header:

// middleware.ts
import { NextResponse, type NextRequest, type NextFetchEvent } from "next/server";

export function middleware(request: NextRequest, event: NextFetchEvent) {
  const fwd = request.headers.get("x-forwarded-for");
  event.waitUntil(
    fetch("https://ingest.kitbase.dev/ingest/v1/server", {
      method: "POST",
      headers: { authorization: `Bearer ${process.env.KITBASE_API_KEY}`, "content-type": "application/json" },
      body: JSON.stringify({ events: [{
        user_agent: request.headers.get("user-agent"),
        ip_address: fwd?.split(",")[0]?.trim(),
        method: request.method,
        host: request.headers.get("host"),
        path: request.nextUrl.pathname,
        referrer: request.headers.get("referer"),
        signature: request.headers.get("signature"),
        signature_input: request.headers.get("signature-input"),
        signature_agent: request.headers.get("signature-agent"),
      }]}),
    }).catch(() => {}), // never block the response
  );

  return NextResponse.next();
}

export const config = {
  // Page requests only — skip static assets and Next internals to avoid noise.
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

That’s the whole integration. Deploy it, and the next crawler to hit a matched route shows up in Kitbase.

What the pieces do

• event.waitUntil(...) lets the forwarding fetch finish after the response is sent. The page is never held up waiting on the network call — the visitor gets their HTML immediately, and the signal ships in the background.
• .catch(() => {}) makes the call fire-and-forget. If Kitbase is briefly unreachable, your site doesn’t error — the request is simply not recorded.
• The signature* headers are part of Web Bot Auth. Forwarding them lets Kitbase cryptographically verify a crawler’s claimed identity when the crawler signs its requests — the strongest form of verification, stronger than an IP-range check alone.
• The matcher limits forwarding to real page requests, skipping _next/static, _next/image, and favicon.ico so you don’t fill your data with asset noise.

flowchart LR
A["Incoming request<br/>human or bot"] --> B["Edge Middleware"]
B --> C["Page response<br/>returned immediately"]
B -.->|"waitUntil<br/>background"| D["Kitbase classifies<br/>server-side"]
D --> E["Bot? store with<br/>path + verdict"]
D --> F["Human? discard<br/>in memory"]

Performance and privacy notes

Two questions always come up, and the design answers both.

Does forwarding every request slow my site? No. The classification work happens on Kitbase’s side, and the event.waitUntil pattern means the network call runs after your response is already on its way to the visitor. Edge Middleware is designed to run on every request; the added work here is a single non-blocking fetch.

Does forwarding every request mean storing every visitor? No. Human visitors’ signals are used only to classify the request in memory and are then discarded — only bot and crawler requests are persisted. For those, the raw IP is stored only if you enable IP logging for the project; otherwise it’s used to derive geolocation (country, region, city) and then dropped. It’s safe to run on every request precisely because humans never end up in the dataset.

What you see afterwards

Once data is flowing, the Kitbase dashboard turns Next.js crawler traffic into something you can actually read:

• Which AI bots visit — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot, and the rest, each identified rather than lumped into “bots.”
• Which pages each bot crawls most — the per-path view that tells you which pages AI bots read most, so you can see whether your best content is reachable or quietly being skipped.
• Verified vs. spoofed — every hit labelled real or impostor using published IP ranges and Web Bot Auth signatures, so a scraper wearing a GPTBot user agent doesn’t inflate your numbers.
• Geography and trends — where crawlers come from and whether volume is rising, flat, or suddenly dropped (a drop to zero usually means an accidental block).

For the full request schema and every attribution field, see the Next.js guide and the API reference. To put your Next.js coverage in context with the rest of the AI crawlers out there, the list of AI crawlers is the field guide.

Hosted on Vercel? Skip the middleware

If your Next.js app runs on Vercel, you can capture crawler requests — static assets and edge routes included — with zero code using a Vercel Log Drain instead of middleware. The log drain streams every request Vercel handles straight to Kitbase, so you get coverage even for routes middleware wouldn’t match. Middleware is the right choice when you want tight control or you’re hosted elsewhere; the log drain is the right choice when you want the least code on Vercel specifically.

FAQ

Does Next.js middleware slow down page loads? Not meaningfully here. The forwarding fetch runs inside event.waitUntil, which executes after the response is sent, so the page isn’t held up. Middleware itself is built to run on every request in the Edge runtime.

Why can’t my Next.js analytics see GPTBot if my site is server-rendered? Because SSR changes how the HTML is generated, not who runs the JavaScript. GPTBot fetches the server-rendered HTML and never executes your client-side analytics snippet, so tag-based tools record nothing. You need server-side or edge-level detection, which is what the middleware provides.

Do I need the signature headers in the middleware? They’re optional but recommended. They carry Web Bot Auth signatures, which let Kitbase cryptographically verify a crawler’s identity when it signs requests — stronger than IP-range verification alone. Forwarding them costs nothing when they’re absent.

What if the Kitbase request fails? It’s fire-and-forget. The .catch(() => {}) swallows any error, so a network blip or Kitbase downtime never affects your visitors — that one request just isn’t recorded.

Should I use middleware or a Vercel Log Drain? On Vercel, the log drain is zero-code and captures static assets too. Off Vercel — or when you want explicit control over what’s forwarded — use the middleware shown here.

Want to see every AI crawler hitting your Next.js app, verified and per-page? Start your free trial — 7 days, no credit card required — and the setup guide has data flowing in minutes.