---
title: "Shipping nginx Access Logs for AI Bot Analytics | Kitbase Blog"
description: "Turn your nginx access log into an AI-crawler dashboard: log request signals as JSON, ship them to Kitbase, and see GPTBot and ClaudeBot server-side."
canonical: https://kitbase.dev/blog/nginx-bot-analytics
---

**If you run your own nginx, the AI crawlers hitting your site are already recorded in the access log. you just need to ship those lines somewhere that can classify them.** GPTBot, ClaudeBot, PerplexityBot, and Googlebot all leave a line in `access.log`; none of them run JavaScript, so your tag-based analytics never saw them. The reliable pattern on nginx is two steps: **log** the handful of signals Kitbase needs as structured JSON, then **ship** that log to the ingestion endpoint in batches. No per-request API calls, no third-party modules.

This is the self-hoster's and VPS operator's path to AI-crawler analytics. If you've ever run `grep -i gptbot access.log` and wished the result were a live, verified, per-page dashboard instead of a wall of text, this is how you get there. It's also the natural upgrade from ad-hoc [log file analysis](/blog/log-file-analysis-ai-era): the same data, continuously classified.

## Why log and ship, not call per request.

nginx can't cleanly make an outbound HTTP call on every request without compiling in extra modules (`ngx_http_lua`, `njs`, and friends) and you don't want a blocking network call in your request path anyway. The robust approach plays to what nginx already does well: it writes a structured line for every request, and a separate lightweight process forwards those lines asynchronously. Your request path stays fast; the shipping happens out of band.

**Log-and-ship: nginx writes, a shipper forwards**

```mermaid
flowchart LR
  A["Request"] --> N["nginx"]
  N -->|"response"| U["Visitor"]
  N --> L["kitbase.log<br/>(JSON lines)"]
  L -->|"tail + batch"| S["Log shipper"]
  S -->|"POST /ingest/v1/server"| K["Kitbase"]
```

## 1. Log the signals as JSON.

Add a JSON `log_format` and write it to a dedicated access log. This follows the [nginx forwarder guide](https://docs.kitbase.dev/crawler-detection/nginx) exactly:

```nginx
log_format kitbase escape=json '{'
    '"user_agent":"$http_user_agent",'
    '"ip_address":"$remote_addr",'
    '"method":"$request_method",'
    '"host":"$host",'
    '"path":"$uri",'
    '"referrer":"$http_referer"'
'}';

access_log /var/log/nginx/kitbase.log kitbase;
```

A few things worth understanding:

- `escape=json` makes nginx safely escape quotes and control characters, so a hostile user-agent can't corrupt the line.
- `$uri` is the **normalized** request path (decoded and free of the query string) which is what you want for grouping by page.
- There's **no timestamp field**. Kitbase stamps each event at receipt time, so you don't have to reconcile clock skew between your server and the ingest endpoint.
- A **dedicated** log (`kitbase.log`, not your main `access.log`) keeps this stream clean and easy to rotate independently.

If nginx sits behind another proxy or load balancer, `$remote_addr` will be the proxy's IP, not the visitor's. Use the first IP from `$http_x_forwarded_for` for `ip_address` instead. the real client IP is what lets Kitbase verify a crawler against its vendor's published ranges.

## 2. Ship the log to Kitbase.

You need one secret and one shipper.

**Prerequisite:** `KITBASE_API_KEY`, your project's **secret** API key (`sk_kitbase_…`). This is the server-side key, not the public browser SDK key.

A shipper tails the log, reads new lines, and POSTs them to `https://ingest.kitbase.dev/ingest/v1/server` wrapped as `{ "events": [ ... ] }`, up to **500** events per call. You have two good options.

### Option A: a production log shipper (recommended).

For anything beyond a hobby box, use a real log shipper [Vector](https://vector.dev) or [Fluent Bit](https://fluentbit.io) with an HTTP sink. These handle the tedious parts you'd otherwise reimplement: tailing across log rotation, retries with backpressure, and batching. Point the sink at the ingest endpoint, add the `Authorization: Bearer sk_kitbase_…` header, and configure it to wrap each batch as `{ "events": [ ... ] }`. That's a durable, restart-safe pipeline.

### Option B: a minimal tailer.

For a single VPS or a quick trial, a tiny Node script is enough. It's the same idea in a few line. Tail the JSON log, buffer parsed lines, and flush a batch on an interval:

```js
// kitbase-shipper.js — tail the nginx JSON log and POST batches

const tail = spawn("tail", ["-n0", "-F", "/var/log/nginx/kitbase.log"]);
let buffer = [];

async function flush() {
  if (buffer.length === 0) return;
  const events = buffer.splice(0, 500);
  await fetch("https://ingest.kitbase.dev/ingest/v1/server", {
    method: "POST",
    headers: { authorization: `Bearer ${process.env.KITBASE_API_KEY}`, "content-type": "application/json" },
    body: JSON.stringify({ events }),
  }).catch(() => {});
}

tail.stdout.on("data", (chunk) => {
  for (const line of chunk.toString().split("\n")) {
    if (line.trim()) try { buffer.push(JSON.parse(line)); } catch {}
  }
});

setInterval(flush, 5000); // flush every 5s
```

Run it under a process manager (`systemd`, `pm2`) so it restarts on reboot. It's deliberately forgiving. Malformed lines are skipped, and a failed POST is dropped rather than crashing the tailer. For a production system where losing a batch matters, prefer Vector or Fluent Bit.

Whichever you choose, Kitbase does the same thing on receipt: it classifies each event, **ignores human requests** entirely, and stores only bot and crawler hits. Forwarding your whole access log doesn't mean storing your whole access log.

## What you get.

Once lines are flowing, the raw log becomes a structured, live view of AI crawler activity:

- **Named crawlers:** every hit matched to a bot from the [list of AI crawlers](/blog/list-of-ai-crawlers), by user-agent token rather than exact string, so a version bump like `GPTBot/1.4` doesn't break detection.
- **Verified vs. spoofed:** each hit's IP checked against the vendor's published ranges, so a scraper sending `GPTBot` from a random VPS is flagged, not counted. To check one line from `kitbase.log` by hand, paste its `user_agent` and `ip_address` into the free [bot verifier](https://kitbase.dev/tools/bot-verifier), which matches the IP against the operator's published ranges and reverse DNS and returns a verified or spoofed verdict.
- **Top crawled paths:** which pages the models read most. This is the same question a [server-log crawler audit](/blog/ai-crawler-audit-server-logs) answers, except continuously and with verification built in.
- **Trends and geography:** crawl volume over time (a sudden drop can reveal an accidental block), and country derived from the IP before it's discarded.

## Portable across your stack.

Log-and-ship is nginx's version of a pattern every host has. If your infrastructure is mixed, the equivalents land in the same dashboard with the same verified/spoofed classification:

- **Cloudflare:** a [Worker forwards requests](/blog/detect-ai-crawlers-cloudflare-workers) with `waitUntil`.
- **Vercel:** a [zero-code Log Drain](/blog/track-bots-vercel-log-drains) streams request logs.

## FAQ

**Do I need a special nginx module to send data to Kitbase?**
No. You use nginx's built-in `log_format` to write JSON lines, then a separate shipper forwards them. No Lua, njs, or compiled modules required.

**Why not just POST from nginx on every request?**
A blocking outbound call in the request path adds latency and a failure mode to every page load. Logging locally and shipping asynchronously keeps nginx fast and decouples your site's uptime from the analytics endpoint's.

**My nginx is behind a load balancer will that mean the IPs be wrong?**
By default `$remote_addr` would be the load balancer's IP. Switch `ip_address` to the first IP in `$http_x_forwarded_for` so Kitbase sees the real client IP and can verify crawlers correctly.

**How many events can I send per request?**
Up to 500 events per POST, wrapped as `{ "events": [ ... ] }`. A production shipper like Vector or Fluent Bit handles batching to that limit for you.

**Will this store data about my human visitors?**
No. Kitbase classifies every forwarded line in memory and discards the human ones. Only bot and crawler requests are stored, and their raw IP is dropped after geolocation unless you enable IP logging.

---

*Want your nginx access log turned into a live AI-crawler dashboard — verified, per-page, no JavaScript required? [Start your free trial](https://app.kitbase.dev/signup/) — 7 days, no credit card required — and ship your first batch in minutes.*