Shipping nginx Access Logs for AI Bot Analytics

If you run your own nginx, the AI crawlers hitting your site are already recorded in the access log — you just need to ship those lines somewhere that can classify them. GPTBot, ClaudeBot, PerplexityBot, and Googlebot all leave a line in access.log; none of them run JavaScript, so your tag-based analytics never saw them. The reliable pattern on nginx is two steps: log the handful of signals Kitbase needs as structured JSON, then ship that log to the ingestion endpoint in batches. No per-request API calls, no third-party modules.

This is the self-hoster’s and VPS operator’s path to AI-crawler analytics. If you’ve ever run grep -i gptbot access.log and wished the result were a live, verified, per-page dashboard instead of a wall of text, this is how you get there. It’s also the natural upgrade from ad-hoc log file analysis: the same data, continuously classified.

Why log-and-ship, not call-per-request

nginx can’t cleanly make an outbound HTTP call on every request without compiling in extra modules (ngx_http_lua, njs, and friends) — and you don’t want a blocking network call in your request path anyway. The robust approach plays to what nginx already does well: it writes a structured line for every request, and a separate lightweight process forwards those lines asynchronously. Your request path stays fast; the shipping happens out of band.

flowchart LR
A["Request"] --> N["nginx"]
N -->|"response"| U["Visitor"]
N --> L["kitbase.log<br/>(JSON lines)"]
L -->|"tail + batch"| S["Log shipper"]
S -->|"POST /ingest/v1/server"| K["Kitbase"]

1. Log the signals as JSON

Add a JSON log_format and write it to a dedicated access log. This follows the nginx forwarder guide exactly:

log_format kitbase escape=json '{'
    '"user_agent":"$http_user_agent",'
    '"ip_address":"$remote_addr",'
    '"method":"$request_method",'
    '"host":"$host",'
    '"path":"$uri",'
    '"referrer":"$http_referer"'
'}';

access_log /var/log/nginx/kitbase.log kitbase;

A few things worth understanding:

• escape=json makes nginx safely escape quotes and control characters, so a hostile user-agent can’t corrupt the line.
• $uri is the normalized request path — decoded and free of the query string — which is what you want for grouping by page.
• There’s no timestamp field. Kitbase stamps each event at receipt time, so you don’t have to reconcile clock skew between your server and the ingest endpoint.
• A dedicated log (kitbase.log, not your main access.log) keeps this stream clean and easy to rotate independently.

If nginx sits behind another proxy or load balancer, $remote_addr will be the proxy’s IP, not the visitor’s. Use the first IP from $http_x_forwarded_for for ip_address instead — the real client IP is what lets Kitbase verify a crawler against its vendor’s published ranges.

2. Ship the log to Kitbase

You need one secret and one shipper.

Prerequisite — KITBASE_API_KEY, your project’s secret API key (sk_kitbase_…). This is the server-side key, not the public browser SDK key.

A shipper tails the log, reads new lines, and POSTs them to https://ingest.kitbase.dev/ingest/v1/server wrapped as { "events": [ ... ] }, up to 500 events per call. You have two good options.

Option A: a production log shipper (recommended)

For anything beyond a hobby box, use a real log shipper — Vector or Fluent Bit — with an HTTP sink. These handle the tedious parts you’d otherwise reimplement: tailing across log rotation, retries with backpressure, and batching. Point the sink at the ingest endpoint, add the Authorization: Bearer sk_kitbase_… header, and configure it to wrap each batch as { "events": [ ... ] }. That’s a durable, restart-safe pipeline.

Option B: a minimal tailer

For a single VPS or a quick trial, a tiny Node script is enough. It’s the same idea in a few lines — tail the JSON log, buffer parsed lines, and flush a batch on an interval:

// kitbase-shipper.js — tail the nginx JSON log and POST batches
import { spawn } from "node:child_process";

const tail = spawn("tail", ["-n0", "-F", "/var/log/nginx/kitbase.log"]);
let buffer = [];

async function flush() {
  if (buffer.length === 0) return;
  const events = buffer.splice(0, 500);
  await fetch("https://ingest.kitbase.dev/ingest/v1/server", {
    method: "POST",
    headers: { authorization: `Bearer ${process.env.KITBASE_API_KEY}`, "content-type": "application/json" },
    body: JSON.stringify({ events }),
  }).catch(() => {});
}

tail.stdout.on("data", (chunk) => {
  for (const line of chunk.toString().split("\n")) {
    if (line.trim()) try { buffer.push(JSON.parse(line)); } catch {}
  }
});

setInterval(flush, 5000); // flush every 5s

Run it under a process manager (systemd, pm2) so it restarts on reboot. It’s deliberately forgiving — malformed lines are skipped, and a failed POST is dropped rather than crashing the tailer. For a production system where losing a batch matters, prefer Vector or Fluent Bit.

Whichever you choose, Kitbase does the same thing on receipt: it classifies each event, ignores human requests entirely, and stores only bot and crawler hits. Forwarding your whole access log doesn’t mean storing your whole access log.

What you get

Once lines are flowing, the raw log becomes a structured, live view of AI crawler activity:

• Named crawlers — every hit matched to a bot from the list of AI crawlers, by user-agent token rather than exact string, so a version bump like GPTBot/1.4 doesn’t break detection.
• Verified vs. spoofed — each hit’s IP checked against the vendor’s published ranges, so a scraper sending GPTBot from a random VPS is flagged, not counted.
• Top crawled paths — which pages the models read most. This is the same question a server-log crawler audit answers, except continuously and with verification built in.
• Trends and geography — crawl volume over time (a sudden drop can reveal an accidental block), and country derived from the IP before it’s discarded.

Portable across your stack

Log-and-ship is nginx’s version of a pattern every host has. If your infrastructure is mixed, the equivalents land in the same dashboard with the same verified/spoofed classification:

• Cloudflare — a Worker forwards requests with waitUntil.
• Vercel — a zero-code Log Drain streams request logs.

FAQ

Do I need a special nginx module to send data to Kitbase? No. You use nginx’s built-in log_format to write JSON lines, then a separate shipper forwards them. No Lua, njs, or compiled modules required.

Why not just POST from nginx on every request? A blocking outbound call in the request path adds latency and a failure mode to every page load. Logging locally and shipping asynchronously keeps nginx fast and decouples your site’s uptime from the analytics endpoint’s.

My nginx is behind a load balancer — will the IPs be wrong? By default $remote_addr would be the load balancer’s IP. Switch ip_address to the first IP in $http_x_forwarded_for so Kitbase sees the real client IP and can verify crawlers correctly.

How many events can I send per request? Up to 500 events per POST, wrapped as { "events": [ ... ] }. A production shipper like Vector or Fluent Bit handles batching to that limit for you.

Will this store data about my human visitors? No. Kitbase classifies every forwarded line in memory and discards the human ones. Only bot and crawler requests are stored, and their raw IP is dropped after geolocation unless you enable IP logging.

Want your nginx access log turned into a live AI-crawler dashboard — verified, per-page, no JavaScript required? Start your free trial — 7 days, no credit card required — and ship your first batch in minutes.