---
title: "Run Your Own AI Crawler Audit: A 30-Minute Server-Log Walkthrough | Kitbase Blog"
description: "A hands-on AI crawler audit: pull 30 days of server logs, count each bot, find top paths, cross-check IP ranges to catch spoofers, and answer the questions that matter."
canonical: https://kitbase.dev/blog/ai-crawler-audit-server-logs/
---

**An AI crawler audit means checking your server logs to answer four things:** which AI bots visit your site, how often, what pages they read, and whether the traffic is real or fake. 
You can do a basic version yourself in about 30 minutes, This guide walks through the manual process.
The same thing [server-side crawler detection](https://docs.kitbase.dev/crawler-detection) automates for you. Doing it by hand once helps you understand what the automated tool is actually measuring.

Everything below works against a standard nginx or Apache combined access log. The point is not to produce a number to brag about, it's to build a factual baseline of AI crawler activity on *your* site so you can spot what's missing, what's blocked, and what's pretending to be something it isn't.

## What you'll need

- **Shell access** to a server (or CDN log export) where the raw access logs live.
- **~30 days of logs.** Logs rotate, so you'll usually be reading `access.log` plus gzipped `access.log.1`, `access.log.2.gz`, and so on. Use `zgrep` to search compressed files transparently.
- **A combined log format** where field 1 is the client IP, field 7 is the request path, and the user agent is the last quoted field. Nearly every default nginx/Apache config uses this.

If you only have JavaScript analytics (Google Analytics, Plausible, and the like), you have nothing to audit yet as those tools never see crawlers, because [AI bots don't run JavaScript](/blog/why-analytics-cant-see-ai-crawlers/). This audit lives entirely in server-side data.

## Step 1: Pull 30 days of logs

First, confirm what you're working with and how far back it goes:

```bash
ls -lh /var/log/nginx/access.log*
```

If your logs rotate weekly, four or five files gets you roughly a month. From here on, use `zgrep` instead of `grep` so compressed archives are included automatically:

```bash
zgrep -ic "bot" /var/log/nginx/access.log*   # rough sanity check: are bots even in here?
```

## Step 2: Count hits per bot

Match on the robots.txt token, not the full user-agent string. 
Note: versions change (GPTBot is on `GPTBot/1.4` at time of writing) but the token is stable. This loop prints a clean per-bot tally:

```bash
for bot in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot PerplexityBot Perplexity-User Googlebot Amazonbot Bytespider CCBot; do
  count=$(zgrep -ic "$bot" /var/log/nginx/access.log*)
  printf "%-18s %s\n" "$bot" "$count"
done
```

One token needs a caveat: `Google-Extended` is a **robots.txt control token, not a separate crawler**. Google fetches with the normal `Googlebot` user agent and uses `Google-Extended` only to decide whether your content trains Gemini so it never appears as its own line in your logs. Control it in robots.txt, but count `Googlebot`.

This tally alone answers question one and two: who visits, and how much. Resist the urge to interpret it yet, a raw count includes spoofers, which Step 4 filters out.

## Step 3: Top paths per bot

For each bot that shows meaningful volume, pull its most-requested paths. Field 7 is the path in a combined log:

```bash
zgrep -i "gptbot" /var/log/nginx/access.log* | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
```

Swap `gptbot` for each bot and read the lists side by side. This is where the audit gets useful, because the shape of the list tells you something:

- **Text-dense pages on top (docs, blog, comparisons)** is the healthy pattern, those are the pages models learn about you from. If they're absent, dig into [which pages AI bots crawl](/blog/which-pages-ai-bots-crawl/) and why yours might be missing.
- **`/robots.txt` and `/llms.txt` fetches** are worth noting, they confirm the bot is checking your rules, and an `/llms.txt` hit is rare direct evidence a bot looked at that file.
- **Different top lists per bot** is normal and informative, Perplexity and OpenAI don't prioritize the same pages.

## Step 4: Cross-check IP ranges to catch spoofers

This is the step that separates an audit from a guess. Anything can send a `GPTBot` user agent, scrapers routinely impersonate well-known crawlers to slip past bot rules. A user-agent match is a *claim*, not a verification. To verify, take the IPs that claimed to be a given bot and check them against that vendor's published egress ranges.

Pull the unique IPs behind a bot's claimed traffic:

```bash
zgrep -i "gptbot" /var/log/nginx/access.log* | awk '{print $1}' | sort -u
```

Then compare each IP against the operator's published list. Most major operators now publish machine-readable IP ranges (in CIDR notation) or support forward-confirmed reverse DNS:

| Bot | robots.txt token | How to verify the IP |
|---|---|---|
| GPTBot | `GPTBot` | [openai.com/gptbot.json](https://openai.com/gptbot.json) |
| OAI-SearchBot | `OAI-SearchBot` | [openai.com/searchbot.json](https://openai.com/searchbot.json) |
| ChatGPT-User | `ChatGPT-User` | [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json) |
| PerplexityBot | `PerplexityBot` | [perplexity.com/perplexitybot.json](https://www.perplexity.com/perplexitybot.json) |
| Perplexity-User | `Perplexity-User` | [perplexity.com/perplexity-user.json](https://www.perplexity.com/perplexity-user.json) |
| ClaudeBot | `ClaudeBot` | Anthropic's [crawler support page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) publishes its IP list |
| Googlebot | `Googlebot` | Reverse + forward DNS to `googlebot.com`, or the [published ranges](https://developers.google.com/search/apis/ipranges/googlebot.json) ([verify guide](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)) |

An IP claiming to be GPTBot from outside `openai.com/gptbot.json` is spoofed, full stop. Googlebot is the exception to the simple IP-list approach, Google recommends [forward-confirmed reverse DNS](/blog/verify-googlebot-spoofing/) (reverse-lookup the IP, confirm the hostname ends in `googlebot.com`, then forward-lookup that hostname back to the same IP) rather than a static list, because its ranges shift.

**Verifying a request that claims to be an AI crawler**

```mermaid
flowchart TD
  A["Log line<br/>UA contains a bot token"] --> B{"Source IP inside the<br/>vendor's published range?"}
  B -->|"Yes"| V["Verified crawler"]
  B -->|"No"| S["Spoofed, a scraper<br/>wearing the bot's name"]
```

Doing this by hand for one bot is instructive. Doing it for ten bots, across shifting ranges, every day, is exactly the toil worth automating, more on that below.

## The audit worksheet

With the raw numbers in hand, work through these questions. The answers, not the commands, are the deliverable:

1. **Which AI bots visit at all?** Any expected bot with a count of zero is either not finding your site or being blocked. GPTBot missing entirely is a red flag on a public marketing site.
2. **Are your best pages the most-crawled ones?** Compare the top-paths lists against your highest-value content. Gaps are reachability problems.
3. **What's the verified-to-spoofed ratio per bot?** A high share of spoofed `GPTBot` hits is scraper activity, a security signal, not AI interest.
4. **Is any bot's volume near zero when it shouldn't be?** A crawler that used to visit and stopped often means an accidental block (a WAF rule, a CDN toggle, a `robots.txt` edit).
5. **Which pages get `/robots.txt` and `/llms.txt` fetches?** Confirms which control files bots actually read.
6. **Are humans polluting the numbers?** If you didn't filter, some "bot" hits are just browsers with odd user agents, another reason server-side classification beats string matching.

Answer those six and you have a real baseline. Notably, none of this tells you whether the crawling *translates into citations*, that's a separate dataset (whether AI answers actually mention you) and a genuinely [open question worth testing on your own data](/blog/does-ai-crawling-predict-citations/). Crawl data is the first stage of the [GEO funnel](/blog/geo-funnel/); the audit measures the input, not the outcome.

## From one-time audit to continuous monitoring

A log audit is just a snapshot so it only shows you that one moment, not what happens after. A future blocked crawler, a new scraper next month, a traffic spike after a model update therefore, A one-time check misses all of it. That's why [ongoing log-file analysis in the AI era](/blog/log-file-analysis-ai-era/) matters more than a single check.

This is where doing it manually falls apart. Keeping track of every bot's IP ranges, verifying each request, and doing this daily is a lot of real engineering work.. [Kitbase's bot & crawler detection](https://docs.kitbase.dev/crawler-detection) does this for you automatically. You forward your server traffic to it, and it sorts every request, filtering out humans, and logging each crawler by bot name, page visited, and whether it's verified or fake. It works with [Next.js](https://docs.kitbase.dev/crawler-detection/nextjs), [WordPress](https://docs.kitbase.dev/crawler-detection/wordpress), [nginx](https://docs.kitbase.dev/crawler-detection/nginx), [Vercel](https://docs.kitbase.dev/crawler-detection/vercel), and more.

Run the manual audit first anyway. It's the fastest way to understand what your logs already contain, and once you've seen it by hand, the automated version stops being a black box.

## FAQ

**How do I audit AI crawlers on my site?**
Pull ~30 days of server access logs, count hits per bot by matching robots.txt tokens (GPTBot, ClaudeBot, PerplexityBot, etc.), list the top paths per bot, and cross-check the source IPs against each vendor's published ranges to separate real crawlers from spoofers. JavaScript analytics can't do this, crawlers don't run JS, so the audit has to happen in server-side logs.

**How can I tell if GPTBot traffic is real or fake?**
Take the client IP from the log line and check it against [openai.com/gptbot.json](https://openai.com/gptbot.json). Any request with a `GPTBot` user agent from an IP outside those CIDR ranges is spoofed. See the [GPTBot guide](/blog/gptbot-explained/) for the full breakdown of OpenAI's crawlers.

**What log format do I need for an AI crawler audit?**
A standard combined access log (the nginx/Apache default) works: client IP in field 1, request path in field 7, and the user-agent string as the last quoted field. Use `zgrep` to include rotated, gzipped logs so you cover the full 30 days.

**How often should I re-run the audit?**
A manual audit is worth repeating monthly at minimum, but the failures that cost the most, for example: an accidental block or a spoofing campaign may happen between audits. Continuous server-side monitoring catches them as they occur rather than weeks later.

**Does a crawler audit tell me my AI visibility?**
No. An audit measures whether engines *read* your pages, which is necessary but not sufficient for them to *cite* you. Measuring citations means querying the engines directly, which is what [AI Visibility](https://docs.kitbase.dev/ai-visibility) does, a separate dataset from crawl logs.

---

*Want the audit worksheet kept current automatically, with every crawler verified and attributed? [Start your free trial](https://app.kitbase.dev/signup/), 7 days, no credit card required, and skip the grep.*