How to Verify Googlebot (and Catch Bots Spoofing It)

To verify Googlebot, don’t trust the user-agent string — verify the IP. Do a reverse DNS lookup on the request’s source IP and confirm it resolves to googlebot.com, google.com, or googleusercontent.com, then run a forward DNS lookup on that hostname and confirm it points back to the same IP. Or skip DNS entirely and match the IP against Google’s published IP-range files. Either method turns “the UA says Googlebot” into “this is provably Googlebot” — because the user-agent header is just a string, and anyone can send it.

That gap between claim and proof is not academic. Impersonating Googlebot is one of the most common scraper tactics on the web, precisely because so many sites treat Googlebot as trusted. Here’s why spoofing works, both of Google’s official verification methods step by step, and how to classify every hit at scale.

Why scrapers impersonate Googlebot

Googlebot is the most-allowlisted bot on the internet. To protect their SEO, site owners routinely configure their infrastructure to give Googlebot a free pass:

• Bot challenges and rate limits get waived for Googlebot, so it can crawl freely without hitting a CAPTCHA or a 429.
• Paywalls and registration gates are sometimes relaxed for Googlebot so pages can be indexed.
• WAF and firewall rules allowlist Googlebot’s traffic to avoid accidentally blocking the crawler that drives organic traffic.

Every one of those accommodations is an incentive to pretend to be Googlebot. A scraper that sends Googlebot in its user-agent hopes to inherit all that trust — bypass the rate limit, slip past the challenge, reach the gated content. That’s why “is this really Googlebot?” is a security question, not just an SEO curiosity, and why Google publishes two ways to answer it definitively.

Method 1: reverse DNS with forward-confirm

This is Google’s recommended manual check, and it needs no downloaded lists. The logic: only Google controls DNS for its crawler hostnames, so a name that resolves and forward-confirms to a Google domain can’t be faked by someone who merely copied the UA.

Step 1 — reverse-lookup the source IP from your log line:

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

Step 2 — check the hostname’s domain. It must end in one of Google’s verified crawler domains:

• googlebot.com
• google.com
• googleusercontent.com

If the reverse lookup returns anything else — an AWS hostname, a residential ISP, NXDOMAIN — stop: the request is not Googlebot.

Step 3 — forward-confirm. Look up the hostname you just got and confirm it resolves back to the original IP:

host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

If the forward lookup returns the same IP you started with, the request is verified Googlebot. If it doesn’t match — or the hostname doesn’t resolve — it’s spoofed. The forward-confirm step is what makes this airtight: reverse DNS alone can be poisoned, but an attacker can’t make a googlebot.com hostname forward-resolve to their own IP.

flowchart TD
A["Request claims Googlebot<br/>take its source IP"] --> B["Reverse DNS: host IP"]
B --> C{"Hostname ends in<br/>googlebot.com / google.com /<br/>googleusercontent.com?"}
C -->|"No"| S["Spoofed"]
C -->|"Yes"| D["Forward DNS: host that hostname"]
D --> E{"Resolves back to<br/>the original IP?"}
E -->|"No"| S
E -->|"Yes"| V["Verified Googlebot"]

Method 2: match against Google’s published IP ranges

DNS lookups add latency and can be rate-limited, so for high-volume checks Google publishes its crawler IP ranges as machine-readable JSON. Match the source IP against the CIDR blocks in the right file and you get the same verdict without any DNS round-trip:

For Google services beyond crawling, the broader corporate range list lives at gstatic.com/ipranges/goog.json. To verify Googlebot specifically, common-crawlers.json is the file you want:

curl -s https://developers.google.com/static/crawling/ipranges/common-crawlers.json

The one operational cost of this method: the files rotate. Google updates the ranges, so a copy you downloaded last quarter will slowly drift out of date and start producing false “spoofed” verdicts on genuine new Googlebot IPs. Refetch on a schedule, or use a tool that does.

A note on which file to check

A subtlety that trips people up: a request wearing the Googlebot UA should verify against common-crawlers.json, not against goog.json. The broad goog.json list includes Google Cloud customer IPs — meaning a scraper running on a Google Cloud VM would “pass” a naive goog.json check while failing the correct crawler-range check. Always match the bot’s token to the narrowest applicable file. The same discipline applies to every vendor; it’s the same reason OpenAI segregates its four bots’ IP files.

Common verification mistakes

• Trusting the user-agent alone. The whole point: Googlebot in the UA proves nothing. This is the number-one mistake, and it’s what spoofers rely on.
• Reverse DNS without forward-confirm. A reverse lookup that says googlebot.com isn’t enough on its own; skip the forward-confirm and you can be fooled by DNS tricks. Always do both directions.
• Checking the wrong IP list. Verifying against goog.json (which includes Google Cloud) instead of common-crawlers.json lets anything hosted on Google Cloud masquerade as Googlebot.
• Using a stale IP file. Google’s ranges change; an un-refreshed list flags real Googlebot as fake. Automate the refetch.
• Forgetting this generalizes. Googlebot is the most-spoofed bot, but the exact same claim-vs-proof problem applies to GPTBot, ClaudeBot, PerplexityBot, and every crawler in the complete AI crawler list. Verify them all the same way.

Automating verified-vs-spoofed at scale

Doing this once, for one IP, is a two-command exercise. Doing it for every hit on a busy site — reverse-and-forward DNS or IP matching, across every crawler vendor, with each vendor’s ranges kept current — is real infrastructure. And you can only do it at all if the traffic reaches something that can see it: Googlebot and other crawlers don’t run JavaScript, so tag-based analytics never records them. This is server-side territory, best examined through your server logs or a tool that ingests them.

Kitbase’s bot and crawler detection runs exactly this pipeline. Forward your server or edge requests, and each one claiming to be a crawler is checked against the vendor’s current published ranges (and cryptographic Web Bot Auth signatures where present) and stored with a verified or spoofed verdict — Googlebot, GPTBot, and every other bot alike. A rising share of spoofed “Googlebot” is a signal you’d otherwise miss: someone is scraping you while wearing Google’s name. The full request-and-response contract is in the API reference. Human visitors are classified in memory and discarded; only bots are stored.

Once verification is automatic, the same dataset answers adjacent questions — is Googlebot’s crawl rate steady or silently dropping (a sign of an accidental block that will erode rankings)? Which of your pages does it actually read? That’s the log-file discipline of server-log analysis in the AI era, with the verified/spoofed verdict already attached.

FAQ

What’s the fastest way to verify Googlebot? Reverse-DNS the source IP with host <IP>, confirm the hostname ends in googlebot.com, google.com, or googleusercontent.com, then forward-confirm that the hostname resolves back to that same IP. Both directions must pass.

Can I verify Googlebot without DNS lookups? Yes. Match the source IP against Google’s published common-crawlers.json range file. It avoids DNS latency but you must refetch the file periodically, since Google rotates the ranges.

Why do scrapers pretend to be Googlebot? Because sites allowlist Googlebot — waiving rate limits, bot challenges, and sometimes paywalls to protect SEO. Impersonating it is an attempt to inherit that trust and bypass those protections.

Is reverse DNS enough by itself? No. Reverse DNS alone can be manipulated. The forward-confirm step — verifying the hostname resolves back to the original IP — is what makes verification tamper-proof.

Does checking goog.json verify Googlebot? Not safely. goog.json includes Google Cloud IPs, so anything hosted on Google Cloud would pass. Verify Googlebot against the narrower common-crawlers.json instead.

Want every “Googlebot” on your site labeled verified or spoofed automatically — plus GPTBot, ClaudeBot, and the rest? Start your free trial — 7 days, no credit card required — and catch the scrapers wearing Google’s name.