Google-Extended vs Googlebot: Blocking AI Training Without Killing Your SEO

Google-Extended is a robots.txt control token, not a crawler. It never fetches a page. It doesn’t have its own user-agent that shows up in your logs. All it does is tell Google whether the content Googlebot already crawled may be used to train Gemini models and ground Google’s generative AI. Googlebot, by contrast, is the crawler that builds Google Search itself — and blocking that is one of the most destructive things you can do to a website.

Confusing the two is a genuine footgun: people who mean to opt out of AI training sometimes reach for the wrong Disallow line and quietly deindex themselves from Google. This guide draws the line precisely — what each one is, what Google-Extended actually controls in 2026, why Googlebot is untouchable, and the AI Overviews subtlety almost everyone gets wrong.

Google-Extended vs Googlebot at a glance

The one-sentence version: Googlebot controls whether Google can find you; Google-Extended controls whether Google can train Gemini on you. They are not substitutes, and they are not on the same layer.

What Google-Extended actually controls in 2026

Per Google’s crawler documentation, Google-Extended lets publishers “manage whether content Google crawls from their sites may be used for training future generations of Gemini models” and for grounding those models. Two clarifications from Google that matter:

• It does not impact a site’s inclusion in Google Search, and it is not used as a ranking signal.
• It has no separate user-agent. Googlebot still does the crawling; Google-Extended is a downstream usage gate applied to what Googlebot already fetched.

So a site that wants to keep every ounce of SEO while opting out of Gemini training uses this, and only this:

User-agent: Google-Extended
Disallow: /

Googlebot keeps crawling and indexing normally. Google just won’t feed that content into Gemini’s training. Note the separate, actively-fetching cousin: Google-CloudVertexBot is a real crawler (with its own token and IP ranges) that fetches pages on demand when a Google Cloud customer builds a grounded Vertex AI Search app pointed at your site — distinct from the passive Google-Extended token.

Why blocking Googlebot is catastrophic

It should be obvious and yet it happens: someone copies an aggressive “block the AI bots” robots.txt from a forum, and it contains User-agent: Googlebot / Disallow: /. The result is not “less AI.” The result is deindexing from Google Search. Your pages fall out of the index, organic traffic collapses, and because Googlebot feeds everything downstream, you also vanish from the surfaces built on the index:

flowchart LR
G["Googlebot<br/>crawls your pages"] --> IDX[("Google Search<br/>index")]
IDX --> R["Search results"]
IDX --> AIO["AI Overviews<br/>and AI Mode"]
G --> GE{"Google-Extended<br/>allowed?"}
GE -->|"Yes"| TR["Gemini / Vertex AI<br/>training + grounding"]
GE -->|"Disallow"| X["Not used for<br/>Gemini training"]

Blocking Googlebot cuts the line at the very top of that diagram — you lose Search results and AI Overviews and any chance of being trained on, all at once. Blocking Google-Extended snips only the bottom-right branch. This is the whole reason the two tokens exist separately: so you can make a training decision without touching your search presence. If SEO matters to you at all, Googlebot’s line in robots.txt should read Allow: / (or simply not be there).

The AI Overviews subtlety everyone gets wrong

Here’s the part that trips up even careful teams: blocking Google-Extended does not remove you from AI Overviews or AI Mode.

AI Overviews are a Search feature. They’re generated from Google’s regular search index — the one Googlebot builds — not from the Gemini training set that Google-Extended governs. So if your goal is “keep my content out of AI Overviews,” Google-Extended does nothing for you. You’d still appear, because you’re still in the index.

The lever that does reach AI Overviews is snippet control — the nosnippet and max-snippet robots meta tags. Those limit how much of your content Google can show, and that limit applies to AI Overviews and AI Mode as well as to regular result snippets. The catch: they also shrink your normal search snippets, which usually hurts click-through. So the honest trade-off looks like this:

Google has continued to develop more granular controls for its AI search features through 2026, so this space is moving; but the fundamental architecture — AI Overviews ride on the Googlebot index, Google-Extended gates only training — is the stable fact to plan around. For the visibility side of AI Overviews, see Google AI Overviews and SEO.

A decision table for the common cases

For the parallel decision on training crawlers from other vendors — GPTBot, ClaudeBot, and friends — see should you block AI crawlers and the line-by-line robots.txt for AI crawlers guide.

Verify what Googlebot is actually doing

Because Google-Extended never appears in logs and Googlebot is the entire ballgame, the thing worth monitoring is Googlebot itself — specifically, that it’s still crawling and that the traffic wearing its name is genuine. Scrapers impersonate Googlebot constantly, and Google publishes IP ranges precisely so you can tell real from fake. The full workflow is in how to verify Googlebot and catch spoofers.

The practical failure mode this catches: a WAF or CDN bot rule starts blocking Googlebot, your crawl rate quietly drops, and rankings erode weeks before anyone connects the dots. JavaScript analytics can’t see Googlebot (it doesn’t run your scripts), so server-side crawler detection is what surfaces the trend — and labels each Googlebot hit verified or spoofed against Google’s published ranges. The complementary question — whether Gemini’s answers actually mention you — is measured separately by AI Visibility.

FAQ

Is Google-Extended a crawler? No. It’s a robots.txt control token with no user-agent and no fetching behavior. It only signals whether content Googlebot already crawled may be used to train Gemini and Vertex AI models.

Will blocking Google-Extended hurt my SEO? No. Google states Google-Extended does not affect inclusion in Google Search and is not a ranking signal. Blocking it opts you out of Gemini training only.

Does blocking Google-Extended remove me from AI Overviews? No. AI Overviews and AI Mode are generated from Google’s search index, which Googlebot builds — not from the training set Google-Extended governs. To limit AI Overview content you’d use nosnippet or max-snippet, which also shrink your regular search snippets.

What happens if I block Googlebot? You deindex from Google Search — losing organic rankings, result listings, and AI Overview appearances all at once. This is almost never what anyone actually intends; the AI-training opt-out is Google-Extended, not Googlebot.

What’s the difference between Google-Extended and Google-CloudVertexBot? Google-Extended is a passive usage token. Google-CloudVertexBot is an active crawler that fetches pages on demand when a Google Cloud customer builds a grounded Vertex AI Search app pointed at your site.

Want to confirm Googlebot is still crawling you — and that the traffic is real, not spoofed? Start your free trial — 7 days, no credit card required — and watch your crawl trends before a silent block costs you rankings.