Google-Extended, Googlebot and AI Overviews: what each control actually affects
Per Google, Google-Extended is a control token, not a crawler: it governs whether crawled content is used to train future Gemini models and for grounding in Gemini Apps and Vertex AI, and it does not affect inclusion in Search. Google's AI features documentation says AI Overviews and AI Mode eligibility follows normal Search indexing and snippet eligibility, so the levers for those are Googlebot access and snippet controls, not Google-Extended.
Last verified 28 October 2026 · Reviewed by Adarsh Kant
Which control affects which Google system?
| Control | What Google says it governs | What it does not govern |
|---|---|---|
| Google-Extended (robots.txt token) | Use of crawled content for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI | Inclusion in Google Search; it is not a ranking signal. Google's crawler page does not address AI Overviews |
| Googlebot (robots.txt token) | Crawling for Google Search indexes | A robots.txt block does not reliably keep a URL out of results |
| noindex (meta or X-Robots-Tag) | Keeps the page out of search results | Needs the page to be crawlable so Google can read it |
| nosnippet / max-snippet:0 | No text snippet or video preview; Google says this also covers AI Overviews and AI Mode and stops content being used as direct input to them | A static image thumbnail may still show |
| max-snippet:N | Limits text snippet to N characters, and per Google applies to AI Overviews and AI Mode too | Does not override more specific permissions such as structured data or a licence agreement with Google |
| data-nosnippet attribute | Excludes specific span, div or section content from snippets | Google's robots meta page does not mention AI features for this attribute |
Does Google-Extended have its own user agent?
No. Google says crawling is done with its existing user agents and the token exists only for control. So you will never see Google-Extended in your access logs, and you cannot test it with a user agent header.
What robots.txt do I copy?
User-agent: Google-Extended
Disallow: /
User-agent: Googlebot
Allow: /How do I limit what AI Overviews and AI Mode can show?
Google's AI features page says to use nosnippet, data-nosnippet, max-snippet or noindex to limit what is shown from a page. It also says a page must be indexed and eligible to appear with a snippet to be included, and that no extra markup or AI-specific files are needed.
<!-- no text snippet anywhere, incl. AI Overviews / AI Mode per Google -->
<meta name="robots" content="nosnippet">
<!-- allow at most 160 characters of text -->
<meta name="robots" content="max-snippet:160">
<!-- hide one block only -->
<div data-nosnippet>Members-only price list</div>
# Same rule for a PDF via HTTP header:
X-Robots-Tag: max-snippet:160What should I verify after making changes?
- Fetch /robots.txt live and confirm the Google-Extended group is present and Googlebot is not disallowed.
- Use Search Console's URL Inspection to view the HTML Googlebot receives; Google says to check this when preview controls seem not to work.
- Confirm meta tags are in the rendered HTML, and that data-nosnippet is well-formed with tags properly closed and not added by JavaScript to existing nodes.
- Remember Googlebot must be able to fetch the URL to see noindex or snippet rules; a robots.txt block hides them.
- Allow time: Google says recrawling can take days to months, though you can request a recrawl.
- To confirm a request is Google, use reverse DNS (googlebot.com, google.com or googleusercontent.com) with a forward lookup, or Google's published IP range files.
Honest limits
What we do not know.
- Google's crawler documentation does not state whether Google-Extended affects AI Overviews or AI Mode, so we do not claim it does or does not beyond what the AI features page implies by treating it as separate.
- We have not published any analysis of CiterLabs' own server logs.
- We have no source showing that snippet controls change how often a page is cited in AI Overviews; the sources document eligibility and limits only.
- We did not test the snippets or measure the effect of any of them on live results.
- Google's AI features and crawler pages change; this reflects the versions opened on 2026-10-28.
- We did not open Google's overview page on special-case or user-triggered fetchers beyond a summary, so we do not list every Google fetcher.
Sources
- Google Search Central: Google's common crawlers (opened 28 October 2026)
- Google Search Central: AI features and your website (opened 28 October 2026)
- Google Search Central: Robots meta tag and X-Robots-Tag (opened 28 October 2026)
- Google Search Central: Introduction to robots.txt (opened 28 October 2026)
- Google Search Central: Verifying Googlebot and other crawlers (opened 28 October 2026)
How this page was made: An AI assistant drafted this page from the cited Google documents, and it is awaiting human review.
Questions
- If I disallow Google-Extended, do I drop out of Google Search?
- Per Google, no. It does not affect inclusion in Search and is not a ranking signal.
- Does nosnippet affect AI Overviews?
- Google's robots meta documentation says nosnippet covers AI Overviews and AI Mode and stops the page being used as direct input to them.
Free, no call needed
See who AI namesin your category today.
Tell us your domain and category. We run 10 buyer questions and send a short report with the raw answers attached.
Get a free AI Visibility CheckOr write to [email protected]