Google Extended User-Agent Bot
Google announced the Google-Extended user agent for managing control to access Gemini (formerly Bard) and Vertex AI.
Web publishers have the capability to block AI tools content on their web pages with robots.txt.
Publisher Control On Generative AI Tools
AI tools access a website’s content and display it to users, but some publishers may want more control over how their content is used by generative AI tools. That’s why Google announced the Google-Extended user agent. Website publishers now have a solution to prevent generative AI tools from accessing their content.
Google Extended on Robots.txt
A standalone product token that web publishers can use to manage whether their sites help improve Gemini (formerly Bard) and Vertex AI generative APIs, including future generations of models that power those products.
User-agent token: Google-Extended
Full user agent string: Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.
Using Google-Extended
# Block all content on website
User-agent: Google-Extended
Disallow: /
