Google announced the Google-Extended user agent for managing control to access Bard and Vertex AI.
Web publishers have the capability to block AI tools content on their web pages with robots.txt.
AI tools access a website’s content and display it to users, but some publishers may want more control over how their content is used by generative AI tools. That’s why Google announced the Google-Extended user agent. Website publishers now have a solution to prevent generative AI tools from accessing their content.
A standalone product token that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products.
User-agent token: Google-Extended
Full user agent string: Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.
# Block all content on website User-agent: Google-Extended Disallow: /