Google announced plans for a major new feature, one that will give publishers and website owners far more control over how AI interacts with their content.
AI has wreaked havoc on many publishers and websites, as many people use the technology in place of traditional search engines. Unfortunately for publishers, AI often still uses their content to generate answers, but without necessarily driving traffic to the website providing that content. There have been efforts to improve the situation, such as incorporating source links in the AI-generated answers, but those measures have had only limited success.
In response, Google is now joining Anthropic, Bing, and OpenAI in unveiling additional controls that websites and publishers can use to better manage their content and how AI utilizes it.
The company described how the new controls work in a developer blog post:
Instead of managing separate instructions for every AI developer, website owners will be able to use a shared ‘policy token’ within robots.txt to set broad preferences across participating AI providers.
When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended. Other AI providers can handle it similarly, making it easy for website owners to opt out across all providers, or to pick and choose.
Existing company-specific controls will continue to function alongside this new joint signal. So website owners can easily express a broad choice using the industry signal, while also tailoring their instructions to individual AI developers like Google through controls such as Google-Extended. Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.
The new mechanism brings website owners the benefit of the AI training definition developed at the IETF while multistakeholder discussions continue on the rest of the AIPREF vocabulary. As more industry players adopt this mechanism, it gives website owners early and expanding access to the preference signal.
This draft proposal reflects feedback from publishers, civil society groups, regulators and other stakeholders that have called for industry-wide controls on AI training. Support for this proposal across major AI providers will enable a future standard that is cross-industry, robust, and durable. This proposal is consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act that recognizes robots.txt as a primary method for web publishers to express machine-readable preferences. We’ll continue to work with these stakeholders to refine and standardize this through the IETF working group.
We aim to support this new control along with other AI providers in 2027, and will provide updates on our progress.
Hopefully these new controls will help mitigate the negative impacts that AI has had on the content industry.