Google Backs a Shared AI Training Control for Every Website

For three years, telling AI companies not to train on your website has meant maintaining a growing list of crawler names in robots.txt, one vendor at a time. On October 30, 2026, Google said it plans to replace that guesswork with a single shared AI training control, a robots.txt signal that participating AI providers will all read the same way. Anthropic, OpenAI and Microsoft also signaled support that day, making this the first time the largest model builders have lined up behind one opt-out mechanism for web publishers.

The change will not take effect overnight. Google says it aims to support the new AI training control in 2027, and the underlying specification is still an Internet-Draft. But the syntax is already public, and site owners who understand the AI training control now will be ready the day enforcement begins.

What Google Announced About the AI Training Control

The announcement came from John Mueller, Google Search Advocate, in a post on the Google Search Central blog. Google wrote that the shared control “will help websites express their AI training preferences across multiple AI providers in a single step,” and that it is based on work from the AIPREF working group at the Internet Engineering Task Force (IETF).

The key detail for anyone already managing AI crawlers is how Google will treat existing settings. When a site uses the shared policy token to opt out of training and does not explicitly mention Google-Extended, “Google plans to interpret that as if the publisher had disallowed Google-Extended.” Google-Extended is the training control the company has offered since 2023, and it is not going away. Google says company-specific controls “will continue to function alongside this new joint signal.”

Google also tied the move to regulation. The post says the proposal is consistent with the EU Code of Practice for General Purpose AI model providers under the EU AI Act, which recognizes robots.txt as a primary way for publishers to state machine-readable preferences. In other words, the AI training control is partly a compliance story, not only a goodwill gesture toward publishers who have spent years asking for one switch instead of twenty.

How the New Policy Token Works

The mechanism is defined in an Internet-Draft titled “Machine-Readable Training Preference Signal,” published on October 30 and authored by Kevin Kelley of Anthropic. It introduces a single policy token, __usage_ai-training, that sits in the User-Agent line of an ordinary robots.txt group. There are no new directives and no new grammar. A site that wants to opt out of training everywhere except its press kit would write this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That example comes straight from the draft. Disallow expresses a preference against AI training for matching paths, and Allow expresses a preference in favor. Because it reuses the existing robots.txt path rules and longest-match precedence, an AI training control can be set per directory rather than site wide. That matters for businesses that are happy to have product pages summarized by assistants but want premium guides, course material or member content left out of training sets.

The draft is also careful about compatibility. Parsers that do not recognize the token simply ignore the group, the same way they ignore rules for any bot they have never heard of. Adding an AI training control to an existing file should not change how search engines or other crawlers read the rest of it.

Wildcard Rules Do Not Count

One detail will catch people out. The draft says wildcard groups, meaning User-Agent: *, “MUST NOT be interpreted as matching the policy token.” If your robots.txt file has no __usage_ai-training group, no training preference has been expressed at all, whatever your catch-all rules say. A blanket Disallow aimed at every bot is not an AI training control under this proposal, and it would block search engines too.

Specific Rules Still Win

The draft says specific expressions should take precedence over general ones, and it lets AI providers treat their own tokens, such as Google-Extended, as more specific than the shared one. Google confirmed it will work that way: sites that explicitly allow Google-Extended remain eligible to appear in Gemini Apps responses and receive clicks via grounding. So you can opt out broadly with the shared AI training control and still carve out an exception for one provider you want to be cited by.

Who Is Backing the Shared AI Training Control

The coordination is the real news. Anthropic wrote the draft and, according to same-day reporting compiled by FourWeekMBA, has already started implementing support. That reporting also says OpenAI committed to honoring a common robots.txt signal for generative AI training in the coming months, and that Microsoft plans to build on the shared preference while adding a separate setting for AI grounding. Google is targeting 2027.

Those timelines are uneven, and the draft itself expires on May 3, 2027 unless it is revised. The IETF Datatracker lists it as an individual submission with no formal standing in the standards process, and Google describes its own commitment as a “draft proposal.” Nobody should assume the syntax is frozen. What has changed is that the companies behind most commercial model training have publicly agreed on the shape of a single AI training control, which was not true a week ago.

What the AI Training Control Does Not Do

This is where many site owners will misread the news. The draft states plainly that the policy token “is not a directive for crawling behavior.” It adds that content usage preferences and traffic management signals are “independent and orthogonal.” Setting an AI training control tells a provider how you want your content used. It does not, by itself, stop that provider's bots from fetching your pages.

Providers may choose to skip pages they cannot use for training, and the draft permits that, but nothing requires it. If AI crawler traffic is eating into your server resources today, the shared token will not fix it. You still need user agent groups for specific crawlers, or blocking and rate limiting at the server or CDN level.

That distinction matters on hosting plans with resource caps. Aggressive crawlers burn CPU time, entry processes and bandwidth, the same shared hosting plan limits that rarely appear on a pricing page. A site can publish a perfect AI training control and still hit its resource ceiling during a crawl spike, because the bot doing the crawling may be gathering pages for search grounding, evaluation or other purposes the token does not cover.

Will This Affect Google Search Rankings?

Google has long said that Google-Extended does not affect a site's inclusion or ranking in Google Search, because Googlebot crawling for Search uses a separate product token. The new AI training control maps onto Google-Extended, so opting out of training should leave organic visibility untouched. The draft reinforces this by saying crawler rules must be read exactly as they were before.

The trade-off sits elsewhere. Google noted that sites allowing Google-Extended are eligible for grounding citations in Gemini Apps. If AI assistants are becoming a referral source in your niche, a blanket opt-out could cost you visibility there. Search itself is volatile enough already: Google's September spam update only finished rolling out on October 8, and swings like the Google algorithm update earlier this year show how much traffic can move without any robots.txt change at all. Keep those two questions separate when you read your analytics.

Why Publishers Pushed for a Single Signal

The patchwork the new token replaces is real. In July 2024, Cloudflare reported that AI bots accessed around 39% of the top one million Internet properties on its network, yet only 2.98% of those properties took measures to block or challenge the requests. Even among the top 10,000 sites, fewer than 9% were blocking.

Part of that gap is effort. Each AI company published its own crawler names, and some split training, search and user-triggered fetching across several bots. Keeping robots.txt current meant tracking announcements from a dozen vendors. Small businesses and bloggers rarely had the time, which is exactly the audience a shared AI training control is meant to serve. The draft also notes that robots.txt files are size constrained and that some deployed files already contain “heavily duplicated groups,” a problem one token is designed to avoid.

Google's post frames it the same way, describing the proposal as a response to feedback from “publishers, civil society groups, regulators and other stakeholders that have called for industry-wide controls on AI training.”

How to Prepare for the AI Training Control

Start by deciding what you actually want. Opting out of training, allowing it, and allowing it only for some sections are all reasonable positions. The right answer depends on whether AI assistants send you customers or simply summarize your work without a click.

Next, audit the robots.txt file you already have. Confirm that Googlebot is not accidentally blocked, note any Google-Extended or GPTBot rules, and remember that an explicit Google-Extended rule will override the shared AI training control for Google. Adding a __usage_ai-training group now is low risk, since parsers that do not recognize it ignore it, but expect the details to change before providers enforce the AI training control in 2027.

Finally, treat crawler load as a separate problem. If your server logs show AI bots hammering your site, rate limit or block them at the server or CDN layer, because a training preference will not slow them down.

What This Means for You

The shared AI training control is the first opt-out that the major model builders have agreed to read the same way, and it turns a dozen vendor-specific rules into one robots.txt group. It is still a draft, Google will not support it until 2027, and it governs how your content is used rather than whether bots crawl it. Set your preference deliberately, keep your Search rules intact, and manage crawler traffic on its own terms. If bot traffic is already straining your current plan, LiteSpeed-powered NVMe web hosting from MonsterMegs gives you the headroom and server-level controls to handle it.