One Setting For AI Training? What The Shared Opt-Out Covers

🔍 Read the full analysis: One Setting For AI Training? What The Shared Opt-Out Covers on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a shared instruction in a website’s robots.txt file asking them not to use its content for generative AI training. The proposed signal could reduce the need for publishers to set separate training preferences for each company, but it is a voluntary request, not a technical barrier or an adopted internet standard.

The shared instruction is a robots.txt group using User-Agent: __usage_ai-training followed by Disallow: /. It comes from an Internet-Draft titled “Machine-Readable Training Preference Signal,” published October 30 by Kevin Kelley, an Anthropic standards lead. The draft proposes a common way for websites to express a preference against training use; it does not itself compel companies to comply.

Companies have described different implementation schedules. Anthropic says it has started implementing support, though it says adding support across its services will take time. OpenAI aims to implement the signal “in the coming months.” Google says it aims to support it in 2027, with its crawler documentation specifying early 2027. Microsoft says it will honor the training preference from February 1, 2027, as it crawls websites.

Google says that when a site uses the shared signal but has no separate Google-Extended rule, it plans to treat the shared preference as if the publisher had disallowed Google-Extended. Microsoft is also introducing a separate page-level preference for AI Grounding—use of web content to inform answers in Bing, Copilot and Microsoft 365. Microsoft says both preferences take effect from February 1, 2027. It says NoArchive and NoCache will stop counting as preferences for training or grounding after March 1, 2027.

One Rule, Several Company Timelines

For website owners, a shared signal could make it easier to communicate a training preference across participating AI providers. Instead of relying only on company-specific crawler names and rules, publishers could add one common robots.txt group. That may reduce the chance that a publisher’s instruction is missed because it did not know which company-specific control to use.

But the practical effect depends on whether each company implements and follows the signal. It is a preference, not enforcement: robots.txt communicates instructions to crawlers, but the draft does not create a technical lock preventing access or use. Publishers may need to keep their existing company-specific rules in place during the rollout, particularly where a provider has not yet switched over.

The distinction between training and other uses also matters. The shared rule addresses training generative AI models; it does not automatically block crawling, remove material from models already trained, or prevent an AI service from drawing on live web pages to answer a user. Microsoft’s separate grounding preference illustrates that these uses may require distinct controls. The announcement also creates no payment right. Microsoft says neither choice guarantees inclusion, traffic or payment.

Products Worth Considering

Samsung 27 Odyssey OLED G5 (G50SF) QHD QD-OLED Gaming Monitor

AmazonView Latest Price

LG UltraGear 32GX850A-B 32-Inch 4K UHD Glossy OLED Gaming Monitor

AmazonView Latest Price

Samsung 27-Inch Odyssey QD-OLED G8 4K 240Hz Gaming Monitor

AmazonView Latest Price

As an affiliate, we earn on qualifying purchases.

How the Shared Signal Emerged

The proposal is an individual Internet-Draft associated with the IETF AI Preferences working group, which works on technical approaches to expressing preferences about AI use of online content. An Internet-Draft is a document under discussion, not by itself an official standard. The published draft says it has no formal standing in the IETF standards process.

The timeline is still significant: the working group met on October 21, before the announcements. At that meeting, one of its chairs said, “We haven’t declared consensus on anything yet.” The October 30 company statements therefore show support for a common signal, but they do not establish that the standards group has approved it.

Existing controls remain relevant during the changeover. OpenAI says its current crawler controls will continue to apply. Google says Google-Extended does not affect Google Search, while Microsoft says its preferences do not change eligibility for Bing search results. These distinctions concern website publishing controls; they are not settings for a person’s conversations with an AI assistant.

““has already started implementing support for this draft””

— Anthropic, in its announcement of the draft

Adoption and Scope Remain Open

Meta’s position is not an implementation commitment. Its communications chief expressed support for publishers’ ability to state preferences, but did not confirm that Meta will follow this signal. The source reporting also found no statement from Apple, Amazon, xAI or Perplexity about whether they plan to honor it. The list of participating companies could change, but broader adoption is not confirmed.

The signal’s status in the standards process is another open issue. The draft is not an official IETF standard, and the working group had not declared consensus at its October 21 meeting. The companies’ announcements establish their stated intentions and schedules; they do not show how consistently the signal will be interpreted across all services or crawlers.

It is also not yet clear how every provider will distinguish model training from other uses of web content in practice. The shared signal is described as covering training, while Microsoft’s separate grounding control addresses content used to inform AI answers. Live-answer use, previously trained models and payment are not covered by the shared training opt-out as described in the announcements.

Publishers Face a Phased Rollout

Publishers can add the shared training group if they want to express an opt-out, but they should check each provider’s current documentation and keep existing company-specific rules until that provider’s support is in place. The dates announced so far are staggered: Anthropic says implementation has begun, OpenAI is aiming for the coming months, and Google and Microsoft point to 2027.

Microsoft’s stated schedule gives sites using NoArchive or NoCache a specific date to review: those tags stop counting as training or grounding preferences after March 1, 2027, under its plan. Sites that want to limit AI grounding will need to consider Microsoft’s separate page-level preference rather than assuming the shared training signal covers answers based on web content.

The next developments to watch are the companies’ actual crawler and documentation updates, any clearer commitment from Meta and other providers, and the IETF working group’s handling of the draft. Until those steps occur, the shared instruction is best understood as a common preference with uneven adoption, not a universal opt-out.

Key Questions

What does the shared AI training opt-out do?

It tells participating companies, through a robots.txt instruction, that a publisher prefers its content not be used to train generative AI models. It is a voluntary signal, not a technical block.

Does it stop AI systems from crawling a website or using it in answers?

No. The draft concerns model training. It does not itself block crawling or cover AI answers that use live web content. Microsoft has announced a separate preference for AI Grounding.

Which companies have committed to honor the signal?

OpenAI, Google, Microsoft and Anthropic announced support, with different implementation schedules. Meta expressed support for the idea but did not confirm that it will honor the signal.

Is the shared signal an official internet standard?

No. It is an individual Internet-Draft, which the source says has no formal standing in the IETF standards process. A working-group chair said on October 21 that the group had not declared consensus.

Should publishers remove their existing crawler rules?

The announcements do not establish that all providers have implemented the shared signal yet. Publishers should check each company’s current guidance and retain relevant existing rules during the phased rollout. Microsoft says NoArchive and NoCache stop counting for training or grounding preferences after March 1, 2027.

Source: ThorstenMeyerAI.com