🔍 Read the full analysis: What The AI Opt-Out Can And Can’t Do For Your Data on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will support a shared robots.txt signal asking that a website’s content not be used to train generative AI models. The signal is voluntary, will roll out on different schedules and does not block crawling, remove material from existing models, or automatically opt content out of AI-generated answers.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will recognize a shared robots.txt preference asking that website content not be used to train generative AI models. The change could let publishers express that preference with one rule instead of maintaining separate instructions for each company, but it is a voluntary request, not a technical block, and it does not cover every way AI services may use web content.
The proposed rule uses the user-agent name __usage_ai-training with a disallow instruction for the site’s content. It appears in an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” written by Anthropic standards lead Kevin Kelley and published on October 30. The draft builds on work by the Internet Engineering Task Force’s AI Preferences working group, but it is not an adopted internet standard.
The companies have set different timelines. Anthropic says it has already started implementing support, while OpenAI says it aims to adopt the signal “in the coming months.” Microsoft says it will honor the training preference from February 1, 2027, as it crawls sites. Google says it aims to support the control in 2027, and its crawler documentation specifies “early 2027.” These are company statements about planned or developing support, not evidence that every service already applies the rule.
Microsoft is also introducing a separate page-level preference for “AI Grounding,” which concerns web content used to inform answers in Bing, Copilot and Microsoft 365. That setting is distinct from the shared training signal. Microsoft says both preferences take effect February 1, 2027, and that NoArchive and NoCache will stop counting as training or grounding preferences after March 1, 2027. The company says these choices do not change eligibility for Bing search results.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
For publishers, a common rule could reduce the work of listing separate crawler instructions for individual AI companies. A site that wants to signal opposition to training can publish the shared token while retaining existing company-specific rules until providers implement the new one. That may make the preference easier to communicate, but robots.txt does not force a company to comply; the signal depends on each provider choosing to honor it.
The distinction matters because “AI use” covers more than model training. A training opt-out does not, by itself, prevent a company from crawling a page or using current web material to produce an answer. Microsoft’s separate grounding preference makes that split explicit. Google says its shared training control will be treated like a disallow for Google-Extended when a site has no Google-Extended rule, while existing controls continue to matter during the rollout. Publishers must decide separately what they want for training, AI answers and search.
The signal also offers no payment mechanism and no promise about traffic or visibility. Microsoft says neither choice guarantees “inclusion, traffic, or payment.” The announcement is about expressing a preference, not negotiating licensing terms or changing how a publisher is compensated.
Products Worth Considering
FLEXISPOT Standing Desk with Monitor Riser & Cable Management
Dell UltraSharp 27 Thunderbolt Hub Monitor (U2726DER)
Samsung Essential S3 27-Inch Curved FHD Monitor
As an affiliate, we earn on qualifying purchases.
From Separate Crawler Rules to One
Before this proposal, a website seeking to limit AI training could need distinct instructions for different crawlers, including OpenAI’s GPTBot, Google-Extended and Anthropic’s ClaudeBot. Microsoft also used page-level tags such as NoArchive and NoCache for related preferences. A shared signal aims to make the training request readable across participating providers, while the old provider-specific rules remain relevant until each company’s support is in place.
The proposal is being discussed through the IETF’s AI Preferences working group. Its draft has no formal standing in the IETF standards process at this point. At a working-group meeting on October 21, one of the chairs said, “We haven’t declared consensus on anything yet.” That means the announcement should not be read as a finalized standard or a settled agreement covering every company.
Meta’s communications chief, Andy Stone, said the company strongly supports publishers’ ability to express preferences and called a common signal an important step. He did not commit Meta to honoring this particular rule. The source account also found no statement on adoption from Apple, Amazon, xAI or Perplexity; the list of participating providers is not settled.
“Anthropic “has already started implementing support for this draft.””
— Anthropic
Compliance and Coverage Remain Open
The rule expresses a publisher’s preference; the announcements do not describe a technical mechanism that prevents a nonparticipating crawler from accessing a site. It is also unclear how consistently each provider will apply the signal across its services, especially while implementations are still being developed. Anthropic has said rollout will take time, and the companies have not all provided the same level of detail about coverage.
Meta has not made an implementation commitment, and the source material identifies no adoption statement from several other major AI companies. The proposal also does not state that it will remove content from models already trained. Nor does the shared training rule automatically cover AI answers based on live web content; Microsoft’s separate grounding setting addresses one distinct use case, but the other providers’ treatment of answer generation is not fully specified here.
Finally, the draft may change as the working group considers it. It has not been adopted as an IETF standard, and the October 21 meeting record indicates that consensus had not been declared. The exact final wording and eventual scope therefore remain subject to further work.
Watch Provider Rollouts Through 2027
Website owners can review their robots.txt files and decide whether they want to publish the shared training preference. Those who add it can keep their existing provider-specific crawler rules while companies move toward implementation, rather than assuming the new line has already replaced them. Sites using Microsoft’s NoArchive or NoCache tags should review Microsoft’s stated dates and plan for the change on March 1, 2027.
The next indicators will be whether providers update their crawler documentation and begin honoring the rule in practice. Google’s stated target is early 2027; Microsoft’s preferences begin February 1, 2027; and OpenAI has given a less specific target of the coming months. Further statements from Meta and other AI companies, along with progress in the IETF working group, will show whether the shared signal becomes broader than the current commitments.
Key Questions
What does the shared AI opt-out ask companies to do?
It uses a robots.txt rule with User-Agent: __usage_ai-training and a disallow instruction to express that a publisher does not want its content used to train generative AI models. Whether a company follows the request depends on that company’s implementation and policy.
Does the rule stop AI companies from crawling my website?
No. The shared signal is described as a training preference; it is not a technical barrier to crawling. The announcement also does not say that it removes content from models that have already been trained.
Does opting out of training stop my pages from appearing in AI answers?
Not automatically. Training and use of live web content for AI answers are separate issues. Microsoft announced a distinct AI Grounding preference for certain services; the shared training signal alone does not establish a general opt-out from AI answers.
When will each company honor the signal?
Anthropic says implementation has started, with rollout across its services still underway. OpenAI aims to implement support in the coming months. Microsoft’s date is February 1, 2027, and Google aims for early 2027. These are announced plans and timelines, not confirmation that all services already comply.
Is the shared rule an official internet standard?
No. It is an individual Internet-Draft connected to the IETF’s AI Preferences work. The draft has no formal standing in the standards process, and a working-group chair said consensus had not been declared at the October 21 meeting.
Source: ThorstenMeyerAI.com
