🔍 Read the full analysis: OpenAI, Google, Microsoft And Anthropic Back An Opt-Out: What To Know on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI, Google, Microsoft and Anthropic have each said they will support a shared robots.txt signal for publishers who do not want their content used to train generative AI models. The opt-out is voluntary, is not yet an official internet standard, and does not block AI answers based on live web content; companies have set different implementation timelines.
OpenAI, Google, Microsoft and Anthropic said they will honor a proposed shared website signal asking that content not be used to train generative AI models. The signal, published October 30 as an Internet-Draft, is intended to replace the need for publishers to maintain separate training opt-outs for each company, but it is a voluntary request, not a technical block or an adopted internet standard.
The draft, titled “Machine-Readable Training Preference Signal,” specifies a shared robots.txt rule group: User-Agent: __usage_ai-training followed by Disallow: /. A publisher adding it would be indicating that participating companies should not use the site’s content for generative AI model training. The draft was written by Anthropic standards lead Kevin Kelley and is part of work associated with the Internet Engineering Task Force’s AI Preferences working group.
The companies have not committed to one common launch date. Anthropic says it has started implementing support, though it says support will take time to reach its services. OpenAI aims to implement the signal “in the coming months.” Google says it is aiming for 2027, with its crawler documentation specifying early 2027. Microsoft says it will honor the signal from February 1, 2027, as it crawls websites.
Microsoft also announced a distinct page-level preference for AI Grounding—the use of web content to inform AI answers in Bing, Copilot and Microsoft 365. Its announcement says both preferences take effect February 1, 2027. Microsoft’s existing NoArchive and NoCache tags will stop counting as training or grounding preferences after March 1, 2027. Google says the new signal will act like a disallow rule for Google-Extended when a publisher has not already specified a Google-Extended rule. Existing company-specific crawler controls remain in place during the changeover.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
For publishers, a common rule could reduce the work of expressing a preference across several large AI companies. Until now, a site seeking to limit training use has generally needed company-specific instructions, such as rules for OpenAI’s GPTBot, Google-Extended and Anthropic’s ClaudeBot. A shared signal offers a simpler way to communicate that preference to companies that adopt it.
But the practical effect depends on each company’s implementation and compliance. Robots.txt communicates a request to crawlers; it does not stop a crawler that chooses not to follow it. The proposal also creates no right to payment and does not guarantee that a publisher’s content will be excluded from every AI product. Microsoft explicitly says its choices do not guarantee inclusion, traffic or payment.
The distinction between training and other uses is important. The shared rule is aimed at foundational model training, not AI answers assembled from live web content. Microsoft’s separate AI Grounding preference addresses one such use for its services, but the other companies’ announcements do not establish that the shared training rule will govern their live-answer systems. Publishers will need to decide separately how they want to handle training, AI answers and search access.
Products Worth Considering
LG UltraGear 27-Inch QHD OLED Gaming Monitor, 240Hz, 0.03ms, G-SYNC Compatible, FreeSync Premium Pro, HDMI 2.1
Samsung 27-inch Odyssey OLED G6 QD-OLED QHD Gaming Monitor
LG UltraGear 32GX850A-B 32-Inch 4K UHD Glossy OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
From Company Rules to One Draft
The development comes after publishers have had to manage a patchwork of crawler preferences. OpenAI, Google and Anthropic each have company-specific controls. Microsoft has relied on page-level NoArchive and NoCache tags for related preferences; it says those tags will be superseded for training and grounding purposes on its stated timetable. The companies say their existing crawler controls continue to apply, so the draft does not immediately replace those settings.
The proposal is an individual Internet-Draft, not an official IETF standard. The source material reports that the draft has no formal standing in the IETF standards process. At a working-group meeting on October 21, a chair said, “We haven’t declared consensus on anything yet.” Support from major companies may encourage adoption, but it does not itself settle the draft’s status or require other services to follow it.
Meta’s communications chief, Andy Stone, said the company supports publishers’ ability to express preferences and called a common signal an important step. He did not state that Meta would honor this specific signal. The source material also reports no commitment from Apple, Amazon, xAI or Perplexity, so participation beyond the four named companies remains unconfirmed.
““has already started implementing support for this draft””
— Anthropic
Adoption and Scope Still Vary
It is not yet clear whether the working group will reach consensus on the draft, whether its wording will change, or when each company will finish implementation across its services. The current commitments also differ: Anthropic says implementation has begun, while the other companies describe future targets. Meta has expressed support for the idea but has not made a clear commitment to honor this signal.
It also remains unclear how broadly the rule will be adopted by AI companies outside the four named firms. The signal is a request rather than an enforcement mechanism, and the announcements do not establish how companies will handle content already used to train existing models. The shared training preference does not cover live-web answers by itself; Microsoft has announced a separate control for that use, while equivalent commitments from the other companies are not specified in the source material.
Watch Rollouts Through 2027
Publishers can add the shared rule if they want to express an opt-out from training, while keeping existing company-specific crawler instructions until providers have implemented the new signal. Sites using Microsoft’s NoArchive or NoCache tags for training or AI Grounding preferences should review Microsoft’s stated March 1, 2027 cutoff and plan to use its new preferences where relevant. Microsoft says its new controls begin taking effect as it crawls from February 1, 2027.
The next milestones are company implementation updates and the IETF working group’s handling of the draft. Google’s target is early 2027; OpenAI has given a less specific “coming months” timeframe. Until adoption and scope are clearer, publishers will need to check each provider’s documentation rather than assume that one line immediately replaces every existing setting.
Key Questions
What does the shared robots.txt signal do?
It tells participating companies that a website does not want its content used for generative AI training. The proposed rule uses the user-agent token __usage_ai-training with Disallow: /.
Does the signal technically prevent AI companies from crawling a site?
No. Robots.txt expresses a preference or request; it is not a technical barrier. Its effect depends on whether a company adopts and follows the signal.
Does opting out stop AI answers from using website content?
Not under this training signal alone. The proposal addresses model training, not answers based on live web content. Microsoft announced a separate AI Grounding preference for some of its services.
When will the companies support the signal?
Anthropic says implementation has started; OpenAI aims to implement it in the coming months. Google is targeting 2027, with documentation saying early 2027. Microsoft says its preference takes effect from February 1, 2027, as it crawls.
Will Meta honor the signal?
Meta’s communications chief voiced support for publishers expressing preferences and for a common signal, but did not confirm that Meta will honor this specific rule.
Source: ThorstenMeyerAI.com
