Google, OpenAI, Anthropic and Microsoft Back AI Training Opt-Out

Google, OpenAI, Anthropic and Microsoft all said on October 30, 2026 that they will honor a new shared robots.txt signal that lets website publishers opt out of having their content used to train generative-AI models with a single directive, instead of maintaining separate rules for every crawler. The mechanism, formally titled "Machine-Readable Training Preference Signal," comes from an individual Internet-Draft filed at the IETF by Kevin Kelley, Anthropic's technical lead on AI standards work. It does not undo any scraping that has already happened, and it has no effect on whether a site appears in search results or rankings — it only governs future crawling done to train AI models.

What happened on October 30

Google's Search Central Blog, Microsoft's Bing blog, and OpenAI's own site all published statements within the same window on October 30, 2026, each saying the company would support the same new preference token in robots.txt. Anthropic did not publish a dedicated blog post; its most direct public statement came from Kevin Kelley himself, posting to the IETF's "ai-control" mailing list the same day to introduce the draft he had just filed.

Google's post, written by Search Advocate John Mueller, put it plainly: "Today, we're announcing our plans to support a shared AI training control in robots.txt. This will help websites express their AI training preferences across multiple AI providers in a single step and is based on work from the AIPREF working group at the Internet Engineering Task Force (IETF)." Google said the new token builds on Google-Extended, the company-specific AI training control it has offered since 2023, and that if a site opts out using the new shared token without separately mentioning Google-Extended, Google "plans to interpret that as if the publisher had disallowed Google-Extended."

OpenAI's post framed the move the same way: a single preference that multiple AI providers can read, rather than separate rules per crawler, with the company saying it "encourages other AI providers to honor the signal" and that it plans to implement support "in the coming months." Microsoft's Bing blog post, written by Principal Product Manager Krishna Madhavan, went furthest on specifics, laying out exact dates and a second, related control covering how content is used inside AI-generated answers (see below).

How the signal actually works

The draft adds no new robots.txt grammar. It reuses the existing RFC 9309 group structure, longest-match path rules, and Allow/Disallow directives that have governed crawler access for decades — it just defines a new, purpose-specific token to put on the User-Agent line: __usage_ai-training. A publisher writes an ordinary-looking robots.txt group keyed to that token, for example blocking training on the whole site but carving out an exception for a press kit:

Any crawler that recognizes the token checks a requested path against the group using the same lookup rules already defined in RFC 9309. A disallowed result means the publisher has expressed a preference against that use; an allowed result means the opposite. The draft explicitly notes that wildcard User-Agent: * groups do not match this token, so general crawler rules and the new training-specific preference stay independent of each other. The draft also allows that private agreements between a publisher and an AI company may override what robots.txt says — so a licensing deal can still supersede the default signal.

What it does not do

Two limits are worth stating clearly, because they are the parts easiest to misread as a blanket "AI opt-out." First, the signal is forward-looking only: the draft ties the preferences that apply to a given page to whatever robots.txt said "in effect when the asset was retrieved," meaning it cannot reach back and erase content a company already scraped and trained on before a publisher ever set the flag. Second, none of the four companies' statements connect this to search visibility. Google's post, Microsoft's post, and the underlying draft all keep the training preference separate from crawling for search indexing — disallowing AI training does not pull a page out of Google Search, Bing, or any other index.

How each company's rollout differs

The companies agree on the mechanism but not on the calendar, and Anthropic, Google, OpenAI and Microsoft are each at a different stage:

Microsoft's second control: training vs. grounding

Microsoft's announcement went beyond the shared training token to introduce a second, independent preference it calls "grounding" — covering whether a publisher's content can be used as live input to produce an AI-generated answer (in Copilot, AI Mode, or AI Overviews), as distinct from being baked into a model's training data. Grounding preferences are set at the page level only, through a meta tag or X-Robots-Tag header such as ai-training:disallow or ai-grounding:disallowextended, and support three values: Allow, DisallowExtended (restricting grounding to Microsoft's own surfaces and excluding third-party grounding services), and Disallow. Both default to Allow when a publisher sets nothing. As Madhavan put it in the Bing post: "Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work." Microsoft also stressed, twice, that neither setting is a guarantee: "Neither choice guarantees inclusion, traffic, or payment."

Microsoft's timeline is also the most concrete of the four. The new preferences take effect as Bing recrawls sites starting February 1, 2027; publishers have until March 1, 2027 to publish a preference (and, for grounding, to record any temporary preference in Bing Webmaster Tools); and a temporary grounding-transition window for sites currently relying on NoArchive or NoCache opens December 15, 2026 and expires September 30, 2027. Microsoft notes that publishing a new setting does not take effect immediately — it only applies once Bing actually recrawls the page, and pinging IndexNow can prompt but not guarantee a recrawl.

Who is behind the draft

The draft itself credits a broader group than the four companies that announced support on October 30. Its acknowledgments section lists contributors affiliated with Google, Microsoft, OpenAI, Cloudflare and Meta, in addition to author Kevin Kelley of Anthropic — signalling that the token was shaped with input well beyond the companies that have so far committed to specific rollout dates. The work sits under the IETF's AI Preferences (AIPREF) working group, which separately maintains a broader vocabulary draft, draft-ietf-aipref-vocab, led by Paul Keller of Open Future and edited by Mozilla's Martin Thomson — a document Google's own post describes as still under "multistakeholder discussions" rather than finished. Kelley's narrower draft was deliberately scoped to just the training signal and its robots.txt syntax, rather than waiting for that fuller vocabulary effort to conclude.

Google's post also tied the move to regulatory context, saying the proposal "is consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act that recognizes robots.txt as a primary method for web publishers to express machine-readable preferences," and that the draft reflects "feedback from publishers, civil society groups, regulators and other stakeholders that have called for industry-wide controls on AI training."

Why a shared token matters

Since generative AI crawlers began scaling up, publishers have had to maintain a growing patchwork of company-specific rules — disallowing GPTBot for OpenAI, Google-Extended for Google's Gemini training, ClaudeBot for Anthropic, and so on — each with its own syntax quirks and its own crawler name to track. A single purpose-coded token collapses that into one line that any participating company's crawler can read the same way, while still letting a publisher layer company-specific exceptions on top if they want more granular control. That is also why this is being pursued as an IETF draft rather than a private agreement between a couple of companies: a standards-track document, even an individual one not yet adopted by the IETF as a whole, gives any crawler operator — not just the four named here — a stable, documented syntax to implement against.

It is a narrower move than it might sound, though. None of the four companies' statements suggest the signal resolves ongoing legal disputes over AI training on previously published content, and the draft's own text allows private licensing agreements to override it anyway. For publishers that have already struck or are negotiating direct licensing deals with AI companies, this is a technical complement to those agreements, not a replacement for them. The AI industry's broader shift toward structured training-data agreements is visible elsewhere too — OpenAI's own infrastructure deal with Google Cloud is a reminder that even competitors are increasingly formalizing how they share resources and data, and OpenAI's earlier move away from training directly on the live internet shows training-data sourcing has already been shifting before this standard existed.

Open questions going into 2027

A few things remain unsettled. The draft is still an individual submission rather than an IETF working-group product, meaning it has not gone through the IETF's formal consensus and standardization process — it currently expires on May 3, 2027 unless renewed or adopted. Google, OpenAI and Microsoft have each committed to honoring the token, but none has published the detailed engineering specifics of how its crawlers will parse edge cases, and enforcement remains voluntary: robots.txt has always been an access-preference protocol that crawlers can choose to respect, not a technical barrier. Meta and Cloudflare are credited as contributors to the draft's development but have not themselves announced rollout dates.

What happens next

Watch Anthropic's own crawler documentation for the first concrete sign of support landing in production, since Kelley's mailing-list post said only that implementation has begun, not that it is complete. OpenAI has given itself "the coming months" with no fixed date, so its first implementation announcement is the next milestone to track there. Google and Microsoft have both pointed to 2027, with Microsoft's February 1, 2027 enforcement date and March 1, 2027 publisher deadline the most concrete dates on the calendar for publishers to prepare robots.txt changes around. Expect further detail to emerge from the IETF's AIPREF working group as the broader vocabulary draft continues through its own standardization process, and watch for whether other large crawler operators — beyond the group credited in the draft's acknowledgments — adopt the same token in the months ahead.

Written byTheo Park

Theo Park runs the AI desk at Pandromeda. He follows model launches from the frontier labs and the open-weight community, tracks the assistants and developer tools built on them, and explains what each release changes on pricing, capability and safety. His reporting leans on primary sources: model cards, technical reports, API documentation and the companies' own announcements.

More from Theo ParkAll AI coverage