# OpenAI and Anthropic join Google, Microsoft in backing one robots.txt AI-training opt-out

Published: 2026-10-31T03:32:24.167Z · Source: IETF Datatracker (draft-kelley-aipref-training-preference-signal-00) (https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/)
Modified: 2026-11-05T19:49:43.000Z
Second source: OpenAI (https://openai.com/index/helping-websites-express-ai-training-preferences)
Source date: 2026-10-30
Flags: OpenAI and Anthropic positions via secondary reports (OpenAI's post and The Verge not retrievable by the desk); vendor commitments are voluntary, timelines vary
Entities: openai, anthropic, google, gemini

> **Correction, 2026-11-05:** The wire covered this announcement twice. The Oct. 30 item, "Google plans to support shared AIPREF AI training opt-out in robots.txt," now redirects here, and its details are incorporated.

OpenAI, Anthropic, Google and Microsoft said on October 30 that they will honor a single robots.txt policy token, usage_ai-training, so a site can opt out of AI model training across all four with one rule (per reports of the announcements, including The Verge via AI Weekly and Unite.AI on OpenAI's post). The token is defined in an individual IETF Internet-Draft, "Machine-Readable Training Preference Signal," authored by Anthropic's Kevin Kelley and dated October 30; the IETF notes such drafts are not endorsed and have no formal standing. OpenAI says it aims to implement the signal "in the coming months" and that its existing robots.txt controls keep applying. Google and Microsoft had already published their own plans the same day, for 2027 and from February 1, 2027 respectively; Anthropic is reported to have started implementing it.

The draft says the token is not a directive for crawling behavior: it expresses a preference about training only, and wildcard User-agent:  groups do not match it. It also leaves content already collected, and answer-time use of pages, outside what it controls.

Google's own post (https://developers.google.com/search/blog/2026/10/ai-training-control) says a site that uses the shared token to opt out of training, and does not name Google-Extended, would be treated as if it had disallowed Google-Extended, and that company-specific controls will keep working beside the shared signal. Google's crawler documentation (https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), changed the same day, says it intends to support the signal in early 2027, and that sites which explicitly allow Google-Extended stay eligible to appear in Gemini Apps responses through grounding. The post gives no date for the other providers it expects to join and names none.

Why it matters: Until now a publisher wanting out of AI training had to name each company's crawler separately; four of the largest AI companies committing to one token turns an IETF draft into the first opt-out most of them say they will read, though only for training, not crawling or answers.

## What this answers

**Which AI companies will honor the shared robots.txt AI training opt-out?**

OpenAI, Anthropic, Google and Microsoft said on October 30, 2026 that they will honor a shared robots.txt policy token, __usage_ai-training, defined in an IETF Internet-Draft by Anthropic's Kevin Kelley. Timelines differ: OpenAI says in the coming months, Microsoft from February 1, 2027, Google in early 2027.

**Does the __usage_ai-training robots.txt token stop AI crawlers from fetching my pages?**

No. The draft says the token is not a directive for crawling behavior; it expresses a preference about AI training only. It is also an individual Internet-Draft, which the IETF does not endorse and which has no formal standing as a standard.

**Will Google support the IETF's shared AI training opt-out in robots.txt?**

Yes. Google said on October 30 that it plans to support a shared AI training control in robots.txt, based on the IETF's AIPREF work, and aims to do so alongside other AI providers in 2027. Its crawler documentation says early 2027.

**Does a shared robots.txt AI training opt-out replace Google-Extended?**

No. Google says existing company-specific controls keep working alongside it. If a site opts out of training with the shared token and does not name Google-Extended, Google plans to treat that as a Google-Extended disallow.


Canonical: https://anythingengineoptimization.com/item/2026-10-31-openai-and-anthropic-join-google-microsoft-in-backing-one-robots-txt/
From Anything Engine Optimization (AEO Wire) — https://anythingengineoptimization.com/ · Standards: https://anythingengineoptimization.com/standards/
