OpenAI, Google, Microsoft and Anthropic said on Friday, October 30, that they will adopt a shared signal that lets a website tell all of their AI crawlers at once that its content should not be used for AI training. The signal lives in the site’s robots.txt file and replaces the current routine of adding a separate rule for each company. Anthropic already supports it, Google and Microsoft plan to follow next year, and OpenAI says it will implement it in the “coming months.”

What was announced

According to The Verge, the common signal is a “policy token” proposed by Kevin Kelley, a standards lead at Anthropic, through the Internet Engineering Task Force (IETF), the body that publishes internet standards. Until now a publisher who wanted out of AI training had to name every crawler individually: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google and so on. A single shared token means one line can express the preference to several providers.

The four companies say their AI crawlers will stay away from a site that carries the token. Meta’s communications head, Andy Stone, said the company supports the idea of “a common signal that conveys such preferences to multiple providers at once,” but did not say whether Meta will honor the new setting.

How it fits with existing controls

The new token builds on work in the IETF’s AI Preferences working group, whose draft vocabulary describes how site owners can state permissions and restrictions for automated processing of their content. That document is still a draft marked as not reflecting working-group consensus, so details can change before anything is final.

Example robots.txt file with a rule blocking one bot entirely and a crawl delay for all others
A robots.txt file is plain text: each group names a user agent and the paths it may not visit. Image: Belbury / Wikimedia Commons, Public domain

Google’s own crawler documentation already points in this direction. It says Google intends to support the shared training-preference signal, written as a user-agent: __usage_ai-training line, in early 2027. The same page explains that Google-Extended is a control token for whether crawled content may be used to train future Gemini models and to ground answers in Gemini Apps, and that it does not affect inclusion or ranking in Google Search.

Microsoft says it will offer more granular choices, separating opt-out from AI training from opt-out from AI grounding, the practice of pointing a model at a verified source at answer time to reduce hallucinations. Microsoft also notes that sites will still appear in Bing results whatever they choose.

What it means for publishers

For a small site, the practical gain is simplicity: one rule, several recipients, and less need to track new crawler names as they appear. For larger publishers, the shared signal is a baseline for negotiations. A site can opt out of training by default and still license its archive to specific companies, as some news organizations have done in deals such as the commercial terms Meta offered to publishers.

The signal has clear limits. It is a request, not a technical barrier, so it works only for companies that choose to honor it. It also applies to future crawling: content already collected is not removed. And the dates matter. Until Google, Microsoft and OpenAI ship support, publishers still need the per-crawler rules they use today. Our guide to controlling AI crawlers in robots.txt lists the current user agents, and the history of robots.txt explains why a 1994 convention is still the main lever.

Other approaches publishers are using

The Verge points to two other efforts. The Really Simple Licensing (RSL) standard lets publishers set terms and charge crawlers that visit. Cloudflare, for its part, blocks AI crawlers by default for its customers. The shared token does not compete with those: it states a preference, while RSL and network-level blocking add pricing and enforcement.

What to watch next

  • Whether the token is published as a final IETF standard, and whether the draft vocabulary changes.
  • The date OpenAI ships support, and whether it covers all of its crawlers or only training.
  • Whether Meta, Apple, Perplexity and smaller AI companies commit to honoring the signal.
  • How courts treat a documented opt-out in the training-data lawsuits we track in our copyright explainer.
  • Whether publishers report fewer unwanted AI requests in their server logs after rollout.

Key takeaways

  • Four major AI labs say they will honor one robots.txt signal for AI-training opt-out.
  • Anthropic supports it now; Google and Microsoft plan support in 2027; OpenAI says “coming months.”
  • Meta welcomed the idea but has not said it will honor the signal.
  • The signal is a voluntary request, applies going forward, and does not remove data already collected.
  • Publishers should keep their per-crawler rules until each company confirms support.

Frequently asked questions

What is the new AI opt-out signal?

It is a shared token placed in a site’s robots.txt file that tells participating AI companies the site does not want its content used for AI training. It is being developed through the IETF.

No. Google says its AI-training control does not affect Search inclusion or ranking, and Microsoft says sites still appear in Bing results regardless of their preferences.

Does the signal delete content that was already used for training?

No. It expresses a preference for future crawling. Content already collected is not removed by adding the token.