For years, a publisher that wanted to keep its work out of generative AI training had to repeat the same chore for each major AI company. It had to learn one company’s control, then another’s, then another’s, and check whether any of them had changed. A rule written for one crawler did not necessarily mean anything to the next, so opting out was a patchwork that was easy to get wrong and hard to keep current, even for large publishers.
On October 30 and 31, 2026, that patchwork began to narrow. Google, Microsoft, OpenAI and Anthropic pledged to honour a single shared robots.txt signal for opting out of generative AI training, according to reporting by The Verge, Engadget, AFP, The Hindu and Malay Mail. The signal is a policy token called __usage_ai-training. In practical terms, a publisher can now express one training preference in one familiar place, and the four companies have said they will respect it, each on its own implementation timeline.
That distinction matters. This is a shared commitment to honour the same signal, not a single switch that takes effect everywhere at once, and not a remedy for training that has already happened. The companies are moving on different schedules, and the token itself is still an individual proposal rather than a formal standard. This article explains what was agreed, how the opt-out works, when each company plans to support it, what it replaces, what it does not do, and how a publisher can put it in place.
The agreement in plain terms
The core idea is deliberately simple. Instead of maintaining a separate opt-out for each AI company, a publisher adds one token to its robots.txt file. That token states a preference about generative AI training, and the four companies that have pledged to honour it will treat that preference as applying to their generative AI training use, within the scope and limits described below.
The backdrop explains why a single token matters. Publishers have pushed back against unauthorized collection of their content for training, amid public debate and legal battles over training data. Meanwhile, controls grew company by company, so opting out meant repeating the work for each developer. A shared token does not settle the larger legal questions or resolve past disputes; it reduces the friction of expressing a forward-looking preference.
Where the token comes from
The token comes from an individual IETF Internet-Draft titled “Machine-Readable Training Preference Signal.” The draft is by Kevin Kelley, Anthropic’s technical lead for standards, who has also been described as the company’s chief standards engineer. It is dated October 30, 2026, and it expires on May 3, 2027.
That provenance must be stated precisely. The draft is an individual submission. The IETF has not endorsed it, and it has no formal standing as an official IETF standard. Its importance comes from the four companies pledging to honour it, not from standards-body approval.
How the one-line opt-out works
The mechanism uses robots.txt, the long-established file that websites use to communicate crawling preferences to automated visitors. A publisher places a robots.txt file at the root of its site, and crawlers that respect robots.txt read it before they fetch pages. The new token fits into that familiar system, rather than requiring publishers to learn an entirely new file format or a new control panel.
The token itself is __usage_ai-training. In the example described in the reporting, a publisher that wants to opt out its whole site from generative AI training would use that token as the user-agent line, followed by a disallow rule for the whole site. A publisher that wants to allow a specific part of the site, even while opting out the rest, can add an allow line for that part. The widely cited example uses a press kit as the carve-out, which shows how a publisher can keep promotional material available while opting out the main body of its work.
User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/
Read line by line, the example is straightforward. The user-agent line names the policy token, not a single company’s crawler. The disallow line covers the whole site. The allow line then creates an exception for one path, in this case the press kit. Publishers can adapt the allow lines to their own structure, but the logic stays the same: disallow broadly, then allow narrowly where an exception is genuinely intended.
Company-by-company rollout timeline
The pledges are shared, but the timelines are not identical, and that is one of the most important details for publishers. Adding the token now expresses a preference, but each company will honour it according to its own schedule, and some companies are further along than others.
Anthropic has already begun implementing the token, making it the earliest mover among the four, which is consistent with the underlying draft coming from an Anthropic standards lead. OpenAI, by contrast, aims to implement the token in the coming months, so publishers should expect a gap between adding the rule and OpenAI honouring it, even though the commitment has been made. Microsoft will support the token from February 1, 2027. Microsoft has also said that its existing NoArchive and NoCache tags will be replaced as AI settings after March 1, 2027, and it offers a separate page-level grounding setting in addition to the robots.txt token. Google aims to support the token in early 2027, and Google will treat the token as blocking Google-Extended, its existing control for AI-related use.
| Company | Stated timeline | What publishers should note |
|---|---|---|
| Anthropic | Already begun implementing | Earliest implementation among the four companies |
| OpenAI | Aims to implement in the coming months | Commitment made, but implementation still ahead |
| Microsoft | From February 1, 2027 | NoArchive and NoCache tags replaced as AI settings after March 1, 2027; separate page-level grounding setting also offered |
| Aims to support in early 2027 | Token will be treated as blocking Google-Extended | |
| Meta | No commitment | Expressed support, but has not committed to honouring the token |
Where Meta stands
Meta’s position is easy to misread. Meta has expressed support for the shared approach, but it has not committed to honouring the token. Support and commitment are not the same thing here, so publishers should not assume Meta will treat the token as an opt-out unless Meta makes that commitment and implements it.
Old per-company controls vs the new shared token
Before this agreement, a publisher that wanted to opt out of generative AI training generally had to tell each company separately. That meant learning each company’s preferred control, writing a separate rule for each one, and keeping all of those rules current. The controls had different names, with examples including GPTBot for OpenAI and Google-Extended for Google. A publisher that completed one company’s opt-out had not automatically completed the others, and a publisher that missed one company had left a gap.
The new shared token collapses that stack into one rule. Instead of repeating the preference in company-specific forms, the publisher states it once, using __usage_ai-training, and the four pledging companies will honour that single statement. That lowers the chance that a missed rule leaves content exposed to one company’s training pipeline while it is excluded from another’s.
| Question | Old per-company approach | New shared token |
|---|---|---|
| How many rules? | One rule per company, repeated separately | One shared rule for the pledging companies |
| Examples of controls | GPTBot, Google-Extended, and other company-specific controls | __usage_ai-training in robots.txt |
| Risk of a gap | High: missing one company’s control left that company uncovered | Lower for pledging companies, but Meta has not committed |
| Maintenance | Each company’s rule had to be checked and updated separately | One file to update, with company timelines still to track |
| Scope | Defined company by company | Generative AI training preference, as pledged by each company |
What it does NOT do
The limits of the new token are just as important as its benefits, and several are stated explicitly in the reporting.
First, the token applies only to future training. It does not undo past scraping, and it does not remove content that has already been collected or used for training. A publisher that adds the token today is expressing a preference about what happens next, not reversing what has already happened. Second, the token does not prevent crawling altogether. It is a training preference, not a general block on all automated access, and crawlers may still visit a site for other purposes governed by other rules.
Third, the token does not prevent content from appearing in Google Search. Google and Microsoft have both said that the preference does not affect search visibility: opting out of training is not the same as opting out of search indexing. Fourth, content is scraped by default until a publisher changes the setting. Silence is not an opt-out.
| Question | Answer under the new token |
|---|---|
| Does it undo past scraping or past training? | No: it applies only to future training |
| Does it block all crawling? | No: it does not prevent crawling altogether |
| Does it remove content from Google Search? | No: it does not prevent content appearing in Google Search |
| Does it affect search visibility? | No: Google and Microsoft say the preference does not affect search visibility |
| Does it apply if a publisher does nothing? | No: content is scraped by default until a publisher changes the setting |
| Does Meta honour it? | Not on current reporting: Meta expressed support but has not committed to honouring it |
A note on legal questions
This article describes a technical preference signal and the pledges made by four companies to honour it. It is not legal advice, and it does not address whether any particular past or future use of content was lawful, what remedies might be available, or how a publisher should handle a dispute. Publishers with legal questions about training data, licensing, or enforcement should consult counsel who can advise on their specific circumstances.
Step-by-step publisher guide
For a publisher that has decided to opt out, the steps are short, but each deserves care. The guide below follows the example described in the reporting.
- Decide the policy first. Confirm whether the whole site should be opted out, or whether specific sections should remain available. Note any carve-outs, such as a press kit, before writing the file.
- Open the existing robots.txt file. Check what rules are already in place, especially any rules governing search crawling, so the new training rule does not conflict with them.
- Add the shared token. Add a block using User-Agent: __usage_ai-training, followed by Disallow: / for a whole-site opt-out.
- Add allow lines only where intended. If a section should remain available, add a narrow Allow line for that path, such as Allow: /press-kit/, and check that the path is correct.
- Publish the file at the site root. Confirm that the file is reachable and that the new block appears exactly as intended, with no typing errors in the token name.
- Record the date and keep a copy. Save the version published and note the date, so the publisher can track each company’s implementation timeline against it.
- Revisit on each company’s timeline. Remember that Anthropic has already begun implementing, OpenAI aims to implement in the coming months, Microsoft will support the token from February 1, 2027, and Google aims to support it in early 2027. Check back as those dates arrive, and review older controls, including Microsoft’s NoArchive and NoCache tags after March 1, 2027, before removing them.
- Do not assume Meta coverage. Meta has expressed support but has not committed to honouring the token, so do not treat the shared rule as covering Meta unless that position changes.
Common mistakes to avoid
The most common mistake would be to misspell the token: it begins with two underscores, and a missing underscore would not match the agreed signal. Another mistake would be an allow line broader than intended, which could reopen a large part of the site. A third would be assuming the token removes content from search or reverses past training, neither of which it does. Reading the file back line by line after publishing catches all three errors.
Frequently asked questions
Is __usage_ai-training now an official IETF standard?
No. The token comes from an individual IETF Internet-Draft titled “Machine-Readable Training Preference Signal,” dated October 30, 2026, and expiring May 3, 2027. It is an individual submission, the IETF has not endorsed it, and it has no formal standing as an official standard. Its practical importance comes from the pledges by Google, Microsoft, OpenAI and Anthropic to honour it, not from formal standards approval.
When will each company actually honour the new token?
The timelines differ. Anthropic has already begun implementing the token. OpenAI aims to implement it in the coming months. Microsoft will support it from February 1, 2027, with its NoArchive and NoCache tags replaced as AI settings after March 1, 2027. Google aims to support it in early 2027, and will treat it as blocking Google-Extended. Publishers should therefore expect company-by-company effect, not a single switch-on date.
Will adding the token hurt my Google Search visibility?
According to the reporting, no. The token does not prevent content from appearing in Google Search, and Google and Microsoft say the preference does not affect search visibility. The token expresses a preference about generative AI training, which is a separate question from search indexing and search presence.
Does the token remove content that was already scraped for training?
No. The token applies only to future training. It does not undo past scraping, and it does not remove content already collected or used. It is a forward-looking preference, so publishers that add it are stating what they want to happen next, not reversing what has already happened.
Do I still need the old per-company controls if I add the shared token?
For the transition period, publishers should be cautious about removing older controls immediately. The shared token collapses the old per-company stack into one rule for the pledging companies, and Google will treat it as blocking Google-Extended, but implementation timelines are staggered and Microsoft’s older tags are only replaced as AI settings after March 1, 2027. Review existing rules carefully, keep the file consistent during the transition, and do not assume Meta coverage, since Meta has expressed support but has not committed to honouring the token. Publishers with legal questions should consult counsel.
Join the discussion
Have a question or a perspective to share?