The new robots.txt AI training opt out lets a small business state its training preference once, in one place, instead of maintaining a separate rule for every AI company. On October 30, 2026, Google, Microsoft, and OpenAI each announced they will honor one shared robots.txt signal for opting out of generative-AI training, based on a draft by Anthropic’s standards lead. This post explains what the line is, when each company will honor it, why training is not search, and the five steps to take now.
Table of Contents
What actually changed on October 30
Until now, opting out of AI training meant naming each company’s crawler or control token one by one: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, Applebot-Extended for Apple, and CCBot for Common Crawl.
The October 30 announcements replace that per-company list with a single shared preference. The mechanism comes from an individual Internet-Draft, “Machine-Readable Training Preference Signal” (draft-kelley-aipref-training-preference-signal-00), written by Kevin Kelley, a technical lead for standards at Anthropic, for the IETF’s AI Preferences working group. First, it is an individual draft, not an adopted standard: the IETF has not endorsed it, it has no formal standing, and it expires on May 3, 2027 unless revised. Second, it binds only companies that choose to follow it; Meta has not committed.
Anthropic’s own position needs care. Kelley has said implementation there has begun, but Anthropic published no standalone announcement; its support is reported through the draft author’s posts, not a newsroom statement. Treat Anthropic as “implementing, details may change,” not a finished commitment.
The one line, explained
The shared signal is a policy token, __usage_ai-training, that goes where a crawler name normally goes in a robots.txt group:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That example opts the whole site out of training except the press kit. The token names a use — training — not a crawler: one group speaks to every participating provider at once. Ordinary path rules still apply inside the group.
Three rules in the draft decide how the group behaves. First, it is not a crawl rule: the draft states the token “is not a directive for crawling behavior,” so a crawler that reads it can keep visiting and caching pages. Second, the catch-all User-agent: * group does not match it — a file with no __usage_ai-training group expresses no training preference at all, so the token must be spelled exactly. Third, software that does not implement the token ignores the group as it ignores any unknown user-agent. That makes it safe to add now.
The draft defines “AI training” narrowly: using an asset to modify the learned parameters of a generative AI model. Training a model used only for classification, ranking, or scoring falls outside it.

When each company will actually honor it
None of the three companies that announced support honors the token yet, and their timelines run into 2027. Do not expect the line to do anything the day you add it.
| Company | Timeline | What to know |
|---|---|---|
| Microsoft (Bing) | From February 1, 2027, as Bing recrawls | Separate page-level AI Grounding preference; NoArchive/NoCache stop counting after March 1, 2027 |
| Planned in 2027; docs say early 2027 | Token alone reads as a Google-Extended disallow unless Google-Extended is named | |
| OpenAI | “The coming months” | Existing GPTBot controls keep working meanwhile |
| Anthropic | Implementation begun, per the draft author | No standalone announcement; details may change |
Microsoft’s announcement is the most concrete, with a deadline attached. Alongside the shared training token, Microsoft adds a separate AI Grounding preference — the retrieval step that pulls live web content into chat answers, distinct from training model weights. Grounding is set per page, by meta tag or HTTP header; Microsoft says restricting it reduces visibility in AI answers while leaving Bing results untouched, as with training opt-out.
If your site still relies on Microsoft’s older NoArchive or NoCache tags to limit AI behavior, March 1, 2027 is your migration deadline: after that date, Bing stops reading those tags as training or grounding signals. A temporary grounding preference in Bing Webmaster Tools opens December 15, 2026, but only for sites still using those tags on that date, and it lasts until Microsoft recrawls the page and finds the new preference, or September 30, 2027, whichever comes first.
Training opt-out is not a search opt-out
This is the distinction the whole topic turns on, and getting it wrong costs visibility. AI companies run separate crawlers for separate jobs: training crawlers collect content that may shape future models, while search crawlers fetch pages so an assistant can show and cite them in answers. OpenAI’s GPTBot collects training data; its OAI-SearchBot decides whether a site can appear in ChatGPT search answers. Anthropic splits ClaudeBot from Claude-SearchBot the same way, PerplexityBot surfaces pages in Perplexity answers rather than training on them, and Google-Extended is a control token, not a crawler — Google says it does not affect inclusion or ranking in Google Search.
Blocking a training crawler therefore does not remove you from AI answers, and blocking a search crawler does not merely stop training — it can remove you from answers entirely. An October 2026 test of 2,000 popular websites found exactly that mistake: 45% of the sites that blocked GPTBot also blocked OAI-SearchBot, quietly opting themselves out of ChatGPT search answers as well as training.
The practical rule is simple: if you want to be cited, explicitly allow the search crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot — while you opt out of training. The shared token helps because it says nothing about search: adding the __usage_ai-training group cannot accidentally block a search crawler; only a Disallow aimed at a search crawler’s own name can.
What to do now: a 5-step checklist
1. Keep your existing rules. The shared token is not live anywhere yet, so the per-company rules you already have — GPTBot, ClaudeBot, Google-Extended — remain what actually works today. Deleting them now would leave you with no working opt-out.
2. Add the shared group alongside them. Non-implementing software ignores the group, so adding it now is harmless and means you are covered the moment each company ships support. Add it at the end of your robots.txt file.
3. Audit for the search-blocking mistake. Open your robots.txt file and search for OAI-SearchBot, Claude-SearchBot, and PerplexityBot. If any is disallowed and you want AI answers to cite you, remove that rule. Keep Googlebot and Bingbot allowed in all cases.
4. Edit the file the safe way for your platform. On WordPress, the file is virtual unless a physical robots.txt file exists in your site root. If Rank Math manages it, go to Rank Math, then General Settings, then Edit robots.txt, and use the raw editor. If Yoast manages it, use SEO, then Tools, then File Editor. If a physical file sits in your web root, edit that file instead — it overrides the virtual one. Keep a copy before you change it, and never put Disallow: / under User-agent: *: that blocks every crawler from every page.
5. Calendar the Microsoft dates and re-check quarterly. December 15, 2026 (temporary Bing preference opens), February 1, 2027 (Microsoft starts honoring the new preferences), and March 1, 2027 (NoArchive and NoCache stop counting). Re-check quarterly through 2027.

What this does not do
Five limits are worth stating plainly. It does not stop crawling: a bot can still fetch and cache a page that opts out of training. It does not remove content already collected or already trained into model weights; it governs future training use, not the past. It does not cover AI answers or grounding except through Microsoft’s separate grounding preference. It does not bind anyone who has not signed up, and scrapers that ignore robots.txt will ignore this too. And it is not yet a standard: an individual draft with a May 2027 expiry is a promising start, not a settled rule.
FAQ
Does adding the new robots.txt line remove my site from ChatGPT or Google search?
No. The shared token covers generative-AI training only, and the draft states it is not a crawl directive. Microsoft says training opt-out does not affect Bing results, and Google says its Google-Extended control does not affect Google Search inclusion or ranking. Only blocking a search crawler such as OAI-SearchBot removes you from that assistant’s answers.
When will the shared opt-out actually work?
On staggered timelines into 2027. Microsoft starts honoring it on February 1, 2027 as Bing recrawls pages; Google aims for early 2027; OpenAI says “in the coming months”; Anthropic has begun implementation but published no standalone announcement. Until each company ships support, your existing per-company rules are what work.
Should I delete my existing GPTBot and Google-Extended rules?
Not yet. The shared token is not live anywhere, so deleting the old rules would leave you with no working opt-out. Keep them, add the shared group alongside them, and revisit once each company’s support is confirmed live.
Does opting out delete content already used for training?
No. Updating robots.txt does not remove content already collected into existing datasets or already trained into model weights. The token states a preference for future training use only.
Conclusion
The shared robots.txt AI training opt-out is genuinely useful and genuinely unfinished: one line that will eventually speak to every participating AI company, arriving on four different schedules, under a draft that still needs to become a standard. For a small business, the right move is unglamorous and cheap — keep the rules that work today, add the shared group now so it is in place when support lands, make sure you have not blocked the search crawlers that bring citations, and calendar Microsoft’s February and March 2027 dates. That protects both halves of what you want: content out of training sets, and your business still findable in AI answers.
Sources
- Search Engine Journal, “Google, Microsoft & OpenAI Plan To Honor Shared AI Training Opt-Out” (November 1, 2026): https://www.searchenginejournal.com/google-microsoft-openai-shared-ai-training-opt-out/595387/
- RR Tech Pulse, “Opt out of AI training in robots.txt” (updated October 31, 2026): https://rrtechpulse.com/stories/robots-txt-ai-training-opt-out-signal/
- The SEO Handbook, “Shared robots.txt AI Training Opt-Out”: https://seohandbook.co.uk/seo-news-updates/shared-ai-training-opt-out-robots-txt/
- Dev.to, “Blocking GPTBot Isn’t Blocking ChatGPT Search: A robots.txt for the 3 Kinds of AI Crawlers”: https://dev.to/posting-dude/blocking-gptbot-isnt-blocking-chatgpt-search-a-robotstxt-for-the-3-kinds-of-ai-crawlers-459a
- TutorialsWebsite, “AI Training Opt-Out: New robots.txt Rule for Your Website”: https://www.tutorialswebsite.com/ai-training-opt-out-robots-txt-rule/
If you have not set up AI basics yet, read our small business AI implementation guide first, and our brand visibility across AI engines guide explains how to check whether AI answers cite you.
Was this article helpful?