Skip to content
← Back to Blog

There Is Now One Line That Opts Your Content Out of AI Training. Decide Which Pages Before You Add It.

Your journal and your advocacy pages should not get the same answer. What the new shared AI training preference covers, when it takes effect, and the Bing deadline.

If your association publishes a journal, standards, research or paid courses, someone on your publications team has probably asked whether you can stop AI companies from training on that content. Until now the honest answer was a list: block this company's bot by name, then that one, then the next one that launches, and hope each company reads its own entry the way you meant it.

On October 30, that started to change. Microsoft said it will honor a single shared line in robots.txt that states your AI training preference for every provider that adopts it, and Google said it intends to support the same line. The line is simple. The decision behind it is not, and it belongs to more people than your web team.

Where this comes from. Microsoft's announcement on the Bing blog, Google's crawler documentation and its documentation changelog, Google's January post on controls for its Search AI features, and the IETF working drafts the signal is based on, all read directly. MLex reported that OpenAI and Anthropic made the same commitment. We could not read either company's own announcement, so treat their timing as unconfirmed.

What the Line Looks Like

To opt your whole site out of AI training, the entry in robots.txt is two lines: User-Agent: __usage_ai-training, then Disallow: /. To opt out one section, such as a journal archive, you replace the slash with that section's path, for example Disallow: /journal/. Allow lines work the same way to carve out exceptions.

The name looks like a crawler, but it is not one. No bot visits your site under that name. It is a label that carries your preference to any AI provider that has agreed to read it, so you no longer have to keep up with every company's bot names.

When It Takes Effect

  • Microsoft. Microsoft says it will begin honoring the new preferences when it crawls pages starting February 1, 2027.
  • Google. Google's documentation says it intends to support the shared signal in early 2027.
  • The standard. The signal comes from the AI Preferences working group at the Internet Engineering Task Force. The draft is still a work in progress, so details can change before it becomes a standard.

Nothing stops you from adding the line now. It costs nothing to publish a preference early, and it will be in place when each company starts reading it.

What It Does Not Do

  • It does not take you out of search. Google states that Google-Extended, its existing training control, does not affect whether your site is included in Google Search or how it ranks. Microsoft says the same for traditional Bing results.
  • It does not take you out of AI answers. Training and AI answers are separate. Microsoft now offers a separate grounding preference that controls whether your pages are used to generate AI answers, and it warns that blocking grounding can reduce your visibility in AI experiences. Google handles its AI features in Search through its own separate controls.
  • It does not bind everyone. It is a voluntary preference. It binds the companies that adopt it, and only in the way they say they will honor it.
  • It does not protect members-only content. Anything that should only be seen by members needs to sit behind a login. A line in robots.txt is a request, not a lock.

The Bing Deadline Most Sites Will Miss

Some publishers used NoArchive or NoCache tags to keep their content out of Bing's AI features. Microsoft says that after March 1, 2027, it will no longer read those tags as a preference for training or grounding. If your site relies on them, you need to publish the new training preference by that date, and for grounding, record a temporary preference in Bing Webmaster Tools. Microsoft says sites that miss the date default to allowing grounding until Bing recrawls the pages.

Decide by Section, Not Site-Wide

The most common mistake we expect is a blanket opt-out added by whoever has access to the server. That is a publications, membership and legal decision, and it usually comes out differently for different parts of the site.

Your advocacy positions, public policy pages, find-a-member directory and event pages are content you want AI tools to know and repeat accurately. Opting those out of training gains you little. Your journal archive, standards, certification materials and paid courses are content you sell or license. Those are the sections where an opt-out protects something.

We wrote earlier about bot blocking that ended up blocking Google. The same risk applies here in a different form: a rule written for one purpose that quietly reaches content nobody meant to include.

What to Do This Month

  • Map it. List the sections of your site and who owns each one.
  • Decide it. Ask publications, membership and legal which sections they want used for AI training, and which they do not.
  • Check what is there. Look at your current robots.txt and any NoArchive or NoCache tags, so the new entry does not contradict the old ones.
  • Publish it. Add the new entry for the sections you chose, then confirm it is live on the production site, not just on staging.

If nobody on your team can answer who decided what is in your robots.txt today, that is the place to start. It is a small file, and it now speaks for your content to every AI company that agrees to read it.

83 Creative

We're a web development studio that works exclusively with trade associations, professional societies, and membership organizations.

← Previous Article Everything Else Runs on This One. Nobody Owns It.