Society
Four major AI companies say they will adopt a signal that lets websites tell them no
OpenAI says it will respect a common signal and, according to The Verge, Google, Microsoft and Anthropic will adopt it too. Websites could decide all at once, but it's still unclear exactly what it covers.

The essentials
- Claimed by one partyOpenAI says it will respect a common signal in robots.txt, the file that shows which programs can crawl a website (OpenAI News, The Verge).
- Claimed by one partyAccording to The Verge, Google, Microsoft and Anthropic also say they will adopt that standard (The Verge).
- Claimed by one partyAccording to The Verge, a website could apply its preferences to several AI companies at once, instead of one by one (The Verge).
- Claimed by one partyAccording to The Verge, the proposal comes from Kevin Kelley, technical standards lead at Anthropic, within the Internet Engineering Task Force (IETF) (The Verge).
Why it matters
Less work for anyone who publishes online: according to The Verge, preferences are currently set company by company (The Verge). If this goes ahead, a single signal would be enough for several companies. How far it reaches will depend on who takes part, because OpenAI talks about participating providers, not the whole industry (OpenAI News).
If you have a website or a blog, you could state your preferences to several AI companies at once. If you only browse the web, it shouldn't directly affect you, in principle.
The details
The Verge calls this signal a "policy token", a kind of permissions label inside robots.txt. For years, that file has shown which crawlers can collect a website, whether for search engines or for archiving it (The Verge). The outlet adds that Search Engine Roundtable reported it first (The Verge).
OpenAI talks only about training preferences and about the companies that take part (OpenAI News). The Verge talks about those preferences, but also about opting out of crawling (The Verge). That's not quite the same thing: not using a website to train an AI is one thing, and not collecting its content is another.
What we don't know
- When the four companies will start applying the signal.
- Whether it covers only training or other uses of crawling as well: The Verge talks about both and OpenAI only about training.
- Which other companies will take part, besides the four mentioned.
Sources
- Making it easier for websites to express AI training preferences
- The biggest names in AI are backing a new way to opt out of scraping
Was this story useful? Yes Not really