OpenAI, Google, Microsoft and Anthropic Adopt Unified AI Crawl Opt-Out Standard
OpenAI, Google, Microsoft, and Anthropic have agreed to adopt a new standard proposed by Anthropic that allows content publishers to opt out of AI training data scraping via robots.txt. This unified mechanism simplifies the process for websites to declare preferences across multiple AI companies, with Anthropic already implementing it and others planning support within months to a year.
Related Entities
Full Report
IT Home reported on November 1 that OpenAI, Google, Microsoft, and Anthropic have recently announced their adoption of a new standard allowing content publishers to opt out of AI web scraping. Through this control mechanism, publishers can uniformly declare their preferences regarding the use of their content for AI model training across multiple AI companies at once, eliminating the need to set individual rules for each provider.
Proposed by Kevin Kelley, Head of Technology at Anthropic, to the Internet Engineering Task Force (IETF), the standard will be integrated into websites' robots.txt files.
For years, robots.txt files have been used to inform web crawlers which site content is permitted or prohibited from being scraped, covering scenarios such as search indexing, web archiving, and AI model training. Currently, websites must separately declare their content usage preferences for AI training to each individual AI company. Against this backdrop, various AI crawlers continue to scrape website content, feeding web materials into AI model training, leaving content publishers overwhelmed.
IT Home noted that while websites currently need to declare training preferences individually to each company, the new standard aims to simplify this process. Anthropic has already adopted it, with OpenAI planning to implement it within the next few months, and Google and Microsoft expected to support it next year. Meta spokesperson Andy Stone also expressed support for establishing such universal signals but did not specify concrete implementation plans.