AI Giants Back New Standard for Unified Scraping Opt-Out

1 reports archived1 independent sourcesupdated 10/31/2026, 04:26:10

Synthesis & Latest Updates

1 Sources Cross-Validated

OpenAI, Google, Microsoft, and Anthropic have agreed to adopt a new IETF-proposed standard allowing websites to use 'policy tokens' in robots.txt to uniformly opt out of AI training scraping across multiple companies. This addresses the inefficiency of sending separate signals to each AI firm. Anthropic has already implemented the signal, while others plan to support it within the coming months or next year.

LATEST/OpenAI, Google, Microsoft, and Anthropic have agreed to adopt a new IETF-proposed standard allowing websites to use 'policy tokens' in robots.txt to uniformly opt out of AI training scraping across multiple companies. This addresses the inefficiency of sending separate signals to each AI firm. Anthropic has already implemented the signal, while others plan to support it within the coming months or next year.

Hourly heat curve

2 fully observed hours

Fewer than 3 fully observed hours; trend not plotted yet.

Coverage timeline

Total 1 reports · Latest first

The Verge AIT2Oct 31, 04:26·68 pts
AI Giants Back New Standard for Unified Scraping Opt-Out
- The new standard uses shared 'policy tokens' in robots.txt based on an IETF draft for cross-vendor control
- Anthropic has already adopted the signal; OpenAI, Google, and Microsoft will follow in coming months or next year
- Microsoft clarified that opting out of AI training does not remove content from Bing search results
View Related Reports & Analysis Original Source