AI Policy Token: the IETF robots.txt Opt-Out Standard
· 5 min read
On October 30, a one-day-old IETF draft became the most consequential robots.txt news since Google, Bing and Yahoo unified crawler rules in 2022. draft-kelley-aipref-training-preference-signal-00, authored by Anthropic standards lead Kevin Kelley, defines a policy token: a way to tell every major AI crawler at once whether your content may be used for model training — with one robots.txt group instead of a dozen per-vendor blocks.
Press reports (The Verge, Search Engine Roundtable) say OpenAI, Google, Microsoft and Anthropic have announced they will adopt the mechanism. Meta voiced support without committing. If your robots.txt still names AI crawlers one by one, this draft is the exit from that treadmill.