Skip to main content

AI Policy Token: the IETF robots.txt Opt-Out Standard

· 5 min read
NGPilot

On October 30, a one-day-old IETF draft became the most consequential robots.txt news since Google, Bing and Yahoo unified crawler rules in 2022. draft-kelley-aipref-training-preference-signal-00, authored by Anthropic standards lead Kevin Kelley, defines a policy token: a way to tell every major AI crawler at once whether your content may be used for model training — with one robots.txt group instead of a dozen per-vendor blocks.

Press reports (The Verge, Search Engine Roundtable) say OpenAI, Google, Microsoft and Anthropic have announced they will adopt the mechanism. Meta voiced support without committing. If your robots.txt still names AI crawlers one by one, this draft is the exit from that treadmill.

The whole mechanism is four lines​

User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That group expresses a preference against AI training for the whole site, except anything under /press-kit/, where the preference flips to in favor. There is no new directive, no new file format, no new grammar. The draft reuses RFC 9309 — the Robots Exclusion Protocol — wholesale: same path language, same longest-match precedence, same group structure. The only invention is a product token that no crawler sends in its User-Agent header: __usage_ai-training.

The double-underscore prefix is deliberate. Every deployed robots.txt parser on the internet already handles unknown product tokens by ignoring their groups — that is what makes backward compatibility free. Legacy parsers skip the group; policy-aware consumers read it.

Three semantic rules worth internalizing​

1. Silence is not consent — and now it is not refusal either. A wildcard User-agent: * block does not express a training preference, and neither does the absence of the token. If no __usage_ai-training group exists, you have expressed no preference at all. The draft makes the difference between "I block all crawlers" (a traffic decision) and "I decline training use" (a usage decision) explicit for the first time.

2. It does not block crawling. This is the part publishers most often get wrong. The policy token is not an exclusion mechanism — GPTBot, ClaudeBot and friends will still fetch your pages unless you block them with their own product tokens. The token governs use of what was fetched. For visibility, that distinction is everything: search and answer engines that rank or cite your content are explicitly outside the definition of AI training (the aipref vocabulary limits training to "modify the learned parameters of a generative AI model" — classification, ranking and scoring are excluded).

3. Your existing per-vendor lines stay. Consumers must keep interpreting their own product tokens exactly as before. If you already allowlist GPTBot or disallow Google-Extended, those rules still apply as crawl rules. And the reverse: those proprietary tokens do not count as a standardized preference — the draft says consumers must not interpret them as such. The token and the crawler blocks are orthogonal signals that happen to live in the same file.

What it is not​

It is not enforcement. The draft is a preference signal — the robots.txt of AI usage, not a paywall. It sits in deliberate contrast to two louder approaches: Cloudflare's default-blocking of AI crawlers, and RSL-style paid-licensing schemes. The token path bets on the opposite trade: standardized, self-service, zero infrastructure, honored voluntarily — which is exactly why four AI vendors could announce support within a day.

What we did: adopted it the same week​

We run ngpilot.com as a docs-and-content site for our Atlassian Marketplace apps, and our AI-visibility stance has been consistent for months: explicit Allow groups for the search and assistant crawlers, llms.txt, and a machine-readable app catalog. The policy token is the standardized expression of the position we already held, so our robots.txt now leads with:

User-agent: __usage_ai-training
Allow: /

Explicit allow, not silence. For a site whose growth strategy is being found and cited by AI systems, having our documentation in training corpora is a feature. If your calculus differs — original research, photography, paywalled work — the same four lines with Disallow express that just as cleanly. The point is not which side you pick; it is that a one-line, path-capable, cross-vendor declaration now exists, and silence is the only stance that requires no decision.

Publisher checklist​

  1. Pick a default stance (allow, disallow, or intentional silence) before a crawler vendor ships support and the choice gets made for you by inattention.
  2. Use paths where it matters. Disallow site-wide with Allow for the pages you want in training data is the pattern the draft was designed around.
  3. Keep your crawler rules separate. Traffic management and usage preferences are independent — don't delete your per-bot lines.
  4. Expect the vocabulary to grow. Only __usage_ai-training is defined today; the aipref working group's vocabulary draft is where future tokens (other usage categories) will land.

Status caveats, stated plainly​

This is a -00 individual draft, one day old, in a working group whose core documents are still evolving. "Will adopt" is a press-release commitment, not shipped code — no crawler implements the token today. The IETF process can reshape or stall it. We published our group anyway because the cost is four lines, the backward compatibility is mathematically guaranteed by how robots.txt parsers already work, and if the standard lands, day-one publishers set the reference examples. If it stalls, we have lost nothing.

Further reading: the draft itself, the aipref working group, and RFC 9309.

Related guides