# AI Policy Token: the IETF robots.txt Opt-Out Standard October 31, 2026 · 5 min read [NGPilot](https://ngpilot.com) On October 30, a one-day-old IETF draft became the most consequential robots.txt news since Google, Bing and Yahoo unified crawler rules in 2022. `draft-kelley-aipref-training-preference-signal-00`, authored by Anthropic standards lead Kevin Kelley, defines a **policy token**: a way to tell every major AI crawler at once whether your content may be used for model training — with one `robots.txt` group instead of a dozen per-vendor blocks. Press reports (The Verge, Search Engine Roundtable) say OpenAI, Google, Microsoft and Anthropic have announced they will adopt the mechanism. Meta voiced support without committing. If your `robots.txt` still names AI crawlers one by one, this draft is the exit from that treadmill. ## The whole mechanism is four lines[​](#the-whole-mechanism-is-four-lines "Direct link to The whole mechanism is four lines") ``` User-agent: __usage_ai-training Disallow: / Allow: /press-kit/ ``` That group expresses a preference **against** AI training for the whole site, **except** anything under `/press-kit/`, where the preference flips to **in favor**. There is no new directive, no new file format, no new grammar. The draft reuses RFC 9309 — the Robots Exclusion Protocol — wholesale: same path language, same longest-match precedence, same group structure. The only invention is a product token that no crawler sends in its `User-Agent` header: `__usage_ai-training`. The double-underscore prefix is deliberate. Every deployed robots.txt parser on the internet already handles unknown product tokens by ignoring their groups — that is what makes backward compatibility free. Legacy parsers skip the group; policy-aware consumers read it. ## Three semantic rules worth internalizing[​](#three-semantic-rules-worth-internalizing "Direct link to Three semantic rules worth internalizing") **1. Silence is not consent — and now it is not refusal either.** A wildcard `User-agent: *` block does *not* express a training preference, and neither does the absence of the token. If no `__usage_ai-training` group exists, you have expressed no preference at all. The draft makes the difference between "I block all crawlers" (a traffic decision) and "I decline training use" (a usage decision) explicit for the first time. **2. It does not block crawling.** This is the part publishers most often get wrong. The policy token is not an exclusion mechanism — GPTBot, ClaudeBot and friends will still fetch your pages unless you block them with their own product tokens. The token governs *use of what was fetched*. For visibility, that distinction is everything: search and answer engines that rank or cite your content are explicitly outside the definition of AI training (the aipref vocabulary limits training to "modify the learned parameters of a generative AI model" — classification, ranking and scoring are excluded). **3. Your existing per-vendor lines stay.** Consumers must keep interpreting their own product tokens exactly as before. If you already allowlist `GPTBot` or disallow `Google-Extended`, those rules still apply as crawl rules. And the reverse: those proprietary tokens do *not* count as a standardized preference — the draft says consumers must not interpret them as such. The token and the crawler blocks are orthogonal signals that happen to live in the same file. ## What it is not[​](#what-it-is-not "Direct link to What it is not") It is not enforcement. The draft is a preference signal — the robots.txt of AI usage, not a paywall. It sits in deliberate contrast to two louder approaches: Cloudflare's default-blocking of AI crawlers, and RSL-style paid-licensing schemes. The token path bets on the opposite trade: standardized, self-service, zero infrastructure, honored voluntarily — which is exactly why four AI vendors could announce support within a day. ## What we did: adopted it the same week[​](#what-we-did-adopted-it-the-same-week "Direct link to What we did: adopted it the same week") We run ngpilot.com as a docs-and-content site for our Atlassian Marketplace apps, and our AI-visibility stance has been consistent for months: explicit `Allow` groups for the search and assistant crawlers, `llms.txt`, and a machine-readable app catalog. The policy token is the standardized expression of the position we already held, so our `robots.txt` now leads with: ``` User-agent: __usage_ai-training Allow: / ``` Explicit allow, not silence. For a site whose growth strategy is being found and cited by AI systems, having our documentation in training corpora is a feature. If your calculus differs — original research, photography, paywalled work — the same four lines with `Disallow` express that just as cleanly. The point is not which side you pick; it is that a one-line, path-capable, cross-vendor declaration now exists, and silence is the only stance that requires no decision. ## Publisher checklist[​](#publisher-checklist "Direct link to Publisher checklist") 1. **Pick a default stance** (allow, disallow, or intentional silence) before a crawler vendor ships support and the choice gets made for you by inattention. 2. **Use paths where it matters.** `Disallow` site-wide with `Allow` for the pages you want in training data is the pattern the draft was designed around. 3. **Keep your crawler rules separate.** Traffic management and usage preferences are independent — don't delete your per-bot lines. 4. **Expect the vocabulary to grow.** Only `__usage_ai-training` is defined today; the aipref working group's vocabulary draft is where future tokens (other usage categories) will land. ## Status caveats, stated plainly[​](#status-caveats-stated-plainly "Direct link to Status caveats, stated plainly") This is a `-00` individual draft, one day old, in a working group whose core documents are still evolving. "Will adopt" is a press-release commitment, not shipped code — no crawler implements the token today. The IETF process can reshape or stall it. We published our group anyway because the cost is four lines, the backward compatibility is mathematically guaranteed by how robots.txt parsers already work, and if the standard lands, day-one publishers set the reference examples. If it stalls, we have lost nothing. Further reading: the [draft itself](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-00), the [aipref working group](https://datatracker.ietf.org/wg/aipref/documents/), and [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309). ## Related guides * [How to Create Mermaid Diagrams in Jira](/blog/how-to-create-mermaid-diagrams-jira.md) * [Export Jira Issues to Excel or CSV](/blog/export-jira-issues-to-excel.md) * [Bulk Download Jira Attachments](/blog/bulk-download-jira-attachments.md) * [How to Migrate Confluence Content](/blog/how-to-migrate-confluence-content.md) * [How to Add a Block Quote in Confluence](/blog/block-quote-confluence.md) Explore NGPILOT[Browse Solutions](/solutions.md)[All Apps](/apps.md)[All Posts](/blog/archive.md)[Atlassian Marketplace →](https://marketplace.atlassian.com/vendors/1226848/?utm_source=ngpilot.com\&utm_medium=blog\&utm_campaign=hub-links) **Tags:** * [seo](/blog/tags/seo.md) * [ai-crawlers](/blog/tags/ai-crawlers.md) * [robots-txt](/blog/tags/robots-txt.md) * [geo](/blog/tags/geo.md) * [ietf](/blog/tags/ietf.md)