Agent Discovery for Static Sites: the Full Playbook
Yesterday we wrote about the IETF policy token and adopted it the day it appeared. That was one line of one file. The natural next question: what does it take to make a plain static site — no backend, no API, no workers — fully discoverable to AI agents end to end?
We spent a day finding out. ngpilot.com is Docusaurus on Cloudflare Pages, deployed from git, with zero server-side code. By the end of the day it passed all nine agent-discovery checks of the isitagentready scanner that apply to a content site. This post is the complete playbook: what we added, what each piece costs, and — just as important — what we refused to fake.
The inventory
| Check | What satisfies it | Where it lives |
|---|---|---|
| robots.txt | valid robots.txt | static/robots.txt |
| AI rules in robots.txt | explicit allow groups per crawler | same file |
| Content Signals | Content-Signal declaration | same file |
| api-catalog | RFC 9727 linkset document | /.well-known/api-catalog |
| Link headers | Link: response header on every page | static/_headers |
| agent-skills | discovery index + real SKILL.md artifacts | /.well-known/agent-skills/ |
| WebMCP | navigator.modelContext.registerTool() | a client module |
| markdown negotiation | Accept: text/markdown served same-URL | Pages _worker.js |
| sitemap | standard sitemap | generated |
Total new code: two JSON files, two one-stanza robots.txt additions, one _headers file, ~100 lines of client TypeScript, and a 64-line worker. No infrastructure changes; everything deploys the way the site already does.
robots.txt now carries three kinds of meaning
The file leads with the aipref policy token (__usage_ai-training, from yesterday's post), keeps explicit Allow groups for the assistant and search crawlers, and closes with a Content-Signal declaration:
Content-Signal: ai-train=yes, search=yes, ai-input=yes
Three stanzas, three vocabularies: the aipref token speaks to the four vendors who announced adoption, the crawler groups speak to today's deployed bots, and Content-Signal (draft-romm-aipref-contentsignals) is the Cloudflare-pushed expression of the same policy. They agree — that is the point. A publisher whose declarations contradict each other across vocabularies is telling every consumer something worse than silence.
The api-catalog: describe what exists, nothing else
RFC 9727's well-known api-catalog is a linkset: an anchor plus service-desc (machine-readable spec) and service-doc (human-readable docs). The honest version for a site with no REST API documents the machine interface the site actually has:
{
"linkset": [{
"anchor": "https://ngpilot.com",
"service-desc": [{ "href": "https://ngpilot.com/openapi.json", "type": "application/json" }],
"service-doc": [{ "href": "https://ngpilot.com/llms.txt", "type": "text/plain" }]
}]
}
The openapi.json is a minimal, accurate OpenAPI 3.1 document describing four real read-only GETs: /llms.txt, /sitemap.xml, and the two blog feeds. Every response schema in it matches what the server truly returns. Writing a spec for endpoints that don't exist would be the documentation equivalent of dead links.
Two Cloudflare details worth copying. First, the Link header ships via a _headers file — no code:
/*
Link: </.well-known/api-catalog>; rel="api-catalog", </openapi.json>; rel="service-desc", </llms.txt>; rel="service-doc"
Second, /.well-known/api-catalog is extensionless, and Cloudflare would serve it as application/octet-stream — but a Content-Type rule in the same _headers file overrides that. The RFC asks for application/linkset+json; one indented line delivers it.
agent-skills: publish skills that exist
The agent-skills discovery draft wants an index at /.well-known/agent-skills/index.json with a URL and SHA-256 digest per skill. The digest is the honesty enforcement: it must match the exact bytes served. We published two skills, both curated from documentation that already existed — one on producing Mermaid diagrams in Confluence, one on navigating the app catalog — and computed the digests from the committed files. If an agent verifies the hash, it passes; the digest isn't decoration, it's a claim.
WebMCP: the site as a tool provider
The WebMCP API lets a page register tools with the browser-resident agent (Chrome's editor for agentic browsing). A static site has real capabilities to expose — we registered three:
search_pages— keyword search over/llms.txt, which is already the site's machine indexget_page_markdown— fetches the.mdmirror of any page, the same mirror every page advertises via<link rel="alternate" type="text/markdown">open_page— navigation
Each tool's execute does exactly what its inputSchema says, against artifacts the server genuinely serves. Registration happens in a client module on mount, with an AbortController signal so tools unregister cleanly.
Markdown negotiation: the one piece that needed a worker
Serving .md at <path>.md is easy (the build generates mirrors, and Cloudflare already served them as text/markdown). Serving the same URL as HTML or markdown depending on the Accept header needs origin logic — the one thing a static site doesn't have.
Cloudflare Pages advanced mode provides it: a _worker.js file in the build output takes control of every request. Ours is 64 lines:
if (wantsMarkdown(request)) {
const res = await env.ASSETS.fetch(new Request(mdUrl));
if (res.status === 200) return withSiteHeaders(res, url.pathname);
}
return withSiteHeaders(await env.ASSETS.fetch(request), url.pathname);
wantsMarkdown parses q-values, so browsers (text/html,…,*/*;q=0.8) never see markdown while agents sending Accept: text/markdown never see HTML.
Two gotchas we hit so you don't have to. wrangler pages dev does not run advanced mode — locally, _worker.js is served as a plain asset and nothing can be tested; push and verify against production. And because _headers application through env.ASSETS.fetch() in advanced mode isn't documented as guaranteed, the worker re-applies our two header rules itself (with guards, so nothing doubles).
What we refused to fake
The scanner also checks for an A2A agent card, an MCP server card, auth.md, Web Bot Auth keys, and DNS-AID records. All of those fail for us, deliberately.
Every one of them is metadata promising an interactive endpoint. An agent card says "speak JSON-RPC to this URL". auth.md says "register here". Publishing those for endpoints that don't exist means some real agent will eventually act on the metadata — connect, POST, credential-hunt — and hit a dead end we created on purpose for a score. The failure states are the honest signal: this site is content, not an agent. If we ever ship a real agent or API, the same checks become one-file implementations and we'll turn them on.
That's the general principle the whole day reinforced: agent discovery metadata is a promise, not a decoration. The standards are cheap to adopt precisely because they assume you'll only declare what's true.
Publisher's order of work
- Fix
robots.txtfirst (policy token, crawler groups, Content-Signal) — twenty minutes, zero risk. - Add the Link header and api-catalog — an hour, all static.
- Publish skills you actually have, digests computed, not invented.
- Wire WebMCP tools onto artifacts your site already serves.
- Only if needed: the
_worker.jsnegotiation step — it's the only piece with a moving part, and the only one local tooling can't fully verify.
The whole stack is one afternoon for any Docusaurus/Hugo/Astro site on Cloudflare Pages, and every declaration in it is one a publisher can stand behind — which is the only version of agent discovery worth having.
Further reading: draft-kelley-aipref-training-preference-signal, RFC 9727, Content Signals, WebMCP, agent-skills discovery RFC, and yesterday's post on the policy token itself.