# Agent Discovery for Static Sites: the Full Playbook
October 31, 2026 ·
7 min read
[NGPilot](https://ngpilot.com)
Yesterday we wrote about the IETF policy token and adopted it the day it appeared. That was one line of one file. The natural next question: what does it take to make a **plain static site** — no backend, no API, no workers — fully discoverable to AI agents end to end?
We spent a day finding out. ngpilot.com is Docusaurus on Cloudflare Pages, deployed from git, with zero server-side code. By the end of the day it passed all nine agent-discovery checks of the isitagentready scanner that apply to a content site. This post is the complete playbook: what we added, what each piece costs, and — just as important — what we refused to fake.
## The inventory[](#the-inventory "Direct link to The inventory")
| Check | What satisfies it | Where it lives |
| ---------------------- | ----------------------------------------- | ---------------------------- |
| robots.txt | valid robots.txt | `static/robots.txt` |
| AI rules in robots.txt | explicit allow groups per crawler | same file |
| Content Signals | `Content-Signal` declaration | same file |
| api-catalog | RFC 9727 linkset document | `/.well-known/api-catalog` |
| Link headers | `Link:` response header on every page | `static/_headers` |
| agent-skills | discovery index + real SKILL.md artifacts | `/.well-known/agent-skills/` |
| WebMCP | `navigator.modelContext.registerTool()` | a client module |
| markdown negotiation | `Accept: text/markdown` served same-URL | Pages `_worker.js` |
| sitemap | standard sitemap | generated |
Total new code: two JSON files, two one-stanza robots.txt additions, one `_headers` file, \~100 lines of client TypeScript, and a 64-line worker. No infrastructure changes; everything deploys the way the site already does.
## robots.txt now carries three kinds of meaning[](#robotstxt-now-carries-three-kinds-of-meaning "Direct link to robots.txt now carries three kinds of meaning")
The file leads with the aipref policy token (`__usage_ai-training`, from yesterday's post), keeps explicit `Allow` groups for the assistant and search crawlers, and closes with a Content-Signal declaration:
```
Content-Signal: ai-train=yes, search=yes, ai-input=yes
```
Three stanzas, three vocabularies: the aipref token speaks to the four vendors who announced adoption, the crawler groups speak to today's deployed bots, and Content-Signal (draft-romm-aipref-contentsignals) is the Cloudflare-pushed expression of the same policy. They agree — that is the point. A publisher whose declarations contradict each other across vocabularies is telling every consumer something worse than silence.
## The api-catalog: describe what exists, nothing else[](#the-api-catalog-describe-what-exists-nothing-else "Direct link to The api-catalog: describe what exists, nothing else")
RFC 9727's well-known api-catalog is a linkset: an anchor plus `service-desc` (machine-readable spec) and `service-doc` (human-readable docs). The honest version for a site with no REST API documents the machine interface the site actually has:
```
{
"linkset": [{
"anchor": "https://ngpilot.com",
"service-desc": [{ "href": "https://ngpilot.com/openapi.json", "type": "application/json" }],
"service-doc": [{ "href": "https://ngpilot.com/llms.txt", "type": "text/plain" }]
}]
}
```
The `openapi.json` is a minimal, accurate OpenAPI 3.1 document describing four real read-only GETs: `/llms.txt`, `/sitemap.xml`, and the two blog feeds. Every response schema in it matches what the server truly returns. Writing a spec for endpoints that don't exist would be the documentation equivalent of dead links.
Two Cloudflare details worth copying. First, the Link header ships via a `_headers` file — no code:
```
/*
Link: ; rel="api-catalog", ; rel="service-desc", ; rel="service-doc"
```
Second, `/.well-known/api-catalog` is extensionless, and Cloudflare would serve it as `application/octet-stream` — but a `Content-Type` rule in the same `_headers` file overrides that. The RFC asks for `application/linkset+json`; one indented line delivers it.
## agent-skills: publish skills that exist[](#agent-skills-publish-skills-that-exist "Direct link to agent-skills: publish skills that exist")
The agent-skills discovery draft wants an index at `/.well-known/agent-skills/index.json` with a URL and SHA-256 digest per skill. The digest is the honesty enforcement: it must match the exact bytes served. We published two skills, both curated from documentation that already existed — one on producing Mermaid diagrams in Confluence, one on navigating the app catalog — and computed the digests from the committed files. If an agent verifies the hash, it passes; the digest isn't decoration, it's a claim.
## WebMCP: the site as a tool provider[](#webmcp-the-site-as-a-tool-provider "Direct link to WebMCP: the site as a tool provider")
The WebMCP API lets a page register tools with the browser-resident agent (Chrome's editor for agentic browsing). A static site has real capabilities to expose — we registered three:
* `search_pages` — keyword search over `/llms.txt`, which is already the site's machine index
* `get_page_markdown` — fetches the `.md` mirror of any page, the same mirror every page advertises via ``
* `open_page` — navigation
Each tool's `execute` does exactly what its `inputSchema` says, against artifacts the server genuinely serves. Registration happens in a client module on mount, with an `AbortController` signal so tools unregister cleanly.
## Markdown negotiation: the one piece that needed a worker[](#markdown-negotiation-the-one-piece-that-needed-a-worker "Direct link to Markdown negotiation: the one piece that needed a worker")
Serving `.md` at `.md` is easy (the build generates mirrors, and Cloudflare already served them as `text/markdown`). Serving the **same URL** as HTML or markdown depending on the `Accept` header needs origin logic — the one thing a static site doesn't have.
Cloudflare Pages advanced mode provides it: a `_worker.js` file in the build output takes control of every request. Ours is 64 lines:
```
if (wantsMarkdown(request)) {
const res = await env.ASSETS.fetch(new Request(mdUrl));
if (res.status === 200) return withSiteHeaders(res, url.pathname);
}
return withSiteHeaders(await env.ASSETS.fetch(request), url.pathname);
```
`wantsMarkdown` parses q-values, so browsers (`text/html,…,*/*;q=0.8`) never see markdown while agents sending `Accept: text/markdown` never see HTML.
Two gotchas we hit so you don't have to. **`wrangler pages dev` does not run advanced mode** — locally, `_worker.js` is served as a plain asset and nothing can be tested; push and verify against production. And because `_headers` application through `env.ASSETS.fetch()` in advanced mode isn't documented as guaranteed, the worker re-applies our two header rules itself (with guards, so nothing doubles).
## What we refused to fake[](#what-we-refused-to-fake "Direct link to What we refused to fake")
The scanner also checks for an A2A agent card, an MCP server card, `auth.md`, Web Bot Auth keys, and DNS-AID records. All of those fail for us, deliberately.
Every one of them is **metadata promising an interactive endpoint**. An agent card says "speak JSON-RPC to this URL". `auth.md` says "register here". Publishing those for endpoints that don't exist means some real agent will eventually act on the metadata — connect, POST, credential-hunt — and hit a dead end we created on purpose for a score. The failure states are the honest signal: this site is content, not an agent. If we ever ship a real agent or API, the same checks become one-file implementations and we'll turn them on.
That's the general principle the whole day reinforced: **agent discovery metadata is a promise, not a decoration**. The standards are cheap to adopt precisely because they assume you'll only declare what's true.
## Publisher's order of work[](#publishers-order-of-work "Direct link to Publisher's order of work")
1. Fix `robots.txt` first (policy token, crawler groups, Content-Signal) — twenty minutes, zero risk.
2. Add the Link header and api-catalog — an hour, all static.
3. Publish skills you actually have, digests computed, not invented.
4. Wire WebMCP tools onto artifacts your site already serves.
5. Only if needed: the `_worker.js` negotiation step — it's the only piece with a moving part, and the only one local tooling can't fully verify.
The whole stack is one afternoon for any Docusaurus/Hugo/Astro site on Cloudflare Pages, and every declaration in it is one a publisher can stand behind — which is the only version of agent discovery worth having.
Further reading: [draft-kelley-aipref-training-preference-signal](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-00), [RFC 9727](https://www.rfc-editor.org/rfc/rfc9727), [Content Signals](https://datatracker.ietf.org/doc/draft-romm-aipref-contentsignals/), [WebMCP](https://webmachinelearning.github.io/webmcp/), [agent-skills discovery RFC](https://github.com/cloudflare/agent-skills-discovery-rfc), and yesterday's post on the [policy token itself](/blog/ai-policy-token-robots-txt.md).
## Related guides
* [How to Create Mermaid Diagrams in Jira](/blog/how-to-create-mermaid-diagrams-jira.md)
* [Export Jira Issues to Excel or CSV](/blog/export-jira-issues-to-excel.md)
* [Bulk Download Jira Attachments](/blog/bulk-download-jira-attachments.md)
* [How to Migrate Confluence Content](/blog/how-to-migrate-confluence-content.md)
* [How to Add a Block Quote in Confluence](/blog/block-quote-confluence.md)
Explore NGPILOT[Browse Solutions](/solutions.md)[All Apps](/apps.md)[All Posts](/blog/archive.md)[Atlassian Marketplace →](https://marketplace.atlassian.com/vendors/1226848/?utm_source=ngpilot.com\&utm_medium=blog\&utm_campaign=hub-links)
**Tags:**
* [seo](/blog/tags/seo.md)
* [ai-crawlers](/blog/tags/ai-crawlers.md)
* [geo](/blog/tags/geo.md)
* [cloudflare-pages](/blog/tags/cloudflare-pages.md)
* [webmcp](/blog/tags/webmcp.md)