feat: agent ready docs (#410)

Summary

Brings the docs site to the public agent-readiness bar (level 0 → level 5 on isitagentready.com) and ships the runtime support that other react-server applications need to do the same. Closes the gap that any pre-rendered react-server site has today: there is no way for an agent asking for Accept: text/markdown on the canonical URL to receive markdown, because the static layer always wins.

The work splits cleanly into three buckets. The docs site grows a set of agent-facing surfaces (well-known endpoints, link headers, content negotiation, an MCP server). The runtime grows the primitives those surfaces depend on (Accept-aware static deferral across every adapter, header propagation across short-circuiting middlewares, a generic fix for rolldown's CJS-imports-JSON wrapping bug). The same primitives let any other application following this PR's patterns reach the same readiness level with a few hundred lines of glue.

Runtime changes

Accept-aware static deferral

Every adapter in @lazarv/react-server previously gave static files unconditional priority over the worker. That makes browsers fast — and silently breaks content negotiation, because middleware never runs for paths whose pre-rendered HTML happens to match. This PR adds a shared shouldDeferToServer(request) / isHtmlRoute(url) pair under adapters/shared/accept.mjs and applies them at every static-first site so that browser navigation and asset traffic keep the fast path while agent traffic with Accept: text/markdown (or any concrete non-HTML media type) flows to the worker.

The Cloudflare adapter checks both predicates before calling env.ASSETS.fetch. The Node-mode static handler in lib/handlers/static.mjs now defers when it would have served text/html and the client clearly prefers something else — covering Bun, Deno, Docker, Azure Functions, AWS Lambda, Firebase, and the singlefile adapter at once. The Vercel adapter gains a has-conditioned route that runs before { handle: "filesystem" } and routes bare paths whose Accept header omits text/html to the function. The Netlify adapter drops the unconditional preferStatic: true from the function config so the in-process static handler can do the per-request decision; users who don't need content negotiation can opt back into the fast path via adapterOptions.functions.config.preferStatic = true. AWS Lambda's tryServeStatic and Docker's bespoke static-first server gain the same isHtmlRoute && shouldDeferToServer guard.

The deferral predicate is deliberately narrow: it requires the URL to look like an HTML route (no extension, or .html/.htm) and the client to explicitly prefer a concrete non-HTML media type at higher q-value than text/html and */*. Browsers always list text/html and never trigger it; image/CSS/JSON requests are excluded by the URL-extension filter; curl with the default */* is treated as "anything is fine" and gets the static reply.

Header propagation across short-circuiting middlewares

react-server now supports running multiple middlewares in sequence — for example, an agent-discovery middleware that sets a Link header followed by a content-negotiation middleware that returns a Response directly. The old behaviour silently dropped headers set on the HTTP context whenever a later middleware short-circuited. A new shared helper at lib/http/middleware-response.mjs#mergeContextHeaders is now invoked from both lib/start/ssr-handler.mjs and lib/dev/ssr-handler.mjs and merges setHeader / appendHeader / headers() output onto the returned Response (the Response's own headers win on conflict, so middlewares that explicitly set Content-Type or Cache-Control retain authority).

Generic fix for the rolldown CJS-imports-JSON wrapping bug

@lazarv/react-server/mcp pulls in @modelcontextprotocol/sdk, which transitively pulls in Express's CJS dependency tree (statuses, mime-types, finalhandler, http-errors). Several of those packages do var data = require('./codes.json') and then iterate Object.keys(data).forEach(k => data[k].toLowerCase()). The bundler's CJS-from-ESM interop wraps imported JSON as { __esModule: true, default: <getter> } and never unwraps it back, so the iteration crashes on the boolean __esModule with r.toLowerCase is not a function. The bug reproduces in examples/mcp on main and is the reason any react-server project trying to host an MCP server has been broken in production.

Docs site changes

Discovery surface

/robots.txt is now a proper file with a User-agent: * block, a Content-Signal: search=yes, ai-input=yes, ai-train=yes directive, and the sitemap. As an open-source documentation site, react-server.dev wants to be indexed and used by AI tools — the signal is intentionally permissive.

A new (agent-discovery).middleware.mjs serves four well-known endpoints with the right content types: /.well-known/api-catalog (RFC 9727 linkset format), /.well-known/agent-skills/index.json (Agent Skills v0.2 index), /.well-known/agent-skills/react-server/SKILL.md (the canonical skill body, imported via ?raw from the monorepo's skills/react-server/SKILL.md), and /.well-known/mcp/server-card.json. The same middleware sets RFC 8288 Link headers on every documentation page advertising the api-catalog, the MCP endpoint, llms.txt, the sitemap, and the agent-skills index.

A new (content-negotiation).middleware.mjs handles Accept: text/markdown by rewriting requests to the existing /md/[...slug] route. The homepage has no /md/ slug entry, so it returns the canonical llms.txt summary as text/markdown instead — that is the right "what is this site" answer for an agent landing at /. Vary: Accept is set on the response so HTML and markdown caches don't poison each other.

The original (i18n).middleware.mjs was reduced back to pure locale resolution. With three middlewares now living side by side (alphabetically ordered: agent-discovery, content-negotiation, i18n), each file does one thing.

MCP server and version source-of-truth

The docs site now hosts a live MCP server at /mcp, dogfooding @lazarv/react-server/mcp. It exposes a search_docs tool (free-text query against the page index), a read_doc tool (fetches any docs page as markdown via the public URL — works on every runtime including Cloudflare Workers without filesystem access), a templated docs-page resource that lists every page, and an explain-topic prompt that orchestrates the two tools.

A new docs/src/version.mjs strips the react-server/ prefix off the package's namespaced version export, giving callers a clean semver. The /mcp server, the MCP server card, and the agent-skills index now all read from this single source — there is no longer a hardcoded 1.0.0 to drift out of sync with the package.

WebMCP browser tools

A "use client" WebMCP.jsx component mounts once in the root layout and registers search_docs and get_docs_page via navigator.modelContext.registerTool. Any in-browser agent (Claude, Cursor, ChatGPT Atlas, Cloudflare Browser-Use) interacting with a docs page can now search the docs and fetch any page as markdown without scraping HTML.