wiki-formant 0.31.0 → 0.33.0

package/README.md

@@ -170,6 +170,12 @@ process.exit(t.summary());
 
 What stays in your repo is fixtures: which tools you expect, what a good answer from each looks like, and which text surfaces you publish. `payloadBudget` weighs every listed call **and** every read-only tool that takes no required arguments, because the one tool nobody thought to list is the one that answers with 3.3 MB; pass a per-call `maxBytes` for a bulk-export tool that is deliberately large. It also asserts that a JSON answer arrived with `structuredContent`.
 
+`standardVersionSources(t, openapiPaths)` is the `versionCoherence` map every S10 surface serves — both agent-card paths, `mcp.json`, the server card and each path the OpenAPI document answers at — so a suite names only the paths it adds. `t.rpcAt(endpoint, …)` and `t.callAt` drive a second MCP endpoint on the same surface with the same backoff, User-Agent and tally.
+
+```ts
+await versionCoherence(t, server.version, standardVersionSources(t, ['/openapi.json', '/api/openapi.json']), init.result?.serverInfo?.version);
+```
+
 `descriptorChecks(t)` defaults to `S10_DESCRIPTORS`, the six JSON descriptors every surface here serves, and `distinctEtagChecks(t, ['llms.txt', 'llms-index.txt', 'llms-full.txt'])` asserts that depths projecting one corpus carry different ETags — every one present, which a bare Set-size check misses.
 
 ## Markdown twins
@@ -220,6 +226,45 @@ const agg = await prisma.page.aggregate({ _count: true, _max: { updatedAt: true
 corpusValidatorsFrom([depth, agg._count], [agg._max.updatedAt]);
 ```
 
+### Route errors
+
+`json`, `errors` and `handleRoute` are the answers a route handler gives when it is not answering with data, as web-standard `Response`s a Next handler returns as they are. Every error body is `{ error, ...extra }`: one origin had these helpers, one had a second convention with a third copy inside a single route, and two answered with whatever `Response.json` was nearest, so one mistake came back three ways depending on the origin.
+
+```ts
+import { errors, handleRoute, HttpError } from 'wiki-formant/http';
+
+export const GET = (req: Request) => handleRoute(async () => {
+  const q = new URL(req.url).searchParams.get('q');
+  if (!q) return errors.badRequest('q is required');
+  if (q.length > 200) throw new HttpError(400, 'q is too long', { code: 'INVALID_PARAMS' });
+  return Response.json(await search(q));
+}, 'Search failed');
+```
+
+A thrown `HttpError` answers with its own status and message. Anything else thrown is logged and answered with a 500 carrying only `errorMsg`, so an exception's text — a database error naming a table — never reaches the caller. `errors.tooManyRequests(sec)` states `Retry-After`.
+
+## Response headers
+
+`wiki-formant/headers` holds the six headers every route sends and the Content-Security-Policy they share, for `next.config.ts` to import:
+
+```ts
+import { contentSecurityPolicy, securityHeaders } from 'wiki-formant/headers';
+
+const csp = contentSecurityPolicy({ 'connect-src': ["'self'", 'https://*.radixdlt.com'] });
+
+const nextConfig = {
+  async headers() {
+    return [{ source: '/:path*', headers: securityHeaders(process.env.NODE_ENV === 'production' ? csp : null) }];
+  },
+};
+```
+
+Four configs had restated these by hand on the grounds that a config could not import an ESM-only package. It could not because no subpath had a `default` condition, and Next resolves a config's imports the way `require` does; every subpath has one now, and Node 24 loads the module there unchanged.
+
+`contentSecurityPolicy` starts from `BASE_CSP`, the policy all four origins shared line for line, and each directive passed replaces its default rather than appending, so the call site reads as the whole list. An empty list drops a directive.
+
+The default `frame-src` is `frameSources(DEFAULT_IFRAME_HOSTS)` — the same list the sanitiser's `iframeHosts` defaults to. The sanitiser's comment had asked every config to keep the two matched by hand, and one did not: its policy lacked the bare `youtube.com` and `youtube-nocookie.com` its sanitiser kept, so those embeds survived cleaning and then rendered blank. A site with its own host list passes it to both.
+
 ## Pagination and versioning
 
 `parsePagination` clamps `page ≥ 1` and `pageSize` to 1–100; `paginatedResponse` always carries `totalPages`. Reshaping that response is a breaking change to every client that pages, which is why it lives here rather than being re-typed per repo.
@@ -333,10 +378,10 @@ group. Both copies were byte-identical across three repos, and they were
 different lists.
 
 ```ts
-import { detectAiBot, aiCrawlerRules } from 'wiki-formant/crawlers';
+import { trackAiBot, crawlerRules } from 'wiki-formant/crawlers';
 
-const bot = detectAiBot(request.headers.get('user-agent'));   // label, or null
-const rules = aiCrawlerRules({ allow: '/', disallow, aiAllow });
+trackAiBot(request, event, () => import('@/lib/track'));   // in proxy.ts: label, or null
+const rules = crawlerRules({ allow: '/', disallow, aiAllow });
 ```
 
 Only one direction of that difference was deliberate: `Applebot-Extended` never
@@ -347,7 +392,11 @@ and a crawler obeys only its most-specific matching group, so an agent with no
 group of its own falls through to `*`. Three wikis were measuring five crawlers
 they had never addressed.
 
-Every group also allows `AGENT_SURFACE_PATHS` — `/api/mcp`, the three `llms` exports, `/openapi.json`, `/.well-known/` — without being told. `aiAllow` is what an origin serves beyond that.
+Every agent group also allows `AGENT_SURFACE_PATHS` — `/api/mcp`, the three `llms` exports, `/openapi.json`, `/.well-known/` — without being told. `aiAllow` is what an origin serves beyond that.
+
+`crawlerRules` returns every group robots.txt needs: the wildcard, one per AI crawler, and one each for `SEARCH_ENGINES`. Search engines get the HTML site and none of its twins — `/*.md$` always, plus `searchDisallow` — because the `*` group keeps the machine surface open to agents, and for Google a twin is a page it already has. Two origins wrote that group by hand and the third never did, so Google crawled its twins. `RSC_DISALLOW`, the `<Link>` prefetch payload, is refused in every group.
+
+`trackAiBot(request, event, load)` counts an `AI Bot Visit` inside `event.waitUntil`. `load` is a dynamic import because the tracker pulls in a database client, which a static import would load for every request the proxy sees.
 
 ## Page metadata
 
@@ -366,6 +415,14 @@ The schema.org nodes sit beside it, because they state the same facts to a diffe
 <JsonLd data={articleLd({ headline: title, url, image, published, modified, publisher: { '@id': `${SITE_URL}/#organization` }, citation: citationsFromReferences(refs) })} />
 ```
 
+`sitePageMetadata({ siteName, handle })` binds the three fields Next drops from every page that sets its own card, with `locale` defaulting to `en_US`. `siteGraphLd` is the Organization, WebSite (with its `SearchAction`) and `WebAPI` as one `@graph` for the root layout, every node carrying an `@id`, and `siteRefs(url)` hands out the same ids for a page's `publisher` and `isPartOf` — so no page writes an inline copy of the publisher that can disagree with the layout's.
+
+```ts
+export const card = sitePageMetadata({ siteName: SITE_NAME, handle: '@example' });
+export const SITE = siteRefs(SITE_URL);
+<JsonLd data={siteGraphLd({ name, url: SITE_URL, logo, sameAs, searchUrl, api })} />
+```
+
 ## Revisions
 
 What changed between two versions of a page, and therefore which semver bump to
@@ -540,7 +597,7 @@ A `codeTabs` tab's `code` is source text in every wiki here, and every view trea
 ```ts
 import { createHtmlSanitizer, sanitizeCoreLeaf } from 'wiki-formant/sanitize';
 
-const clean = createHtmlSanitizer({ iframeHosts: FRAME_HOSTS });   // pair with CSP frame-src
+const clean = createHtmlSanitizer({ iframeHosts: FRAME_HOSTS });   // and frameSources(FRAME_HOSTS) in the CSP
 const safe = mapBlockTree(blocks, b => sanitizeCoreLeaf(b, clean), BLOCK_SHAPE);
 ```
 
@@ -644,15 +701,16 @@ its readers asked for. The queries that return zero rows are the valuable ones
 `wiki-formant/passkey` is the door to a site's own admin pages: a passkey and
 nothing else, no account, password or email. `createPasskeyGate({ sql, rpName })`
 returns the whole of it. `gate.route` is the sign-in route,
-`gate.signedIn(cookieValue)` is the check a page makes, and `gate.cookie` is the
-cookie's name. The two tables (`Passkey`, `PasskeyToken`) are stated as Prisma
+`gate.signedIn(cookieValue)` is the check a page makes, `gate.open()` is true
+while no passkey is saved, and `gate.cookie` is the cookie's name. The two tables (`Passkey`, `PasskeyToken`) are stated as Prisma
 models at the top of `src/passkey.ts`, and every token is kept only as its hash.
 `@simplewebauthn/server` is an optional peer.
 
-There is no sign-up. The first passkey is saved from a single-use link that the
-`passkey-invite` bin prints from a machine holding `DATABASE_URL`, so the only
-way in is through someone who can already reach the database. A passkey is bound
-to the host it was saved on, so localhost needs its own.
+There is no sign-up. While no passkey is saved, the page saves the first one
+from whoever opens it, so the owner should do that as soon as the door ships.
+Every passkey after that is saved from a single-use link that the
+`passkey-invite` bin prints from a machine holding `DATABASE_URL`. A passkey is
+bound to the host it was saved on, so localhost needs its own.
 
 ```ts
 // src/lib/admin.ts
@@ -663,7 +721,7 @@ export const POST = gate.route;
 
 // src/app/stats/page.tsx
 if (!(await gate.signedIn((await cookies()).get(gate.cookie)?.value)))
-  return <PasskeyButton endpoint="/api/passkey" invite={invite} />;
+  return <PasskeyButton endpoint="/api/passkey" invite={invite} create={await gate.open()} />;
 const days = statsDays((await searchParams).days); // 1, 7, 30 or 90, else 30
 return <Stats digest={await digest(sql, { days })} days={days} />;
 ```
@@ -676,7 +734,7 @@ ranked list, and `?days=` links between windows. It ships no JavaScript.
 
 ## Base stylesheet
 
-`wiki-formant/base.css` is the layout the package's markup does not work without, and nothing else: columns that stack until there is room, stored tab panels that show one at a time, a copy button pinned to its block's corner and visible on focus and on touch, a stored table's scroll box, the `aria-sort` arrow as a mask over `currentColor`, and the stats page's grid and bars. No colour and no scale, so a design system's own rules override it at equal specificity.
+`wiki-formant/base.css` is the layout the package's markup does not work without, and nothing else: columns that stack until there is room, stored tab panels that show one at a time, a copy button pinned to its block's corner and visible on focus and on touch, a table's scroll box, the breadcrumb row, the `aria-sort` arrow as a mask over `currentColor`, and the stats page's grid and bars. It also carries the shape and motion of the primitives every site re-implemented and forked — `.spinner` (a `currentColor` ring sized by `--spinner-size`), `.skeleton`'s pulse, `.empty-state`, the wrap rule for long tokens in inline `code` — and the reduced-motion guard none of them had. No colour and no scale, so a design system's own rules override it at equal specificity.
 
 ```css
 @import "wiki-formant/base.css" layer(components);
@@ -693,6 +751,12 @@ already re-exported one module twice. The emitted `.d.ts` files are the referenc
 There is no hand-maintained symbol list here, because the one that used to be here
 drifted from them.
 
+`wiki-formant/tsconfig.base.json` is the compiler floor: an app's `tsconfig.json` extends it and keeps only `paths`, `include`, `exclude` and any licensed extra, since tsconfig resolves those against the file that declares them.
+
+```json
+{ "extends": "wiki-formant/tsconfig.base.json", "compilerOptions": { "paths": { "@/*": ["./src/*"] } }, "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", "src/app/.well-known/**/*.ts", ".next/types/**/*.ts", ".next/dev/types/**/*.ts"], "exclude": ["node_modules"] }
+```
+
 `wiki-formant/db` holds `pgPoolConfig(url)`, the options every app here hands to
 `new pg.Pool()`: ten clients on Supabase's 6543 transaction pooler, three on the
 5432 session pooler, whose fifteen-client cap a build's workers share.
@@ -722,12 +786,35 @@ npx analytics-digest --days 30 --handle radixwiki --siblings caper.network,acuiq
 npx analytics-digest --days 7 --gap "Symptom Miss:term"   # adds an event to the search gaps
 ```
 
-`passkey-invite` prints a single-use link that saves a passkey for the page it names, good for a day (`--days n` for longer). Run it the same way:
+`passkey-invite` prints a single-use link that saves another passkey for the page it names, good for a day (`--days n` for longer). Run it the same way:
 
 ```sh
 npx passkey-invite https://caper.network/stats
 ```
 
+`fts-ddl <table>` rebuilds a table's `search_tsv` generated column and its GIN index from the expression `wiki-formant/search` queries, over `DIRECT_URL` (else `DATABASE_URL` on 5432). Run it before declaring the column in `schema.prisma`; the bin's header says why.
+
+```sh
+npx fts-ddl pages
+```
+
+## A reusable workflow
+
+`.github/workflows/publish-mcp-registry.yml` republishes an origin's server to the MCP registry when the version in its `server.json` moves. Each app keeps its own trigger and calls it with its domain:
+
+```yaml
+on:
+  push: { branches: [main], paths: ['server.json'] }
+  workflow_dispatch:
+jobs:
+  publish:
+    uses: tutmoses/wiki-formant/.github/workflows/publish-mcp-registry.yml@main
+    with:
+      domain: example.com
+    secrets:
+      MCP_REGISTRY_PRIVATE_KEY: ${{ secrets.MCP_REGISTRY_PRIVATE_KEY }}
+```
+
 ## Licence
 
 MIT.

package/bin/fts-ddl.mjs

@@ -0,0 +1,79 @@
+#!/usr/bin/env node
+// bin/fts-ddl.mjs – (re)build a table's `search_tsv` generated column and its
+// GIN index, from the same expression `wiki-formant/search` queries.
+//
+// Run from the app's root, where `pg` is installed and `.env` holds the
+// database URL. Two wikis carried this script with nothing between them but
+// the table name.
+//
+//   npx fts-ddl pages
+//   npx fts-ddl wiki_pages
+//
+// The column is rebuilt every run rather than skipped when it exists: a skip
+// is blind to a changed EXPRESSION, and Postgres normalises the stored one, so
+// comparing it to the string can only produce false rebuilds. DROP+ADD in one
+// transaction means no reader sees the table without it. Run this BEFORE
+// declaring the column in schema.prisma, never after:
+//
+//   1. `prisma db push` drops the column if it is not declared
+//      (`Unsupported("tsvector")?`), and drops the index if
+//      `@@index([searchTsv], type: Gin)` is not declared either.
+//   2. Push must never create the column: Prisma emits a plain
+//      `ADD COLUMN search_tsv tsvector` with no generation expression, which
+//      stays empty forever and fails silently.
+//   3. With both declared, push still wants `ALTER COLUMN "search_tsv" DROP
+//      DEFAULT`, and Postgres refuses it (42601, "is a generated column"). That
+//      error is the expected outcome, not a problem to fix.
+//
+// Connection: DIRECT_URL (the session pooler on 5432) when set, else
+// DATABASE_URL moved from 6543 to 5432. Both run DDL; the session port is the
+// one that never hangs on a long statement.
+import pg from 'pg';
+import { searchTsvDdl } from '../dist/search.js';
+
+const table = process.argv[2];
+if (!table || !/^[a-z_][a-z0-9_]*$/.test(table)) {
+  console.error('usage: npx fts-ddl <table>');
+  process.exit(1);
+}
+
+if (!process.env.DIRECT_URL && !process.env.DATABASE_URL) {
+  try {
+    process.loadEnvFile('.env');
+  } catch {
+    // reported below
+  }
+}
+const url = process.env.DIRECT_URL || process.env.DATABASE_URL?.replace(':6543', ':5432');
+if (!url) {
+  console.error('no DIRECT_URL or DATABASE_URL in the environment or ./.env');
+  process.exit(1);
+}
+
+// A bad route hangs rather than erroring without a connect timeout.
+const client = new pg.Client({
+  connectionString: url.replace(/\?.*$/, ''),
+  ssl: { rejectUnauthorized: false },
+  connectionTimeoutMillis: 15000,
+});
+await client.connect();
+const q = s => client.query(s).then(r => r.rows);
+try {
+  const ddl = searchTsvDdl(table);
+  await q('BEGIN');
+  for (const statement of ddl.column) await q(statement);
+  await q('COMMIT');
+  console.log('rebuilt generated column search_tsv');
+  await q(ddl.index);
+  console.log('GIN index present');
+  console.table(await q(`select column_name, data_type, is_generated from information_schema.columns
+    where table_name='${table}' and column_name='search_tsv'`));
+  console.table(await q(`select count(*)::int rows, count(*) filter (where search_tsv is null)::int null_tsv from ${table}`));
+  console.table(await q(`select pg_size_pretty(pg_relation_size('${table}_search_tsv_idx')) index_size`));
+} catch (e) {
+  await q('ROLLBACK').catch(() => {});
+  console.error(e.message);
+  process.exitCode = 1;
+} finally {
+  await client.end();
+}

package/dist/analytics.d.ts

@@ -196,9 +196,9 @@ export interface Tracker {
  * The tracker holds `sql`, so the module that builds it imports the app's
  * database client. A proxy must keep reaching it through a dynamic import
  * inside `waitUntil`, never a static one, or every request the proxy sees
- * loads Prisma:
+ * loads Prisma — `trackAiBot` (`wiki-formant/crawlers`) is that call:
  *
- *   event.waitUntil(import('@/lib/track').then(m => m.trackEvent('AI Bot Visit', url, { bot }, request.headers)));
+ *   trackAiBot(request, event, () => import('@/lib/track'));
  */
 export declare function createTracker({ sql, defer, enabled, siteUrl, }: TrackerOptions): Tracker;
 export {};

package/dist/analytics.js

@@ -272,9 +272,9 @@ export function searchQueryProps({ query, results, surface, }) {
  * The tracker holds `sql`, so the module that builds it imports the app's
  * database client. A proxy must keep reaching it through a dynamic import
  * inside `waitUntil`, never a static one, or every request the proxy sees
- * loads Prisma:
+ * loads Prisma — `trackAiBot` (`wiki-formant/crawlers`) is that call:
  *
- *   event.waitUntil(import('@/lib/track').then(m => m.trackEvent('AI Bot Visit', url, { bot }, request.headers)));
+ *   trackAiBot(request, event, () => import('@/lib/track'));
  */
 export function createTracker({ sql, defer, enabled = typeof process !== 'undefined' && process.env.VERCEL_ENV === 'production', siteUrl = '/', }) {
     const trackEvent = (name, url, props, headers) => enabled ? recordEvent(sql, { name, url, props, headers }) : Promise.resolve();

package/dist/base.css

@@ -46,10 +46,63 @@
 .tab-panel:not(.active) { display: none; }
 [data-tabs]:not([data-tabs-init]) [data-tab-item] ~ [data-tab-item] { display: none; }
 
-/* ---- table scroll boxes (scrollTables) -----------------------------------
-   A stored table wider than its column scrolls inside its own box, at every
-   width, instead of painting past the column. */
+/* ---- table scroll boxes (scrollTables, React tables) -----------------------
+   A table wider than its column scrolls inside its own box, at every width,
+   instead of painting past the column. */
 .table-scroll { overflow-x: auto; }
+.data-table-wrap { overflow-x: auto; }
+
+/* ---- Breadcrumbs ------------------------------------------------------------
+   The trail is a <nav> around an <ol>; the row is the list, not the nav.
+   Three sites restated this row, each a little differently. */
+.breadcrumbs ol {
+  display: flex;
+  flex-wrap: wrap;
+  align-items: center;
+  gap: 0.25rem;
+  margin: 0;
+  padding-left: 0;
+  list-style: none;
+}
+.breadcrumbs .separator { user-select: none; }
+
+/* ---- spinner, skeleton, empty state ----------------------------------------
+   The shape and the motion; colour, size and surface are the site's. The
+   spinner is a ring in currentColor sized by `--spinner-size`, the skeleton a
+   pulse over whatever background the site gives it. Four stylesheets had
+   forked the spinner three ways: 0.6s and 0.8s, currentColor and a hard-coded
+   accent, one with no keyframe in any layer. */
+@keyframes spin { to { transform: rotate(360deg); } }
+@keyframes pulse { 50% { opacity: 0.5; } }
+.spinner {
+  display: inline-block;
+  flex: none;
+  width: var(--spinner-size, 1em);
+  height: var(--spinner-size, 1em);
+  border: 2px solid currentColor;
+  border-top-color: transparent;
+  border-radius: 9999px;
+  animation: spin 0.6s linear infinite;
+}
+.skeleton { animation: pulse 2s cubic-bezier(0.4, 0, 0.6, 1) infinite; }
+.empty-state { padding-block: 3rem; text-align: center; }
+
+/* ---- long tokens ----------------------------------------------------------
+   An address or hash is one word, so its min-content width is the whole
+   string and it runs off a phone. A <pre> keeps scrolling: wrapping a listing
+   is not reading it. */
+:not(pre) > code { overflow-wrap: anywhere; word-break: break-word; }
+
+/* ---- reduced motion ---------------------------------------------------------
+   No site here honoured it. The spinner slows rather than stops, because a
+   still ring no longer says anything is happening; the pulse and smooth
+   scrolling stop. `!important` because a site's own later rule in the same
+   layer would otherwise put the motion back. */
+@media (prefers-reduced-motion: reduce) {
+  html { scroll-behavior: auto !important; }
+  .spinner { animation-duration: 2.4s !important; }
+  .skeleton { animation: none !important; }
+}
 
 /* ---- code copy buttons (addCopyButtons) -----------------------------------
    The script positions the <pre>; this pins the button in its corner. Shown

package/dist/conformance.d.ts

@@ -70,6 +70,14 @@ export interface Tester {
     rpc(method: string, params?: unknown): Promise<Rpc>;
     /** `tools/call` shorthand. */
     call(name: string, args?: Record<string, unknown>): Promise<Rpc>;
+    /**
+     * `rpc` against another MCP endpoint on the same surface — a per-tenant
+     * server beside the main one — with the same backoff, User-Agent and tally.
+     * The one surface with a second endpoint had re-implemented the backoff for
+     * it, without the User-Agent.
+     */
+    rpcAt(endpoint: string, method: string, params?: unknown): Promise<Rpc>;
+    callAt(endpoint: string, name: string, args?: Record<string, unknown>): Promise<Rpc>;
     /** The decoded body of a tool result: parsed JSON, raw text, or the error. */
     payload(r: Rpc): unknown;
     /** Record an assertion and print it as it happens. */
@@ -107,6 +115,13 @@ export declare function versionCoherence(t: Tester, expected: string, sources: R
     url: string;
     at: (json: Record<string, unknown>) => unknown;
 }>, initVersion?: string): Promise<void>;
+/** The version sources every S10 surface serves, for `versionCoherence`. */
+export declare function standardVersionSources(t: Pick<Tester, 'base' | 'endpoint'>, 
+/** Every path the OpenAPI document is served at. */
+openapi?: readonly string[]): Record<string, {
+    url: string;
+    at: (json: Record<string, unknown>) => unknown;
+}>;
 /**
  * Both A2A well-known paths must serve the identical card. v0.3 renamed
  * agent.json to agent-card.json and defined no fallback either way, so an origin

package/dist/conformance.js

@@ -45,10 +45,10 @@ export function createTester(opts) {
     // bounded number of times. This used to say "once" and recurse forever at a
     // fixed 5s, so a suite pointed at a budget it could never get under hung.
     const RETRIES = 4;
-    const rpc = async (method, params) => {
+    const rpcAt = async (url, method, params) => {
         for (let attempt = 1;; attempt++) {
             calls++;
-            const res = await fetch(endpoint, {
+            const res = await fetch(url, {
                 method: 'POST',
                 headers: { 'Content-Type': 'application/json', 'User-Agent': opts.clientName },
                 body: JSON.stringify({ jsonrpc: '2.0', id: calls, method, params }),
@@ -61,11 +61,14 @@ export function createTester(opts) {
             await new Promise(r => setTimeout(r, Math.min(60, Number.isFinite(wait) && wait > 0 ? wait : 5) * 1000));
         }
     };
+    const rpc = (method, params) => rpcAt(endpoint, method, params);
     return {
         base,
         endpoint,
         rpc,
         call: (name, args = {}) => rpc('tools/call', { name, arguments: args }),
+        rpcAt,
+        callAt: (url, name, args = {}) => rpcAt(url, 'tools/call', { name, arguments: args }),
         payload(r) {
             const text = r.result?.content?.[0]?.text;
             if (text == null)
@@ -241,6 +244,22 @@ export async function versionCoherence(t, expected, sources, initVersion) {
     const all = Object.values(found).filter(v => v !== undefined);
     t.check('version coherence', all.every(v => v === expected), `expected=${expected} ` + Object.entries(found).map(([k, v]) => `${k}=${v}`).join(' '));
 }
+/** The version sources every S10 surface serves, for `versionCoherence`. */
+export function standardVersionSources(t, 
+/** Every path the OpenAPI document is served at. */
+openapi = ['/openapi.json', '/.well-known/openapi.json']) {
+    const version = (j) => j.version;
+    return {
+        card: { url: `${t.base}/.well-known/agent-card.json`, at: version },
+        legacyCard: { url: `${t.base}/.well-known/agent.json`, at: version },
+        mcpManifest: { url: `${t.base}/.well-known/mcp.json`, at: version },
+        serverCard: { url: `${t.endpoint}/server-card`, at: version },
+        ...Object.fromEntries(openapi.map(path => [
+            `openapi:${path}`,
+            { url: `${t.base}${path}`, at: (j) => j.info?.version },
+        ])),
+    };
+}
 /**
  * Both A2A well-known paths must serve the identical card. v0.3 renamed
  * agent.json to agent-card.json and defined no fallback either way, so an origin

package/dist/crawlers.d.ts

@@ -1,3 +1,5 @@
+import type { Tracker } from './analytics.js';
+import type { HeaderReader } from './rate-limit.js';
 export interface AiCrawler {
     /** Matched as a substring of the User-Agent, and the robots.txt token. */
     token: string;
@@ -36,16 +38,48 @@ export interface RobotsGroup {
  */
 export declare const AGENT_SURFACE_PATHS: readonly string[];
 /**
- * The wildcard group followed by one group per crawler.
+ * The React Server Components payload a `<Link>` prefetches. A crawler that
+ * renders a page runs those prefetches too — 43% of Googlebot's requests on
+ * one wiki, against 10% for HTML — and nothing that reads robots.txt can use
+ * one. Every group gets it.
+ */
+export declare const RSC_DISALLOW: readonly string[];
+/** The search engines that get the HTML site and none of its twins. */
+export declare const SEARCH_ENGINES: readonly string[];
+/**
+ * Every group robots.txt needs: the wildcard, one per AI crawler, one per
+ * search engine.
  *
  * Every named agent gets an explicit `disallow`. Omitting it is the failure the
  * comment in all three `robots.ts` files warned about and all three then
  * committed for five agents: a named group that disallows nothing grants that
  * agent everything, because it stops matching `*` the moment it matches itself.
+ *
+ * The search-engine groups exist because `*` keeps the machine surface open to
+ * an agent the origin has never heard of, which is right for an agent and wrong
+ * for Google: a `.md` twin or its JSON is a page it already has in another
+ * format. Two origins wrote that group by hand and the third never did, so
+ * Google crawled its twins.
  */
-export declare function aiCrawlerRules(opts: {
+export declare function crawlerRules(opts: {
     allow: string | string[];
     disallow: string | string[];
     /** Beyond `AGENT_SURFACE_PATHS`, which every group gets regardless. */
     aiAllow?: string | string[];
+    /** What search engines are refused beyond `disallow`. `/*.md$` always is. */
+    searchDisallow?: string | string[];
 }): RobotsGroup[];
+/**
+ * Count an AI crawler's visit from a proxy, without holding up its response.
+ *
+ * `load` is a dynamic import of the module that records it, because that
+ * module pulls in a database client, and a static import would load it for
+ * every request the proxy sees, bots or not. Three proxies wrote these lines.
+ * Returns the bot's label, or null.
+ */
+export declare function trackAiBot(request: {
+    url: string;
+    headers: HeaderReader;
+}, event: {
+    waitUntil(promise: Promise<unknown>): void;
+}, load: () => Promise<Pick<Tracker, 'trackEvent'>>): string | null;

package/dist/crawlers.js

@@ -67,15 +67,33 @@ export const AGENT_SURFACE_PATHS = [
     '/.well-known/',
 ];
 /**
- * The wildcard group followed by one group per crawler.
+ * The React Server Components payload a `<Link>` prefetches. A crawler that
+ * renders a page runs those prefetches too — 43% of Googlebot's requests on
+ * one wiki, against 10% for HTML — and nothing that reads robots.txt can use
+ * one. Every group gets it.
+ */
+export const RSC_DISALLOW = ['/*?_rsc=', '/*&_rsc='];
+/** The search engines that get the HTML site and none of its twins. */
+export const SEARCH_ENGINES = ['Googlebot', 'Bingbot'];
+/**
+ * Every group robots.txt needs: the wildcard, one per AI crawler, one per
+ * search engine.
  *
  * Every named agent gets an explicit `disallow`. Omitting it is the failure the
  * comment in all three `robots.ts` files warned about and all three then
  * committed for five agents: a named group that disallows nothing grants that
  * agent everything, because it stops matching `*` the moment it matches itself.
+ *
+ * The search-engine groups exist because `*` keeps the machine surface open to
+ * an agent the origin has never heard of, which is right for an agent and wrong
+ * for Google: a `.md` twin or its JSON is a page it already has in another
+ * format. Two origins wrote that group by hand and the third never did, so
+ * Google crawled its twins.
  */
-export function aiCrawlerRules(opts) {
+export function crawlerRules(opts) {
     const aiAllow = [...new Set([...[opts.aiAllow ?? []].flat(), ...AGENT_SURFACE_PATHS])];
+    const disallow = [...new Set([...[opts.disallow].flat(), ...RSC_DISALLOW])];
+    const searchDisallow = [...new Set([...disallow, '/*.md$', ...[opts.searchDisallow ?? []].flat()])];
     return [
         // `aiAllow` rides on the default group too, not only on the named roster.
         // The agent surface an origin advertises has to be reachable by a caller it
@@ -83,11 +101,22 @@ export function aiCrawlerRules(opts) {
         // their own agent card named `/api/mcp`, so an MCP client that identified
         // honestly as itself was told the endpoint was off limits. The roster is
         // for indexing policy, not for hiding a documented endpoint.
-        { userAgent: '*', allow: [...new Set([...[opts.allow].flat(), ...aiAllow])], disallow: opts.disallow },
-        ...aiCrawlerTokens().map(userAgent => ({
-            userAgent,
-            allow: aiAllow,
-            disallow: opts.disallow,
-        })),
+        { userAgent: '*', allow: [...new Set([...[opts.allow].flat(), ...aiAllow])], disallow },
+        ...aiCrawlerTokens().map(userAgent => ({ userAgent, allow: aiAllow, disallow })),
+        ...SEARCH_ENGINES.map(userAgent => ({ userAgent, allow: '/', disallow: searchDisallow })),
     ];
 }
+/**
+ * Count an AI crawler's visit from a proxy, without holding up its response.
+ *
+ * `load` is a dynamic import of the module that records it, because that
+ * module pulls in a database client, and a static import would load it for
+ * every request the proxy sees, bots or not. Three proxies wrote these lines.
+ * Returns the bot's label, or null.
+ */
+export function trackAiBot(request, event, load) {
+    const bot = detectAiBot(request.headers.get('user-agent'));
+    if (bot)
+        event.waitUntil(load().then(m => m.trackEvent('AI Bot Visit', request.url, { bot }, request.headers)));
+    return bot;
+}

package/dist/headers.d.ts

@@ -0,0 +1,35 @@
+/** A header as Next's `headers()` wants it. */
+export interface HeaderEntry {
+    key: string;
+    value: string;
+}
+/**
+ * The embed hosts the editor's iframe, YouTube, tweet and map nodes produce:
+ * the sanitiser's default `iframeHosts` and the default `frame-src`, so an
+ * embed that clears one clears the other.
+ */
+export declare const DEFAULT_IFRAME_HOSTS: readonly string[];
+/** `frame-src` sources for a host list — the one a sanitiser was given. */
+export declare const frameSources: (hosts?: readonly string[]) => string[];
+/** CSP directive names this builder writes, in the order it writes them. */
+export type CspDirective = 'default-src' | 'base-uri' | 'object-src' | 'frame-ancestors' | 'form-action' | 'script-src' | 'style-src' | 'font-src' | 'img-src' | 'media-src' | 'connect-src' | 'frame-src' | 'worker-src' | 'manifest-src';
+/**
+ * The policy all four origins shared line for line. Everything a site adds is
+ * a source list it hands `contentSecurityPolicy`, which replaces the default
+ * for that directive rather than appending, so a reader sees the whole list.
+ */
+export declare const BASE_CSP: Readonly<Partial<Record<CspDirective, readonly string[]>>>;
+/**
+ * The policy string: `BASE_CSP` with each directive in `directives` replacing
+ * its default. An empty list drops the directive.
+ */
+export declare function contentSecurityPolicy(directives?: Partial<Record<CspDirective, readonly string[]>>): string;
+/**
+ * The six headers every route sends, plus the policy when one is given.
+ *
+ * Pass the policy only where it is enforced — production. Next's dev runtime
+ * is inline and `eval`, so every origin here exempts dev, and the switch is the
+ * caller's because only the caller knows what dev means for it. An origin that
+ * sets its policy per request (a nonce) passes none.
+ */
+export declare function securityHeaders(csp?: string | null): HeaderEntry[];