wiki-formant 0.20.0 → 0.22.0

package/README.md

@@ -145,6 +145,12 @@ handler: async (args, ctx) => {
 
 `config.gate` is envelope-level middleware: it may withhold entries before dispatch and merge its own responses back afterwards. A payment gate has to sit there rather than in a handler, because the demand *replaces* the call and the receipt rides on the envelope.
 
+### Tool arguments and the corpus tool
+
+`readArgs` reads a tool's raw arguments as typed, clamped values and records every value it had to override, so a result can say `adjustments: [{ param: 'limit', requested: -4, used: 1, … }]` instead of silently answering a different question. Defaults belong in the tool description; the result echoes only overrides.
+
+`wiki-formant/corpus` is the arithmetic behind a `get_full_corpus` tool: a `sizeOnly` preflight with per-branch and largest-page breakdowns, then page-aligned slices under a `maxChars` budget (`CORPUS_BUDGET`) with `truncated`, `nextSkip` and `clippedPage`. The corpus itself — which rows, how a page becomes a section — is the wiki's; `sliceCorpus` takes the sections it builds.
+
 ## Conformance
 
 `wiki-formant/conformance` is the half of an MCP conformance run that is not about any one server's tools: a JSON-RPC client that backs off on a 429, the transport assertions (CORS preflight, `GET`→405, a notification answering 202 with no body, `-32700`, the batch cap, honest `capabilities`, version negotiation), version coherence across every descriptor a surface publishes, A2A card parity, conditional-GET and `robots` checks, and a pass/fail table with an exit code.
@@ -159,6 +165,8 @@ process.exit(t.summary());
 
 What stays in your repo is fixtures: which tools you expect, what a good answer from each looks like, and which text surfaces you publish. `payloadBudget` weighs every listed call **and** every read-only tool that takes no required arguments, because the one tool nobody thought to list is the one that answers with 3.3 MB; pass a per-call `maxBytes` for a bulk-export tool that is deliberately large. It also asserts that a JSON answer arrived with `structuredContent`.
 
+`descriptorChecks(t)` defaults to `S10_DESCRIPTORS`, the six JSON descriptors every surface here serves, and `distinctEtagChecks(t, ['llms.txt', 'llms-index.txt', 'llms-full.txt'])` asserts that depths projecting one corpus carry different ETags — every one present, which a bare Set-size check misses.
+
 ## Markdown twins
 
 Pass an `etag` and answer `notModified` before rendering: a twin is the single most recrawled URL a page has, so a twin with no validator is a full render on every pass, forever — the same arithmetic that justifies the corpus ETag, applied per page.
@@ -186,6 +194,8 @@ Block trees stay in your app — every project owns its own type set. Give this
 
 The `llms.txt` / `llms-index.txt` / `llms-full.txt` trio are the most-recrawled URLs a wiki serves and the most expensive to render. Without a corpus-derived ETag, every AI crawler pays full price on every pass, forever.
 
+A URL that answers JSON or markdown by `Accept` asks `wantsMarkdown(request)` and sends `VARY_ACCEPT` on both branches. Without the Vary, a shared cache hands one client the other's format.
+
 ```ts
 import { corpusEtag, notModified, textHeaders } from 'wiki-formant/http';
 
@@ -264,6 +274,7 @@ are unit-tested without a DOM.
 `wiki-formant/react` carries `'use client'`, and that is a module-level boundary: anything exported from it hydrates in the consumer's tree whether or not it uses a hook. `wiki-formant/react-server` is the same React, without the directive — for the parts of a wiki that are pure functions of their props and should ship no JavaScript at all.
 
 `FacetBar` renders the rows `createTaxonomy` already produces. This is why the taxonomy exports a rows model rather than markup: the rows could always cross the boundary and, until this subpath existed, the markup could not, so all three wikis hand-rendered it and two put `aria-pressed` on an `<a>`. `Breadcrumbs` renders the trail and its `BreadcrumbList` JSON-LD together, because a trail whose structured data is written somewhere else is a trail that will one day disagree with its own markup — which is the case Google penalises. It takes a `base` origin: structured-data URLs must be absolute and a package cannot know the site.
+`FacetBar` takes a `count` class to render each count as its own element, and `alphaFirst` to lead with the A–Z row. Its JSON-LD goes out through `JsonLd`, which is exported for every other payload a page emits: it re-encodes each `<` as `\u003c`, because these payloads carry authored titles and an authored `</script>` would otherwise close the tag.
 
 `PageNav` is the previous/next pair at the foot of an article — the sequential read the infobox rail's lateral links do not cover. Ordering is the caller's, because it is the one part that is never portable: a wiki's sequence is its section's configured sort, a knowledge base's is a taxonomy walk. Pair it with `adjacentPages` from `wiki-formant/pagination` over a list you already hold — neither wiki needs a query for it, and the two indexed lookups the neighbours used to cost were the reason one of them dropped the control.
 
@@ -308,6 +319,19 @@ and a crawler obeys only its most-specific matching group, so an agent with no
 group of its own falls through to `*`. Three wikis were measuring five crawlers
 they had never addressed.
 
+Every group also allows `AGENT_SURFACE_PATHS` — `/api/mcp`, the three `llms` exports, `/openapi.json`, `/.well-known/` — without being told. `aiAllow` is what an origin serves beyond that.
+
+## Page metadata
+
+`wiki-formant/metadata`'s `pageMetadata` builds a page's canonical, markdown-twin alternate, Open Graph and Twitter card from one input. Next replaces those objects per route segment rather than merging them, and fills a missing twitter card from `openGraph` only when no `twitter` object was inherited — so under a layout that sets its own card, a page that sets one and forgets the other shows the layout's. Two wikis wrote a helper around that, covering different halves.
+
+```ts
+export const generateMetadata = () => ({
+  title,
+  ...pageMetadata({ title, description, url, type: 'article', image: ogImageUrl(title), siteName, handle, markdownTwin: true }),
+});
+```
+
 ## Revisions
 
 What changed between two versions of a page, and therefore which semver bump to
@@ -318,10 +342,7 @@ between the copies; they differed only in what each had learned since.
 const diff = computeRevisionDiff({
   currentVersion: page.version,
   oldContent, newContent, oldTitle, newTitle,
-  containers: b =>
-    b.type === 'infobox' ? [{ path: 'blocks', blocks: b.blocks }]
-    : b.type === 'columns' ? b.columns.map((c, i) => ({ path: `columns.${i}.blocks`, blocks: c.blocks }))
-    : null,
+  // containers: defaults to coreBlockGroups — `columns.i.blocks` and an infobox's `blocks`
 });
 ```
 
@@ -330,9 +351,9 @@ const diff = computeRevisionDiff({
 - **Container keys are never diffed as attributes.** They are walked as their own
   entries, and comparing them here would report an infobox as edited every time
   anything inside it changed — turning a prose edit into a structural bump.
-- **The path segment is a parameter**, because `root.1.columns.0.blocks.2` is
-  what a reviewing UI anchors on and only the consumer knows how its containers
-  are addressed.
+- **The path segment is part of the contract**, because `root.1.columns.0.blocks.2`
+  is what a reviewing UI anchors on. A wiki with a container beyond the core two
+  passes its own `containers` and says how it is addressed.
 
 One copy also built two Maps keyed by a recursive `JSON.stringify` of every
 block, on every save, and never read either one. Matching is by id and always
@@ -355,6 +376,7 @@ carries no build date rather than a fictional one.
 
 ## Licence declarations
 
+
 S10 wants a licence on every surface, and each repo satisfied that by writing the
 same block again. What is genuinely per-project is the *scope* — which half of a
 site the grant covers and what it excludes — so that is the parameter.
@@ -364,6 +386,8 @@ const license = ccBy40({ siteName: 'AcuiQ', siteUrl: SITE_URL });
 const block = licenseBlock({ license, scope: 'The protocol compilation and prose', excludes });
 ```
 
+The same `License` goes to `agentCard({ license, licenseScope })` and to `openApiLicense(license)` for a spec's `info.license`. The three cards and three specs here had each projected it differently, one from a hand-typed name.
+
 ## Rendered-article passes
 
 `wiki-formant/dom` holds what runs against an article element after it is in the document. Not React, so not in `react.tsx`.
@@ -377,12 +401,57 @@ const off = onTweetResize(h => sizeTweetEmbeds(el, h));
 
 `addCopyButton` was **byte-identical** in two BlockRenderers, down to the SVG path data. Its idempotence guard now lives inside the function rather than in a `pre:not(:has(…))` at the call site, where it can be — and was — retyped.
 
+In React, `useArticlePasses(ref, [blocks])` runs the three passes and `useTweetEmbeds(ref, [html])` the embed pair, from `wiki-formant/react`. Three renderers had the effect written out and disagreed on its dependencies: two ran once on mount, so a page swapped in without a remount kept its old tables unsortable.
+
 `activateTabGroups` turns stored `[data-tabs]` markup into a working tab group. The editor persists tabs as nested divs, which is the right thing to store — it survives a markdown twin, a plain HTML render and a reader with JavaScript off, all of which show every tab in order. Making one of them pressable is a reader-side job, and it sits beside the other passes rather than inside a component.
 
 `sortTables` makes the tables stored in article HTML sortable by their headers. They arrive as a string a `dangerouslySetInnerHTML` wrote, so React never sees their rows and cannot sort them. A column is dates if every filled cell starts with one, numbers if every one does, and text otherwise — one stray value makes the whole column text, which beats sorting half of it by one rule and half by another. A third press restores the author's order, which is often chronological or ranked and otherwise needs a reload. Label/value tables and tables with merged cells are left alone. The markup it writes — `aria-sort` on the cell, a `.sort-header` button inside it — is the markup a React sortable header should write too, so both kinds of table draw their arrows from one stylesheet rule.
 
 `TWITTER_ORIGIN` is written down once. It is both the embed host and the allow-list `onTweetResize` checks before believing a posted height, and it had been spelled out at four call sites across two repos. Any page can `postMessage`; only the embed host may size the embed.
 
+## Search
+
+`proseSql`, `searchTsvSql`, `HEADLINE_OPTIONS` and `FTS_RANK_NORMALIZATION` are what a literal tier and a full-text tier must agree on. `searchTsvDdl(table)` is the statements that build the generated `search_tsv` column and its index — dropped and re-added in one transaction every run, because a skip-if-present script is blind to a changed expression.
+
+## Block trees
+
+Every wiki here stores the same two containers the same way — `columns[i].blocks` and an infobox's `blocks` — and between them had hand-written that walk seven times. Bind it once and hand it to every walk:
+
+```ts
+import { coreBlockShape, leafBlocks, mapBlockTree } from 'wiki-formant/blocks';
+
+export const BLOCK_SHAPE = coreBlockShape<Block>();
+mapBlockTree(blocks, processLeaf, BLOCK_SHAPE);
+leafBlocks(blocks, BLOCK_SHAPE.containers);
+```
+
+`computeRevisionDiff` defaults to the same shape, path-addressed, so a repo whose containers are the core two passes none.
+
+The dispatch over a repo's own block union stays in that repo; the bodies come from here. `wiki-formant/text` has a prose body for every core leaf — `statsToText`, `linkGridToText` and `pageListToText` joined the originals when two of the three wikis turned out to extract nothing from them, so their MCP `get_page` answered a hub page as nearly empty.
+
+`createBlockValidator` returns `blockIssues` beside the boolean validators: the same walk, reporting where each failure is.
+
+```ts
+const issues = blockIssues(body.content);
+if (issues.length) return badRequest(`Invalid content: ${describeBlockIssues(issues)}`);
+// Invalid content: [2].columns[0].blocks[1]: malformed linkGrid block
+```
+
+## Sanitising stored HTML
+
+`wiki-formant/sanitize` is the allowlist between an author's saved HTML and a reader's browser. The block views render three HTML fields themselves, and a repo renders `content.text` beside them, so the guard ships with the renderer. `sanitize-html` is an optional peer; run it server-side, on the render path, so it covers rows written before it existed and no sanitiser ships to the client.
+
+```ts
+import { createHtmlSanitizer, sanitizeCoreLeaf } from 'wiki-formant/sanitize';
+
+const clean = createHtmlSanitizer({ iframeHosts: FRAME_HOSTS });   // pair with CSP frame-src
+const safe = mapBlockTree(blocks, b => sanitizeCoreLeaf(b, clean), BLOCK_SHAPE);
+```
+
+The default list is derived from what the editor nodes in `wiki-formant/tiptap` store — the embed wrappers' data attributes, the tab markup `activateTabGroups` reads back, the table classes — plus the presentational SVG subset the infographics pipeline embeds. Extend it with `tags`, `attributes`, `classes` and `schemesByTag` derived from your stored HTML, never from memory: a list written from memory erases content on the first render.
+
+Classes pass by name only. Limiting `style` to layout and paint is worth nothing while `class` is free, because every site's own stylesheet ships `fixed inset-0 z-50`, and those draw a fake prompt over the chrome as well as `position` does.
+
 ## Editor nodes
 
 `wiki-formant/tiptap` carries the custom nodes both wiki editors had written twice: `Iframe`, `YouTube` (the stock extension plus the paste rule it does not ship with), `TwitterEmbed`, `createMapEmbed`, `createCodeBlock` and `createTabs`. The four `@tiptap/*` packages are optional peers, so a consumer taking only the taxonomy still installs a package with no runtime dependencies.
@@ -399,6 +468,14 @@ The ones that take config take it because that is exactly where the two copies d
 
 `createTabs` returns `TabGroup` and `TabItem` together: `tabGroup`'s content expression is `tabItem+`, so registering one without the other leaves a node type the schema cannot satisfy. A pasted short map link inserts immediately with `about:blank` and swaps its `src` when the redirect resolves — pasting must not block on a network hop, and the node has to exist for the reader to see anything happen.
 
+`createHeadingIds({ slug })` decorates each heading in the editor with the id its published copy will carry, through `uniqueHeadingId` — the dedupe `injectHeadingIds` uses — so a rail listing headings mid-edit links to the anchors readers will get. `uploadImageTo('/api/upload')` is the `uploadImage` two editors had written identically.
+
+The redirect resolves through the wiki's own route, because the editor cannot read it cross-origin. `resolveMapUrl` is the client half and `resolveMapHandler` the whole route: exact shortener hosts, one hop, an allowlisted landing host, a timeout, and your sign-in check as `authorize`. One of the two copies it replaced matched `goo.gl` as a substring and followed every redirect for anyone who asked.
+
+```ts
+export const GET = resolveMapHandler({ authorize: async () => !!(await currentUser()) });
+```
+
 ## Analytics
 
 Two lanes, one events helper. `mcpCallProps` reads a tool name out of a JSON-RPC

package/dist/block-views.d.ts

@@ -79,19 +79,16 @@ export declare function StatsView({ items, columns, }: {
 /**
  * A maintenance notice.
  *
- * The label and the fallback message arrive as `meta` because they are this
- * wiki's editorial voice — "You can help RADIX Wiki by expanding it" has a name
- * in it. The markup, the `role="note"` and the variant class are shared, and
- * were identical in both repos. `icon` is optional: one wiki sets one, and a
- * component library that hardcoded an icon set would make its consumers install
- * that icon set.
+ * The label is `BANNER_LABELS`, looked up here: three renderers each did the
+ * unknown-variant fallback and the lookup, and one restated all six labels.
+ * The fallback `message` arrives as a prop because it is the wiki's editorial
+ * voice — "You can help RADIX Wiki by expanding it" has a name in it. `icon`
+ * is optional: a component library that hardcoded an icon set would make its
+ * consumers install that icon set.
  */
-export declare function BannerView({ variant, text, meta, icon, }: {
+export declare function BannerView({ variant, text, message, icon, }: {
     variant: string;
     text?: string | null;
-    meta: {
-        label: string;
-        message: string;
-    };
+    message: string;
     icon?: ReactNode;
 }): import("react").JSX.Element;

package/dist/block-views.js

@@ -22,6 +22,7 @@ import { Fragment, useState } from 'react';
 import { Anchor } from './react-server.js';
 import { safeLinkHref } from './validation.js';
 import { cx } from './html.js';
+import { BANNER_LABELS, bannerVariant } from './text.js';
 // ---- codeTabs ---------------------------------------------------------------
 /**
  * Tabbed code samples. Every tab's body stays mounted and the inactive ones are
@@ -106,13 +107,14 @@ export function StatsView({ items, columns = 4, }) {
 /**
  * A maintenance notice.
  *
- * The label and the fallback message arrive as `meta` because they are this
- * wiki's editorial voice — "You can help RADIX Wiki by expanding it" has a name
- * in it. The markup, the `role="note"` and the variant class are shared, and
- * were identical in both repos. `icon` is optional: one wiki sets one, and a
- * component library that hardcoded an icon set would make its consumers install
- * that icon set.
+ * The label is `BANNER_LABELS`, looked up here: three renderers each did the
+ * unknown-variant fallback and the lookup, and one restated all six labels.
+ * The fallback `message` arrives as a prop because it is the wiki's editorial
+ * voice — "You can help RADIX Wiki by expanding it" has a name in it. `icon`
+ * is optional: a component library that hardcoded an icon set would make its
+ * consumers install that icon set.
  */
-export function BannerView({ variant, text, meta, icon, }) {
-    return (_jsxs("div", { className: cx('editorial-banner', `editorial-banner-${variant}`), role: "note", children: [icon, _jsxs("p", { className: "editorial-banner-body", children: [_jsxs("strong", { children: [meta.label, "."] }), " ", text?.trim() || meta.message] })] }));
+export function BannerView({ variant, text, message, icon, }) {
+    const known = bannerVariant(variant);
+    return (_jsxs("div", { className: cx('editorial-banner', `editorial-banner-${known}`), role: "note", children: [icon, _jsxs("p", { className: "editorial-banner-body", children: [_jsxs("strong", { children: [BANNER_LABELS[known], "."] }), " ", text?.trim() || message] })] }));
 }

package/dist/blocks.d.ts

@@ -1,3 +1,4 @@
+import type { BlockGroup } from './revisions.js';
 export interface CodeTab {
     label: string;
     language?: string;
@@ -57,13 +58,42 @@ export interface BlockTreeShape<B> {
     rebuild: (block: B, groups: B[][]) => B;
 }
 /**
- * Every block in the tree, transformed, with the tree's shape preserved.
+ * The core containers' nested groups, each with the path segment that
+ * addresses it: `columns` holds `columns[i].blocks`, `infobox` holds `blocks`.
+ * Every other type is a leaf.
+ *
+ * All three wikis store these two identically, and between them had written
+ * this walk seven times — once per markdown twin, text export, revision diff,
+ * resolver and checker. A walk that forgot a container shipped a code block
+ * unhighlighted or dropped a link from the checker. The path-addressed form is
+ * what `computeRevisionDiff` defaults to.
+ */
+export declare function coreBlockGroups<B extends {
+    type: string;
+}>(block: B): BlockGroup<B>[] | null;
+/**
+ * `coreBlockGroups` as a `BlockTreeShape`: read the groups, and put mapped
+ * groups back where they came from. Bind it once per repo,
+ * `const BLOCK_SHAPE = coreBlockShape<Block>()`, and hand it to every walk.
+ */
+export declare function coreBlockShape<B extends {
+    type: string;
+}>(): BlockTreeShape<B>;
+/** Every leaf in document order, with the containers flattened away. */
+export declare function leafBlocks<B>(blocks: readonly B[], containers: (block: B) => B[][] | null): B[];
+/**
+ * Every leaf in the tree, transformed, with the tree's shape preserved.
  *
  * The three-branch walk this replaces — columns to `columns[].blocks`, infobox
  * to `.blocks`, everything else atomic — was written by hand for heading
  * injection, syntax highlighting, server-side data resolution, feed rendering
  * and twice more besides. Each copy was correct and each had to be found again
  * whenever a container type was added.
+ *
+ * It recurses, so `map` only ever sees leaves, however deep. It used to stop
+ * one level down and hand a nested container to `map` as if it were a leaf —
+ * harmless while every validator here rejects nesting, and a hole the moment a
+ * sanitising pass is the thing being mapped.
  */
 export declare function mapBlockTree<B>(blocks: readonly B[], map: (block: B) => B, shape: BlockTreeShape<B>): B[];
 /** The async twin. Highlighting and data resolution both await per leaf. */

package/dist/blocks.js

@@ -19,36 +19,95 @@ import { htmlToMarkdown, inlineToMarkdown } from './markdown.js';
  */
 export function renderBlockTree(blocks, opts) {
     const { atomic, containers, groupSeparator = '\n\n' } = opts;
-    return blocks
-        .map(block => {
+    const render = (block) => {
         const groups = containers?.(block);
         if (!groups)
             return atomic(block);
         return groups
-            .map(group => group.map(atomic).filter(Boolean).join(groupSeparator))
+            .map(group => group.map(render).filter(Boolean).join(groupSeparator))
             .filter(Boolean)
             .join(groupSeparator);
-    })
+    };
+    return blocks
+        .map(render)
         .filter(Boolean)
         .join('\n\n')
         .replace(/\n{3,}/g, '\n\n')
         .trim();
 }
+const list = (v) => (Array.isArray(v) ? v : []);
 /**
- * Every block in the tree, transformed, with the tree's shape preserved.
+ * The core containers' nested groups, each with the path segment that
+ * addresses it: `columns` holds `columns[i].blocks`, `infobox` holds `blocks`.
+ * Every other type is a leaf.
+ *
+ * All three wikis store these two identically, and between them had written
+ * this walk seven times — once per markdown twin, text export, revision diff,
+ * resolver and checker. A walk that forgot a container shipped a code block
+ * unhighlighted or dropped a link from the checker. The path-addressed form is
+ * what `computeRevisionDiff` defaults to.
+ */
+export function coreBlockGroups(block) {
+    const b = block;
+    if (b.type === 'columns') {
+        return list(b.columns).map((col, i) => ({
+            path: `columns.${i}.blocks`,
+            blocks: list(col?.blocks),
+        }));
+    }
+    if (b.type === 'infobox')
+        return [{ path: 'blocks', blocks: list(b.blocks) }];
+    return null;
+}
+/**
+ * `coreBlockGroups` as a `BlockTreeShape`: read the groups, and put mapped
+ * groups back where they came from. Bind it once per repo,
+ * `const BLOCK_SHAPE = coreBlockShape<Block>()`, and hand it to every walk.
+ */
+export function coreBlockShape() {
+    return {
+        containers: block => coreBlockGroups(block)?.map(group => group.blocks) ?? null,
+        rebuild: (block, groups) => {
+            const b = block;
+            if (b.type === 'columns') {
+                return {
+                    ...block,
+                    columns: list(b.columns).map((col, i) => ({ ...col, blocks: groups[i] ?? [] })),
+                };
+            }
+            if (b.type === 'infobox')
+                return { ...block, blocks: groups[0] ?? [] };
+            return block;
+        },
+    };
+}
+/** Every leaf in document order, with the containers flattened away. */
+export function leafBlocks(blocks, containers) {
+    return blocks.flatMap(block => {
+        const groups = containers(block);
+        return groups ? groups.flatMap(group => leafBlocks(group, containers)) : [block];
+    });
+}
+/**
+ * Every leaf in the tree, transformed, with the tree's shape preserved.
  *
  * The three-branch walk this replaces — columns to `columns[].blocks`, infobox
  * to `.blocks`, everything else atomic — was written by hand for heading
  * injection, syntax highlighting, server-side data resolution, feed rendering
  * and twice more besides. Each copy was correct and each had to be found again
  * whenever a container type was added.
+ *
+ * It recurses, so `map` only ever sees leaves, however deep. It used to stop
+ * one level down and hand a nested container to `map` as if it were a leaf —
+ * harmless while every validator here rejects nesting, and a hole the moment a
+ * sanitising pass is the thing being mapped.
  */
 export function mapBlockTree(blocks, map, shape) {
     return blocks.map(block => {
         const groups = shape.containers(block);
         if (!groups)
             return map(block);
-        return shape.rebuild(block, groups.map(group => group.map(map)));
+        return shape.rebuild(block, groups.map(group => mapBlockTree(group, map, shape)));
     });
 }
 /** The async twin. Highlighting and data resolution both await per leaf. */
@@ -57,7 +116,7 @@ export async function mapBlockTreeAsync(blocks, map, shape) {
         const groups = shape.containers(block);
         if (!groups)
             return map(block);
-        const mapped = await Promise.all(groups.map(group => Promise.all(group.map(map))));
+        const mapped = await Promise.all(groups.map(group => mapBlockTreeAsync(group, map, shape)));
         return shape.rebuild(block, mapped);
     }));
 }

package/dist/conformance.d.ts

@@ -134,7 +134,19 @@ export declare function conditionalGetChecks(t: Tester, paths: readonly string[]
  * document moving, which is worse than none. The body-derived ETag is the whole
  * validator, and it is enough.
  */
-export declare function descriptorChecks(t: Tester, paths: readonly string[]): Promise<void>;
+/** The JSON descriptors every S10 origin serves: the two agent-card paths, the OpenAPI pair, the MCP manifest and the server card. */
+export declare const S10_DESCRIPTORS: readonly string[];
+export declare function descriptorChecks(t: Tester, paths?: readonly string[]): Promise<void>;
+/**
+ * Documents that project one corpus at different depths carry different ETags.
+ *
+ * Depths sharing a validator pass every conditional-GET check and still never
+ * move when only one of them changes. Every one must also HAVE an ETag: a Set
+ * of three nulls is size one and fails, but one null among two tags would pass
+ * a size check alone — the copy of this check one repo carried had exactly
+ * that hole.
+ */
+export declare function distinctEtagChecks(t: Tester, paths: readonly string[], label?: string): Promise<void>;
 /**
  * Every tool carries behavioural hints, and the ones that write say so.
  *

package/dist/conformance.js

@@ -312,7 +312,16 @@ export async function conditionalGetChecks(t, paths) {
  * document moving, which is worse than none. The body-derived ETag is the whole
  * validator, and it is enough.
  */
-export async function descriptorChecks(t, paths) {
+/** The JSON descriptors every S10 origin serves: the two agent-card paths, the OpenAPI pair, the MCP manifest and the server card. */
+export const S10_DESCRIPTORS = [
+    '.well-known/agent-card.json',
+    '.well-known/agent.json',
+    '.well-known/openapi.json',
+    'openapi.json',
+    '.well-known/mcp.json',
+    'api/mcp/server-card',
+];
+export async function descriptorChecks(t, paths = S10_DESCRIPTORS) {
     for (const path of paths) {
         const url = path.startsWith('http') ? path : `${t.base}/${path.replace(/^\//, '')}`;
         const label = path.replace(t.base, '');
@@ -326,6 +335,19 @@ export async function descriptorChecks(t, paths) {
         t.check(`${label} states a freshness`, /max-age=\d+/.test(fresh.headers.get('cache-control') ?? ''), `cache-control: ${fresh.headers.get('cache-control')}`);
     }
 }
+/**
+ * Documents that project one corpus at different depths carry different ETags.
+ *
+ * Depths sharing a validator pass every conditional-GET check and still never
+ * move when only one of them changes. Every one must also HAVE an ETag: a Set
+ * of three nulls is size one and fails, but one null among two tags would pass
+ * a size check alone — the copy of this check one repo carried had exactly
+ * that hole.
+ */
+export async function distinctEtagChecks(t, paths, label = 'llms depths have distinct ETags') {
+    const tags = await Promise.all(paths.map(p => fetch(`${t.base}/${p.replace(/^\//, '')}`).then(r => r.headers.get('etag'))));
+    t.check(label, tags.every(Boolean) && new Set(tags).size === paths.length, tags.join(' '));
+}
 /**
  * Every tool carries behavioural hints, and the ones that write say so.
  *

package/dist/corpus.d.ts

@@ -0,0 +1,65 @@
+/** One page as it appears in the document. */
+export interface CorpusSection {
+    /** The page's address, `tag/path/slug`. */
+    path: string;
+    /** Its branch, for the preflight's per-branch breakdown. */
+    tagPath: string;
+    /** Heading, URL, date and body, exactly as the document will carry them. */
+    section: string;
+}
+/** `maxChars` bounds: the default, the floor and the ceiling. Quote them in the tool's schema. */
+export declare const CORPUS_BUDGET: {
+    readonly default: 200000;
+    readonly min: 1000;
+    readonly max: 1000000;
+};
+export interface CorpusSliceOptions {
+    /** The returned document's heading. */
+    title: string;
+    /** Sizes and breakdowns only, no document. */
+    sizeOnly?: boolean;
+    /** First section to include: the previous call's `nextSkip`. */
+    skip?: number;
+    /** Character budget for `document`. */
+    maxChars?: number;
+    /** The preflight's closing advice: what is cheaper than pulling everything. */
+    hint?: string;
+}
+/**
+ * A preflight (`sizeOnly`) or one page-aligned slice of `sections`.
+ *
+ * `truncated` is true whenever content was cut: whole pages left over, or one
+ * page clipped mid-way because it alone exceeds the budget. `nextSkip` appears
+ * only when paging forward can return something; a clipped page is skipped
+ * past, since resuming on it would stall paging forever.
+ */
+export declare function sliceCorpus(sections: readonly CorpusSection[], opts: CorpusSliceOptions): {
+    branches: {
+        pages: number;
+        chars: number;
+        path: string;
+    }[];
+    largestPages: {
+        path: string;
+        chars: number;
+    }[];
+    hint: string;
+    totalPages: number;
+    characters: number;
+    estimatedTokens: number;
+    tokenNote: string;
+} | {
+    document: string;
+    clippedPage?: string | undefined;
+    clippedHint?: string | undefined;
+    nextSkip?: number | undefined;
+    skip: number;
+    includedPages: number;
+    omittedPages: number;
+    returnedCharacters: number;
+    truncated: boolean;
+    totalPages: number;
+    characters: number;
+    estimatedTokens: number;
+    tokenNote: string;
+};

package/dist/corpus.js

@@ -0,0 +1,82 @@
+// corpus.ts — a wiki's whole corpus, sliced to something a tool result can hold.
+//
+// /llms-full.txt can afford to be 3 MB: a fetch streams to disk under an ETag.
+// A tool result lands in a context window, so the MCP twin of that export needs
+// a preflight, a character budget and a page-aligned resume. Two wikis wrote
+// that loop and called the second a copy of the first "field for field". It was
+// not: the fields had different names, one reported `truncated: false` after
+// clipping the last page — while its own prompt told agents to page until
+// `truncated` was false — and the other sized its preflight on page bodies but
+// sliced on whole sections, so the preflight undercounted what it would send.
+//
+// The corpus itself — which rows, in what order, how a page becomes a section —
+// stays with each wiki. This is the arithmetic over the sections it hands in.
+/** `maxChars` bounds: the default, the floor and the ceiling. Quote them in the tool's schema. */
+export const CORPUS_BUDGET = { default: 200_000, min: 1_000, max: 1_000_000 };
+const CLIP_MARKER = '\n\n[…page clipped at maxChars…]';
+/**
+ * A preflight (`sizeOnly`) or one page-aligned slice of `sections`.
+ *
+ * `truncated` is true whenever content was cut: whole pages left over, or one
+ * page clipped mid-way because it alone exceeds the budget. `nextSkip` appears
+ * only when paging forward can return something; a clipped page is skipped
+ * past, since resuming on it would stall paging forever.
+ */
+export function sliceCorpus(sections, opts) {
+    const { title, sizeOnly = false, skip = 0, maxChars = CORPUS_BUDGET.default, hint } = opts;
+    const characters = sections.reduce((n, s) => n + s.section.length, 0);
+    const head = {
+        totalPages: sections.length,
+        characters,
+        estimatedTokens: Math.round(characters / 4),
+        tokenNote: 'estimatedTokens is characters/4, a rough guide only.',
+    };
+    if (sizeOnly) {
+        const branches = new Map();
+        for (const s of sections) {
+            const b = branches.get(s.tagPath) ?? { pages: 0, chars: 0 };
+            branches.set(s.tagPath, { pages: b.pages + 1, chars: b.chars + s.section.length });
+        }
+        return {
+            ...head,
+            branches: [...branches.entries()].map(([path, b]) => ({ path, ...b })).sort((a, b) => b.chars - a.chars),
+            largestPages: sections
+                .map(s => ({ path: s.path, chars: s.section.length }))
+                .sort((a, b) => b.chars - a.chars)
+                .slice(0, 5),
+            hint: hint ?? 'Pull with maxChars, or one branch at a time with tagPath.',
+        };
+    }
+    const parts = [];
+    let used = 0;
+    let index = skip;
+    let clippedPage;
+    for (; index < sections.length; index++) {
+        const s = sections[index];
+        if (used + s.section.length > maxChars) {
+            if (!parts.length) {
+                parts.push(`${s.section.slice(0, maxChars)}${CLIP_MARKER}`);
+                clippedPage = s.path;
+                used = maxChars;
+                index++;
+            }
+            break;
+        }
+        parts.push(s.section);
+        used += s.section.length;
+    }
+    const morePages = index < sections.length;
+    return {
+        ...head,
+        skip,
+        includedPages: parts.length,
+        omittedPages: sections.length - index,
+        returnedCharacters: used,
+        truncated: morePages || clippedPage !== undefined,
+        ...(morePages ? { nextSkip: index } : {}),
+        ...(clippedPage
+            ? { clippedPage, clippedHint: 'This page alone exceeds maxChars; raise maxChars to read the rest of it.' }
+            : {}),
+        document: [`# ${title}\n\n> pages ${skip + 1}-${index} of ${sections.length}`, ...parts].join('\n\n'),
+    };
+}

package/dist/crawlers.d.ts

@@ -29,6 +29,12 @@ export interface RobotsGroup {
     allow?: string | string[];
     disallow?: string | string[];
 }
+/**
+ * The agent surface an S10 origin serves, allowed in every group. All three
+ * `robots.ts` files listed these by hand, and a path missing from one list is
+ * an endpoint the origin advertises and then closes to the callers it named.
+ */
+export declare const AGENT_SURFACE_PATHS: readonly string[];
 /**
  * The wildcard group followed by one group per crawler.
  *
@@ -40,5 +46,6 @@ export interface RobotsGroup {
 export declare function aiCrawlerRules(opts: {
     allow: string | string[];
     disallow: string | string[];
-    aiAllow: string | string[];
+    /** Beyond `AGENT_SURFACE_PATHS`, which every group gets regardless. */
+    aiAllow?: string | string[];
 }): RobotsGroup[];