wiki-formant 0.77.0 → 0.78.0

package/README.md

@@ -260,6 +260,7 @@ The drift checks hold an MCP server to what its own site does, so a tool cannot
 - `roundTrip(t, label, { read, write })`: what a read tool hands out, the write tool takes back unchanged, and the next read matches.
 - `openapiWriteParity(t, [[path, method, tool], …])`: every field an OpenAPI write documents, the matching tool takes.
 - `rolaOriginCheck(t)`: a challenge tool's ROLA recipe names the origin under test.
+- `freshAfterWriteChecks(t, urls, write, { shows?, gone? })`: each URL is read, `write` runs, and each then carries `shows` and not `gone`. It catches a twin or `llms.txt` the CDN keeps past the save that changed it, which every check that reads a surface once passes. Run it against a deployment: with no CDN in front it passes trivially.
 
 `standardVersionSources(t, openapiPaths)` is the `versionCoherence` map every S10 surface serves — both agent-card paths, `mcp.json`, the server card and each path the OpenAPI document answers at — so a suite names only the paths it adds. `t.rpcAt(endpoint, …)` and `t.callAt` drive a second MCP endpoint on the same surface with the same backoff, User-Agent and tally.
 
@@ -267,7 +268,7 @@ The drift checks hold an MCP server to what its own site does, so a tool cannot
 await versionCoherence(t, server.version, standardVersionSources(t, ['/openapi.json', '/api/openapi.json']), init.result?.serverInfo?.version);
 ```
 
-`descriptorChecks(t)` defaults to `S10_DESCRIPTORS`, the six JSON descriptors every surface here serves, and `distinctEtagChecks(t, ['llms.txt', 'llms-index.txt', 'llms-full.txt'])` asserts that depths projecting one corpus carry different ETags — every one present, which a bare Set-size check misses.
+`descriptorChecks(t)` defaults to `S10_DESCRIPTORS`, the six JSON descriptors every surface here serves, and `distinctEtagChecks(t, ['llms.txt', 'llms-full.txt'])` asserts that depths projecting one corpus carry different ETags — every one present, which a bare Set-size check misses.
 
 ## Markdown twins
 
@@ -290,13 +291,19 @@ return new Response(
 
 `noindex` sends `X-Robots-Tag: noindex`, for a twin of an HTML page: a search engine that reads it has the page already. A document with no HTML page, an AGENTS.md, leaves it off.
 
+Put `cacheTagHeaders(tags)` in `extra`, with the tags your save path already revalidates. A route handler is held at Vercel's CDN by its `s-maxage` alone, so without `Vercel-Cache-Tag` the twin outlives every save by its edge window while the ISR page beside it is fresh. The same goes for `llms.txt`, a search index and a sitemap: any text an agent reads back after writing.
+
+```ts
+headers: markdownHeaders(lastModified, { etag, noindex: true, extra: { ...VARY_ACCEPT, ...cacheTagHeaders(pageTag(path)) } })
+```
+
 Block trees stay in your app — every project owns its own type set. Give this module HTML and it gives you markdown.
 
 > A trap worth naming: if you serve twins via a rewrite, Next drops the destination query string. The rewrite must carry the `.md` extension through to the destination path, or the twin silently serves JSON.
 
 ## Conditional GET
 
-The `llms.txt` / `llms-index.txt` / `llms-full.txt` trio are the most-recrawled URLs a wiki serves and the most expensive to render. Without a corpus-derived ETag, every AI crawler pays full price on every pass, forever.
+`llms.txt` and the markdown twins are the most-recrawled URLs a wiki serves, and a corpus export the most expensive to render. Without a corpus-derived ETag, every AI crawler pays full price on every pass, forever.
 
 A URL that answers JSON or markdown by `Accept` asks `wantsMarkdown(request)` and sends `VARY_ACCEPT` on both branches. Without the Vary, a shared cache hands one client the other's format.
 
@@ -497,7 +504,7 @@ and a crawler obeys only its most-specific matching group, so an agent with no
 group of its own falls through to `*`. Three wikis were measuring five crawlers
 they had never addressed.
 
-Every agent group also allows the agent surface — `/api/mcp`, the three `llms` exports, `/openapi.json`, `/.well-known/` — without being told. `aiAllow` is what an origin serves beyond that.
+Every agent group also allows the agent surface — `/api/mcp`, `/llms.txt`, `/openapi.json`, `/.well-known/` — without being told. `aiAllow` is what an origin serves beyond that.
 
 `crawlerRules` returns every group robots.txt needs: the wildcard, one per AI crawler, and one for each search engine. Search engines get the HTML site and none of its twins — `/*.md$` always, plus `searchDisallow` — because the `*` group keeps the machine surface open to agents, and for Google a twin is a page it already has. Two origins wrote that group by hand and the third never did, so Google crawled its twins. The `<Link>` prefetch payload (`?_rsc=`) is refused in every group.
 
@@ -761,7 +768,7 @@ const safe = mapBlockTree(blocks, b => sanitizeCoreLeaf(b, clean), BLOCK_SHAPE);
 
 The default list is derived from what the editor nodes in `wiki-formant/tiptap` store — the embed wrappers' data attributes, the tab markup `activateTabGroups` reads back, the table classes — plus the presentational SVG subset the infographics pipeline embeds. Extend it with `tags`, `attributes`, `classes` and `schemesByTag` derived from your stored HTML, never from memory: a list written from memory erases content on the first render.
 
-A YouTube iframe comes out as the `YouTube` node's markup, `<div data-youtube-video><iframe …></iframe></div>`, whoever wrote it: an agent, an import, a pasted embed code. The editor then opens it as that node, and `base.css` draws it as the same full-width 16:9 player as one the editor wrote, as it does a map embed. Two wikis had drawn that player in their own stylesheets and a third drew any iframe its own way; corners, borders and the editor's selection ring stay the site's.
+A YouTube iframe comes out as the `YouTube` node's markup, `<div data-youtube-video><iframe …></iframe></div>`, whoever wrote it: an agent, an import, a pasted embed code. The editor then opens it as that node, and `base.css` draws it as the same full-width 16:9 player as one the editor wrote, as it does a map embed. Two wikis had drawn that player in their own stylesheets and a third drew any iframe its own way; spacing, corners, borders and the editor's selection ring stay the site's.
 
 Classes pass by name only. Limiting `style` to layout and paint is worth nothing while `class` is free, because every site's own stylesheet ships `fixed inset-0 z-50`, and those draw a fake prompt over the chrome as well as `position` does.
 

package/dist/base.css

@@ -434,13 +434,11 @@ pre:hover > .code-copy-btn,
 /* ---- embeds ------------------------------------------------------------------
    The players the `wiki-formant/tiptap` nodes store, filling their column at a
    screen's proportions. The node writes a width and height onto the iframe,
-   which `aspect-ratio` with `height: auto` overrides. Corners, borders and the
-   editor's selection ring are the site's. */
-:is(div[data-youtube-video], .map-embed) { margin: 1em 0; }
+   which `aspect-ratio` with `height: auto` overrides. Spacing, corners,
+   borders and the editor's selection ring are the site's. */
 :is(div[data-youtube-video], .map-embed) iframe {
   display: block;
   width: 100%;
   height: auto;
   aspect-ratio: 16 / 9;
-  border: 0;
 }

package/dist/conformance.d.ts

@@ -218,6 +218,22 @@ export declare function rowParity(t: Tester, label: string, mcp: unknown, rest:
  * adds where the page's own blocks open with no h1.
  */
 export declare function twinParity(t: Tester, label: string, twinUrl: string, content: string): Promise<void>;
+/**
+ * A write reaches what an agent reads back. Each URL is read first, so the CDN
+ * holds a copy, then `write` runs, and each must then carry `shows` and no
+ * longer carry `gone`.
+ *
+ * Every other check here reads a surface once, so a twin cached past the save
+ * that changed it passed all of them: the pages were ISR and fresh, while the
+ * route handlers beside them answered from their edge window for an hour. A
+ * purge takes a moment to reach every region, so each URL gets five seconds,
+ * nothing near an edge window. With no CDN in front, as on a dev server, it
+ * passes trivially; run it against a deployment.
+ */
+export declare function freshAfterWriteChecks(t: Tester, urls: readonly string[], write: () => Promise<unknown>, expect: {
+    shows?: string;
+    gone?: string;
+}): Promise<void>;
 /**
  * What an edit starts from, on any server built on the wiki read tools: rows
  * carry the id a pageList block takes, `get_page` with `blocks: true` returns

package/dist/conformance.js

@@ -538,6 +538,35 @@ export async function twinParity(t, label, twinUrl, content) {
         body = body.replace(/^# [^\n]*\n+/, '');
     t.check(label, res.ok && !!text && body === text, !res.ok ? `${res.status} ${url}` : !text ? 'nothing to compare: the MCP text is empty' : body === text ? `${text.length} chars` : firstDifference(text, body));
 }
+/**
+ * A write reaches what an agent reads back. Each URL is read first, so the CDN
+ * holds a copy, then `write` runs, and each must then carry `shows` and no
+ * longer carry `gone`.
+ *
+ * Every other check here reads a surface once, so a twin cached past the save
+ * that changed it passed all of them: the pages were ISR and fresh, while the
+ * route handlers beside them answered from their edge window for an hour. A
+ * purge takes a moment to reach every region, so each URL gets five seconds,
+ * nothing near an edge window. With no CDN in front, as on a dev server, it
+ * passes trivially; run it against a deployment.
+ */
+export async function freshAfterWriteChecks(t, urls, write, expect) {
+    const at = (u) => (u.startsWith('http') ? u : `${t.base}/${u.replace(/^\//, '')}`);
+    const fresh = (body) => (!expect.shows || body.includes(expect.shows)) && (!expect.gone || !body.includes(expect.gone));
+    await Promise.all(urls.map(u => fetch(at(u)).then(r => r.text())));
+    await write();
+    for (const u of urls) {
+        let res = null;
+        let body = '';
+        for (let i = 0; i < 5 && !(res?.ok && fresh(body)); i++) {
+            if (i)
+                await new Promise(r => setTimeout(r, 1000));
+            res = await fetch(at(u));
+            body = await res.text();
+        }
+        t.check(`${u.replace(t.base, '')} after a write`, (!!expect.shows || !!expect.gone) && !!res?.ok && fresh(body), !expect.shows && !expect.gone ? 'nothing to compare: neither shows nor gone' : `${res?.status} ${res?.headers.get('x-vercel-cache') ?? ''} ${fresh(body) ? 'fresh' : 'the copy from before the write'}`);
+    }
+}
 /**
  * What an edit starts from, on any server built on the wiki read tools: rows
  * carry the id a pageList block takes, `get_page` with `blocks: true` returns

package/dist/crawlers.js

@@ -63,8 +63,6 @@ export function detectAiBot(userAgent) {
 export const AGENT_SURFACE_PATHS = [
     '/api/mcp',
     '/llms.txt',
-    '/llms-index.txt',
-    '/llms-full.txt',
     '/openapi.json',
     '/.well-known/',
 ];

package/dist/http.d.ts

@@ -62,6 +62,20 @@ export declare function markdownHeaders(lastModified?: string | null, opts?: {
     noindex?: boolean;
     extra?: Record<string, string>;
 }): Record<string, string>;
+/**
+ * The tag that ties a route handler's response to the writes that change it.
+ * Spread it into the response's headers, beside its `Cache-Control`.
+ *
+ * A route handler is held at Vercel's CDN by its `s-maxage` alone: the tags
+ * `revalidateTag` purges are a page's, and nothing names this response among
+ * them. So a twin, llms.txt or a search index outlived every save by its edge
+ * window, and an agent that read a page back after writing it saw the old
+ * words — four origins shipped that, each with ISR pages that were fresh. A
+ * response carrying `Vercel-Cache-Tag` is purged with the tag, so pass the
+ * tags the save path already revalidates. `freshAfterWriteChecks` holds a
+ * deployment to it.
+ */
+export declare function cacheTagHeaders(tags: string | readonly string[]): Record<string, string>;
 /**
  * Whether a request to a URL that serves both JSON and markdown asked for the
  * markdown: `?format=text`, or an Accept naming text/markdown or text/plain.

package/dist/http.js

@@ -1,10 +1,9 @@
 // http.ts — conditional-GET plumbing for text corpus endpoints, and the error
 // answers every route handler gives (at the foot).
 //
-// The three llms depths (llms.txt / llms-index.txt / llms-full.txt) are the
-// most-recrawled URLs a wiki serves and the most expensive to render. With a
-// corpus-derived ETag a recrawl costs a 304 instead of a full corpus build.
-// Without one, every AI crawler pays full price on every pass, forever.
+// llms.txt and the markdown twins are the most-recrawled URLs a wiki serves.
+// With a corpus-derived ETag a recrawl costs a 304 instead of a full corpus
+// build. Without one, every AI crawler pays full price on every pass, forever.
 import { isoDate } from './html.js';
 /** A stable ETag from whatever the corpus revision is (count + newest stamp). */
 export function corpusEtag(parts) {
@@ -125,6 +124,22 @@ export function markdownHeaders(lastModified, opts = {}) {
         ...opts.extra,
     };
 }
+/**
+ * The tag that ties a route handler's response to the writes that change it.
+ * Spread it into the response's headers, beside its `Cache-Control`.
+ *
+ * A route handler is held at Vercel's CDN by its `s-maxage` alone: the tags
+ * `revalidateTag` purges are a page's, and nothing names this response among
+ * them. So a twin, llms.txt or a search index outlived every save by its edge
+ * window, and an agent that read a page back after writing it saw the old
+ * words — four origins shipped that, each with ISR pages that were fresh. A
+ * response carrying `Vercel-Cache-Tag` is purged with the tag, so pass the
+ * tags the save path already revalidates. `freshAfterWriteChecks` holds a
+ * deployment to it.
+ */
+export function cacheTagHeaders(tags) {
+    return { 'Vercel-Cache-Tag': [tags].flat().join(',') };
+}
 /**
  * Whether a request to a URL that serves both JSON and markdown asked for the
  * markdown: `?format=text`, or an Accept naming text/markdown or text/plain.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "wiki-formant",
-  "version": "0.77.0",
+  "version": "0.78.0",
   "description": "The portable half of a wiki: derived taxonomy and facet controls, a version-negotiating MCP transport, markdown twins, block rendering, a rich-text editor engine, conditional-GET plumbing, and the headers and CSP every origin sends.",
   "type": "module",
   "license": "MIT",