wiki-formant 0.83.1 → 0.85.0

package/README.md

@@ -37,9 +37,9 @@ const { items, total } = await store.search('validators', { take: 20 });
 await store.save({ tagPath, slug, content, authorId, message, baseVersion: page.version });
 ```
 
-- **A save is one statement.** The page update and its revision are a data-modifying CTE guarded on the version the save read. A save that lost a race writes nothing and throws a 409 `HttpError`; pass `baseVersion` and a save opened on an old version is refused the same way, instead of erasing the edit in between.
+- **A save is one statement.** The page update and its revision are a data-modifying CTE guarded on the version the save read. A save that lost a race writes nothing and throws a 409 `HttpError`; pass `baseVersion` and a save opened on an old version, or on a page deleted since, is refused the same way, instead of erasing the edit in between or re-creating the page. A rename onto a slug another save took meanwhile is a 409 too.
 - **The bump is the diff's.** Added, removed or moved blocks are `major`, changed prose or title `minor`; `change` overrides it for a script that knows better. A metadata-only save moves no version and writes no revision.
-- **History is diffed on read.** A revision stores the page as it was saved, never a diff, so a script's revision and the editor's carry the same history. `history(pageId, { limit })` reads one row more than it returns, so the oldest entry shown is diffed against its real predecessor.
+- **History is diffed on read.** A revision stores the page as it was saved, never a diff, so a script's revision and the editor's carry the same history. `history(pageId, { limit })` reads one row more than it returns, so the oldest entry shown is diffed against its real predecessor. It and `revision` answer only for a page of the store's own site.
 - **Listings carry a snippet, not the article.** `list`, `recent` and `byIds` drop `content` for one line of it; a category of 200 cards otherwise ships 200 block arrays.
 - **Search is four tiers**: title prefix, title, literal prose, then full text on `search_tsv` with Postgres' own headline as the snippet. Hidden paths are left out unless the search names one; `content: true` keeps each row's blocks.
 - **A path under a hidden one is hidden too**, in every listing, whether or not the tree names it.
@@ -93,7 +93,7 @@ const { items, total } = await searchWiki(reading, 'validators', { page: 1, size
 - Every row carries the page `id` too, which a `pageList` block takes. `get_page` with `blocks: true` returns the stored blocks, the page `id`, its whole metadata and `version`: what an edit starts from. `forEdit` adds what the app's write tool takes beside them, and `authors` names each revision's author in `get_page_history`.
 - `text` should be the serializer the page's markdown twin uses, and `resolve` the resolver its page uses to fill what is read live (a page list's pages): `get_page`, `get_pages` and the default corpus read text through both, so a page reads the same over MCP as at its twin, which `twinParity` checks.
 - `privateHidden` closes the back office to every tool; without it a hidden page is unlisted but readable by path, as its URL is.
-- `metadata`, `label`, `corpus`, `noMatch` and `skills` are the app's: what a row adds, how a category is named, the sections its llms-full.txt is made of, what an empty search says, and the A2A skills.
+- `metadata`, `label`, `corpus`, `noMatch` and `skills` are the app's: what a row adds, how a category is named, the sections `get_full_corpus` slices, what an empty search says, and the A2A skills.
 - Writes stay with each app, because who may write is the one thing no two of them agree on.
 
 ## Taxonomy
@@ -268,7 +268,7 @@ The drift checks hold an MCP server to what its own site does, so a tool cannot
 await versionCoherence(t, server.version, standardVersionSources(t, ['/openapi.json', '/api/openapi.json']), init.result?.serverInfo?.version);
 ```
 
-`descriptorChecks(t)` defaults to `S10_DESCRIPTORS`, the six JSON descriptors every surface here serves, and `distinctEtagChecks(t, ['llms.txt', 'llms-full.txt'])` asserts that depths projecting one corpus carry different ETags — every one present, which a bare Set-size check misses.
+`descriptorChecks(t)` defaults to `S10_DESCRIPTORS`, the six JSON descriptors every surface here serves.
 
 ## Markdown twins
 
@@ -377,7 +377,7 @@ The default `frame-src` is `DEFAULT_IFRAME_HOSTS`, the same list the sanitiser's
 
 `parsePagination` clamps `page ≥ 1` and `pageSize` to 1–100; `paginatedResponse` always carries `totalPages`. Reshaping that response is a breaking change to every client that pages, which is why it lives here rather than being re-typed per repo.
 
-`parseVersion` / `bump` / `compareVersions` handle revision semver tolerantly — a page always has a version, even when the column holds `null` or junk.
+`parseVersion` / `bump` handle revision semver tolerantly — a page always has a version, even when the column holds `null` or junk.
 
 ## React
 
@@ -422,7 +422,7 @@ const { ref, onScroll, at, step, go } = useSnapRow();
 
 `useClickOutside` is fifteen lines and was in all three. Only one had the `offsetParent` check, and without it a container hidden at the current breakpoint still answers outside-clicks — so on a phone, a tap anywhere dismisses the popover the reader is looking at, because the hidden desktop copy got there first.
 
-`useUnsavedChanges(dirty)` asks before an edit with unsaved changes is left: the browser's own prompt on a reload or a closed tab, and the `confirmLeave()` it returns for a Back link or Cancel, which a client-side navigation never sends through `beforeunload`. If a save leaves by a full page load, pass false once it has landed, or the save's own exit asks. `useSaveShortcut(save)` binds ⌘S and Ctrl+S, reading `save` when the key is pressed.
+`useUnsavedChanges(dirty)` asks before an edit with unsaved changes is left: the browser's own prompt on a reload or a closed tab, and the `confirmLeave()` it returns for a Back link or Cancel, which a client-side navigation never sends through `beforeunload`. If a save leaves by a full page load, pass false once it has landed, or the save's own exit asks. `useSaveShortcut(save)` binds ⌘S and Ctrl+S, reading `save` when the key is pressed. `useSearchShortcut(open)` is Wikipedia's "/" to search, ignored while the reader types in a field: `open` focuses the field or opens what holds it, and returns false to leave the key alone, which `focusShown(input)` does for a field hidden at this width.
 
 ### The listbox contract
 
@@ -765,6 +765,14 @@ A `codeTabs` tab's `code` is source text in every wiki here, and every view trea
 
 `relativeTime(then, now, { style })` is `compact` (`3h`), `short` (`3h ago`) or `long` (`3 days ago`, day-grained for pages cached for hours), with `absoluteAfterDays` to hand over to the date itself. `now` is always passed in, never read, so a server render and its hydration agree. `formatDay` is always UTC: a stored day formatted in the server's zone, or the browser's, is the previous day for half the world. It reads `en-US` unless given a `locale`: `formatDay(day, { locale: 'en-GB', month: 'long' })` is `19 September 2026`. Both are in `wiki-formant/freshness`, with `isoDate`.
 
+## Freshness and the maintenance queues
+
+`freshnessBanner(page, nowMs, { maxAgeDays, against })` is the synthetic `outdated` banner a page carries once it has gone unchecked for `maxAgeDays` (180), or null. Its notice asks the reader to re-check the facts against `current sources`; the two Radix wikis pass `against: 'current sources and the live ledger'`, which a shop's wiki has no use for.
+
+`wiki-formant/queues` gathers the signals each page shows into Wikipedia's tracking categories: `maintenanceQueues(pages, { href, required, nowMs, verify, listedIds })` returns the outdated, orphaned, unsourced and missing-facts queues, empty ones dropped, from one pass over the pages' content. Pass every page but the home page and the back office, since an orphan is a page no other page links to. `href` is a page's address as its links name it, `required` the facts its category requires, `verify` how a page is stamped as checked here (it closes the outdated queue's sentence), and `listedIds` the ids a list block names its pages by. `MaintenanceQueues` (`wiki-formant/react-server`) draws them, a heading and a count each, under classes `maintenance-*`; the page around them is the wiki's, `noindex`. All four apps serve one at `/maintenance` (on miow, the owner page).
+
+`Cite` (`wiki-formant/react-server`) is "Cite" in an article's meta line, a `<details>` opening on `citation({ publisher, title, date, url })`: `AcuiQ. "Point codes." October 2, 2026. https://…`.
+
 ## Sanitising stored HTML
 
 `wiki-formant/sanitize` is the allowlist between an author's saved HTML and a reader's browser. The block views render two HTML fields themselves — `linkGrid` descriptions and `references` text — and a repo renders `content.text` beside them, so the guard ships with the renderer. `codeTabs` is not one of them: its code is source. Run it server-side, on the render path, so it covers rows written before it existed and no sanitiser ships to the client.
@@ -1177,7 +1185,7 @@ npx fts-ddl pages
 npx fts-ddl '"WikiPage"'
 ```
 
-`mark-verified` stamps `lastVerifiedAt` on pages a sweep checked and found current, so `freshnessBanner` stops calling them outdated, and `maintenance-log` is the state I/O of a wiki-sweep routine whose memory lives in one page's `metadata.state`: `read`, `write` (the whole state, or `-` for stdin), `compact` and `backlog-add`, which appends under a row lock so another routine's item cannot be lost to a concurrent `write`. Every write compacts and re-renders the page, holding at most as many FLAG FOR A HUMAN items as the page shows; a revision is written at most once a day. `write` refuses a state whose `totalRuns` is below the stored one, which is a state read before another run wrote. Both work on the package's own `WikiPage` and `WikiRevision`. Run them from the app's root, like `passkey-invite`:
+`mark-verified` stamps `lastVerifiedAt` on pages a sweep checked and found current, so `freshnessBanner` stops calling them outdated, and `maintenance-log` is the state I/O of a wiki-sweep routine whose memory lives in one page's `metadata.state`: `read`, `write` (the whole state, or `-` for stdin), `compact` and `backlog-add`, which is how another routine files an item. Every mode holds the row lock from read to write, and `backlog-add` records what it filed in `state.filed`, so a `write` of a state read before the filing merges the item back in rather than dropping it (`lateBacklog`). Only `metadata.state` is written; the page's other metadata is kept. Every write compacts and re-renders the page, holding at most as many FLAG FOR A HUMAN items as the page shows; a revision is written at most once a day. `write` refuses a state whose `totalRuns` is below the stored one, which is a state read before another run wrote. Both work on the package's own `WikiPage` and `WikiRevision`. Run them from the app's root, like `passkey-invite`:
 
 ```sh
 npx mark-verified contents/tech/radix-engine scrypto       # one page

package/bin/maintenance-log.mjs

@@ -26,11 +26,12 @@
 // page in place, because a full copy per run is what made radix-wiki's log
 // over half its revisions table. `write` refuses a state whose totalRuns is
 // below the stored one: that state was read before another run wrote.
-// `backlog-add` is how another routine files an
-// item: `write` replaces the whole state, so an append through it could lose a
-// concurrent run's, and the row lock serialises the two. A revision bumps the
-// page's version a minor.
-import { PINNED, addBacklog, compact, render, staleWrite } from '../dist/maintenance.js';
+// `backlog-add` is how another routine files an item. It records what it filed
+// in `state.filed`, and a `write` whose state was read before the filing merges
+// the item back in, so a run's whole-state write cannot drop it. Every mode
+// reads and writes under the row lock. Only `state` in the page's metadata is
+// written; other keys are kept. A revision bumps the page's version a minor.
+import { PINNED, addBacklog, compact, lateBacklog, render, staleWrite } from '../dist/maintenance.js';
 import { bump } from '../dist/versioning.js';
 import { arg, connect, invalidate } from './_cli.mjs';
 
@@ -48,6 +49,8 @@ if (!['read', 'write', 'compact', 'backlog-add'].includes(mode) || slugAt < 1 ||
 const pages = '"WikiPage"';
 const revisions = '"WikiRevision"';
 const where = `"tagPath" = $1 AND slug = $2 AND "siteId" = $3`;
+/** The new state into the page's metadata, beside whatever else it holds. `lastVerified` was a stamp nothing read. */
+const withState = `(metadata - 'lastVerified') || jsonb_build_object('state', $2::jsonb)`;
 const address = [at.slice(0, slugAt), at.slice(slugAt + 1), arg('--site') ?? ''];
 
 const readStdin = async () => {
@@ -79,14 +82,14 @@ try {
       await q('ROLLBACK');
       console.log('  nothing to add (all duplicates)');
     } else {
-      const next = compact({ ...state, backlog: [...(state.backlog ?? []), ...added] });
+      const next = compact({ ...state, backlog: [...(state.backlog ?? []), ...added], filed: [...(state.filed ?? []), ...added.map(a => a.item)] });
       console.log(`  ${dry ? '[dry] ' : ''}+${added.length} backlog item(s), ${next.backlog.length} open`);
       for (const a of added) console.log(`     ${a.item}`);
       if (dry) await q('ROLLBACK');
       else {
-        await q(`UPDATE ${pages} SET content = $1, metadata = $2, "updatedAt" = now() WHERE id = $3`, [
+        await q(`UPDATE ${pages} SET content = $1, metadata = ${withState}, "updatedAt" = now() WHERE id = $3`, [
           JSON.stringify(render(next, routine)),
-          JSON.stringify({ state: next, lastVerified: today }),
+          JSON.stringify(next),
           page.id,
         ]);
         await q('COMMIT');
@@ -94,15 +97,22 @@ try {
       }
     }
   } else {
-    const [page] = await q(`SELECT * FROM ${pages} WHERE ${where}`, address);
+    const text = mode === 'write' ? (payload === undefined || payload === '-' ? await readStdin() : payload) : '';
+    if (mode === 'write' && !text.trim()) throw new Error('write needs the state as JSON, as an argument or on stdin with `write -`');
+    // Locked from the read to the write, so a `backlog-add` lands before or after, never in between.
+    await q('BEGIN');
+    const [page] = await q(`SELECT * FROM ${pages} WHERE ${where} FOR UPDATE`, address);
     if (!page) throw new Error(`no log page at ${at}`);
-    let input = page.metadata?.state ?? {};
+    const stored = page.metadata?.state ?? {};
+    let input = stored;
     if (mode === 'write') {
-      const text = payload === undefined || payload === '-' ? await readStdin() : payload;
-      if (!text.trim()) throw new Error('write needs the state as JSON, as an argument or on stdin with `write -`');
       input = JSON.parse(text);
-      const stale = staleWrite(page.metadata?.state ?? {}, input);
+      const stale = staleWrite(stored, input);
       if (stale) throw new Error(stale);
+      const late = lateBacklog(stored, input, today);
+      for (const l of late) console.log(`  kept, filed since this state was read: ${l.item}`);
+      const { filed: _, ...rest } = input;
+      input = { ...rest, backlog: [...(input.backlog ?? []), ...late] };
     }
     const state = compact(input);
     const content = render(state, routine);
@@ -130,11 +140,11 @@ try {
         `content ${JSON.stringify(content).length} B${revise ? `, revision v${version}` : ', no revision today'}`,
     );
 
-    if (!dry) {
-      await q('BEGIN');
+    if (dry) await q('ROLLBACK');
+    else {
       await q(
-        `UPDATE ${pages} SET content = $1, metadata = $2, "updatedAt" = now(), version = $4 WHERE id = $3`,
-        [JSON.stringify(content), JSON.stringify({ state, lastVerified: today }), page.id, version],
+        `UPDATE ${pages} SET content = $1, metadata = ${withState}, "updatedAt" = now(), version = $4 WHERE id = $3`,
+        [JSON.stringify(content), JSON.stringify(state), page.id, version],
       );
       if (revise) {
         await q(

package/bin/mark-verified.mjs

@@ -29,8 +29,9 @@ try {
   client = await connect();
   const { rowCount } = all
     ? await client.query(
-        `UPDATE "WikiPage" SET "lastVerifiedAt" = now() WHERE "siteId" = $3 AND ("tagPath" = $1 OR "tagPath" LIKE $2)`,
-        [tagPath, `${tagPath}/%`, site],
+        // starts_with, not LIKE: a `_` or `%` in the path would match other branches.
+        `UPDATE "WikiPage" SET "lastVerifiedAt" = now() WHERE "siteId" = $3 AND ("tagPath" = $1 OR starts_with("tagPath", $2))`,
+        [tagPath, `${tagPath}/`, site],
       )
     : await client.query(
         `UPDATE "WikiPage" SET "lastVerifiedAt" = now() WHERE "siteId" = $3 AND "tagPath" = $1 AND slug = $2`,

package/dist/a2a.js

@@ -10,7 +10,7 @@
 //
 // Both revisions in the field are spoken: v0.3 (`message/send`, a Message with
 // `kind`) and v1.0 (`SendMessage`, a `{ message }` with proto-JSON enums).
-import { McpToolError } from './mcp.js';
+import { MCP_LEGACY_PROTOCOL_VERSION, McpToolError } from './mcp.js';
 const SEND = { 'message/send': '0.3', SendMessage: '1.0' };
 /** Task methods: none can succeed, because no task is ever created. */
 const TASK = new Set(['tasks/get', 'tasks/cancel', 'tasks/resubscribe', 'GetTask', 'CancelTask', 'SubscribeToTask']);
@@ -31,7 +31,7 @@ const UNSUPPORTED = new Set([
     'GetExtendedAgentCard',
 ]);
 /** @internal */
-export const isA2aMethod = (method) => method in SEND || TASK.has(method) || UNSUPPORTED.has(method);
+export const isA2aMethod = (method) => Object.hasOwn(SEND, method) || TASK.has(method) || UNSUPPORTED.has(method);
 const error = (id, code, message) => ({ jsonrpc: '2.0', id, error: { code, message } });
 /** The text of a message's text parts, whichever revision shaped them. */
 function textOf(message) {
@@ -53,14 +53,14 @@ async function reply(text, config) {
         const out = await tool.handler({ [config.answer.argument]: text }, {
             meta: {},
             setMeta: () => { },
-            protocolVersion: '2025-11-25',
+            protocolVersion: MCP_LEGACY_PROTOCOL_VERSION,
         });
         const lead = `${config.serverInfo.name} ${tool.name} for ${JSON.stringify(text)}. For more, call this endpoint over MCP: "initialize", then "tools/list".`;
         if (typeof out === 'string')
             return { text: `${lead}\n\n${out}` };
         // The text twin stays beside the data, as it does on an MCP tool result:
         // a client that reads only text parts still has to be able to read the answer.
-        return { text: `${lead}\n\n${JSON.stringify(out, null, 2)}`, ...(out && typeof out === 'object' && !Array.isArray(out) ? { data: out } : {}) };
+        return { text: `${lead}\n\n${JSON.stringify(out ?? null, null, 2)}`, ...(out && typeof out === 'object' && !Array.isArray(out) ? { data: out } : {}) };
     }
     catch (err) {
         if (err instanceof McpToolError)

package/dist/analytics.js

@@ -134,15 +134,18 @@ export async function collect(request, { sql, defer, site, rateLimit: limit = BE
     if (BOT.test(headers.get('user-agent') ?? '') || !rateLimit(clientKey('beacon', headers), limit).ok)
         return done;
     const body = (await request.json().catch(() => null));
-    const path = typeof body?.path === 'string' ? normalisePath(body.path) : null;
-    if (!body || !path)
+    const raw = typeof body?.path === 'string' ? body.path : null;
+    const path = raw === null ? null : normalisePath(raw);
+    if (!body || raw === null || !path)
         return done;
     const key = site ? await site(request, path) : '';
     if (key === null)
         return done;
+    // The recorders normalise what they are given, so they get the path as sent:
+    // decoded twice, `/100%25-off` was dropped and `/what%3Fnow` cut to `/what`.
     defer(() => typeof body.name === 'string' && body.name
-        ? recordEvent(sql, { name: body.name, props: beaconProps(body.props), url: path, headers, site: key })
-        : recordView(sql, { headers, path, referrer: body.referrer, utm: body.utm, site: key }));
+        ? recordEvent(sql, { name: body.name, props: beaconProps(body.props), url: raw, headers, site: key })
+        : recordView(sql, { headers, path: raw, referrer: body.referrer, utm: body.utm, site: key }));
     return done;
 }
 /** Searches in a wiki's own search box that found nothing (`searchQueryProps` events). */

package/dist/block-views.d.ts

@@ -82,7 +82,13 @@ export declare function PageRefsView({ refs, empty, link: Link, renderItem, }: {
  * through its HTML pipeline, and running it in the other would rewrite anchors
  * that were minted under different rules.
  */
-export declare function ReferencesView({ items, title, processHtml, }: {
+export declare function ReferencesView({ id, items, title, processHtml, }: {
+    /**
+     * The block's id, which keys the heading's: `references-<id>`. Without it the
+     * heading is `references-heading`, which a second references block on the
+     * page would repeat, labelling both sections with the first one's heading.
+     */
+    id?: string;
     items?: readonly (ReferenceItem & {
         id?: string;
     })[];

package/dist/block-views.js

@@ -89,11 +89,12 @@ export function PageRefsView({ refs, empty, link: Link = Anchor, renderItem, })
  * through its HTML pipeline, and running it in the other would rewrite anchors
  * that were minted under different rules.
  */
-export function ReferencesView({ items, title = 'References', processHtml = (html) => html, }) {
+export function ReferencesView({ id, items, title = 'References', processHtml = (html) => html, }) {
     const rows = items ?? [];
     if (!rows.length)
         return null;
-    return (_jsxs("section", { className: "references-block", "aria-labelledby": "references-heading", children: [_jsx("h2", { id: "references-heading", children: title }), _jsx("ol", { className: "references-list", children: rows.map((item, i) => {
+    const headingId = id ? `references-${id}` : 'references-heading';
+    return (_jsxs("section", { className: "references-block", "aria-labelledby": headingId, children: [_jsx("h2", { id: headingId, children: title }), _jsx("ol", { className: "references-list", children: rows.map((item, i) => {
                     const url = safeLinkHref(item.url);
                     return (_jsxs("li", { id: `ref-${i + 1}`, className: "reference-item", children: [_jsx("a", { href: `#cite-${i + 1}`, className: "ref-backlink", "aria-label": "Back to citation", children: "\u2191" }), ' ', _jsx("span", { dangerouslySetInnerHTML: { __html: processHtml(item.text) } }), url && (_jsxs(_Fragment, { children: [' ', _jsx("a", { href: url, target: "_blank", rel: "noopener", className: "reference-link", "aria-label": "Open source", children: "\u2197" })] }))] }, item.id ?? i));
                 }) })] }));

package/dist/blocks.js

@@ -164,7 +164,10 @@ export function codeTabsToMarkdown(tabs) {
         .map(t => {
         const longest = Math.max(0, ...(t.code.match(/`+/g) ?? []).map(run => run.length));
         const fence = '`'.repeat(Math.max(3, longest + 1));
-        return `**${t.label}**\n\n${fence}${t.language || ''}\n${t.code.trim()}\n${fence}`;
+        // Blank lines go from either end, but not the first line's indentation:
+        // in YAML or Python it is part of the code.
+        const code = t.code.replace(/^\s*\n/, '').trimEnd();
+        return `**${t.label}**\n\n${fence}${t.language || ''}\n${code}\n${fence}`;
     })
         .join('\n\n');
 }

package/dist/conformance.d.ts

@@ -152,16 +152,6 @@ export declare const S10_DESCRIPTORS: readonly string[];
  * validator, and it is enough.
  */
 export declare function descriptorChecks(t: Tester, paths?: readonly string[]): Promise<void>;
-/**
- * Documents that project one corpus at different depths carry different ETags.
- *
- * Depths sharing a validator pass every conditional-GET check and still never
- * move when only one of them changes. Every one must also HAVE an ETag: a Set
- * of three nulls is size one and fails, but one null among two tags would pass
- * a size check alone — the copy of this check one repo carried had exactly
- * that hole.
- */
-export declare function distinctEtagChecks(t: Tester, paths: readonly string[], label?: string): Promise<void>;
 /**
  * Every tool carries behavioural hints, and the ones that write say so.
  *

package/dist/conformance.js

@@ -367,19 +367,6 @@ export async function descriptorChecks(t, paths = S10_DESCRIPTORS) {
         t.check(`${label} states a freshness`, /max-age=\d+/.test(fresh.headers.get('cache-control') ?? ''), `cache-control: ${fresh.headers.get('cache-control')}`);
     }
 }
-/**
- * Documents that project one corpus at different depths carry different ETags.
- *
- * Depths sharing a validator pass every conditional-GET check and still never
- * move when only one of them changes. Every one must also HAVE an ETag: a Set
- * of three nulls is size one and fails, but one null among two tags would pass
- * a size check alone — the copy of this check one repo carried had exactly
- * that hole.
- */
-export async function distinctEtagChecks(t, paths, label = 'llms depths have distinct ETags') {
-    const tags = await Promise.all(paths.map(p => fetch(`${t.base}/${p.replace(/^\//, '')}`).then(r => r.headers.get('etag'))));
-    t.check(label, tags.every(Boolean) && new Set(tags).size === paths.length, tags.join(' '));
-}
 /**
  * Every tool carries behavioural hints, and the ones that write say so.
  *
@@ -452,7 +439,11 @@ export async function payloadBudget(t, calls, maxBytes = 120_000) {
 export async function robotsChecks(t, paths) {
     const body = await (await fetch(`${t.base}/robots.txt`)).text();
     const rules = [];
+    // A group is a run of User-agent lines and the rules after it, so `*` anywhere
+    // in the run takes the group's rules; the next User-agent after a rule opens
+    // a new group.
     let inStar = false;
+    let inRules = false;
     for (const raw of body.split('\n')) {
         const line = raw.replace(/#.*$/, '').trim();
         const [field, ...rest] = line.split(':');
@@ -460,10 +451,15 @@ export async function robotsChecks(t, paths) {
             continue;
         const key = field.trim().toLowerCase();
         const value = rest.join(':').trim();
-        if (key === 'user-agent')
-            inStar = value === '*';
-        else if (inStar && (key === 'allow' || key === 'disallow') && value) {
-            rules.push({ allow: key === 'allow', path: value });
+        if (key === 'user-agent') {
+            if (inRules)
+                [inStar, inRules] = [false, false];
+            inStar ||= value === '*';
+        }
+        else if (key !== 'sitemap') {
+            inRules = true;
+            if (inStar && (key === 'allow' || key === 'disallow') && value)
+                rules.push({ allow: key === 'allow', path: value });
         }
     }
     for (const path of paths) {

package/dist/corpus.js

@@ -1,6 +1,6 @@
 // corpus.ts — a wiki's whole corpus, sliced to something a tool result can hold.
 //
-// /llms-full.txt can afford to be 3 MB: a fetch streams to disk under an ETag.
+// A file export can afford to be 3 MB: a fetch streams to disk under an ETag.
 // A tool result lands in a context window, so the MCP twin of that export needs
 // a preflight, a character budget and a page-aligned resume. Two wikis wrote
 // that loop and called the second a copy of the first "field for field". It was

package/dist/freshness.d.ts

@@ -12,6 +12,16 @@ export declare function daysSince(date: Date | string | null | undefined, now: n
  * edit is a weaker signal than a verification, but it is a signal.
  */
 export declare function isStale(page: FreshnessInput, now: number, maxAgeDays?: number): boolean;
+export interface FreshnessOptions {
+    /** Older than this, unverified, and the page is stale. */
+    maxAgeDays?: number;
+    /**
+     * What the notice asks a reader to re-check the facts against: `current
+     * sources` unless the wiki has more to name. The two Radix wikis add the live
+     * ledger, which a shop's wiki or a medical one has no use for.
+     */
+    against?: string;
+}
 /**
  * A synthetic `outdated` banner block for a stale page, or null when it is
  * fresh — the shape every wiki's `banner` type already stores, so it drops
@@ -20,7 +30,7 @@ export declare function isStale(page: FreshnessInput, now: number, maxAgeDays?:
  * during a client render would let a page near the boundary be stale on the
  * server and fresh in the browser.
  */
-export declare function freshnessBanner(page: FreshnessInput, nowMs: number, maxAgeDays?: number): {
+export declare function freshnessBanner(page: FreshnessInput, nowMs: number, { maxAgeDays, against }?: FreshnessOptions): {
     id: string;
     type: 'banner';
     variant: 'outdated';

package/dist/freshness.js

@@ -32,17 +32,17 @@ export function isStale(page, now, maxAgeDays = DEFAULT_MAX_AGE_DAYS) {
     return age === null || age > maxAgeDays;
 }
 /**
- * The sentence the notice carries. Both wikis shipped this string identically,
- * down to the ISO date and the closing request, so it lives here rather than
- * being re-typed either side of the extraction.
+ * The sentence the notice carries. Both Radix wikis shipped this string
+ * identically, down to the ISO date and the closing request; what to re-check
+ * against is the one part that differs between wikis.
  *
  * @internal
  */
-export function freshnessNotice(page) {
+export function freshnessNotice(page, against = 'current sources') {
     const when = page.lastVerifiedAt
         ? `last verified ${isoDate(new Date(page.lastVerifiedAt))}`
         : 'not yet verified against sources';
-    return `This page was ${when} and may be out of date. Please help re-check its facts against current sources and the live ledger.`;
+    return `This page was ${when} and may be out of date. Please help re-check its facts against ${against}.`;
 }
 /**
  * A synthetic `outdated` banner block for a stale page, or null when it is
@@ -52,10 +52,10 @@ export function freshnessNotice(page) {
  * during a client render would let a page near the boundary be stale on the
  * server and fresh in the browser.
  */
-export function freshnessBanner(page, nowMs, maxAgeDays = DEFAULT_MAX_AGE_DAYS) {
+export function freshnessBanner(page, nowMs, { maxAgeDays = DEFAULT_MAX_AGE_DAYS, against } = {}) {
     if (!isStale(page, nowMs, maxAgeDays))
         return null;
-    return { id: '__freshness__', type: 'banner', variant: 'outdated', text: freshnessNotice(page) };
+    return { id: '__freshness__', type: 'banner', variant: 'outdated', text: freshnessNotice(page, against) };
 }
 const toMs = (d) => (typeof d === 'number' ? d : d instanceof Date ? d.getTime() : Date.parse(d));
 /**

package/dist/http.js

@@ -151,7 +151,15 @@ export function cacheTagHeaders(tags) {
  */
 export function wantsMarkdown(request) {
     return (new URL(request.url).searchParams.get('format') === 'text' ||
-        /text\/(markdown|plain)/.test(request.headers.get('accept') ?? ''));
+        (request.headers.get('accept') ?? '').split(',').some(acceptsText));
+}
+/** One Accept media range naming markdown or plain text at a weight above zero: `q=0` means "not this". */
+function acceptsText(range) {
+    const [type, ...params] = range.toLowerCase().split(';').map(s => s.replace(/\s/g, ''));
+    if (type !== 'text/markdown' && type !== 'text/plain')
+        return false;
+    const q = params.find(p => p.startsWith('q='));
+    return !q || Number(q.slice(2)) > 0;
 }
 /** The header every response from a content-negotiated URL carries. */
 export const VARY_ACCEPT = { Vary: 'Accept' };
@@ -215,7 +223,9 @@ export function cleanSnippet(text, max = 160) {
 export function pageLine(opts) {
     const excerpt = opts.excerpt ? `: ${cleanSnippet(opts.excerpt)}` : '';
     const stamp = opts.updated ? isoDate(opts.updated) : '';
-    return `- [${opts.title}](${opts.url})${excerpt}${stamp ? ` _(updated ${stamp})_` : ''}`;
+    // A bracket in a title ("[RFC] Fees") would close the link text early.
+    const title = opts.title.replace(/[[\]\\]/g, '\\$&');
+    return `- [${title}](${opts.url})${excerpt}${stamp ? ` _(updated ${stamp})_` : ''}`;
 }
 /**
  * The validators for an aggregate: the ETag seeded from `seed` (the depth or

package/dist/link-check.d.ts

@@ -8,6 +8,12 @@ export interface Probe {
     note?: string;
     contentType?: string;
     bytes?: number;
+    /**
+     * The host refused a script rather than the page being gone: a bot challenge,
+     * a 429, or a 403/503 whose body is a challenge page. Report it as
+     * unverifiable, never as dead.
+     */
+    walled?: boolean;
 }
 export interface ProbeOptions {
     /** Ordinary per-attempt budget. */

package/dist/link-check.js

@@ -106,23 +106,41 @@ export function perHost(url, fn) {
     hostQueues.set(host, next.catch(() => undefined));
     return next;
 }
+/**
+ * A browser's request headers. A scholarly publisher or a standards body 403s
+ * Node's own user agent whatever the page's health; acuiq2's sweep, which
+ * checks PubMed and journal citations, sent these and saw a fraction of the
+ * false deaths.
+ */
+const BROWSER = {
+    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36',
+    Accept: 'text/html,application/xhtml+xml,*/*;q=0.8',
+    'Accept-Language': 'en',
+};
+const CHALLENGE = /recaptcha|cf-chl|challenge-platform|just a moment|captcha/i;
 async function attempt(url, budgetMs) {
     // One signal for both requests, so the budget covers the HEAD *and* the GET
     // rather than resetting between them. `engines.node` is >= 20, so this needs
     // no controller and no `clearTimeout` in a `finally` that could be forgotten.
     const signal = AbortSignal.timeout(budgetMs);
     try {
-        let res = await fetch(url, { method: 'HEAD', redirect: 'follow', signal });
+        let res = await fetch(url, { method: 'HEAD', redirect: 'follow', headers: BROWSER, signal });
         // Some hosts reject HEAD outright; retry with GET before believing a 4xx.
+        let body = '';
         if (res.status >= 400) {
-            res = await fetch(url, { method: 'GET', redirect: 'follow', signal });
+            res = await fetch(url, { method: 'GET', redirect: 'follow', headers: BROWSER, signal });
+            if (res.status >= 400)
+                body = await res.text().catch(() => '');
         }
+        // A 429 is the host rationing us, which says nothing about the page.
+        const walled = res.headers.get('cf-mitigated') === 'challenge' || res.status === 429 || ([403, 503].includes(res.status) && CHALLENGE.test(body));
         return {
             url,
             status: res.status,
             ok: res.status < 400,
             contentType: (res.headers.get('content-type') || '').split(';')[0]?.trim(),
             bytes: Number(res.headers.get('content-length') || 0),
+            ...(walled && { walled }),
         };
     }
     catch (err) {

package/dist/maintenance.d.ts

@@ -36,6 +36,8 @@ export interface LogState {
     backlog?: BacklogEntry[];
     feedback?: FeedbackEntry[];
     runHistory?: RunEntry[];
+    /** What `backlog-add` filed since the last `write`, by item: how a write tells a late item from one it closed. */
+    filed?: string[];
     [key: string]: unknown;
 }
 /** Drops what is closed or a diary entry, and caps what is append-only. Newest first. */
@@ -47,6 +49,15 @@ export declare function compact(state: LogState): LogState;
  * with it silently erases what the later run recorded.
  */
 export declare function staleWrite(stored: LogState, next: LogState): string | null;
+/**
+ * The items another routine filed with `backlog-add` after `next` was read,
+ * which `next` therefore cannot carry: in the stored `filed` list and not in
+ * the one `next` was read with. A run's `write` replaces the whole state, so
+ * without this an item filed mid-run was lost. Merged rather than refused,
+ * because a run that has done its work should not be sent back over another
+ * routine's note.
+ */
+export declare function lateBacklog(stored: LogState, next: LogState, today: string): BacklogEntry[];
 /** New backlog items, deduplicated against what is there, dated today unless they carry a date. */
 export declare function addBacklog(state: LogState, items: readonly BacklogEntry[], today: string): BacklogEntry[];
 /** The page's content blocks, re-rendered from state on every write. `routine` names who keeps it. */

package/dist/maintenance.js

@@ -67,6 +67,19 @@ export function staleWrite(stored, next) {
         ? `refusing a state with totalRuns ${has} over the stored ${had}: it predates a run that has already written. Read the log again and reapply this run's changes.`
         : null;
 }
+/**
+ * The items another routine filed with `backlog-add` after `next` was read,
+ * which `next` therefore cannot carry: in the stored `filed` list and not in
+ * the one `next` was read with. A run's `write` replaces the whole state, so
+ * without this an item filed mid-run was lost. Merged rather than refused,
+ * because a run that has done its work should not be sent back over another
+ * routine's note.
+ */
+export function lateBacklog(stored, next, today) {
+    const filed = new Set(stored.filed ?? []);
+    const seen = new Set(next.filed ?? []);
+    return addBacklog(next, (stored.backlog ?? []).filter(b => filed.has(itemOf(b)) && !seen.has(itemOf(b))), today);
+}
 /** New backlog items, deduplicated against what is there, dated today unless they carry a date. */
 export function addBacklog(state, items, today) {
     const seen = new Set((state.backlog ?? []).map(itemOf));
@@ -94,9 +107,10 @@ export function render(state, routine, newId = () => crypto.randomUUID()) {
     const block = (text) => ({ id: newId(), type: 'content', text });
     const label = (x) => (typeof x === 'string' ? x : (x.handle ?? x.name ?? x.url ?? ''));
     const sourceRows = sources.slice(0, SHOWN.sources).map(([key, v]) => {
-        const watch = (v.list ?? []).map(label).filter(Boolean);
+        // `v?.`: the state is a routine's own JSON, and `compact` keeps a null entry.
+        const watch = (v?.list ?? []).map(label).filter(Boolean);
         const watching = watch.length ? ` — watching: ${esc(watch.slice(0, 8).join(', '))}${watch.length > 8 ? '…' : ''}` : '';
-        return `<tr><td>${esc(key)}</td><td>${esc(v.lastChecked || '–')}</td><td>${esc(v.note)}${watching}</td></tr>`;
+        return `<tr><td>${esc(key)}</td><td>${esc(v?.lastChecked || '–')}</td><td>${esc(v?.note)}${watching}</td></tr>`;
     });
     const runRows = (state.runHistory ?? []).slice(-SHOWN.runs).reverse().map(r => {
         const tweet = r.tweet && /^https?:\/\//.test(r.tweet) ? `<a href="${esc(r.tweet)}">link</a>` : esc(r.tweet);

package/dist/markdown.js

@@ -45,7 +45,7 @@ const NAMED_ENTITIES = {
 export function decodeEntities(s) {
     return s.replace(/&(?:#x([0-9a-fA-F]{1,6})|#(\d{1,7})|([a-zA-Z][a-zA-Z0-9]{1,9}));/g, (m, hex, dec, name) => {
         if (name)
-            return NAMED_ENTITIES[name] ?? m;
+            return Object.hasOwn(NAMED_ENTITIES, name) ? NAMED_ENTITIES[name] : m;
         const code = hex ? parseInt(hex, 16) : Number(dec);
         return code <= 0x10ffff ? String.fromCodePoint(code) : m;
     });
@@ -146,15 +146,17 @@ export function htmlToMarkdown(html) {
 }
 /** YAML frontmatter block, `---` fences included. */
 export function frontmatter(fields) {
-    const esc = (s) => s.replace(/"/g, '\\"');
+    // A YAML double-quoted scalar: a backslash starts an escape (`\b` is a
+    // backspace) and a raw newline ends the line, so both are escaped with the quote.
+    const esc = (s) => s.replace(/\\/g, '\\\\').replace(/"/g, '\\"').replace(/\r?\n/g, '\\n');
     return [
         '---',
         `title: "${esc(fields.title)}"`,
-        `url: "${fields.url}"`,
+        `url: "${esc(fields.url)}"`,
         ...(fields.updated ? [`updated: ${isoDate(fields.updated)}`] : []),
         ...(fields.lastVerified ? [`last_verified: ${isoDate(fields.lastVerified)}`] : []),
         ...(fields.license
-            ? [`license: ${fields.license.spdx}`, `license_url: "${fields.license.url}"`]
+            ? [`license: ${fields.license.spdx}`, `license_url: "${esc(fields.license.url)}"`]
             : []),
         ...Object.entries(fields.extra ?? {})
             .filter(([, v]) => v !== undefined && v !== '')