wiki-formant 0.50.0 → 0.52.0

package/README.md

@@ -2,7 +2,7 @@
 
 The portable half of a wiki: derived taxonomy, a spec-correct MCP transport, markdown twins, and the conditional-GET plumbing an agent surface needs.
 
-Zero runtime dependencies. Web-standard `Request`/`Response`, so it runs unchanged on Next route handlers, Hono, Bun, Deno and workers.
+The editor, history, sanitiser, chart and passkey modules bring their libraries as dependencies (`@tiptap/*`, `fast-diff`, `sanitize-html`, `lightweight-charts`, `@simplewebauthn/*`), since every wiki here uses all five. What only some apps use stays an optional peer: `pg`, `react`, `jose` and `@radixdlt/rola` for ROLA, `sharp` and `@vercel/blob` for uploads. Web-standard `Request`/`Response`, so it runs unchanged on Next route handlers, Hono, Bun, Deno and workers.
 
 ```bash
 npm install wiki-formant
@@ -193,10 +193,12 @@ return new Response(
       license: { spdx: 'CC-BY-4.0', url: 'https://creativecommons.org/licenses/by/4.0/' } },
     blocksToMarkdown(page.content), // your block types, your function
   ),
-  { headers: markdownHeaders(lastModified, { etag }) },
+  { headers: markdownHeaders(lastModified, { etag, noindex: true }) },
 );
 ```
 
+`noindex` sends `X-Robots-Tag: noindex`, for a twin of an HTML page: a search engine that reads it has the page already. A document with no HTML page, an AGENTS.md, leaves it off.
+
 Block trees stay in your app — every project owns its own type set. Give this module HTML and it gives you markdown.
 
 > A trap worth naming: if you serve twins via a rewrite, Next drops the destination query string. The rewrite must carry the `.md` extension through to the destination path, or the twin silently serves JSON.
@@ -263,6 +265,12 @@ const nextConfig = {
 
 Four configs had restated these by hand on the grounds that a config could not import an ESM-only package. It could not because no subpath had a `default` condition, and Next resolves a config's imports the way `require` does; every subpath has one now, and Node 24 loads the module there unchanged.
 
+`withWikiDefaults(config, { csp })` wraps the whole config: the headers on every route with `csp` enforced in production, AVIF before WebP from the image optimiser, Turbopack's build cache off (Vercel restored a stale one and shipped a stylesheet two commits old), and `distDir` from `STUDIO_DIST_DIR` for radix-studio's second dev server. What the app sets wins, and its own `headers()` follow these.
+
+```ts
+export default withWikiDefaults({ async redirects() { /* … */ } }, { csp });
+```
+
 `contentSecurityPolicy` starts from the policy all four origins shared line for line, and each directive passed replaces its default rather than appending, so the call site reads as the whole list. An empty list drops a directive.
 
 The default `frame-src` is `DEFAULT_IFRAME_HOSTS`, the same list the sanitiser's `iframeHosts` defaults to. The sanitiser's comment had asked every config to keep the two matched by hand, and one did not: its policy lacked the bare `youtube.com` and `youtube-nocookie.com` its sanitiser kept, so those embeds survived cleaning and then rendered blank. A site with its own host list passes it to the sanitiser and states the same hosts in its `frame-src`.
@@ -414,7 +422,7 @@ npx indexnow https://radix.wiki --hours 6
 npx indexnow https://radix.wiki --all        # every row: a new origin or a new key
 ```
 
-Rows with no `lastModified` go only in a full submission. A 403 or 422 from the engine exits 1.
+Rows with no `lastModified` go only in a full submission. A 403 or 422 from the engine exits 1. A sitemap index is read one level down, as `check-pages` reads it.
 
 ## Page metadata
 
@@ -485,7 +493,7 @@ was. That half is not here.
 
 What each revision changed, as a history view shows it. `revisions` stays free
 of dependencies because every save imports it; this is the reading half, and it
-needs `fast-diff`, an optional peer.
+needs `fast-diff`.
 
 ```tsx
 import { withChanges } from 'wiki-formant/history';
@@ -560,7 +568,7 @@ In React, `useArticlePasses(ref, [blocks])` runs the three passes and `useTweetE
 
 `sortTables` makes the tables stored in article HTML sortable by their headers. They arrive as a string a `dangerouslySetInnerHTML` wrote, so React never sees their rows and cannot sort them. A column is dates if every filled cell starts with one, numbers if every one does, and text otherwise — one stray value makes the whole column text, which beats sorting half of it by one rule and half by another. A third press restores the author's order, which is often chronological or ranked and otherwise needs a reload. Label/value tables and tables with merged cells are left alone. The markup it writes — `aria-sort` on the cell, a `.sort-header` button inside it — is the markup a React sortable header should write too, so both kinds of table draw their arrows from one stylesheet rule.
 
-`scrollTables(html)` is the one table pass that runs on the string, before render: it wraps each stored table in a `.table-scroll` box, and `base.css` lets that box scroll sideways. A table box ignores `overflow`, so a table with an address or a URL in a cell is as wide as that token whatever its column, and a scroll box added by an effect would arrive after the table had already been laid out too wide. Call it on the render path beside `injectHeadingIds`. A table already in a box keeps it, so it is safe on HTML that has been through it once.
+`scrollTables(html)` is the one table pass that runs on the string, before render: it wraps each stored table in a `.table-scroll` box, and `base.css` lets that box scroll sideways. A table box ignores `overflow`, so a table with an address or a URL in a cell is as wide as that token whatever its column, and a scroll box added by an effect would arrive after the table had already been laid out too wide. Call it on the render path beside `injectHeadingIds`. A table already in a box keeps it, so it is safe on HTML that has been through it once. `articleHtml(html, { used, slug, anchor, links })` (`wiki-formant/headings`) is that chain: heading ids, the scroll boxes, and, given `links`, `normaliseLinks`.
 
 The embed's origin is written down once. It is both the embed host and the allow-list `onTweetResize` checks before believing a posted height, and it had been spelled out at four call sites across two repos. Any page can `postMessage`; only the embed host may size the embed.
 
@@ -623,7 +631,7 @@ A `codeTabs` tab's `code` is source text in every wiki here, and every view trea
 
 ## Sanitising stored HTML
 
-`wiki-formant/sanitize` is the allowlist between an author's saved HTML and a reader's browser. The block views render two HTML fields themselves — `linkGrid` descriptions and `references` text — and a repo renders `content.text` beside them, so the guard ships with the renderer. `codeTabs` is not one of them: its code is source. `sanitize-html` is an optional peer; run it server-side, on the render path, so it covers rows written before it existed and no sanitiser ships to the client.
+`wiki-formant/sanitize` is the allowlist between an author's saved HTML and a reader's browser. The block views render two HTML fields themselves — `linkGrid` descriptions and `references` text — and a repo renders `content.text` beside them, so the guard ships with the renderer. `codeTabs` is not one of them: its code is source. Run it server-side, on the render path, so it covers rows written before it existed and no sanitiser ships to the client.
 
 ```ts
 import { createHtmlSanitizer, sanitizeCoreLeaf } from 'wiki-formant/sanitize';
@@ -653,7 +661,7 @@ The bytes say what the file is, never the name or type, and SVG is refused. Ever
 
 ## Editor nodes
 
-`wiki-formant/tiptap` carries the custom nodes both wiki editors had written twice: `Iframe`, `YouTube` (the stock extension plus the paste rule it does not ship with), `TwitterEmbed`, `createMapEmbed`, `createCodeBlock` and `createTabs`. The four `@tiptap/*` packages are optional peers, so a consumer taking only the taxonomy still installs a package with no runtime dependencies.
+`wiki-formant/tiptap` carries the custom nodes both wiki editors had written twice: `Iframe`, `YouTube` (the stock extension plus the paste rule it does not ship with), `TwitterEmbed`, `createMapEmbed`, `createCodeBlock` and `createTabs`.
 
 ```ts
 const CodeBlock = createCodeBlock({
@@ -668,16 +676,19 @@ The language list defaults to `CODE_LANGS` and `DEFAULT_LANG` from `wiki-formant
 
 `createHeadingIds({ slug })` decorates each heading in the editor with the id its published copy will carry, through `uniqueHeadingId` — the dedupe `injectHeadingIds` uses — so a rail listing headings mid-edit links to the anchors readers will get. `uploadImageTo('/api/upload')` is the `uploadImage` two editors had written identically.
 
-`toolbarActions([...])` picks the formatting commands, in order, with the labels a screen reader announces — two toolbars titled buttons with the internal key, so one said "codeBlock". Render each through `ToolbarButton` (named, and `aria-pressed`, since these are toggles), and run a table command with `runTableAction`. Upload and embed stay with the caller; they are the parts that differ.
-
-`useUrlPrompt(editor, { resolveMapUrl })` is the link and embed address field under a toolbar, in place of a `window.prompt` that blocks the page and cannot show the link the cursor is in. Three editors had written it and one still used the prompt; only one copy both opened a link field on the current address and unlinked on an emptied one. The hook carries both, closes the field when the editor it serves changes (a toolbar shared between blocks would otherwise apply it to the wrong one), and leaves the markup to you.
+`toolbarActions([...])` picks the formatting commands, in order, with the labels a screen reader announces — two toolbars titled buttons with the internal key, so one said "codeBlock". `WikiToolbar` lays them out the way three wikis had each assembled them by hand: the format buttons, Remove link while the cursor is in a link, upload and embed, the insert buttons, and the table controls while the cursor is in a table. Every button is a `ToolbarButton` (named, and `aria-pressed`, since these are toggles); the link button opens the address field and reads as on while it is open. Which buttons, icons and classes are yours; a toolbar shaped differently builds from `ToolbarButton` and `runTableAction` directly.
 
 ```tsx
-const url = useUrlPrompt(editor, { resolveMapUrl });
-<ToolbarButton label="Link" pressed={url.kind === 'link'} onPress={() => url.toggle('link')}>…</ToolbarButton>
-{url.kind && <input autoFocus value={url.value} onChange={e => url.setValue(e.target.value)} onKeyDown={url.onKeyDown} />}
+<WikiToolbar editor={editor} isActive={isActive} prompt={url} actions={FORMAT} trailing={INSERT} onUpload={triggerUpload} embed
+  icons={{ ...ICONS, unlink: <Unlink />, upload: <Upload />, embed: <Globe /> }}
+  classNames={{ toolbar: 'toolbar', button: 'toolbar-btn', pressed: 'toolbar-btn-on', divider: 'toolbar-divider' }} />
+<UrlPromptField prompt={url} classNames={{ row: 'row', input: 'input', apply: 'btn', cancel: 'btn-quiet' }} />
 ```
 
+`useUrlPrompt(editor, { resolveMapUrl })` is the link and embed address field under a toolbar, in place of a `window.prompt` that blocks the page and cannot show the link the cursor is in. Three editors had written it and one still used the prompt; only one copy both opened a link field on the current address and unlinked on an emptied one. The hook carries both, closes the field when the editor it serves changes (a toolbar shared between blocks would otherwise apply it to the wrong one), and leaves the markup to you.
+
+`UrlPromptField` is its markup: an address input on which Enter applies and Escape closes, and Apply and Cancel.
+
 A link is stored as its `href` and the `link` class, nothing else. Stock Tiptap writes `target="_blank" rel="noopener noreferrer nofollow"` onto every link, internal ones included, which on a site throws away its own link equity and opens in-site navigation in new tabs. Whether a link is external depends on the host serving the page, so `normaliseLinks` decides it at render time. A legacy link loses both attributes the next time its page is saved.
 
 The redirect resolves through the wiki's own route, because the editor cannot read it cross-origin. `resolveMapUrl` is the client half and `resolveMapHandler` the whole route: exact shortener hosts, one hop, an allowlisted landing host, a timeout, and your sign-in check as `authorize`. One of the two copies it replaced matched `goo.gl` as a substring and followed every redirect for anyone who asked.
@@ -686,6 +697,36 @@ The redirect resolves through the wiki's own route, because the editor cannot re
 export const GET = resolveMapHandler({ authorize: async () => !!(await currentUser()) });
 ```
 
+## Block editor
+
+`wiki-formant/block-editor` is the block list an author edits and the fields of the core types. Three wikis had written it, 500 to 700 lines each, and had drifted: one dropped a stored code language its list did not name, one let a typed count run past the row cap, one named every remove button "Remove", one never marked the block being edited. The behaviour is here once; the look arrives as a kit.
+
+```tsx
+const KIT: BlockEditorKit = {
+  classNames: { list: 'stack', card: 'block-card', action: 'icon-btn', input: 'input', row: 'row', … },
+  icons: { add: <Plus />, up: <ChevronUp />, down: <ChevronDown />, duplicate: <Copy />, remove: <Trash2 /> },
+  Choice: ToggleGroup,   // optional: a <select> otherwise
+  Menu: PopoverMenu,     // optional: an inline panel otherwise
+};
+const SCHEMA: BlockSchema<Block> = {
+  meta: type => BLOCK_META[type], create: createBlock, duplicate: duplicateBlock,
+  types: PAGE_TYPES, nestedTypes: ATOMIC_TYPES, single: ['infobox'],
+  renderBody: (block, onUpdate) => {
+    switch (block.type) {
+      case 'codeTabs': return <CodeTabsFields block={block} onUpdate={onUpdate} />;
+      case 'columns': return <ColumnsFields block={block} onUpdate={onUpdate} />;
+      case 'daoTimeline': return <DaoTimelineEdit block={block} onUpdate={onUpdate} />;
+      …
+    }
+  },
+};
+<BlockEditor blocks={blocks} onChange={setBlocks} schema={SCHEMA} kit={KIT} />
+```
+
+The dispatch stays a `switch` over the wiki's own union, so a new type is a compile error there. `CodeTabsFields`, `StatsFields`, `ReferencesFields`, `LinkGridFields`, `BannerFields`, `PageListFields`, `ColumnsFields` and `InfoboxFields` take the core blocks `wiki-formant/blocks` declares; the two containers hold the same list, offering `nestedTypes`. `Field`, `Choice` and `LimitField` are exported for a wiki's own editors, so its native types wear the same kit. A card is marked `data-selected` while it is being edited and `data-nested` inside a container; a type the build does not know is shown by its raw type and carried through untouched. Keep the kit and the schema stable — a module constant or a `useMemo` — since both reach every card through context, and a fresh one re-renders every card on every keystroke.
+
+The core blocks are `ContentBlock`, `CodeTabsBlock`, `StatsBlock`, `LinkGridBlock`, `ReferencesBlock`, `BannerBlock`, `ColumnsBlock<A>` and `InfoboxBlock<A>`, `A` being the wiki's atomic union. A wiki builds its union from them beside its own `recentPages` and `pageList`, which hold its own page rows or a section, and its native types.
+
 ## Analytics
 
 Page views and events, counted in the site's own Postgres instead of a hosted
@@ -757,7 +798,6 @@ returns the whole of it. `gate.route` is the sign-in route,
 `gate.signedIn(cookieValue)` is the check a page makes, `gate.open()` is true
 while no passkey is saved, and `gate.cookie` is the cookie's name. The two tables (`Passkey`, `PasskeyToken`) ship as Prisma
 models in `prisma/passkey.prisma`, and every token is kept only as its hash.
-`@simplewebauthn/server` is an optional peer.
 
 There is no sign-up. While no passkey is saved, the page saves the first one
 from whoever opens it, so the owner should do that as soon as the door ships.
@@ -773,14 +813,18 @@ export const gate = createPasskeyGate({ sql, rpName: 'Caper' });
 export const POST = gate.route;
 
 // src/app/stats/page.tsx
-if (!(await gate.signedIn((await cookies()).get(gate.cookie)?.value)))
-  return <PasskeyButton endpoint="/api/passkey" invite={invite} create={await gate.open()} />;
-const q = statsQuery(await searchParams); // ?days= (1, 7, 30, 90, 365 or all, else 30) and the filters
-return <Stats digest={await digest(sql, q)} {...q} />;
+const q = statsQuery(params); // ?days= (1, 7, 30, 90, 365 or all, else 30) and the filters
+return (
+  <StatsGate gate={gate} session={(await cookies()).get(gate.cookie)?.value} invite={params.invite} host="caper.network">
+    {async () => <Stats digest={await digest(sql, q)} {...q} />}
+  </StatsGate>
+);
 ```
 
+`StatsGate` (`wiki-formant/stats`) renders `children` for a signed-in reader and, for anyone else, the line saying what the prompt will do and the `PasskeyButton` that does it; `classNames` places the text, the button's box, the button and its error.
+
 `PasskeyButton` (`wiki-formant/passkey-button`) is the browser half and has its
-own subpath, because it imports the optional peer `@simplewebauthn/browser`.
+own subpath, because it imports `@simplewebauthn/browser`.
 `Stats` (`wiki-formant/stats`) renders a `digest` as one page: the
 headline figures, each with its change from the window before
 (`.stats-delta[data-better]`, a share or for the bounce rate points, for the
@@ -803,8 +847,7 @@ events only from their visitors. Agent tool calls and missed searches are
 events no visit reaches, so their rows are not links. `ViewDay` keeps totals,
 not visits, so a narrowed page starts at `counted_from`, the first day the site
 counted itself, says so, and drops the comparison with the window before. The chart is `TimeChart` with a count
-axis from zero, so `lightweight-charts` is an optional peer that a stats page
-needs installed.
+axis from zero.
 
 ## ROLA sign-in
 
@@ -899,7 +942,19 @@ package's tests still reach it, and the `.d.ts` leaves it out.
 
 `wiki-formant/db` holds `pgPoolConfig(url)`, the options every app here hands to
 `new pg.Pool()`: ten clients on Supabase's 6543 transaction pooler, three on the
-5432 session pooler, whose fifteen-client cap a build's workers share.
+5432 session pooler, whose fifteen-client cap a build's workers share. It also
+holds `prismaConfig(root, schema?)`, what an app's prisma.config.ts hands
+`defineConfig`: `.env` loaded from `root`, the schema folder (or one file) and
+migrations under it, and DIRECT_URL preferred, because the Prisma CLI's
+connections hang on the transaction pooler.
+
+```ts
+// prisma.config.ts
+import { defineConfig } from 'prisma/config';
+import { prismaConfig } from 'wiki-formant/db';
+
+export default defineConfig(prismaConfig(import.meta.dirname));
+```
 
 `rateLimit` takes `blockMs` (and a `blockKey`, default the bucket's own) for a
 caller already shown to be abusive: running dry blocks the key outright for that
@@ -928,7 +983,7 @@ npx analytics-digest --days 30 --handle radixwiki --siblings caper.network,acuiq
 npx analytics-digest --days 7 --gap "Symptom Miss:term"   # adds an event to the search gaps
 ```
 
-`passkey-invite` prints a single-use link that saves another passkey for the page it names, good for a day (`--days n` for longer). Run it the same way:
+`passkey-invite` prints a single-use link that saves another passkey for the page it names, good for a day (`--days n` for longer). No routine calls it: the site's owner runs it by hand, from an app's checkout, when a second device or a second person needs into the stats page. Run it the same way:
 
 ```sh
 npx passkey-invite https://caper.network/stats
@@ -941,6 +996,12 @@ npx prisma-models            # write both into prisma/schema
 npx prisma-models --check    # in the build, before prisma generate
 ```
 
+An app that keeps one schema file and writes the models into it by hand, because each needs a relation the shipped one cannot carry, checks them in place: `--in` names the file and compares model by model, ignoring comments and spacing, and `--tenant` lets each `siteId` drop its default and gain a `@relation` on it.
+
+```sh
+npx prisma-models analytics --check --in prisma/schema.prisma --tenant   # miow
+```
+
 `fts-ddl <table>` rebuilds a table's `search_tsv` generated column and its GIN index from the expression `wiki-formant/search` queries, over `DIRECT_URL` (else `DATABASE_URL` on 5432). Run it before declaring the column in `schema.prisma`; the bin's header says why.
 
 ```sh

package/bin/_cli.mjs

@@ -0,0 +1,50 @@
+// bin/_cli.mjs – what the bins share: reading a flag, and a client on the app's
+// database. Five bins had each written the `.env` fallback and the `pg.Client`
+// setup, and four the flag reader.
+
+const argv = process.argv.slice(2);
+
+/** The value after `name`, or undefined. */
+export const arg = name => {
+  const i = argv.indexOf(name);
+  return i >= 0 ? argv[i + 1] : undefined;
+};
+
+/** Every value after each `name`, for a flag given more than once. */
+export const all = name => argv.flatMap((a, i) => (a === name && argv[i + 1] ? [argv[i + 1]] : []));
+
+/**
+ * A connected `pg.Client` on DATABASE_URL, read from the environment first and
+ * `./.env` second, so a bin runs from the app's root, where `pg` is installed.
+ * `direct` prefers DIRECT_URL and moves DATABASE_URL from the transaction
+ * pooler on 6543 to the session one on 5432: DDL hangs on the first. Throws
+ * when no URL is set.
+ */
+export async function connect({ direct = false, ...options } = {}) {
+  const url = () => (direct ? process.env.DIRECT_URL || process.env.DATABASE_URL?.replace(':6543', ':5432').replace(/\?.*$/, '') : process.env.DATABASE_URL);
+  if (!url()) {
+    try {
+      process.loadEnvFile('.env');
+    } catch {
+      // reported below
+    }
+  }
+  const connectionString = url();
+  if (!connectionString) throw new Error(`no ${direct ? 'DIRECT_URL or DATABASE_URL' : 'DATABASE_URL'} in the environment or ./.env`);
+  // Imported once a URL is known: `pg` is the app's, not this package's.
+  const { default: pg } = await import('pg');
+  const client = new pg.Client({ connectionString, ...options });
+  await client.connect();
+  return client;
+}
+
+/** `vercel cache invalidate --tag`, reported either way. */
+export async function invalidate(tag) {
+  const { execFileSync } = await import('node:child_process');
+  try {
+    execFileSync('vercel', ['cache', 'invalidate', '--tag', tag, '--yes'], { stdio: 'ignore' });
+    console.log(`cache: ${tag} tag invalidated`);
+  } catch (e) {
+    console.warn(`cache: ${tag} tag NOT invalidated (${e.message}); run \`vercel cache invalidate --tag ${tag}\``);
+  }
+}

package/bin/analytics-digest.mjs

@@ -10,31 +10,12 @@
 //   npx analytics-digest --days 30 --handle radixwiki --siblings caper.network,acuiq.com
 //   npx analytics-digest --days 7 --site <id>             # a multi-tenant app's site
 //   npx analytics-digest --days 7 --gap "Symptom Miss:term"   # adds to the search gaps
-import pg from 'pg';
 import { digest, SEARCH_GAPS } from '../dist/analytics.js';
+import { all, arg, connect } from './_cli.mjs';
 
-const args = process.argv.slice(2);
-const arg = name => {
-  const i = args.indexOf(name);
-  return i >= 0 ? args[i + 1] : undefined;
-};
-const all = name => args.flatMap((a, i) => (a === name && args[i + 1] ? [args[i + 1]] : []));
-
-if (!process.env.DATABASE_URL) {
-  try {
-    process.loadEnvFile('.env');
-  } catch {
-    // reported below
-  }
-}
-if (!process.env.DATABASE_URL) {
-  console.log(JSON.stringify({ error: 'no DATABASE_URL in the environment or ./.env' }));
-  process.exit(0);
-}
-
-const client = new pg.Client({ connectionString: process.env.DATABASE_URL });
+let client;
 try {
-  await client.connect();
+  client = await connect();
   const sql = (query, ...values) => client.query(query, values).then(r => r.rows);
   const result = await digest(sql, {
     site: arg('--site') ?? '',
@@ -53,5 +34,5 @@ try {
 } catch (e) {
   console.log(JSON.stringify({ error: e.message }));
 } finally {
-  await client.end().catch(() => {});
+  await client?.end().catch(() => {});
 }

package/bin/check-pages.mjs

@@ -28,12 +28,11 @@
 // which is the right default, since an unlisted route is one nobody asked a
 // crawler to read.
 import { outlineIssues } from '../dist/headings.js';
+import { readSitemap } from '../dist/indexnow.js';
+import { decodeEntities } from '../dist/markdown.js';
 import { TITLE_BUDGET, TITLE_LIMIT } from '../dist/metadata.js';
+import { arg } from './_cli.mjs';
 
-const arg = name => {
-  const i = process.argv.indexOf(name);
-  return i >= 0 ? process.argv[i + 1] : undefined;
-};
 const has = name => process.argv.includes(name);
 
 const site = (arg('--site') || '').replace(/\/+$/, '');
@@ -52,20 +51,9 @@ const get = async url => {
   return { status: res.status, body: res.ok ? await res.text() : '' };
 };
 
-// The sitemap is the URL list, and a sitemap index is one more fetch deep.
+// The sitemap is the URL list.
 async function sitemapUrls(origin) {
-  const seen = [];
-  const read = async url => {
-    const { status, body } = await get(url);
-    if (status !== 200) return;
-    const locs = [...body.matchAll(/<loc>\s*([^<\s]+)\s*<\/loc>/g)].map(m => m[1]);
-    if (/<sitemapindex/i.test(body)) {
-      for (const child of locs.slice(0, 20)) await read(child);
-    } else {
-      seen.push(...locs);
-    }
-  };
-  await read(`${origin}/sitemap.xml`);
+  const seen = (await readSitemap(`${origin}/sitemap.xml`, url => fetch(url, { headers: { 'user-agent': UA } })).catch(() => [])).map(e => e.url);
   // A sitemap states absolute production URLs by definition, so pointing
   // --site at a dev server would otherwise crawl production and report the
   // build you were trying to replace. The path is what the sitemap contributes;
@@ -88,8 +76,6 @@ const descriptionOf = html => {
   const m = html.match(/<meta[^>]+name=["']description["'][^>]*>/i);
   return m ? (m[0].match(/content=["']([\s\S]*?)["']/i)?.[1] ?? '').trim() : null;
 };
-const decode = s =>
-  s.replace(/&amp;/g, '&').replace(/&lt;/g, '<').replace(/&gt;/g, '>').replace(/&quot;/g, '"').replace(/&#0?39;|&apos;/g, "'");
 
 const urls = await sitemapUrls(site);
 if (!urls.length) {
@@ -126,7 +112,7 @@ for (const url of checked) {
     });
   }
 
-  const title = decode(titleOf(page.body));
+  const title = decodeEntities(titleOf(page.body));
   if (!title) faults.push({ url, fault: 'noTitle', detail: '' });
   else {
     if (title.length > TITLE_LIMIT) faults.push({ url, fault: 'titleTooLong', detail: `${title.length} chars: ${title}` });

package/bin/fts-ddl.mjs

@@ -28,8 +28,8 @@
 // Connection: DIRECT_URL (the session pooler on 5432) when set, else
 // DATABASE_URL moved from 6543 to 5432. Both run DDL; the session port is the
 // one that never hangs on a long statement.
-import pg from 'pg';
 import { searchTsvDdl } from '../dist/search.js';
+import { connect } from './_cli.mjs';
 
 const table = process.argv[2];
 if (!table || !/^[a-z_][a-z0-9_]*$/.test(table)) {
@@ -37,26 +37,11 @@ if (!table || !/^[a-z_][a-z0-9_]*$/.test(table)) {
   process.exit(1);
 }
 
-if (!process.env.DIRECT_URL && !process.env.DATABASE_URL) {
-  try {
-    process.loadEnvFile('.env');
-  } catch {
-    // reported below
-  }
-}
-const url = process.env.DIRECT_URL || process.env.DATABASE_URL?.replace(':6543', ':5432');
-if (!url) {
-  console.error('no DIRECT_URL or DATABASE_URL in the environment or ./.env');
-  process.exit(1);
-}
-
 // A bad route hangs rather than erroring without a connect timeout.
-const client = new pg.Client({
-  connectionString: url.replace(/\?.*$/, ''),
-  ssl: { rejectUnauthorized: false },
-  connectionTimeoutMillis: 15000,
+const client = await connect({ direct: true, ssl: { rejectUnauthorized: false }, connectionTimeoutMillis: 15000 }).catch(e => {
+  console.error(e.message);
+  process.exit(1);
 });
-await client.connect();
 const q = s => client.query(s).then(r => r.rows);
 try {
   const ddl = searchTsvDdl(table);

package/bin/indexnow.mjs

@@ -9,7 +9,7 @@
 //   npx indexnow https://radix.wiki --hours 6
 //   npx indexnow https://radix.wiki --all        # every row: a new origin or a new key
 import { readdirSync, readFileSync } from 'node:fs';
-import { sitemapEntries, changedUrls, submitIndexNow } from '../dist/indexnow.js';
+import { readSitemap, changedUrls, submitIndexNow } from '../dist/indexnow.js';
 
 const args = process.argv.slice(2);
 const origin = args.find(a => /^https?:\/\//.test(a))?.replace(/\/$/, '');
@@ -40,12 +40,10 @@ if (live.trim() !== key) {
   process.exit(1);
 }
 
-const sitemap = await fetch(`${origin}/sitemap.xml`);
-if (!sitemap.ok) {
-  console.error(`${origin}/sitemap.xml answered ${sitemap.status}`);
+const entries = await readSitemap(`${origin}/sitemap.xml`).catch(e => {
+  console.error(e.message);
   process.exit(1);
-}
-const entries = sitemapEntries(await sitemap.text());
+});
 const urls = args.includes('--all') ? entries.map(e => e.url) : changedUrls(entries, Date.now() - hours * 3_600_000);
 const statuses = await submitIndexNow(key, urls);
 console.log(JSON.stringify({ origin, submitted: urls.length, statuses }));

package/bin/maintenance-log.mjs

@@ -31,16 +31,12 @@
 // concurrent run's, and the row lock serialises the two. A revision's version
 // is the page's `version` column bumped a minor where the table has one, else
 // the latest revision's.
-import { execFileSync } from 'node:child_process';
 import { randomUUID } from 'node:crypto';
 import { addBacklog, compact, ident, render, scopeSql } from '../dist/maintenance.js';
 import { bump } from '../dist/versioning.js';
+import { arg, connect, invalidate } from './_cli.mjs';
 
 const args = process.argv.slice(2);
-const arg = name => {
-  const i = args.indexOf(name);
-  return i >= 0 ? args[i + 1] : undefined;
-};
 const [mode, payload] = args.filter((a, i) => !a.startsWith('--') && !args[i - 1]?.match(/^--(?!dry-run)/));
 const dry = args.includes('--dry-run');
 const at = arg('--at') ?? '';
@@ -58,31 +54,17 @@ const scope = scopeSql(arg('--scope'), 3);
 const where = `tag_path = $1 AND slug = $2 AND ${scope.sql}`;
 const address = [at.slice(0, slugAt), at.slice(slugAt + 1), ...scope.values];
 
-if (!process.env.DATABASE_URL) {
-  try {
-    process.loadEnvFile('.env');
-  } catch {
-    // reported below
-  }
-}
-if (!process.env.DATABASE_URL) {
-  console.error('no DATABASE_URL in the environment or ./.env');
-  process.exit(1);
-}
-
 const readStdin = async () => {
   let data = '';
   for await (const chunk of process.stdin) data += chunk;
   return data;
 };
 
-// Imported once the command is known good: `pg` is the app's, not this package's.
-const { default: pg } = await import('pg');
-const client = new pg.Client({ connectionString: process.env.DATABASE_URL });
+let client;
 const q = (text, values) => client.query(text, values).then(r => r.rows);
 let wrote = false;
 try {
-  await client.connect();
+  client = await connect();
   const today = new Date().toISOString().slice(0, 10);
 
   if (mode === 'read') {
@@ -181,18 +163,11 @@ try {
     }
   }
 } catch (e) {
-  await client.query('ROLLBACK').catch(() => {});
+  await client?.query('ROLLBACK').catch(() => {});
   console.error(e.message);
   process.exitCode = 1;
 } finally {
-  await client.end().catch(() => {});
+  await client?.end().catch(() => {});
   const tag = arg('--invalidate');
-  if (wrote && tag) {
-    try {
-      execFileSync('vercel', ['cache', 'invalidate', '--tag', tag, '--yes'], { stdio: 'ignore' });
-      console.log(`cache: ${tag} tag invalidated`);
-    } catch (e) {
-      console.warn(`cache: ${tag} tag NOT invalidated (${e.message}); run \`vercel cache invalidate --tag ${tag}\``);
-    }
-  }
+  if (wrote && tag) await invalidate(tag);
 }

package/bin/mark-verified.mjs

@@ -13,14 +13,10 @@
 //   --table <table>             default pages
 //   --scope <column>[=<value>]  only rows where the column is NULL, or equals the value
 //   --invalidate <tag>          `vercel cache invalidate --tag` after a write
-import { execFileSync } from 'node:child_process';
 import { ident, scopeSql } from '../dist/maintenance.js';
+import { arg, connect, invalidate } from './_cli.mjs';
 
 const args = process.argv.slice(2);
-const arg = name => {
-  const i = args.indexOf(name);
-  return i >= 0 ? args[i + 1] : undefined;
-};
 const all = args.includes('--all');
 const [tagPath, slug] = args.filter((a, i) => !a.startsWith('--') && !['--table', '--scope', '--invalidate'].includes(args[i - 1]));
 
@@ -31,23 +27,9 @@ if (!tagPath || (!all && slug === undefined)) {
 const table = ident(arg('--table') ?? 'pages', '--table');
 const scope = scopeSql(arg('--scope'), 3);
 
-if (!process.env.DATABASE_URL) {
-  try {
-    process.loadEnvFile('.env');
-  } catch {
-    // reported below
-  }
-}
-if (!process.env.DATABASE_URL) {
-  console.error('no DATABASE_URL in the environment or ./.env');
-  process.exit(1);
-}
-
-// Imported once the command is known good: `pg` is the app's, not this package's.
-const { default: pg } = await import('pg');
-const client = new pg.Client({ connectionString: process.env.DATABASE_URL });
+let client;
 try {
-  await client.connect();
+  client = await connect();
   const { rowCount } = all
     ? await client.query(
         `UPDATE ${table} SET last_verified_at = now() WHERE (tag_path = $1 OR tag_path LIKE $2) AND ${scope.sql}`,
@@ -59,17 +41,10 @@ try {
       );
   console.log(`Stamped last_verified_at on ${rowCount} page(s).`);
   const tag = arg('--invalidate');
-  if (rowCount && tag) {
-    try {
-      execFileSync('vercel', ['cache', 'invalidate', '--tag', tag, '--yes'], { stdio: 'ignore' });
-      console.log(`cache: ${tag} tag invalidated`);
-    } catch (e) {
-      console.warn(`cache: ${tag} tag NOT invalidated (${e.message}); run \`vercel cache invalidate --tag ${tag}\``);
-    }
-  }
+  if (rowCount && tag) await invalidate(tag);
 } catch (e) {
   console.error(e.message);
   process.exitCode = 1;
 } finally {
-  await client.end().catch(() => {});
+  await client?.end().catch(() => {});
 }

package/bin/passkey-invite.mjs

@@ -8,36 +8,26 @@
 //
 //   npx passkey-invite https://caper.network/stats
 //   npx passkey-invite http://localhost:3000/stats --days 7
-import pg from 'pg';
 import { createPasskeyGate } from '../dist/passkey.js';
+import { arg, connect } from './_cli.mjs';
 
-const args = process.argv.slice(2);
-const page = args.find(a => /^https?:\/\//.test(a));
-const days = Number(args[args.indexOf('--days') + 1] ?? 1);
+const page = process.argv.slice(2).find(a => /^https?:\/\//.test(a));
 
 if (!page) {
   console.error('usage: passkey-invite <page url> [--days n]');
   process.exit(1);
 }
-if (!process.env.DATABASE_URL) {
-  try {
-    process.loadEnvFile('.env');
-  } catch {
-    // reported below
-  }
-}
-if (!process.env.DATABASE_URL) {
-  console.error('no DATABASE_URL in the environment or ./.env');
-  process.exit(1);
-}
 
-const client = new pg.Client({ connectionString: process.env.DATABASE_URL });
+let client;
 try {
-  await client.connect();
+  client = await connect();
   const sql = (query, ...values) => client.query(query, values).then(r => r.rows);
   const url = new URL(page);
-  url.searchParams.set('invite', await createPasskeyGate({ sql, rpName: '' }).invite(args.includes('--days') ? days : 1));
+  url.searchParams.set('invite', await createPasskeyGate({ sql, rpName: '' }).invite(Number(arg('--days') ?? 1)));
   console.log(url.href);
+} catch (e) {
+  console.error(e.message);
+  process.exitCode = 1;
 } finally {
-  await client.end().catch(() => {});
+  await client?.end().catch(() => {});
 }