@heroiclands/package-build 24.0.0 → 24.0.1

package/CHANGELOG.md

@@ -1,5 +1,19 @@
 # @heroiclands/package-build
 
+## 24.0.1
+
+### Patch Changes
+
+**Sites**
+
+- The sitemap and site search list only the pages a reader can reach from the homepage, so a page can be published and read at its address before anything links it.
+- Related links, previous and next, and the other generated link sections point only to pages a reader can reach.
+- `site build` finishes writing the search index before it reports success.
+
+**Queries**
+
+- A query on a note's `data` reads the same however other notes are shaped; adding a field to one note no longer breaks unrelated queries.
+
 ## 24.0.0
 
 ### Major Changes

package/docs/commands.md

@@ -657,12 +657,14 @@ The rules are scoped to those hostnames, so a site deployed under a domain of
 its own stays indexable.
 
 With `_headers` written, the rendered site is indexed for search. Pagefind
-reads every page under `<out>/<contentPackage>/` and writes the index into
+reads the pages under `<out>/<contentPackage>/` that are reachable from the
+homepage (see [the pages a reader finds](guides/site.md#decide-which-pages-a-reader-finds))
+and writes the index into
 `<out>/<contentPackage>/pagefind/`, so it is deployed with the pages and served
 at `/<contentPackage>/pagefind/`; the theme loads `pagefind.js` from there,
 relative to the site's base URL, and the browser bundle derives the prefix from
 its own address, so nothing about the prefix is configured on either side. What
-is indexed is the body of every page and the homepage; the theme marks its own
+is indexed is the body of every reachable page and the homepage; the theme marks its own
 chrome — header, footer, breadcrumbs, infobox, the related-pages card —
 `data-pagefind-ignore`, and carries `data-pagefind-filter` attributes for a
 page's `type` and `package`, so a search can be narrowed to one kind of page.
@@ -2577,7 +2579,7 @@ emptied first:
   `site.sections` names at its section, every other note at
   `<type>-<shortcode>/` — each minified;
 - `404.html`, the page-not-found page `site.notfound` describes;
-- `sitemap.xml`, every page by its full address;
+- `sitemap.xml`, the full address of every page reachable from the homepage;
 - the theme's stylesheets, scripts, icon font and images, and `css/code.css`;
 - the script fonts the notes set text in, under
   `fonts/heroiclands/<version>/`;
@@ -2595,8 +2597,8 @@ page-not-found page and reported as a warning located at that entry of
 the content tree is an error located at its entry, and writes nothing.
 
 Then, as `package-build site-root` does, `_headers` is written at `<out>/` and
-the pages are indexed for search into `pagefind/` inside the site's directory,
-unless `site.search` is `false`. With `site.mergeIndexes`, the search reads
+the pages reachable from the homepage are indexed for search into `pagefind/`
+inside the site's directory, unless `site.search` is `false`. With `site.mergeIndexes`, the search reads
 each package site's index beside its own. What a page tells the search is set
 out in the [site guide](guides/site.md#search).
 

package/docs/guides/site.md

@@ -223,6 +223,49 @@ The build writes these keys; [what the theme draws](../reference/theme-pages.md)
 describes the panel each one feeds — the infobox, the hero band, the related
 card, and search — and the generated sections the theme draws as any other.
 
+## Decide which pages a reader finds
+
+Every note is written as a page, linked or not, and a page is served at its
+address. What a reader can find is what the homepage leads to: the sitemap and
+the site search cover the pages reachable from the homepage, directly or
+through the pages it links, and no others.
+
+Reachability is read from the links the rendered pages carry, never from the
+notes' sources. A page is reachable when an authored link leads to it:
+
+- a link in the homepage or in the body of a reachable note, including the
+  links a `pagelist` or a `sql` table writes into the body;
+- a link in the menu or the rest of the site chrome;
+- a link in a listing: the pages of a section page, a tag page, and the posts
+  an Atom feed lists, once a reachable page links the section, the tag or the
+  feed. A tag chip on a reachable page links its tag page when `site.tags` is
+  on, so every page carrying the tag is then reachable.
+
+The links the build generates do not make a page reachable. These are the
+Related card, the previous and next links, the infobox links, and the sections
+generated after a note's text (Within, Governed by, Governed places,
+Chronology, Events, Accounts, What followed, In song and story). Each is drawn
+without a link to a page outside the reachable set, and a section left with
+nothing under it is not drawn. An infobox value that names an unreachable page
+is set as plain text. A link written in prose or produced by a query is kept
+wherever it points, and it is what makes its target reachable.
+
+A page that is not reachable is the way to preview a page in place. Write the
+note, build the site, and read the page at its address on the live site; link
+it from the homepage or from a linked page when it is ready to publish. For a
+note `doc-charlie` in the package `setting`:
+
+1. Add `doc-charlie` to the content tree. It is written at
+   `/setting/doc-charlie/` and appears in neither the sitemap nor the search.
+2. Read it at `/setting/doc-charlie/`.
+3. Link it, for example `[[doc-charlie|Charlie]]` in a linked note, to publish
+   it. The next build lists it in the sitemap and the search.
+
+An address is a short code and the site serves no directory listing, so an
+unlinked page is found only by someone who knows its address. The content
+index that `package-build content-index` writes is a release asset and is not
+part of the site.
+
 ## Show marked text in its font
 
 A site shows text marked with `font=` and one of the faces of

package/engine/site-build.mjs

@@ -99,6 +99,7 @@ import { checkHomepageCount, homepageFrontmatter, homepageTitle, isHomepage } fr
 import { publishesContentPages, siteBase } from "../content-config.mjs";
 import { seeAlsoPages } from "./see-also.mjs";
 import { homepageLinkTargets, relatedPages } from "./related-pages.mjs";
+import { GENERATED_MARK } from "./site-reach.mjs";
 import { drawSiteMaps, inlineDrawingFigure } from "./site-maps.mjs";
 import { generatedDrawing } from "./generated-sections.mjs";
 import { IMAGE_PATTERN } from "./content-images.mjs";
@@ -784,7 +785,13 @@ export function renderSitePage(
     const tableErrors = expanded.findings.tables;
     const expressionErrors = expanded.findings.expressions;
     const wikiErrors = expanded.findings.links;
-    const resolvedBody = expanded.markdown;
+    // The author's text and the sections the build generated after it are
+    // told apart by a comment, so the site writer can leave the generated
+    // links out of what makes a page reachable.
+    const resolvedBody =
+        expanded.generated ?
+            `${expanded.authored}\n\n${GENERATED_MARK}\n\n${expanded.generated}`
+        :   expanded.markdown;
     for (const error of expanded.findings.captions)
         captionErrors.push({
             file: page.file,

package/engine/site-reach.mjs

@@ -0,0 +1,213 @@
+/*
+ * This file is part of the Song of Heroic Lands (SoHL) system for Foundry VTT.
+ * Copyright (c) 2024-2026 Tom Rodriguez ("Toasty") — <toasty@heroiclands.org>
+ *
+ * This work is licensed under the GNU General Public License v3.0 (GPLv3).
+ * You may copy, modify, and distribute it under the terms of that license.
+ *
+ * For full terms, see the LICENSE.md file in the project root or visit:
+ * https://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * SPDX-License-Identifier: GPL-3.0-or-later
+ */
+
+/**
+ * **The pages a reader can reach.** Every note is written as a page, but what
+ * a reader finds is what the homepage leads to: the sitemap and the search
+ * index cover the pages reachable from the homepage by the links the rendered
+ * HTML carries, and no others. An unlinked page is still served at its
+ * address, so an author reads a page in place before linking it to publish it.
+ *
+ * Reachability is read from the rendered pages, never from the source notes:
+ * the `<a>` and `<link>` targets that lie inside the site's base path, the
+ * menu and the chrome included. A link that lands on a feed makes the pages
+ * the feed lists reachable too, because a feed is a list of links.
+ *
+ * @module
+ */
+
+/** A URL with a scheme, or a protocol-relative one: not a path inside the site. */
+const EXTERNAL = /^(?:[a-z][a-z0-9+.-]*:|\/\/)/i;
+
+/** An `<a>` or `<link>` and its `href`, quoted either way or bare. */
+const LINK = /<(?:a|link)\b[^>]*?\shref=(?:"([^"]*)"|'([^']*)'|([^\s>]+))/gi;
+
+/**
+ * The internal targets a page links.
+ *
+ * @param {string} html - A rendered page.
+ * @param {string} from - The page's own address, beginning with `/`.
+ * @param {string} base - The site's base path, beginning and ending with `/`.
+ * @returns {string[]} Absolute paths inside the base, without fragment or query.
+ */
+export function internalLinks(html, from, base) {
+    const found = [];
+    for (const match of html.matchAll(LINK)) {
+        const href = (match[1] ?? match[2] ?? match[3] ?? "").trim().replace(/&amp;/g, "&");
+        if (!href || href.startsWith("#") || EXTERNAL.test(href)) continue;
+        let pathname;
+        try {
+            pathname = new URL(href, `http://site.invalid${from}`).pathname;
+        } catch {
+            continue;
+        }
+        if (pathname.startsWith(base)) found.push(decodeURI(pathname));
+    }
+    return found;
+}
+
+/**
+ * The address a link to a file names a page by: `…/index.html` is its
+ * directory, and a bare directory ends in a slash.
+ *
+ * @param {string} pathname - An absolute path.
+ * @returns {string} The page address it would be keyed by.
+ */
+export function pageAddress(pathname) {
+    if (pathname.endsWith("/index.html")) return pathname.slice(0, -"index.html".length);
+    return pathname.endsWith("/") || /\.[a-z0-9]+$/i.test(pathname) ? pathname : `${pathname}/`;
+}
+
+/**
+ * The pages reachable from the homepage.
+ *
+ * @param {object} options - The inputs.
+ * @param {string} options.base - The site's base path, beginning and ending
+ *   with `/`; the homepage is at the base.
+ * @param {(url: string) => string|undefined} options.html - A page's rendered
+ *   HTML by its address, or `undefined` where the site has no such page.
+ * @param {(pathname: string) => readonly string[]} [options.expand] - The page
+ *   addresses a linked non-page file, such as a feed, lists.
+ * @returns {Set<string>} The address of every reachable page, the homepage
+ *   included.
+ */
+export function reachablePages({ base, html, expand = () => [] }) {
+    /** @type {Set<string>} */
+    const reached = new Set();
+    const queue = [base];
+    const seenFiles = new Set();
+    while (queue.length) {
+        const url = queue.pop();
+        if (reached.has(url)) continue;
+        const body = html(url);
+        if (body === undefined) continue;
+        reached.add(url);
+        for (const target of internalLinks(body, url, base)) {
+            const address = pageAddress(target);
+            if (html(address) !== undefined) {
+                queue.push(address);
+            } else if (!seenFiles.has(target)) {
+                seenFiles.add(target);
+                for (const listed of expand(target)) queue.push(pageAddress(listed));
+            }
+        }
+    }
+    return reached;
+}
+
+/**
+ * The page addresses an Atom feed lists, as paths.
+ *
+ * @param {string} xml - The feed.
+ * @returns {string[]} The `href` of every `<link>` that is an absolute URL,
+ *   as its path.
+ */
+export function feedPaths(xml) {
+    const paths = [];
+    for (const match of xml.matchAll(/<link\b[^>]*?\shref="([^"]*)"/g)) {
+        try {
+            paths.push(new URL(match[1].replace(/&amp;/g, "&")).pathname);
+        } catch {
+            // A relative or malformed link names no page.
+        }
+    }
+    return paths;
+}
+
+/**
+ * The comment that separates a page's authored text from the sections the
+ * build generated after it, in the HTML the renderer writes.
+ *
+ * @type {string}
+ */
+export const GENERATED_MARK = "<!--generated-sections-->";
+
+/**
+ * A page body split at {@link GENERATED_MARK}.
+ *
+ * @param {string} html - A page body.
+ * @returns {{authored: string, generated: string}} The author's text, and the
+ *   generated sections after it, `""` where there are none.
+ */
+export function splitContent(html) {
+    const text = String(html ?? "");
+    const at = text.indexOf(GENERATED_MARK);
+    if (at < 0) return { authored: text, generated: "" };
+    return { authored: text.slice(0, at), generated: text.slice(at + GENERATED_MARK.length) };
+}
+
+/** An `<a>` element with an `href`, whole. */
+const ANCHOR = /<a\b[^>]*?\shref=(?:"([^"]*)"|'([^']*)'|([^\s>]+))[^>]*>[\s\S]*?<\/a>/gi;
+
+/**
+ * Generated sections with the entries that lead outside a set of pages taken
+ * out.
+ *
+ * An entry is a list item, a table row, or a link in a paragraph's comma
+ * separated run; a paragraph, list or table left with no entry goes, and so
+ * does a section left with nothing under its heading. A link to anything that
+ * is not a page of the site is kept, and so is text that was never a link.
+ *
+ * @param {string} html - The generated sections, rendered.
+ * @param {(href: string) => boolean} outside - Whether an `href` names a page
+ *   of the site that is not reachable.
+ * @returns {string} The sections, or `""` when nothing is left.
+ */
+export function pruneGenerated(html, outside) {
+    const dead = (anchor) => {
+        const href = new RegExp(ANCHOR.source, "i").exec(anchor);
+        const value = href ? (href[1] ?? href[2] ?? href[3] ?? "").replace(/&amp;/g, "&") : "";
+        return value !== "" && outside(value);
+    };
+    const holdsDead = (fragment) => [...fragment.matchAll(ANCHOR)].some((match) => dead(match[0]));
+
+    // Each section runs from its heading to the next one.
+    const parts = String(html ?? "").split(/(?=<h1\b)/);
+    const kept = parts.map((part) => {
+        if (!holdsDead(part)) return part;
+        let out = part
+            .replace(/<li\b[^>]*>(?:(?!<\/li>)[\s\S])*<\/li>\s*/g, (item) =>
+                holdsDead(item) ? "" : item,
+            )
+            .replace(/<tr\b[^>]*>(?:(?!<\/tr>)[\s\S])*<\/tr>\s*/g, (row) =>
+                holdsDead(row) ? "" : row,
+            )
+            .replace(/<p\b[^>]*>(?:(?!<\/p>)[\s\S])*<\/p>\s*/g, (paragraph) => {
+                if (!holdsDead(paragraph)) return paragraph;
+                const trimmed = paragraph.replace(ANCHOR, (anchor) =>
+                    dead(anchor) ? "\u0000" : anchor,
+                );
+                const entries = trimmed
+                    .replace(/<\/?p\b[^>]*>/g, "")
+                    .replace(/^\s*<strong>[^<]*<\/strong>/, "")
+                    .replace(/\u0000/g, "")
+                    .replace(/[\s,]+/g, " ")
+                    .replace(/<[^>]*>/g, "")
+                    .trim();
+                if (!entries) return "";
+                return trimmed
+                    .replace(/\u0000,\s*/g, "")
+                    .replace(/,\s*\u0000/g, "")
+                    .replace(/\u0000/g, "");
+            });
+        out = out
+            .replace(/<(ul|ol)\b[^>]*>\s*<\/\1>\s*/g, "")
+            .replace(/<tbody\b[^>]*>\s*<\/tbody>\s*/g, "")
+            .replace(/<table\b[^>]*>(?:(?!<tr\b)[\s\S])*<\/table>\s*/g, (table) =>
+                /<tr\b/.test(table) ? table : "",
+            );
+        const rest = out.replace(/^<h1\b[^>]*>[\s\S]*?<\/h1>/, "").replace(/<!--[\s\S]*?-->/g, "");
+        return rest.trim() === "" ? "" : out;
+    });
+    return kept.join("");
+}

package/engine/site-render.mjs

@@ -50,6 +50,12 @@
  * | `feed: {section, title}` | `<section>/atom.xml`, the section's pages, newest first |
  * | `sitemaps: [...]` | `sitemap-index.xml`, naming this site's sitemap and each one listed, and `robots.txt` naming them all |
  *
+ * Every page is written, but `sitemap.xml` lists only the pages reachable from
+ * the homepage by the links the rendered pages carry; see
+ * {@link module:engine/site-reach}. A feed is a link, so the pages it lists
+ * are reachable when a page links the feed, and a section or tag page lists
+ * pages it links to the same way.
+ *
  * A sitemap index and `robots.txt` belong at the root of an origin, so a site
  * served under a path refuses `sitemaps`.
  *
@@ -76,6 +82,7 @@ import {
 import { buildSite, gatesFailed } from "./site-build.mjs";
 import { readCachedNavigation, siteSettings } from "./site-config.mjs";
 import { atomFeed, robotsTxt, sitemapIndexXml, sitemapXml } from "./site-files.mjs";
+import { pageAddress, pruneGenerated, reachablePages, splitContent } from "./site-reach.mjs";
 import { prepareTreeSqlTables } from "./sql-tables.mjs";
 import { placeGeography, searchMetadata } from "./site-search.mjs";
 import {
@@ -298,8 +305,9 @@ export function writeThemeFiles(out, staticDir = SITE_STATIC_DIR) {
  * @param {string} [options.staticDir] - The theme's own files, copied first.
  * @param {ReadonlySet<string>} [options.only] - The addresses of the pages to
  *   draw again into a site already written. Nothing else is touched.
- * @returns {{files: string[], pages: string[], findings: object[]}} Every file
- *   written, the address of every page, and the layouts' findings.
+ * @returns {{files: string[], pages: string[], reachable: string[],
+ *   findings: object[]}} Every file written, the address of every page, the
+ *   address of each page reachable from the homepage, and the layouts' findings.
  * @throws {Error} When a page's address is outside the site, or a site served
  *   under a path asks for a sitemap index.
  */
@@ -311,32 +319,95 @@ export function writeSite({ site, pages, out, report, staticDir = SITE_STATIC_DI
     const tree = siteTree(site, pages);
     const published = new Set(tree.map((page) => page.url));
     const feedUrl = site.feed ? relUrl(site, `${site.feed.section}/atom.xml`) : "";
-    const theme = createSiteTheme({
-        site: {
-            ...site,
-            feed:
-                site.feed ?
-                    {
-                        title: site.feed.title ?? site.title,
-                        url: feedUrl,
-                        section: text(site.feed.section),
-                    }
-                :   undefined,
-            pages: tree
-                .filter((page) => page.kind === "page")
-                .map((page) => ({
-                    url: page.url,
-                    title: page.title,
-                    section: text(page.section),
-                    type: page.params?.type,
-                    subType: page.params?.subType,
-                    date: page.date ?? page.params?.date,
-                    content: page.content,
-                })),
-            pageExists: (url) => published.has(url) || published.has(`${url.replace(/\/?$/, "/")}`),
-        },
-        report,
+    const base = basePath(site);
+    const origin = (url) => new URL(url, site.baseURL).href;
+
+    // The address of the page a URL names, or undefined where it names none of
+    // this site's.
+    const pageOfUrl = (url) => {
+        let pathname;
+        try {
+            const resolved = new URL(String(url), origin(base));
+            if (resolved.origin !== new URL(site.baseURL).origin) return undefined;
+            pathname = decodeURI(resolved.pathname);
+        } catch {
+            return undefined;
+        }
+        const address = pageAddress(pathname);
+        return published.has(address) ? address : undefined;
+    };
+
+    // A page is drawn twice. The first draw has no generated link to a page,
+    // and the pages it links are the reachable ones; the second draws the
+    // generated sections again with the links that stay inside that set.
+    const makeTheme = ({ visible, contentOf, report: sink }) =>
+        createSiteTheme({
+            site: {
+                ...site,
+                visible,
+                feed:
+                    site.feed ?
+                        {
+                            title: site.feed.title ?? site.title,
+                            url: feedUrl,
+                            section: text(site.feed.section),
+                        }
+                    :   undefined,
+                pages: tree
+                    .filter((page) => page.kind === "page")
+                    .map((page) => ({
+                        url: page.url,
+                        title: page.title,
+                        section: text(page.section),
+                        type: page.params?.type,
+                        subType: page.params?.subType,
+                        date: page.date ?? page.params?.date,
+                        content: contentOf(page),
+                    })),
+                pageExists: (url) =>
+                    published.has(url) || published.has(`${url.replace(/\/?$/, "/")}`),
+            },
+            report: sink,
+        });
+    const authoredOf = (page) => splitContent(page.content).authored;
+    const first = makeTheme({
+        visible: (url) => pageOfUrl(url) === undefined,
+        contentOf: authoredOf,
+    });
+    const firstPass = new Map();
+    for (const page of tree)
+        firstPass.set(
+            page.url,
+            first.render({
+                ...page,
+                content: page.content === undefined ? undefined : authoredOf(page),
+                permalink: origin(page.url),
+            }),
+        );
+
+    const section = site.feed ? text(site.feed.section) : "";
+    const time = (page) => new Date(text(page.date ?? page.params?.date)).getTime() || 0;
+    const posts =
+        site.feed ?
+            tree
+                .filter((page) => page.kind === "page" && text(page.section) === section)
+                .sort((a, b) => time(b) - time(a))
+        :   [];
+    const reachable = reachablePages({
+        base,
+        html: (url) => firstPass.get(url),
+        expand: (target) => (site.feed && target === feedUrl ? posts.map((post) => post.url) : []),
     });
+    const outside = (url) => {
+        const address = pageOfUrl(url);
+        return address !== undefined && !reachable.has(address);
+    };
+    const contentOf = (page) => {
+        if (page.content === undefined) return page.content;
+        const { authored, generated } = splitContent(page.content);
+        return authored + pruneGenerated(generated, outside);
+    };
+    const theme = makeTheme({ visible: (url) => !outside(url), contentOf, report });
 
     const files = [];
     const write = (file, content) => {
@@ -344,15 +415,22 @@ export function writeSite({ site, pages, out, report, staticDir = SITE_STATIC_DI
         fs.writeFileSync(file, content);
         files.push(file);
     };
-    const origin = (url) => new URL(url, site.baseURL).href;
-    const draw = (page) =>
-        write(
-            pageFile(site, out, page.url),
-            minifyHtml(theme.render({ ...page, permalink: origin(page.url) })),
+    const draw = (page) => {
+        const html = minifyHtml(
+            theme.render({ ...page, content: contentOf(page), permalink: origin(page.url) }),
         );
+        write(pageFile(site, out, page.url), html);
+    };
+    const result = (extra = {}) => ({
+        files,
+        pages: tree.map((page) => page.url),
+        reachable: [...reachable],
+        findings: theme.findings,
+        ...extra,
+    });
     if (only) {
         for (const page of tree) if (only.has(page.url)) draw(page);
-        return { files, pages: tree.map((page) => page.url), findings: theme.findings };
+        return result();
     }
 
     fs.rmSync(out, { recursive: true, force: true });
@@ -373,19 +451,16 @@ export function writeSite({ site, pages, out, report, staticDir = SITE_STATIC_DI
     write(
         path.join(out, "sitemap.xml"),
         sitemapXml(
-            tree.map((page) => ({
-                loc: origin(page.url),
-                lastmod: page.params?.lastmod ?? page.date ?? page.params?.date,
-            })),
+            tree
+                .filter((page) => reachable.has(page.url))
+                .map((page) => ({
+                    loc: origin(page.url),
+                    lastmod: page.params?.lastmod ?? page.date ?? page.params?.date,
+                })),
         ),
     );
 
     if (site.feed) {
-        const section = text(site.feed.section);
-        const time = (page) => new Date(text(page.date ?? page.params?.date)).getTime() || 0;
-        const posts = tree
-            .filter((page) => page.kind === "page" && text(page.section) === section)
-            .sort((a, b) => time(b) - time(a));
         write(
             path.join(out, ...section.split("/").filter(Boolean), "atom.xml"),
             atomFeed({
@@ -394,7 +469,7 @@ export function writeSite({ site, pages, out, report, staticDir = SITE_STATIC_DI
                 self: origin(feedUrl),
                 author: site.params?.author,
                 entries: posts.map((post) => {
-                    const words = plainText(post.content).split(" ").filter(Boolean);
+                    const words = plainText(contentOf(post)).split(" ").filter(Boolean);
                     return {
                         title: post.title,
                         url: origin(post.url),
@@ -405,7 +480,7 @@ export function writeSite({ site, pages, out, report, staticDir = SITE_STATIC_DI
                             (words.length > SUMMARY_WORDS ?
                                 `${words.slice(0, SUMMARY_WORDS).join(" ")}…`
                             :   words.join(" ")),
-                        content: post.content,
+                        content: contentOf(post),
                     };
                 }),
             }),
@@ -420,7 +495,7 @@ export function writeSite({ site, pages, out, report, staticDir = SITE_STATIC_DI
         write(path.join(out, "robots.txt"), robotsTxt([index, ...all]));
     }
 
-    return { files, pages: tree.map((page) => page.url), findings: theme.findings };
+    return result();
 }
 
 /**

package/engine/site-root.mjs

@@ -51,7 +51,8 @@
 import fs from "node:fs";
 import path from "node:path";
 import * as pagefind from "pagefind";
-import { gunzipSync } from "node:zlib";
+
+import { feedPaths, reachablePages } from "./site-reach.mjs";
 
 import { formatDiagnostic } from "./diagnostics.mjs";
 
@@ -149,11 +150,40 @@ export function headers({ pkg, base, fonts = false } = {}) {
  * binary.
  *
  * @callback SiteIndexer
- * @param {{site: string, output: string}} where - The rendered site, and the
- *   directory inside it the index is written to.
+ * @param {{site: string, output: string, base?: string}} where - The rendered
+ *   site, the directory inside it the index is written to, and the path the
+ *   site is served at. With a `base`, only the pages reachable from the
+ *   homepage are indexed.
  * @returns {Promise<{pages: number}>} How many pages were indexed.
  */
 
+/**
+ * The addresses of the pages reachable from a rendered site's homepage.
+ *
+ * @param {string} site - The rendered site's directory, served at `base`.
+ * @param {string} base - The path the site is served at.
+ * @returns {Set<string>} Page addresses, each beginning with `base`.
+ */
+function reachableFromDisk(site, base) {
+    const file = (url) => path.join(site, ...url.slice(base.length).split("/").filter(Boolean));
+    const read = (target) => {
+        try {
+            return fs.readFileSync(target, "utf8");
+        } catch {
+            return undefined;
+        }
+    };
+    return reachablePages({
+        base,
+        html: (url) => read(path.join(file(url), "index.html")),
+        expand: (target) => {
+            if (!target.endsWith(".xml")) return [];
+            const xml = read(file(target));
+            return xml === undefined ? [] : feedPaths(xml);
+        },
+    });
+}
+
 /**
  * Build the search index of a rendered site with Pagefind.
  *
@@ -169,12 +199,17 @@ export function headers({ pkg, base, fonts = false } = {}) {
  * - The binary missing is reported by the package as an install failure of
  *   its own; it is rethrown naming the platform package to install.
  *
+ * With a `base`, the index covers the pages reachable from the homepage by
+ * the links the rendered HTML carries, and no others: an unlinked page is
+ * served but not searchable. Each reachable page is handed to Pagefind as its
+ * own file, so a page outside the set is never read.
+ *
  * The output directory is emptied first, so an index of an earlier build's
  * pages never survives beside the current one.
  *
  * @type {SiteIndexer}
  */
-export async function indexSite({ site, output }) {
+export async function indexSite({ site, output, base }) {
     if (!fs.statSync(site, { throwIfNoEntry: false })?.isDirectory()) {
         throw new Error(`${site} is not a directory, so there is nothing to index`);
     }
@@ -201,33 +236,40 @@ export async function indexSite({ site, output }) {
     }
     let pages;
     try {
-        const added = await index.addDirectory({ path: site });
-        if (added.errors.length) throw new Error(added.errors.join("; "));
-        pages = added.page_count;
+        if (base) {
+            pages = 0;
+            for (const url of reachableFromDisk(site, base)) {
+                const sourcePath = path.posix.join(
+                    ...url.slice(base.length).split("/"),
+                    "index.html",
+                );
+                const content = fs.readFileSync(path.join(site, sourcePath), "utf8");
+                const added = await index.addHTMLFile({ sourcePath, content });
+                if (added.errors.length) throw new Error(added.errors.join("; "));
+                pages += 1;
+            }
+        } else {
+            const added = await index.addDirectory({ path: site });
+            if (added.errors.length) throw new Error(added.errors.join("; "));
+            pages = added.page_count;
+        }
         if (!pages) {
             throw new Error(`no page under ${site} was indexed — the site holds no HTML`);
         }
-        const written = await index.writeFiles({ outputPath: output });
-        if (written.errors.length) throw new Error(written.errors.join("; "));
+        // The files are taken from the index in memory and written here,
+        // because `writeFiles` can answer while the binary is still flushing
+        // them, which leaves an index that a reader loads half-written.
+        const { files, errors } = await index.getFiles();
+        if (errors.length) throw new Error(errors.join("; "));
+        if (!files.length) throw new Error("the search index holds no files");
+        for (const file of files) {
+            const target = path.join(output, file.path);
+            fs.mkdirSync(path.dirname(target), { recursive: true });
+            fs.writeFileSync(target, file.content);
+        }
     } finally {
         await pagefind.close();
     }
-    // Pagefind may finish its API call while fragment writes are still visible
-    // as incomplete gzip streams under heavy parallel load. A successful site
-    // build must hand deployment complete fragments.
-    const fragmentDir = path.join(output, "fragment");
-    const deadline = Date.now() + 5000;
-    for (;;) {
-        try {
-            const fragments = fs.readdirSync(fragmentDir);
-            if (!fragments.length) throw new Error("no search fragments were written");
-            for (const name of fragments) gunzipSync(fs.readFileSync(path.join(fragmentDir, name)));
-            break;
-        } catch (error) {
-            if (Date.now() >= deadline) throw error;
-            await new Promise((resolve) => setTimeout(resolve, 25));
-        }
-    }
     return { pages };
 }
 
@@ -289,7 +331,7 @@ export async function writeSiteRoot({ pkg, base, out, search = true, indexer = i
     }
     let pages;
     try {
-        ({ pages } = await indexer({ site: rendered, output: dir }));
+        ({ pages } = await indexer({ site: rendered, output: dir, base: prefix }));
     } catch (err) {
         const message = err instanceof Error ? err.message : String(err);
         // A directory, so the locator is the file field alone: there is no

package/engine/site-serve.mjs

@@ -45,6 +45,7 @@ import path from "node:path";
 import { siteBuildFailed } from "./site-build.mjs";
 import { SITE_THEME_DIR } from "./site-theme.mjs";
 import { SITE_STATIC_DIR, buildPackageSite, writeSite, writeThemeFiles } from "./site-render.mjs";
+import { siteBase } from "../content-config.mjs";
 import { SEARCH_DIR, indexSite } from "./site-root.mjs";
 
 /** The address a served page listens on for the server's reloads. */
@@ -165,7 +166,7 @@ export async function serveSite({
 
     const indexForSearch = async () => {
         if (!search || config.site.search === false || !state) return;
-        await indexSite({ site: out, output: path.join(out, SEARCH_DIR) });
+        await indexSite({ site: out, output: path.join(out, SEARCH_DIR), base: siteBase(config) });
     };
 
     await buildAll();

package/engine/site-theme.mjs

@@ -37,7 +37,9 @@
  * `draftNotice`, `scriptFonts`), its `menu`, and `pages`, every regular page
  * as `{url, title, section, type, subType, date, content}`. `tagPages` says the site
  * writes a page for each tag, and `pageExists` answers whether a path inside
- * the site is published. `feed` is the site's Atom feed as `{title, url,
+ * the site is published. `visible`, where the site sets it, answers whether a
+ * generated link may point at a URL; the Related card, the previous and next
+ * links and the infobox links leave out, or set as plain text, what it refuses. `feed` is the site's Atom feed as `{title, url,
  * section}`, `section` being the one its posts are listed in, and
  * `mergeIndexes` the search bundles of other sites its search reads beside
  * its own.
@@ -453,11 +455,14 @@ export function createSiteTheme({ site, dir = SITE_THEME_DIR, bannerData, report
         return { bannerUrl: url, title, tagline };
     };
 
-    /** One infobox entry. */
+    /** Whether a generated link may point at a URL: any, where the site sets no `visible`. */
+    const visible = (url) => (typeof site.visible === "function" ? site.visible(url) : true);
+
+    /** One infobox entry; one that leads outside the reachable pages is its text alone. */
     const entryOf = (entry) => ({
         lead: text(entry?.lead),
         text: text(entry?.text),
-        url: entry?.url ? safeUrl(entry.url) : "",
+        url: entry?.url && visible(entry.url) ? safeUrl(entry.url) : "",
     });
 
     /** The value of one row, by its kind. */
@@ -552,7 +557,7 @@ export function createSiteTheme({ site, dir = SITE_THEME_DIR, bannerData, report
         const deliberate = [];
         const merged = [];
         for (const ref of related.seeAlso ?? []) {
-            if (seen.has(ref.url)) continue;
+            if (seen.has(ref.url) || !visible(ref.url)) continue;
             seen.add(ref.url);
             deliberate.push({
                 url: safeUrl(ref.url),
@@ -561,7 +566,7 @@ export function createSiteTheme({ site, dir = SITE_THEME_DIR, bannerData, report
             });
         }
         for (const ref of [...(related.mentions ?? []), ...(related.backlinks ?? [])]) {
-            if (seen.has(ref.url)) continue;
+            if (seen.has(ref.url) || !visible(ref.url)) continue;
             seen.add(ref.url);
             merged.push(ref);
         }
@@ -595,7 +600,15 @@ export function createSiteTheme({ site, dir = SITE_THEME_DIR, bannerData, report
 
     /** The prev/next links of a page. */
     const navOf = (page, narrow) => {
-        const { prev, next } = neighbours(page, narrow);
+        const { prev, next } = neighbours(
+            {
+                ...page,
+                siblings: (page.siblings ?? []).filter(
+                    (item) => item.url === page.url || visible(item.url),
+                ),
+            },
+            narrow,
+        );
         if (!prev && !next) return undefined;
         const link = (item) => item && { url: safeUrl(item.url), title: text(item.title) };
         return { prev: link(prev), next: link(next) };

package/engine/sql-tables.mjs

@@ -603,11 +603,7 @@ async function createEventRelation(connection, file, records, prefix, audience)
     fs.writeFileSync(file, rows.map((row) => JSON.stringify(row)).join("\n"));
     // `whenIn` and `untilIn` are keyed by calendar Address, and read as structs
     // whatever their number of keys, so `whenIn['<calendar>']` always binds.
-    await createTableFromFile(
-        connection,
-        `${prefix}events`,
-        readJsonAuto(file).replace(/\)$/, ", map_inference_threshold=-1)"),
-    );
+    await createTableFromFile(connection, `${prefix}events`, readJsonAuto(file));
 }
 
 /**
@@ -941,13 +937,20 @@ async function createRowRelation(connection, file, rows, view, columns, types) {
  * type's fields with `NULL` where a record lacks one. A dependency's index has
  * exactly the same shape and needs exactly the same reading.
  *
+ * `map_inference_threshold=-1` keeps every object a struct. Left at its default,
+ * DuckDB reads an object whose keys are sparsely spread across rows (the mean
+ * share of rows carrying a key falls under `field_appearance_threshold`) as a
+ * map, so `data.<key>` changes meaning with the corpus: one more field on one
+ * note can turn a plain comparison into a JSON conversion error.
+ *
  * @param {string} file - The JSONL to read.
  * @returns {string} The `SELECT … FROM read_json_auto(…)` clause.
  */
 function readJsonAuto(file) {
     return (
         `SELECT * FROM read_json_auto('${file.replace(/'/g, "''")}', ` +
-        `format='newline_delimited', union_by_name=true, maximum_object_size=20000000)`
+        `format='newline_delimited', union_by_name=true, maximum_object_size=20000000, ` +
+        `map_inference_threshold=-1)`
     );
 }
 

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@heroiclands/package-build",
-    "version": "24.0.0",
+    "version": "24.0.1",
     "description": "Shared toolchain for building and shipping a HeroicLands Foundry VTT package — content compilation, manifest, localization, staging, bundle, release and deployment.",
     "license": "GPL-3.0-or-later",
     "type": "module",

package/types/engine/site-reach.d.mts

@@ -0,0 +1,76 @@
+/**
+ * The internal targets a page links.
+ *
+ * @param {string} html - A rendered page.
+ * @param {string} from - The page's own address, beginning with `/`.
+ * @param {string} base - The site's base path, beginning and ending with `/`.
+ * @returns {string[]} Absolute paths inside the base, without fragment or query.
+ */
+export function internalLinks(html: string, from: string, base: string): string[];
+/**
+ * The address a link to a file names a page by: `…/index.html` is its
+ * directory, and a bare directory ends in a slash.
+ *
+ * @param {string} pathname - An absolute path.
+ * @returns {string} The page address it would be keyed by.
+ */
+export function pageAddress(pathname: string): string;
+/**
+ * The pages reachable from the homepage.
+ *
+ * @param {object} options - The inputs.
+ * @param {string} options.base - The site's base path, beginning and ending
+ *   with `/`; the homepage is at the base.
+ * @param {(url: string) => string|undefined} options.html - A page's rendered
+ *   HTML by its address, or `undefined` where the site has no such page.
+ * @param {(pathname: string) => readonly string[]} [options.expand] - The page
+ *   addresses a linked non-page file, such as a feed, lists.
+ * @returns {Set<string>} The address of every reachable page, the homepage
+ *   included.
+ */
+export function reachablePages({ base, html, expand }: {
+    base: string;
+    html: (url: string) => string | undefined;
+    expand?: ((pathname: string) => readonly string[]) | undefined;
+}): Set<string>;
+/**
+ * The page addresses an Atom feed lists, as paths.
+ *
+ * @param {string} xml - The feed.
+ * @returns {string[]} The `href` of every `<link>` that is an absolute URL,
+ *   as its path.
+ */
+export function feedPaths(xml: string): string[];
+/**
+ * A page body split at {@link GENERATED_MARK}.
+ *
+ * @param {string} html - A page body.
+ * @returns {{authored: string, generated: string}} The author's text, and the
+ *   generated sections after it, `""` where there are none.
+ */
+export function splitContent(html: string): {
+    authored: string;
+    generated: string;
+};
+/**
+ * Generated sections with the entries that lead outside a set of pages taken
+ * out.
+ *
+ * An entry is a list item, a table row, or a link in a paragraph's comma
+ * separated run; a paragraph, list or table left with no entry goes, and so
+ * does a section left with nothing under its heading. A link to anything that
+ * is not a page of the site is kept, and so is text that was never a link.
+ *
+ * @param {string} html - The generated sections, rendered.
+ * @param {(href: string) => boolean} outside - Whether an `href` names a page
+ *   of the site that is not reachable.
+ * @returns {string} The sections, or `""` when nothing is left.
+ */
+export function pruneGenerated(html: string, outside: (href: string) => boolean): string;
+/**
+ * The comment that separates a page's authored text from the sections the
+ * build generated after it, in the HTML the renderer writes.
+ *
+ * @type {string}
+ */
+export const GENERATED_MARK: string;

package/types/engine/site-render.d.mts

@@ -50,8 +50,9 @@ export function writeThemeFiles(out: string, staticDir?: string): string[];
  * @param {string} [options.staticDir] - The theme's own files, copied first.
  * @param {ReadonlySet<string>} [options.only] - The addresses of the pages to
  *   draw again into a site already written. Nothing else is touched.
- * @returns {{files: string[], pages: string[], findings: object[]}} Every file
- *   written, the address of every page, and the layouts' findings.
+ * @returns {{files: string[], pages: string[], reachable: string[],
+ *   findings: object[]}} Every file written, the address of every page, the
+ *   address of each page reachable from the homepage, and the layouts' findings.
  * @throws {Error} When a page's address is outside the site, or a site served
  *   under a path asks for a sitemap index.
  */
@@ -65,6 +66,7 @@ export function writeSite({ site, pages, out, report, staticDir, only }: {
 }): {
     files: string[];
     pages: string[];
+    reachable: string[];
     findings: object[];
 };
 /**

package/types/engine/site-root.d.mts

@@ -42,6 +42,7 @@ export function headers({ pkg, base, fonts }?: {
 export function indexSite(where: {
     site: string;
     output: string;
+    base?: string;
 }): Promise<{
     pages: number;
 }>;
@@ -125,6 +126,7 @@ export const ORIGIN_SUFFIX: string;
 export type SiteIndexer = (where: {
     site: string;
     output: string;
+    base?: string;
 }) => Promise<{
     pages: number;
 }>;