@heroiclands/package-build 23.7.1 → 23.9.0

package/CHANGELOG.md

@@ -1,5 +1,46 @@
 # @heroiclands/package-build
 
+## 23.9.0
+
+### Minor Changes
+
+**Dictionaries and glossaries**
+
+- Language dictionaries and declared culture or affiliation glossaries appear automatically in journals, sites and books; remove SQL fences for these defaults.
+- Dictionaries use one full alphabetical list with field tags and one key; existing anchored Dictionary, Lexicon and Glossary headings keep their introductions and receive the lists.
+- Language notes with nonempty `data.words`, and cultures or affiliations with nonempty `data.glossary`, need no prose body; glossary sections show only their owner's declared entries.
+- Glossary entries naming dictionary words show their class and word link, with dictionary senses or the entry's context definition; sole-owner sense labels are omitted on that owner's page, while other attribution remains.
+- Common marks belong only to glossary terms; select or filter `terms.common`, because queries using `dictionary.common` fail.
+- Custom culture views select the culture's own entries and reached ranks or offices, applying common marks only to entries the culture owns.
+- Glossary entries without definitions must name a word in their resolved language or a reachable rank or office; specify the correct language or supply a definition when a name cannot resolve.
+
+**Links and publishing**
+
+- Correct links to headings inside HTML comments or raw HTML, which publish no section; builds also refuse section links absent from a dependency's published notes, including pages without prose anchors or journal documents.
+- Rebuild and fetch package indexes so dependent packages can link to generated sections and SQL-expression anchors; readable item documentation anchors target their documentation journals.
+- Engine publishers with dynamic content must await `prepareContentIndex` and pass its publication to `emitContentIndex` before writing metadata.
+
+### Patch Changes
+
+- Language dictionaries can show optional noun plurals and one verb pattern beside existing gender labels; languages without these details retain their output.
+- Dictionary fragments without a key omit the verb table; key-only fragments print abbreviations alone.
+- Built words show element senses in dictionary etymologies and pass the built-piece check, including declared variants, inherited elements and front-vowel affix forms.
+- Public dictionaries omit GM-only citations, preventing false link errors from hidden notes.
+
+## 23.8.0
+
+### Minor Changes
+
+Lint, Markdown checks and formatting now refuse malformed divs, alerts and poetry. Leave a blank line after a quote or alert before closing `:::`; formatting preserves malformed files for correction.
+
+### Patch Changes
+
+- Authors can suggest related notes with optional explanations, shown in reading order on websites, in Foundry journals, and in books.
+- End-to-end Cypress tests use the same pinned Chrome for Testing locally and in CI, with a reusable download cache and explicit browser overrides.
+- Container tests wait for S3 to answer and explain how to recover safely when switching checkouts leaves an incompatible storage server.
+- Content lint reports index, SQL and validation times in its final summary, making slow stages visible in build logs.
+- Content lint and pack compilation spend less time repeating SQL queries and parsing notes, especially in large content packages.
+
 ## 23.7.1
 
 ### Patch Changes

package/README.md

@@ -16,6 +16,33 @@ npm install -D @heroiclands/package-build
 
 The generated project includes a homepage, website and book targets, npm scripts, and a local `.gitignore`. [Getting started](docs/getting-started.md) routes you to a [system](docs/guides/system-project.md), [module](docs/guides/module-project.md), or [content authoring](docs/authoring/first-note.md) path. The [content-language-server](https://github.com/HeroicLands/content-language-server) provides editor navigation and completion.
 
+## Working on this repository
+
+Run `npm ci` in each fresh clone or worktree. Its `prepare` step installs the
+committed `.githooks/` with `git config core.hooksPath .githooks`; run
+`npm run prepare` to install the hooks again without reinstalling dependencies.
+Git uses the hook files in the current worktree.
+
+These are the canonical POSIX shell hooks from
+[HeroicLands/.github](https://github.com/HeroicLands/.github#local-git-hooks-and-actionsgithooks).
+They need Git, POSIX utilities and an available UTF-8 locale, and run without
+Node at commit time. `pre-commit` refuses commits on `main` and `master`;
+`commit-msg` refuses assistant coauthor credits and standalone assistant
+signatures while allowing ordinary prose and human coauthors. Both forges run
+the pinned canonical drift action against every installed hook file.
+
+Local opt-outs use Git configuration: `hooks.protectedBranch false` or
+`hooks.allowCommitOnMain true` for the branch guard, and
+`hooks.noAttribution false` for the message guard. Prefix each with `git config`
+to set it for this clone. `git commit --no-verify` skips both checks once.
+Server and pull-request enforcement still apply.
+
+The published `githooks/` bundle is a separate optional API described in
+[project setup](docs/project-setup.md#optional-published-hook-bundle). This
+repository’s setup installs the two canonical shell hooks; it installs no
+pre-push hook. Run `npm test` and the other checks directly, or use
+`package-build ci` for a container run.
+
 ## Documentation
 
 The documentation lives in the Markdown files in this repository. Start with [the documentation home](docs/index.md). This toolchain does not build its own site or book. A Hugo theme dependency would form a cycle because the theme uses package-build for releases.
@@ -54,4 +81,9 @@ Every icon and image is covered by a provenance record stating `attribution`, `l
 
 Leading captions, generic fenced divs, inline spans, and poetry code fences are described in [links and markup](docs/authoring/links-and-markup.md).
 
+Cypress end-to-end suites use pinned Chrome for Testing 155.0.8059.84 by default
+on macOS arm64 and Linux x64, downloaded once into a reusable cache. Select an
+installed browser with `package-build e2e fast -- --browser electron` (also
+available for `run` and `open`). See [browser selection and cache settings](docs/guides/e2e.md#choose-a-browser).
+
 The package is GPL-3.0-or-later for code and CC-BY-SA-4.0 for original content. See [LICENSE.md](LICENSE.md).

package/bin/content-commands.mjs

@@ -101,17 +101,23 @@ import { HM3_ITEM_FIELDS } from "../hm3/item-fields.mjs";
 import { ENGINE_NOTE_SCHEMAS } from "../engine/note-schemas.mjs";
 import { schemaSubtypeOf } from "../engine/subtype-registry.mjs";
 import { NOTE_VOCABULARY } from "../engine/note-vocabulary.mjs";
+import { markupFindings } from "../engine/markup-validation.mjs";
 import { checkFormatting, checkPrettierConventions, lintMarkdown } from "../engine/prose-lint.mjs";
 import { lintProse, scoreProseTree } from "../engine/readability-lint.mjs";
 import { loadPackageBuildConfig } from "../config.mjs";
-import { emitContentIndex, indexRecordsFor, isAssetRecord } from "../engine/content-index.mjs";
+import {
+    emitContentIndex,
+    prepareContentIndex,
+    indexRecordsFor,
+    isAssetRecord,
+} from "../engine/content-index.mjs";
 import {
     buildSite,
     gatesFailed,
     formatUnaddressableFinding as formatUnaddressable,
 } from "../engine/site-build.mjs";
 import { auditLinks, buildLinkIndex, walkReachability } from "../engine/content-links.mjs";
-import { prepareTreeSqlTables } from "../engine/sql-tables.mjs";
+import { prepareLinkSqlTables, prepareTreeSqlTables } from "../engine/sql-tables.mjs";
 // The one place a link finding is worded, shared with both builds.
 import { linkFindingMessage } from "../engine/wikilink-syntax.mjs";
 import {
@@ -917,6 +923,7 @@ function lintCommand() {
                 // reader needs a line to open, not a stack.
                 const corpusProblems = [];
                 const parsed = new Map();
+                const indexStarted = performance.now();
                 const records = indexRecordsFor({
                     contentBase: root,
                     config,
@@ -924,12 +931,23 @@ function lintCommand() {
                     problems: corpusProblems,
                     parsed,
                 });
+                const indexMs = performance.now() - indexStarted;
                 for (const problem of corpusProblems) emitDiagnostic(problem);
                 // An error whatever the command's own strictness: the note is
                 // absent from every answer below, so reporting it and exiting 0
                 // would call the tree clean while silently omitting a note.
                 if (corpusProblems.length) process.exitCode = 1;
 
+                const sqlStarted = performance.now();
+                const sqlTables = await prepareTreeSqlTables(root, {
+                    config,
+                    records,
+                    parsed,
+                    skipDirectories: config.skipDirectories,
+                });
+                const sqlMs = performance.now() - sqlStarted;
+                const validationStarted = performance.now();
+
                 // The package is passed for the homepage rule, which
                 // names the address a tree with no front page fails to serve.
                 const addresses = lintContentTree(root, {
@@ -945,12 +963,7 @@ function lintCommand() {
                     config,
                     records,
                     skipDirectories: config.skipDirectories,
-                    sqlTables: await prepareTreeSqlTables(root, {
-                        config,
-                        records,
-                        parsed,
-                        skipDirectories: config.skipDirectories,
-                    }),
+                    sqlTables,
                 });
                 // Which system blocks this tree carries, derived from what it
                 // declares it ships for. Read before the lint so the
@@ -1182,6 +1195,7 @@ function lintCommand() {
                 });
 
                 const findings = [
+                    ...index.notes.flatMap((note) => markupFindings(note.raw, note.file)),
                     ...addresses.findings,
                     ...frontmatter.findings,
                     ...schemaFindings,
@@ -1198,6 +1212,7 @@ function lintCommand() {
                 // Only an **error** fails the run. `reportFindings` draws that
                 // line, so the rule is the shared one rather than a second
                 // copy here. Each finding names its own file.
+                const validationMs = performance.now() - validationStarted;
                 const errors = reportFindings(findings, {});
                 if (errors) {
                     log.error(`${findings.length} finding(s) across ${addresses.notes} note(s).`);
@@ -1214,6 +1229,10 @@ function lintCommand() {
                             `${addresses.notes} note(s)).`,
                     );
                 }
+                log.info(
+                    `Stage timings: index ${indexMs.toFixed(1)} ms; ` +
+                        `SQL ${sqlMs.toFixed(1)} ms; validation ${validationMs.toFixed(1)} ms.`,
+                );
             } catch (err) {
                 reportFailure(err);
                 process.exitCode = 1;
@@ -1424,7 +1443,7 @@ function linksCommand() {
                     config,
                     records,
                     ...scope,
-                    sqlTables: await prepareTreeSqlTables(contentBase, {
+                    sqlTables: await prepareLinkSqlTables(contentBase, {
                         config,
                         records,
                         parsed,
@@ -1602,12 +1621,15 @@ function contentIndexCommand() {
                 type: "string",
             });
         },
-        handler: (argv) => {
+        handler: async (argv) => {
             try {
                 const config = loadPackConfig();
+                const contentBase = argv.root ?? config.paths.content;
+                const publication = await prepareContentIndex({ config, contentBase });
                 const { file, notes, bytes, full, draft, stub, dictionary, terms } =
                     emitContentIndex({
                         config,
+                        publication,
                         ...(argv.root ? { contentBase: argv.root } : {}),
                         ...(argv.out ? { outDir: argv.out } : {}),
                     });

package/ci/check-banners.mjs

@@ -16,21 +16,16 @@
  *
  * `partials/hero-banner.html` needs to know which banner images exist, and Hugo
  * cannot ask a remote asset host that question at build time — so the answer is
- * declared in `data/banners.yaml` instead. A declaration nothing
- * checks is kept in step by hope, and the hole it was written to close reopens
- * the first time it drifts. It already had: when this check was written, four
- * names present in a local working copy of the artwork repository were not
- * published, and three that were published appeared in no local copy at all.
- *
- * So this fetches every declared name and fails on any non-200. It is the test
- * the template cannot make, moved to the one place that can make it.
+ * declared in `data/banners.yaml` instead. This checks every declaration and
+ * accepts HTTP 200 or 206. HTTP 404 and 410 identify absent artwork; other HTTP
+ * statuses and transport errors leave publication unverified and fail the check.
  *
  * It also refuses a name in `data/banner-kinds.yaml` that `banners.yaml` does
  * not declare, before any request is made: a mapping to undrawn art is a dead
  * URL on every page of that kind.
  *
  * Usage:
- *   node utils/check-banners.mjs [--base <url>] [--concurrency <n>]
+ *   node ci/check-banners.mjs [--base <url>] [--concurrency <n>]
  *
  * The host defaults to the Heroic Lands CDN and can be pointed anywhere with
  * `--base` or `BANNER_BASE_URL`, so a consumer publishing its own artwork set
@@ -158,6 +153,20 @@ function checkMapping(available) {
     if (undeclared.length) process.exit(1);
 }
 
+/** Fetch wraps transport errors in a cause, which can hold several connection failures. */
+function networkErrorDetails(error, seen = new Set()) {
+    if (!error || seen.has(error)) return [];
+    seen.add(error);
+    const message = [error.code, error.message].filter(Boolean).join(": ");
+    return [
+        ...(message ? [message] : []),
+        ...networkErrorDetails(error.cause, seen),
+        ...(Array.isArray(error.errors) ?
+            error.errors.flatMap((cause) => networkErrorDetails(cause, seen))
+        :   []),
+    ];
+}
+
 async function probe(url) {
     try {
         // HEAD first — it is what the question actually is. Some object stores
@@ -172,10 +181,15 @@ async function probe(url) {
         }
         return {
             ok: res.status === 200 || res.status === 206,
-            status: String(res.status),
+            missing: res.status === 404 || res.status === 410,
+            status: `HTTP ${res.status}`,
         };
     } catch (err) {
-        return { ok: false, status: `network error: ${err.message}` };
+        return {
+            ok: false,
+            missing: false,
+            status: `network error: ${networkErrorDetails(err).join("; ") || String(err)}`,
+        };
     }
 }
 
@@ -190,26 +204,30 @@ async function main() {
     const workers = Array.from({ length: Math.min(concurrency, queue.length) }, async () => {
         for (let name = queue.shift(); name !== undefined; name = queue.shift()) {
             const url = `${root}/${name}.webp`;
-            const { ok, status } = await probe(url);
-            if (!ok) failures.push({ name, url, status });
+            const { ok, missing, status } = await probe(url);
+            if (!ok) failures.push({ name, url, missing, status });
         }
     });
     await Promise.all(workers);
 
     if (failures.length) {
         failures.sort((a, b) => a.name.localeCompare(b.name));
-        for (const { name, url, status } of failures) {
+        for (const { name, url, missing, status } of failures) {
             diag(
                 lineOf(source, name),
-                `declared banner ${name} is not published (${status}): ${url}`,
+                `declared banner ${name} ${missing ? "is not published" : "could not be verified"} (${status}): ${url}`,
             );
         }
+        const missing = failures.filter((failure) => failure.missing).length;
+        const unverified = failures.length - missing;
         console.error(
-            `check-banners: ${failures.length} of ${available.length} declared banners are missing from ${base}.`,
-        );
-        console.error(
-            "Publish the artwork, or drop the name — an undrawn name here is a 404 on every page that resolves to it.",
+            `check-banners: ${failures.length} of ${available.length} declared banners failed verification at ${base} (${missing} missing, ${unverified} unverified).`,
         );
+        if (missing) console.error("Publish the missing artwork or correct its inventory entry.");
+        if (unverified)
+            console.error(
+                "Check the asset host and request failures before changing the inventory.",
+            );
         process.exit(1);
     }
 

package/container.mjs

@@ -492,8 +492,8 @@ export function s3ShellArgs(plan, command) {
 }
 
 /**
- * Start the S3 server on its network and create the bucket, unless it is
- * already running, and copy the `awsConfig` into the data root.
+ * Start or reuse the S3 server on its network, wait for readiness, ensure the
+ * bucket exists, and copy the `awsConfig` into the data root.
  *
  * @param {object} opts
  * @param {S3Plan} opts.plan - The S3 plan.
@@ -506,7 +506,8 @@ export function s3ShellArgs(plan, command) {
  * @param {(ms: number) => void} [opts.sleep] - Waits.
  * @param {number} [opts.attempts] - How often to ask the server whether it is
  *   ready.
- * @throws {Error} When an input file is missing or the server never answers.
+ * @throws {Error} When an input is not a readable regular file, a reused
+ *   identities mount is incompatible, or the S3 HTTP service never answers.
  */
 export function ensureS3({
     plan,
@@ -521,6 +522,10 @@ export function ensureS3({
         if (!fs.existsSync(file)) {
             throw new Error(`FOUNDRYVTT_CONTAINER_S3 needs ${file}, which does not exist.`);
         }
+        if (!fs.statSync(file).isFile()) {
+            throw new Error(`FOUNDRYVTT_CONTAINER_S3 needs ${file} to be a regular file.`);
+        }
+        fs.accessSync(file, fs.constants.R_OK);
     }
     const configDir = path.join(dataRoot, "Config");
     fs.mkdirSync(configDir, { recursive: true });
@@ -540,6 +545,32 @@ export function ensureS3({
         "{{.Names}}",
     ]);
     if (present.split("\n").includes(plan.server)) {
+        let identitiesMount;
+        try {
+            const mounts = JSON.parse(
+                capture(["inspect", "--format", "{{json .Mounts}}", plan.server]),
+            );
+            identitiesMount = mounts.find(
+                (mount) => mount.Destination === "/etc/seaweedfs/s3.json",
+            );
+        } catch {
+            // An unreadable inspection cannot establish that this server is safe to reuse.
+        }
+        const source = identitiesMount?.Source;
+        const canonical = (file) =>
+            fs.existsSync(file) ? fs.realpathSync(file) : path.resolve(file);
+        if (
+            identitiesMount?.Type !== "bind" ||
+            typeof source !== "string" ||
+            canonical(source) !== canonical(plan.identities)
+        ) {
+            throw new Error(
+                `S3 server '${plan.server}' has an incompatible identities mount ` +
+                    `(found ${source ?? "no readable bind"}; expected ${plan.identities}). ` +
+                    `Restore the original checkout, or remove only this disposable S3 container ` +
+                    `with 'docker rm -f ${plan.server}' and rerun. Removing it discards its bucket data.`,
+            );
+        }
         const running = capture([
             "ps",
             "--filter",
@@ -548,15 +579,22 @@ export function ensureS3({
             "{{.Names}}",
         ]);
         if (!running.split("\n").includes(plan.server)) run(["start", plan.server]);
-        return;
+    } else {
+        log(`Starting S3 server '${plan.server}' (${S3_IMAGE}) as ${S3_ALIAS}:${S3_PORT}`);
+        const status = run(s3RunArgs(plan));
+        if (status !== 0) throw new Error(`Starting the S3 server failed with status ${status}.`);
     }
-    log(`Starting S3 server '${plan.server}' (${S3_IMAGE}) as ${S3_ALIAS}:${S3_PORT}`);
-    const status = run(s3RunArgs(plan));
-    if (status !== 0) throw new Error(`Starting the S3 server failed with status ${status}.`);
-    // `weed shell` exits 0 whether or not the filer is up; it names the filers
-    // it found once there is one.
+    // A ready filer does not mean the S3 listener has started. SeaweedFS's
+    // unauthenticated health route runs on that listener.
+    const probe = [
+        "exec",
+        plan.server,
+        "sh",
+        "-c",
+        `wget -q -T 2 -O /dev/null http://127.0.0.1:${S3_PORT}/healthz && echo ready`,
+    ];
     for (let attempt = 0; ; attempt++) {
-        if (capture(s3ShellArgs(plan, "s3.bucket.list")).includes("filers: [")) break;
+        if (capture(probe).trim() === "ready") break;
         if (attempt >= attempts) {
             throw new Error(`The S3 server '${plan.server}' did not become ready.`);
         }