@heroiclands/package-build 24.1.0 → 25.0.0

package/docs/guides/writing-in-invented-scripts.md

@@ -158,6 +158,7 @@ language note says which, as one map and one face:
 
 ```yaml
 data:
+  roman: "ABDEFGHIKLMNOPRSTUVWYZabdefghiklmnoprstuvwyzÁÉÍÓÚÝáéíóúýËëÀà0123456789."
   writtenIn: { map: semaron, font: semaron-kuriale }
 ```
 
@@ -300,7 +301,6 @@ data:
       - { from: "y", to: "󰉅", before: vowel } # U+F0245
       - ["y", "󰈷"] # U+F0237
       - [".", "󰉨"] # U+F0268
-      - [",", "󰉨"] # U+F0268
     fonts: [{ font: semaron-kuriale }]
 ---
 
@@ -339,8 +339,8 @@ them, none included. A mapping that drops a language's vowels writes `[a, ""]`.
   `y` is the consonant letter before a vowel and the vowel `i` elsewhere.
 - **Capitals and marks need rows of their own.** A row matches the character as
   written, so `L` and `l` are two rows, and a script with no capitals gives both
-  the same code point. The full stop and the comma are characters too: a run
-  that holds one needs a row for it, or the face-coverage check reports it.
+  the same code point. A period is a character too and needs its own row.
+  A language's `data.roman` decides which source characters may appear.
 - **A character no row matches passes through unchanged,** and the face-coverage
   check then reports it. A mapping for a face that draws nothing on roman code
   points lists every letter the spelling uses.
@@ -381,14 +381,19 @@ A row with a fault is left out of the map and the rest apply.
 
 ## How a mapping treats punctuation
 
+The language note's `data.roman` lists its exact permitted Roman source
+characters. Include capitals, accents, digits and punctuation explicitly;
+`package-build lint` checks them before mapping. A map cannot admit a character
+that the language's inventory excludes. There are no automatic punctuation
+substitutions.
+
 The toolchain is used with one convention for marks: **mood, questions and
 quotation are carried by words, not marks,** unless a language's note rules
 otherwise. A mapping therefore lists the marks its script has and nothing more.
-In the sample, a comma and a full stop are the script's one point:
+In the sample, only a period writes the script's pause point:
 
 ```yaml
 - [".", "\U000F0268"] # U+F0268
-- [",", "\U000F0268"] # U+F0268
 ```
 
 (Formatting rewrites the escapes, as above.) A character with no row passes
@@ -398,10 +403,11 @@ mapped run is an error that names the mark and the mapping:
 > `` `?` (U+003F) is not drawn by the font `semaron-kuriale`, and the map `semaron` has no row for it ``
 
 The remedy is to write the question or the quotation in words, as the language
-does, and leave the mark out of the run. A language whose note rules
-otherwise gets rows for its marks in the mapping, and the check stops
-reporting them. The same holds for the digits, the hyphen and the catch: a row
-for each makes the mark part of the script; none leaves it out.
+does, and leave the mark out of the run. A language whose note permits a mark
+includes it in `data.roman`; its script needs a row for that mark as well.
+Vylari permits period alone as punctuation. Its lexical catch is written with
+a grave-accented vowel, as in `kàralzon`, and that vowel needs its own mapping
+row; an apostrophe is not another spelling of it.
 
 ## The dictionary first
 
@@ -793,7 +799,7 @@ paragraphs: indentation, line breaks and spans survive formatting untouched.
 
 ````markdown
 ```verse {lang=vyl font=semaron-kuriale map=semaron .nativefont}
-Lokazor lonak,
+Lokazor lonak.
   lokamen zaron.
 ```
 ````
@@ -1114,6 +1120,7 @@ name: { full: Vylari Language }
 type: skill
 subType: language
 data:
+  roman: "ABDEFGHIKLMNOPRSTUVWYZabdefghiklmnoprstuvwyzÁÉÍÓÚÝáéíóúýËëÀà0123456789."
   writtenIn: { map: semaron, font: semaron-kuriale }
   core: true
   fields:
@@ -1177,7 +1184,7 @@ The lintel over the door of the oath road is cut with a couplet, and the
 stonecutter set the couplet in the script of the court, so that every traveller
 could read it as the court reads it.
 
-[Lokazor lonak thiruthrimar, lokamen zaron.]{lang=vyl font=semaron-kuriale map=semaron .nativefont}
+[Lokazor lonak thiruthrimar. lokamen zaron.]{lang=vyl font=semaron-kuriale map=semaron .nativefont}
 ```
 
 Linting reports the unknown word, at its first character:
@@ -1198,6 +1205,7 @@ name: { full: Vylari Language }
 type: skill
 subType: language
 data:
+  roman: "ABDEFGHIKLMNOPRSTUVWYZabdefghiklmnoprstuvwyzÁÉÍÓÚÝáéíóúýËëÀà0123456789."
   writtenIn: { map: semaron, font: semaron-kuriale }
   core: true
   fields:
@@ -1251,7 +1259,7 @@ The lint is now clean, and the conversion can be read. `package-build transliter
 prints the code points the build writes for the couplet:
 
 ```console
-$ package-build transliterate semaron --lang vyl --hex "Lokazor lonak, lokamen zaron."
+$ package-build transliterate semaron --lang vyl --hex "Lokazor lonak. lokamen zaron."
 U+F020A U+F023D U+F0239 U+F0230 U+F0246 U+F023D U+F023F U+0020 U+F023A U+F023D U+F023C U+F0230 U+F0239 U+F0268 U+0020 U+F023A U+F023D U+F0239 U+F0230 U+F023B U+F0233 U+F023C U+0020 U+F0246 U+F0248 U+F023F U+F023D U+F023C U+F0268
 ```
 
@@ -1264,8 +1272,8 @@ The site sets the run as
 ```
 
 where the characters are the code points above: `Lokazor` begins with the
-capital `L` of the rows, `zaron` is the exception's long vowel, and the comma
-and full stop are the script's one point.
+capital `L` of the rows, `zaron` is the exception's long vowel, and each period
+writes the script's pause point.
 
 A poem in columns and a poem with its metre checked sit on one note. Both parse
 with no finding: the first sets native Vylari in column one and English over Roman
@@ -1341,7 +1349,7 @@ The door is marked with the same words as the lintel. Although Semaron's
 convention is rubric, this Vylari span uses no Markdown emphasis: spans with a
 non-English language cannot contain bold or italic markup.
 
-[Lokazor lonak, zaron.]{lang=vyl font=semaron-kuriale map=semaron .nativefont}
+[Lokazor lonak. zaron.]{lang=vyl font=semaron-kuriale map=semaron .nativefont}
 ```
 
 Finally, the core report lists what the language still lacks:

package/docs/reference/format-details.md

@@ -2002,6 +2002,30 @@ in that column are grey; each row resets that choice. `numstep` counts logical
 rows, and `***` separates stanzas. Language and native script choices remain
 on the authored inline runs, independently for every reading.
 
+**A language note declares its Roman source inventory in `data.roman`.** It is
+one literal NFC string containing each permitted character once, including every
+capital, accent, digit and punctuation mark the language uses. For example,
+`roman: "kaKàÀ.1"` permits exactly those seven characters. There is no range
+syntax, automatic case expansion, or inventory inherited from a parent language.
+An empty string explicitly declares that the language has no written characters.
+Spaces, tabs and line breaks are structural and are not listed in the inventory.
+A missing declaration on a configured language note is an error even when no
+passage uses the language. A directly typed native sign is separate from
+Roman input only when the run explicitly selects `.nativefont`, the character
+belongs to a recognized setting-script encoding, and the selected face draws it.
+A font's ability to draw Roman punctuation does not admit that punctuation.
+
+`package-build lint` checks the visible source of language-tagged spans, divs,
+verse and scroll fences against that inventory before any map or native-word
+replacement can hide a character. Both the configured name and its BCP 47 tag
+select the same inventory. A nested language selects its own; link destinations,
+attributes, column numbers, indentation periods and `***` stanza separators are
+structure. Escaping a character does not admit it. Source spelling must be NFC;
+errors retain the original file position. Unassociated external tags such as
+`en` and `zxx` keep their ordinary behavior. Standalone `package-build markdown`
+checks markup without loading language inventories. Structured frontmatter words
+are separate from prose runs.
+
 **A face draws the code points its `fonts.json` entry lists in `codepoints`,
 and whitespace.** After the map, any other character in a run set in a face is
 an error, located at the source character it came from; through a map, at the
@@ -6095,6 +6119,7 @@ SoHL sets `system.quantity` to `1` for each compendium item.
 | `family`                                   | `Address`                                                                                                                      | `language` only: the parent language, naming a skill note with `subType: language`; this language reads the parent's words as its own — see [a language's words](#a-languages-words)           |
 | `abbreviation`                             | `string`                                                                                                                       | `language` only: the tag a dictionary prints for it, as `Prov.`; its full name where unstated — see [dictionaries and glossaries](#dictionaries-and-glossaries)                                |
 | `core`                                     | `boolean`                                                                                                                      | `language` only: `true` where the language must cover the [core vocabulary](#the-core-vocabulary)                                                                                              |
+| `roman`                                    | literal NFC string, may be empty                                                                                               | Exact permitted Roman source characters, including both cases, accents, digits and punctuation; required on each configured language note. No family inheritance or range syntax.              |
 | `grammar`                                  | `{ plural?, verbs?, agreement?, participles? }`                                                                                | Optional noun plurals, one verb pattern, agreement and participles; supplied shapes only are checked                                                                                           |
 | `grammar.plural`                           | `{ ending, becomes }[]`                                                                                                        | Literal suffix replacements in authored order; the first match wins                                                                                                                            |
 | `grammar.plural[].ending`                  | `string`, required, may be empty                                                                                               | Bare suffix to replace; empty matches every noun                                                                                                                                               |

package/docs/reference/note-types.md

@@ -468,6 +468,7 @@ These fields are accepted by every note type. A field's value can still be irrel
 | `data.fields`           | list of `{ name, holds }`                                                                                                                    | The fields the language's words are grouped under. A language skill note only.                                                                                                                                                                                                                                                                                                                        |
 | `data.words`            | list of `{ form, class, field, gloss, … }`                                                                                                   | The words this language states, one map each, in authored order; its automatic dictionary sorts them alphabetically. A language skill note only. See [a language's words](format-details.md#a-languages-words).                                                                                                                                                                                       |
 | `data.affixes`          | list of `{ form, gloss, attaches?, examples? }`                                                                                              | The affixes a word's `built` pieces may name, each with its gloss. A language skill note only.                                                                                                                                                                                                                                                                                                        |
+| `data.roman`            | a literal NFC string of permitted characters, including case, accents, digits and punctuation                                                | The exact Roman source characters this language permits, each once. No ranges, case expansion or inherited inventory; spaces and line breaks are structural. An empty string declares no written characters. Required on each configured language note.                                                                                                                                               |
 | `data.grammar`          | a map — `{ plural?, verbs?, agreement?, participles?, native? }`                                                                             | Optional noun plurals, one verb pattern, agreement, participles and the endings' logographic writings, for dictionary presentation, the mapped-run word check and logographic maps. Only the shape of supplied keys is checked; no word must follow or state a rule. A language skill note only.                                                                                                      |
 | `data.tongues`          | list of `{ tag, meaning, language? }`                                                                                                        | The tags a register row's `tongue` may name, each with what it means. A language skill note only.                                                                                                                                                                                                                                                                                                     |
 | `data.register`         | list of `{ name, tongue?, kind?, built?, gloss?, why? }`                                                                                     | Every name the setting gives in the language's reach, with its tongue and pieces; the notes carrying each name are derived. A language skill note only.                                                                                                                                                                                                                                               |

package/engine/build-assets.mjs

@@ -0,0 +1,179 @@
+/*
+ * This file is part of the Song of Heroic Lands (SoHL) system for Foundry VTT.
+ * Copyright (c) 2024-2026 Tom Rodriguez ("Toasty") — <toasty@heroiclands.org>
+ *
+ * This work is licensed under the GNU General Public License v3.0 (GPLv3).
+ * You may copy, modify, and distribute it under the terms of that license.
+ *
+ * For full terms, see the LICENSE.md file in the project root or visit:
+ * https://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * SPDX-License-Identifier: GPL-3.0-or-later
+ */
+
+import fs from "node:fs";
+import path from "node:path";
+import { pathToFileURL } from "node:url";
+import { loadPackageBuildConfig } from "../config.mjs";
+import { loadPackConfig, packConfigPath } from "./pack-config.mjs";
+import { checkForeignAssetBindings } from "./asset-bindings.mjs";
+import { formatDiagnostic, positionOfYamlPath } from "./diagnostics.mjs";
+import { stageAssets } from "../stage.mjs";
+import { rowsStagingPictureRoots, stagePackageAssets } from "./asset-shipping.mjs";
+import { baseStyleStageEntry } from "./base-styles.mjs";
+import { checkScriptFonts, copyScriptFonts, STAGED_FONTS_DIR } from "./script-font-delivery.mjs";
+
+/**
+ * A finding for each `packageBuild.assets` row that would stage a picture root
+ * itself, located at the row's `from` when the configuration is YAML.
+ *
+ * @param {readonly {from: string, to: string}[]} rows - The configured rows.
+ * @param {{rootDir: string, assetsBase: string}} opts - Where they resolve.
+ * @returns {object[]} The findings.
+ */
+function pictureRootRowFindings(rows, { rootDir, assetsBase }) {
+    const offending = rowsStagingPictureRoots(rows, { rootDir, assetsBase });
+    if (!offending.length) return [];
+    const file = packConfigPath();
+    const text =
+        file.endsWith(".yaml") || file.endsWith(".yml") ? fs.readFileSync(file, "utf8") : "";
+    return offending.map(({ index, from, root }) => ({
+        file,
+        ...(text ? positionOfYamlPath(text, ["packageBuild", "assets", index, "from"]) : {}),
+        severity: "error",
+        message:
+            `\`packageBuild.assets[${index}]\` copies \`${from}\`, which reaches ${root}/ — ` +
+            "the stage copies the icon and image roots itself, flat and filtered by " +
+            "each file's provenance, so a row copying them would ship every file " +
+            "unfiltered. Remove the row",
+    }));
+}
+
+/**
+ * The repository's `packageBuild.assetTransform`, loaded.
+ *
+ * One loader for every command that writes a picture, so the stage and the CDN
+ * hold the same bytes for the same file.
+ *
+ * @param {object} config - The resolved package-build configuration.
+ * @returns {Promise<((sourcePath: string) => string|null|undefined)|undefined>}
+ *   The transform, or `undefined` when the repository names none.
+ */
+export async function loadAssetTransform(config) {
+    if (!config.assetTransform) return undefined;
+    const module = await import(pathToFileURL(config.assetTransform).href).catch((err) => {
+        throw new Error(
+            `cannot load \`packageBuild.assetTransform\` (${config.assetTransform}): ${err.message}`,
+        );
+    });
+    const transform = module.transform;
+    if (typeof transform !== "function") {
+        throw new Error(
+            `\`packageBuild.assetTransform\` ` +
+                `(${config.assetTransform}) exports no \`transform\` ` +
+                `function. It must export ` +
+                `\`transform(sourcePath) -> string | null\`.`,
+        );
+    }
+    return transform;
+}
+
+/**
+ * Stage static files, provenance-filtered pictures, base styles and fonts.
+ * @param {object} [options]
+ * @param {ReturnType<typeof loadPackageBuildConfig>} [options.config]
+ * @param {import("../content-config.mjs").ContentBuildConfig} [options.packConfig]
+ * @param {import("./content-session.mjs").ContentSession} [options.session]
+ */
+export async function stagePackageBuildAssets({
+    config = loadPackageBuildConfig(),
+    packConfig = loadPackConfig(),
+    session,
+} = {}) {
+    const assetsBase = packConfig.paths.assets;
+    const findings = [
+        ...pictureRootRowFindings(config.assets, {
+            rootDir: config.rootDir,
+            assetsBase,
+        }),
+        ...checkForeignAssetBindings(packConfig, { assets: config.assets }),
+    ];
+    if (findings.length) {
+        const error = new Error(findings.map(formatDiagnostic).join("\n"));
+        error.located = true;
+        throw error;
+    }
+
+    const transform = await loadAssetTransform(config);
+
+    // `to:` is relative to the staged package root, so a
+    // repository's table reads `lang`, not `build/stage/lang`.
+    const entries = config.assets.map(({ from, to }) => [from, path.join(config.stageDir, to)]);
+    let { entries: count, files } = stageAssets(entries, {
+        cwd: config.rootDir,
+        transform,
+    });
+
+    // The picture roots, staged by the toolchain rather than by a row:
+    // flat, filtered by each file's provenance, and listed in the
+    // package's asset set.
+    const pictures = stagePackageAssets({
+        assetsBase,
+        contentPackage: packConfig.contentPackage,
+        stageDir: path.resolve(config.rootDir, config.stageDir),
+        // The manifest's title names the set; a package with none is
+        // named by its content package.
+        title:
+            typeof config.manifest?.title === "string" && config.manifest.title.trim() ?
+                config.manifest.title
+            :   packConfig.contentPackage,
+        transform,
+        shippingLicenses: config.shippingLicenses,
+    });
+    if (pictures.findings.length) {
+        const error = new Error(pictures.findings.map(formatDiagnostic).join("\n"));
+        error.located = true;
+        throw error;
+    }
+    files += pictures.staged.length;
+    if (pictures.staged.length || pictures.excluded.length) {
+        console.log(
+            `Pictures: ${pictures.staged.length} staged, ` +
+                `${pictures.excluded.length} left out by provenance.`,
+        );
+    }
+
+    // Staged in its own pass, without the repository's transform: the
+    // sheet is this toolchain's file rather than one of the
+    // repository's assets, and a `transform` written to retheme a
+    // package's own SVGs has no business being handed it.
+    if (config.baseStyles) {
+        const staged = stageAssets([baseStyleStageEntry(config.stageDir)], {
+            cwd: config.rootDir,
+        });
+        count += staged.entries;
+        files += staged.files;
+    }
+
+    // The installed fonts, when the package has installed the fonts
+    // package and a note sets text in one of its faces: the
+    // stylesheet, the WOFF2 files and the licence, which the
+    // manifest's `styles` names.
+    const fontFindings = checkScriptFonts(packConfig, session);
+    if (fontFindings.length) {
+        const error = new Error(fontFindings.map(formatDiagnostic).join("\n"));
+        error.located = true;
+        throw error;
+    }
+    const fonts = copyScriptFonts(
+        packConfig,
+        path.resolve(config.rootDir, config.stageDir, STAGED_FONTS_DIR),
+        ["woff2"],
+        session,
+    );
+    if (fonts) {
+        count += 1;
+        files += fonts;
+    }
+    console.log(`✅ Static assets staged (${count} entries, ${files} files).`);
+}

package/engine/calendar-artifacts.mjs

@@ -77,16 +77,20 @@ function readIndex(file) {
  *
  * @param {object} opts
  * @param {object} opts.config - Resolved content configuration.
+ * @param {Map<string, {raw: string}>} [opts.sources] - Captured authored text for diagnostics.
  * @param {string} opts.calendariaVersion - Version whose import format is targeted.
  * @returns {{directory: string, calendars: number, files: number}}
  */
-export function emitCalendarArtifacts({ config, calendariaVersion } = {}) {
+export function emitCalendarArtifacts({ config, calendariaVersion, sources } = {}) {
     if (!config) throw new Error("calendar emission requires a content configuration");
     if (!/^\d+\.\d+\.\d+$/.test(calendariaVersion ?? "")) {
         throw new Error("calendar emission requires a Calendaria version such as 1.4.2");
     }
 
     const root = config.rootDir;
+    const rawSource = (file) =>
+        sources?.get(path.resolve(root, file))?.raw ??
+        fs.readFileSync(path.resolve(root, file), "utf8");
     const contentBase = path.resolve(root, config.paths.content);
     const indexFile = path.resolve(
         root,
@@ -108,7 +112,7 @@ export function emitCalendarArtifacts({ config, calendariaVersion } = {}) {
     );
     const names = new Map();
     for (const note of calendars) {
-        const raw = fs.readFileSync(path.resolve(root, note.file), "utf8");
+        const raw = rawSource(note.file);
         const shortcode = note.fm.shortcode;
         if (typeof shortcode !== "string" || !/^[a-z0-9]+$/.test(shortcode)) {
             failAt(
@@ -135,7 +139,7 @@ export function emitCalendarArtifacts({ config, calendariaVersion } = {}) {
         failAt(
             note.file,
             "calendar notes require a world note declaring data.year",
-            fs.readFileSync(path.resolve(root, note.file), "utf8"),
+            rawSource(note.file),
             ["subType"],
         );
     }

package/engine/compendiums.mjs

@@ -96,6 +96,7 @@ export async function compilePacks({
         contentBase: config.paths.content,
         skipDirectories: config.skipDirectories,
     });
+    const measured = (name, work) => (session ? session.timings.measure(name, work) : work());
     const packNames = sourcePacks.filter((name) => !packName || name === packName);
 
     // A prebuilt pack's JSON is checked in rather than generated, so it is
@@ -114,12 +115,14 @@ export async function compilePacks({
     // content tree, and a package that ships only prebuilt packs has none at
     // all — so running it would fail the build over a tree nothing reads.
     if (needsGeneration) {
-        const errors = await generatePacksJson({
-            only: packName,
-            config,
-            session,
-            reportPreparation,
-        });
+        const errors = await measured("rendering/documents", () =>
+            generatePacksJson({
+                only: packName,
+                config,
+                session,
+                reportPreparation,
+            }),
+        );
         if (errors > 0) {
             throw new Error(
                 `Pack JSON generation reported ${errors} error(s); refusing to ` +
@@ -141,11 +144,13 @@ export async function compilePacks({
 
         const stage = path.join(stageDest, name);
         log.info(`Pack ${name}: compiling to LevelDB at ${stage}`);
-        await compilePack(source, stage, {
-            recursive: true,
-            log: false,
-            transformEntry: cleanPackEntry,
-        });
+        await measured("pack compilation", () =>
+            compilePack(source, stage, {
+                recursive: true,
+                log: false,
+                transformEntry: cleanPackEntry,
+            }),
+        );
 
         // A Scene's map image lives on an embedded Level, stored under its own
         // LevelDB key. Nothing in Foundry ties the two together on read: a

package/engine/content-charset.mjs

@@ -176,6 +176,17 @@ const SCRIPT_BLOCKS = Object.freeze([
     [0x13430, 0x1345f], // the Egyptian format controls Zethu uses
 ]);
 
+/**
+ * Whether a code point belongs to a declared setting-script encoding block.
+ * This identifies native input independently of a language's Roman inventory;
+ * a selected native face must still draw the individual character.
+ * @param {number} cp - A Unicode code point.
+ * @returns {boolean}
+ */
+export function isSettingScriptCodePoint(cp) {
+    return SCRIPT_BLOCKS.some(([from, to]) => cp >= from && cp <= to);
+}
+
 /**
  * Whether the charset admits a code point anywhere in a note.
  *
@@ -184,10 +195,7 @@ const SCRIPT_BLOCKS = Object.freeze([
  */
 export function isAllowedCodePoint(cp) {
     return (
-        isLetterTier(cp) ||
-        TYPOGRAPHY.has(cp) ||
-        NOTATION.has(cp) ||
-        SCRIPT_BLOCKS.some(([from, to]) => cp >= from && cp <= to)
+        isLetterTier(cp) || TYPOGRAPHY.has(cp) || NOTATION.has(cp) || isSettingScriptCodePoint(cp)
     );
 }
 

package/engine/content-format.yaml

@@ -200,6 +200,7 @@ types:
             - other
     skill:
         data:
+            - roman
             - grammar
             - templatePriority
             - masteryLevel