@heroiclands/package-build 7.0.0 → 8.0.0

package/CHANGELOG.md

@@ -1,5 +1,231 @@
 # @heroiclands/package-build
 
+## 8.0.0
+
+### Major Changes
+
+- d324b5a: Stop emitting five `system` fields no SoHL DataModel declares (#60).
+  
+  The comparison this release adds was run against sohl's published schema at
+  0.8.2 and found five, on its first run:
+  
+  | type             | emitted, undeclared                                      |
+  | ---------------- | -------------------------------------------------------- |
+  | `affliction`     | `isTreated`                                              |
+  | `trauma`         | `isTreated`, `isBleeding`                                |
+  | `projectilegear` | `impactBase.overrideDice`, `impactBase.overrideModifier` |
+  
+  Every one was discarded when the document was constructed, on every compiled
+  document, with nothing said — which is the whole of what #60 is about.
+  
+  **Two of them were never storable.** `isTreated` and `isBleeding` are _derived_
+  on the logic classes: `AfflictionLogic.isTreated` is `treatmentDate != null`,
+  and `TraumaLogic.isBleeding` is `bloodLossAdvanceDurationBase != null`. So the
+  builder wrote a constant Foundry threw away while the field it is computed from
+  went unwritten — both directions of the same defect, on the same field. Nothing
+  replaces them: an untreated affliction is one whose `treatmentDate` is unset,
+  which is already the initial value.
+  
+  **Three were authored fields that vanished.** `trauma.isTreated`,
+  `trauma.isBleeding` and the two projectile overrides carried a frontmatter
+  `name`, so a note could write them — and the value went nowhere.
+  
+  **The projectile overrides are removed rather than reported upstream as missing
+  fields**, because nothing anywhere wants them: no DataModel declares them, no
+  logic class reads them, no localization key names them. A
+  launcher-versus-ammunition override may be worth having, but it would have to be
+  designed in the system first, and a content builder cannot be where it is
+  invented.
+  
+  **Checked before removing, because three were authored.** Dropping an authored
+  field turns a note that writes it from a silent loss into an unknown-key error,
+  so all six content trees were searched first: `sohl`, `thalorna`, `kethira`,
+  `harnensemble`, `harnadventures` and `hm3` write none of the five.
+  
+  **Verified.** Against sohl 0.8.2's published schema, `undeclared` falls from
+  five to zero. The twelve remaining findings are the advisory direction —
+  fields a subtype declares that no builder emits, `treatmentDate` among them —
+  and are reported rather than fatal.
+  
+  **Bump**
+  
+  _Major._ Three of the five were part of the authored frontmatter vocabulary, and
+  a note writing one is now an unknown-key error rather than a value silently
+  dropped. No content in this organisation writes them, but a consumer outside it
+  would have to delete the keys — and would find its documents unchanged, since
+  they never reached a saved document in the first place.
+
+### Minor Changes
+
+- d324b5a: Run the emitted-versus-declared field comparison in `content-build lint` (#60).
+  
+  The comparison shipped in 7.0.0 with nothing calling it, because no system had
+  published its field sets yet. `sohl` now does, so this reads the artifact and
+  runs both directions.
+  
+  **Which system, at which version, is already settled.** `stats.systemId` and
+  `stats.systemVersion` are derived rather than authored (#48) — a system package
+  is its own system, a module takes the one it requires, and the version is the
+  `compatibility.verified` it pins. So there is no second piece of configuration
+  to disagree with the first about whose schema to check against.
+  
+  Two places to find it:
+  
+  - **A system** reads its own `schema.json`, generated from its `src/`.
+  - **A module** reads the copy `content-build deps fetch` caches from the archive
+    of the version it pins — which is what makes the comparison happen at
+    `verified` rather than against whatever the system's `main` holds today.
+  
+  **The fetch now keeps the schema.** Both fetch paths already unpacked the
+  dependency's archive, but only one kept the result: a download unzips into
+  `<cache>/package/` and leaves it, while `deps fetch --from` unzips into a
+  temporary directory and deletes it. A reader looking in the unpacked tree would
+  have found the schema for one and not the other — so it is copied to one known
+  place beside the extracted items instead.
+  
+  **An absent schema is announced, not skipped in silence.** A system before its
+  first schema build, and a module pinning a version released before the artifact
+  existed, both have nothing to check against. That is not an error — but a check
+  that quietly does nothing is indistinguishable from one that passed, and this
+  issue exists because a defect went unnoticed for a release. So the run says so:
+  
+  ```text
+  No published schema for sohl@0.8.2, so emitted `system` fields are unchecked.
+  A system generates its own; a module gets one from `content-build deps fetch`.
+  ```
+  
+  **Also lands the five fixes the comparison found.** They were pushed after
+  #116's merge and so were not part of it: `affliction` and `trauma` emitted
+  `isTreated`, `trauma` emitted `isBleeding`, and `projectilegear` emitted
+  `impactBase.overrideDice` and `impactBase.overrideModifier` — none of which any
+  DataModel declares. Without them this change would have turned `sohl`'s own
+  lint red on the defects it was written to find.
+  
+  **Verified against both shapes.** Against `sohl`, the run reports zero errors
+  and twelve advisory warnings, and `lint` passes. Against `sohl-kethira-basic`,
+  whose pinned 0.8.2 archive predates the artifact, it announces the skip and
+  reports only that repository's pre-existing findings.
+  
+  **Bump**
+  
+  _Minor._ New reporting on an existing command, and a fetch that keeps one more
+  file. The error direction can fail a build that passed before — but only for a
+  package whose dependency publishes a schema, which no released version does yet.
+- ee9a9a8: Compare a builder's emitted `system` fields against the receiving DataModel
+  (#60) — the comparison half.
+  
+  Foundry discards an unknown `system` key when a document is constructed, and
+  says nothing: the value is absent at load while the build that wrote it reported
+  success. Both directions of that mismatch have already happened here, both
+  compiled clean, and both were found by set-subtracting compiled documents'
+  `system` keys against `defineSchema()` **by hand**.
+  
+  **The emitted half needs neither compilation nor parsing.** `field-spec.mjs`
+  already makes the field list the only statement of the mapping — "the
+  declaration is the builder" — so every `system` path a type can emit is
+  `field.to`, known statically. Nothing compiles a document to find out.
+  
+  **The declared half arrives as data, pinned to the declared version.** A system
+  publishes its field sets as an artifact and this reads it, the shape the link
+  manifest already uses for addresses. Against `compatibility.verified`, never the
+  system's `main`: `affiliation.subType` _is_ defined on sohl `main` and simply
+  unreleased, while `sohl-kethira-basic` pins `0.8.2` — so a check against `main`
+  passes and the field still evaporates for all 21 of its deities.
+  
+  **`own` and `inherited` are recorded apart, and the two directions read
+  different sets.** A subtype's schema spreads its parent's, so `notes`, `docHtml`
+  and the rest land on every subtype; they are the system's own runtime concerns
+  and no content builder is expected to emit them.
+  
+  | direction             | read against                                                       | severity |
+  | --------------------- | ------------------------------------------------------------------ | -------- |
+  | emitted, not declared | `own` ∪ `inherited` — the field must exist somewhere               | error    |
+  | declared, not emitted | `own` only — what the subtype adds is what its builder answers for | report   |
+  
+  Collapsing them would report every inherited field on every type: a wall of
+  findings that are all correct and none actionable.
+  
+  **A false positive the real schema caught before this shipped.** Run against
+  sohl's actual `mysticalability`, the _declared, not emitted_ direction reported
+  `charges.value` and `charges.max` on a type that populates them correctly — the
+  builder writes `charges` as a whole object and never names the leaves beneath
+  it. A declared path is now covered when the builder emits any ancestor. The
+  first real schema tried produced two false findings, which is exactly the kind
+  that teaches people to ignore a report.
+  
+  **Verified against the real declarations.** With sohl's `mysticalability`
+  schema transcribed from source, the comparison reports nothing; with #35's
+  `assocMysteryCode` reinstated, it reports exactly that field.
+  
+  **What this does not do yet.** It does not read an artifact from disk, and
+  nothing runs it in a build — those wait on a system actually publishing its
+  schemas, which is sohl's half and a separate change. The comparison, the format
+  and both regression cases are pinned here so that half has something to satisfy.
+  
+  **Bump**
+  
+  _Minor._ New surface — `engine/schema-check.mjs` and its exports — and nothing
+  existing changes behaviour. No consumer runs the comparison until an artifact
+  exists to run it against.
+- 14cd092: Add `package-build schema`, so a system publishes its DataModel field sets from
+  here rather than from its own copy of an extractor.
+  
+  The consuming half of this contract shipped in 7.0.0: `content-build lint`
+  subtracts what a package's builders emit from what a document will actually
+  receive, because Foundry discards an unknown `system` key at construction and
+  says nothing about it. The producing half lived in the first system that needed
+  it, which meant the second system to need it would have copied 491 lines — and,
+  worse, would have copied a hardcoded `SCHEMA_ARTIFACT_VERSION`, a constant this
+  package owns. Two producers stamping a third repository's constant by hand is
+  the drift worth removing before it happens rather than after: the version is now
+  imported by the producer, not restated.
+  
+  **Why here and not in each system.** A DataModel's schema is only introspectable
+  inside Foundry — `defineSchema()` returns field classes that do not exist in
+  Node — so the field sets have to be read out of the source as an AST.
+  TypeScript's parser reads plain JavaScript too, and this package already pins
+  that compiler for `coverage.mjs`. Putting the reader here means a
+  JavaScript-only system does not acquire a TypeScript pin merely to describe its
+  own data models.
+  
+  **Declared, because the two layouts in use disagree.**
+  
+  ```yaml
+  packageBuild:
+    schema:
+      Item: { from: module/data/item-models.js, registry: itemModels }
+      Actor: { from: module/data/actor-models.js, registry: actorModels }
+  ```
+  
+  One system keeps both registries in a single configuration module; the other
+  keeps one per file. Neither layout is more correct, and a convention guessing
+  between them would fail by reading _nothing_ rather than by complaining — which
+  is the worst failure available here, since an empty schema passes every check.
+  A registry that maps nothing is refused for the same reason.
+  
+  **Four spellings of inheritance, all followed.** `...Super.defineSchema()`,
+  `...super.defineSchema()`, `Object.assign(super.defineSchema(), {…})`, and a
+  subclass with no `defineSchema()` at all. The last is a real and complete
+  declaration — `class MiscGearModel extends GearModel {}` — and reading it as
+  "declares nothing" would make every field of a whole subtype look undeclared.
+  `SchemaField` nesting is recorded as dotted paths whether written bare or as
+  `fields.SchemaField`, since both spellings are in use.
+  
+  **A schema with nothing to compare against now says so.** The emitted side of
+  the check is the `fields:` of `itemBuilders`, so a package whose compendium
+  content is committed JSON rather than built from field declarations has an empty
+  one — and every field the system declares would have been reported as unemitted.
+  That is hundreds of findings whose only content is that the package does not
+  build documents that way, which is not news and not a defect. It is announced
+  once instead, for the same reason the absent-schema case is: a check that quietly
+  does nothing reads exactly like one that passed. The moment a builder declares
+  `fields:`, the comparison starts running on its own.
+  
+  **Bump**
+  
+  _Minor._ A new command, a new optional configuration key, and a `lint` that
+  reports strictly less than before.
+
 ## 7.0.0
 
 ### Major Changes

package/bin/content-build.mjs

@@ -73,6 +73,12 @@ import {
 import { renderItemFieldReference } from "../engine/field-reference.mjs";
 import { lintContentTree } from "../engine/content-lint.mjs";
 import { lintFrontmatter } from "../engine/frontmatter-lint.mjs";
+import {
+    compareFields,
+    resolveSchemaArtifact,
+    undeclaredMessage,
+    unemittedMessage,
+} from "../engine/schema-check.mjs";
 // The one vocabulary, loaded whole. Every content project authors the full type
 // set — an adventure module ships skills, beings and magic swords — so no
 // consumer gets a subset (#19, #20).
@@ -386,9 +392,79 @@ function lintCommand() {
                     references: argv.references,
                 });
 
+                // What the builders emit, against what the receiving system
+                // declares (#60). Reported here rather than at compile: it is
+                // a property of the *declarations*, not of any one note, so it
+                // is the same answer for every document and belongs where a
+                // reader is already being told about the vocabulary.
+                const schema = resolveSchemaArtifact(config);
+                const schemaFindings = [];
+                if (!schema && config.stats?.systemId) {
+                    // Said out loud, because a check that quietly does nothing
+                    // is indistinguishable from one that passed — and this
+                    // whole issue exists because a defect went unnoticed for a
+                    // release. A system before its first schema build, or a
+                    // module pinning a version released before the artifact
+                    // existed, lands here.
+                    log.info(
+                        `No published schema for ${config.stats.systemId}` +
+                            `@${config.stats.systemVersion ?? "?"}, so emitted ` +
+                            `\`system\` fields are unchecked. A system ` +
+                            `generates its own; a module gets one from ` +
+                            `\`content-build deps fetch\`.`,
+                    );
+                }
+                const fieldSpecs = config.itemFields ?? {};
+                const comparable = Object.keys(fieldSpecs).length > 0;
+                if (schema && !comparable) {
+                    // A schema, and nothing to compare it against. The emitted
+                    // side of this check is the `fields:` of `itemBuilders`, so
+                    // a package whose compendium content is committed JSON
+                    // rather than built from field declarations has an empty
+                    // one — and every field the system declares would be
+                    // reported as unemitted. That is hundreds of findings whose
+                    // only content is that this package does not build
+                    // documents this way, which is not news and not a defect.
+                    //
+                    // Said out loud rather than skipped in silence, for the
+                    // same reason the absent-schema case is: a check that
+                    // quietly does nothing reads exactly like one that passed.
+                    log.info(
+                        `Read ${config.stats?.systemId ?? "the system"}'s ` +
+                            `schema, but this package declares no ` +
+                            `\`itemBuilders\` field specifications — so there ` +
+                            `is nothing to compare it against and the ` +
+                            `emitted-versus-declared check does not apply.`,
+                    );
+                }
+                if (schema && comparable) {
+                    const { undeclared, unemitted } = compareFields({
+                        builders: fieldSpecs,
+                        artifact: schema.artifact,
+                    });
+                    for (const f of undeclared) {
+                        schemaFindings.push({
+                            file: schema.source,
+                            severity: "error",
+                            message: undeclaredMessage(f),
+                        });
+                    }
+                    for (const f of unemitted) {
+                        // Advisory: a field the system fills at runtime, or one
+                        // added ahead of the content that will use it, is not a
+                        // defect.
+                        emitDiagnostic({
+                            file: schema.source,
+                            severity: "warning",
+                            message: unemittedMessage(f),
+                        });
+                    }
+                }
+
                 const findings = [
                     ...addresses.findings,
                     ...frontmatter.findings,
+                    ...schemaFindings,
                 ];
                 for (const finding of findings) emitDiagnostic(finding);
                 if (findings.length) {

package/bin/package-build.mjs

@@ -80,6 +80,8 @@ import { hideBin } from "yargs/helpers";
 import { loadPackageBuildConfig } from "../config.mjs";
 import { loadPackConfig, packConfigPath } from "../engine/pack-config.mjs";
 import { cleanBuildArtifacts, stageAssets } from "../stage.mjs";
+import { buildSchemaArtifact } from "../engine/schema-extract.mjs";
+import { SCHEMA_ARTIFACT_FILE } from "../engine/foreign-catalog.mjs";
 import { validateLangSource } from "../lang.mjs";
 import {
     analyzeCoverage,
@@ -312,6 +314,109 @@ function assetsCommand() {
     };
 }
 
+/**
+ * Format generated text the way the repository formats everything else.
+ *
+ * Not cosmetic. A generated file that Prettier would reformat leaves
+ * `lint:format` and the generator's own `--check` each demanding what the other
+ * forbids, and the repository cannot be made green. Resolving the config from
+ * the *output path* is what makes one implementation here serve repositories
+ * with different Prettier settings.
+ *
+ * Imported on use, as `prose-lint.mjs` does, so that commands which never
+ * format do not pay to load it.
+ *
+ * @param {string} text - The unformatted content.
+ * @param {string} filepath - Where it will be written.
+ * @returns {Promise<string>} The formatted content.
+ */
+async function formatGenerated(text, filepath) {
+    const prettier = await import("prettier");
+    const config = await prettier.resolveConfig(filepath);
+    return prettier.format(text, { ...config, filepath });
+}
+
+/**
+ * `schema` — publish this package's DataModel field sets as `schema.json`.
+ *
+ * The producing half of the check `content-build lint` runs: Foundry discards
+ * an unknown `system` key at construction and says nothing, so a content build
+ * needs to know what a document will actually receive (#60). It cannot ask a
+ * running Foundry, and it cannot read `defineSchema()` from a sibling checkout,
+ * so the system publishes the field sets as data — the same shape the link
+ * manifest already uses for addresses.
+ *
+ * `--check` fails when the committed copy disagrees with what the source would
+ * produce now, because a generated file nothing checks drifts from its
+ * generator silently — and this one is read by other repositories.
+ *
+ * @returns {object} The yargs command module.
+ */
+function schemaCommand() {
+    return {
+        command: "schema",
+        describe: "Publish this package's DataModel field sets as schema.json",
+        builder: (y) =>
+            y.option("check", {
+                type: "boolean",
+                default: false,
+                describe:
+                    "Fail when the committed schema.json is out of date " +
+                    "rather than rewriting it",
+            }),
+        handler: handler(async (argv) => {
+            const config = loadPackageBuildConfig();
+            if (!config.schema.length) {
+                console.log(
+                    "package-build: no `packageBuild.schema` declared; " +
+                        "nothing to publish.",
+                );
+                return;
+            }
+
+            const pkg = readPackageJson(config);
+            const artifact = buildSchemaArtifact({
+                rootDir: config.rootDir,
+                registries: config.schema,
+                packageId: config.packageId,
+                version: pkg.version,
+            });
+
+            const out = path.join(config.rootDir, SCHEMA_ARTIFACT_FILE);
+            const text = await formatGenerated(JSON.stringify(artifact), out);
+            const counts = Object.entries(artifact.documents)
+                .map(
+                    ([kind, subtypes]) =>
+                        `${Object.keys(subtypes).length} ${kind}`,
+                )
+                .join(", ");
+
+            if (argv.check) {
+                const current =
+                    fs.existsSync(out) ? fs.readFileSync(out, "utf8") : null;
+                if (current !== text) {
+                    die(
+                        `${SCHEMA_ARTIFACT_FILE} does not match what this ` +
+                            `package's data models would produce — regenerate ` +
+                            `it with \`package-build schema\`.`,
+                    );
+                }
+                console.log(
+                    `✅ ${SCHEMA_ARTIFACT_FILE} is up to date ` +
+                        `(${counts} subtypes).`,
+                );
+                return;
+            }
+
+            fs.writeFileSync(out, text, "utf8");
+            console.log(
+                `✅ Wrote ${SCHEMA_ARTIFACT_FILE} for ${artifact.system} ` +
+                    `${artifact.systemVersion} (${counts} subtypes).`,
+            );
+        }),
+    };
+}
+
 /**
  * `manifest` — generate `system.json` / `module.json` into the build stage.
  *
@@ -927,6 +1032,7 @@ yargs(hideBin(process.argv))
     .command(cleanCommand())
     .command(assetsCommand())
     .command(manifestCommand())
+    .command(schemaCommand())
     .command(langCommand())
     .command(bundleCommand())
     .command(releaseCommand())

package/config.mjs

@@ -72,6 +72,7 @@ const SECTION_KEYS = [
     "assetTransform",
     "manifest",
     "manifestFlags",
+    "schema",
     "clean",
     "lang",
     "deploy",
@@ -227,6 +228,41 @@ function normalizeAsset(value, index) {
     });
 }
 
+/**
+ * Validate the DataModel registries `package-build schema` reads.
+ *
+ * Each entry names a file and the binding in it that maps subtype to DataModel
+ * class. Two keys rather than a convention because the two repositories that
+ * declare this disagree on both: one keeps `ITEM_DM_DEF` and `ACTOR_DM_DEF` in
+ * a single configuration module, the other keeps `itemModels` and `actorModels`
+ * in a file apiece. Neither layout is more correct, and guessing between them
+ * would fail by reading nothing rather than by complaining.
+ *
+ * The document type is the key, so a package that models only Items simply says
+ * so — there is no empty `Actor` entry to write.
+ *
+ * @param {unknown} value - The `schema` block, or `undefined`.
+ * @returns {Readonly<object[]>} Registry entries; empty when absent.
+ */
+function normalizeSchema(value) {
+    if (value === undefined) return Object.freeze([]);
+    if (!isMapping(value)) fail("packageBuild.schema", "must be a mapping");
+    const input = /** @type {Record<string, unknown>} */ (value);
+
+    const entries = Object.entries(input).map(([documentType, entry]) => {
+        const where = `packageBuild.schema.${documentType}`;
+        if (!isMapping(entry)) fail(where, "must be a mapping");
+        const row = /** @type {Record<string, unknown>} */ (entry);
+        rejectUnknownKeys(row, ["from", "registry"], `${where}.`);
+        return Object.freeze({
+            documentType,
+            from: requireNonEmptyString(row.from, `${where}.from`),
+            registry: requireNonEmptyString(row.registry, `${where}.registry`),
+        });
+    });
+    return Object.freeze(entries);
+}
+
 /**
  * Validate the manifest specification.
  *
@@ -695,6 +731,7 @@ export function resolvePackageBuildConfig(shared) {
                     "packageBuild.release.artifact",
                 ),
         assets: Object.freeze(assets),
+        schema: normalizeSchema(section.schema),
         assetTransform:
             section.assetTransform === undefined ?
                 null

package/engine/foreign-catalog.mjs

@@ -97,6 +97,51 @@ export function catalogDir(config, id, version) {
  */
 const itemsDir = (dir) => path.join(dir, "items");
 
+/**
+ * The file a system publishes its `system` field sets as (#60).
+ *
+ * @type {string}
+ */
+export const SCHEMA_ARTIFACT_FILE = "schema.json";
+
+/**
+ * Where a cached dependency's published schema sits, if it shipped one.
+ *
+ * @param {object} config - The resolved configuration.
+ * @param {string} id - The dependency's package id.
+ * @param {string} version - Its resolved version.
+ * @returns {string} The path, whether or not it exists.
+ */
+export function cachedSchemaPath(config, id, version) {
+    return path.join(catalogDir(config, id, version), SCHEMA_ARTIFACT_FILE);
+}
+
+/**
+ * Keep the dependency's published schema beside its extracted items.
+ *
+ * **Copied to one known place rather than read from where it landed.** The two
+ * fetch paths leave the unpacked archive in different states — a download
+ * unzips into `<cache>/package/` and keeps it, while `--from` unzips into a
+ * temporary directory and deletes it — so a reader that went looking in the
+ * unpacked tree would find the schema for one and not the other, which is the
+ * kind of difference that shows up as an unexplained skipped check.
+ *
+ * Absent is not an error: a system that has not adopted the artifact yet is
+ * simply unchecked, and saying so is {@link module:engine/schema-check}'s job
+ * rather than the fetch's.
+ *
+ * @param {string} root - The unpacked package root.
+ * @param {string} dir - The dependency's cache directory.
+ * @returns {boolean} Whether one was published.
+ */
+function cacheSchemaArtifact(root, dir) {
+    const src = path.join(root, SCHEMA_ARTIFACT_FILE);
+    if (!fs.existsSync(src)) return false;
+    fs.mkdirSync(dir, { recursive: true });
+    fs.copyFileSync(src, path.join(dir, SCHEMA_ARTIFACT_FILE));
+    return true;
+}
+
 /**
  * Whether a dependency's cache is present and complete.
  *
@@ -288,6 +333,7 @@ export async function fetchCatalog(config, rel) {
     await downloadAndUnzip(download, raw);
 
     await extractItemPacks(rel.id, version, manifest, raw, dir);
+    cacheSchemaArtifact(raw, dir);
     return dir;
 }
 
@@ -385,6 +431,7 @@ export async function fetchCatalogFromPath(config, rel, source) {
         const dir = catalogDir(config, rel.id, version);
         fs.rmSync(dir, { recursive: true, force: true });
         await extractItemPacks(rel.id, version, manifest, root, dir);
+        cacheSchemaArtifact(root, dir);
         log.info(`${rel.id}@${version}: cached from ${source}`);
         return dir;
     } finally {