@heroiclands/package-build 15.0.0 → 16.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +143 -0
- package/CONTENT.md +203 -0
- package/bin/content-build.mjs +54 -0
- package/content-config.mjs +10 -0
- package/engine/content-address.mjs +21 -36
- package/engine/content-index.mjs +439 -0
- package/engine/frontmatter-lint.mjs +86 -1
- package/engine/index.mjs +3 -0
- package/package.json +1 -1
- package/types/content-config.d.mts +9 -0
- package/types/engine/content-address.d.mts +22 -34
- package/types/engine/content-index.d.mts +194 -0
- package/types/engine/index.d.mts +1 -0
|
@@ -0,0 +1,439 @@
|
|
|
1
|
+
/*
|
|
2
|
+
* This file is part of the Song of Heroic Lands (SoHL) system for Foundry VTT.
|
|
3
|
+
* Copyright (c) 2024-2026 Tom Rodriguez ("Toasty") — <toasty@heroiclands.org>
|
|
4
|
+
*
|
|
5
|
+
* This work is licensed under the GNU General Public License v3.0 (GPLv3).
|
|
6
|
+
* You may copy, modify, and distribute it under the terms of that license.
|
|
7
|
+
*
|
|
8
|
+
* For full terms, see the LICENSE.md file in the project root or visit:
|
|
9
|
+
* https://www.gnu.org/licenses/gpl-3.0.html
|
|
10
|
+
*
|
|
11
|
+
* SPDX-License-Identifier: GPL-3.0-or-later
|
|
12
|
+
*/
|
|
13
|
+
|
|
14
|
+
/**
|
|
15
|
+
* Emitting this package's content index (#224).
|
|
16
|
+
*
|
|
17
|
+
* Every content build already walks the whole note tree and parses every note's
|
|
18
|
+
* frontmatter — the pack compilers, the site build, and the content-table
|
|
19
|
+
* expander each do it — and every one of them throws the result away when it
|
|
20
|
+
* finishes. So nothing outside a build can ask a question about the content:
|
|
21
|
+
* "which beings carry no `kbcat`?", "what does this table actually select?",
|
|
22
|
+
* "did that type rename leave anything behind?" have no answer short of writing
|
|
23
|
+
* a throwaway script that re-walks the tree. Eight dead Bestiary tables shipped
|
|
24
|
+
* for weeks behind exactly that gap (#223).
|
|
25
|
+
*
|
|
26
|
+
* This module publishes the walk. One line of JSON per note, in
|
|
27
|
+
* [JSON Lines](https://jsonlines.org/) — the whole frontmatter, plus where the
|
|
28
|
+
* note sits in the tree.
|
|
29
|
+
*
|
|
30
|
+
* **The record is the note, not a projection of it.** Frontmatter is
|
|
31
|
+
* heterogeneous and open: in `sohl` it spreads 242 distinct leaf paths unevenly
|
|
32
|
+
* over 15 types, from 9 on a `macro` to 72 on a `being`, and adding a field to
|
|
33
|
+
* one type is ordinary authoring. Any format that fixes a column set would turn
|
|
34
|
+
* that authoring into a schema migration, so nothing here selects, flattens, or
|
|
35
|
+
* renames — a reader addresses `sohl.body.weight.base` because that is what the
|
|
36
|
+
* note says, which is also, not by accident, exactly what a `dataview` query
|
|
37
|
+
* writes.
|
|
38
|
+
*
|
|
39
|
+
* **JSON Lines rather than a database.** The artifact has to survive its build
|
|
40
|
+
* and be usable by anything — a person with `jq`, an editor, a CI check,
|
|
41
|
+
* another package's build. A line-per-note text file needs no server, no
|
|
42
|
+
* driver, and no schema; it diffs in a pull request, so a migration that
|
|
43
|
+
* quietly empties a category shows up as a diff rather than as a silently
|
|
44
|
+
* different binary; and it is readable by every language without an install.
|
|
45
|
+
* SQL is not forfeited by the choice — DuckDB reads JSON Lines directly, with
|
|
46
|
+
* nested access — whereas a stored schema would forfeit the open shape.
|
|
47
|
+
*
|
|
48
|
+
* **Byte-stable, because it is meant to be rebuilt.** {@link emitContentIndex}
|
|
49
|
+
* is reachable on its own (`content-build content-index`) and costs a
|
|
50
|
+
* frontmatter parse, not a build, so the honest expectation is that anyone
|
|
51
|
+
* regenerates it whenever they want rather than treating it as precious. That
|
|
52
|
+
* only holds if two runs over an unchanged tree produce an identical file, so
|
|
53
|
+
* records are ordered by content path with the note id breaking any tie — the
|
|
54
|
+
* same total order {@link selectRows} imposes for the same reason — and every
|
|
55
|
+
* object's keys are sorted, at every depth. A walk order is a directory-read
|
|
56
|
+
* order, and directory-read order is not a fact about the content.
|
|
57
|
+
*
|
|
58
|
+
* **Derived, never a source.** The index is written under `build/`, is
|
|
59
|
+
* gitignored with the rest of it, and nothing may be authored against it. It is
|
|
60
|
+
* emphatically not in `paths.stage`: that tree is mirrored destructively into a
|
|
61
|
+
* Foundry data root, so anything left there ships inside the installed system
|
|
62
|
+
* to every player, and a build artifact has no business there.
|
|
63
|
+
*
|
|
64
|
+
* @module
|
|
65
|
+
*/
|
|
66
|
+
|
|
67
|
+
import fs from "node:fs";
|
|
68
|
+
import path from "node:path";
|
|
69
|
+
|
|
70
|
+
import unidecode from "unidecode";
|
|
71
|
+
|
|
72
|
+
import { addressSlug } from "./content-address.mjs";
|
|
73
|
+
import { canonicalKey } from "./kb-manifest.mjs";
|
|
74
|
+
import { walkMarkdownTree } from "./helpers.mjs";
|
|
75
|
+
import { loadPackConfig } from "./pack-config.mjs";
|
|
76
|
+
|
|
77
|
+
/**
|
|
78
|
+
* The keys this module adds to a record, which a note therefore may not carry
|
|
79
|
+
* itself.
|
|
80
|
+
*
|
|
81
|
+
* `package` is the note's distribution unit — the configured `contentPackage`,
|
|
82
|
+
* since a note declaring its own is a hard error (package-build#56) — and it
|
|
83
|
+
* matches what the content-table expander puts on the same field, so a query
|
|
84
|
+
* reads the same value from either. `file` namespaces the note's place in the
|
|
85
|
+
* tree, again matching the expander's `file.*`.
|
|
86
|
+
*
|
|
87
|
+
* Both are checked rather than assumed: `folder` is real frontmatter on most
|
|
88
|
+
* notes, so the neighbouring names are close enough to a real key that a silent
|
|
89
|
+
* overwrite is a plausible future rather than a hypothetical one.
|
|
90
|
+
*
|
|
91
|
+
* @type {ReadonlyArray<string>}
|
|
92
|
+
*/
|
|
93
|
+
export const DERIVED_KEYS = Object.freeze([
|
|
94
|
+
"package",
|
|
95
|
+
"file",
|
|
96
|
+
"address",
|
|
97
|
+
"anchors",
|
|
98
|
+
"nameAscii",
|
|
99
|
+
"aliasesAscii",
|
|
100
|
+
]);
|
|
101
|
+
|
|
102
|
+
/**
|
|
103
|
+
* A heading, and the `{#slug}` anchor it declares.
|
|
104
|
+
*
|
|
105
|
+
* Kept identical to the pair {@link splitPages} matches, because the two must
|
|
106
|
+
* agree about what an anchor is: that pass decides which sections become
|
|
107
|
+
* addressable journal pages, and an index naming an anchor it does not produce
|
|
108
|
+
* would advertise a link that resolves nowhere. `tests/content-index.test.ts`
|
|
109
|
+
* asserts the two find the same anchors, so drift fails the suite rather than
|
|
110
|
+
* shipping.
|
|
111
|
+
*/
|
|
112
|
+
const HEADING = /^\s*(#{1,6})\s+(.+?)\s*#*\s*$/;
|
|
113
|
+
const ANCHOR = /^(.*?)\s*\{#([^}]+)\}\s*$/;
|
|
114
|
+
|
|
115
|
+
/**
|
|
116
|
+
* The `{#slug}` anchors a note's body declares, with where each one sits.
|
|
117
|
+
*
|
|
118
|
+
* Only headings carrying an explicit anchor are collected. A bare `#` heading
|
|
119
|
+
* also starts a journal page, but it declares no slug, so nothing can address
|
|
120
|
+
* it with `#…` — listing it would offer a link that cannot be written.
|
|
121
|
+
*
|
|
122
|
+
* @param {string} body - The note's markdown body, frontmatter already removed.
|
|
123
|
+
* @param {number} [bodyLine] - The 1-based file line the body starts on, from
|
|
124
|
+
* `parseMarkdownFile`. Anchors are reported at their position in the **file**,
|
|
125
|
+
* so an editor can jump straight to one; passing nothing numbers from the body.
|
|
126
|
+
* @returns {Array<{slug: string, name: string, level: number, line: number}>}
|
|
127
|
+
* In document order.
|
|
128
|
+
*/
|
|
129
|
+
export function collectAnchors(body, bodyLine = 1) {
|
|
130
|
+
const anchors = [];
|
|
131
|
+
let inCodeBlock = false;
|
|
132
|
+
const lines = String(body ?? "").split("\n");
|
|
133
|
+
|
|
134
|
+
for (let i = 0; i < lines.length; i++) {
|
|
135
|
+
// A fenced block's contents are not headings, and `#` is a comment in
|
|
136
|
+
// most of what gets fenced.
|
|
137
|
+
if (lines[i].trim().startsWith("```")) {
|
|
138
|
+
inCodeBlock = !inCodeBlock;
|
|
139
|
+
continue;
|
|
140
|
+
}
|
|
141
|
+
if (inCodeBlock) continue;
|
|
142
|
+
|
|
143
|
+
const heading = HEADING.exec(lines[i]);
|
|
144
|
+
if (!heading) continue;
|
|
145
|
+
const anchor = ANCHOR.exec(heading[2].trim());
|
|
146
|
+
if (!anchor) continue;
|
|
147
|
+
|
|
148
|
+
const slug = anchor[2].trim();
|
|
149
|
+
if (!slug) continue;
|
|
150
|
+
anchors.push({
|
|
151
|
+
slug,
|
|
152
|
+
name: anchor[1].trim(),
|
|
153
|
+
level: heading[1].length,
|
|
154
|
+
line: bodyLine + i,
|
|
155
|
+
});
|
|
156
|
+
}
|
|
157
|
+
return anchors;
|
|
158
|
+
}
|
|
159
|
+
|
|
160
|
+
/**
|
|
161
|
+
* The address a wikilink writes to reach a note, or `null` when it has none.
|
|
162
|
+
*
|
|
163
|
+
* A wikilink target is an address: `being-aurochs` locally, or
|
|
164
|
+
* `sohl-being-aurochs` from another package (`readQualifier` also accepts
|
|
165
|
+
* `being/aurochs`, the same two fields with a different separator). Both forms
|
|
166
|
+
* are already derivable from `type` and `shortcode`, which every record
|
|
167
|
+
* carries — so this field adds no information. What it adds is the *rule*:
|
|
168
|
+
* the lowercasing and the hyphen join live in one place, and a consumer that
|
|
169
|
+
* reimplements them slightly differently gets a lookup that matches nothing and
|
|
170
|
+
* says nothing about why. That is a real failure, not a hypothetical one — it
|
|
171
|
+
* is precisely how a resolver keyed on a bare `type/shortcode` silently misses
|
|
172
|
+
* every canonical `pkg-type-shortcode` entry.
|
|
173
|
+
*
|
|
174
|
+
* Derived by the same functions the link manifest and the site build use, so an
|
|
175
|
+
* index cannot disagree with either about where a note lives.
|
|
176
|
+
*
|
|
177
|
+
* @param {Record<string, any>} frontmatter - The note's parsed frontmatter.
|
|
178
|
+
* @param {string} contentPackage - The package the tree compiles as.
|
|
179
|
+
* @returns {{slug: string, canonical: string}|null} `slug` is what goes inside
|
|
180
|
+
* `[[…]]` within this package; `canonical` is the package-qualified key the
|
|
181
|
+
* manifest files the note under. `null` for a note with no type or no
|
|
182
|
+
* shortcode, which has no address at all and is stated as such rather than
|
|
183
|
+
* left for every reader to rediscover.
|
|
184
|
+
*/
|
|
185
|
+
export function noteAddress(frontmatter, contentPackage) {
|
|
186
|
+
let slug;
|
|
187
|
+
try {
|
|
188
|
+
slug = addressSlug(frontmatter);
|
|
189
|
+
} catch {
|
|
190
|
+
// Unaddressable is ordinary — a template, a stub, a note that carries
|
|
191
|
+
// no shortcode — and not this pass's business to report. The manifest
|
|
192
|
+
// emitter already reports it, where it means a missing published page.
|
|
193
|
+
return null;
|
|
194
|
+
}
|
|
195
|
+
return {
|
|
196
|
+
slug,
|
|
197
|
+
canonical: canonicalKey(contentPackage, frontmatter.type, frontmatter.shortcode),
|
|
198
|
+
};
|
|
199
|
+
}
|
|
200
|
+
|
|
201
|
+
/**
|
|
202
|
+
* Recursively sort an object's keys, so serialization is order-independent.
|
|
203
|
+
*
|
|
204
|
+
* Arrays keep their order — it is authored — but every object inside one is
|
|
205
|
+
* sorted too. Anything that is not a plain object is returned as it is.
|
|
206
|
+
*
|
|
207
|
+
* @param {unknown} value - The value to normalize.
|
|
208
|
+
* @returns {unknown} The value with every plain object's keys in sorted order.
|
|
209
|
+
*/
|
|
210
|
+
export function sortKeysDeep(value) {
|
|
211
|
+
if (Array.isArray(value)) return value.map(sortKeysDeep);
|
|
212
|
+
if (value === null || typeof value !== "object") return value;
|
|
213
|
+
// A Date or any other exotic object would lose itself in a rebuild from
|
|
214
|
+
// entries, and YAML frontmatter can produce one.
|
|
215
|
+
if (Object.getPrototypeOf(value) !== Object.prototype) return value;
|
|
216
|
+
/** @type {Record<string, unknown>} */
|
|
217
|
+
const out = {};
|
|
218
|
+
for (const key of Object.keys(value).sort()) out[key] = sortKeysDeep(value[key]);
|
|
219
|
+
return out;
|
|
220
|
+
}
|
|
221
|
+
|
|
222
|
+
/**
|
|
223
|
+
* A note's display name reduced to printable 7-bit ASCII.
|
|
224
|
+
*
|
|
225
|
+
* Content names carry the setting's orthography — `Kûrbúl Helm`, `Hârn`,
|
|
226
|
+
* `Kèthîra` — and nobody types them. A reader searching the index, or an editor
|
|
227
|
+
* completing a wikilink, needs a form that matches what a keyboard produces, so
|
|
228
|
+
* the record states one rather than leaving every consumer to invent it (and to
|
|
229
|
+
* invent a *different* one, which is how two searches over the same data come
|
|
230
|
+
* to disagree).
|
|
231
|
+
*
|
|
232
|
+
* **Transliterated, not stripped.** `unidecode` — the same table
|
|
233
|
+
* {@link slugify} already runs, so an ASCII name and a slug can never disagree
|
|
234
|
+
* about a character — carries a letter across rather than deleting it:
|
|
235
|
+
* diacritics fold (`â`→`a`, `è`→`e`), ligatures expand (`æ`→`ae`, `Œ`→`OE`,
|
|
236
|
+
* `ß`→`ss`), the runic letters spell out (`þ`→`th`, `Þ`→`Th`, `ð`→`d`), and
|
|
237
|
+
* even a vulgar fraction becomes readable (`¾`→`3/4`). Deleting them instead
|
|
238
|
+
* would collapse `Kûrbúl` to `Krbl`, which is worse than the original.
|
|
239
|
+
*
|
|
240
|
+
* Anything still outside printable ASCII after that becomes a space, and runs
|
|
241
|
+
* of whitespace collapse — a space rather than nothing, so a character that
|
|
242
|
+
* transliterates away cannot silently weld two words together.
|
|
243
|
+
*
|
|
244
|
+
* The value is emitted even when it equals the name, so a consumer matching on
|
|
245
|
+
* it never has to branch on whether the name happened to be ASCII already.
|
|
246
|
+
*
|
|
247
|
+
* @param {unknown} name - The note's `name.full`.
|
|
248
|
+
* @returns {string|null} The ASCII form, or `null` when there is no name, or
|
|
249
|
+
* nothing printable survives.
|
|
250
|
+
*/
|
|
251
|
+
export function asciiName(name) {
|
|
252
|
+
if (typeof name !== "string") return null;
|
|
253
|
+
const folded = unidecode(name)
|
|
254
|
+
.replace(/[^\x20-\x7E]/g, " ")
|
|
255
|
+
.replace(/\s+/g, " ")
|
|
256
|
+
.trim();
|
|
257
|
+
return folded === "" ? null : folded;
|
|
258
|
+
}
|
|
259
|
+
|
|
260
|
+
/**
|
|
261
|
+
* A note's `name.aliases` reduced to printable 7-bit ASCII, in order.
|
|
262
|
+
*
|
|
263
|
+
* An alias is the name a reader is at least as likely to reach for as the
|
|
264
|
+
* canonical one — `Killer Whale` for an orca, `Ice Bear` for a polar bear,
|
|
265
|
+
* `Ix'balam` for a jaguar — so anything searching or completing over the index
|
|
266
|
+
* has to match them too, and needs the same keyboard-typeable form
|
|
267
|
+
* {@link asciiName} gives the primary name.
|
|
268
|
+
*
|
|
269
|
+
* Order is the authored order, so a caller can pair an entry with the alias it
|
|
270
|
+
* came from. An alias that is not a non-empty string, or that leaves nothing
|
|
271
|
+
* printable behind, is dropped rather than left as a hole — the array is a set
|
|
272
|
+
* of names to match, and a null in it is not one.
|
|
273
|
+
*
|
|
274
|
+
* @param {unknown} aliases - The note's `name.aliases`; may be absent or null.
|
|
275
|
+
* @returns {Array<string>} Possibly empty, never null: a note with no aliases
|
|
276
|
+
* has an empty set of them, which is a fact rather than a missing value, and
|
|
277
|
+
* a consumer iterating it should not have to check first.
|
|
278
|
+
*/
|
|
279
|
+
export function asciiAliases(aliases) {
|
|
280
|
+
if (!Array.isArray(aliases)) return [];
|
|
281
|
+
return aliases.map((alias) => asciiName(alias)).filter((alias) => alias !== null);
|
|
282
|
+
}
|
|
283
|
+
|
|
284
|
+
/**
|
|
285
|
+
* Build one index record from a note's frontmatter and its place in the tree.
|
|
286
|
+
*
|
|
287
|
+
* @param {object} options - Options.
|
|
288
|
+
* @param {Record<string, any>} options.frontmatter - The note's parsed frontmatter.
|
|
289
|
+
* @param {string} options.relPath - Its path below the content root, POSIX-separated.
|
|
290
|
+
* @param {string} options.contentPackage - The package the tree compiles as.
|
|
291
|
+
* @param {string} [options.body] - The note's markdown body, for its anchors.
|
|
292
|
+
* @param {number} [options.bodyLine] - The 1-based file line the body starts on.
|
|
293
|
+
* @returns {Record<string, any>} The record, keys sorted at every depth.
|
|
294
|
+
* @throws {Error} When the note carries a key this module derives, which would
|
|
295
|
+
* otherwise be overwritten without a word.
|
|
296
|
+
*/
|
|
297
|
+
export function buildIndexRecord({ frontmatter, relPath, contentPackage, body, bodyLine }) {
|
|
298
|
+
for (const key of DERIVED_KEYS) {
|
|
299
|
+
if (Object.hasOwn(frontmatter ?? {}, key)) {
|
|
300
|
+
throw new Error(
|
|
301
|
+
`${relPath}: \`${key}:\` is derived by the content index and ` +
|
|
302
|
+
`cannot be authored — rename the frontmatter field`,
|
|
303
|
+
);
|
|
304
|
+
}
|
|
305
|
+
}
|
|
306
|
+
|
|
307
|
+
const posix = relPath.split(path.sep).join("/");
|
|
308
|
+
const folder = posix.includes("/") ? posix.slice(0, posix.lastIndexOf("/")) : "";
|
|
309
|
+
const address = noteAddress(frontmatter, contentPackage);
|
|
310
|
+
|
|
311
|
+
return /** @type {Record<string, any>} */ (
|
|
312
|
+
sortKeysDeep({
|
|
313
|
+
...frontmatter,
|
|
314
|
+
package: contentPackage,
|
|
315
|
+
address,
|
|
316
|
+
nameAscii: asciiName(frontmatter?.name?.full),
|
|
317
|
+
aliasesAscii: asciiAliases(frontmatter?.name?.aliases),
|
|
318
|
+
// Each anchor carries the link that reaches it, so a section is
|
|
319
|
+
// addressable from the index without anyone re-deriving how an
|
|
320
|
+
// anchor is spelled — and its file line, so an editor can jump
|
|
321
|
+
// there rather than search for the heading.
|
|
322
|
+
anchors: collectAnchors(body, bodyLine).map((a) => ({
|
|
323
|
+
...a,
|
|
324
|
+
link: address ? `${address.slug}#${a.slug}` : null,
|
|
325
|
+
})),
|
|
326
|
+
file: {
|
|
327
|
+
// Relative to the content root, and deliberately not absolute.
|
|
328
|
+
// An absolute path is a fact about the machine that built the
|
|
329
|
+
// index, not about the content: it would differ between two
|
|
330
|
+
// checkouts of the same tree, so the file would stop being
|
|
331
|
+
// byte-stable, and a published copy would carry someone's home
|
|
332
|
+
// directory and be wrong for every reader. Anyone holding the
|
|
333
|
+
// index knows the root it was built from, and `root + path` is
|
|
334
|
+
// the absolute form whenever it is wanted.
|
|
335
|
+
path: posix,
|
|
336
|
+
folder,
|
|
337
|
+
name: path.basename(posix, ".md"),
|
|
338
|
+
},
|
|
339
|
+
})
|
|
340
|
+
);
|
|
341
|
+
}
|
|
342
|
+
|
|
343
|
+
/**
|
|
344
|
+
* Read a content tree into index records, in the order they will be written.
|
|
345
|
+
*
|
|
346
|
+
* @param {string} contentBase - The content tree to walk.
|
|
347
|
+
* @param {object} options - Options.
|
|
348
|
+
* @param {string} options.contentPackage - The package the tree compiles as.
|
|
349
|
+
* @param {Array<string>} [options.skipDirectories] - Directory names to skip.
|
|
350
|
+
* @returns {Array<Record<string, any>>} The records, in a total order that does
|
|
351
|
+
* not depend on directory-read order.
|
|
352
|
+
*/
|
|
353
|
+
export function collectContentIndex(contentBase, { contentPackage, skipDirectories }) {
|
|
354
|
+
const records = [];
|
|
355
|
+
const walkOpts = skipDirectories ? { skipDirectories } : {};
|
|
356
|
+
|
|
357
|
+
for (const { frontmatter, body, bodyLine, absPath } of walkMarkdownTree(
|
|
358
|
+
contentBase,
|
|
359
|
+
walkOpts,
|
|
360
|
+
)) {
|
|
361
|
+
records.push(
|
|
362
|
+
buildIndexRecord({
|
|
363
|
+
frontmatter: frontmatter ?? {},
|
|
364
|
+
relPath: path.relative(contentBase, absPath),
|
|
365
|
+
contentPackage,
|
|
366
|
+
body,
|
|
367
|
+
bodyLine,
|
|
368
|
+
}),
|
|
369
|
+
);
|
|
370
|
+
}
|
|
371
|
+
|
|
372
|
+
// Content path, then the note id. The walk yields in directory-read order,
|
|
373
|
+
// which is not a fact about the content, and a rebuild that reorders lines
|
|
374
|
+
// would make every regeneration look like a change.
|
|
375
|
+
records.sort(
|
|
376
|
+
(a, b) =>
|
|
377
|
+
String(a.file.path).localeCompare(String(b.file.path), "en") ||
|
|
378
|
+
String(a.id ?? "").localeCompare(String(b.id ?? ""), "en"),
|
|
379
|
+
);
|
|
380
|
+
return records;
|
|
381
|
+
}
|
|
382
|
+
|
|
383
|
+
/**
|
|
384
|
+
* Serialize records as JSON Lines.
|
|
385
|
+
*
|
|
386
|
+
* @param {Array<Record<string, any>>} records - From {@link collectContentIndex}.
|
|
387
|
+
* @returns {string} One compact JSON object per line, newline-terminated. An
|
|
388
|
+
* empty set serializes to the empty string rather than to a lone newline, so
|
|
389
|
+
* the file is exactly the lines it holds.
|
|
390
|
+
*/
|
|
391
|
+
export function serializeContentIndex(records) {
|
|
392
|
+
if (records.length === 0) return "";
|
|
393
|
+
return `${records.map((r) => JSON.stringify(r)).join("\n")}\n`;
|
|
394
|
+
}
|
|
395
|
+
|
|
396
|
+
/**
|
|
397
|
+
* Emit this package's content index.
|
|
398
|
+
*
|
|
399
|
+
* @param {object} [options] - Options.
|
|
400
|
+
* @param {string} [options.contentBase] - The content tree; defaults to the
|
|
401
|
+
* configured `paths.content`.
|
|
402
|
+
* @param {string} [options.outDir] - Where to write; defaults to the configured
|
|
403
|
+
* `paths.contentIndex`.
|
|
404
|
+
* @param {object} [options.config] - A resolved configuration; loaded when omitted.
|
|
405
|
+
* @returns {{file: string, notes: number, bytes: number}} Where it was written,
|
|
406
|
+
* how many notes it holds, and its size.
|
|
407
|
+
* @throws {Error} When the content tree is absent, or when it yields no note at
|
|
408
|
+
* all — an empty index is indistinguishable from a mis-pointed tree, and a
|
|
409
|
+
* reader would take it as the authoritative statement that this package has
|
|
410
|
+
* no content.
|
|
411
|
+
*/
|
|
412
|
+
export function emitContentIndex({ contentBase, outDir, config } = {}) {
|
|
413
|
+
const resolved = config ?? loadPackConfig();
|
|
414
|
+
const tree = contentBase ?? resolved.paths.content;
|
|
415
|
+
const dir = outDir ?? resolved.paths.contentIndex;
|
|
416
|
+
const contentPackage = resolved.contentPackage;
|
|
417
|
+
|
|
418
|
+
if (!fs.existsSync(tree)) {
|
|
419
|
+
throw new Error(`no content tree at ${tree}`);
|
|
420
|
+
}
|
|
421
|
+
|
|
422
|
+
const records = collectContentIndex(tree, {
|
|
423
|
+
contentPackage,
|
|
424
|
+
skipDirectories: resolved.skipDirectories,
|
|
425
|
+
});
|
|
426
|
+
if (records.length === 0) {
|
|
427
|
+
throw new Error(
|
|
428
|
+
`${tree} yielded no notes, so the index would state that this ` +
|
|
429
|
+
`package has no content`,
|
|
430
|
+
);
|
|
431
|
+
}
|
|
432
|
+
|
|
433
|
+
const text = serializeContentIndex(records);
|
|
434
|
+
const file = path.join(dir, `${contentPackage}.jsonl`);
|
|
435
|
+
fs.mkdirSync(dir, { recursive: true });
|
|
436
|
+
fs.writeFileSync(file, text);
|
|
437
|
+
|
|
438
|
+
return { file, notes: records.length, bytes: Buffer.byteLength(text) };
|
|
439
|
+
}
|
|
@@ -56,7 +56,12 @@
|
|
|
56
56
|
*/
|
|
57
57
|
|
|
58
58
|
import { authoredFields } from "./field-spec.mjs";
|
|
59
|
-
import {
|
|
59
|
+
import {
|
|
60
|
+
resolveFieldValue,
|
|
61
|
+
systemBlock,
|
|
62
|
+
SYSTEM_BLOCK_KEYS,
|
|
63
|
+
unknownBlockKeys,
|
|
64
|
+
} from "./system-block.mjs";
|
|
60
65
|
import { positionInFrontmatter, positionOfFrontmatterPath } from "./diagnostics.mjs";
|
|
61
66
|
import { checkHomepageAddressFields } from "./homepage.mjs";
|
|
62
67
|
import { RETIRED_TYPES } from "./ids.mjs";
|
|
@@ -499,6 +504,83 @@ function authoredValue(fm, key) {
|
|
|
499
504
|
return fm && Object.hasOwn(fm, key) ? fm[key] : undefined;
|
|
500
505
|
}
|
|
501
506
|
|
|
507
|
+
/**
|
|
508
|
+
* Two embedded items on one actor may not share `(type, shortcode)` (#228).
|
|
509
|
+
*
|
|
510
|
+
* SoHL treats `(type, shortcode)` as a **logical identity**, not a lookup
|
|
511
|
+
* convenience: two documents of one type bearing one shortcode denote *the same
|
|
512
|
+
* entity*, whatever their `_id`s or field values. It is unique within four
|
|
513
|
+
* scopes, one of which is an actor's own embedded items — and the invariant
|
|
514
|
+
* exists to keep that identity well-defined. Two colliding entries make "the
|
|
515
|
+
* same thing" ambiguous, and every match resolving by it — compendium↔world
|
|
516
|
+
* reconciliation, archetype shadowing, `fvttFindItemByShortcode`, cohort
|
|
517
|
+
* membership, expression and effect references — becomes unsound.
|
|
518
|
+
*
|
|
519
|
+
* Nothing else catches it. The compiler resolves each entry independently and
|
|
520
|
+
* distinguishes the two only when seeding `_id`, so the collision compiles to
|
|
521
|
+
* two documents with distinct ids and ships unremarked.
|
|
522
|
+
*
|
|
523
|
+
* **Decidable from frontmatter alone**, which is why it belongs here rather
|
|
524
|
+
* than in the compiler. An entry's effective key is
|
|
525
|
+
* `system.shortcode ?? shortcode`: a top-level `shortcode` merely selects the
|
|
526
|
+
* template the entry is written from and is never written to the document,
|
|
527
|
+
* while a template's own `system.shortcode` is its address by construction. So
|
|
528
|
+
* neither the catalogue nor a compile is needed to know what an entry will
|
|
529
|
+
* carry.
|
|
530
|
+
*
|
|
531
|
+
* Only entries naming both a type and a key are compared. One naming neither —
|
|
532
|
+
* a stand-alone entry still missing its `system.shortcode` — is the compiler's
|
|
533
|
+
* finding to make, and reporting it twice helps nobody.
|
|
534
|
+
*
|
|
535
|
+
* @param {object} note - A note from the link index (`{fm, file, raw}`).
|
|
536
|
+
* @param {string} blockName - The system block whose `items` to check.
|
|
537
|
+
* @returns {object[]} One finding per collision, at the later entry.
|
|
538
|
+
*/
|
|
539
|
+
function checkEmbeddedShortcodes(note, blockName) {
|
|
540
|
+
const findings = [];
|
|
541
|
+
const block = systemBlock(note.fm ?? {}, blockName);
|
|
542
|
+
const entries = block?.items;
|
|
543
|
+
if (!Array.isArray(entries)) return findings;
|
|
544
|
+
|
|
545
|
+
/** `type\0key` → the index that claimed it first. */
|
|
546
|
+
const claimed = new Map();
|
|
547
|
+
entries.forEach((entry, index) => {
|
|
548
|
+
if (!entry || typeof entry !== "object" || Array.isArray(entry)) return;
|
|
549
|
+
const type = entry.type;
|
|
550
|
+
const system = entry.system;
|
|
551
|
+
const key =
|
|
552
|
+
system && typeof system === "object" && !Array.isArray(system) ?
|
|
553
|
+
(system.shortcode ?? entry.shortcode)
|
|
554
|
+
: entry.shortcode;
|
|
555
|
+
if (!type || !key) return;
|
|
556
|
+
|
|
557
|
+
const address = `${type}${key}`;
|
|
558
|
+
const first = claimed.get(address);
|
|
559
|
+
if (first === undefined) {
|
|
560
|
+
claimed.set(address, index);
|
|
561
|
+
return;
|
|
562
|
+
}
|
|
563
|
+
findings.push({
|
|
564
|
+
file: note.file,
|
|
565
|
+
...positionOfFrontmatterPath(note.raw ?? "", [blockName, "items", index]),
|
|
566
|
+
severity: "error",
|
|
567
|
+
message:
|
|
568
|
+
`"${type}:${key}" is already the shortcode of ` +
|
|
569
|
+
`\`${blockName}.items[${first}]\` on this actor; ` +
|
|
570
|
+
`(type, shortcode) identifies *which entity* an item is, and ` +
|
|
571
|
+
`must be unique among an actor's embedded items, so the two ` +
|
|
572
|
+
`denote one thing and every lookup by it is ambiguous. Give ` +
|
|
573
|
+
`this entry its own \`system.shortcode\`` +
|
|
574
|
+
(entry.shortcode && !entry.system?.shortcode ?
|
|
575
|
+
` — a top-level \`shortcode\` only selects the template ` +
|
|
576
|
+
`this entry is written from and never reaches the document`
|
|
577
|
+
: "") +
|
|
578
|
+
`, or delete it if it is a duplicate.`,
|
|
579
|
+
});
|
|
580
|
+
});
|
|
581
|
+
return findings;
|
|
582
|
+
}
|
|
583
|
+
|
|
502
584
|
/**
|
|
503
585
|
* Check one note against its type's schema.
|
|
504
586
|
*
|
|
@@ -732,6 +814,9 @@ export function lintNote(note, { schemas, index, vocabulary, systems = DEFAULT_S
|
|
|
732
814
|
// for `sohl`, the note type's own field names, which are still the position
|
|
733
815
|
// the corpus authors them at until #126 moves them.
|
|
734
816
|
for (const [blockName, spec] of Object.entries(systems ?? {})) {
|
|
817
|
+
// Two embedded items denoting one entity (#228). Per block, because
|
|
818
|
+
// `items` is a block key and a second system's actor carries its own.
|
|
819
|
+
findings.push(...checkEmbeddedShortcodes(note, blockName));
|
|
735
820
|
const accepted = new Set([
|
|
736
821
|
...UNIVERSAL_KEYS,
|
|
737
822
|
...(spec?.known ?? []),
|
package/engine/index.mjs
CHANGED
|
@@ -92,6 +92,9 @@ export * as kbManifest from "./kb-manifest.mjs";
|
|
|
92
92
|
/** Deriving this package's own link manifest from its content tree. */
|
|
93
93
|
export * as manifestEmit from "./manifest-emit.mjs";
|
|
94
94
|
|
|
95
|
+
/** Publishing the note tree as a queryable JSON Lines index. */
|
|
96
|
+
export * as contentIndex from "./content-index.mjs";
|
|
97
|
+
|
|
95
98
|
/** Publishing a content tree as a website: the pass, and its integrity gates. */
|
|
96
99
|
export * as siteBuild from "./site-build.mjs";
|
|
97
100
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@heroiclands/package-build",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "16.0.0",
|
|
4
4
|
"description": "Shared toolchain for building and shipping a HeroicLands Foundry VTT package — content compilation, manifest, localization, staging, bundle, release and deployment.",
|
|
5
5
|
"license": "GPL-3.0-or-later",
|
|
6
6
|
"type": "module",
|
|
@@ -41,6 +41,7 @@ export namespace DEFAULT_PATHS {
|
|
|
41
41
|
let content: "assets/content";
|
|
42
42
|
let manifests: "assets/manifests";
|
|
43
43
|
let manifestOut: "build/manifests";
|
|
44
|
+
let contentIndex: "build/content-index";
|
|
44
45
|
let packJson: "build/packs-json";
|
|
45
46
|
let stage: "build/stage/packs";
|
|
46
47
|
let unpack: "build/tmp/packs";
|
|
@@ -296,6 +297,13 @@ export type PathsInput = {
|
|
|
296
297
|
* `manifests` directory.
|
|
297
298
|
*/
|
|
298
299
|
manifestOut?: string | undefined;
|
|
300
|
+
/**
|
|
301
|
+
* Where `content-index` writes this
|
|
302
|
+
* package's note index. Outbound, and a
|
|
303
|
+
* derived artifact — never a source, and
|
|
304
|
+
* never inside `stage`.
|
|
305
|
+
*/
|
|
306
|
+
contentIndex?: string | undefined;
|
|
299
307
|
/**
|
|
300
308
|
* Build-only per-entry JSON intermediate.
|
|
301
309
|
*/
|
|
@@ -316,6 +324,7 @@ export type ResolvedPaths = {
|
|
|
316
324
|
content: string;
|
|
317
325
|
manifests: string;
|
|
318
326
|
manifestOut: string;
|
|
327
|
+
contentIndex: string;
|
|
319
328
|
packJson: string;
|
|
320
329
|
stage: string;
|
|
321
330
|
unpack: string;
|
|
@@ -21,44 +21,32 @@
|
|
|
21
21
|
*/
|
|
22
22
|
export function addressSlug(fm: object): string;
|
|
23
23
|
/**
|
|
24
|
-
* A note's address
|
|
24
|
+
* A note's address: `<type>-<shortcode>/`, e.g. `affliction-aconite/`.
|
|
25
25
|
*
|
|
26
|
-
*
|
|
27
|
-
*
|
|
28
|
-
*
|
|
29
|
-
* decides nothing about where it publishes.
|
|
30
|
-
*
|
|
31
|
-
* @param {object} fm - Parsed frontmatter.
|
|
32
|
-
* @returns {string} The mount-relative address, with a trailing slash.
|
|
33
|
-
* @throws {Error} When the note has no address.
|
|
34
|
-
*/
|
|
35
|
-
export function contentAddress(fm: object): string;
|
|
36
|
-
/**
|
|
37
|
-
* A note's address relative to its **package**, e.g. `affliction-aconite/`.
|
|
38
|
-
*
|
|
39
|
-
* This is the form the link manifest records and the site build emits pages at,
|
|
40
|
-
* and it is one function because those two must agree — a manifest asserting an
|
|
26
|
+
* This is the one form a note is addressed by. It is what the link manifest
|
|
27
|
+
* records as an entry's `path` and what the site build emits the page at, and
|
|
28
|
+
* it is one function because those two must agree — a manifest asserting an
|
|
41
29
|
* address the site does not publish resolves at build time and 404s for the
|
|
42
30
|
* reader, which is the failure this module exists to prevent.
|
|
43
31
|
*
|
|
44
|
-
* **
|
|
45
|
-
*
|
|
46
|
-
*
|
|
47
|
-
* is
|
|
48
|
-
*
|
|
49
|
-
*
|
|
50
|
-
*
|
|
51
|
-
*
|
|
52
|
-
*
|
|
53
|
-
*
|
|
54
|
-
*
|
|
55
|
-
*
|
|
56
|
-
*
|
|
57
|
-
*
|
|
58
|
-
*
|
|
59
|
-
*
|
|
60
|
-
*
|
|
61
|
-
*
|
|
32
|
+
* **The address is relative to the package**, and to nothing finer. A consumer
|
|
33
|
+
* composing a URL prepends where the package is served (`/<package>/`); a
|
|
34
|
+
* consumer composing a manifest entry measures against that same base. Nothing
|
|
35
|
+
* else is prepended: `prefix` says where the content tree *mounts inside the
|
|
36
|
+
* package* — the Hugo directory its pages are written under — and an address is
|
|
37
|
+
* `(type, shortcode)`, a package-wide identity that takes no mount, so `sohl`
|
|
38
|
+
* publishes `/sohl/affliction-aconite/` from a file written under `kb/`. The
|
|
39
|
+
* `type-` half is what keeps that flat namespace clear of the package's fixed
|
|
40
|
+
* mounts — `/<package>/` for the landing, `/<package>/api/` for generated API
|
|
41
|
+
* docs, neither of which contains a hyphen or names a type.
|
|
42
|
+
*
|
|
43
|
+
* **It is a pure function of the frontmatter**, and takes no options. Nothing
|
|
44
|
+
* about the file the note was read from reaches it: the `README.md` convention
|
|
45
|
+
* that made one note address a whole section is retired with the section itself
|
|
46
|
+
* (#204), so every note is addressed alike and there is one rule and no branch.
|
|
47
|
+
* It took an address scheme until #215, to validate a `landing` rule it then
|
|
48
|
+
* discarded; with that key retired, `prefix` was the only thing left in the
|
|
49
|
+
* scheme and the paragraph above is the reason it never applied.
|
|
62
50
|
*
|
|
63
51
|
* @param {object} fm - Parsed frontmatter.
|
|
64
52
|
* @returns {string} The package-relative address, with a trailing slash and no
|