@heroiclands/package-build 15.0.0 → 16.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +143 -0
- package/CONTENT.md +203 -0
- package/bin/content-build.mjs +54 -0
- package/content-config.mjs +10 -0
- package/engine/content-address.mjs +21 -36
- package/engine/content-index.mjs +439 -0
- package/engine/frontmatter-lint.mjs +86 -1
- package/engine/index.mjs +3 -0
- package/package.json +1 -1
- package/types/content-config.d.mts +9 -0
- package/types/engine/content-address.d.mts +22 -34
- package/types/engine/content-index.d.mts +194 -0
- package/types/engine/index.d.mts +1 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,148 @@
|
|
|
1
1
|
# @heroiclands/package-build
|
|
2
2
|
|
|
3
|
+
## 16.0.0
|
|
4
|
+
|
|
5
|
+
### Major Changes
|
|
6
|
+
|
|
7
|
+
- d734ac9: **Two embedded items on one actor may no longer share `(type, shortcode)`.**
|
|
8
|
+
`content-build lint` reports each collision as an error, at the later entry.
|
|
9
|
+
|
|
10
|
+
SoHL treats `(type, shortcode)` as a **logical identity** rather than a lookup
|
|
11
|
+
convenience: two documents of one type bearing one shortcode denote _the same
|
|
12
|
+
entity_, whatever their `_id`s or field values. It is unique within four scopes,
|
|
13
|
+
one of which is an actor's own embedded items — and the invariant exists to keep
|
|
14
|
+
that identity well-defined. Two colliding entries make "the same thing"
|
|
15
|
+
ambiguous, and every match that resolves by it — compendium↔world
|
|
16
|
+
reconciliation, archetype shadowing, `fvttFindItemByShortcode`, cohort
|
|
17
|
+
membership, expression and effect references — becomes unsound.
|
|
18
|
+
|
|
19
|
+
Nothing caught it. The compiler resolves each entry independently and
|
|
20
|
+
distinguishes the two only when seeding `_id`, so a collision compiled to two
|
|
21
|
+
documents with distinct ids and shipped unremarked:
|
|
22
|
+
|
|
23
|
+
```yaml
|
|
24
|
+
items:
|
|
25
|
+
- { shortcode: swim, type: skill, system: { masteryLevelBase: 30 } }
|
|
26
|
+
- { shortcode: swim, type: skill, initSkillMult: 1 }
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Neither entry overrides `system.shortcode`, so both inherit `swim` from the
|
|
30
|
+
template and compile to two `skill` items keyed `swim` on one actor.
|
|
31
|
+
|
|
32
|
+
**The rule is decidable from frontmatter alone**, which is why it is a lint and
|
|
33
|
+
not a compile step. An entry's effective key is `system.shortcode ?? shortcode`:
|
|
34
|
+
a top-level `shortcode` merely selects the template the entry is written from
|
|
35
|
+
and never reaches the document, while a template's own `system.shortcode` is its
|
|
36
|
+
address by construction. Neither the catalogue nor a compile is needed to know
|
|
37
|
+
what an entry will carry.
|
|
38
|
+
|
|
39
|
+
Two entries written from one template are still fine when the second says which
|
|
40
|
+
entity it is — `{ shortcode: Dgr, type: weapongear, name: "Dagger 2", system: { shortcode: Dgr2 } }`.
|
|
41
|
+
The key is the _pair_, so two different types may share a shortcode, and an
|
|
42
|
+
entry naming no key at all is left to the compiler, which already reports it.
|
|
43
|
+
|
|
44
|
+
**Major**, because a tree carrying a collision goes red on adoption. Known at
|
|
45
|
+
the time of writing: `Song-of-Heroic-Lands-FoundryVTT` 1 actor,
|
|
46
|
+
`sohl-thalorna` 15, `sohl-kethira-basic` 0.
|
|
47
|
+
- 2e0d32e: **A note's address has one name: `packageAddress`.** `engine/content-address.mjs`
|
|
48
|
+
exported it twice — as `contentAddress` and as `packageAddress` — and the two
|
|
49
|
+
bodies had become byte-identical:
|
|
50
|
+
|
|
51
|
+
```js
|
|
52
|
+
// before // after
|
|
53
|
+
contentAddress(fm); // "doc-gear/" packageAddress(fm); // "doc-gear/"
|
|
54
|
+
packageAddress(fm); // "doc-gear/"
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
`contentAddress` is removed. Import `packageAddress` from
|
|
58
|
+
`@heroiclands/package-build/engine/content-address` instead; the two returned
|
|
59
|
+
the same string, so nothing else changes.
|
|
60
|
+
|
|
61
|
+
The names once meant different things — a note's address _within the content
|
|
62
|
+
tree_, and its address _relative to the package_. They could differ while a
|
|
63
|
+
`README.md` addressed its section rather than itself and while a page's URL was
|
|
64
|
+
derived from `name.full`; the first went with the landing rules (#204, #208) and
|
|
65
|
+
the second when a page's URL became its address (#181). What was left was two
|
|
66
|
+
exported names for one notion, on a public subpath, with `contentAddress`'s
|
|
67
|
+
documentation still describing an address "below the knowledgebase mount" that
|
|
68
|
+
it no longer returned.
|
|
69
|
+
|
|
70
|
+
`packageAddress` is the name that survives because it still says something true
|
|
71
|
+
and load-bearing: the string is measured **from the package**, so a caller
|
|
72
|
+
composing a URL or a manifest `path` prepends where the package is served. That
|
|
73
|
+
is the one qualification a reader of the call site needs, and it is exactly the
|
|
74
|
+
distinction the `url:` change in 15.0.0 turned on. Closes #226.
|
|
75
|
+
|
|
76
|
+
### Minor Changes
|
|
77
|
+
|
|
78
|
+
- e131f7b: Publish the note tree as a queryable index, instead of throwing every build's walk away.
|
|
79
|
+
|
|
80
|
+
Every content build parses every note's frontmatter — the pack compilers, the site
|
|
81
|
+
build, and the content-table expander each do it — and every one of them discards
|
|
82
|
+
the result. Nothing outside a build could therefore ask a question about the
|
|
83
|
+
content: _which beings carry no `kbcat`_, _what does this table actually select_,
|
|
84
|
+
_did that type rename leave anything behind_ each needed a throwaway script that
|
|
85
|
+
re-walked the tree. Eight dead tables shipped for weeks behind exactly that gap.
|
|
86
|
+
|
|
87
|
+
`content-build content-index` emits one JSON Lines record per note — the whole
|
|
88
|
+
frontmatter, plus a derived `file` (`path`, `folder`, `name`) and the configured
|
|
89
|
+
`package` — so a question is one line of `jq`, and so an editor, a CI check, or
|
|
90
|
+
another package's build can read the content without re-deriving it.
|
|
91
|
+
|
|
92
|
+
**The record is the note, not a projection of it.** Nothing is selected, flattened,
|
|
93
|
+
or renamed; a reader addresses `sohl.body.weight.base` because that is what the note
|
|
94
|
+
says, which is also exactly what a `dataview` query writes. The refusal to impose a
|
|
95
|
+
schema is deliberate: `sohl`'s frontmatter spreads 242 distinct leaf paths unevenly
|
|
96
|
+
over 15 types, from 9 on a `macro` to 72 on a `being`, so a fixed column set would
|
|
97
|
+
turn ordinary authoring into a schema migration. `package` and `file` are the two
|
|
98
|
+
derived keys, and a note carrying either is an error rather than a silent overwrite.
|
|
99
|
+
|
|
100
|
+
**Derived, disposable, byte-stable.** It writes to the new `paths.contentIndex`
|
|
101
|
+
(`build/content-index/<package>.jsonl`) — never `paths.stage`, which is mirrored
|
|
102
|
+
into a Foundry data root — so nothing may be authored against it and it need never
|
|
103
|
+
be committed. Regenerating costs a frontmatter parse rather than a build, which is
|
|
104
|
+
why it is a command of its own; and because rebuilding is the intended use, records
|
|
105
|
+
are ordered by content path with the note id breaking ties and every object's keys
|
|
106
|
+
are sorted at every depth, so a rebuild over an unchanged tree is a no-op. A tree
|
|
107
|
+
that yields no note is an error, not an empty index: a reader takes the file as
|
|
108
|
+
authoritative, and "this package has no content" is indistinguishable from a
|
|
109
|
+
mis-pointed tree.
|
|
110
|
+
|
|
111
|
+
**Every note's address, and every anchor it defines.** A record states
|
|
112
|
+
`address.slug` (what goes inside `[[…]]` locally) and `address.canonical` (the
|
|
113
|
+
package-qualified key the manifest files it under), plus every `{#slug}` anchor the
|
|
114
|
+
body declares — each with its heading name, level, **line in the file**, and the
|
|
115
|
+
`slug#anchor` link that reaches it. Neither address is new information, since both
|
|
116
|
+
derive from `type` and `shortcode`; what the fields add is the rule, derived through
|
|
117
|
+
the same `addressSlug` and `canonicalKey` the manifest and site build use, so an
|
|
118
|
+
index cannot disagree with either about where a note lives. The payoff is that a
|
|
119
|
+
wikilink — anchor and all — becomes checkable by lookup rather than by re-parsing
|
|
120
|
+
the tree, and an editor can jump to a section instead of searching for it. What
|
|
121
|
+
counts as an anchor is kept identical to what `splitPages` matches, and a test
|
|
122
|
+
asserts the two agree.
|
|
123
|
+
|
|
124
|
+
**A keyboard-typeable form of every name.** `nameAscii` states `name.full`
|
|
125
|
+
reduced to printable 7-bit ASCII — `Kûrbúl ¾-Helm` → `Kurbul 3/4-Helm`, `Kèthîra`
|
|
126
|
+
→ `Kethira`, `Ærling` → `AErling`, `Þorn` → `Thorn`, `Straße` → `Strasse`. Names
|
|
127
|
+
carry the setting's orthography and nobody types them, so anything searching or
|
|
128
|
+
completing over the index needs a form a keyboard produces; stating one means
|
|
129
|
+
every consumer folds the same way instead of each inventing its own and two
|
|
130
|
+
searches over the same data disagreeing. It transliterates rather than strips,
|
|
131
|
+
through the same `unidecode` table `slugify` already runs — so an ASCII name and
|
|
132
|
+
a slug cannot disagree about a character, and `Kûrbúl` does not become `Krbl`.
|
|
133
|
+
Emitted even when it equals the name, so a consumer never branches on whether a
|
|
134
|
+
name happened to be ASCII; `null` only when the note has no name. `aliasesAscii`
|
|
135
|
+
does the same for `name.aliases` in the authored order — an alias is the name a
|
|
136
|
+
reader is at least as likely to reach for (`Killer Whale` for an orca), so a
|
|
137
|
+
search has to match it too — and is an empty array rather than null when a note
|
|
138
|
+
has none.
|
|
139
|
+
|
|
140
|
+
`file.path` stays relative to the content root and is deliberately never absolute:
|
|
141
|
+
an absolute path is a fact about the machine that built the index, so it would break
|
|
142
|
+
byte-stability between checkouts and publish someone's home directory.
|
|
143
|
+
|
|
144
|
+
Closes #224.
|
|
145
|
+
|
|
3
146
|
## 15.0.0
|
|
4
147
|
|
|
5
148
|
### Major Changes
|
package/CONTENT.md
CHANGED
|
@@ -73,6 +73,9 @@ paths:
|
|
|
73
73
|
manifests: assets/manifests
|
|
74
74
|
# Where `manifest` writes this package's own. Outbound, and a build artifact.
|
|
75
75
|
manifestOut: build/manifests
|
|
76
|
+
# Where `content-index` writes this package's note index. Derived and
|
|
77
|
+
# disposable — never a source, and never inside `stage`.
|
|
78
|
+
contentIndex: build/content-index
|
|
76
79
|
packJson: build/packs-json
|
|
77
80
|
stage: build/stage/packs
|
|
78
81
|
unpack: build/tmp/packs
|
|
@@ -713,6 +716,7 @@ npx content-build links [root] [--manifests <dir>]
|
|
|
713
716
|
npx content-build format [paths..] [--write]
|
|
714
717
|
npx content-build markdown [paths..] [--fix]
|
|
715
718
|
npx content-build manifest [root] [--out <dir>]
|
|
719
|
+
npx content-build content-index [root] [--out <dir>]
|
|
716
720
|
npx content-build site [--out <dir>]
|
|
717
721
|
npx content-build reachability <dir> [file] [--index <shortcode>]
|
|
718
722
|
npx content-build addresses diff --from <zip|dir> [--strict]
|
|
@@ -728,6 +732,7 @@ npx content-build addresses diff --from <zip|dir> [--strict]
|
|
|
728
732
|
| `format` | Prettier, with the shared configuration. See [Prose: formatting and markdown](#prose-formatting-and-markdown). |
|
|
729
733
|
| `markdown` | markdownlint, with the shared rule set — the structure Prettier is indifferent to. |
|
|
730
734
|
| `manifest` | Emit this package's cross-package link manifest. See [Publishing a link manifest](#publishing-a-link-manifest). |
|
|
735
|
+
| `content-index` | Emit this package's note index as JSON Lines. See [Publishing a content index](#publishing-a-content-index). |
|
|
731
736
|
| `site` | Publish the content tree as a website. See [Publishing a website](#publishing-a-website). |
|
|
732
737
|
| `reachability` | Walk outward from an index note and report what no path reaches, for a tree meant to be navigable from one entry point. |
|
|
733
738
|
| `addresses` | Report every published item address this build has stopped publishing. See [Diffing published addresses](#diffing-published-addresses). |
|
|
@@ -1390,6 +1395,204 @@ It is **reported and omitted**, never guessed: the command prints one located
|
|
|
1390
1395
|
diagnostic per note and still writes the file, because a note with no address is
|
|
1391
1396
|
ordinary while a manifest entry pointing at a page that does not exist is not.
|
|
1392
1397
|
|
|
1398
|
+
## Publishing a content index
|
|
1399
|
+
|
|
1400
|
+
Every content build walks the whole note tree and parses every note's
|
|
1401
|
+
frontmatter — the pack compilers, the site build, and the content-table expander
|
|
1402
|
+
each do it — and every one of them throws the result away. So nothing outside a
|
|
1403
|
+
build can ask a question about the content. "Which beings carry no `kbcat`?",
|
|
1404
|
+
"what does this table actually select?", "did that type rename leave anything
|
|
1405
|
+
behind?" have no answer short of writing a throwaway script that re-walks the
|
|
1406
|
+
tree, which is how eight dead Bestiary tables came to ship for weeks unnoticed.
|
|
1407
|
+
|
|
1408
|
+
`content-index` publishes the walk:
|
|
1409
|
+
|
|
1410
|
+
```bash
|
|
1411
|
+
npx content-build content-index
|
|
1412
|
+
# sohl → build/content-index/sohl.jsonl (1606 notes, 1578 KiB)
|
|
1413
|
+
```
|
|
1414
|
+
|
|
1415
|
+
One line of [JSON Lines](https://jsonlines.org/) per note, holding the note's
|
|
1416
|
+
whole frontmatter plus where it sits in the tree:
|
|
1417
|
+
|
|
1418
|
+
```json
|
|
1419
|
+
{
|
|
1420
|
+
"type": "being",
|
|
1421
|
+
"shortcode": "aurochs",
|
|
1422
|
+
"package": "sohl",
|
|
1423
|
+
"file": { "path": "Bestiary/Animal/Aurochs.md", "folder": "Bestiary/Animal", "name": "Aurochs" },
|
|
1424
|
+
"sohl": { "kbcat": "animal", "body": { "weight": { "base": 1500 } } }
|
|
1425
|
+
}
|
|
1426
|
+
```
|
|
1427
|
+
|
|
1428
|
+
so a question is one line of `jq`:
|
|
1429
|
+
|
|
1430
|
+
```bash
|
|
1431
|
+
jq -r 'select(.type == "being" and .sohl.kbcat == "animal") | .shortcode' \
|
|
1432
|
+
build/content-index/sohl.jsonl
|
|
1433
|
+
```
|
|
1434
|
+
|
|
1435
|
+
### The record is the note, not a projection of it
|
|
1436
|
+
|
|
1437
|
+
Nothing is selected, flattened, or renamed. A reader addresses
|
|
1438
|
+
`sohl.body.weight.base` because that is what the note says — which is also,
|
|
1439
|
+
not by accident, exactly what a `dataview` content-table query writes.
|
|
1440
|
+
|
|
1441
|
+
That is a deliberate refusal to impose a schema, and the tree is why. In `sohl`,
|
|
1442
|
+
frontmatter spreads **242 distinct leaf paths** unevenly over **15 types**, from
|
|
1443
|
+
9 on a `macro` to 72 on a `being`, and adding a field to one type is ordinary
|
|
1444
|
+
authoring. A format with a fixed column set would turn that authoring into a
|
|
1445
|
+
schema migration; a document format has no such problem.
|
|
1446
|
+
|
|
1447
|
+
Two keys are **derived** rather than authored, and a note carrying either is an
|
|
1448
|
+
error rather than a silent overwrite:
|
|
1449
|
+
|
|
1450
|
+
| Key | What it holds |
|
|
1451
|
+
| --------- | -------------------------------------------------------------------------------------------------- |
|
|
1452
|
+
| `package` | The configured `contentPackage`. A note may not declare its own, and the expander reads the same. |
|
|
1453
|
+
| `file` | `path`, `folder` and `name` below the content root — the same `file.*` a content-table query uses. |
|
|
1454
|
+
|
|
1455
|
+
The location is namespaced under `file` precisely because `folder` is real
|
|
1456
|
+
frontmatter on most notes; a record states both, and they mean different things.
|
|
1457
|
+
|
|
1458
|
+
### Every note's address, and every anchor it defines
|
|
1459
|
+
|
|
1460
|
+
A record states the address a wikilink writes to reach the note, and every
|
|
1461
|
+
`{#slug}` anchor its body declares:
|
|
1462
|
+
|
|
1463
|
+
```json
|
|
1464
|
+
{
|
|
1465
|
+
"address": { "slug": "being-aurochs", "canonical": "sohl-being-aurochs" },
|
|
1466
|
+
"file": { "path": "Bestiary/Animal/Aurochs.md", "folder": "Bestiary/Animal", "name": "Aurochs" },
|
|
1467
|
+
"anchors": [
|
|
1468
|
+
{
|
|
1469
|
+
"slug": "appearance",
|
|
1470
|
+
"name": "Appearance",
|
|
1471
|
+
"level": 1,
|
|
1472
|
+
"line": 348,
|
|
1473
|
+
"link": "being-aurochs#appearance"
|
|
1474
|
+
},
|
|
1475
|
+
{
|
|
1476
|
+
"slug": "dossier",
|
|
1477
|
+
"name": "Dossier",
|
|
1478
|
+
"level": 1,
|
|
1479
|
+
"line": 352,
|
|
1480
|
+
"link": "being-aurochs#dossier"
|
|
1481
|
+
}
|
|
1482
|
+
]
|
|
1483
|
+
}
|
|
1484
|
+
```
|
|
1485
|
+
|
|
1486
|
+
A record also states `nameAscii`, the note's `name.full` reduced to printable
|
|
1487
|
+
7-bit ASCII:
|
|
1488
|
+
|
|
1489
|
+
| `name.full` | `nameAscii` |
|
|
1490
|
+
| --------------- | ----------------- |
|
|
1491
|
+
| `Kûrbúl ¾-Helm` | `Kurbul 3/4-Helm` |
|
|
1492
|
+
| `Kèthîra` | `Kethira` |
|
|
1493
|
+
| `Ærling` | `AErling` |
|
|
1494
|
+
| `Þorn` | `Thorn` |
|
|
1495
|
+
| `Ðunhold` | `Dunhold` |
|
|
1496
|
+
| `Straße` | `Strasse` |
|
|
1497
|
+
|
|
1498
|
+
Names carry the setting's orthography and nobody types them, so anything
|
|
1499
|
+
searching or completing over the index needs a form a keyboard produces. Stating
|
|
1500
|
+
one means every consumer matches the same way, rather than each inventing a
|
|
1501
|
+
slightly different fold and two searches over the same data disagreeing.
|
|
1502
|
+
|
|
1503
|
+
It **transliterates rather than strips**, through the same `unidecode` table
|
|
1504
|
+
`slugify` already runs — so an ASCII name and a slug can never disagree about a
|
|
1505
|
+
character. Diacritics fold, ligatures expand, the runic letters spell out, and a
|
|
1506
|
+
vulgar fraction becomes readable. Deleting the marks instead would reduce
|
|
1507
|
+
`Kûrbúl` to `Krbl`, which is worse than the original for anyone trying to
|
|
1508
|
+
recognise it. Whatever is still outside printable ASCII afterwards becomes a
|
|
1509
|
+
space and runs of whitespace collapse — a space rather than nothing, so a
|
|
1510
|
+
character that transliterates away cannot weld two words together.
|
|
1511
|
+
|
|
1512
|
+
The value is emitted even when it equals the name, so a consumer matching on it
|
|
1513
|
+
never has to branch on whether the name happened to be ASCII already; it is
|
|
1514
|
+
`null` only when the note has no name at all. On the `sohl` tree, 25 of 1,606
|
|
1515
|
+
notes differ from their `name.full`.
|
|
1516
|
+
|
|
1517
|
+
`aliasesAscii` does the same for `name.aliases`, in the authored order. An alias
|
|
1518
|
+
is the name a reader is at least as likely to reach for as the canonical one —
|
|
1519
|
+
`Killer Whale` for an orca, `Ice Bear` for a polar bear, `Ix'balam` for a
|
|
1520
|
+
jaguar — so anything searching the index has to match them too. It is an **empty
|
|
1521
|
+
array**, never null, when a note has no aliases: an empty set of names is a fact
|
|
1522
|
+
rather than a missing value, and a consumer iterating it should not have to check
|
|
1523
|
+
first. An entry that is not a non-empty string is dropped rather than left as a
|
|
1524
|
+
hole, since the array is a set of names to match and a null is not one.
|
|
1525
|
+
|
|
1526
|
+
`address.slug` is what goes inside `[[…]]` within the package; `address.canonical`
|
|
1527
|
+
is the package-qualified key the link manifest files the note under. Both are
|
|
1528
|
+
`null` for a note with no type or no shortcode, which has no address at all — the
|
|
1529
|
+
record says so rather than leaving each reader to rediscover the rule.
|
|
1530
|
+
|
|
1531
|
+
**Neither is new information** — both derive from `type` and `shortcode`, which
|
|
1532
|
+
every record already carries. What the fields add is the _rule_: the lowercasing
|
|
1533
|
+
and the hyphen join live in one place, derived by the same `addressSlug` and
|
|
1534
|
+
`canonicalKey` the manifest and the site build use, so an index cannot disagree
|
|
1535
|
+
with either about where a note lives. A consumer that reimplements the join
|
|
1536
|
+
slightly differently gets a lookup matching nothing and no explanation — which is
|
|
1537
|
+
exactly how a resolver keyed on a bare `type/shortcode` silently misses every
|
|
1538
|
+
canonical `pkg-type-shortcode` entry.
|
|
1539
|
+
|
|
1540
|
+
**Anchors make a link checkable without a build.** Because the index states every
|
|
1541
|
+
anchor a note defines, `[[being-aurochs#dossier]]` can be confirmed — or shown
|
|
1542
|
+
dead — by a lookup, rather than by re-parsing the tree. Each anchor also carries
|
|
1543
|
+
its **line in the file**, so an editor jumps straight to the heading instead of
|
|
1544
|
+
searching for it, and a diagnostic about a section can name a real position.
|
|
1545
|
+
|
|
1546
|
+
Only headings carrying an explicit `{#slug}` are listed. A bare `#` heading also
|
|
1547
|
+
starts a journal page, but declares no slug, so nothing can address it with `#…`
|
|
1548
|
+
and listing it would offer a link that cannot be written. What counts as an anchor
|
|
1549
|
+
is kept identical to what `splitPages` matches — that pass decides which sections
|
|
1550
|
+
become addressable journal pages — and a test asserts the two agree, so drift
|
|
1551
|
+
fails the suite rather than advertising a link that resolves nowhere.
|
|
1552
|
+
|
|
1553
|
+
**The path stays relative.** `file.path` is below the content root and is
|
|
1554
|
+
deliberately never absolute: an absolute path is a fact about the machine that
|
|
1555
|
+
built the index rather than about the content, so it would differ between two
|
|
1556
|
+
checkouts of the same tree — costing the byte-stability the artifact depends on —
|
|
1557
|
+
and a published copy would carry someone's home directory and be wrong for every
|
|
1558
|
+
reader. Anyone holding the index knows the root it was built from, and
|
|
1559
|
+
`root + file.path` is the absolute form whenever it is wanted.
|
|
1560
|
+
|
|
1561
|
+
### Why JSON Lines, and not a database
|
|
1562
|
+
|
|
1563
|
+
The artifact has to survive the build that made it and be usable by anything — a
|
|
1564
|
+
person with `jq`, an editor, a CI check, another package's build. A
|
|
1565
|
+
line-per-note text file needs no server, no driver, and no schema; it is
|
|
1566
|
+
readable by every language without an install; and it **diffs**, so a migration
|
|
1567
|
+
that quietly empties a category shows up as a reviewable change rather than as a
|
|
1568
|
+
silently different binary.
|
|
1569
|
+
|
|
1570
|
+
Choosing it forfeits no SQL: DuckDB reads JSON Lines directly, with nested
|
|
1571
|
+
access, so `FROM read_json_auto('build/content-index/sohl.jsonl')` is a query
|
|
1572
|
+
away. A stored schema would forfeit the open shape, which is the asymmetry that
|
|
1573
|
+
decides it.
|
|
1574
|
+
|
|
1575
|
+
### It is derived, disposable, and byte-stable
|
|
1576
|
+
|
|
1577
|
+
The index is written under `build/`, gitignored with the rest of it, and
|
|
1578
|
+
**nothing may be authored against it**. It is deliberately not in `paths.stage`:
|
|
1579
|
+
that tree is mirrored destructively into a Foundry data root, so anything left
|
|
1580
|
+
there ships inside the installed system to every player.
|
|
1581
|
+
|
|
1582
|
+
Regenerating costs a frontmatter parse rather than a build, so the intended way
|
|
1583
|
+
to use it is to rebuild it whenever it looks stale — which is why it is a
|
|
1584
|
+
command of its own and not only a build step, and why it need never be
|
|
1585
|
+
committed.
|
|
1586
|
+
|
|
1587
|
+
That only holds if a rebuild is a no-op when nothing changed, so the output is
|
|
1588
|
+
**byte-stable**: records are ordered by content path with the note id breaking
|
|
1589
|
+
any tie, and every object's keys are sorted at every depth. A walk order is a
|
|
1590
|
+
directory-read order, and directory-read order is not a fact about the content.
|
|
1591
|
+
|
|
1592
|
+
An empty tree is an **error**, not an empty index. A reader takes the file as
|
|
1593
|
+
authoritative, and an index stating that a package has no content is
|
|
1594
|
+
indistinguishable from one built against a mis-pointed tree.
|
|
1595
|
+
|
|
1393
1596
|
## Publishing a website
|
|
1394
1597
|
|
|
1395
1598
|
```bash
|
package/bin/content-build.mjs
CHANGED
|
@@ -97,6 +97,7 @@ import { ENGINE_NOTE_SCHEMAS } from "../engine/note-schemas.mjs";
|
|
|
97
97
|
import { NOTE_VOCABULARY } from "../engine/note-vocabulary.mjs";
|
|
98
98
|
import { checkFormatting, lintMarkdown } from "../engine/prose-lint.mjs";
|
|
99
99
|
import { emitLinkManifest } from "../engine/manifest-emit.mjs";
|
|
100
|
+
import { emitContentIndex } from "../engine/content-index.mjs";
|
|
100
101
|
import {
|
|
101
102
|
buildSite,
|
|
102
103
|
gatesFailed,
|
|
@@ -228,6 +229,7 @@ const argv = yargs(hideBin(process.argv))
|
|
|
228
229
|
.command(formatCommand())
|
|
229
230
|
.command(markdownCommand())
|
|
230
231
|
.command(manifestCommand())
|
|
232
|
+
.command(contentIndexCommand())
|
|
231
233
|
.command(siteCommand())
|
|
232
234
|
.command(reachabilityCommand())
|
|
233
235
|
.command(addressesCommand())
|
|
@@ -1195,6 +1197,58 @@ function manifestCommand() {
|
|
|
1195
1197
|
};
|
|
1196
1198
|
}
|
|
1197
1199
|
|
|
1200
|
+
/**
|
|
1201
|
+
* `content-build content-index` — emit this package's note index.
|
|
1202
|
+
*
|
|
1203
|
+
* Every build already walks the tree and parses every note's frontmatter, then
|
|
1204
|
+
* throws the result away, so nothing outside a build can ask a question about
|
|
1205
|
+
* the content (#224). This publishes that walk as JSON Lines: one record per
|
|
1206
|
+
* note, carrying the whole frontmatter plus the note's place in the tree.
|
|
1207
|
+
*
|
|
1208
|
+
* It is a command of its own rather than only a build step because the point of
|
|
1209
|
+
* the artifact is that anyone can regenerate it at will — it costs a
|
|
1210
|
+
* frontmatter parse, not a build. That is also what lets it stay uncommitted:
|
|
1211
|
+
* something reproducible in under a second does not need to be kept.
|
|
1212
|
+
*
|
|
1213
|
+
* @returns {object} The yargs command module.
|
|
1214
|
+
*/
|
|
1215
|
+
// eslint-disable-next-line
|
|
1216
|
+
function contentIndexCommand() {
|
|
1217
|
+
return {
|
|
1218
|
+
command: "content-index [root]",
|
|
1219
|
+
describe: "Emit this package's note index as JSON Lines",
|
|
1220
|
+
builder: (yargs) => {
|
|
1221
|
+
yargs.positional("root", {
|
|
1222
|
+
describe: "Content tree to read. Defaults to the configured contentBase.",
|
|
1223
|
+
type: "string",
|
|
1224
|
+
});
|
|
1225
|
+
yargs.option("out", {
|
|
1226
|
+
describe:
|
|
1227
|
+
"Directory to write into. Defaults to the configured " +
|
|
1228
|
+
"`paths.contentIndex`.",
|
|
1229
|
+
type: "string",
|
|
1230
|
+
});
|
|
1231
|
+
},
|
|
1232
|
+
handler: (argv) => {
|
|
1233
|
+
try {
|
|
1234
|
+
const config = loadPackConfig();
|
|
1235
|
+
const { file, notes, bytes } = emitContentIndex({
|
|
1236
|
+
config,
|
|
1237
|
+
...(argv.root ? { contentBase: argv.root } : {}),
|
|
1238
|
+
...(argv.out ? { outDir: argv.out } : {}),
|
|
1239
|
+
});
|
|
1240
|
+
log.info(
|
|
1241
|
+
`${config.contentPackage} → ${path.relative(process.cwd(), file)} ` +
|
|
1242
|
+
`(${notes} notes, ${Math.round(bytes / 1024)} KiB)`,
|
|
1243
|
+
);
|
|
1244
|
+
} catch (err) {
|
|
1245
|
+
reportFailure(err);
|
|
1246
|
+
process.exitCode = 1;
|
|
1247
|
+
}
|
|
1248
|
+
},
|
|
1249
|
+
};
|
|
1250
|
+
}
|
|
1251
|
+
|
|
1198
1252
|
/**
|
|
1199
1253
|
* `content-build site` — publish the content tree as a website.
|
|
1200
1254
|
*
|
package/content-config.mjs
CHANGED
|
@@ -90,6 +90,11 @@ export const DEFAULT_PATHS = /** @type {const} */ ({
|
|
|
90
90
|
content: "assets/content",
|
|
91
91
|
manifests: "assets/manifests",
|
|
92
92
|
manifestOut: "build/manifests",
|
|
93
|
+
// Where `content-index` writes this package's note index. Under `build/`
|
|
94
|
+
// because it is derived and disposable — regenerating it costs a
|
|
95
|
+
// frontmatter parse — and emphatically not under `stage`, which is mirrored
|
|
96
|
+
// into a Foundry data root (#224).
|
|
97
|
+
contentIndex: "build/content-index",
|
|
93
98
|
packJson: "build/packs-json",
|
|
94
99
|
stage: "build/stage/packs",
|
|
95
100
|
unpack: "build/tmp/packs",
|
|
@@ -305,6 +310,10 @@ export function publishesContentPages(config) {
|
|
|
305
310
|
* build artifact — the published copy is
|
|
306
311
|
* the one a consumer vendors into its
|
|
307
312
|
* `manifests` directory.
|
|
313
|
+
* @property {string} [contentIndex] Where `content-index` writes this
|
|
314
|
+
* package's note index. Outbound, and a
|
|
315
|
+
* derived artifact — never a source, and
|
|
316
|
+
* never inside `stage`.
|
|
308
317
|
* @property {string} [packJson] Build-only per-entry JSON intermediate.
|
|
309
318
|
* @property {string} [stage] Compiled LevelDB packs.
|
|
310
319
|
* @property {string} [unpack] Where `unpack` extracts JSON back to.
|
|
@@ -317,6 +326,7 @@ export function publishesContentPages(config) {
|
|
|
317
326
|
* @property {string} content
|
|
318
327
|
* @property {string} manifests
|
|
319
328
|
* @property {string} manifestOut
|
|
329
|
+
* @property {string} contentIndex
|
|
320
330
|
* @property {string} packJson
|
|
321
331
|
* @property {string} stage
|
|
322
332
|
* @property {string} unpack
|
|
@@ -79,47 +79,32 @@ export function addressSlug(fm) {
|
|
|
79
79
|
}
|
|
80
80
|
|
|
81
81
|
/**
|
|
82
|
-
* A note's address
|
|
82
|
+
* A note's address: `<type>-<shortcode>/`, e.g. `affliction-aconite/`.
|
|
83
83
|
*
|
|
84
|
-
*
|
|
85
|
-
*
|
|
86
|
-
*
|
|
87
|
-
* decides nothing about where it publishes.
|
|
88
|
-
*
|
|
89
|
-
* @param {object} fm - Parsed frontmatter.
|
|
90
|
-
* @returns {string} The mount-relative address, with a trailing slash.
|
|
91
|
-
* @throws {Error} When the note has no address.
|
|
92
|
-
*/
|
|
93
|
-
export function contentAddress(fm) {
|
|
94
|
-
return `${addressSlug(fm)}/`;
|
|
95
|
-
}
|
|
96
|
-
|
|
97
|
-
/**
|
|
98
|
-
* A note's address relative to its **package**, e.g. `affliction-aconite/`.
|
|
99
|
-
*
|
|
100
|
-
* This is the form the link manifest records and the site build emits pages at,
|
|
101
|
-
* and it is one function because those two must agree — a manifest asserting an
|
|
84
|
+
* This is the one form a note is addressed by. It is what the link manifest
|
|
85
|
+
* records as an entry's `path` and what the site build emits the page at, and
|
|
86
|
+
* it is one function because those two must agree — a manifest asserting an
|
|
102
87
|
* address the site does not publish resolves at build time and 404s for the
|
|
103
88
|
* reader, which is the failure this module exists to prevent.
|
|
104
89
|
*
|
|
105
|
-
* **
|
|
106
|
-
*
|
|
107
|
-
*
|
|
108
|
-
* is
|
|
109
|
-
*
|
|
110
|
-
*
|
|
111
|
-
*
|
|
112
|
-
*
|
|
113
|
-
*
|
|
114
|
-
*
|
|
115
|
-
* of the package's fixed mounts — `/<package>/` for the landing,
|
|
116
|
-
* `/<package>/api/` for generated API docs, neither of which contains a hyphen
|
|
117
|
-
* or names a type.
|
|
90
|
+
* **The address is relative to the package**, and to nothing finer. A consumer
|
|
91
|
+
* composing a URL prepends where the package is served (`/<package>/`); a
|
|
92
|
+
* consumer composing a manifest entry measures against that same base. Nothing
|
|
93
|
+
* else is prepended: `prefix` says where the content tree *mounts inside the
|
|
94
|
+
* package* — the Hugo directory its pages are written under — and an address is
|
|
95
|
+
* `(type, shortcode)`, a package-wide identity that takes no mount, so `sohl`
|
|
96
|
+
* publishes `/sohl/affliction-aconite/` from a file written under `kb/`. The
|
|
97
|
+
* `type-` half is what keeps that flat namespace clear of the package's fixed
|
|
98
|
+
* mounts — `/<package>/` for the landing, `/<package>/api/` for generated API
|
|
99
|
+
* docs, neither of which contains a hyphen or names a type.
|
|
118
100
|
*
|
|
119
|
-
* **It
|
|
120
|
-
*
|
|
121
|
-
*
|
|
122
|
-
*
|
|
101
|
+
* **It is a pure function of the frontmatter**, and takes no options. Nothing
|
|
102
|
+
* about the file the note was read from reaches it: the `README.md` convention
|
|
103
|
+
* that made one note address a whole section is retired with the section itself
|
|
104
|
+
* (#204), so every note is addressed alike and there is one rule and no branch.
|
|
105
|
+
* It took an address scheme until #215, to validate a `landing` rule it then
|
|
106
|
+
* discarded; with that key retired, `prefix` was the only thing left in the
|
|
107
|
+
* scheme and the paragraph above is the reason it never applied.
|
|
123
108
|
*
|
|
124
109
|
* @param {object} fm - Parsed frontmatter.
|
|
125
110
|
* @returns {string} The package-relative address, with a trailing slash and no
|