@fro.bot/wiki-write-core 0.1.4 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +35 -4
- package/dist/frontmatter-block.d.ts +49 -0
- package/dist/frontmatter-block.js +62 -0
- package/dist/frontmatter.d.ts +5 -1
- package/dist/frontmatter.js +11 -5
- package/dist/gate-contract.js +1 -1
- package/dist/gate-contract.json +1 -1
- package/dist/repo-page-move.js +18 -10
- package/dist/wiki-ingest.js +92 -24
- package/dist/wiki-utils.d.ts +2 -2
- package/dist/wiki-utils.js +10 -10
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -15,7 +15,13 @@ pnpm add @fro.bot/wiki-write-core
|
|
|
15
15
|
- Node `>=24`.
|
|
16
16
|
- **ESM-only.** The exports declare `import` and `types` conditions, with no `require` condition and no CommonJS build. CommonJS consumers fail at package resolution by design.
|
|
17
17
|
|
|
18
|
-
The package ships unbundled ESM JavaScript with declaration files
|
|
18
|
+
The published package ships unbundled ESM JavaScript with declaration files in `dist/`. No install-time build step is required.
|
|
19
|
+
|
|
20
|
+
In the repository the package resolves to its TypeScript source: `exports` maps every subpath to `./src/<name>.ts`, and the only file committed under `dist/` is `gate-contract.json`. The release build compiles `dist/` and stages a manifest that differs from the committed one in `exports` alone, pointing each subpath at `./dist/<name>.js` with `./dist/<name>.d.ts` as its types. `npm pack` and `npm publish` run from that staged directory, so the committed manifest is never rewritten.
|
|
21
|
+
|
|
22
|
+
`bun run check:wiki-write-core-package` is the local and CI check: it builds the package, fails when the committed `dist/gate-contract.json` differs from the fresh build, packs the staged package, unpacks the tarball into a scratch `node_modules`, then resolves and imports every `exports` subpath under Node and type-checks a consumer module that imports them all. `bun run build` rewrites the committed marker after a change under `src/`, `tsconfig.build.json` or the manifest's `exports` or `files`.
|
|
23
|
+
|
|
24
|
+
`--stage <dir>` replaces whatever is at `<dir>`, so it refuses anything but a target that resolves (through symlinks) to somewhere strictly under the OS temp directory or `RUNNER_TEMP`, and it replaces an existing directory only when a previous stage left its `wiki-write-core-stage.marker` there. A root must be absolute and must not resolve to a filesystem root (so `RUNNER_TEMP=/` is refused), and a stage that contains a hidden file or directory is refused, because `upload-artifact` drops them and the publish job's digest would no longer match. The target is judged before anything is built or written, so a refused run leaves the committed `dist/gate-contract.json` alone, and again just before the replacement. Refusals happen before anything is created, moved or removed. If a replacement fails and the original cannot be restored either, the original is kept in a `.wiki-write-core-dist-backup-*` sibling and the error names it.
|
|
19
25
|
|
|
20
26
|
## Fro Bot defaults
|
|
21
27
|
|
|
@@ -27,9 +33,9 @@ The package targets the Fro Bot control plane and carries its coordinates as def
|
|
|
27
33
|
|
|
28
34
|
When a call receives no Octokit client (`params.octokit`), it reads `GITHUB_TOKEN` from the environment and fails with a remediation message if the variable is unset.
|
|
29
35
|
|
|
30
|
-
The build also emits `dist/gate-contract.json` with `version` from `GATE_CONTRACT_VERSION` and `sourceTreeHash` from the same computed source hash embedded in `gate-contract.js`.
|
|
36
|
+
The build also emits `dist/gate-contract.json` with `version` from `GATE_CONTRACT_VERSION` and `sourceTreeHash` from the same computed source hash embedded in `gate-contract.js`. This is the one file committed under `dist/`: downstream writers fetch it from `main`, so it stays at that path. `bun run check:wiki-write-core-package` verifies the committed marker against a fresh build, and the same marker ships in the tarball. `version` is the gate criterion: a mismatch means a deliberate contract break and refuses writes. `sourceTreeHash` is diagnostic only: it moves with ordinary source changes and reports drift in the pinned copy, but must never block writes.
|
|
31
37
|
|
|
32
|
-
The package exports `GATE_CONTRACT_VERSION` and `GATE_SOURCE_TREE_HASH` from its root entrypoint. The build embeds a deterministic hash covering every non-test TypeScript file under `src/`, the resolved `tsconfig.build.json` configuration including inherited settings, and canonicalized `exports` and `files` values
|
|
38
|
+
The package exports `GATE_CONTRACT_VERSION` and `GATE_SOURCE_TREE_HASH` from its root entrypoint. In the repository that entrypoint is source, so `GATE_SOURCE_TREE_HASH` is the placeholder `__SOURCE_TREE_HASH__`; only the built `dist/gate-contract.js` in the tarball carries the hash. The build embeds a deterministic hash covering every non-test TypeScript file under `src/`, the resolved `tsconfig.build.json` configuration including inherited settings, and canonicalized `exports` and `files` values of the published manifest (`exports` in its `dist` form). Object-key order and `files` array order do not affect the hash. The hash does not cover tests, this README, other manifest fields such as dependencies or version, or runtime dependencies. Because it hashes the _resolved_ compiler configuration, a TypeScript upgrade that changes an inherited default or the `--showConfig` serialization moves the hash with no semantic change to the shipped code; a hash that shifts across a compiler bump is expected and does not by itself indicate gate drift.
|
|
33
39
|
|
|
34
40
|
`GATE_SOURCE_TREE_HASH` is the automatic drift detector: compare it when you need to know whether the shipped gate inputs changed. `GATE_CONTRACT_VERSION` is a deliberate, human-set marker for an intentional consumer-visible contract break; it is not expected to change for every source edit. Neither value proves that dependencies or the runtime behave identically. Consumers should compare the source hash for ordinary gate-input drift and the contract version when deciding whether an intentional contract migration is required.
|
|
35
41
|
|
|
@@ -39,6 +45,28 @@ The package exports `GATE_CONTRACT_VERSION` and `GATE_SOURCE_TREE_HASH` from its
|
|
|
39
45
|
|
|
40
46
|
`WikiLintFindingKind` includes `correction-eroded` for a blocking active-correction failure and `correction-needs-reconfirmation` for an advisory lifecycle or formatting-only change. Both retain the existing `{kind, path, message, target?}` runtime shape and fingerprint derivation. Consumers with exhaustive matches must handle both kinds.
|
|
41
47
|
|
|
48
|
+
A page that opens a frontmatter block and never closes it is reported as `invalid-frontmatter`; it was reported as `missing-frontmatter` while the block was read as no block. Wiki repair counts such a page as out of scope, as it does for any `invalid-frontmatter` finding.
|
|
49
|
+
|
|
50
|
+
## Frontmatter block
|
|
51
|
+
|
|
52
|
+
`splitFrontmatterBlock(content)` (exported from the root entry and from `@fro.bot/wiki-write-core/frontmatter`) finds the YAML frontmatter block at the start of a document. It does not parse YAML. The root entry already exports the wiki page reader as `splitFrontmatter`, so the splitter has another name.
|
|
53
|
+
|
|
54
|
+
- The block opens only when the content starts with a line that is exactly `---`. A trailing space, a byte-order mark or a blank line before it means there is no block.
|
|
55
|
+
- A line ending is LF or CRLF; a lone CR is part of its line.
|
|
56
|
+
- The block closes at the first later line that is `---` followed only by spaces or tabs. `---x`, `----` and `--- text` are content of the block.
|
|
57
|
+
- The result is `{kind: 'none'}`, `{kind: 'malformed'}` (opened and never closed, which includes a file that is only `---`) or `{kind: 'block', opening, text, textEnding, closing, body}`. A block carries the pieces that put the document back: `opening + text + textEnding + closing + body` is the original content. `text` is what a YAML parser reads (it is `''` for an empty block); `textEnding` is the line ending of the last line of `text`, or `''` for a block with no lines; `closing` is the closing line with its spaces, tabs and line ending. A caller that rewrites a line of `text` rebuilds the file from the result alone and keeps every other byte.
|
|
58
|
+
- The scan visits each line once, so its running time grows linearly with the content, an unclosed block included.
|
|
59
|
+
|
|
60
|
+
The package's own readers (`parseFrontmatterDocument`, `splitFrontmatter`, the ingest's page parser and the repo-page move) all use it. Each keeps its error texts and its body handling. An empty block is an empty mapping for the document reader and the ingest parser; a block that parses to `null` (whitespace, comments, `null`, `...`) is not a mapping and gets each reader's rule for a non-mapping. The wiki page reader reports an unclosed block as `Invalid frontmatter: missing closing --- delimiter`.
|
|
61
|
+
|
|
62
|
+
## Target page sources
|
|
63
|
+
|
|
64
|
+
When the ingest writes the page of the `repo:` target and that page lists no sources (no `sources` key, or an empty list), the page lists `{url, accessed}` with the target's `https://github.com/<owner>/<name>` URL and the date of the ingest's timestamp. When the page on the branch already has an entry for the target, its whole `sources` list is carried over as it is, so a repeat write changes nothing, a repeat after a migration included. The stamp applies only when the owner and name are plain GitHub names (ASCII letters, digits, `.`, `_` and `-`, not `.` or `..`); for any other target the page is written without one. A page that lists any source is never changed, and neither is a `sources` value that is not a list. When the ingest migrates a page to the target's slug by a trusted node ID, the merged page also gains the target's URL if no entry matches it, and keeps its other entries. No other repo page in the write is stamped, and a write whose target is a page path (the dashboard's case) is never stamped. Whether an entry matches the target uses the privacy gate's URL rule: `https`, host `github.com`, no port, owner and name equal without case, any path after them.
|
|
65
|
+
|
|
66
|
+
## Gate contract version
|
|
67
|
+
|
|
68
|
+
`GATE_CONTRACT_VERSION` stays at 1 for the splitter and the reader changes. A bump makes every dashboard save fail until the dashboard ships a release that carries the same version, and the changes here do not need that: no page on `data` changes meaning, the dashboard already has a message for both lint finding kinds, and the dashboard writes with its own installed copy of this package. `dist/gate-contract.json` still changes, because its `sourceTreeHash` follows the source.
|
|
69
|
+
|
|
42
70
|
## Releasing
|
|
43
71
|
|
|
44
72
|
A merge to `main` that changes `packages/wiki-write-core/` releases the package through `.github/workflows/release-wiki-write-core.yaml`. Versions are the `wiki-write-core-v<version>` tags (for example `wiki-write-core-v0.1.1`), and each tag has a GitHub release.
|
|
@@ -55,10 +83,13 @@ Only commits that touch `packages/wiki-write-core/` are analyzed. The commit typ
|
|
|
55
83
|
- A scoped suppression wins over a breaking marker: `build(dev)!:` releases nothing.
|
|
56
84
|
- While the package is 0.x it never releases a major version; a breaking change is a minor bump.
|
|
57
85
|
- A commit containing `[skip release]` (or `[release skip]`) is ignored by the release analysis.
|
|
86
|
+
- Pull requests are squash-merged with `squash_merge_commit_title: PR_TITLE` (`common-settings.yaml`), so the PR title is the commit header the analysis reads. Pick the title type for the release you want.
|
|
87
|
+
- A PR that changes only tests, docs other than the README, or other files under `packages/wiki-write-core/` that should not ship needs a non-releasing title (`test:`, `docs:`, `chore:`, `refactor:` or `build(dev):`). A `fix:` or `build:` title there publishes a patch.
|
|
88
|
+
- A PR that changes nothing under `packages/wiki-write-core/` does not count toward the release type, whatever its own type. The release runs on every push to `main` and reads every package commit since the last release tag, so such a merge can still publish an earlier package change that has not been released yet.
|
|
58
89
|
|
|
59
90
|
### Publishing
|
|
60
91
|
|
|
61
|
-
The `publish` job uses npm trusted publishing (OIDC) from `release-wiki-write-core.yaml` in the `release` environment, and publishes with provenance. There is no npm token. It publishes the
|
|
92
|
+
The `publish` job uses npm trusted publishing (OIDC) from `release-wiki-write-core.yaml` in the `release` environment, and publishes with provenance. There is no npm token. It publishes the package the `build` job staged: the compiled `dist/` and a manifest whose `exports` point at it. The `build` job proves the packed tarball resolves, imports and type-checks every export. The `publish` job runs no repository script. The `build` job exposes a digest of the staged tree (sha256 over the `<sha256> <path>` listing of every file, in byte order of the path) as a job output, which comes from the runner that built the package and not from the artifact store. The `publish` job recomputes it over what it downloaded and refuses unless the two are equal, before it reads or installs anything. It then checks that the staged manifest is the committed one with `exports` rewritten (each `./src/<name>.ts` becomes `./dist/<name>.js` with `./dist/<name>.d.ts` as its types, and no target has a `.`, `..` or empty segment) and replaces the checkout's `package.json` and `dist/` with the staged files. semantic-release then changes only the manifest's `version` (`npm version`, which the digest does not cover because it is checked first), and `npm publish` packs the same files the `build` job's tarball check proved.
|
|
62
93
|
|
|
63
94
|
Verify a release:
|
|
64
95
|
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* What `splitFrontmatterBlock` found at the start of a document.
|
|
3
|
+
*
|
|
4
|
+
* - `none`: the content does not open a block.
|
|
5
|
+
* - `malformed`: the content opens a block and never closes it. It carries no content, so a caller cannot read
|
|
6
|
+
* the pieces of a block that was not closed.
|
|
7
|
+
* - `block`: the content opens and closes a block. The five strings put the content back byte for byte:
|
|
8
|
+
* `opening + text + textEnding + closing + body` is the original content.
|
|
9
|
+
*
|
|
10
|
+
* The pieces are strings, not offsets, so an editor that rewrites a line of the block rebuilds the file from the
|
|
11
|
+
* result alone, with the line endings, trailing spaces and body it found.
|
|
12
|
+
*/
|
|
13
|
+
export type FrontmatterBlockSplit = {
|
|
14
|
+
readonly kind: 'none';
|
|
15
|
+
} | {
|
|
16
|
+
readonly kind: 'malformed';
|
|
17
|
+
} | {
|
|
18
|
+
readonly kind: 'block';
|
|
19
|
+
/** The opening line with its line ending: `---` and LF or CRLF. */
|
|
20
|
+
readonly opening: string;
|
|
21
|
+
/**
|
|
22
|
+
* The lines between the opening and the closing line, each with the line ending it has, except the last line,
|
|
23
|
+
* whose ending is `textEnding`. It is what a YAML parser reads. It is `''` for a block with no lines and for
|
|
24
|
+
* a block of one blank line, which `textEnding` tells apart.
|
|
25
|
+
*/
|
|
26
|
+
readonly text: string;
|
|
27
|
+
/** The line ending of the last line of `text`: `\n`, `\r\n`, or `''` when the block has no lines. */
|
|
28
|
+
readonly textEnding: string;
|
|
29
|
+
/** The closing line: `---`, any spaces or tabs after it, and its line ending. It has none at the end of the content. */
|
|
30
|
+
readonly closing: string;
|
|
31
|
+
/** Everything after the closing line, untouched. */
|
|
32
|
+
readonly body: string;
|
|
33
|
+
};
|
|
34
|
+
/**
|
|
35
|
+
* Finds the YAML frontmatter block at the start of `content`. It does not parse the YAML.
|
|
36
|
+
*
|
|
37
|
+
* The block rule:
|
|
38
|
+
* - The content starts with a line that is exactly `---`. A space, a byte-order mark, a blank line or any other
|
|
39
|
+
* character before it or after it, on that line, means there is no block.
|
|
40
|
+
* - A line ending is LF or CRLF. A lone CR is part of its line.
|
|
41
|
+
* - The block ends at the first later line that is `---` followed only by spaces or tabs. A line such as `---x`,
|
|
42
|
+
* `----` or `--- text` is part of the block.
|
|
43
|
+
* - A block can be empty: `---` directly followed by `---` gives empty text.
|
|
44
|
+
* - A block that is opened and never closed is malformed, and so is a file that is only `---`.
|
|
45
|
+
*
|
|
46
|
+
* The content is cut at its line feeds once and each piece is visited once, so the running time is linear in the
|
|
47
|
+
* length of the content, and the loop is bounded by the number of pieces.
|
|
48
|
+
*/
|
|
49
|
+
export declare function splitFrontmatterBlock(content: string): FrontmatterBlockSplit;
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Finds the YAML frontmatter block at the start of `content`. It does not parse the YAML.
|
|
3
|
+
*
|
|
4
|
+
* The block rule:
|
|
5
|
+
* - The content starts with a line that is exactly `---`. A space, a byte-order mark, a blank line or any other
|
|
6
|
+
* character before it or after it, on that line, means there is no block.
|
|
7
|
+
* - A line ending is LF or CRLF. A lone CR is part of its line.
|
|
8
|
+
* - The block ends at the first later line that is `---` followed only by spaces or tabs. A line such as `---x`,
|
|
9
|
+
* `----` or `--- text` is part of the block.
|
|
10
|
+
* - A block can be empty: `---` directly followed by `---` gives empty text.
|
|
11
|
+
* - A block that is opened and never closed is malformed, and so is a file that is only `---`.
|
|
12
|
+
*
|
|
13
|
+
* The content is cut at its line feeds once and each piece is visited once, so the running time is linear in the
|
|
14
|
+
* length of the content, and the loop is bounded by the number of pieces.
|
|
15
|
+
*/
|
|
16
|
+
export function splitFrontmatterBlock(content) {
|
|
17
|
+
const pieces = content.split('\n');
|
|
18
|
+
const lastIndex = pieces.length - 1;
|
|
19
|
+
// `offset` is where the piece being read starts. `openingLength` is the length of the opening line with its line
|
|
20
|
+
// feed, and `textEnd` is where the lines before the piece end, before their line ending.
|
|
21
|
+
let offset = 0;
|
|
22
|
+
let openingLength = 0;
|
|
23
|
+
let textEnd = 0;
|
|
24
|
+
for (const [index, piece] of pieces.entries()) {
|
|
25
|
+
// A piece that ends with a carriage return has a CRLF ending when a line feed followed it, that is, unless it is
|
|
26
|
+
// the last piece.
|
|
27
|
+
const line = index < lastIndex && piece.endsWith('\r') ? piece.slice(0, -1) : piece;
|
|
28
|
+
if (index === 0) {
|
|
29
|
+
if (line !== '---')
|
|
30
|
+
return { kind: 'none' };
|
|
31
|
+
openingLength = piece.length + 1;
|
|
32
|
+
textEnd = openingLength;
|
|
33
|
+
}
|
|
34
|
+
else if (isCloseLine(line)) {
|
|
35
|
+
const lineEnd = offset + piece.length + 1;
|
|
36
|
+
return {
|
|
37
|
+
kind: 'block',
|
|
38
|
+
opening: content.slice(0, openingLength),
|
|
39
|
+
text: content.slice(openingLength, textEnd),
|
|
40
|
+
textEnding: content.slice(textEnd, offset),
|
|
41
|
+
closing: content.slice(offset, lineEnd),
|
|
42
|
+
body: content.slice(lineEnd),
|
|
43
|
+
};
|
|
44
|
+
}
|
|
45
|
+
else {
|
|
46
|
+
textEnd = offset + line.length;
|
|
47
|
+
}
|
|
48
|
+
offset += piece.length + 1;
|
|
49
|
+
}
|
|
50
|
+
return { kind: 'malformed' };
|
|
51
|
+
}
|
|
52
|
+
/** Whether `line` is `---` followed only by spaces and tabs. */
|
|
53
|
+
function isCloseLine(line) {
|
|
54
|
+
if (!line.startsWith('---'))
|
|
55
|
+
return false;
|
|
56
|
+
for (let at = 3; at < line.length; at += 1) {
|
|
57
|
+
const character = line.charAt(at);
|
|
58
|
+
if (character !== ' ' && character !== '\t')
|
|
59
|
+
return false;
|
|
60
|
+
}
|
|
61
|
+
return true;
|
|
62
|
+
}
|
package/dist/frontmatter.d.ts
CHANGED
|
@@ -1,8 +1,12 @@
|
|
|
1
|
+
export { splitFrontmatterBlock, type FrontmatterBlockSplit } from './frontmatter-block.js';
|
|
1
2
|
export interface FrontmatterDocument {
|
|
2
3
|
readonly values: Record<string, unknown>;
|
|
3
4
|
readonly body: string;
|
|
4
5
|
}
|
|
5
|
-
/**
|
|
6
|
+
/**
|
|
7
|
+
* Parse a validated wiki document while retaining its raw body. A document with no block, or with a block that is
|
|
8
|
+
* never closed, is missing its frontmatter. A block with no lines is an empty mapping; any other block must parse to one.
|
|
9
|
+
*/
|
|
6
10
|
export declare function parseFrontmatterDocument(content: string): FrontmatterDocument;
|
|
7
11
|
/** Reconstruct a wiki document with system-owned frontmatter and normalized body. */
|
|
8
12
|
export declare function renderFrontmatterDocument(values: Record<string, unknown>, body: string): string;
|
package/dist/frontmatter.js
CHANGED
|
@@ -1,14 +1,20 @@
|
|
|
1
1
|
import { parse, stringify } from 'yaml';
|
|
2
|
-
|
|
2
|
+
import { splitFrontmatterBlock } from "./frontmatter-block.js";
|
|
3
|
+
export { splitFrontmatterBlock } from "./frontmatter-block.js";
|
|
4
|
+
/**
|
|
5
|
+
* Parse a validated wiki document while retaining its raw body. A document with no block, or with a block that is
|
|
6
|
+
* never closed, is missing its frontmatter. A block with no lines is an empty mapping; any other block must parse to one.
|
|
7
|
+
*/
|
|
3
8
|
export function parseFrontmatterDocument(content) {
|
|
4
|
-
const
|
|
5
|
-
if (
|
|
9
|
+
const split = splitFrontmatterBlock(content);
|
|
10
|
+
if (split.kind !== 'block') {
|
|
6
11
|
throw new Error('validated wiki page is missing frontmatter');
|
|
7
12
|
}
|
|
8
|
-
|
|
13
|
+
// Only a block with no lines is an empty mapping. A block of one blank line parses to null, like any blank text.
|
|
14
|
+
const parsed = split.text === '' && split.textEnding === '' ? {} : parse(split.text);
|
|
9
15
|
if (!isRecord(parsed))
|
|
10
16
|
throw new Error('validated wiki page frontmatter is not an object');
|
|
11
|
-
return { values: parsed, body:
|
|
17
|
+
return { values: parsed, body: split.body };
|
|
12
18
|
}
|
|
13
19
|
/** Reconstruct a wiki document with system-owned frontmatter and normalized body. */
|
|
14
20
|
export function renderFrontmatterDocument(values, body) {
|
package/dist/gate-contract.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
1
|
// Bump deliberately when an intentional consumer-visible gate contract break ships.
|
|
2
2
|
export const GATE_CONTRACT_VERSION = 1;
|
|
3
3
|
// Replaced with the deterministic source-tree hash by the package build.
|
|
4
|
-
export const GATE_SOURCE_TREE_HASH = '
|
|
4
|
+
export const GATE_SOURCE_TREE_HASH = '637b237dc25880625d73dde771514ff4ff64fd845cb6dd1a07c61184570de419';
|
package/dist/gate-contract.json
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":1,"sourceTreeHash":"
|
|
1
|
+
{"version":1,"sourceTreeHash":"637b237dc25880625d73dde771514ff4ff64fd845cb6dd1a07c61184570de419"}
|
package/dist/repo-page-move.js
CHANGED
|
@@ -13,6 +13,7 @@
|
|
|
13
13
|
* repository URL. The body is never consulted.
|
|
14
14
|
* - `log.md` history and `knowledge/wiki/README.md` are never rewritten.
|
|
15
15
|
*/
|
|
16
|
+
import { splitFrontmatterBlock } from "./frontmatter-block.js";
|
|
16
17
|
import { parseFrontmatterDocument, renderFrontmatterDocument } from "./frontmatter.js";
|
|
17
18
|
import { parseRelatedFlow } from "./line-scanners.js";
|
|
18
19
|
import { appendLogEntry, rebuildWikiIndex, validateWikilinks, WikiIngestError } from "./wiki-ingest.js";
|
|
@@ -293,21 +294,21 @@ function repairRelated(content, oldSlug, newSlug) {
|
|
|
293
294
|
}
|
|
294
295
|
/**
|
|
295
296
|
* Edit only the `related:` lines of the frontmatter text, so the rest of the page keeps its bytes.
|
|
296
|
-
* Returns `undefined` for a layout it does not recognize
|
|
297
|
+
* Returns `undefined` for a layout it does not recognize, and for a block with a CRLF line ending anywhere in it,
|
|
298
|
+
* the closing line included, which the edit would otherwise leave half rewritten; the caller then re-renders structurally.
|
|
297
299
|
*/
|
|
298
300
|
function editRelatedText(content, oldSlug, newSlug) {
|
|
299
|
-
const
|
|
300
|
-
|
|
301
|
-
if (frontmatter === undefined)
|
|
301
|
+
const split = splitFrontmatterBlock(content);
|
|
302
|
+
if (split.kind !== 'block' || hasCrlf(split))
|
|
302
303
|
return undefined;
|
|
303
|
-
const lines =
|
|
304
|
+
const lines = split.text.split('\n');
|
|
304
305
|
const flowIndex = lines.findIndex(line => parseRelatedFlow(line) !== undefined);
|
|
305
306
|
if (flowIndex !== -1) {
|
|
306
307
|
const inner = parseRelatedFlow(lines[flowIndex] ?? '') ?? '';
|
|
307
308
|
const items = inner.split(',').map(item => item.trim());
|
|
308
309
|
const mapped = dedupe(items.map(item => (stripQuotes(item) === oldSlug ? swapQuoted(item, newSlug) : item)));
|
|
309
310
|
lines[flowIndex] = `related: [${mapped.join(', ')}]`;
|
|
310
|
-
return
|
|
311
|
+
return rebuild(split, lines.join('\n'));
|
|
311
312
|
}
|
|
312
313
|
const keyIndex = lines.findIndex(line => RELATED_BLOCK_KEY.test(line));
|
|
313
314
|
if (keyIndex === -1)
|
|
@@ -327,7 +328,7 @@ function editRelatedText(content, oldSlug, newSlug) {
|
|
|
327
328
|
kept.push(item.value === oldSlug ? `${item.prefix}${item.quote}${newSlug}${item.quote}${item.suffix}` : line);
|
|
328
329
|
}
|
|
329
330
|
lines.splice(keyIndex + 1, cursor - keyIndex - 1, ...kept);
|
|
330
|
-
return
|
|
331
|
+
return rebuild(split, lines.join('\n'));
|
|
331
332
|
}
|
|
332
333
|
/** Parse one ` - value` line of a block sequence without a backtracking regex. */
|
|
333
334
|
function parseBlockItem(line) {
|
|
@@ -346,9 +347,16 @@ function parseBlockItem(line) {
|
|
|
346
347
|
const quoted = raw.length >= 2 && (first === "'" || first === '"') && raw.endsWith(first);
|
|
347
348
|
return { prefix, quote: quoted ? (first ?? '') : '', value: quoted ? raw.slice(1, -1) : raw, suffix };
|
|
348
349
|
}
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
return
|
|
350
|
+
/** Whether the opening line, the block text, the line before the close or the closing line ends with CRLF. */
|
|
351
|
+
function hasCrlf(block) {
|
|
352
|
+
return (block.opening.endsWith('\r\n') ||
|
|
353
|
+
block.textEnding === '\r\n' ||
|
|
354
|
+
block.text.includes('\r\n') ||
|
|
355
|
+
block.closing.endsWith('\r\n'));
|
|
356
|
+
}
|
|
357
|
+
/** The page with the block text replaced and every other byte of it kept. */
|
|
358
|
+
function rebuild(block, text) {
|
|
359
|
+
return block.opening + text + block.textEnding + block.closing + block.body;
|
|
352
360
|
}
|
|
353
361
|
function readRelated(content) {
|
|
354
362
|
try {
|
package/dist/wiki-ingest.js
CHANGED
|
@@ -9,6 +9,8 @@ import { assertBranchWritable } from "./branch-safety.js";
|
|
|
9
9
|
import { verifyCorrectionSurvival } from "./corrections-survival.js";
|
|
10
10
|
import { CorrectionStoreError, readCorrections } from "./corrections.js";
|
|
11
11
|
import { bootstrapDataBranch as defaultBootstrapDataBranch, } from "./data-branch-bootstrap.js";
|
|
12
|
+
import { splitFrontmatterBlock } from "./frontmatter-block.js";
|
|
13
|
+
import { parseFrontmatterDocument } from "./frontmatter.js";
|
|
12
14
|
import { findIndexEntryLines } from "./line-scanners.js";
|
|
13
15
|
import { assertReposFile } from "./schemas.js";
|
|
14
16
|
import { computeRepoSlug } from "./wiki-slug.js";
|
|
@@ -661,23 +663,16 @@ export function appendLogEntry(existingLog, params) {
|
|
|
661
663
|
return normalizeText(`${base}\n## [${stamp}] ${logOperation} | ${params.target}\n\n${params.summary}\n\n${sources}\n`);
|
|
662
664
|
}
|
|
663
665
|
function parseFrontmatter(path, content) {
|
|
664
|
-
const
|
|
665
|
-
if (
|
|
666
|
+
const split = splitFrontmatterBlock(content);
|
|
667
|
+
if (split.kind !== 'block') {
|
|
666
668
|
throw new WikiIngestError({
|
|
667
669
|
code: 'INVALID_FRONTMATTER',
|
|
668
670
|
message: `${path} is missing YAML frontmatter`,
|
|
669
671
|
remediation: 'Add frontmatter with type, title, created, and updated fields before ingesting the page.',
|
|
670
672
|
});
|
|
671
673
|
}
|
|
672
|
-
|
|
673
|
-
|
|
674
|
-
throw new WikiIngestError({
|
|
675
|
-
code: 'INVALID_FRONTMATTER',
|
|
676
|
-
message: `${path} frontmatter could not be extracted`,
|
|
677
|
-
remediation: 'Ensure the page begins with a valid YAML frontmatter block.',
|
|
678
|
-
});
|
|
679
|
-
}
|
|
680
|
-
const parsed = parse(frontmatterText);
|
|
674
|
+
// Only a block with no lines is an empty mapping. A block of one blank line parses to null, like any blank text.
|
|
675
|
+
const parsed = split.text === '' && split.textEnding === '' ? {} : parse(split.text);
|
|
681
676
|
if (!isRecord(parsed)) {
|
|
682
677
|
throw new WikiIngestError({
|
|
683
678
|
code: 'INVALID_FRONTMATTER',
|
|
@@ -715,6 +710,7 @@ function prepareWikiPage(page, existingFiles, params, deletedPaths, migrations)
|
|
|
715
710
|
const targetSlug = repoTargetSlug(params.target);
|
|
716
711
|
const trackedNodeId = params.trackedRepoNodeIds?.get(pageSlug);
|
|
717
712
|
const isTargetPage = targetSlug === pageSlug;
|
|
713
|
+
const stamp = isTargetPage ? buildTargetStamp(params, existingFiles[page.path]) : undefined;
|
|
718
714
|
const identity = resolvePageIdentity({
|
|
719
715
|
trackedNodeId,
|
|
720
716
|
targetNodeId: isTargetPage ? params.targetNodeId : undefined,
|
|
@@ -727,7 +723,7 @@ function prepareWikiPage(page, existingFiles, params, deletedPaths, migrations)
|
|
|
727
723
|
: undefined;
|
|
728
724
|
return {
|
|
729
725
|
path: page.path,
|
|
730
|
-
content: updateRepoPageFrontmatter(page.content, preserveNodeId),
|
|
726
|
+
content: updateRepoPageFrontmatter(page.content, preserveNodeId, stamp),
|
|
731
727
|
};
|
|
732
728
|
}
|
|
733
729
|
const nodeMatch = identity.trusted ? findRepoPageByNodeId(existingFiles, identity.nodeId) : undefined;
|
|
@@ -747,12 +743,13 @@ function prepareWikiPage(page, existingFiles, params, deletedPaths, migrations)
|
|
|
747
743
|
nodeId: identity.nodeId,
|
|
748
744
|
title: repoTargetTitle(params.target) ?? incomingFrontmatter.title,
|
|
749
745
|
aliases: [nodeMatch.slug],
|
|
746
|
+
stamp,
|
|
750
747
|
}),
|
|
751
748
|
};
|
|
752
749
|
}
|
|
753
750
|
return {
|
|
754
751
|
path: page.path,
|
|
755
|
-
content: updateRepoPageFrontmatter(page.content, identity.nodeId),
|
|
752
|
+
content: updateRepoPageFrontmatter(page.content, identity.nodeId, stamp),
|
|
756
753
|
};
|
|
757
754
|
}
|
|
758
755
|
function resolvePageIdentity(params) {
|
|
@@ -812,17 +809,98 @@ function mergeRepoPageContent(incomingContent, historicalContents, changes) {
|
|
|
812
809
|
body = body === '' ? historicalBody : `${body}\n\n${historicalBody}`;
|
|
813
810
|
}
|
|
814
811
|
}
|
|
812
|
+
// The ingest matched the old page to the target by a trusted node ID, so the merged page is the target's.
|
|
813
|
+
if (changes.stamp !== undefined)
|
|
814
|
+
addTargetSource(values, changes.stamp);
|
|
815
815
|
return renderFrontmatterDocument(values, body);
|
|
816
816
|
}
|
|
817
|
-
function updateRepoPageFrontmatter(content, nodeId) {
|
|
817
|
+
function updateRepoPageFrontmatter(content, nodeId, stamp) {
|
|
818
818
|
const document = parseFrontmatterDocument(content);
|
|
819
819
|
if (nodeId === undefined)
|
|
820
820
|
delete document.values.node_id;
|
|
821
821
|
else
|
|
822
822
|
document.values.node_id = nodeId;
|
|
823
823
|
delete document.values.database_id;
|
|
824
|
+
if (stamp !== undefined)
|
|
825
|
+
stampEmptySources(document.values, stamp);
|
|
824
826
|
return renderFrontmatterDocument(document.values, document.body);
|
|
825
827
|
}
|
|
828
|
+
function buildTargetStamp(params, existingContent) {
|
|
829
|
+
const parts = repoTargetParts(params.target);
|
|
830
|
+
// Only a target whose owner and name are plain GitHub names is listed on its page. Anything else is written
|
|
831
|
+
// without a stamp, as before, and never makes the ingest throw.
|
|
832
|
+
if (parts === undefined || !isPlainGitHubName(parts.owner) || !isPlainGitHubName(parts.name))
|
|
833
|
+
return undefined;
|
|
834
|
+
return {
|
|
835
|
+
owner: parts.owner,
|
|
836
|
+
name: parts.name,
|
|
837
|
+
url: `https://github.com/${parts.owner}/${parts.name}`,
|
|
838
|
+
accessed: params.timestamp.toISOString().slice(0, 10),
|
|
839
|
+
existingContent,
|
|
840
|
+
};
|
|
841
|
+
}
|
|
842
|
+
/** A GitHub owner or repository name: ASCII letters, digits, `.`, `_` and `-`, and not `.` or `..`. */
|
|
843
|
+
function isPlainGitHubName(value) {
|
|
844
|
+
return value !== '.' && value !== '..' && /^[\w.-]+$/u.test(value);
|
|
845
|
+
}
|
|
846
|
+
/**
|
|
847
|
+
* Lists the target on its own page when the page lists no sources: no `sources` key, or an empty list. A page that
|
|
848
|
+
* lists any source is never changed, and neither is a `sources` value that is not a list. When the page on the
|
|
849
|
+
* branch already lists the target, the whole `sources` list it has is carried over as it is, dates and other
|
|
850
|
+
* entries included, so a repeat write changes nothing. Otherwise the page gets the one new entry.
|
|
851
|
+
*/
|
|
852
|
+
function stampEmptySources(values, stamp) {
|
|
853
|
+
const sources = values.sources;
|
|
854
|
+
if (sources !== undefined && !Array.isArray(sources))
|
|
855
|
+
return;
|
|
856
|
+
if (sources !== undefined && sources.length > 0)
|
|
857
|
+
return;
|
|
858
|
+
values.sources = findExistingSourcesWithTarget(stamp) ?? [{ url: stamp.url, accessed: stamp.accessed }];
|
|
859
|
+
}
|
|
860
|
+
/** Adds the target to a migrated page's sources when no entry matches it, keeping every other entry. */
|
|
861
|
+
function addTargetSource(values, stamp) {
|
|
862
|
+
const sources = values.sources;
|
|
863
|
+
if (sources !== undefined && !Array.isArray(sources))
|
|
864
|
+
return;
|
|
865
|
+
const listed = sources ?? [];
|
|
866
|
+
if (listed.some(entry => sourceMatchesTarget(entry, stamp)))
|
|
867
|
+
return;
|
|
868
|
+
values.sources = [...listed, { url: stamp.url, accessed: stamp.accessed }];
|
|
869
|
+
}
|
|
870
|
+
/** The `sources` list of the page on the branch, when it has an entry for the target. */
|
|
871
|
+
function findExistingSourcesWithTarget(stamp) {
|
|
872
|
+
if (stamp.existingContent === undefined)
|
|
873
|
+
return undefined;
|
|
874
|
+
try {
|
|
875
|
+
const sources = parseFrontmatterDocument(stamp.existingContent).values.sources;
|
|
876
|
+
return Array.isArray(sources) && sources.some(entry => sourceMatchesTarget(entry, stamp)) ? sources : undefined;
|
|
877
|
+
}
|
|
878
|
+
catch {
|
|
879
|
+
return undefined;
|
|
880
|
+
}
|
|
881
|
+
}
|
|
882
|
+
function sourceMatchesTarget(entry, stamp) {
|
|
883
|
+
return isRecord(entry) && typeof entry.url === 'string' && sourceUrlMatchesRepo(entry.url, stamp.owner, stamp.name);
|
|
884
|
+
}
|
|
885
|
+
/**
|
|
886
|
+
* Whether `sourceUrl` refers to the repository `owner`/`name` on github.com by the rule the privacy gate applies to
|
|
887
|
+
* a page's sources (`sourceUrlMatchesRepo` in `check-wiki-private-presence.ts`): `https`, host `github.com`, no
|
|
888
|
+
* port, owner and name equal without case, and any path after them. This package cannot import the gate, so the
|
|
889
|
+
* rule is copied and a test pins the copy against the gate's own cases.
|
|
890
|
+
*/
|
|
891
|
+
function sourceUrlMatchesRepo(sourceUrl, owner, name) {
|
|
892
|
+
let parsed;
|
|
893
|
+
try {
|
|
894
|
+
parsed = new URL(sourceUrl);
|
|
895
|
+
}
|
|
896
|
+
catch {
|
|
897
|
+
return false;
|
|
898
|
+
}
|
|
899
|
+
if (parsed.protocol !== 'https:' || parsed.hostname !== 'github.com' || parsed.port !== '')
|
|
900
|
+
return false;
|
|
901
|
+
const segments = parsed.pathname.split('/').filter(segment => segment.length > 0);
|
|
902
|
+
return segments[0]?.toLowerCase() === owner.toLowerCase() && segments[1]?.toLowerCase() === name.toLowerCase();
|
|
903
|
+
}
|
|
826
904
|
function getExistingRepoNodeId(content, path) {
|
|
827
905
|
if (content === undefined)
|
|
828
906
|
return undefined;
|
|
@@ -859,16 +937,6 @@ function mergeFrontmatterArray(records, key) {
|
|
|
859
937
|
}
|
|
860
938
|
return values;
|
|
861
939
|
}
|
|
862
|
-
function parseFrontmatterDocument(content) {
|
|
863
|
-
const match = /^---\n([\s\S]+?)\n---\n?/u.exec(content);
|
|
864
|
-
if (match === null || match[1] === undefined) {
|
|
865
|
-
throw new Error('validated wiki page is missing frontmatter');
|
|
866
|
-
}
|
|
867
|
-
const parsed = parse(match[1]);
|
|
868
|
-
if (!isRecord(parsed))
|
|
869
|
-
throw new Error('validated wiki page frontmatter is not an object');
|
|
870
|
-
return { values: parsed, body: content.slice(match[0].length) };
|
|
871
|
-
}
|
|
872
940
|
function isRepoPagePath(path) {
|
|
873
941
|
return path.startsWith(`${WIKI_ROOT}/repos/`) && path.endsWith('.md');
|
|
874
942
|
}
|
package/dist/wiki-utils.d.ts
CHANGED
|
@@ -20,8 +20,8 @@ export interface SplitFrontmatterResult {
|
|
|
20
20
|
/**
|
|
21
21
|
* Split raw wiki page content into frontmatter and body.
|
|
22
22
|
*
|
|
23
|
-
* Stricter than a naive split: malformed YAML frontmatter is
|
|
24
|
-
* rather than silently discarded, so downstream safety/lint logic can observe
|
|
23
|
+
* Stricter than a naive split: malformed YAML frontmatter, and a block that is opened and never closed, are
|
|
24
|
+
* surfaced via `error` rather than silently discarded, so downstream safety/lint logic can observe them.
|
|
25
25
|
*/
|
|
26
26
|
export declare function splitFrontmatter(content: string): SplitFrontmatterResult;
|
|
27
27
|
/** Collect `[[target]]` and `[[target|label]]` wikilink targets from page body content. */
|
package/dist/wiki-utils.js
CHANGED
|
@@ -2,38 +2,38 @@ import { Buffer } from 'node:buffer';
|
|
|
2
2
|
import { basename } from 'node:path';
|
|
3
3
|
import { parse } from 'yaml';
|
|
4
4
|
import { findCodeRegions, findRegionAt } from "./code-regions.js";
|
|
5
|
+
import { splitFrontmatterBlock } from "./frontmatter-block.js";
|
|
5
6
|
/** Canonical root directory for wiki page content. Shared by every wiki script. */
|
|
6
7
|
export const WIKI_ROOT = 'knowledge/wiki';
|
|
7
8
|
/**
|
|
8
9
|
* Split raw wiki page content into frontmatter and body.
|
|
9
10
|
*
|
|
10
|
-
* Stricter than a naive split: malformed YAML frontmatter is
|
|
11
|
-
* rather than silently discarded, so downstream safety/lint logic can observe
|
|
11
|
+
* Stricter than a naive split: malformed YAML frontmatter, and a block that is opened and never closed, are
|
|
12
|
+
* surfaced via `error` rather than silently discarded, so downstream safety/lint logic can observe them.
|
|
12
13
|
*/
|
|
13
14
|
export function splitFrontmatter(content) {
|
|
14
|
-
const
|
|
15
|
-
if (
|
|
15
|
+
const split = splitFrontmatterBlock(content);
|
|
16
|
+
if (split.kind === 'none') {
|
|
16
17
|
return { frontmatter: {}, body: content.trim() };
|
|
17
18
|
}
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
return { frontmatter: {}, body: content.trim() };
|
|
19
|
+
if (split.kind === 'malformed') {
|
|
20
|
+
return { frontmatter: {}, body: content.trim(), error: 'Invalid frontmatter: missing closing --- delimiter' };
|
|
21
21
|
}
|
|
22
22
|
let parsed;
|
|
23
23
|
try {
|
|
24
|
-
parsed = parse(
|
|
24
|
+
parsed = parse(split.text);
|
|
25
25
|
}
|
|
26
26
|
catch (error) {
|
|
27
27
|
const message = error instanceof Error ? error.message : 'Unknown YAML parse error';
|
|
28
28
|
return {
|
|
29
29
|
frontmatter: {},
|
|
30
|
-
body:
|
|
30
|
+
body: split.body.trim(),
|
|
31
31
|
error: `Invalid YAML frontmatter: ${message}`,
|
|
32
32
|
};
|
|
33
33
|
}
|
|
34
34
|
return {
|
|
35
35
|
frontmatter: isRecord(parsed) ? parsed : {},
|
|
36
|
-
body:
|
|
36
|
+
body: split.body.trim(),
|
|
37
37
|
};
|
|
38
38
|
}
|
|
39
39
|
/**
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@fro.bot/wiki-write-core",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.2.0",
|
|
4
4
|
"description": "Shared validation and commit primitives for Fro Bot wiki writes",
|
|
5
5
|
"homepage": "https://github.com/fro-bot/.github/tree/main/packages/wiki-write-core#readme",
|
|
6
6
|
"bugs": {
|