@msareen/knowledge-hub-builder 0.2.0 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +1 -0
- package/README.md +28 -10
- package/SPEC.md +14 -3
- package/package.json +3 -1
- package/scripts/cli.ts +1 -0
- package/scripts/doctor.ts +177 -0
- package/scripts/hubs.ts +1 -1
- package/scripts/ingest/acquire.ts +1 -1
- package/scripts/init.ts +8 -1
- package/scripts/lib/extract.ts +36 -4
- package/scripts/lib/relocate.ts +1 -1
- package/scripts/lib/upgrade.ts +17 -5
- package/scripts/lib/util.ts +1 -1
- package/scripts/lint.ts +176 -73
- package/skills/ingest/SKILL.md +1 -1
- package/skills/lint/SKILL.md +24 -2
- package/templates/hub/gitignore +2 -2
package/AGENTS.md
CHANGED
|
@@ -137,6 +137,7 @@ From outside, pass `--hub <dir>` or set `$KHB_HUB`.
|
|
|
137
137
|
| Command | Purpose |
|
|
138
138
|
|---|---|
|
|
139
139
|
| `khb lint` | validate structure against `skills/lint/SKILL.md` |
|
|
140
|
+
| `khb doctor` | read-only report on the hub: version, location, per-bundle concept/raw/backlog counts, transcriber, and the command that fixes each finding. Writes nothing, and does not duplicate `lint` |
|
|
140
141
|
| `khb upgrade` | refresh this hub's package-owned contract docs — runs by itself whenever `khb.json`'s stamped version differs from the installed khb, so these docs always match the CLI |
|
|
141
142
|
| `khb visualize [--port N] [--no-open]` | serve the live bundle graph in your browser; aliases `vis`, `viz` |
|
|
142
143
|
| `khb new-bundle <name>` | scaffold + register a bundle |
|
package/README.md
CHANGED
|
@@ -251,26 +251,43 @@ Commands can be run directly or requested through the matching agent skill.
|
|
|
251
251
|
| `khb new-bundle <name> ["scope"]` | Create and register a bundle |
|
|
252
252
|
| `khb ingest [bundle] [--force] [--skip-ocr] [--skip-audio]` | Acquire and extract declared sources |
|
|
253
253
|
| `khb lint` | Validate routing, bundle structure, and OKF metadata |
|
|
254
|
+
| `khb doctor` | Read-only report: version, location, per-bundle counts, catalog backlog, transcriber |
|
|
254
255
|
| `khb visualize [--port N] [--no-open]` (aliases: `vis`, `viz`) | Serve the live bundle graph on a random free port and open it in your default browser — pan/zoomable cross-bundle map, drill into a bundle for its folder-clustered concepts, rebuild-on-refresh, exits when you close the tab |
|
|
255
256
|
| `khb export <bundle> [dest]` | Export one standalone bundle |
|
|
256
257
|
|
|
257
|
-
|
|
258
|
+
Run against a hub outside the current directory with `--hub <dir>` or `$KHB_HUB`.
|
|
258
259
|
|
|
259
|
-
|
|
260
|
-
- `--skip-audio`
|
|
260
|
+
### `khb doctor` — what state is this hub in?
|
|
261
261
|
|
|
262
|
-
|
|
262
|
+
A single read-only report. It writes nothing and repairs nothing; each finding names the
|
|
263
|
+
command that does.
|
|
264
|
+
|
|
265
|
+
```text
|
|
266
|
+
Hub name, description, stamped version vs installed, location, registered
|
|
267
|
+
Bundles per bundle: concepts, raw/ files, log.md rows, catalog backlog, pending
|
|
268
|
+
Extraction which formats are bundled, and whether a transcriber is ready
|
|
269
|
+
Findings what needs attention, each with its fix
|
|
270
|
+
```
|
|
271
|
+
|
|
272
|
+
Every check existed already, spread across the margins of commands that each knew one of
|
|
273
|
+
them — a move is announced by whatever you happen to run next, the `khb update` hint comes
|
|
274
|
+
out of `khb upgrade`, the uncurated row count out of `khb ingest`, and the transcriber probe
|
|
275
|
+
only ever spoke during a run that needed it. `doctor` asks for the whole picture without
|
|
276
|
+
changing anything to get it.
|
|
277
|
+
|
|
278
|
+
It is not a validator: `khb lint` still owns structure and OKF conformance, and `doctor`
|
|
279
|
+
points at it rather than repeating a rule.
|
|
263
280
|
|
|
264
281
|
### Moving between hubs
|
|
265
282
|
|
|
266
|
-
These
|
|
283
|
+
These run **outside** any hub, from any terminal:
|
|
267
284
|
|
|
268
285
|
| Command | Purpose |
|
|
269
286
|
|---|---|
|
|
270
287
|
| `khb` | Open a hub. One registered hub asks; several show a list and take a pick; none walks you through creating the first |
|
|
271
288
|
| `khb list [--json]` | Every hub on this machine, with its description and path |
|
|
272
|
-
| `khb go
|
|
273
|
-
| `khb agent [name] [--command X] [--args "…"]` | Which agent `khb go` launches — `claude`, `codex`, anything on your PATH, or `none` to just print the path |
|
|
289
|
+
| `khb go [name\|N] [--path] [--no-agent] [--agent X]` | Open one by name or list position. `--path` prints just the path, for `cd "$(khb go --path work)"`; `--no-agent` prints the path and the `cd` line without launching anything; `--agent X` launches `X` for this run only, leaving the configured default alone |
|
|
290
|
+
| `khb agent [name\|none] [--command X] [--args "…"]` | Which agent `khb go` launches — `claude`, `codex`, anything on your PATH, or `none` to just print the path |
|
|
274
291
|
| `khb update [new-path] [--path\|-p] [--schema\|-s] [--from <old>] [--dry-run]` | Repair the hub: path references after a move, and/or backfill `sources.yaml` to the current schema. No flag runs both |
|
|
275
292
|
| `khb forget <name>` | Drop a hub from the list. The folder is untouched |
|
|
276
293
|
|
|
@@ -278,7 +295,8 @@ These four run **outside** any hub, from any terminal:
|
|
|
278
295
|
cold terminal ends with an agent open on the right folder. No program can change its
|
|
279
296
|
parent shell's directory, which is why the `cd` line is printed rather than performed.
|
|
280
297
|
|
|
281
|
-
The list lives in `~/.khb/hubs-config.json` (`%USERPROFILE%\.khb` on Windows
|
|
298
|
+
The list lives in `~/.khb/hubs-config.json` (`%USERPROFILE%\.khb` on Windows; `$KHB_HOME`
|
|
299
|
+
overrides the directory, for a portable install or a test run). It holds
|
|
282
300
|
paths and one launch command — no knowledge — and fills itself in: any khb command run
|
|
283
301
|
inside a hub registers it, so hubs you already had show up without a migration step.
|
|
284
302
|
Delete the file and the next command in each hub puts it back.
|
|
@@ -324,7 +342,7 @@ the same spelling of the new path; matches must end at a path boundary, so movin
|
|
|
324
342
|
never touches a sibling `…/older`; and the new path is matched too and rewritten to itself,
|
|
325
343
|
which is what makes overlapping moves safe — lifting a hub out of its parent, or pushing it
|
|
326
344
|
down into a subdirectory of where it stood — and makes a second run a no-op.
|
|
327
|
-
`.git/`, `node_modules/` and the
|
|
345
|
+
`.git/`, `node_modules/` and the `.ingest-cache/` cache are skipped, as are binaries.
|
|
328
346
|
|
|
329
347
|
**`--schema`/`-s`** — a bundle's `sources.yaml` can predate a field khb's since learned about
|
|
330
348
|
(e.g. `exclude:`), with no way to discover it short of reading the docs. This backfills
|
|
@@ -373,7 +391,7 @@ everything under `bundles/`.
|
|
|
373
391
|
|
|
374
392
|
## Privacy
|
|
375
393
|
|
|
376
|
-
`raw/` and the extraction cache under
|
|
394
|
+
`raw/` and the extraction cache under `.ingest-cache/` are gitignored. `log.md` is committed and
|
|
377
395
|
records source paths, which may be absolute. Ignore `log.md` before the first commit if
|
|
378
396
|
those paths are sensitive.
|
|
379
397
|
|
package/SPEC.md
CHANGED
|
@@ -72,7 +72,7 @@ my-knowledge/ # ~/OneDrive/my-knowledge, a private repo, a shar
|
|
|
72
72
|
│ │ ├── index.md # (each may carry its own index)
|
|
73
73
|
│ │ └── <concept>.md
|
|
74
74
|
│ └── raw/ # ingested/extracted material, pre-curation (gitignored)
|
|
75
|
-
├──
|
|
75
|
+
├── .ingest-cache/extracted/ # hub-wide extraction cache, keyed by content hash (gitignored)
|
|
76
76
|
│
|
|
77
77
|
│ ── below: package-owned copies, refreshed by `khb upgrade`, never hand-edited ──
|
|
78
78
|
├── AGENTS.md # common contract; Codex discovers this directly
|
|
@@ -121,14 +121,21 @@ acting on them** — an agent can never read a protocol the CLI no longer implem
|
|
|
121
121
|
│ ├── new-bundle.ts # scaffold from .bundle_template, register in outer.index.md
|
|
122
122
|
│ ├── export.ts # bundle + common patterns → standalone shareable folder
|
|
123
123
|
│ ├── lint.ts # enforce skills/lint/SKILL.md across the hub
|
|
124
|
+
│ ├── doctor.ts # read-only state report; writes nothing, repairs nothing
|
|
124
125
|
│ ├── visualize.ts # serve the live bundle graph from indexes + refs
|
|
125
126
|
│ ├── ingest/ # folder.ts / files.ts / web.ts → acquire.ts → bundle/raw
|
|
126
127
|
│ └── lib/
|
|
128
|
+
│ ├── args.ts # argv helpers; what they consume they remove, leaving positionals
|
|
129
|
+
│ ├── create.ts # making a hub, shared by `khb init` and the first-run wizard
|
|
127
130
|
│ ├── extract.ts # every local extractor + the content-hash cache
|
|
131
|
+
│ ├── graph.ts # graph data for the visualizer — read-only, never writes
|
|
132
|
+
│ ├── graph-page.ts # the visualizer's browser UI, rendered from that data
|
|
128
133
|
│ ├── ledger.ts # log.md read/write
|
|
134
|
+
│ ├── log.ts # progress reporting: each unit announces itself before it runs
|
|
129
135
|
│ ├── paths.ts # package-side paths — importing it never needs a hub
|
|
130
136
|
│ ├── registry.ts # ~/.khb/hubs-config.json: where this machine's hubs are
|
|
131
137
|
│ ├── relocate.ts # khb update --path's path rewriter — pure text, no judgement
|
|
138
|
+
│ ├── scaffold.ts # bundle creation + lookup, shared by new-bundle and ingest
|
|
132
139
|
│ ├── schema.ts # khb update --schema: sources.yaml field diff/apply
|
|
133
140
|
│ ├── upgrade.ts # the refresh itself: `khb upgrade` and the drift check
|
|
134
141
|
│ └── util.ts # hub resolution + shared helpers
|
|
@@ -272,7 +279,7 @@ unattended:
|
|
|
272
279
|
changes nothing. Only old and new naming *the same directory* is refused, there being no
|
|
273
280
|
move to repair.
|
|
274
281
|
|
|
275
|
-
`.git/`, `node_modules/` and the
|
|
282
|
+
`.git/`, `node_modules/` and the `.ingest-cache/` extraction cache are not walked; binary files and
|
|
276
283
|
anything over 8 MB are skipped. `--dry-run` reports the file-by-file hit count and writes
|
|
277
284
|
nothing.
|
|
278
285
|
|
|
@@ -483,7 +490,7 @@ raw files, with one hard rule — subagents write concept docs, the orchestrator
|
|
|
483
490
|
|
|
484
491
|
Extraction is deterministic, so `khb` owns all of it. Common formats use bundled pure-JS
|
|
485
492
|
libraries with no system install; results are cached hub-wide by content hash
|
|
486
|
-
(
|
|
493
|
+
(`.ingest-cache/extracted/<sha256>.md`) and reused across bundles. Nothing here contacts a model:
|
|
487
494
|
tesseract and whisper are local binaries, expensive in CPU but reproducible, which is what
|
|
488
495
|
puts them on the CLI side of the §Division-of-labor line.
|
|
489
496
|
|
|
@@ -526,6 +533,10 @@ drops, and rewrites the raw file with `extract_tool: claude-vision`.
|
|
|
526
533
|
- every concept doc is listed in an index and carries OKF frontmatter (`type` required)
|
|
527
534
|
- `refs.md` targets exist; no cross-bundle inline links from concept docs
|
|
528
535
|
- index files contain links only (routing, not content)
|
|
536
|
+
- intra-bundle links from a concept doc resolve to a file that exists (warning — OKF
|
|
537
|
+
tolerates a link to knowledge not yet written)
|
|
538
|
+
- `log.md` agrees with the bundle: a `curated` value naming a missing concept is an error,
|
|
539
|
+
a `raw` value naming a missing file or a `raw/` file with no row is a warning
|
|
529
540
|
|
|
530
541
|
## 8. Visualizer
|
|
531
542
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@msareen/knowledge-hub-builder",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.2",
|
|
4
4
|
"description": "Knowledge Hub Builder — a bundle-of-bundles knowledge base you build with an agent",
|
|
5
5
|
"keywords": ["knowledge-base", "okf", "agent", "wiki", "bundles", "claude", "codex", "rag"],
|
|
6
6
|
"license": "MIT",
|
|
@@ -35,6 +35,8 @@
|
|
|
35
35
|
"scripts": {
|
|
36
36
|
"khb": "bun scripts/cli.ts",
|
|
37
37
|
"lint": "bun scripts/cli.ts lint",
|
|
38
|
+
"upgrade": "bun scripts/cli.ts upgrade",
|
|
39
|
+
"doctor": "bun scripts/cli.ts doctor",
|
|
38
40
|
"visualize": "bun scripts/cli.ts visualize",
|
|
39
41
|
"new-bundle": "bun scripts/cli.ts new-bundle",
|
|
40
42
|
"export": "bun scripts/cli.ts export",
|
package/scripts/cli.ts
CHANGED
|
@@ -13,6 +13,7 @@ const COMMANDS: Record<string, { load: () => Promise<unknown>; usage: string; de
|
|
|
13
13
|
desc: "acquire + extract declared sources → raw/",
|
|
14
14
|
},
|
|
15
15
|
lint: { load: () => import("./lint"), usage: "khb lint", desc: "validate the hub against skills/lint/SKILL.md" },
|
|
16
|
+
doctor: { load: () => import("./doctor"), usage: "khb doctor", desc: "read-only report on this hub's state" },
|
|
16
17
|
visualize: {
|
|
17
18
|
load: () => import("./visualize"),
|
|
18
19
|
usage: "khb visualize [--port N] [--no-open]",
|
|
@@ -0,0 +1,177 @@
|
|
|
1
|
+
// khb doctor — one read-only report on the state of a hub.
|
|
2
|
+
//
|
|
3
|
+
// Every check here already existed, scattered across the preambles of commands that each
|
|
4
|
+
// knew one of them: cli.ts announces a move and a version drift, `khb upgrade` prints the
|
|
5
|
+
// `khb update` hint, `khb ingest` counts the uncurated rows on its way out, and the
|
|
6
|
+
// transcriber probe only ever spoke during a run that needed it. So the answer to "what
|
|
7
|
+
// state is this hub in?" was: run several commands that change things and read their
|
|
8
|
+
// margins. This command asks nothing of the hub but to look at it.
|
|
9
|
+
//
|
|
10
|
+
// It writes nothing. That is the point, and it is also the boundary: `doctor` reports and
|
|
11
|
+
// names the command that repairs, but never repairs. `khb lint` stays the structural
|
|
12
|
+
// validator — doctor counts and points at it rather than duplicating a rule.
|
|
13
|
+
import { HUB, BUNDLES, listBundles, read, join, existsSync } from "./lib/util";
|
|
14
|
+
import { readLedger } from "./lib/ledger";
|
|
15
|
+
import { staleLocations, hubVersion } from "./lib/upgrade";
|
|
16
|
+
import { diffSourcesYamlAll } from "./lib/schema";
|
|
17
|
+
import { transcriberStatus } from "./lib/extract";
|
|
18
|
+
import { version, MARKER, markerIn } from "./lib/paths";
|
|
19
|
+
import { listHubs, canonical } from "./lib/registry";
|
|
20
|
+
import { section, detail, totalElapsed } from "./lib/log";
|
|
21
|
+
import { rejectUnknownFlags } from "./lib/args";
|
|
22
|
+
import { readdirSync, statSync } from "node:fs";
|
|
23
|
+
import { relative } from "node:path";
|
|
24
|
+
|
|
25
|
+
rejectUnknownFlags(process.argv.slice(2), "khb doctor");
|
|
26
|
+
|
|
27
|
+
/** Findings are advisory: doctor's exit code reports whether it ran, not what it found. */
|
|
28
|
+
const findings: string[] = [];
|
|
29
|
+
const flag = (msg: string, fix?: string) => findings.push(fix ? `${msg}\n fix: ${fix}` : msg);
|
|
30
|
+
|
|
31
|
+
console.log(`khb doctor → ${HUB}`);
|
|
32
|
+
|
|
33
|
+
// ---- Hub identity -----------------------------------------------------------------------
|
|
34
|
+
const marker = (() => {
|
|
35
|
+
try {
|
|
36
|
+
return JSON.parse(read(join(HUB, markerIn(HUB) ?? MARKER))) as Record<string, unknown>;
|
|
37
|
+
} catch {
|
|
38
|
+
return {} as Record<string, unknown>;
|
|
39
|
+
}
|
|
40
|
+
})();
|
|
41
|
+
|
|
42
|
+
section("Hub");
|
|
43
|
+
detail(`name ${(marker.name as string) || "(unset — khb.json 'name')"}`);
|
|
44
|
+
detail(`description ${(marker.description as string) || "(unset — khb.json 'description')"}`);
|
|
45
|
+
|
|
46
|
+
// Drift is normally self-healing: cli.ts refreshes a hub before any in-hub command, this one
|
|
47
|
+
// included, so a mismatch here means the refresh was suppressed rather than that it is due.
|
|
48
|
+
const stamped = hubVersion(HUB);
|
|
49
|
+
const installed = version();
|
|
50
|
+
detail(
|
|
51
|
+
`khb version ${stamped ?? "unstamped"}` +
|
|
52
|
+
(stamped === installed ? ` (matches installed)` : ` — installed is ${installed}`),
|
|
53
|
+
);
|
|
54
|
+
if (stamped !== installed)
|
|
55
|
+
flag(
|
|
56
|
+
`hub is stamped ${stamped ?? "unstamped"} but khb is ${installed}; its contract docs may be a version behind.`,
|
|
57
|
+
process.env.KHB_NO_AUTO_UPGRADE ? "unset KHB_NO_AUTO_UPGRADE, or run: khb upgrade" : "khb upgrade",
|
|
58
|
+
);
|
|
59
|
+
|
|
60
|
+
// ---- Location and registry --------------------------------------------------------------
|
|
61
|
+
// A hub moved more than once before anyone repaired it carries every former home, and
|
|
62
|
+
// `khb update --path` rewrites them all in one pass — so say how many there are rather than
|
|
63
|
+
// showing the most recent and implying it is the only one.
|
|
64
|
+
const stale = staleLocations(HUB);
|
|
65
|
+
detail(
|
|
66
|
+
`location ${
|
|
67
|
+
stale.length
|
|
68
|
+
? `moved from ${stale[stale.length - 1]}${stale.length > 1 ? ` (+${stale.length - 1} earlier)` : ""}`
|
|
69
|
+
: "matches the marker"
|
|
70
|
+
}`,
|
|
71
|
+
);
|
|
72
|
+
if (stale.length)
|
|
73
|
+
flag(
|
|
74
|
+
`this hub has moved; absolute paths recorded inside it still name ${stale.length > 1 ? "former locations" : "its former location"}.`,
|
|
75
|
+
"khb update --path (khb update --path --dry-run to preview)",
|
|
76
|
+
);
|
|
77
|
+
|
|
78
|
+
const registered = listHubs().some((entry) => canonical(entry.path) === canonical(HUB));
|
|
79
|
+
detail(`registered ${registered ? "yes" : "no — 'khb list' and 'khb go' will not offer it"}`);
|
|
80
|
+
|
|
81
|
+
// ---- sources.yaml schema ----------------------------------------------------------------
|
|
82
|
+
const schemaDiffs = diffSourcesYamlAll(HUB);
|
|
83
|
+
if (schemaDiffs.length) {
|
|
84
|
+
const fields = schemaDiffs.reduce((total, diff) => total + diff.changes.length, 0);
|
|
85
|
+
flag(
|
|
86
|
+
`${fields} sources.yaml field(s) across ${schemaDiffs.length} bundle(s) predate the current schema.`,
|
|
87
|
+
"khb update --schema (khb update --schema --dry-run to preview)",
|
|
88
|
+
);
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
// ---- Bundles ----------------------------------------------------------------------------
|
|
92
|
+
const RESERVED = ["index.md", "log.md", "refs.md"];
|
|
93
|
+
|
|
94
|
+
/** Concept docs: every .md in the bundle outside raw/ that is not a reserved filename. */
|
|
95
|
+
function conceptCount(dir: string): number {
|
|
96
|
+
const walk = (current: string): string[] =>
|
|
97
|
+
readdirSync(current).flatMap((entry: string) => {
|
|
98
|
+
const path = join(current, entry);
|
|
99
|
+
if (statSync(path).isDirectory()) return entry === "raw" ? [] : walk(path);
|
|
100
|
+
return [relative(dir, path).replaceAll("\\", "/")];
|
|
101
|
+
});
|
|
102
|
+
return walk(dir).filter(
|
|
103
|
+
(file) => file.endsWith(".md") && !RESERVED.includes(file.split("/").pop()!),
|
|
104
|
+
).length;
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
const bundles = listBundles();
|
|
108
|
+
section(`Bundles (${bundles.length})`);
|
|
109
|
+
|
|
110
|
+
if (!bundles.length) {
|
|
111
|
+
detail("none yet — khb new-bundle <name> \"scope\"");
|
|
112
|
+
} else {
|
|
113
|
+
const summaries = bundles.map((bundle) => {
|
|
114
|
+
const dir = join(BUNDLES, bundle);
|
|
115
|
+
const ledger = readLedger(dir);
|
|
116
|
+
const rawDir = join(dir, "raw");
|
|
117
|
+
const rawFiles = existsSync(rawDir)
|
|
118
|
+
? (readdirSync(rawDir, { recursive: true }) as string[]).filter((file) => file.endsWith(".md"))
|
|
119
|
+
.length
|
|
120
|
+
: 0;
|
|
121
|
+
const rows = [...ledger.values()];
|
|
122
|
+
return {
|
|
123
|
+
bundle,
|
|
124
|
+
concepts: conceptCount(dir),
|
|
125
|
+
rawFiles,
|
|
126
|
+
rows: rows.length,
|
|
127
|
+
// The catalog backlog in the ledger's own terms — "in raw/ but not yet distilled into
|
|
128
|
+
// a concept doc" — so a row must have a raw file to be part of it. A row with neither
|
|
129
|
+
// is *pending*, a different state with a different fix, and counting it in both would
|
|
130
|
+
// overstate the work cataloging can actually pick up.
|
|
131
|
+
backlog: rows.filter((row) => row.raw && !row.curated).length,
|
|
132
|
+
pending: rows.filter((row) => !row.raw).length,
|
|
133
|
+
};
|
|
134
|
+
});
|
|
135
|
+
|
|
136
|
+
const nameWidth = Math.max(6, ...summaries.map((summary) => summary.bundle.length));
|
|
137
|
+
detail(`${"bundle".padEnd(nameWidth)} concepts raw/ rows backlog pending`);
|
|
138
|
+
for (const summary of summaries)
|
|
139
|
+
detail(
|
|
140
|
+
`${summary.bundle.padEnd(nameWidth)} ${String(summary.concepts).padStart(8)} ` +
|
|
141
|
+
`${String(summary.rawFiles).padStart(4)} ${String(summary.rows).padStart(4)} ` +
|
|
142
|
+
`${String(summary.backlog).padStart(7)} ${String(summary.pending).padStart(7)}`,
|
|
143
|
+
);
|
|
144
|
+
|
|
145
|
+
const backlog = summaries.reduce((total, summary) => total + summary.backlog, 0);
|
|
146
|
+
const pending = summaries.reduce((total, summary) => total + summary.pending, 0);
|
|
147
|
+
if (backlog)
|
|
148
|
+
flag(
|
|
149
|
+
`${backlog} row(s) in raw/ but not yet cataloged, across ` +
|
|
150
|
+
`${summaries.filter((summary) => summary.backlog).length} bundle(s).`,
|
|
151
|
+
"ask an agent to catalog the bundle (skills/catalog/SKILL.md)",
|
|
152
|
+
);
|
|
153
|
+
// An empty `raw` is a source khb saw and could not convert — a missing extractor, a
|
|
154
|
+
// protected file, or a --skip flag. It is not a failed run, but it is work still owed.
|
|
155
|
+
if (pending)
|
|
156
|
+
flag(
|
|
157
|
+
`${pending} source(s) acquired but not extracted (empty 'raw' in log.md).`,
|
|
158
|
+
"khb ingest <bundle> — after installing whatever the row's reason names",
|
|
159
|
+
);
|
|
160
|
+
}
|
|
161
|
+
|
|
162
|
+
// ---- Extraction -------------------------------------------------------------------------
|
|
163
|
+
section("Extraction");
|
|
164
|
+
detail("bundled text, PDF, DOCX, ODT, XLSX, PPTX, OCR (images + scanned PDFs), captions");
|
|
165
|
+
const transcriber = await transcriberStatus();
|
|
166
|
+
detail(`transcriber ${transcriber.detail}`);
|
|
167
|
+
if (!transcriber.ready)
|
|
168
|
+
flag(`no transcriber is ready, so audio and video will pend.`, transcriber.fix);
|
|
169
|
+
|
|
170
|
+
// ---- Findings ---------------------------------------------------------------------------
|
|
171
|
+
section(findings.length ? `Findings (${findings.length})` : "Findings");
|
|
172
|
+
if (!findings.length) detail("none — nothing here needs attention.");
|
|
173
|
+
else for (const finding of findings) detail(`- ${finding}`);
|
|
174
|
+
|
|
175
|
+
section("Next");
|
|
176
|
+
detail("khb lint structural and OKF validation (doctor does not duplicate it)");
|
|
177
|
+
console.log(`\ndoctor: ${findings.length} finding(s) across ${bundles.length} bundle(s) in ${totalElapsed()}`);
|
package/scripts/hubs.ts
CHANGED
|
@@ -461,7 +461,7 @@ function repairPaths(newPath: string, fromOpt: string | undefined, dryRun: boole
|
|
|
461
461
|
detail(`scanning ${newPath} …`);
|
|
462
462
|
const { scanned, hits, failed } = rewritePaths(newPath, froms, newPath, {
|
|
463
463
|
dryRun,
|
|
464
|
-
onStart: (n) => detail(`${n} file(s) to check (skipping .git/, node_modules/,
|
|
464
|
+
onStart: (n) => detail(`${n} file(s) to check (skipping .git/, node_modules/, .ingest-cache/)`),
|
|
465
465
|
});
|
|
466
466
|
|
|
467
467
|
const total = hits.reduce((n, h) => n + h.count, 0);
|
|
@@ -223,7 +223,7 @@ export function report(c: Counters) {
|
|
|
223
223
|
line(c.moved, "moved/renamed — existing raw file and catalog entry kept");
|
|
224
224
|
line(c.copied, "text file(s) copied");
|
|
225
225
|
line(c.extracted, "extracted");
|
|
226
|
-
line(c.fromCache, "reused from the extraction cache (
|
|
226
|
+
line(c.fromCache, "reused from the extraction cache (.ingest-cache/extracted/)");
|
|
227
227
|
line(c.ocrd, "read by OCR");
|
|
228
228
|
line(c.transcribed, "transcribed");
|
|
229
229
|
line(c.captioned, "read from a caption sidecar (no transcription needed)");
|
package/scripts/init.ts
CHANGED
|
@@ -50,7 +50,14 @@ if (upgrading) {
|
|
|
50
50
|
|
|
51
51
|
const { from, to, synced, pruned, renamed } = upgradeHub(HUB);
|
|
52
52
|
console.log(`Upgraded ${HUB}: ${from ?? "?"} -> ${to}`);
|
|
53
|
-
|
|
53
|
+
// An empty list is not an empty result: in the khb development repo the package *is* the
|
|
54
|
+
// hub, so every managed path is its own source and there is genuinely nothing to copy.
|
|
55
|
+
// Printing a bare "refreshed:" there reads as a failure rather than as the no-op it is.
|
|
56
|
+
console.log(
|
|
57
|
+
synced.length
|
|
58
|
+
? ` refreshed: ${synced.join(", ")}`
|
|
59
|
+
: ` refreshed: nothing to copy — this hub is its own package`,
|
|
60
|
+
);
|
|
54
61
|
if (renamed) console.log(` renamed: ${renamed} -> ${MARKER}`);
|
|
55
62
|
if (pruned.length) console.log(` removed (no longer part of the contract): ${pruned.join(", ")}`);
|
|
56
63
|
console.log(`Your bundles/ and outer.index.md were not touched. Next: khb lint`);
|
package/scripts/lib/extract.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
// Content-addressed extraction cache:
|
|
1
|
+
// Content-addressed extraction cache: .ingest-cache/extracted/<sha256>.md
|
|
2
2
|
//
|
|
3
3
|
// Binary formats have to be converted before anything can read them. That conversion is
|
|
4
4
|
// expensive and deterministic per content hash, so it is cached hub-wide rather than
|
|
@@ -21,10 +21,10 @@
|
|
|
21
21
|
// and named in the provenance header, so curation can escalate to a vision read of the
|
|
22
22
|
// source instead of trusting garbled text.
|
|
23
23
|
import { mkdirSync, writeFileSync, readFileSync, existsSync, rmSync, readdirSync } from "node:fs";
|
|
24
|
-
import {
|
|
24
|
+
import { INGEST_CACHE, join, basename } from "./util";
|
|
25
25
|
import { note } from "./log";
|
|
26
26
|
|
|
27
|
-
export const EXTRACTED = join(
|
|
27
|
+
export const EXTRACTED = join(INGEST_CACHE, "extracted");
|
|
28
28
|
|
|
29
29
|
/** Below this many characters per page, a PDF is a picture of a document, not a document. */
|
|
30
30
|
const SCANNED_CHARS_PER_PAGE = 20;
|
|
@@ -611,7 +611,7 @@ export async function transcribeCached(path: string, hash: string, model = "base
|
|
|
611
611
|
|
|
612
612
|
// Both engines write files rather than to stdout; give each run a scratch directory of
|
|
613
613
|
// its own so a stray sibling transcript is never mistaken for this one's.
|
|
614
|
-
const out = join(
|
|
614
|
+
const out = join(INGEST_CACHE, "tmp", hash.slice(0, 12));
|
|
615
615
|
mkdirSync(out, { recursive: true });
|
|
616
616
|
try {
|
|
617
617
|
return eng.kind === "vno"
|
|
@@ -623,3 +623,35 @@ export async function transcribeCached(path: string, hash: string, model = "base
|
|
|
623
623
|
rmSync(out, { recursive: true, force: true });
|
|
624
624
|
}
|
|
625
625
|
}
|
|
626
|
+
|
|
627
|
+
/**
|
|
628
|
+
* The transcriber situation as a fact to report, rather than as a step in a run.
|
|
629
|
+
*
|
|
630
|
+
* `asrEngine()` above is the ingest path: it caches its answer for the process and speaks
|
|
631
|
+
* up on stderr, both of which are right in the middle of a run and wrong for `khb doctor`,
|
|
632
|
+
* which prints its own report and must not warn on the side. Same probe order — vno, then
|
|
633
|
+
* whisper, then faster-whisper — so the two never disagree about what would actually run.
|
|
634
|
+
*/
|
|
635
|
+
export async function transcriberStatus(): Promise<{ ready: boolean; detail: string; fix?: string }> {
|
|
636
|
+
const vno = await vnoStatus();
|
|
637
|
+
if (vno.state === "ready") return { ready: true, detail: "vno (whisper.cpp)" };
|
|
638
|
+
|
|
639
|
+
const whisper =
|
|
640
|
+
(await runCli(["whisper", "--help"])) ? "whisper"
|
|
641
|
+
: (await runCli(["faster-whisper", "--help"])) ? "faster-whisper"
|
|
642
|
+
: "";
|
|
643
|
+
|
|
644
|
+
if (vno.state === "unset-up") {
|
|
645
|
+
const missing = vno.blockers.length ? `: ${vno.blockers.join(", ")}` : "";
|
|
646
|
+
// Amber, exactly as in a run: an unset-up vno with whisper behind it is not a problem.
|
|
647
|
+
return whisper
|
|
648
|
+
? { ready: true, detail: `${whisper} — vno installed but not set up${missing}`, fix: "vno setup" }
|
|
649
|
+
: { ready: false, detail: `vno installed but not set up${missing}`, fix: "vno setup" };
|
|
650
|
+
}
|
|
651
|
+
if (whisper) return { ready: true, detail: whisper };
|
|
652
|
+
return {
|
|
653
|
+
ready: false,
|
|
654
|
+
detail: "none on PATH — audio and video pend as rows, everything else ingests",
|
|
655
|
+
fix: "npm install -g @msareen/voice-notes-organizer (or: pip install -U openai-whisper)",
|
|
656
|
+
};
|
|
657
|
+
}
|
package/scripts/lib/relocate.ts
CHANGED
|
@@ -10,7 +10,7 @@ import { join, relative } from "node:path";
|
|
|
10
10
|
import { ticker } from "./log";
|
|
11
11
|
|
|
12
12
|
/** Directories never walked: VCS internals, installed packages, regenerable caches. */
|
|
13
|
-
const SKIP_DIRS = new Set([".git", "node_modules", "
|
|
13
|
+
const SKIP_DIRS = new Set([".git", "node_modules", ".ingest-cache"]);
|
|
14
14
|
|
|
15
15
|
/** A file bigger than this is a corpus artefact, not something holding a path reference. */
|
|
16
16
|
const MAX_BYTES = 8 * 1024 * 1024;
|
package/scripts/lib/upgrade.ts
CHANGED
|
@@ -32,16 +32,28 @@ export function hubVersion(hub: string): string | undefined {
|
|
|
32
32
|
}
|
|
33
33
|
}
|
|
34
34
|
|
|
35
|
-
/**
|
|
35
|
+
/**
|
|
36
|
+
* Copy every package-owned contract file into the hub, replacing what is there.
|
|
37
|
+
*
|
|
38
|
+
* A file is skipped when source and destination are the same path. That is not a corner
|
|
39
|
+
* case: the khb development repo is its own hub (see the `note` in this repo's `khb.json`),
|
|
40
|
+
* so `PKG` and `hub` are one directory there and every managed path resolves to itself.
|
|
41
|
+
* `cpSync` rejects that outright with EINVAL, which turned any version drift in the dev
|
|
42
|
+
* repo into a crash on *every* in-hub command — the working copy is already the source of
|
|
43
|
+
* truth, so the honest answer is that there is nothing to copy.
|
|
44
|
+
*/
|
|
36
45
|
export function syncManaged(hub: string): string[] {
|
|
37
46
|
const done: string[] = [];
|
|
38
|
-
|
|
39
|
-
|
|
47
|
+
const same = (left: string, right: string) =>
|
|
48
|
+
process.platform === "win32" ? left.toLowerCase() === right.toLowerCase() : left === right;
|
|
49
|
+
for (const managed of MANAGED) {
|
|
50
|
+
const src = join(PKG, managed);
|
|
40
51
|
if (!existsSync(src)) continue;
|
|
41
|
-
const dest = join(hub,
|
|
52
|
+
const dest = join(hub, managed);
|
|
53
|
+
if (same(resolve(src), resolve(dest))) continue;
|
|
42
54
|
mkdirSync(dirname(dest), { recursive: true });
|
|
43
55
|
cpSync(src, dest, { recursive: true, force: true });
|
|
44
|
-
done.push(statSync(src).isDirectory() ? `${
|
|
56
|
+
done.push(statSync(src).isDirectory() ? `${managed}/` : managed);
|
|
45
57
|
}
|
|
46
58
|
return done;
|
|
47
59
|
}
|
package/scripts/lib/util.ts
CHANGED
|
@@ -25,7 +25,7 @@ function resolveHub(): string {
|
|
|
25
25
|
|
|
26
26
|
export const HUB = resolveHub();
|
|
27
27
|
export const BUNDLES = join(HUB, "bundles");
|
|
28
|
-
export const
|
|
28
|
+
export const INGEST_CACHE = join(HUB, ".ingest-cache");
|
|
29
29
|
export { TEMPLATE, markerIn } from "./paths";
|
|
30
30
|
|
|
31
31
|
export function listBundles(): string[] {
|
package/scripts/lint.ts
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
// khb lint — enforce skills/lint/SKILL.md (structural rules + OKF v0.1 conformance) across the hub
|
|
2
2
|
import { HUB, BUNDLES, listBundles, read, mdLinks, refTargets, join, existsSync } from "./lib/util";
|
|
3
|
+
import { readLedger } from "./lib/ledger";
|
|
3
4
|
import { detail, section, totalElapsed } from "./lib/log";
|
|
4
5
|
import { readdirSync, statSync } from "node:fs";
|
|
5
6
|
import { dirname, relative } from "node:path";
|
|
@@ -13,156 +14,258 @@ rejectUnknownFlags(process.argv.slice(2), "khb lint");
|
|
|
13
14
|
const OKF_FIELDS = new Set(["type", "title", "description", "resource", "tags", "timestamp"]);
|
|
14
15
|
|
|
15
16
|
/** Accepts a YAML-parsed Date (unquoted) or an ISO-8601 string (quoted). */
|
|
16
|
-
const isTimestamp = (
|
|
17
|
-
|
|
17
|
+
const isTimestamp = (value: unknown) =>
|
|
18
|
+
value instanceof Date
|
|
19
|
+
? !isNaN(value.getTime())
|
|
20
|
+
: typeof value === "string" && !isNaN(Date.parse(value));
|
|
18
21
|
|
|
19
22
|
let errors = 0, warnings = 0;
|
|
20
23
|
const err = (rule: string, msg: string) => { errors++; console.error(`ERROR ${rule}: ${msg}`); };
|
|
21
24
|
const warn = (rule: string, msg: string) => { warnings++; console.warn(`warn ${rule}: ${msg}`); };
|
|
22
25
|
|
|
23
|
-
|
|
26
|
+
/**
|
|
27
|
+
* Drop everything that is markup *about* markdown rather than markdown: HTML comments, and
|
|
28
|
+
* every code span or fenced block.
|
|
29
|
+
*
|
|
30
|
+
* Code has to go before any link is extracted. A doc explaining the index form writes
|
|
31
|
+
* `` `* [Title](path.md) - description` `` as an example, and a link rule that cannot tell
|
|
32
|
+
* an example from a link reports it as a dead one — which in a project whose concept docs
|
|
33
|
+
* document its own conventions is a false positive on exactly the docs most worth writing.
|
|
34
|
+
* The same reasoning covers L6: a code sample *showing* a forbidden cross-bundle link is
|
|
35
|
+
* teaching the rule, not breaking it.
|
|
36
|
+
*
|
|
37
|
+
* One pattern handles spans and fences alike: a run of N backticks closes on the next run
|
|
38
|
+
* of exactly N, so ``` fences and the `` `…` `` form that quotes inner backticks both pair
|
|
39
|
+
* correctly. An unbalanced backtick simply fails to match and leaves the text alone.
|
|
40
|
+
*/
|
|
41
|
+
const stripNonProse = (markdown: string) =>
|
|
42
|
+
markdown.replace(/<!--[\s\S]*?-->/g, "").replace(/(`+)[\s\S]*?\1/g, "");
|
|
43
|
+
|
|
24
44
|
const RESERVED = ["index.md", "log.md", "refs.md"]; // refs.md is KHB-reserved
|
|
25
45
|
const bundles = listBundles();
|
|
26
46
|
const outerIndex = read(join(HUB, "outer.index.md"));
|
|
27
47
|
|
|
28
48
|
/** All files under dir (relative paths), skipping raw/. */
|
|
29
49
|
function walk(dir: string, base = dir): string[] {
|
|
30
|
-
return readdirSync(dir).flatMap((
|
|
31
|
-
const
|
|
32
|
-
if (statSync(
|
|
33
|
-
return [relative(base,
|
|
50
|
+
return readdirSync(dir).flatMap((entry: string) => {
|
|
51
|
+
const path = join(dir, entry);
|
|
52
|
+
if (statSync(path).isDirectory()) return entry === "raw" ? [] : walk(path, base);
|
|
53
|
+
return [relative(base, path).replaceAll("\\", "/")];
|
|
34
54
|
});
|
|
35
55
|
}
|
|
36
56
|
|
|
57
|
+
/**
|
|
58
|
+
* A markdown link target resolved to a path relative to the bundle root, or undefined when
|
|
59
|
+
* it names nothing in this bundle's files: an external URL, a `mailto:`, or a bare `#anchor`
|
|
60
|
+
* pointing inside the linking document itself.
|
|
61
|
+
*
|
|
62
|
+
* `/from/bundle/root.md` is the form AGENTS.md prefers; anything else is relative to the
|
|
63
|
+
* file doing the linking. A trailing `#section` names a place *within* the target, not a
|
|
64
|
+
* different file, so it is dropped before the path is resolved — without that, every
|
|
65
|
+
* `[text](concept.md#heading)` reads as a link to a file that does not exist.
|
|
66
|
+
*/
|
|
67
|
+
function resolveLink(bundleRoot: string, fromRelative: string, target: string): string | undefined {
|
|
68
|
+
if (/^[a-z][a-z0-9+.-]*:/i.test(target) || target.startsWith("//")) return undefined;
|
|
69
|
+
const path = target.split("#")[0].split("?")[0].trim();
|
|
70
|
+
if (!path) return undefined;
|
|
71
|
+
const resolved = path.startsWith("/")
|
|
72
|
+
? path.slice(1)
|
|
73
|
+
: relative(bundleRoot, join(bundleRoot, dirname(fromRelative), path)).replaceAll("\\", "/");
|
|
74
|
+
return resolved.replace(/\/$/, "");
|
|
75
|
+
}
|
|
76
|
+
|
|
37
77
|
console.log(`khb lint → ${HUB}`);
|
|
38
78
|
detail(`${bundles.length} bundle(s): ${bundles.join(", ") || "none"}`);
|
|
39
79
|
|
|
40
|
-
for (const [
|
|
41
|
-
const dir = join(BUNDLES,
|
|
80
|
+
for (const [bundleIndex, bundle] of bundles.entries()) {
|
|
81
|
+
const dir = join(BUNDLES, bundle);
|
|
42
82
|
// Name the bundle before its findings: an unattributed "ERROR L4" in a fifty-bundle hub
|
|
43
83
|
// sends you grepping, and a clean bundle should still show that it was actually checked.
|
|
44
|
-
section(`[${
|
|
84
|
+
section(`[${bundleIndex + 1}/${bundles.length}] ${bundle}`);
|
|
45
85
|
|
|
46
86
|
// L2 name
|
|
47
|
-
if (!/^[a-z0-9][a-z0-9-]*$/.test(
|
|
87
|
+
if (!/^[a-z0-9][a-z0-9-]*$/.test(bundle)) err("L2", `bad bundle name '${bundle}'`);
|
|
48
88
|
|
|
49
89
|
// L1 required files
|
|
50
|
-
for (const
|
|
51
|
-
if (!existsSync(join(dir,
|
|
90
|
+
for (const required of ["index.md", "refs.md", "sources.yaml"])
|
|
91
|
+
if (!existsSync(join(dir, required))) err("L1", `${bundle}: missing ${required}`);
|
|
52
92
|
|
|
53
93
|
// L3 registered in outer index
|
|
54
|
-
if (!outerIndex.includes(`bundles/${
|
|
94
|
+
if (!outerIndex.includes(`bundles/${bundle}/`))
|
|
95
|
+
err("L3", `${bundle}: not listed in outer.index.md`);
|
|
55
96
|
|
|
56
97
|
const files = existsSync(dir) ? walk(dir) : [];
|
|
57
|
-
const mdFiles = files.filter((
|
|
58
|
-
const concepts = mdFiles.filter((
|
|
59
|
-
const indexes = mdFiles.filter((
|
|
98
|
+
const mdFiles = files.filter((file) => file.endsWith(".md"));
|
|
99
|
+
const concepts = mdFiles.filter((file) => !RESERVED.includes(file.split("/").pop()!));
|
|
100
|
+
const indexes = mdFiles.filter((file) => file.split("/").pop() === "index.md");
|
|
60
101
|
detail(`${concepts.length} concept doc(s), ${indexes.length} index file(s)`);
|
|
61
102
|
|
|
62
103
|
// Collect all index link targets, resolved to bundle-relative paths
|
|
63
104
|
const indexed = new Set<string>();
|
|
64
|
-
for (const
|
|
65
|
-
const
|
|
66
|
-
for (const
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
: relative(dir, join(dir, dirname(idx), l.target)).replaceAll("\\", "/");
|
|
71
|
-
indexed.add(resolved.replace(/\/$/, ""));
|
|
105
|
+
for (const indexFile of indexes) {
|
|
106
|
+
const markdown = stripNonProse(read(join(dir, indexFile)));
|
|
107
|
+
for (const link of mdLinks(markdown)) {
|
|
108
|
+
const resolved = resolveLink(dir, indexFile, link.target);
|
|
109
|
+
if (resolved === undefined) continue;
|
|
110
|
+
indexed.add(resolved);
|
|
72
111
|
// L4b index links resolve (warning — OKF tolerates not-yet-written knowledge)
|
|
73
112
|
if (!existsSync(join(dir, resolved)))
|
|
74
|
-
warn("L4", `${
|
|
113
|
+
warn("L4", `${bundle}: ${indexFile} links to missing ${resolved}`);
|
|
75
114
|
}
|
|
76
115
|
}
|
|
77
116
|
|
|
78
117
|
// L4a every concept is indexed somewhere
|
|
79
|
-
for (const
|
|
80
|
-
if (!indexed.has(
|
|
118
|
+
for (const concept of concepts)
|
|
119
|
+
if (!indexed.has(concept)) err("L4", `${bundle}: ${concept} not listed in any index.md`);
|
|
81
120
|
|
|
82
|
-
for (const
|
|
83
|
-
const body = read(join(dir,
|
|
121
|
+
for (const concept of concepts) {
|
|
122
|
+
const body = read(join(dir, concept));
|
|
84
123
|
|
|
85
124
|
// L9 OKF conformance: frontmatter must parse and carry a usable field set.
|
|
86
125
|
// Frontmatter is the machine-readable half of a concept — routing, filtering and any
|
|
87
126
|
// future index generator read it — so a typo'd key is a silent data loss, not a style nit.
|
|
88
|
-
const
|
|
89
|
-
if (
|
|
127
|
+
const frontmatter = body.match(/^---\n([\s\S]*?)\n---/)?.[1];
|
|
128
|
+
if (frontmatter === undefined)
|
|
129
|
+
err("L9", `${bundle}: ${concept} has no YAML frontmatter (OKF requires it)`);
|
|
90
130
|
else {
|
|
91
131
|
let meta: Record<string, unknown> | undefined;
|
|
92
132
|
try {
|
|
93
|
-
meta = (parseYaml(
|
|
94
|
-
} catch (
|
|
95
|
-
err(
|
|
133
|
+
meta = (parseYaml(frontmatter) ?? {}) as Record<string, unknown>;
|
|
134
|
+
} catch (error) {
|
|
135
|
+
err(
|
|
136
|
+
"L9",
|
|
137
|
+
`${bundle}: ${concept} frontmatter is not valid YAML — ${(error as Error).message.split("\n")[0]}`,
|
|
138
|
+
);
|
|
96
139
|
}
|
|
97
140
|
if (meta) {
|
|
98
|
-
const str = (
|
|
141
|
+
const str = (key: string) =>
|
|
142
|
+
typeof meta![key] === "string" ? (meta![key] as string).trim() : "";
|
|
99
143
|
// type is the one OKF hard requirement; the rest degrade to warnings so an
|
|
100
144
|
// in-progress hub still lints clean while its authors fill things in.
|
|
101
|
-
if (!str("type")) err("L9", `${
|
|
102
|
-
for (const
|
|
103
|
-
if (!str(
|
|
145
|
+
if (!str("type")) err("L9", `${bundle}: ${concept} frontmatter missing required 'type'`);
|
|
146
|
+
for (const key of ["title", "description"])
|
|
147
|
+
if (!str(key)) warn("L9", `${bundle}: ${concept} frontmatter missing '${key}'`);
|
|
104
148
|
if ("tags" in meta && !Array.isArray(meta.tags))
|
|
105
|
-
err("L9", `${
|
|
106
|
-
if (Array.isArray(meta.tags) && meta.tags.some((
|
|
107
|
-
err("L9", `${
|
|
149
|
+
err("L9", `${bundle}: ${concept} 'tags' must be a YAML list, not ${typeof meta.tags}`);
|
|
150
|
+
if (Array.isArray(meta.tags) && meta.tags.some((tag) => typeof tag !== "string"))
|
|
151
|
+
err("L9", `${bundle}: ${concept} 'tags' must contain only strings`);
|
|
108
152
|
if ("timestamp" in meta && !isTimestamp(meta.timestamp))
|
|
109
|
-
warn("L9", `${
|
|
110
|
-
for (const
|
|
111
|
-
if (!OKF_FIELDS.has(
|
|
153
|
+
warn("L9", `${bundle}: ${concept} 'timestamp' is not an ISO-8601 datetime`);
|
|
154
|
+
for (const key of Object.keys(meta))
|
|
155
|
+
if (!OKF_FIELDS.has(key)) warn("L9", `${bundle}: ${concept} unknown frontmatter key '${key}'`);
|
|
112
156
|
}
|
|
113
157
|
}
|
|
114
158
|
|
|
115
159
|
// L6 no cross-bundle links from concept docs
|
|
116
|
-
for (const
|
|
117
|
-
if (/(^|\/)bundles\//.test(
|
|
118
|
-
err("L6", `${
|
|
160
|
+
for (const link of mdLinks(stripNonProse(body))) {
|
|
161
|
+
if (/(^|\/)bundles\//.test(link.target) || link.target.startsWith("../../")) {
|
|
162
|
+
err("L6", `${bundle}: ${concept} links into another bundle (${link.target}) — use refs.md`);
|
|
163
|
+
continue;
|
|
164
|
+
}
|
|
165
|
+
// L11 in-bundle concept links resolve. Concepts link to each other as the bundle's
|
|
166
|
+
// actual structure — the catalog cross-link pass and the query skill's back-links to
|
|
167
|
+
// a synthesis's sources both live in these links, and a synthesis nobody can reach
|
|
168
|
+
// from its sources is a dead end. Only the index side of this was ever checked.
|
|
169
|
+
// A warning, like L4b and for the same reason: a link to a concept somebody intends
|
|
170
|
+
// to write next is not-yet-written knowledge, which OKF tolerates by design.
|
|
171
|
+
const resolved = resolveLink(dir, concept, link.target);
|
|
172
|
+
if (resolved !== undefined && !existsSync(join(dir, resolved)))
|
|
173
|
+
warn("L11", `${bundle}: ${concept} links to missing ${resolved}`);
|
|
119
174
|
}
|
|
120
175
|
}
|
|
121
176
|
|
|
122
177
|
// L7 ref targets exist
|
|
123
178
|
if (existsSync(join(dir, "refs.md"))) {
|
|
124
|
-
for (const
|
|
125
|
-
if (!bundles.includes(
|
|
179
|
+
for (const target of refTargets(read(join(dir, "refs.md"))))
|
|
180
|
+
if (!bundles.includes(target)) err("L7", `${bundle}: refs.md targets missing bundle '${target}'`);
|
|
126
181
|
}
|
|
127
182
|
|
|
128
|
-
//
|
|
183
|
+
// Enumerated once, bundle-relative (`raw/<type>/<file>.md`) — the spelling the ledger
|
|
184
|
+
// stores, so L10 can compare the two sides without renormalizing on every row.
|
|
129
185
|
const rawDir = join(dir, "raw");
|
|
130
|
-
|
|
131
|
-
|
|
186
|
+
const rawFiles = existsSync(rawDir)
|
|
187
|
+
? (readdirSync(rawDir, { recursive: true }) as string[])
|
|
188
|
+
.filter((file) => file.endsWith(".md"))
|
|
189
|
+
.map((file) => `raw/${file.replaceAll("\\", "/")}`)
|
|
190
|
+
: [];
|
|
191
|
+
|
|
192
|
+
// L8 raw provenance (warning)
|
|
193
|
+
if (rawFiles.length) {
|
|
132
194
|
detail(`${rawFiles.length} raw/ file(s) checked for provenance`);
|
|
133
|
-
for (const
|
|
195
|
+
for (const rawFile of rawFiles) {
|
|
134
196
|
try {
|
|
135
|
-
|
|
136
|
-
const
|
|
137
|
-
|
|
138
|
-
|
|
197
|
+
const head = read(join(dir, rawFile));
|
|
198
|
+
const provenance = head.match(/^---\n([\s\S]*?)\n---/)?.[1];
|
|
199
|
+
if (provenance === undefined) {
|
|
200
|
+
warn("L8", `${bundle}: ${rawFile} missing provenance header`);
|
|
201
|
+
continue;
|
|
202
|
+
}
|
|
139
203
|
// `source` is the whole point of the header: it is how a bad extraction gets re-read.
|
|
140
|
-
if (!/^source:\s*\S/m.test(
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
204
|
+
if (!/^source:\s*\S/m.test(provenance))
|
|
205
|
+
warn("L8", `${bundle}: ${rawFile} provenance missing 'source'`);
|
|
206
|
+
const quality = provenance.match(/^quality:\s*(\S+)/m)?.[1];
|
|
207
|
+
if (quality && quality !== "high" && quality !== "low")
|
|
208
|
+
warn("L8", `${bundle}: ${rawFile} quality '${quality}' is not high|low`);
|
|
144
209
|
} catch {}
|
|
145
210
|
}
|
|
146
211
|
}
|
|
212
|
+
|
|
213
|
+
// L10 ledger integrity. log.md is the durable record across both halves of the workflow,
|
|
214
|
+
// and its empty `curated` cells *are* the catalog backlog — but nothing has ever checked
|
|
215
|
+
// that its paths still name anything, so a concept renamed after cataloging leaves a row
|
|
216
|
+
// claiming work that can no longer be found, and neither side notices.
|
|
217
|
+
const ledger = readLedger(dir);
|
|
218
|
+
if (ledger.size) {
|
|
219
|
+
detail(`${ledger.size} log.md row(s) checked`);
|
|
220
|
+
const claimed = new Set<string>();
|
|
221
|
+
for (const row of ledger.values()) {
|
|
222
|
+
if (row.raw) {
|
|
223
|
+
claimed.add(row.raw);
|
|
224
|
+
// raw/ is gitignored and re-derivable, so a hub that was cloned rather than ingested
|
|
225
|
+
// legitimately has every row and no files at all. Only hold a row to its raw file
|
|
226
|
+
// once raw/ has actually been populated; an empty one is that ordinary state.
|
|
227
|
+
if (rawFiles.length && !existsSync(join(dir, row.raw)))
|
|
228
|
+
warn("L10", `${bundle}: log.md row '${row.source}' names missing ${row.raw}`);
|
|
229
|
+
}
|
|
230
|
+
// `declined` is the documented way to close a row without writing a concept
|
|
231
|
+
// (skills/catalog/SKILL.md §5); anything else is a path the row claims to have written.
|
|
232
|
+
// An error, unlike the link rules: there is no not-yet-written case here, since the
|
|
233
|
+
// column is only filled once the concept exists.
|
|
234
|
+
if (row.curated && row.curated !== "declined") {
|
|
235
|
+
const curatedPaths = row.curated.split(",").map((path) => path.trim()).filter(Boolean);
|
|
236
|
+
for (const curated of curatedPaths)
|
|
237
|
+
if (!existsSync(join(dir, curated)))
|
|
238
|
+
err("L10", `${bundle}: log.md row '${row.source}' claims missing concept ${curated}`);
|
|
239
|
+
}
|
|
240
|
+
}
|
|
241
|
+
// An extracted file no row names is invisible work: it is not offered as backlog, so it
|
|
242
|
+
// stays uncurated without ever appearing to be outstanding.
|
|
243
|
+
for (const rawFile of rawFiles)
|
|
244
|
+
if (!claimed.has(rawFile)) warn("L10", `${bundle}: ${rawFile} has no log.md row`);
|
|
245
|
+
} else if (rawFiles.length) {
|
|
246
|
+
warn("L10", `${bundle}: ${rawFiles.length} file(s) in raw/ but log.md records none of them`);
|
|
247
|
+
}
|
|
147
248
|
}
|
|
148
249
|
|
|
149
250
|
// L3 reverse: outer index entries exist
|
|
150
|
-
for (const
|
|
151
|
-
const
|
|
152
|
-
if (
|
|
251
|
+
for (const link of mdLinks(outerIndex)) {
|
|
252
|
+
const match = link.target.match(/^bundles\/([a-z0-9-]+)\//);
|
|
253
|
+
if (match && !bundles.includes(match[1]))
|
|
254
|
+
err("L3", `outer.index.md lists missing bundle '${match[1]}'`);
|
|
153
255
|
}
|
|
154
256
|
|
|
155
257
|
// L5 index prose check (rough): paragraph-length prose in index files
|
|
156
|
-
function proseCheck(name: string,
|
|
157
|
-
for (const block of
|
|
158
|
-
const
|
|
159
|
-
if (!
|
|
160
|
-
if (
|
|
258
|
+
function proseCheck(name: string, markdown: string) {
|
|
259
|
+
for (const block of stripNonProse(markdown).split(/\n\s*\n/)) {
|
|
260
|
+
const text = block.trim();
|
|
261
|
+
if (!text || /^[#|\-*]/.test(text) || text.startsWith("---")) continue;
|
|
262
|
+
if (text.split(/\s+/).length > 30)
|
|
263
|
+
warn("L5", `${name}: paragraph-length prose in an index file`);
|
|
161
264
|
}
|
|
162
265
|
}
|
|
163
266
|
proseCheck("outer.index.md", outerIndex);
|
|
164
|
-
for (const
|
|
165
|
-
if (existsSync(join(BUNDLES,
|
|
267
|
+
for (const bundle of bundles)
|
|
268
|
+
if (existsSync(join(BUNDLES, bundle, "index.md")))
|
|
269
|
+
proseCheck(`${bundle}/index.md`, read(join(BUNDLES, bundle, "index.md")));
|
|
166
270
|
|
|
167
271
|
console.log(`\nlint: ${errors} error(s), ${warnings} warning(s) across ${bundles.length} bundle(s) in ${totalElapsed()}`);
|
|
168
|
-
process.exit(errors ? 1 : 0);
|
package/skills/ingest/SKILL.md
CHANGED
|
@@ -154,7 +154,7 @@ curation, not transcription.
|
|
|
154
154
|
| `.mp3 .wav .m4a .mp4 .mov .mkv` | local `vno` (whisper.cpp), else `whisper` / `faster-whisper` | **low** |
|
|
155
155
|
| `.vtt .srt` | built-in caption reader | high |
|
|
156
156
|
|
|
157
|
-
Extracted text is cached hub-wide by content hash at
|
|
157
|
+
Extracted text is cached hub-wide by content hash at `.ingest-cache/extracted/<sha256>.md`, so the
|
|
158
158
|
same file appearing in two bundles converts once.
|
|
159
159
|
|
|
160
160
|
**A recording next to its captions is one source, not two.** `talk.vtt` (or `talk.en.vtt`,
|
package/skills/lint/SKILL.md
CHANGED
|
@@ -12,7 +12,7 @@ description: Validate KHB structure (routing integrity, bundle shape, OKF confor
|
|
|
12
12
|
log it in `bundles/meta/notes/decisions.md`. No meta bundle means no decision log — do
|
|
13
13
|
not create one to have somewhere to write.
|
|
14
14
|
|
|
15
|
-
## The rules (L1–
|
|
15
|
+
## The rules (L1–L11)
|
|
16
16
|
|
|
17
17
|
Enforced by `khb lint`. Combines KHB routing rules with
|
|
18
18
|
OKF v0.1 conformance (see the OKF spec). Reserved filenames: `index.md`, `log.md`
|
|
@@ -33,7 +33,8 @@ is a **concept document**.
|
|
|
33
33
|
`outer.index.md` exists on disk.
|
|
34
34
|
- L4. Every concept doc is listed in at least one of the bundle's `index.md` files
|
|
35
35
|
(error). Index links pointing at missing files are a warning only — OKF treats
|
|
36
|
-
broken links as not-yet-written knowledge.
|
|
36
|
+
broken links as not-yet-written knowledge. A `#section` suffix names a place inside
|
|
37
|
+
the target and is dropped before the path is checked.
|
|
37
38
|
- L5. Index files contain routing only: headings, bullet/table link lines, one-line
|
|
38
39
|
descriptions. Paragraph-length prose is a violation (warning).
|
|
39
40
|
|
|
@@ -42,6 +43,12 @@ is a **concept document**.
|
|
|
42
43
|
- L6. No markdown link from a concept doc into another bundle's files. Cross-bundle
|
|
43
44
|
pointers live in `refs.md` only.
|
|
44
45
|
- L7. Every target bundle named in `refs.md` exists.
|
|
46
|
+
- L11. Markdown links *within* a bundle resolve to a file that exists (warning). Concept
|
|
47
|
+
links are the bundle's real structure — the catalog cross-link pass and the back-links
|
|
48
|
+
the query skill writes from a synthesis to its sources are both made of them, and a
|
|
49
|
+
synthesis nobody can reach from its sources is a dead end. A warning rather than an
|
|
50
|
+
error, for L4's reason: a link to a concept somebody means to write next is
|
|
51
|
+
not-yet-written knowledge.
|
|
45
52
|
|
|
46
53
|
### Provenance
|
|
47
54
|
|
|
@@ -49,6 +56,21 @@ is a **concept document**.
|
|
|
49
56
|
non-empty `source:`, and `quality:` — if set — reading exactly `high` or `low`.
|
|
50
57
|
`source` is what makes a bad extraction recoverable, so a raw file without one is
|
|
51
58
|
uncatalogable, not merely untidy.
|
|
59
|
+
- L10. `log.md` still describes what is on disk. It is the durable record across ingest
|
|
60
|
+
and catalog, and its empty `curated` cells *are* the catalog backlog, so a row that has
|
|
61
|
+
come loose from its files misreports the work outstanding:
|
|
62
|
+
- a `curated` path names a file that exists (**error**). `declined` is the documented
|
|
63
|
+
way to close a row without a concept and is accepted as-is; anything else is a path
|
|
64
|
+
the row claims to have written, and unlike a link there is no not-yet-written case —
|
|
65
|
+
the column is filled only once the concept exists. Renaming a concept after
|
|
66
|
+
cataloging is what usually breaks it.
|
|
67
|
+
- a row's `raw` path names a file that exists (warning), checked **only** when `raw/`
|
|
68
|
+
has files in it. `raw/` is gitignored and re-derivable, so a hub that was cloned
|
|
69
|
+
rather than ingested has every row and no files at all — that is an ordinary state,
|
|
70
|
+
not a finding.
|
|
71
|
+
- every `.md` under `raw/` has a row (warning). An extracted file no row names is
|
|
72
|
+
invisible work: never offered as backlog, so it stays uncurated without ever looking
|
|
73
|
+
outstanding.
|
|
52
74
|
|
|
53
75
|
### OKF conformance
|
|
54
76
|
|
package/templates/hub/gitignore
CHANGED
|
@@ -4,8 +4,8 @@ bundles/*/raw/
|
|
|
4
4
|
# Agent-specific machine-local permissions; shared skills remain tracked.
|
|
5
5
|
.claude/settings.local.json
|
|
6
6
|
|
|
7
|
-
# Extraction cache (
|
|
8
|
-
/
|
|
7
|
+
# Extraction cache (.ingest-cache/extracted/<sha256>.md) — re-derivable from the sources.
|
|
8
|
+
/.ingest-cache/
|
|
9
9
|
|
|
10
10
|
# Generated. The leading slash matters: a bare `export/` also matches the `skills/export/`
|
|
11
11
|
# these files ship with, and silently drops the export skill from the hub's own history.
|