@msareen/knowledge-hub-builder 0.2.0 → 0.2.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENTS.md CHANGED
@@ -137,6 +137,7 @@ From outside, pass `--hub <dir>` or set `$KHB_HUB`.
137
137
  | Command | Purpose |
138
138
  |---|---|
139
139
  | `khb lint` | validate structure against `skills/lint/SKILL.md` |
140
+ | `khb doctor` | read-only report on the hub: version, location, per-bundle concept/raw/backlog counts, transcriber, and the command that fixes each finding. Writes nothing, and does not duplicate `lint` |
140
141
  | `khb upgrade` | refresh this hub's package-owned contract docs — runs by itself whenever `khb.json`'s stamped version differs from the installed khb, so these docs always match the CLI |
141
142
  | `khb visualize [--port N] [--no-open]` | serve the live bundle graph in your browser; aliases `vis`, `viz` |
142
143
  | `khb new-bundle <name>` | scaffold + register a bundle |
package/README.md CHANGED
@@ -251,26 +251,43 @@ Commands can be run directly or requested through the matching agent skill.
251
251
  | `khb new-bundle <name> ["scope"]` | Create and register a bundle |
252
252
  | `khb ingest [bundle] [--force] [--skip-ocr] [--skip-audio]` | Acquire and extract declared sources |
253
253
  | `khb lint` | Validate routing, bundle structure, and OKF metadata |
254
+ | `khb doctor` | Read-only report: version, location, per-bundle counts, catalog backlog, transcriber |
254
255
  | `khb visualize [--port N] [--no-open]` (aliases: `vis`, `viz`) | Serve the live bundle graph on a random free port and open it in your default browser — pan/zoomable cross-bundle map, drill into a bundle for its folder-clustered concepts, rebuild-on-refresh, exits when you close the tab |
255
256
  | `khb export <bundle> [dest]` | Export one standalone bundle |
256
257
 
257
- Additional ingest flags:
258
+ Run against a hub outside the current directory with `--hub <dir>` or `$KHB_HUB`.
258
259
 
259
- - `--skip-ocr`
260
- - `--skip-audio`
260
+ ### `khb doctor` — what state is this hub in?
261
261
 
262
- Run against a hub outside the current directory with `--hub <dir>` or `$KHB_HUB`.
262
+ A single read-only report. It writes nothing and repairs nothing; each finding names the
263
+ command that does.
264
+
265
+ ```text
266
+ Hub name, description, stamped version vs installed, location, registered
267
+ Bundles per bundle: concepts, raw/ files, log.md rows, catalog backlog, pending
268
+ Extraction which formats are bundled, and whether a transcriber is ready
269
+ Findings what needs attention, each with its fix
270
+ ```
271
+
272
+ Every check existed already, spread across the margins of commands that each knew one of
273
+ them — a move is announced by whatever you happen to run next, the `khb update` hint comes
274
+ out of `khb upgrade`, the uncurated row count out of `khb ingest`, and the transcriber probe
275
+ only ever spoke during a run that needed it. `doctor` asks for the whole picture without
276
+ changing anything to get it.
277
+
278
+ It is not a validator: `khb lint` still owns structure and OKF conformance, and `doctor`
279
+ points at it rather than repeating a rule.
263
280
 
264
281
  ### Moving between hubs
265
282
 
266
- These four run **outside** any hub, from any terminal:
283
+ These run **outside** any hub, from any terminal:
267
284
 
268
285
  | Command | Purpose |
269
286
  |---|---|
270
287
  | `khb` | Open a hub. One registered hub asks; several show a list and take a pick; none walks you through creating the first |
271
288
  | `khb list [--json]` | Every hub on this machine, with its description and path |
272
- | `khb go <name\|N> [--path]` | Open one by name or list position. `--path` prints just the path, for `cd "$(khb go --path work)"` |
273
- | `khb agent [name] [--command X] [--args "…"]` | Which agent `khb go` launches — `claude`, `codex`, anything on your PATH, or `none` to just print the path |
289
+ | `khb go [name\|N] [--path] [--no-agent] [--agent X]` | Open one by name or list position. `--path` prints just the path, for `cd "$(khb go --path work)"`; `--no-agent` prints the path and the `cd` line without launching anything; `--agent X` launches `X` for this run only, leaving the configured default alone |
290
+ | `khb agent [name\|none] [--command X] [--args "…"]` | Which agent `khb go` launches — `claude`, `codex`, anything on your PATH, or `none` to just print the path |
274
291
  | `khb update [new-path] [--path\|-p] [--schema\|-s] [--from <old>] [--dry-run]` | Repair the hub: path references after a move, and/or backfill `sources.yaml` to the current schema. No flag runs both |
275
292
  | `khb forget <name>` | Drop a hub from the list. The folder is untouched |
276
293
 
@@ -278,7 +295,8 @@ These four run **outside** any hub, from any terminal:
278
295
  cold terminal ends with an agent open on the right folder. No program can change its
279
296
  parent shell's directory, which is why the `cd` line is printed rather than performed.
280
297
 
281
- The list lives in `~/.khb/hubs-config.json` (`%USERPROFILE%\.khb` on Windows). It holds
298
+ The list lives in `~/.khb/hubs-config.json` (`%USERPROFILE%\.khb` on Windows; `$KHB_HOME`
299
+ overrides the directory, for a portable install or a test run). It holds
282
300
  paths and one launch command — no knowledge — and fills itself in: any khb command run
283
301
  inside a hub registers it, so hubs you already had show up without a migration step.
284
302
  Delete the file and the next command in each hub puts it back.
@@ -324,7 +342,7 @@ the same spelling of the new path; matches must end at a path boundary, so movin
324
342
  never touches a sibling `…/older`; and the new path is matched too and rewritten to itself,
325
343
  which is what makes overlapping moves safe — lifting a hub out of its parent, or pushing it
326
344
  down into a subdirectory of where it stood — and makes a second run a no-op.
327
- `.git/`, `node_modules/` and the `inbox/` cache are skipped, as are binaries.
345
+ `.git/`, `node_modules/` and the `.ingest-cache/` cache are skipped, as are binaries.
328
346
 
329
347
  **`--schema`/`-s`** — a bundle's `sources.yaml` can predate a field khb's since learned about
330
348
  (e.g. `exclude:`), with no way to discover it short of reading the docs. This backfills
@@ -373,7 +391,7 @@ everything under `bundles/`.
373
391
 
374
392
  ## Privacy
375
393
 
376
- `raw/` and the extraction cache under `inbox/` are gitignored. `log.md` is committed and
394
+ `raw/` and the extraction cache under `.ingest-cache/` are gitignored. `log.md` is committed and
377
395
  records source paths, which may be absolute. Ignore `log.md` before the first commit if
378
396
  those paths are sensitive.
379
397
 
package/SPEC.md CHANGED
@@ -72,7 +72,7 @@ my-knowledge/ # ~/OneDrive/my-knowledge, a private repo, a shar
72
72
  │ │ ├── index.md # (each may carry its own index)
73
73
  │ │ └── <concept>.md
74
74
  │ └── raw/ # ingested/extracted material, pre-curation (gitignored)
75
- ├── inbox/extracted/ # hub-wide extraction cache, keyed by content hash (gitignored)
75
+ ├── .ingest-cache/extracted/ # hub-wide extraction cache, keyed by content hash (gitignored)
76
76
 
77
77
  │ ── below: package-owned copies, refreshed by `khb upgrade`, never hand-edited ──
78
78
  ├── AGENTS.md # common contract; Codex discovers this directly
@@ -121,14 +121,21 @@ acting on them** — an agent can never read a protocol the CLI no longer implem
121
121
  │ ├── new-bundle.ts # scaffold from .bundle_template, register in outer.index.md
122
122
  │ ├── export.ts # bundle + common patterns → standalone shareable folder
123
123
  │ ├── lint.ts # enforce skills/lint/SKILL.md across the hub
124
+ │ ├── doctor.ts # read-only state report; writes nothing, repairs nothing
124
125
  │ ├── visualize.ts # serve the live bundle graph from indexes + refs
125
126
  │ ├── ingest/ # folder.ts / files.ts / web.ts → acquire.ts → bundle/raw
126
127
  │ └── lib/
128
+ │ ├── args.ts # argv helpers; what they consume they remove, leaving positionals
129
+ │ ├── create.ts # making a hub, shared by `khb init` and the first-run wizard
127
130
  │ ├── extract.ts # every local extractor + the content-hash cache
131
+ │ ├── graph.ts # graph data for the visualizer — read-only, never writes
132
+ │ ├── graph-page.ts # the visualizer's browser UI, rendered from that data
128
133
  │ ├── ledger.ts # log.md read/write
134
+ │ ├── log.ts # progress reporting: each unit announces itself before it runs
129
135
  │ ├── paths.ts # package-side paths — importing it never needs a hub
130
136
  │ ├── registry.ts # ~/.khb/hubs-config.json: where this machine's hubs are
131
137
  │ ├── relocate.ts # khb update --path's path rewriter — pure text, no judgement
138
+ │ ├── scaffold.ts # bundle creation + lookup, shared by new-bundle and ingest
132
139
  │ ├── schema.ts # khb update --schema: sources.yaml field diff/apply
133
140
  │ ├── upgrade.ts # the refresh itself: `khb upgrade` and the drift check
134
141
  │ └── util.ts # hub resolution + shared helpers
@@ -272,7 +279,7 @@ unattended:
272
279
  changes nothing. Only old and new naming *the same directory* is refused, there being no
273
280
  move to repair.
274
281
 
275
- `.git/`, `node_modules/` and the `inbox/` extraction cache are not walked; binary files and
282
+ `.git/`, `node_modules/` and the `.ingest-cache/` extraction cache are not walked; binary files and
276
283
  anything over 8 MB are skipped. `--dry-run` reports the file-by-file hit count and writes
277
284
  nothing.
278
285
 
@@ -483,7 +490,7 @@ raw files, with one hard rule — subagents write concept docs, the orchestrator
483
490
 
484
491
  Extraction is deterministic, so `khb` owns all of it. Common formats use bundled pure-JS
485
492
  libraries with no system install; results are cached hub-wide by content hash
486
- (`inbox/extracted/<sha256>.md`) and reused across bundles. Nothing here contacts a model:
493
+ (`.ingest-cache/extracted/<sha256>.md`) and reused across bundles. Nothing here contacts a model:
487
494
  tesseract and whisper are local binaries, expensive in CPU but reproducible, which is what
488
495
  puts them on the CLI side of the §Division-of-labor line.
489
496
 
@@ -526,6 +533,10 @@ drops, and rewrites the raw file with `extract_tool: claude-vision`.
526
533
  - every concept doc is listed in an index and carries OKF frontmatter (`type` required)
527
534
  - `refs.md` targets exist; no cross-bundle inline links from concept docs
528
535
  - index files contain links only (routing, not content)
536
+ - intra-bundle links from a concept doc resolve to a file that exists (warning — OKF
537
+ tolerates a link to knowledge not yet written)
538
+ - `log.md` agrees with the bundle: a `curated` value naming a missing concept is an error,
539
+ a `raw` value naming a missing file or a `raw/` file with no row is a warning
529
540
 
530
541
  ## 8. Visualizer
531
542
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@msareen/knowledge-hub-builder",
3
- "version": "0.2.0",
3
+ "version": "0.2.2",
4
4
  "description": "Knowledge Hub Builder — a bundle-of-bundles knowledge base you build with an agent",
5
5
  "keywords": ["knowledge-base", "okf", "agent", "wiki", "bundles", "claude", "codex", "rag"],
6
6
  "license": "MIT",
@@ -35,6 +35,8 @@
35
35
  "scripts": {
36
36
  "khb": "bun scripts/cli.ts",
37
37
  "lint": "bun scripts/cli.ts lint",
38
+ "upgrade": "bun scripts/cli.ts upgrade",
39
+ "doctor": "bun scripts/cli.ts doctor",
38
40
  "visualize": "bun scripts/cli.ts visualize",
39
41
  "new-bundle": "bun scripts/cli.ts new-bundle",
40
42
  "export": "bun scripts/cli.ts export",
package/scripts/cli.ts CHANGED
@@ -13,6 +13,7 @@ const COMMANDS: Record<string, { load: () => Promise<unknown>; usage: string; de
13
13
  desc: "acquire + extract declared sources → raw/",
14
14
  },
15
15
  lint: { load: () => import("./lint"), usage: "khb lint", desc: "validate the hub against skills/lint/SKILL.md" },
16
+ doctor: { load: () => import("./doctor"), usage: "khb doctor", desc: "read-only report on this hub's state" },
16
17
  visualize: {
17
18
  load: () => import("./visualize"),
18
19
  usage: "khb visualize [--port N] [--no-open]",
@@ -0,0 +1,177 @@
1
+ // khb doctor — one read-only report on the state of a hub.
2
+ //
3
+ // Every check here already existed, scattered across the preambles of commands that each
4
+ // knew one of them: cli.ts announces a move and a version drift, `khb upgrade` prints the
5
+ // `khb update` hint, `khb ingest` counts the uncurated rows on its way out, and the
6
+ // transcriber probe only ever spoke during a run that needed it. So the answer to "what
7
+ // state is this hub in?" was: run several commands that change things and read their
8
+ // margins. This command asks nothing of the hub but to look at it.
9
+ //
10
+ // It writes nothing. That is the point, and it is also the boundary: `doctor` reports and
11
+ // names the command that repairs, but never repairs. `khb lint` stays the structural
12
+ // validator — doctor counts and points at it rather than duplicating a rule.
13
+ import { HUB, BUNDLES, listBundles, read, join, existsSync } from "./lib/util";
14
+ import { readLedger } from "./lib/ledger";
15
+ import { staleLocations, hubVersion } from "./lib/upgrade";
16
+ import { diffSourcesYamlAll } from "./lib/schema";
17
+ import { transcriberStatus } from "./lib/extract";
18
+ import { version, MARKER, markerIn } from "./lib/paths";
19
+ import { listHubs, canonical } from "./lib/registry";
20
+ import { section, detail, totalElapsed } from "./lib/log";
21
+ import { rejectUnknownFlags } from "./lib/args";
22
+ import { readdirSync, statSync } from "node:fs";
23
+ import { relative } from "node:path";
24
+
25
+ rejectUnknownFlags(process.argv.slice(2), "khb doctor");
26
+
27
+ /** Findings are advisory: doctor's exit code reports whether it ran, not what it found. */
28
+ const findings: string[] = [];
29
+ const flag = (msg: string, fix?: string) => findings.push(fix ? `${msg}\n fix: ${fix}` : msg);
30
+
31
+ console.log(`khb doctor → ${HUB}`);
32
+
33
+ // ---- Hub identity -----------------------------------------------------------------------
34
+ const marker = (() => {
35
+ try {
36
+ return JSON.parse(read(join(HUB, markerIn(HUB) ?? MARKER))) as Record<string, unknown>;
37
+ } catch {
38
+ return {} as Record<string, unknown>;
39
+ }
40
+ })();
41
+
42
+ section("Hub");
43
+ detail(`name ${(marker.name as string) || "(unset — khb.json 'name')"}`);
44
+ detail(`description ${(marker.description as string) || "(unset — khb.json 'description')"}`);
45
+
46
+ // Drift is normally self-healing: cli.ts refreshes a hub before any in-hub command, this one
47
+ // included, so a mismatch here means the refresh was suppressed rather than that it is due.
48
+ const stamped = hubVersion(HUB);
49
+ const installed = version();
50
+ detail(
51
+ `khb version ${stamped ?? "unstamped"}` +
52
+ (stamped === installed ? ` (matches installed)` : ` — installed is ${installed}`),
53
+ );
54
+ if (stamped !== installed)
55
+ flag(
56
+ `hub is stamped ${stamped ?? "unstamped"} but khb is ${installed}; its contract docs may be a version behind.`,
57
+ process.env.KHB_NO_AUTO_UPGRADE ? "unset KHB_NO_AUTO_UPGRADE, or run: khb upgrade" : "khb upgrade",
58
+ );
59
+
60
+ // ---- Location and registry --------------------------------------------------------------
61
+ // A hub moved more than once before anyone repaired it carries every former home, and
62
+ // `khb update --path` rewrites them all in one pass — so say how many there are rather than
63
+ // showing the most recent and implying it is the only one.
64
+ const stale = staleLocations(HUB);
65
+ detail(
66
+ `location ${
67
+ stale.length
68
+ ? `moved from ${stale[stale.length - 1]}${stale.length > 1 ? ` (+${stale.length - 1} earlier)` : ""}`
69
+ : "matches the marker"
70
+ }`,
71
+ );
72
+ if (stale.length)
73
+ flag(
74
+ `this hub has moved; absolute paths recorded inside it still name ${stale.length > 1 ? "former locations" : "its former location"}.`,
75
+ "khb update --path (khb update --path --dry-run to preview)",
76
+ );
77
+
78
+ const registered = listHubs().some((entry) => canonical(entry.path) === canonical(HUB));
79
+ detail(`registered ${registered ? "yes" : "no — 'khb list' and 'khb go' will not offer it"}`);
80
+
81
+ // ---- sources.yaml schema ----------------------------------------------------------------
82
+ const schemaDiffs = diffSourcesYamlAll(HUB);
83
+ if (schemaDiffs.length) {
84
+ const fields = schemaDiffs.reduce((total, diff) => total + diff.changes.length, 0);
85
+ flag(
86
+ `${fields} sources.yaml field(s) across ${schemaDiffs.length} bundle(s) predate the current schema.`,
87
+ "khb update --schema (khb update --schema --dry-run to preview)",
88
+ );
89
+ }
90
+
91
+ // ---- Bundles ----------------------------------------------------------------------------
92
+ const RESERVED = ["index.md", "log.md", "refs.md"];
93
+
94
+ /** Concept docs: every .md in the bundle outside raw/ that is not a reserved filename. */
95
+ function conceptCount(dir: string): number {
96
+ const walk = (current: string): string[] =>
97
+ readdirSync(current).flatMap((entry: string) => {
98
+ const path = join(current, entry);
99
+ if (statSync(path).isDirectory()) return entry === "raw" ? [] : walk(path);
100
+ return [relative(dir, path).replaceAll("\\", "/")];
101
+ });
102
+ return walk(dir).filter(
103
+ (file) => file.endsWith(".md") && !RESERVED.includes(file.split("/").pop()!),
104
+ ).length;
105
+ }
106
+
107
+ const bundles = listBundles();
108
+ section(`Bundles (${bundles.length})`);
109
+
110
+ if (!bundles.length) {
111
+ detail("none yet — khb new-bundle <name> \"scope\"");
112
+ } else {
113
+ const summaries = bundles.map((bundle) => {
114
+ const dir = join(BUNDLES, bundle);
115
+ const ledger = readLedger(dir);
116
+ const rawDir = join(dir, "raw");
117
+ const rawFiles = existsSync(rawDir)
118
+ ? (readdirSync(rawDir, { recursive: true }) as string[]).filter((file) => file.endsWith(".md"))
119
+ .length
120
+ : 0;
121
+ const rows = [...ledger.values()];
122
+ return {
123
+ bundle,
124
+ concepts: conceptCount(dir),
125
+ rawFiles,
126
+ rows: rows.length,
127
+ // The catalog backlog in the ledger's own terms — "in raw/ but not yet distilled into
128
+ // a concept doc" — so a row must have a raw file to be part of it. A row with neither
129
+ // is *pending*, a different state with a different fix, and counting it in both would
130
+ // overstate the work cataloging can actually pick up.
131
+ backlog: rows.filter((row) => row.raw && !row.curated).length,
132
+ pending: rows.filter((row) => !row.raw).length,
133
+ };
134
+ });
135
+
136
+ const nameWidth = Math.max(6, ...summaries.map((summary) => summary.bundle.length));
137
+ detail(`${"bundle".padEnd(nameWidth)} concepts raw/ rows backlog pending`);
138
+ for (const summary of summaries)
139
+ detail(
140
+ `${summary.bundle.padEnd(nameWidth)} ${String(summary.concepts).padStart(8)} ` +
141
+ `${String(summary.rawFiles).padStart(4)} ${String(summary.rows).padStart(4)} ` +
142
+ `${String(summary.backlog).padStart(7)} ${String(summary.pending).padStart(7)}`,
143
+ );
144
+
145
+ const backlog = summaries.reduce((total, summary) => total + summary.backlog, 0);
146
+ const pending = summaries.reduce((total, summary) => total + summary.pending, 0);
147
+ if (backlog)
148
+ flag(
149
+ `${backlog} row(s) in raw/ but not yet cataloged, across ` +
150
+ `${summaries.filter((summary) => summary.backlog).length} bundle(s).`,
151
+ "ask an agent to catalog the bundle (skills/catalog/SKILL.md)",
152
+ );
153
+ // An empty `raw` is a source khb saw and could not convert — a missing extractor, a
154
+ // protected file, or a --skip flag. It is not a failed run, but it is work still owed.
155
+ if (pending)
156
+ flag(
157
+ `${pending} source(s) acquired but not extracted (empty 'raw' in log.md).`,
158
+ "khb ingest <bundle> — after installing whatever the row's reason names",
159
+ );
160
+ }
161
+
162
+ // ---- Extraction -------------------------------------------------------------------------
163
+ section("Extraction");
164
+ detail("bundled text, PDF, DOCX, ODT, XLSX, PPTX, OCR (images + scanned PDFs), captions");
165
+ const transcriber = await transcriberStatus();
166
+ detail(`transcriber ${transcriber.detail}`);
167
+ if (!transcriber.ready)
168
+ flag(`no transcriber is ready, so audio and video will pend.`, transcriber.fix);
169
+
170
+ // ---- Findings ---------------------------------------------------------------------------
171
+ section(findings.length ? `Findings (${findings.length})` : "Findings");
172
+ if (!findings.length) detail("none — nothing here needs attention.");
173
+ else for (const finding of findings) detail(`- ${finding}`);
174
+
175
+ section("Next");
176
+ detail("khb lint structural and OKF validation (doctor does not duplicate it)");
177
+ console.log(`\ndoctor: ${findings.length} finding(s) across ${bundles.length} bundle(s) in ${totalElapsed()}`);
package/scripts/hubs.ts CHANGED
@@ -461,7 +461,7 @@ function repairPaths(newPath: string, fromOpt: string | undefined, dryRun: boole
461
461
  detail(`scanning ${newPath} …`);
462
462
  const { scanned, hits, failed } = rewritePaths(newPath, froms, newPath, {
463
463
  dryRun,
464
- onStart: (n) => detail(`${n} file(s) to check (skipping .git/, node_modules/, inbox/)`),
464
+ onStart: (n) => detail(`${n} file(s) to check (skipping .git/, node_modules/, .ingest-cache/)`),
465
465
  });
466
466
 
467
467
  const total = hits.reduce((n, h) => n + h.count, 0);
@@ -223,7 +223,7 @@ export function report(c: Counters) {
223
223
  line(c.moved, "moved/renamed — existing raw file and catalog entry kept");
224
224
  line(c.copied, "text file(s) copied");
225
225
  line(c.extracted, "extracted");
226
- line(c.fromCache, "reused from the extraction cache (inbox/extracted/)");
226
+ line(c.fromCache, "reused from the extraction cache (.ingest-cache/extracted/)");
227
227
  line(c.ocrd, "read by OCR");
228
228
  line(c.transcribed, "transcribed");
229
229
  line(c.captioned, "read from a caption sidecar (no transcription needed)");
package/scripts/init.ts CHANGED
@@ -50,7 +50,14 @@ if (upgrading) {
50
50
 
51
51
  const { from, to, synced, pruned, renamed } = upgradeHub(HUB);
52
52
  console.log(`Upgraded ${HUB}: ${from ?? "?"} -> ${to}`);
53
- console.log(` refreshed: ${synced.join(", ")}`);
53
+ // An empty list is not an empty result: in the khb development repo the package *is* the
54
+ // hub, so every managed path is its own source and there is genuinely nothing to copy.
55
+ // Printing a bare "refreshed:" there reads as a failure rather than as the no-op it is.
56
+ console.log(
57
+ synced.length
58
+ ? ` refreshed: ${synced.join(", ")}`
59
+ : ` refreshed: nothing to copy — this hub is its own package`,
60
+ );
54
61
  if (renamed) console.log(` renamed: ${renamed} -> ${MARKER}`);
55
62
  if (pruned.length) console.log(` removed (no longer part of the contract): ${pruned.join(", ")}`);
56
63
  console.log(`Your bundles/ and outer.index.md were not touched. Next: khb lint`);
@@ -1,4 +1,4 @@
1
- // Content-addressed extraction cache: inbox/extracted/<sha256>.md
1
+ // Content-addressed extraction cache: .ingest-cache/extracted/<sha256>.md
2
2
  //
3
3
  // Binary formats have to be converted before anything can read them. That conversion is
4
4
  // expensive and deterministic per content hash, so it is cached hub-wide rather than
@@ -21,10 +21,10 @@
21
21
  // and named in the provenance header, so curation can escalate to a vision read of the
22
22
  // source instead of trusting garbled text.
23
23
  import { mkdirSync, writeFileSync, readFileSync, existsSync, rmSync, readdirSync } from "node:fs";
24
- import { INBOX, join, basename } from "./util";
24
+ import { INGEST_CACHE, join, basename } from "./util";
25
25
  import { note } from "./log";
26
26
 
27
- export const EXTRACTED = join(INBOX, "extracted");
27
+ export const EXTRACTED = join(INGEST_CACHE, "extracted");
28
28
 
29
29
  /** Below this many characters per page, a PDF is a picture of a document, not a document. */
30
30
  const SCANNED_CHARS_PER_PAGE = 20;
@@ -611,7 +611,7 @@ export async function transcribeCached(path: string, hash: string, model = "base
611
611
 
612
612
  // Both engines write files rather than to stdout; give each run a scratch directory of
613
613
  // its own so a stray sibling transcript is never mistaken for this one's.
614
- const out = join(INBOX, "tmp", hash.slice(0, 12));
614
+ const out = join(INGEST_CACHE, "tmp", hash.slice(0, 12));
615
615
  mkdirSync(out, { recursive: true });
616
616
  try {
617
617
  return eng.kind === "vno"
@@ -623,3 +623,35 @@ export async function transcribeCached(path: string, hash: string, model = "base
623
623
  rmSync(out, { recursive: true, force: true });
624
624
  }
625
625
  }
626
+
627
+ /**
628
+ * The transcriber situation as a fact to report, rather than as a step in a run.
629
+ *
630
+ * `asrEngine()` above is the ingest path: it caches its answer for the process and speaks
631
+ * up on stderr, both of which are right in the middle of a run and wrong for `khb doctor`,
632
+ * which prints its own report and must not warn on the side. Same probe order — vno, then
633
+ * whisper, then faster-whisper — so the two never disagree about what would actually run.
634
+ */
635
+ export async function transcriberStatus(): Promise<{ ready: boolean; detail: string; fix?: string }> {
636
+ const vno = await vnoStatus();
637
+ if (vno.state === "ready") return { ready: true, detail: "vno (whisper.cpp)" };
638
+
639
+ const whisper =
640
+ (await runCli(["whisper", "--help"])) ? "whisper"
641
+ : (await runCli(["faster-whisper", "--help"])) ? "faster-whisper"
642
+ : "";
643
+
644
+ if (vno.state === "unset-up") {
645
+ const missing = vno.blockers.length ? `: ${vno.blockers.join(", ")}` : "";
646
+ // Amber, exactly as in a run: an unset-up vno with whisper behind it is not a problem.
647
+ return whisper
648
+ ? { ready: true, detail: `${whisper} — vno installed but not set up${missing}`, fix: "vno setup" }
649
+ : { ready: false, detail: `vno installed but not set up${missing}`, fix: "vno setup" };
650
+ }
651
+ if (whisper) return { ready: true, detail: whisper };
652
+ return {
653
+ ready: false,
654
+ detail: "none on PATH — audio and video pend as rows, everything else ingests",
655
+ fix: "npm install -g @msareen/voice-notes-organizer (or: pip install -U openai-whisper)",
656
+ };
657
+ }
@@ -10,7 +10,7 @@ import { join, relative } from "node:path";
10
10
  import { ticker } from "./log";
11
11
 
12
12
  /** Directories never walked: VCS internals, installed packages, regenerable caches. */
13
- const SKIP_DIRS = new Set([".git", "node_modules", "inbox"]);
13
+ const SKIP_DIRS = new Set([".git", "node_modules", ".ingest-cache"]);
14
14
 
15
15
  /** A file bigger than this is a corpus artefact, not something holding a path reference. */
16
16
  const MAX_BYTES = 8 * 1024 * 1024;
@@ -32,16 +32,28 @@ export function hubVersion(hub: string): string | undefined {
32
32
  }
33
33
  }
34
34
 
35
- /** Copy every package-owned contract file into the hub, replacing what is there. */
35
+ /**
36
+ * Copy every package-owned contract file into the hub, replacing what is there.
37
+ *
38
+ * A file is skipped when source and destination are the same path. That is not a corner
39
+ * case: the khb development repo is its own hub (see the `note` in this repo's `khb.json`),
40
+ * so `PKG` and `hub` are one directory there and every managed path resolves to itself.
41
+ * `cpSync` rejects that outright with EINVAL, which turned any version drift in the dev
42
+ * repo into a crash on *every* in-hub command — the working copy is already the source of
43
+ * truth, so the honest answer is that there is nothing to copy.
44
+ */
36
45
  export function syncManaged(hub: string): string[] {
37
46
  const done: string[] = [];
38
- for (const f of MANAGED) {
39
- const src = join(PKG, f);
47
+ const same = (left: string, right: string) =>
48
+ process.platform === "win32" ? left.toLowerCase() === right.toLowerCase() : left === right;
49
+ for (const managed of MANAGED) {
50
+ const src = join(PKG, managed);
40
51
  if (!existsSync(src)) continue;
41
- const dest = join(hub, f);
52
+ const dest = join(hub, managed);
53
+ if (same(resolve(src), resolve(dest))) continue;
42
54
  mkdirSync(dirname(dest), { recursive: true });
43
55
  cpSync(src, dest, { recursive: true, force: true });
44
- done.push(statSync(src).isDirectory() ? `${f}/` : f);
56
+ done.push(statSync(src).isDirectory() ? `${managed}/` : managed);
45
57
  }
46
58
  return done;
47
59
  }
@@ -25,7 +25,7 @@ function resolveHub(): string {
25
25
 
26
26
  export const HUB = resolveHub();
27
27
  export const BUNDLES = join(HUB, "bundles");
28
- export const INBOX = join(HUB, "inbox");
28
+ export const INGEST_CACHE = join(HUB, ".ingest-cache");
29
29
  export { TEMPLATE, markerIn } from "./paths";
30
30
 
31
31
  export function listBundles(): string[] {
package/scripts/lint.ts CHANGED
@@ -1,5 +1,6 @@
1
1
  // khb lint — enforce skills/lint/SKILL.md (structural rules + OKF v0.1 conformance) across the hub
2
2
  import { HUB, BUNDLES, listBundles, read, mdLinks, refTargets, join, existsSync } from "./lib/util";
3
+ import { readLedger } from "./lib/ledger";
3
4
  import { detail, section, totalElapsed } from "./lib/log";
4
5
  import { readdirSync, statSync } from "node:fs";
5
6
  import { dirname, relative } from "node:path";
@@ -13,156 +14,258 @@ rejectUnknownFlags(process.argv.slice(2), "khb lint");
13
14
  const OKF_FIELDS = new Set(["type", "title", "description", "resource", "tags", "timestamp"]);
14
15
 
15
16
  /** Accepts a YAML-parsed Date (unquoted) or an ISO-8601 string (quoted). */
16
- const isTimestamp = (v: unknown) =>
17
- v instanceof Date ? !isNaN(v.getTime()) : typeof v === "string" && !isNaN(Date.parse(v));
17
+ const isTimestamp = (value: unknown) =>
18
+ value instanceof Date
19
+ ? !isNaN(value.getTime())
20
+ : typeof value === "string" && !isNaN(Date.parse(value));
18
21
 
19
22
  let errors = 0, warnings = 0;
20
23
  const err = (rule: string, msg: string) => { errors++; console.error(`ERROR ${rule}: ${msg}`); };
21
24
  const warn = (rule: string, msg: string) => { warnings++; console.warn(`warn ${rule}: ${msg}`); };
22
25
 
23
- const stripComments = (md: string) => md.replace(/<!--[\s\S]*?-->/g, "");
26
+ /**
27
+ * Drop everything that is markup *about* markdown rather than markdown: HTML comments, and
28
+ * every code span or fenced block.
29
+ *
30
+ * Code has to go before any link is extracted. A doc explaining the index form writes
31
+ * `` `* [Title](path.md) - description` `` as an example, and a link rule that cannot tell
32
+ * an example from a link reports it as a dead one — which in a project whose concept docs
33
+ * document its own conventions is a false positive on exactly the docs most worth writing.
34
+ * The same reasoning covers L6: a code sample *showing* a forbidden cross-bundle link is
35
+ * teaching the rule, not breaking it.
36
+ *
37
+ * One pattern handles spans and fences alike: a run of N backticks closes on the next run
38
+ * of exactly N, so ``` fences and the `` `…` `` form that quotes inner backticks both pair
39
+ * correctly. An unbalanced backtick simply fails to match and leaves the text alone.
40
+ */
41
+ const stripNonProse = (markdown: string) =>
42
+ markdown.replace(/<!--[\s\S]*?-->/g, "").replace(/(`+)[\s\S]*?\1/g, "");
43
+
24
44
  const RESERVED = ["index.md", "log.md", "refs.md"]; // refs.md is KHB-reserved
25
45
  const bundles = listBundles();
26
46
  const outerIndex = read(join(HUB, "outer.index.md"));
27
47
 
28
48
  /** All files under dir (relative paths), skipping raw/. */
29
49
  function walk(dir: string, base = dir): string[] {
30
- return readdirSync(dir).flatMap((f) => {
31
- const p = join(dir, f);
32
- if (statSync(p).isDirectory()) return f === "raw" ? [] : walk(p, base);
33
- return [relative(base, p).replaceAll("\\", "/")];
50
+ return readdirSync(dir).flatMap((entry: string) => {
51
+ const path = join(dir, entry);
52
+ if (statSync(path).isDirectory()) return entry === "raw" ? [] : walk(path, base);
53
+ return [relative(base, path).replaceAll("\\", "/")];
34
54
  });
35
55
  }
36
56
 
57
+ /**
58
+ * A markdown link target resolved to a path relative to the bundle root, or undefined when
59
+ * it names nothing in this bundle's files: an external URL, a `mailto:`, or a bare `#anchor`
60
+ * pointing inside the linking document itself.
61
+ *
62
+ * `/from/bundle/root.md` is the form AGENTS.md prefers; anything else is relative to the
63
+ * file doing the linking. A trailing `#section` names a place *within* the target, not a
64
+ * different file, so it is dropped before the path is resolved — without that, every
65
+ * `[text](concept.md#heading)` reads as a link to a file that does not exist.
66
+ */
67
+ function resolveLink(bundleRoot: string, fromRelative: string, target: string): string | undefined {
68
+ if (/^[a-z][a-z0-9+.-]*:/i.test(target) || target.startsWith("//")) return undefined;
69
+ const path = target.split("#")[0].split("?")[0].trim();
70
+ if (!path) return undefined;
71
+ const resolved = path.startsWith("/")
72
+ ? path.slice(1)
73
+ : relative(bundleRoot, join(bundleRoot, dirname(fromRelative), path)).replaceAll("\\", "/");
74
+ return resolved.replace(/\/$/, "");
75
+ }
76
+
37
77
  console.log(`khb lint → ${HUB}`);
38
78
  detail(`${bundles.length} bundle(s): ${bundles.join(", ") || "none"}`);
39
79
 
40
- for (const [bi, b] of bundles.entries()) {
41
- const dir = join(BUNDLES, b);
80
+ for (const [bundleIndex, bundle] of bundles.entries()) {
81
+ const dir = join(BUNDLES, bundle);
42
82
  // Name the bundle before its findings: an unattributed "ERROR L4" in a fifty-bundle hub
43
83
  // sends you grepping, and a clean bundle should still show that it was actually checked.
44
- section(`[${bi + 1}/${bundles.length}] ${b}`);
84
+ section(`[${bundleIndex + 1}/${bundles.length}] ${bundle}`);
45
85
 
46
86
  // L2 name
47
- if (!/^[a-z0-9][a-z0-9-]*$/.test(b)) err("L2", `bad bundle name '${b}'`);
87
+ if (!/^[a-z0-9][a-z0-9-]*$/.test(bundle)) err("L2", `bad bundle name '${bundle}'`);
48
88
 
49
89
  // L1 required files
50
- for (const f of ["index.md", "refs.md", "sources.yaml"])
51
- if (!existsSync(join(dir, f))) err("L1", `${b}: missing ${f}`);
90
+ for (const required of ["index.md", "refs.md", "sources.yaml"])
91
+ if (!existsSync(join(dir, required))) err("L1", `${bundle}: missing ${required}`);
52
92
 
53
93
  // L3 registered in outer index
54
- if (!outerIndex.includes(`bundles/${b}/`)) err("L3", `${b}: not listed in outer.index.md`);
94
+ if (!outerIndex.includes(`bundles/${bundle}/`))
95
+ err("L3", `${bundle}: not listed in outer.index.md`);
55
96
 
56
97
  const files = existsSync(dir) ? walk(dir) : [];
57
- const mdFiles = files.filter((f) => f.endsWith(".md"));
58
- const concepts = mdFiles.filter((f) => !RESERVED.includes(f.split("/").pop()!));
59
- const indexes = mdFiles.filter((f) => f.split("/").pop() === "index.md");
98
+ const mdFiles = files.filter((file) => file.endsWith(".md"));
99
+ const concepts = mdFiles.filter((file) => !RESERVED.includes(file.split("/").pop()!));
100
+ const indexes = mdFiles.filter((file) => file.split("/").pop() === "index.md");
60
101
  detail(`${concepts.length} concept doc(s), ${indexes.length} index file(s)`);
61
102
 
62
103
  // Collect all index link targets, resolved to bundle-relative paths
63
104
  const indexed = new Set<string>();
64
- for (const idx of indexes) {
65
- const md = stripComments(read(join(dir, idx)));
66
- for (const l of mdLinks(md)) {
67
- if (l.target.startsWith("http")) continue;
68
- const resolved = l.target.startsWith("/")
69
- ? l.target.slice(1)
70
- : relative(dir, join(dir, dirname(idx), l.target)).replaceAll("\\", "/");
71
- indexed.add(resolved.replace(/\/$/, ""));
105
+ for (const indexFile of indexes) {
106
+ const markdown = stripNonProse(read(join(dir, indexFile)));
107
+ for (const link of mdLinks(markdown)) {
108
+ const resolved = resolveLink(dir, indexFile, link.target);
109
+ if (resolved === undefined) continue;
110
+ indexed.add(resolved);
72
111
  // L4b index links resolve (warning — OKF tolerates not-yet-written knowledge)
73
112
  if (!existsSync(join(dir, resolved)))
74
- warn("L4", `${b}: ${idx} links to missing ${resolved}`);
113
+ warn("L4", `${bundle}: ${indexFile} links to missing ${resolved}`);
75
114
  }
76
115
  }
77
116
 
78
117
  // L4a every concept is indexed somewhere
79
- for (const c of concepts)
80
- if (!indexed.has(c)) err("L4", `${b}: ${c} not listed in any index.md`);
118
+ for (const concept of concepts)
119
+ if (!indexed.has(concept)) err("L4", `${bundle}: ${concept} not listed in any index.md`);
81
120
 
82
- for (const c of concepts) {
83
- const body = read(join(dir, c));
121
+ for (const concept of concepts) {
122
+ const body = read(join(dir, concept));
84
123
 
85
124
  // L9 OKF conformance: frontmatter must parse and carry a usable field set.
86
125
  // Frontmatter is the machine-readable half of a concept — routing, filtering and any
87
126
  // future index generator read it — so a typo'd key is a silent data loss, not a style nit.
88
- const fm = body.match(/^---\n([\s\S]*?)\n---/)?.[1];
89
- if (fm === undefined) err("L9", `${b}: ${c} has no YAML frontmatter (OKF requires it)`);
127
+ const frontmatter = body.match(/^---\n([\s\S]*?)\n---/)?.[1];
128
+ if (frontmatter === undefined)
129
+ err("L9", `${bundle}: ${concept} has no YAML frontmatter (OKF requires it)`);
90
130
  else {
91
131
  let meta: Record<string, unknown> | undefined;
92
132
  try {
93
- meta = (parseYaml(fm) ?? {}) as Record<string, unknown>;
94
- } catch (e) {
95
- err("L9", `${b}: ${c} frontmatter is not valid YAML — ${(e as Error).message.split("\n")[0]}`);
133
+ meta = (parseYaml(frontmatter) ?? {}) as Record<string, unknown>;
134
+ } catch (error) {
135
+ err(
136
+ "L9",
137
+ `${bundle}: ${concept} frontmatter is not valid YAML — ${(error as Error).message.split("\n")[0]}`,
138
+ );
96
139
  }
97
140
  if (meta) {
98
- const str = (k: string) => (typeof meta![k] === "string" ? (meta![k] as string).trim() : "");
141
+ const str = (key: string) =>
142
+ typeof meta![key] === "string" ? (meta![key] as string).trim() : "";
99
143
  // type is the one OKF hard requirement; the rest degrade to warnings so an
100
144
  // in-progress hub still lints clean while its authors fill things in.
101
- if (!str("type")) err("L9", `${b}: ${c} frontmatter missing required 'type'`);
102
- for (const k of ["title", "description"])
103
- if (!str(k)) warn("L9", `${b}: ${c} frontmatter missing '${k}'`);
145
+ if (!str("type")) err("L9", `${bundle}: ${concept} frontmatter missing required 'type'`);
146
+ for (const key of ["title", "description"])
147
+ if (!str(key)) warn("L9", `${bundle}: ${concept} frontmatter missing '${key}'`);
104
148
  if ("tags" in meta && !Array.isArray(meta.tags))
105
- err("L9", `${b}: ${c} 'tags' must be a YAML list, not ${typeof meta.tags}`);
106
- if (Array.isArray(meta.tags) && meta.tags.some((t) => typeof t !== "string"))
107
- err("L9", `${b}: ${c} 'tags' must contain only strings`);
149
+ err("L9", `${bundle}: ${concept} 'tags' must be a YAML list, not ${typeof meta.tags}`);
150
+ if (Array.isArray(meta.tags) && meta.tags.some((tag) => typeof tag !== "string"))
151
+ err("L9", `${bundle}: ${concept} 'tags' must contain only strings`);
108
152
  if ("timestamp" in meta && !isTimestamp(meta.timestamp))
109
- warn("L9", `${b}: ${c} 'timestamp' is not an ISO-8601 datetime`);
110
- for (const k of Object.keys(meta))
111
- if (!OKF_FIELDS.has(k)) warn("L9", `${b}: ${c} unknown frontmatter key '${k}'`);
153
+ warn("L9", `${bundle}: ${concept} 'timestamp' is not an ISO-8601 datetime`);
154
+ for (const key of Object.keys(meta))
155
+ if (!OKF_FIELDS.has(key)) warn("L9", `${bundle}: ${concept} unknown frontmatter key '${key}'`);
112
156
  }
113
157
  }
114
158
 
115
159
  // L6 no cross-bundle links from concept docs
116
- for (const l of mdLinks(stripComments(body))) {
117
- if (/(^|\/)bundles\//.test(l.target) || l.target.startsWith("../../"))
118
- err("L6", `${b}: ${c} links into another bundle (${l.target}) — use refs.md`);
160
+ for (const link of mdLinks(stripNonProse(body))) {
161
+ if (/(^|\/)bundles\//.test(link.target) || link.target.startsWith("../../")) {
162
+ err("L6", `${bundle}: ${concept} links into another bundle (${link.target}) — use refs.md`);
163
+ continue;
164
+ }
165
+ // L11 in-bundle concept links resolve. Concepts link to each other as the bundle's
166
+ // actual structure — the catalog cross-link pass and the query skill's back-links to
167
+ // a synthesis's sources both live in these links, and a synthesis nobody can reach
168
+ // from its sources is a dead end. Only the index side of this was ever checked.
169
+ // A warning, like L4b and for the same reason: a link to a concept somebody intends
170
+ // to write next is not-yet-written knowledge, which OKF tolerates by design.
171
+ const resolved = resolveLink(dir, concept, link.target);
172
+ if (resolved !== undefined && !existsSync(join(dir, resolved)))
173
+ warn("L11", `${bundle}: ${concept} links to missing ${resolved}`);
119
174
  }
120
175
  }
121
176
 
122
177
  // L7 ref targets exist
123
178
  if (existsSync(join(dir, "refs.md"))) {
124
- for (const t of refTargets(read(join(dir, "refs.md"))))
125
- if (!bundles.includes(t)) err("L7", `${b}: refs.md targets missing bundle '${t}'`);
179
+ for (const target of refTargets(read(join(dir, "refs.md"))))
180
+ if (!bundles.includes(target)) err("L7", `${bundle}: refs.md targets missing bundle '${target}'`);
126
181
  }
127
182
 
128
- // L8 raw provenance (warning)
183
+ // Enumerated once, bundle-relative (`raw/<type>/<file>.md`) — the spelling the ledger
184
+ // stores, so L10 can compare the two sides without renormalizing on every row.
129
185
  const rawDir = join(dir, "raw");
130
- if (existsSync(rawDir)) {
131
- const rawFiles = (readdirSync(rawDir, { recursive: true }) as string[]).filter((f) => f.endsWith(".md"));
186
+ const rawFiles = existsSync(rawDir)
187
+ ? (readdirSync(rawDir, { recursive: true }) as string[])
188
+ .filter((file) => file.endsWith(".md"))
189
+ .map((file) => `raw/${file.replaceAll("\\", "/")}`)
190
+ : [];
191
+
192
+ // L8 raw provenance (warning)
193
+ if (rawFiles.length) {
132
194
  detail(`${rawFiles.length} raw/ file(s) checked for provenance`);
133
- for (const f of readdirSync(rawDir, { recursive: true }) as string[]) {
195
+ for (const rawFile of rawFiles) {
134
196
  try {
135
- if (!f.endsWith(".md")) continue;
136
- const head = read(join(rawDir, f));
137
- const rfm = head.match(/^---\n([\s\S]*?)\n---/)?.[1];
138
- if (rfm === undefined) { warn("L8", `${b}: raw/${f} missing provenance header`); continue; }
197
+ const head = read(join(dir, rawFile));
198
+ const provenance = head.match(/^---\n([\s\S]*?)\n---/)?.[1];
199
+ if (provenance === undefined) {
200
+ warn("L8", `${bundle}: ${rawFile} missing provenance header`);
201
+ continue;
202
+ }
139
203
  // `source` is the whole point of the header: it is how a bad extraction gets re-read.
140
- if (!/^source:\s*\S/m.test(rfm)) warn("L8", `${b}: raw/${f} provenance missing 'source'`);
141
- const q = rfm.match(/^quality:\s*(\S+)/m)?.[1];
142
- if (q && q !== "high" && q !== "low")
143
- warn("L8", `${b}: raw/${f} quality '${q}' is not high|low`);
204
+ if (!/^source:\s*\S/m.test(provenance))
205
+ warn("L8", `${bundle}: ${rawFile} provenance missing 'source'`);
206
+ const quality = provenance.match(/^quality:\s*(\S+)/m)?.[1];
207
+ if (quality && quality !== "high" && quality !== "low")
208
+ warn("L8", `${bundle}: ${rawFile} quality '${quality}' is not high|low`);
144
209
  } catch {}
145
210
  }
146
211
  }
212
+
213
+ // L10 ledger integrity. log.md is the durable record across both halves of the workflow,
214
+ // and its empty `curated` cells *are* the catalog backlog — but nothing has ever checked
215
+ // that its paths still name anything, so a concept renamed after cataloging leaves a row
216
+ // claiming work that can no longer be found, and neither side notices.
217
+ const ledger = readLedger(dir);
218
+ if (ledger.size) {
219
+ detail(`${ledger.size} log.md row(s) checked`);
220
+ const claimed = new Set<string>();
221
+ for (const row of ledger.values()) {
222
+ if (row.raw) {
223
+ claimed.add(row.raw);
224
+ // raw/ is gitignored and re-derivable, so a hub that was cloned rather than ingested
225
+ // legitimately has every row and no files at all. Only hold a row to its raw file
226
+ // once raw/ has actually been populated; an empty one is that ordinary state.
227
+ if (rawFiles.length && !existsSync(join(dir, row.raw)))
228
+ warn("L10", `${bundle}: log.md row '${row.source}' names missing ${row.raw}`);
229
+ }
230
+ // `declined` is the documented way to close a row without writing a concept
231
+ // (skills/catalog/SKILL.md §5); anything else is a path the row claims to have written.
232
+ // An error, unlike the link rules: there is no not-yet-written case here, since the
233
+ // column is only filled once the concept exists.
234
+ if (row.curated && row.curated !== "declined") {
235
+ const curatedPaths = row.curated.split(",").map((path) => path.trim()).filter(Boolean);
236
+ for (const curated of curatedPaths)
237
+ if (!existsSync(join(dir, curated)))
238
+ err("L10", `${bundle}: log.md row '${row.source}' claims missing concept ${curated}`);
239
+ }
240
+ }
241
+ // An extracted file no row names is invisible work: it is not offered as backlog, so it
242
+ // stays uncurated without ever appearing to be outstanding.
243
+ for (const rawFile of rawFiles)
244
+ if (!claimed.has(rawFile)) warn("L10", `${bundle}: ${rawFile} has no log.md row`);
245
+ } else if (rawFiles.length) {
246
+ warn("L10", `${bundle}: ${rawFiles.length} file(s) in raw/ but log.md records none of them`);
247
+ }
147
248
  }
148
249
 
149
250
  // L3 reverse: outer index entries exist
150
- for (const l of mdLinks(outerIndex)) {
151
- const m = l.target.match(/^bundles\/([a-z0-9-]+)\//);
152
- if (m && !bundles.includes(m[1])) err("L3", `outer.index.md lists missing bundle '${m[1]}'`);
251
+ for (const link of mdLinks(outerIndex)) {
252
+ const match = link.target.match(/^bundles\/([a-z0-9-]+)\//);
253
+ if (match && !bundles.includes(match[1]))
254
+ err("L3", `outer.index.md lists missing bundle '${match[1]}'`);
153
255
  }
154
256
 
155
257
  // L5 index prose check (rough): paragraph-length prose in index files
156
- function proseCheck(name: string, md: string) {
157
- for (const block of stripComments(md).split(/\n\s*\n/)) {
158
- const t = block.trim();
159
- if (!t || /^[#|\-*]/.test(t) || t.startsWith("---")) continue;
160
- if (t.split(/\s+/).length > 30) warn("L5", `${name}: paragraph-length prose in an index file`);
258
+ function proseCheck(name: string, markdown: string) {
259
+ for (const block of stripNonProse(markdown).split(/\n\s*\n/)) {
260
+ const text = block.trim();
261
+ if (!text || /^[#|\-*]/.test(text) || text.startsWith("---")) continue;
262
+ if (text.split(/\s+/).length > 30)
263
+ warn("L5", `${name}: paragraph-length prose in an index file`);
161
264
  }
162
265
  }
163
266
  proseCheck("outer.index.md", outerIndex);
164
- for (const b of bundles)
165
- if (existsSync(join(BUNDLES, b, "index.md"))) proseCheck(`${b}/index.md`, read(join(BUNDLES, b, "index.md")));
267
+ for (const bundle of bundles)
268
+ if (existsSync(join(BUNDLES, bundle, "index.md")))
269
+ proseCheck(`${bundle}/index.md`, read(join(BUNDLES, bundle, "index.md")));
166
270
 
167
271
  console.log(`\nlint: ${errors} error(s), ${warnings} warning(s) across ${bundles.length} bundle(s) in ${totalElapsed()}`);
168
- process.exit(errors ? 1 : 0);
@@ -154,7 +154,7 @@ curation, not transcription.
154
154
  | `.mp3 .wav .m4a .mp4 .mov .mkv` | local `vno` (whisper.cpp), else `whisper` / `faster-whisper` | **low** |
155
155
  | `.vtt .srt` | built-in caption reader | high |
156
156
 
157
- Extracted text is cached hub-wide by content hash at `inbox/extracted/<sha256>.md`, so the
157
+ Extracted text is cached hub-wide by content hash at `.ingest-cache/extracted/<sha256>.md`, so the
158
158
  same file appearing in two bundles converts once.
159
159
 
160
160
  **A recording next to its captions is one source, not two.** `talk.vtt` (or `talk.en.vtt`,
@@ -12,7 +12,7 @@ description: Validate KHB structure (routing integrity, bundle shape, OKF confor
12
12
  log it in `bundles/meta/notes/decisions.md`. No meta bundle means no decision log — do
13
13
  not create one to have somewhere to write.
14
14
 
15
- ## The rules (L1–L9)
15
+ ## The rules (L1–L11)
16
16
 
17
17
  Enforced by `khb lint`. Combines KHB routing rules with
18
18
  OKF v0.1 conformance (see the OKF spec). Reserved filenames: `index.md`, `log.md`
@@ -33,7 +33,8 @@ is a **concept document**.
33
33
  `outer.index.md` exists on disk.
34
34
  - L4. Every concept doc is listed in at least one of the bundle's `index.md` files
35
35
  (error). Index links pointing at missing files are a warning only — OKF treats
36
- broken links as not-yet-written knowledge.
36
+ broken links as not-yet-written knowledge. A `#section` suffix names a place inside
37
+ the target and is dropped before the path is checked.
37
38
  - L5. Index files contain routing only: headings, bullet/table link lines, one-line
38
39
  descriptions. Paragraph-length prose is a violation (warning).
39
40
 
@@ -42,6 +43,12 @@ is a **concept document**.
42
43
  - L6. No markdown link from a concept doc into another bundle's files. Cross-bundle
43
44
  pointers live in `refs.md` only.
44
45
  - L7. Every target bundle named in `refs.md` exists.
46
+ - L11. Markdown links *within* a bundle resolve to a file that exists (warning). Concept
47
+ links are the bundle's real structure — the catalog cross-link pass and the back-links
48
+ the query skill writes from a synthesis to its sources are both made of them, and a
49
+ synthesis nobody can reach from its sources is a dead end. A warning rather than an
50
+ error, for L4's reason: a link to a concept somebody means to write next is
51
+ not-yet-written knowledge.
45
52
 
46
53
  ### Provenance
47
54
 
@@ -49,6 +56,21 @@ is a **concept document**.
49
56
  non-empty `source:`, and `quality:` — if set — reading exactly `high` or `low`.
50
57
  `source` is what makes a bad extraction recoverable, so a raw file without one is
51
58
  uncatalogable, not merely untidy.
59
+ - L10. `log.md` still describes what is on disk. It is the durable record across ingest
60
+ and catalog, and its empty `curated` cells *are* the catalog backlog, so a row that has
61
+ come loose from its files misreports the work outstanding:
62
+ - a `curated` path names a file that exists (**error**). `declined` is the documented
63
+ way to close a row without a concept and is accepted as-is; anything else is a path
64
+ the row claims to have written, and unlike a link there is no not-yet-written case —
65
+ the column is filled only once the concept exists. Renaming a concept after
66
+ cataloging is what usually breaks it.
67
+ - a row's `raw` path names a file that exists (warning), checked **only** when `raw/`
68
+ has files in it. `raw/` is gitignored and re-derivable, so a hub that was cloned
69
+ rather than ingested has every row and no files at all — that is an ordinary state,
70
+ not a finding.
71
+ - every `.md` under `raw/` has a row (warning). An extracted file no row names is
72
+ invisible work: never offered as backlog, so it stays uncurated without ever looking
73
+ outstanding.
52
74
 
53
75
  ### OKF conformance
54
76
 
@@ -4,8 +4,8 @@ bundles/*/raw/
4
4
  # Agent-specific machine-local permissions; shared skills remain tracked.
5
5
  .claude/settings.local.json
6
6
 
7
- # Extraction cache (inbox/extracted/<sha256>.md) — re-derivable from the sources.
8
- /inbox/
7
+ # Extraction cache (.ingest-cache/extracted/<sha256>.md) — re-derivable from the sources.
8
+ /.ingest-cache/
9
9
 
10
10
  # Generated. The leading slash matters: a bare `export/` also matches the `skills/export/`
11
11
  # these files ship with, and silently drops the export skill from the hub's own history.