@msareen/knowledge-hub-builder 0.1.4 → 0.1.7

package/AGENTS.md

@@ -135,7 +135,7 @@ From outside, pass `--hub <dir>` or set `$KHB_HUB`.
 | Command | Purpose |
 |---|---|
 | `khb lint` | validate structure against `skills/lint/SKILL.md` |
-| `khb upgrade` | refresh this hub's package-owned contract docs |
+| `khb upgrade` | refresh this hub's package-owned contract docs — runs by itself whenever `khb.json`'s stamped version differs from the installed khb, so these docs always match the CLI |
 | `khb visualize` | regenerate `visualizer/graph.html` |
 | `khb new-bundle <name>` | scaffold + register a bundle |
 | `khb ingest <bundle>` | acquire + extract every source in `sources.yaml` → `raw/`; maintains `log.md`. Name the bundle — with none it lists the hub's bundles and stops, unless there is nothing to choose between (no bundles, or only `default`), where it uses `default` |
@@ -158,6 +158,10 @@ The single exception is a hub with nothing to choose between: with no bundles at
 unnamed ingest lands in `default`, created on the spot, and where `default` is the only
 bundle it lands there as it stands.
 
+When a run finishes, **offer the catalog pass** for the bundle just ingested — report what
+landed and how many rows are uncurated, then ask before curating anything. Neither stopping
+silently at the summary nor cataloging unasked is right.
+
 **Catalog** (`skills/catalog/SKILL.md`) is the judgement half, one bundle at a time: read
 each `raw/` file, split it into concepts, give each OKF frontmatter, link them, register
 them in `index.md`. When the runtime supports parallel agents, fan them out over the raw

package/README.md

@@ -1,7 +1,7 @@
 # knowledge-hub-builder
 
 <p align="center">
-  <img src="images/demo.gif" alt="KHB creates a hub, ingests sources, routes a question to a concept, and validates the result" width="544">
+  <img src="images/demo.gif" alt="khb init in the shell, then an agent session: the ingest skill asks which bundle owns the material, runs khb ingest as a tool call, offers to catalog, writes the concept docs, then answers a question from them" width="600">
 </p>
 
 **KHB (Knowledge Hub Builder)** is a local, markdown-based knowledge base maintained with
@@ -69,20 +69,25 @@ after ingestion.
 `khb ingest <bundle>` reads `sources.yaml` and writes extracted markdown under
 `bundles/<bundle>/raw/`.
 
-Supported without additional system tools:
+Supported out of the box, with no system tools and no further installation:
 
 - text, markdown, CSV, JSON, and YAML
 - PDF and DOCX
 - ODT, XLSX, and PPTX
+- OCR for images and scanned PDFs, applied automatically when a PDF has no text layer
 
-Optional local tools add:
+OCR ships with KHB: `@hyzyla/pdfium`, `sharp`, and `tesseract.js` are ordinary dependencies,
+so installing KHB pulls them down whether or not you ever ingest a scan. That costs roughly
+75 MB of WASM plus `sharp`'s native binaries — the price of an ingest that never stalls
+waiting for a setup step.
 
-- OCR for images and scanned PDFs: `@hyzyla/pdfium`, `sharp`, and `tesseract.js`
-- audio and video transcription: a `whisper` or `faster-whisper` executable on `PATH`
+One extractor is genuinely optional:
 
-If an optional extractor is missing, KHB leaves a pending row in `log.md` and prints the
-required setup. OCR packages must be installed where the KHB package resolves modules; the
-CLI prints that location.
+- audio and video transcription needs a `whisper` or `faster-whisper` executable on `PATH`
+  (`pip install -U openai-whisper`)
+
+Without it, KHB leaves a pending row in `log.md` and prints the required setup rather than
+failing the run.
 
 Every raw markdown file carries provenance:
 
@@ -208,6 +213,12 @@ files under `skills/` when developing KHB.
 changing `bundles/` or `outer.index.md`. `khb export` includes the same compatibility
 layout in a standalone bundle export.
 
+The refresh is also automatic: every command that works on a hub first compares the `khb`
+version stamped in `khb.json` against the installed package, and upgrades the hub in place
+if they differ, printing one line to stderr. So updating the package is enough — the hub's
+contract docs can never be a version behind the CLI acting on them. Set
+`KHB_NO_AUTO_UPGRADE=1` to suppress the check and leave the hub as it is.
+
 ## Commands
 
 Commands can be run directly or requested through the matching agent skill.
@@ -215,7 +226,7 @@ Commands can be run directly or requested through the matching agent skill.
 | Command | Purpose |
 |---|---|
 | `khb init [dir]` | Create a hub |
-| `khb upgrade` | Refresh package-owned contracts and skills |
+| `khb upgrade` | Refresh package-owned contracts and skills (also runs automatically on version drift) |
 | `khb new-bundle <name> ["scope"]` | Create and register a bundle |
 | `khb ingest [bundle] [--force]` | Acquire and extract declared sources |
 | `khb lint` | Validate routing, bundle structure, and OKF metadata |
@@ -229,14 +240,15 @@ Additional ingest flags:
 
 Run against a hub outside the current directory with `--hub <dir>` or `$KHB_HUB`.
 
-To update the installed package and then refresh a hub:
+To update the installed package:
 
 ```bash
 bun update -g @msareen/knowledge-hub-builder
-cd ~/my-knowledge
-khb upgrade
 ```
 
+Each hub refreshes itself the next time you run any command in it. `khb upgrade` does the
+same thing on demand, with a fuller report.
+
 ## Hub Layout
 
 ```text

package/SPEC.md

@@ -92,6 +92,14 @@ agent is opened on the hub folder and must find its rules there, without knowing
 `khb` is installed. They are package-owned: `khb upgrade` overwrites them in place and
 leaves `bundles/` and `outer.index.md` alone.
 
+Copies drift, so the copy is kept honest by the CLI rather than by the user's memory.
+`khb.json` stamps the version that wrote the hub's copies; before running any command that
+touches a hub, `khb` compares that stamp to the installed package version and, if they
+differ, performs the upgrade in place and says so on stderr. Only `init` (no hub yet) and
+`upgrade` (which is the operation) skip the check, and `KHB_NO_AUTO_UPGRADE=1` disables it.
+The invariant it buys: **a hub's contract docs always state the same version as the `khb`
+acting on them** — an agent can never read a protocol the CLI no longer implements.
+
 ### 2b. The package — `@msareen/knowledge-hub-builder`, installed once
 
 > **None of the following is in your hub.** This section describes the *installed tool*,
@@ -105,7 +113,7 @@ leaves `bundles/` and `outer.index.md` alone.
 @msareen/knowledge-hub-builder/     # installed once, globally — NOT part of a hub
 ├── package.json               # bin: khb → scripts/cli.ts
 ├── scripts/
-│   ├── cli.ts                 # subcommand dispatch; owns the global --hub flag
+│   ├── cli.ts                 # subcommand dispatch; --hub flag; the version drift check
 │   ├── init.ts                # khb init / khb upgrade
 │   ├── new-bundle.ts          # scaffold from .bundle_template, register in outer.index.md
 │   ├── export.ts              # bundle + common patterns → standalone shareable folder
@@ -116,6 +124,7 @@ leaves `bundles/` and `outer.index.md` alone.
 │       ├── extract.ts         # every local extractor + the content-hash cache
 │       ├── ledger.ts          # log.md read/write
 │       ├── paths.ts           # package-side paths — importing it never needs a hub
+│       ├── upgrade.ts         # the refresh itself: `khb upgrade` and the drift check
 │       └── util.ts            # hub resolution + shared helpers
 ├── .bundle_template/          # copied by `khb new-bundle`
 ├── templates/hub/             # copied by `khb init`
@@ -273,11 +282,15 @@ puts them on the CLI side of the §Division-of-labor line.
 | DOCX | `mammoth`, `pandoc` if present | bundled | high |
 | ODT, PPTX | `fflate` + XML | bundled | high |
 | XLSX | `fflate` → one markdown table per sheet | bundled | high |
-| scanned PDF | `pdfium` + `tesseract.js` (WASM) | opt-in, ~75 MB | low |
-| Images (png/jpg/webp/tif) | `tesseract.js` | opt-in, ~75 MB | low |
+| scanned PDF | `pdfium` + `tesseract.js` (WASM) | bundled, ~75 MB | low |
+| Images (png/jpg/webp/tif) | `tesseract.js` | bundled, ~75 MB | low |
 | Audio, video | `whisper` / `faster-whisper` | opt-in, pip | low |
 
-Missing optional deps degrade to a ledger row with an empty `raw` and a printed install
+The OCR stack is bundled rather than opt-in: an ingest that stops to ask for an install is
+worse than an install that carries WASM nobody uses. Transcription stays opt-in because it is
+a Python executable, not something a JS package manager can pull down.
+
+A missing dep degrades to a ledger row with an empty `raw` and a printed install
 hint — never to a failed run. `quality: low` output is a standing invitation for the catalog
 pass to re-read the original: a vision read of a chart or a scanned table recovers what OCR
 drops, and rewrites the raw file with `extract_tool: claude-vision`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@msareen/knowledge-hub-builder",
-  "version": "0.1.4",
+  "version": "0.1.7",
   "description": "Knowledge Hub Builder — a bundle-of-bundles knowledge base you build with an agent",
   "keywords": ["knowledge-base", "okf", "agent", "wiki", "bundles", "claude", "codex", "rag"],
   "license": "MIT",

package/scripts/cli.ts

@@ -1,7 +1,7 @@
 #!/usr/bin/env bun
 // khb — the CLI. Subcommands are loaded lazily: `init` must run before a hub exists,
 // so nothing that resolves a hub may be imported at module scope.
-import { version } from "./lib/paths";
+import { version, findHub, markerIn, MARKER } from "./lib/paths";
 
 const COMMANDS: Record<string, { load: () => Promise<unknown>; help: string }> = {
   init: { load: () => import("./init"), help: "khb init [dir]                  create a hub here (or in dir)" },
@@ -59,6 +59,28 @@ if (!entry) {
   process.exit(1);
 }
 
+// Version drift: a hub carries package-owned copies of the agent contract, and a hub
+// stamped at an older version than the installed khb is stating an older contract than
+// the one the CLI now implements. Rather than let the two disagree, refresh the hub in
+// place before running the command — `khb upgrade` touches nothing the user wrote.
+// `init` has no hub yet, `upgrade` does this itself, and $KHB_NO_AUTO_UPGRADE opts out.
+if (cmd !== "init" && cmd !== "upgrade" && !process.env.KHB_NO_AUTO_UPGRADE) {
+  const hub = findHub();
+  if (hub) {
+    const { hubVersion, upgradeHub } = await import("./lib/upgrade");
+    // A marker under a pre-rename name is drift too, even at a matching version.
+    if (hubVersion(hub) !== version() || markerIn(hub) !== MARKER) {
+      const { from, to, pruned, renamed } = upgradeHub(hub);
+      // stderr, so a command's own output stays pipeable.
+      console.error(
+        `khb: hub was built by ${from ?? "an unknown version"}, khb is ${to} — refreshed its contract docs.`,
+      );
+      if (renamed) console.error(`khb:   renamed ${renamed} -> khb.json`);
+      if (pruned.length) console.error(`khb:   removed (no longer part of the contract): ${pruned.join(", ")}`);
+    }
+  }
+}
+
 // Subcommand modules parse process.argv.slice(2) themselves — reshape it so they see
 // their own arguments and not the subcommand name.
 process.argv = [process.argv[0], process.argv[1], ...argv];

package/scripts/init.ts

@@ -5,63 +5,23 @@
 // copied INTO the hub — an agent opened on the hub folder must be able to read them
 // without knowing where khb is installed. Those copies are package-owned: `upgrade`
 // overwrites them.
-import { cpSync, mkdirSync, writeFileSync, existsSync, statSync, rmSync } from "node:fs";
-import { join, resolve, basename, dirname } from "node:path";
-import { PKG, HUB_TEMPLATE, MANAGED, RETIRED, MARKER, markerIn, version } from "./lib/paths";
+//
+// The mechanism itself lives in lib/upgrade.ts, because cli.ts also runs it on version
+// drift before any hub command.
+import { cpSync, mkdirSync, existsSync } from "node:fs";
+import { join, resolve, basename } from "node:path";
+import { HUB_TEMPLATE, MARKER, markerIn } from "./lib/paths";
+import { upgradeHub, syncManaged, stamp } from "./lib/upgrade";
 
 const upgrading = process.env.KHB_SUBCOMMAND === "upgrade";
 const [dirArg] = process.argv.slice(2);
 
-/** Copy every package-owned contract file into the hub, replacing what is there. */
-function syncManaged(hub: string): string[] {
-  const done: string[] = [];
-  for (const f of MANAGED) {
-    const src = join(PKG, f);
-    if (!existsSync(src)) continue;
-    const dest = join(hub, f);
-    mkdirSync(dirname(dest), { recursive: true });
-    cpSync(src, dest, { recursive: true, force: true });
-    done.push(statSync(src).isDirectory() ? `${f}/` : f);
-  }
-  return done;
-}
-
-/** Drop package-owned files that later versions stopped shipping. */
-function pruneRetired(hub: string): string[] {
-  const gone: string[] = [];
-  for (const f of RETIRED) {
-    const p = join(hub, f);
-    if (!existsSync(p)) continue;
-    rmSync(p, { recursive: true, force: true });
-    gone.push(f);
-  }
-  return gone;
-}
-
-function stamp(hub: string, created?: string) {
-  writeFileSync(
-    join(hub, MARKER),
-    JSON.stringify(
-      { khb: version(), created: created ?? new Date().toISOString(), upgraded: new Date().toISOString() },
-      null,
-      2,
-    ) + "\n",
-  );
-}
-
 if (upgrading) {
   const { HUB } = await import("./lib/util"); // resolves the hub, or exits with guidance
-  // The hub may still carry a marker name from an older version; stamp() writes MARKER,
-  // so drop the old file rather than leaving the hub with two.
-  const found = markerIn(HUB)!;
-  const before = JSON.parse(await Bun.file(join(HUB, found)).text());
-  if (found !== MARKER) rmSync(join(HUB, found));
-  const synced = syncManaged(HUB);
-  const pruned = pruneRetired(HUB);
-  stamp(HUB, before.created);
-  console.log(`Upgraded ${HUB}: ${before.khb ?? before.bkr ?? "?"} -> ${version()}`);
+  const { from, to, synced, pruned, renamed } = upgradeHub(HUB);
+  console.log(`Upgraded ${HUB}: ${from ?? "?"} -> ${to}`);
   console.log(`  refreshed: ${synced.join(", ")}`);
-  if (found !== MARKER) console.log(`  renamed: ${found} -> ${MARKER}`);
+  if (renamed) console.log(`  renamed: ${renamed} -> ${MARKER}`);
   if (pruned.length) console.log(`  removed (no longer part of the contract): ${pruned.join(", ")}`);
   console.log(`Your bundles/ and outer.index.md were not touched. Next: khb lint`);
 } else {

package/scripts/lib/paths.ts

@@ -1,7 +1,7 @@
 // Package-side paths. Importing this must never require a hub to exist — `khb init`
 // runs before there is one. Hub-side paths live in util.ts.
 import { fileURLToPath } from "node:url";
-import { join } from "node:path";
+import { join, dirname, resolve } from "node:path";
 import { readFileSync, existsSync } from "node:fs";
 
 /** Root of the installed @msareen/knowledge-hub-builder package (NOT the user's hub). */
@@ -29,6 +29,26 @@ export const LEGACY_MARKERS = ["bkr.json"];
 export const markerIn = (dir: string): string | undefined =>
   [MARKER, ...LEGACY_MARKERS].find((m) => existsSync(join(dir, m)));
 
+/**
+ * The hub this invocation acts on, or undefined if there is none. Precedence:
+ * $KHB_HUB (set from --hub by cli.ts) > nearest ancestor of cwd holding a marker.
+ * Soft by design — util.ts turns "none" into an error with guidance, while the version
+ * drift check in cli.ts must stay silent for commands that run outside a hub.
+ */
+export function findHub(): string | undefined {
+  const explicit = process.env.KHB_HUB;
+  if (explicit) {
+    const dir = resolve(explicit);
+    return markerIn(dir) ? dir : undefined;
+  }
+  for (let dir = process.cwd(); ; ) {
+    if (markerIn(dir)) return dir;
+    const up = dirname(dir);
+    if (up === dir) return undefined;
+    dir = up;
+  }
+}
+
 /**
  * Package-owned files copied into every hub by `khb init` and refreshed by
  * `khb upgrade`. These are the agent contract — the hub needs its own copies so an

package/scripts/lib/upgrade.ts

@@ -0,0 +1,93 @@
+// The upgrade mechanism: refresh a hub's package-owned contract docs to match the
+// installed khb. Lives here rather than in init.ts because two callers need it — the
+// explicit `khb upgrade`, and the drift check cli.ts runs before every hub command.
+//
+// Nothing here may import util.ts: the drift check runs before a hub is resolved, and
+// util.ts resolves one or exits.
+import { cpSync, mkdirSync, writeFileSync, readFileSync, existsSync, statSync, rmSync } from "node:fs";
+import { join, dirname } from "node:path";
+import { PKG, MANAGED, RETIRED, MARKER, markerIn, version } from "./paths";
+
+export type UpgradeResult = {
+  /** Version recorded in the hub's marker before the upgrade, if it recorded one. */
+  from?: string;
+  to: string;
+  synced: string[];
+  pruned: string[];
+  /** Set when a legacy marker name was renamed to khb.json. */
+  renamed?: string;
+};
+
+/** Read the khb version a hub was last stamped with. Undefined if it records none. */
+export function hubVersion(hub: string): string | undefined {
+  const found = markerIn(hub);
+  if (!found) return undefined;
+  try {
+    const marker = JSON.parse(readFileSync(join(hub, found), "utf8"));
+    return marker.khb ?? marker.bkr; // bkr: the pre-rename field name
+  } catch {
+    return undefined; // unreadable marker — treat as drifted, upgrade will restamp it
+  }
+}
+
+/** Copy every package-owned contract file into the hub, replacing what is there. */
+export function syncManaged(hub: string): string[] {
+  const done: string[] = [];
+  for (const f of MANAGED) {
+    const src = join(PKG, f);
+    if (!existsSync(src)) continue;
+    const dest = join(hub, f);
+    mkdirSync(dirname(dest), { recursive: true });
+    cpSync(src, dest, { recursive: true, force: true });
+    done.push(statSync(src).isDirectory() ? `${f}/` : f);
+  }
+  return done;
+}
+
+/** Drop package-owned files that later versions stopped shipping. */
+function pruneRetired(hub: string): string[] {
+  const gone: string[] = [];
+  for (const f of RETIRED) {
+    const p = join(hub, f);
+    if (!existsSync(p)) continue;
+    rmSync(p, { recursive: true, force: true });
+    gone.push(f);
+  }
+  return gone;
+}
+
+/** Write khb.json with the installed version, preserving the hub's creation date. */
+export function stamp(hub: string, created?: string) {
+  writeFileSync(
+    join(hub, MARKER),
+    JSON.stringify(
+      { khb: version(), created: created ?? new Date().toISOString(), upgraded: new Date().toISOString() },
+      null,
+      2,
+    ) + "\n",
+  );
+}
+
+/**
+ * Bring `hub`'s package-owned files up to the installed version. Touches only what the
+ * package owns: bundles/, outer.index.md and anything else the user wrote are untouched.
+ */
+export function upgradeHub(hub: string): UpgradeResult {
+  // The hub may still carry a marker name from an older version; stamp() writes MARKER,
+  // so drop the old file rather than leaving the hub with two.
+  const found = markerIn(hub)!;
+  let created: string | undefined;
+  let from: string | undefined;
+  try {
+    const before = JSON.parse(readFileSync(join(hub, found), "utf8"));
+    created = before.created;
+    from = before.khb ?? before.bkr;
+  } catch {
+    /* unreadable marker: rewritten below with today's date */
+  }
+  if (found !== MARKER) rmSync(join(hub, found));
+  const synced = syncManaged(hub);
+  const pruned = pruneRetired(hub);
+  stamp(hub, created);
+  return { from, to: version(), synced, pruned, renamed: found === MARKER ? undefined : found };
+}

package/scripts/lib/util.ts

@@ -1,7 +1,7 @@
 import { readdirSync, readFileSync, existsSync, statSync, mkdirSync, writeFileSync } from "node:fs";
 import { createHash } from "node:crypto";
-import { join, basename, dirname, resolve } from "node:path";
-import { MARKER, markerIn } from "./paths";
+import { join, basename, resolve } from "node:path";
+import { MARKER, markerIn, findHub } from "./paths";
 
 /**
  * Find the hub root — the folder holding khb.json, outer.index.md and bundles/.
@@ -10,20 +10,12 @@ import { MARKER, markerIn } from "./paths";
  * ancestor of cwd containing the marker.
  */
 function resolveHub(): string {
+  const found = findHub();
+  if (found) return found;
   const explicit = process.env.KHB_HUB;
   if (explicit) {
-    const dir = resolve(explicit);
-    if (!markerIn(dir)) {
-      console.error(`Not a KHB hub (no ${MARKER}): ${dir}`);
-      process.exit(1);
-    }
-    return dir;
-  }
-  for (let dir = process.cwd(); ; ) {
-    if (markerIn(dir)) return dir;
-    const up = dirname(dir);
-    if (up === dir) break;
-    dir = up;
+    console.error(`Not a KHB hub (no ${MARKER}): ${resolve(explicit)}`);
+    process.exit(1);
   }
   console.error(`No KHB hub found in ${process.cwd()} or any parent directory.`);
   console.error(`Create one:   khb init <dir>`);

package/skills/ingest/SKILL.md

@@ -141,16 +141,22 @@ curation, not transcription.
 Extracted text is cached hub-wide by content hash at `inbox/extracted/<sha256>.md`, so the
 same file appearing in two bundles converts once.
 
-OCR and transcription need optional dependencies. When they are missing khb says so once and
-records the affected files as pending rather than failing the run:
+**OCR needs no setup.** `@hyzyla/pdfium`, `sharp` and `tesseract.js` are dependencies of khb
+itself, so a scanned PDF or a photographed page is read on the first run, in any hub, without
+asking the user to install anything.
+
+Transcription is the one route that can be absent: it wants a `whisper` or `faster-whisper`
+executable on `PATH`.
 
 ```
-bun add @hyzyla/pdfium sharp tesseract.js    # OCR — ~75 MB WASM, no system binary
 pip install -U openai-whisper                # transcription (faster-whisper also works)
 ```
 
-Install them where `khb` resolves modules from — for a global install that is the khb
-package directory, not your hub. khb prints the exact `cd … && bun add …` to use.
+When any extractor is unavailable khb says so once and records the affected files as pending
+rather than failing the run — a `log.md` row with an empty `raw`, waiting for the dependency.
+If khb ever prints a `bun add` hint for the OCR packages, its own install tree is incomplete;
+install them where `khb` resolves modules from — for a global install that is the khb package
+directory, not your hub — and khb prints the exact `cd … && bun add …` to use.
 
 ## 4. Sources khb cannot reach
 
@@ -229,11 +235,30 @@ Still on you, not khb: a source **modified in place** keeps its `curated` value,
 concept derived from it does not re-enter the backlog even though its material changed.
 Watch for `raw/` files whose content shifted and re-catalog them deliberately.
 
-## Hand off
+## Hand off — offer the catalog pass
 
 Ingest is done when the summary shows nothing unexpectedly pending. Report to the user what
-landed, what didn't and why, and how many rows are uncurated — then continue with the
-[catalog skill](../catalog/SKILL.md) to turn `raw/` into concept docs.
+landed, what didn't and why, and how many `log.md` rows are now uncurated.
+
+Then **offer to catalog, and wait for the answer.** Raw text is not yet knowledge — a bundle
+left at the end of ingest has a backlog and nothing citable — so never stop silently on the
+summary, and never start cataloging unasked either. Name the bundle and the size of the
+backlog in the offer, so the answer is informed:
+
+> Ingest landed 94 files in `real-estate/raw/`; 94 rows are uncurated. Shall I catalog them
+> into concept docs now?
+
+Take the answer at face value:
+
+- **yes** → continue with the [catalog skill](../catalog/SKILL.md), on that bundle, reading
+  the backlog from `log.md`.
+- **no, or not now** → stop. The ledger is the durable backlog, so nothing is lost; say that
+  the uncurated rows are waiting whenever they want to pick it up.
+- **only part of it** — one folder, one document, the low-quality files first → catalog that
+  subset and leave the rest of the rows uncurated.
+
+Offer once, for the bundle you just ingested. Do not offer to catalog a bundle this run did
+not touch, and do not roll a "yes" onward into a second bundle's backlog.
 
 ## Hygiene