@msareen/knowledge-hub-builder 0.1.5 → 0.1.7

package/AGENTS.md

@@ -158,6 +158,10 @@ The single exception is a hub with nothing to choose between: with no bundles at
 unnamed ingest lands in `default`, created on the spot, and where `default` is the only
 bundle it lands there as it stands.
 
+When a run finishes, **offer the catalog pass** for the bundle just ingested — report what
+landed and how many rows are uncurated, then ask before curating anything. Neither stopping
+silently at the summary nor cataloging unasked is right.
+
 **Catalog** (`skills/catalog/SKILL.md`) is the judgement half, one bundle at a time: read
 each `raw/` file, split it into concepts, give each OKF frontmatter, link them, register
 them in `index.md`. When the runtime supports parallel agents, fan them out over the raw

package/README.md

@@ -1,7 +1,7 @@
 # knowledge-hub-builder
 
 <p align="center">
-  <img src="images/demo.gif" alt="KHB creates a hub, ingests sources, routes a question to a concept, and validates the result" width="544">
+  <img src="images/demo.gif" alt="khb init in the shell, then an agent session: the ingest skill asks which bundle owns the material, runs khb ingest as a tool call, offers to catalog, writes the concept docs, then answers a question from them" width="600">
 </p>
 
 **KHB (Knowledge Hub Builder)** is a local, markdown-based knowledge base maintained with
@@ -69,20 +69,25 @@ after ingestion.
 `khb ingest <bundle>` reads `sources.yaml` and writes extracted markdown under
 `bundles/<bundle>/raw/`.
 
-Supported without additional system tools:
+Supported out of the box, with no system tools and no further installation:
 
 - text, markdown, CSV, JSON, and YAML
 - PDF and DOCX
 - ODT, XLSX, and PPTX
+- OCR for images and scanned PDFs, applied automatically when a PDF has no text layer
 
-Optional local tools add:
+OCR ships with KHB: `@hyzyla/pdfium`, `sharp`, and `tesseract.js` are ordinary dependencies,
+so installing KHB pulls them down whether or not you ever ingest a scan. That costs roughly
+75 MB of WASM plus `sharp`'s native binaries — the price of an ingest that never stalls
+waiting for a setup step.
 
-- OCR for images and scanned PDFs: `@hyzyla/pdfium`, `sharp`, and `tesseract.js`
-- audio and video transcription: a `whisper` or `faster-whisper` executable on `PATH`
+One extractor is genuinely optional:
 
-If an optional extractor is missing, KHB leaves a pending row in `log.md` and prints the
-required setup. OCR packages must be installed where the KHB package resolves modules; the
-CLI prints that location.
+- audio and video transcription needs a `whisper` or `faster-whisper` executable on `PATH`
+  (`pip install -U openai-whisper`)
+
+Without it, KHB leaves a pending row in `log.md` and prints the required setup rather than
+failing the run.
 
 Every raw markdown file carries provenance:
 

package/SPEC.md

@@ -282,11 +282,15 @@ puts them on the CLI side of the §Division-of-labor line.
 | DOCX | `mammoth`, `pandoc` if present | bundled | high |
 | ODT, PPTX | `fflate` + XML | bundled | high |
 | XLSX | `fflate` → one markdown table per sheet | bundled | high |
-| scanned PDF | `pdfium` + `tesseract.js` (WASM) | opt-in, ~75 MB | low |
-| Images (png/jpg/webp/tif) | `tesseract.js` | opt-in, ~75 MB | low |
+| scanned PDF | `pdfium` + `tesseract.js` (WASM) | bundled, ~75 MB | low |
+| Images (png/jpg/webp/tif) | `tesseract.js` | bundled, ~75 MB | low |
 | Audio, video | `whisper` / `faster-whisper` | opt-in, pip | low |
 
-Missing optional deps degrade to a ledger row with an empty `raw` and a printed install
+The OCR stack is bundled rather than opt-in: an ingest that stops to ask for an install is
+worse than an install that carries WASM nobody uses. Transcription stays opt-in because it is
+a Python executable, not something a JS package manager can pull down.
+
+A missing dep degrades to a ledger row with an empty `raw` and a printed install
 hint — never to a failed run. `quality: low` output is a standing invitation for the catalog
 pass to re-read the original: a vision read of a chart or a scanned table recovers what OCR
 drops, and rewrites the raw file with `extract_tool: claude-vision`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@msareen/knowledge-hub-builder",
-  "version": "0.1.5",
+  "version": "0.1.7",
   "description": "Knowledge Hub Builder — a bundle-of-bundles knowledge base you build with an agent",
   "keywords": ["knowledge-base", "okf", "agent", "wiki", "bundles", "claude", "codex", "rag"],
   "license": "MIT",

package/skills/ingest/SKILL.md

@@ -141,16 +141,22 @@ curation, not transcription.
 Extracted text is cached hub-wide by content hash at `inbox/extracted/<sha256>.md`, so the
 same file appearing in two bundles converts once.
 
-OCR and transcription need optional dependencies. When they are missing khb says so once and
-records the affected files as pending rather than failing the run:
+**OCR needs no setup.** `@hyzyla/pdfium`, `sharp` and `tesseract.js` are dependencies of khb
+itself, so a scanned PDF or a photographed page is read on the first run, in any hub, without
+asking the user to install anything.
+
+Transcription is the one route that can be absent: it wants a `whisper` or `faster-whisper`
+executable on `PATH`.
 
 ```
-bun add @hyzyla/pdfium sharp tesseract.js    # OCR — ~75 MB WASM, no system binary
 pip install -U openai-whisper                # transcription (faster-whisper also works)
 ```
 
-Install them where `khb` resolves modules from — for a global install that is the khb
-package directory, not your hub. khb prints the exact `cd … && bun add …` to use.
+When any extractor is unavailable khb says so once and records the affected files as pending
+rather than failing the run — a `log.md` row with an empty `raw`, waiting for the dependency.
+If khb ever prints a `bun add` hint for the OCR packages, its own install tree is incomplete;
+install them where `khb` resolves modules from — for a global install that is the khb package
+directory, not your hub — and khb prints the exact `cd … && bun add …` to use.
 
 ## 4. Sources khb cannot reach
 
@@ -229,11 +235,30 @@ Still on you, not khb: a source **modified in place** keeps its `curated` value,
 concept derived from it does not re-enter the backlog even though its material changed.
 Watch for `raw/` files whose content shifted and re-catalog them deliberately.
 
-## Hand off
+## Hand off — offer the catalog pass
 
 Ingest is done when the summary shows nothing unexpectedly pending. Report to the user what
-landed, what didn't and why, and how many rows are uncurated — then continue with the
-[catalog skill](../catalog/SKILL.md) to turn `raw/` into concept docs.
+landed, what didn't and why, and how many `log.md` rows are now uncurated.
+
+Then **offer to catalog, and wait for the answer.** Raw text is not yet knowledge — a bundle
+left at the end of ingest has a backlog and nothing citable — so never stop silently on the
+summary, and never start cataloging unasked either. Name the bundle and the size of the
+backlog in the offer, so the answer is informed:
+
+> Ingest landed 94 files in `real-estate/raw/`; 94 rows are uncurated. Shall I catalog them
+> into concept docs now?
+
+Take the answer at face value:
+
+- **yes** → continue with the [catalog skill](../catalog/SKILL.md), on that bundle, reading
+  the backlog from `log.md`.
+- **no, or not now** → stop. The ledger is the durable backlog, so nothing is lost; say that
+  the uncurated rows are waiting whenever they want to pick it up.
+- **only part of it** — one folder, one document, the low-quality files first → catalog that
+  subset and leave the rest of the rows uncurated.
+
+Offer once, for the bundle you just ingested. Do not offer to catalog a bundle this run did
+not touch, and do not roll a "yes" onward into a second bundle's backlog.
 
 ## Hygiene