@msareen/knowledge-hub-builder 0.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (57) hide show
  1. package/.agents/skills/catalog/SKILL.md +7 -0
  2. package/.agents/skills/export/SKILL.md +7 -0
  3. package/.agents/skills/ingest/SKILL.md +7 -0
  4. package/.agents/skills/lint/SKILL.md +7 -0
  5. package/.agents/skills/new-bundle/SKILL.md +7 -0
  6. package/.agents/skills/query/SKILL.md +7 -0
  7. package/.agents/skills/visualize/SKILL.md +7 -0
  8. package/.bundle_template/index.md +9 -0
  9. package/.bundle_template/log.md +10 -0
  10. package/.bundle_template/raw/.gitkeep +15 -0
  11. package/.bundle_template/refs.md +6 -0
  12. package/.bundle_template/sources.yaml +13 -0
  13. package/.claude/skills/catalog/SKILL.md +7 -0
  14. package/.claude/skills/export/SKILL.md +7 -0
  15. package/.claude/skills/ingest/SKILL.md +7 -0
  16. package/.claude/skills/lint/SKILL.md +7 -0
  17. package/.claude/skills/new-bundle/SKILL.md +7 -0
  18. package/.claude/skills/query/SKILL.md +7 -0
  19. package/.claude/skills/visualize/SKILL.md +7 -0
  20. package/AGENTS.md +167 -0
  21. package/CLAUDE.md +13 -0
  22. package/README.md +289 -0
  23. package/SPEC.md +354 -0
  24. package/document/faq.md +156 -0
  25. package/package.json +52 -0
  26. package/scripts/cli.ts +66 -0
  27. package/scripts/export.ts +42 -0
  28. package/scripts/ingest/acquire.ts +189 -0
  29. package/scripts/ingest/exts.ts +29 -0
  30. package/scripts/ingest/files.ts +29 -0
  31. package/scripts/ingest/folder.ts +44 -0
  32. package/scripts/ingest/index.ts +125 -0
  33. package/scripts/ingest/protect.ts +42 -0
  34. package/scripts/ingest/web.ts +54 -0
  35. package/scripts/init.ts +93 -0
  36. package/scripts/lib/args.ts +8 -0
  37. package/scripts/lib/extract.ts +384 -0
  38. package/scripts/lib/graph-page.ts +477 -0
  39. package/scripts/lib/graph.ts +117 -0
  40. package/scripts/lib/ledger.ts +136 -0
  41. package/scripts/lib/log.ts +53 -0
  42. package/scripts/lib/paths.ts +55 -0
  43. package/scripts/lib/scaffold.ts +56 -0
  44. package/scripts/lib/util.ts +154 -0
  45. package/scripts/lint.ts +165 -0
  46. package/scripts/new-bundle.ts +17 -0
  47. package/scripts/visualize.ts +104 -0
  48. package/skills/catalog/SKILL.md +164 -0
  49. package/skills/export/SKILL.md +31 -0
  50. package/skills/ingest/SKILL.md +229 -0
  51. package/skills/lint/SKILL.md +69 -0
  52. package/skills/new-bundle/SKILL.md +25 -0
  53. package/skills/query/SKILL.md +114 -0
  54. package/skills/visualize/SKILL.md +33 -0
  55. package/templates/hub/gitattributes +12 -0
  56. package/templates/hub/gitignore +12 -0
  57. package/templates/hub/outer.index.md +13 -0
@@ -0,0 +1,7 @@
1
+ ---
2
+ name: catalog
3
+ description: Curate one KHB bundle's raw material into linked OKF concept documents. Use after ingest or when asked to catalog or organize a bundle.
4
+ ---
5
+
6
+ Read and follow the [canonical workflow](../../../skills/catalog/SKILL.md) in full; treat
7
+ all of its instructions as part of this skill.
@@ -0,0 +1,7 @@
1
+ ---
2
+ name: export
3
+ description: Export a KHB bundle as a standalone shareable folder. Use when asked to share or ship one bundle.
4
+ ---
5
+
6
+ Read and follow the [canonical workflow](../../../skills/export/SKILL.md) in full; treat
7
+ all of its instructions as part of this skill.
@@ -0,0 +1,7 @@
1
+ ---
2
+ name: ingest
3
+ description: Acquire external material into a KHB bundle as provenance-bearing markdown. Use when asked to add, import, pull, or refresh source data.
4
+ ---
5
+
6
+ Read and follow the [canonical workflow](../../../skills/ingest/SKILL.md) in full; treat
7
+ all of its instructions as part of this skill.
@@ -0,0 +1,7 @@
1
+ ---
2
+ name: lint
3
+ description: Validate and repair KHB structure and OKF conformance. Use after structural edits or when asked to check the knowledge base.
4
+ ---
5
+
6
+ Read and follow the [canonical workflow](../../../skills/lint/SKILL.md) in full; treat all
7
+ of its instructions as part of this skill.
@@ -0,0 +1,7 @@
1
+ ---
2
+ name: new-bundle
3
+ description: Create and register a KHB bundle. Use only when the user explicitly wants a new owner or knowledge area.
4
+ ---
5
+
6
+ Read and follow the [canonical workflow](../../../skills/new-bundle/SKILL.md) in full;
7
+ treat all of its instructions as part of this skill.
@@ -0,0 +1,7 @@
1
+ ---
2
+ name: query
3
+ description: Answer a question from a KHB hub by routing through its indexes and concept documents. Use for knowledge-base lookups.
4
+ ---
5
+
6
+ Read and follow the [canonical workflow](../../../skills/query/SKILL.md) in full; treat all
7
+ of its instructions as part of this skill.
@@ -0,0 +1,7 @@
1
+ ---
2
+ name: visualize
3
+ description: Regenerate the KHB bundle graph. Use when asked to visualize the hub or after bundle references change.
4
+ ---
5
+
6
+ Read and follow the [canonical workflow](../../../skills/visualize/SKILL.md) in full; treat
7
+ all of its instructions as part of this skill.
@@ -0,0 +1,9 @@
1
+ ---
2
+ okf_version: '0.1'
3
+ ---
4
+
5
+ # {{name}}
6
+
7
+ <!-- OKF index: progressive disclosure. List every concept doc / subdirectory as:
8
+ * [Title](path.md) - one-line description (from the doc's frontmatter)
9
+ Organize concepts in whatever subdirectories fit the domain. -->
@@ -0,0 +1,10 @@
1
+ # {{name}} — ingest log
2
+
3
+ Ingestion ledger, one row per source. `khb ingest` maintains `source`, `sha256`,
4
+ `fetched` and `raw`; the agent fills `curated` while cataloging (skills/catalog/SKILL.md).
5
+
6
+ Empty `raw` = seen but not extracted (protected, unreadable, or skipped by a flag).
7
+ Empty `curated` = in raw/ but not yet distilled into a concept doc.
8
+
9
+ | source | sha256 | fetched | raw | curated |
10
+ |---|---|---|---|---|
@@ -0,0 +1,15 @@
1
+ # raw/ — ingested source material, NOT canonical.
2
+ #
3
+ # Phase 1 of skills/ingest/SKILL.md drops faithful copies here, one subdirectory per source
4
+ # type: raw/folder/, raw/web/, raw/confluence/, raw/ado/, raw/pdf/ ...
5
+ # Every .md file starts with a provenance header (lint L8):
6
+ #
7
+ # ---
8
+ # source: <url | path | tool query>
9
+ # fetched: <ISO timestamp>
10
+ # ---
11
+ #
12
+ # Re-acquiring overwrites this tree idempotently. It is gitignored
13
+ # (bundles/*/raw/) and never cited in answers — curate into concept docs and
14
+ # cite those instead. This file only keeps the directory present in the
15
+ # template; delete it once the bundle has real raw content.
@@ -0,0 +1,6 @@
1
+ # {{name}} — refs
2
+
3
+ Cross-bundle references. The only way out of this bundle.
4
+
5
+ | Bundle | Why | Note (optional) |
6
+ |---|---|---|
@@ -0,0 +1,13 @@
1
+ # Ingestion sources for {{name}}.
2
+ # folder/web: scripted — bun run ingest {{name}}
3
+ # confluence/ado: listed as provenance; agent ingests via MCP/CLI (AGENTS.md §Ingestion)
4
+ sources: []
5
+ # - type: folder
6
+ # path: /abs/path/to/some-folder
7
+ # - type: web
8
+ # urls: []
9
+ # - type: confluence
10
+ # space: SPACEKEY
11
+ # - type: ado
12
+ # org: myorg
13
+ # project: MyProject
@@ -0,0 +1,7 @@
1
+ ---
2
+ name: catalog
3
+ description: Curate one KHB bundle's raw material into linked OKF concept documents. Use after ingest or when asked to catalog or organize a bundle.
4
+ ---
5
+
6
+ Read and follow the [canonical workflow](../../../skills/catalog/SKILL.md) in full; treat
7
+ all of its instructions as part of this skill.
@@ -0,0 +1,7 @@
1
+ ---
2
+ name: export
3
+ description: Export a KHB bundle as a standalone shareable folder. Use when asked to share or ship one bundle.
4
+ ---
5
+
6
+ Read and follow the [canonical workflow](../../../skills/export/SKILL.md) in full; treat
7
+ all of its instructions as part of this skill.
@@ -0,0 +1,7 @@
1
+ ---
2
+ name: ingest
3
+ description: Acquire external material into a KHB bundle as provenance-bearing markdown. Use when asked to add, import, pull, or refresh source data.
4
+ ---
5
+
6
+ Read and follow the [canonical workflow](../../../skills/ingest/SKILL.md) in full; treat
7
+ all of its instructions as part of this skill.
@@ -0,0 +1,7 @@
1
+ ---
2
+ name: lint
3
+ description: Validate and repair KHB structure and OKF conformance. Use after structural edits or when asked to check the knowledge base.
4
+ ---
5
+
6
+ Read and follow the [canonical workflow](../../../skills/lint/SKILL.md) in full; treat all
7
+ of its instructions as part of this skill.
@@ -0,0 +1,7 @@
1
+ ---
2
+ name: new-bundle
3
+ description: Create and register a KHB bundle. Use only when the user explicitly wants a new owner or knowledge area.
4
+ ---
5
+
6
+ Read and follow the [canonical workflow](../../../skills/new-bundle/SKILL.md) in full;
7
+ treat all of its instructions as part of this skill.
@@ -0,0 +1,7 @@
1
+ ---
2
+ name: query
3
+ description: Answer a question from a KHB hub by routing through its indexes and concept documents. Use for knowledge-base lookups.
4
+ ---
5
+
6
+ Read and follow the [canonical workflow](../../../skills/query/SKILL.md) in full; treat all
7
+ of its instructions as part of this skill.
@@ -0,0 +1,7 @@
1
+ ---
2
+ name: visualize
3
+ description: Regenerate the KHB bundle graph. Use when asked to visualize the hub or after bundle references change.
4
+ ---
5
+
6
+ Read and follow the [canonical workflow](../../../skills/visualize/SKILL.md) in full; treat
7
+ all of its instructions as part of this skill.
package/AGENTS.md ADDED
@@ -0,0 +1,167 @@
1
+ # AGENTS.md — common contract for Claude, Codex, and other agents
2
+
3
+ You are inside a **KHB hub** — a bundle-of-bundles knowledge system. Full design: `SPEC.md`.
4
+
5
+ The hub is this folder: `khb.json`, `outer.index.md`, and `bundles/`. The knowledge is
6
+ here; the `khb` tooling is installed separately and holds none of it. This file and its
7
+ siblings (`SPEC.md`, `skills/`) are package-owned copies refreshed by `khb upgrade` —
8
+ never edit them, edit bundle content instead. Every workflow protocol lives wholly inside
9
+ `skills/<name>/SKILL.md`: query, ingest, catalog, lint, new-bundle, export, visualize.
10
+ Those files are plain markdown — any agent can read one directly, whether or not its
11
+ runtime has a notion of "skills".
12
+
13
+ ## What a bundle is
14
+
15
+ A bundle is a **logical unit defined by its owner** — a person, a team, a project, a client
16
+ — not a subject classification. It holds as many topics as its owner has; topics are
17
+ organized inside it with subdirectories. A bundle whose contents look heterogeneous is
18
+ working as intended.
19
+
20
+ Creating, splitting or merging bundles is a human decision, always. Never do it on your own
21
+ initiative, in any workflow, however obviously a subject seems to want its own home. On
22
+ explicit instruction, carve one out and move the files; otherwise leave the shape alone.
23
+
24
+ ## How to navigate
25
+
26
+ 1. Start at `outer.index.md`. Pick exactly one bundle for the question. Do not browse.
27
+ 2. If that doesn't settle it, escalate per `skills/query/SKILL.md`: grep the bundle
28
+ `index.md` files, then concept front matter, then **ask the user** which bundle.
29
+ Never guess silently.
30
+ 3. Enter the bundle via its `index.md`, then read only the concepts/notes it routes you to.
31
+
32
+ This file is the single common contract — bundles carry no per-bundle agent rules.
33
+ Claude loads it through `CLAUDE.md`; Codex loads it directly.
34
+
35
+ ## Hard rules
36
+
37
+ - **One bundle at a time.** If a question spans two bundles, follow the collation protocol
38
+ in `skills/query/SKILL.md`: resolve in the first bundle, follow its `refs.md` to the
39
+ second, and enter the second **through its own index.md**. Never deep-link from one bundle's notes into
40
+ another's. Join answers in your response, not in the files.
41
+ - **Indexes are routing only.** Never add knowledge content to `outer.index.md` or any
42
+ `index.md` — add a concept doc and link it from the index.
43
+ - **Writing (OKF v0.1):** a concept = one markdown file, anywhere in the bundle except
44
+ reserved names (`index.md`, `log.md`, `refs.md`) and `raw/`. Group concepts in
45
+ subdirectories that fit the domain (`tables/`, `metrics/`, `notes/`, …). Every
46
+ concept doc starts with front matter — `type` is required:
47
+
48
+ ```yaml
49
+ ---
50
+ type: Metric # free-form: Table, Playbook, Decision Log, Reference, …
51
+ title: Display name
52
+ description: One-line summary (reused by index generators)
53
+ resource: <canonical URI of the underlying asset, if any>
54
+ tags: [a, b]
55
+ timestamp: 2026-07-19T00:00:00Z
56
+ ---
57
+ ```
58
+
59
+ List every concept in an `index.md` (`* [Title](path.md) - description`). Link
60
+ concepts to each other with plain markdown links, `/path/from/bundle/root.md`
61
+ preferred; relationship meaning goes in the surrounding prose.
62
+ - **Cross-bundle relationships** go in `refs.md` only — never inline links across bundles.
63
+ - **Decisions:** if this hub has a `meta` bundle, any change to how the hub itself works
64
+ gets a dated line in `bundles/meta/notes/decisions.md`.
65
+ - **`raw/` is not canonical.** It's ingested source material awaiting curation. Cite
66
+ concept docs; use `raw/` only when curating.
67
+ - **New bundle:** `khb new-bundle <name>` — never hand-copy `.bundle_template`.
68
+ - After structural edits run `khb lint` and fix what it reports.
69
+
70
+ ## Division of labor — khb vs agent
71
+
72
+ One hard boundary governs every workflow and every future change to the tooling:
73
+
74
+ - **`khb` converts bytes to text.** Hashing, caching, file plumbing, ledger-keeping, and
75
+ *every* local extractor: PDF/DOCX/ODT/XLSX/PPTX libraries, tesseract OCR for scans and
76
+ images, whisper for audio and video. All of it deterministic, offline, and free of
77
+ charge. It **never contacts a model** — not directly, not by shelling out.
78
+ - **The agent decides what the text means.** Splitting a document into concepts, labeling
79
+ and linking them, curating `raw/` into the wiki, escalating a bad OCR to a vision read,
80
+ and judging when a query has produced a new concept worth keeping.
81
+
82
+ The line is *conversion vs. interpretation*, not cheap vs. expensive: tesseract and whisper
83
+ belong in `khb` despite costing real CPU, because their output is reproducible and needs no
84
+ judgement. The split exists so every intelligent decision is auditable — it leaves a
85
+ provenance header and a `log.md` row. When extending khb, keep conversion in the CLI and
86
+ interpretation in an agent pass. Never add a `khb … --auto-label` / `--summarize` flag that
87
+ calls a model; that rots the boundary.
88
+
89
+ ## Where skills live — one canonical file, thin adapters
90
+
91
+ A skill exists **once**, in `skills/<name>/SKILL.md`, and that one file carries the entire
92
+ protocol. Runtimes that have a notion of "skills" discover it through a parallel adapter
93
+ that only points at it:
94
+
95
+ ```
96
+ skills/<name>/SKILL.md ← canonical: the whole protocol
97
+ .claude/skills/<name>/SKILL.md ← adapter: frontmatter + a link
98
+ .agents/skills/<name>/SKILL.md ← adapter: frontmatter + a link
99
+ ```
100
+
101
+ Each adapter is the same handful of lines — nothing else belongs in one:
102
+
103
+ ```markdown
104
+ ---
105
+ name: <name>
106
+ description: <one line; this is what the runtime matches on when deciding to load the skill>
107
+ ---
108
+
109
+ Read and follow the [canonical workflow](../../../skills/<name>/SKILL.md) in full; treat all
110
+ of its instructions as part of this skill.
111
+ ```
112
+
113
+ The `../../../` climbs from `.claude/skills/<name>/` back to the hub root. The two adapters
114
+ are byte-identical to each other; only the canonical file differs from them.
115
+
116
+ **Adding a hub-specific skill** follows the same shape — create all three files, keep
117
+ `<name>` identical across them, and put every instruction in the canonical file only. The
118
+ adapter `description` is a triggering surface, so write it as *when to reach for this*, not
119
+ as a summary of what it does; the canonical file's own frontmatter can be longer and more
120
+ precise, since it is read after the decision to load has been made.
121
+
122
+ Never let protocol text drift into an adapter: two copies of a workflow means an agent can
123
+ load the stale one and have no way to tell. If a runtime needs something the link cannot
124
+ express, extend the canonical file and keep the adapter a pointer.
125
+
126
+ `khb upgrade` refreshes all three trees, so the built-in skills are package-owned — but it
127
+ overwrites paths it ships and does not delete unknown ones, so a skill you add to a hub
128
+ under a name khb does not use survives upgrades intact.
129
+
130
+ ## Tooling
131
+
132
+ Run `khb` from anywhere inside the hub — it finds the hub by walking up to `khb.json`.
133
+ From outside, pass `--hub <dir>` or set `$KHB_HUB`.
134
+
135
+ | Command | Purpose |
136
+ |---|---|
137
+ | `khb lint` | validate structure against `skills/lint/SKILL.md` |
138
+ | `khb upgrade` | refresh this hub's package-owned contract docs |
139
+ | `khb visualize` | regenerate `visualizer/graph.html` |
140
+ | `khb new-bundle <name>` | scaffold + register a bundle |
141
+ | `khb ingest [bundle]` | acquire + extract every source in `sources.yaml` → `raw/`; maintains `log.md`. No bundle named → `default`, created if absent |
142
+ | `khb export <bundle> [dest]` | standalone copy: bundle + common patterns, shareable alone |
143
+
144
+ There is no `khb catalog` command — cataloging is entirely a judgement pass.
145
+
146
+ ## Ingest, then catalog — two steps, in that order
147
+
148
+ **Ingest** (`skills/ingest/SKILL.md`) is mechanical and flat: `khb ingest <bundle>` pulls
149
+ every declared source into `raw/` as markdown with a provenance header, extracting
150
+ everything it can locally — into the named bundle, or into `default` when none is named — text, PDF, DOCX, ODT, XLSX, PPTX, images by OCR, audio and
151
+ video by whisper. Sources behind an authenticated API (Confluence, ADO, git hosts) you pull
152
+ yourself via MCP/CLI into the same `raw/` shape. Ingest never interprets content.
153
+
154
+ **Catalog** (`skills/catalog/SKILL.md`) is the judgement half, one bundle at a time: read
155
+ each `raw/` file, split it into concepts, give each OKF frontmatter, link them, register
156
+ them in `index.md`. When the runtime supports parallel agents, fan them out over the raw
157
+ files; only the orchestrating agent writes `index.md`, `log.md` and `refs.md`.
158
+
159
+ Each bundle's `log.md` is the durable ledger across both steps — rows with an empty
160
+ `curated` column are the catalog backlog, so keep it current. A raw file carrying
161
+ `quality: low` came from OCR or a transcript: distrust it, and re-read the original named in
162
+ its `source:` header when the text looks wrong.
163
+
164
+ ## Parallel work
165
+
166
+ Two agents may work two different bundles concurrently — bundles are independent by
167
+ construction. Do not have two agents write the same bundle simultaneously.
package/CLAUDE.md ADDED
@@ -0,0 +1,13 @@
1
+ # CLAUDE.md
2
+
3
+ KHB is agent-agnostic; this file is only a Claude-specific shim.
4
+
5
+ @AGENTS.md
6
+
7
+ Claude discovers the workflow adapters in `.claude/skills/`. Each adapter points to the
8
+ canonical protocol in `skills/<name>/SKILL.md`; use them for querying, ingesting,
9
+ cataloging, linting, creating, visualizing, and exporting bundles.
10
+ Note the split: `ingest` acquires and extracts (mechanical), `catalog` turns `raw/` into
11
+ concept docs (judgement, no CLI command).
12
+ Each canonical `skills/<name>/SKILL.md` is self-contained: the whole protocol is in that
13
+ one file, with no root-level companion doc to chase.
package/README.md ADDED
@@ -0,0 +1,289 @@
1
+ # knowledge-hub-builder
2
+
3
+ <p align="center">
4
+ <img src="images/demo.gif" alt="KHB creates a hub, ingests sources, routes a question to a concept, and validates the result" width="544">
5
+ </p>
6
+
7
+ **KHB (Knowledge Hub Builder)** is a local, markdown-based knowledge base maintained with
8
+ Claude Code, Codex, or another coding agent.
9
+
10
+ Knowledge is divided into **bundles** owned by a person, team, project, or client. Each
11
+ bundle can contain many topics. A small outer index routes the agent to one bundle, and the
12
+ bundle index routes it to the relevant concept documents.
13
+
14
+ KHB keeps two jobs separate:
15
+
16
+ - `khb ingest` converts source files into markdown. It is mechanical and does not call a
17
+ model.
18
+ - The agent catalogs that material into concepts, links, and indexes. This requires
19
+ judgement and has no CLI command.
20
+
21
+ Full design: [SPEC.md](SPEC.md). Agent contract: [AGENTS.md](AGENTS.md). Common workflow
22
+ questions: [FAQ](document/faq.md).
23
+
24
+ ## Quick Start
25
+
26
+ KHB requires [Bun](https://bun.sh).
27
+
28
+ ```bash
29
+ bun install -g @msareen/knowledge-hub-builder
30
+
31
+ khb init ~/my-knowledge
32
+ cd ~/my-knowledge
33
+
34
+ khb new-bundle personal "My accounts, plans, records, and reference material"
35
+ ```
36
+
37
+ Add sources to `bundles/personal/sources.yaml`:
38
+
39
+ ```yaml
40
+ sources:
41
+ - type: folder
42
+ path: /absolute/path/to/documents
43
+ - type: files
44
+ paths:
45
+ - /absolute/path/to/one.pdf
46
+ - /absolute/path/to/two.xlsx
47
+ - type: web
48
+ urls:
49
+ - https://example.com/reference
50
+ ```
51
+
52
+ Then either run the workflow directly:
53
+
54
+ ```bash
55
+ khb ingest personal
56
+ ```
57
+
58
+ or ask the agent:
59
+
60
+ > Ingest and catalog the personal bundle, then run the KHB lint.
61
+
62
+ The agent discovers the KHB workflow skills from the hub and performs the catalog pass
63
+ after ingestion.
64
+
65
+ ## How It Works
66
+
67
+ ### 1. Ingest
68
+
69
+ `khb ingest <bundle>` reads `sources.yaml` and writes extracted markdown under
70
+ `bundles/<bundle>/raw/`.
71
+
72
+ Supported without additional system tools:
73
+
74
+ - text, markdown, CSV, JSON, and YAML
75
+ - PDF and DOCX
76
+ - ODT, XLSX, and PPTX
77
+
78
+ Optional local tools add:
79
+
80
+ - OCR for images and scanned PDFs: `@hyzyla/pdfium`, `sharp`, and `tesseract.js`
81
+ - audio and video transcription: a `whisper` or `faster-whisper` executable on `PATH`
82
+
83
+ If an optional extractor is missing, KHB leaves a pending row in `log.md` and prints the
84
+ required setup. OCR packages must be installed where the KHB package resolves modules; the
85
+ CLI prints that location.
86
+
87
+ Every raw markdown file carries provenance:
88
+
89
+ ```yaml
90
+ ---
91
+ source: "/absolute/path/to/statement.pdf"
92
+ fetched: 2026-07-23T09:14:02Z
93
+ sha256: db2ee470c95d
94
+ extract_tool: "tesseract.js"
95
+ quality: low
96
+ ---
97
+ ```
98
+
99
+ `quality: low` means OCR or transcription may be inaccurate. The agent can return to the
100
+ original `source` during cataloging.
101
+
102
+ KHB also maintains `bundles/<bundle>/log.md`:
103
+
104
+ - unchanged source hashes are skipped on later runs;
105
+ - a source that moved or was renamed is recognised by its hash and its existing row is
106
+ re-pointed at the new path, keeping its `raw` file and its `curated` value — it does not
107
+ come back as a second row to catalog again;
108
+ - `raw` identifies the extracted file;
109
+ - an empty `raw` means extraction is pending;
110
+ - an empty `curated` means the raw file has not been cataloged.
111
+
112
+ Authenticated systems such as Confluence, Azure DevOps, or private git hosts remain an
113
+ agent integration boundary. Declare them in `sources.yaml`; the agent acquires them through
114
+ the available MCP server or CLI and preserves the same raw-file provenance shape.
115
+
116
+ ### 2. Catalog
117
+
118
+ Ask the agent to catalog one bundle. It reads uncataloged rows from `log.md`, turns the raw
119
+ material into concept documents, and updates the bundle index.
120
+
121
+ A concept is one markdown file with OKF frontmatter:
122
+
123
+ ```markdown
124
+ ---
125
+ type: Playbook
126
+ title: Quarterly tax filing
127
+ description: Steps and deadlines for estimated quarterly tax.
128
+ tags: [tax, recurring]
129
+ ---
130
+
131
+ # Steps
132
+
133
+ ...
134
+
135
+ # Citations
136
+
137
+ - raw/folder/tax-notes.md
138
+ ```
139
+
140
+ Only `type` is required. `title` and `description` are recommended. `tags`, when present,
141
+ must be a YAML list of strings.
142
+
143
+ The catalog workflow:
144
+
145
+ 1. Reuses the bundle's existing vocabulary and directories.
146
+ 2. Splits or merges source material by concept, not by source filename.
147
+ 3. Links related concepts inside the same bundle.
148
+ 4. Lists every concept in an `index.md`.
149
+ 5. Fills the source row's `curated` value in `log.md`.
150
+ 6. Runs `khb lint`.
151
+
152
+ When the runtime supports subagents, cataloging may process independent raw files in
153
+ parallel. The orchestrating agent remains the only writer of shared index and ledger files.
154
+
155
+ ### 3. Query
156
+
157
+ Ask a question in natural language. The query workflow follows:
158
+
159
+ ```text
160
+ outer.index.md -> bundles/<name>/index.md -> concept documents
161
+ ```
162
+
163
+ The agent answers from curated concepts, not `raw/`. For a question that genuinely spans
164
+ bundles, it follows `refs.md` and enters the second bundle through its own index.
165
+
166
+ Indexes contain routing only. Knowledge belongs in concept documents.
167
+
168
+ ## Bundles
169
+
170
+ A bundle is a unit of ownership, not a subject category. Examples:
171
+
172
+ - `personal`
173
+ - `team-payments`
174
+ - `client-acme`
175
+ - `project-atlas`
176
+
177
+ Create one only when you intend to:
178
+
179
+ ```bash
180
+ khb new-bundle team-payments "Payments team roadmap, incidents, and vendor decisions"
181
+ ```
182
+
183
+ KHB never creates, splits, or merges named bundles based on their contents. When
184
+ `khb ingest` is run without a bundle name, it uses a `default` landing bundle and creates it
185
+ if needed.
186
+
187
+ Cross-bundle relationships belong in `refs.md`; concept documents must not link directly
188
+ into another bundle.
189
+
190
+ ## Claude And Codex
191
+
192
+ KHB has one source of truth for agent behavior:
193
+
194
+ - `AGENTS.md` is the common contract. Codex loads it directly.
195
+ - `CLAUDE.md` imports `AGENTS.md` for Claude Code.
196
+ - `skills/<name>/SKILL.md` contains each canonical workflow.
197
+ - `.agents/skills/` contains Codex discovery adapters.
198
+ - `.claude/skills/` contains Claude discovery adapters.
199
+
200
+ The adapter files are small pointers, not copies of the workflows. Edit only the canonical
201
+ files under `skills/` when developing KHB.
202
+
203
+ `khb init` copies this managed contract into a hub. `khb upgrade` refreshes it without
204
+ changing `bundles/` or `outer.index.md`. `khb export` includes the same compatibility
205
+ layout in a standalone bundle export.
206
+
207
+ ## Commands
208
+
209
+ Commands can be run directly or requested through the matching agent skill.
210
+
211
+ | Command | Purpose |
212
+ |---|---|
213
+ | `khb init [dir]` | Create a hub |
214
+ | `khb upgrade` | Refresh package-owned contracts and skills |
215
+ | `khb new-bundle <name> ["scope"]` | Create and register a bundle |
216
+ | `khb ingest [bundle] [--force]` | Acquire and extract declared sources |
217
+ | `khb lint` | Validate routing, bundle structure, and OKF metadata |
218
+ | `khb visualize [--port N] [--no-open]` (aliases: `vis`, `viz`) | Serve the live bundle graph on a random free port and open it in your default browser — pan/zoomable cross-bundle map, drill into a bundle for its folder-clustered concepts, rebuild-on-refresh, exits when you close the tab |
219
+ | `khb export <bundle> [dest]` | Export one standalone bundle |
220
+
221
+ Additional ingest flags:
222
+
223
+ - `--skip-ocr`
224
+ - `--skip-audio`
225
+
226
+ Run against a hub outside the current directory with `--hub <dir>` or `$KHB_HUB`.
227
+
228
+ To update the installed package and then refresh a hub:
229
+
230
+ ```bash
231
+ bun update -g @msareen/knowledge-hub-builder
232
+ cd ~/my-knowledge
233
+ khb upgrade
234
+ ```
235
+
236
+ ## Hub Layout
237
+
238
+ ```text
239
+ khb.json
240
+ outer.index.md
241
+ AGENTS.md
242
+ CLAUDE.md
243
+ SPEC.md
244
+ skills/<name>/SKILL.md
245
+ .agents/skills/<name>/SKILL.md
246
+ .claude/skills/<name>/SKILL.md
247
+ bundles/<name>/
248
+ index.md
249
+ refs.md
250
+ sources.yaml
251
+ log.md
252
+ <group>/<concept>.md
253
+ raw/
254
+ ```
255
+
256
+ The package owns the contract files and skill directories. You own `outer.index.md` and
257
+ everything under `bundles/`.
258
+
259
+ ## Privacy
260
+
261
+ `raw/` and the extraction cache under `inbox/` are gitignored. `log.md` is committed and
262
+ records source paths, which may be absolute. Ignore `log.md` before the first commit if
263
+ those paths are sensitive.
264
+
265
+ `khb export` copies the complete bundle, including `log.md`.
266
+
267
+ KHB extraction runs locally and does not call a model. Agent cataloging and querying use
268
+ the model provider configured in Claude Code, Codex, or the active agent runtime.
269
+
270
+ ## Development
271
+
272
+ This repository is both the npm package and a small KHB hub used to exercise the tooling.
273
+
274
+ ```bash
275
+ bun install
276
+ bun run lint
277
+ bun scripts/cli.ts help
278
+ bun scripts/cli.ts init /tmp/scratch-hub
279
+ ```
280
+
281
+ `khb.json`, `bundles/`, and `outer.index.md` are excluded from the published package by the
282
+ `files` allowlist in `package.json`.
283
+
284
+ ## Lineage
285
+
286
+ KHB combines [Karpathy's LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f)
287
+ with [Google's Open Knowledge Format](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf).
288
+ KHB adds the bundle router, cross-bundle reference rules, local ingestion tooling, and
289
+ standalone bundle export.