@msareen/knowledge-hub-builder 0.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/catalog/SKILL.md +7 -0
- package/.agents/skills/export/SKILL.md +7 -0
- package/.agents/skills/ingest/SKILL.md +7 -0
- package/.agents/skills/lint/SKILL.md +7 -0
- package/.agents/skills/new-bundle/SKILL.md +7 -0
- package/.agents/skills/query/SKILL.md +7 -0
- package/.agents/skills/visualize/SKILL.md +7 -0
- package/.bundle_template/index.md +9 -0
- package/.bundle_template/log.md +10 -0
- package/.bundle_template/raw/.gitkeep +15 -0
- package/.bundle_template/refs.md +6 -0
- package/.bundle_template/sources.yaml +13 -0
- package/.claude/skills/catalog/SKILL.md +7 -0
- package/.claude/skills/export/SKILL.md +7 -0
- package/.claude/skills/ingest/SKILL.md +7 -0
- package/.claude/skills/lint/SKILL.md +7 -0
- package/.claude/skills/new-bundle/SKILL.md +7 -0
- package/.claude/skills/query/SKILL.md +7 -0
- package/.claude/skills/visualize/SKILL.md +7 -0
- package/AGENTS.md +167 -0
- package/CLAUDE.md +13 -0
- package/README.md +289 -0
- package/SPEC.md +354 -0
- package/document/faq.md +156 -0
- package/package.json +52 -0
- package/scripts/cli.ts +66 -0
- package/scripts/export.ts +42 -0
- package/scripts/ingest/acquire.ts +189 -0
- package/scripts/ingest/exts.ts +29 -0
- package/scripts/ingest/files.ts +29 -0
- package/scripts/ingest/folder.ts +44 -0
- package/scripts/ingest/index.ts +125 -0
- package/scripts/ingest/protect.ts +42 -0
- package/scripts/ingest/web.ts +54 -0
- package/scripts/init.ts +93 -0
- package/scripts/lib/args.ts +8 -0
- package/scripts/lib/extract.ts +384 -0
- package/scripts/lib/graph-page.ts +477 -0
- package/scripts/lib/graph.ts +117 -0
- package/scripts/lib/ledger.ts +136 -0
- package/scripts/lib/log.ts +53 -0
- package/scripts/lib/paths.ts +55 -0
- package/scripts/lib/scaffold.ts +56 -0
- package/scripts/lib/util.ts +154 -0
- package/scripts/lint.ts +165 -0
- package/scripts/new-bundle.ts +17 -0
- package/scripts/visualize.ts +104 -0
- package/skills/catalog/SKILL.md +164 -0
- package/skills/export/SKILL.md +31 -0
- package/skills/ingest/SKILL.md +229 -0
- package/skills/lint/SKILL.md +69 -0
- package/skills/new-bundle/SKILL.md +25 -0
- package/skills/query/SKILL.md +114 -0
- package/skills/visualize/SKILL.md +33 -0
- package/templates/hub/gitattributes +12 -0
- package/templates/hub/gitignore +12 -0
- package/templates/hub/outer.index.md +13 -0
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: catalog
|
|
3
|
+
description: Curate one KHB bundle's raw material into linked OKF concept documents. Use after ingest or when asked to catalog or organize a bundle.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Read and follow the [canonical workflow](../../../skills/catalog/SKILL.md) in full; treat
|
|
7
|
+
all of its instructions as part of this skill.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: export
|
|
3
|
+
description: Export a KHB bundle as a standalone shareable folder. Use when asked to share or ship one bundle.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Read and follow the [canonical workflow](../../../skills/export/SKILL.md) in full; treat
|
|
7
|
+
all of its instructions as part of this skill.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: ingest
|
|
3
|
+
description: Acquire external material into a KHB bundle as provenance-bearing markdown. Use when asked to add, import, pull, or refresh source data.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Read and follow the [canonical workflow](../../../skills/ingest/SKILL.md) in full; treat
|
|
7
|
+
all of its instructions as part of this skill.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: lint
|
|
3
|
+
description: Validate and repair KHB structure and OKF conformance. Use after structural edits or when asked to check the knowledge base.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Read and follow the [canonical workflow](../../../skills/lint/SKILL.md) in full; treat all
|
|
7
|
+
of its instructions as part of this skill.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: new-bundle
|
|
3
|
+
description: Create and register a KHB bundle. Use only when the user explicitly wants a new owner or knowledge area.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Read and follow the [canonical workflow](../../../skills/new-bundle/SKILL.md) in full;
|
|
7
|
+
treat all of its instructions as part of this skill.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: query
|
|
3
|
+
description: Answer a question from a KHB hub by routing through its indexes and concept documents. Use for knowledge-base lookups.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Read and follow the [canonical workflow](../../../skills/query/SKILL.md) in full; treat all
|
|
7
|
+
of its instructions as part of this skill.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: visualize
|
|
3
|
+
description: Regenerate the KHB bundle graph. Use when asked to visualize the hub or after bundle references change.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Read and follow the [canonical workflow](../../../skills/visualize/SKILL.md) in full; treat
|
|
7
|
+
all of its instructions as part of this skill.
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
---
|
|
2
|
+
okf_version: '0.1'
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
# {{name}}
|
|
6
|
+
|
|
7
|
+
<!-- OKF index: progressive disclosure. List every concept doc / subdirectory as:
|
|
8
|
+
* [Title](path.md) - one-line description (from the doc's frontmatter)
|
|
9
|
+
Organize concepts in whatever subdirectories fit the domain. -->
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
# {{name}} — ingest log
|
|
2
|
+
|
|
3
|
+
Ingestion ledger, one row per source. `khb ingest` maintains `source`, `sha256`,
|
|
4
|
+
`fetched` and `raw`; the agent fills `curated` while cataloging (skills/catalog/SKILL.md).
|
|
5
|
+
|
|
6
|
+
Empty `raw` = seen but not extracted (protected, unreadable, or skipped by a flag).
|
|
7
|
+
Empty `curated` = in raw/ but not yet distilled into a concept doc.
|
|
8
|
+
|
|
9
|
+
| source | sha256 | fetched | raw | curated |
|
|
10
|
+
|---|---|---|---|---|
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# raw/ — ingested source material, NOT canonical.
|
|
2
|
+
#
|
|
3
|
+
# Phase 1 of skills/ingest/SKILL.md drops faithful copies here, one subdirectory per source
|
|
4
|
+
# type: raw/folder/, raw/web/, raw/confluence/, raw/ado/, raw/pdf/ ...
|
|
5
|
+
# Every .md file starts with a provenance header (lint L8):
|
|
6
|
+
#
|
|
7
|
+
# ---
|
|
8
|
+
# source: <url | path | tool query>
|
|
9
|
+
# fetched: <ISO timestamp>
|
|
10
|
+
# ---
|
|
11
|
+
#
|
|
12
|
+
# Re-acquiring overwrites this tree idempotently. It is gitignored
|
|
13
|
+
# (bundles/*/raw/) and never cited in answers — curate into concept docs and
|
|
14
|
+
# cite those instead. This file only keeps the directory present in the
|
|
15
|
+
# template; delete it once the bundle has real raw content.
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
# Ingestion sources for {{name}}.
|
|
2
|
+
# folder/web: scripted — bun run ingest {{name}}
|
|
3
|
+
# confluence/ado: listed as provenance; agent ingests via MCP/CLI (AGENTS.md §Ingestion)
|
|
4
|
+
sources: []
|
|
5
|
+
# - type: folder
|
|
6
|
+
# path: /abs/path/to/some-folder
|
|
7
|
+
# - type: web
|
|
8
|
+
# urls: []
|
|
9
|
+
# - type: confluence
|
|
10
|
+
# space: SPACEKEY
|
|
11
|
+
# - type: ado
|
|
12
|
+
# org: myorg
|
|
13
|
+
# project: MyProject
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: catalog
|
|
3
|
+
description: Curate one KHB bundle's raw material into linked OKF concept documents. Use after ingest or when asked to catalog or organize a bundle.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Read and follow the [canonical workflow](../../../skills/catalog/SKILL.md) in full; treat
|
|
7
|
+
all of its instructions as part of this skill.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: export
|
|
3
|
+
description: Export a KHB bundle as a standalone shareable folder. Use when asked to share or ship one bundle.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Read and follow the [canonical workflow](../../../skills/export/SKILL.md) in full; treat
|
|
7
|
+
all of its instructions as part of this skill.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: ingest
|
|
3
|
+
description: Acquire external material into a KHB bundle as provenance-bearing markdown. Use when asked to add, import, pull, or refresh source data.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Read and follow the [canonical workflow](../../../skills/ingest/SKILL.md) in full; treat
|
|
7
|
+
all of its instructions as part of this skill.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: lint
|
|
3
|
+
description: Validate and repair KHB structure and OKF conformance. Use after structural edits or when asked to check the knowledge base.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Read and follow the [canonical workflow](../../../skills/lint/SKILL.md) in full; treat all
|
|
7
|
+
of its instructions as part of this skill.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: new-bundle
|
|
3
|
+
description: Create and register a KHB bundle. Use only when the user explicitly wants a new owner or knowledge area.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Read and follow the [canonical workflow](../../../skills/new-bundle/SKILL.md) in full;
|
|
7
|
+
treat all of its instructions as part of this skill.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: query
|
|
3
|
+
description: Answer a question from a KHB hub by routing through its indexes and concept documents. Use for knowledge-base lookups.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Read and follow the [canonical workflow](../../../skills/query/SKILL.md) in full; treat all
|
|
7
|
+
of its instructions as part of this skill.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: visualize
|
|
3
|
+
description: Regenerate the KHB bundle graph. Use when asked to visualize the hub or after bundle references change.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Read and follow the [canonical workflow](../../../skills/visualize/SKILL.md) in full; treat
|
|
7
|
+
all of its instructions as part of this skill.
|
package/AGENTS.md
ADDED
|
@@ -0,0 +1,167 @@
|
|
|
1
|
+
# AGENTS.md — common contract for Claude, Codex, and other agents
|
|
2
|
+
|
|
3
|
+
You are inside a **KHB hub** — a bundle-of-bundles knowledge system. Full design: `SPEC.md`.
|
|
4
|
+
|
|
5
|
+
The hub is this folder: `khb.json`, `outer.index.md`, and `bundles/`. The knowledge is
|
|
6
|
+
here; the `khb` tooling is installed separately and holds none of it. This file and its
|
|
7
|
+
siblings (`SPEC.md`, `skills/`) are package-owned copies refreshed by `khb upgrade` —
|
|
8
|
+
never edit them, edit bundle content instead. Every workflow protocol lives wholly inside
|
|
9
|
+
`skills/<name>/SKILL.md`: query, ingest, catalog, lint, new-bundle, export, visualize.
|
|
10
|
+
Those files are plain markdown — any agent can read one directly, whether or not its
|
|
11
|
+
runtime has a notion of "skills".
|
|
12
|
+
|
|
13
|
+
## What a bundle is
|
|
14
|
+
|
|
15
|
+
A bundle is a **logical unit defined by its owner** — a person, a team, a project, a client
|
|
16
|
+
— not a subject classification. It holds as many topics as its owner has; topics are
|
|
17
|
+
organized inside it with subdirectories. A bundle whose contents look heterogeneous is
|
|
18
|
+
working as intended.
|
|
19
|
+
|
|
20
|
+
Creating, splitting or merging bundles is a human decision, always. Never do it on your own
|
|
21
|
+
initiative, in any workflow, however obviously a subject seems to want its own home. On
|
|
22
|
+
explicit instruction, carve one out and move the files; otherwise leave the shape alone.
|
|
23
|
+
|
|
24
|
+
## How to navigate
|
|
25
|
+
|
|
26
|
+
1. Start at `outer.index.md`. Pick exactly one bundle for the question. Do not browse.
|
|
27
|
+
2. If that doesn't settle it, escalate per `skills/query/SKILL.md`: grep the bundle
|
|
28
|
+
`index.md` files, then concept front matter, then **ask the user** which bundle.
|
|
29
|
+
Never guess silently.
|
|
30
|
+
3. Enter the bundle via its `index.md`, then read only the concepts/notes it routes you to.
|
|
31
|
+
|
|
32
|
+
This file is the single common contract — bundles carry no per-bundle agent rules.
|
|
33
|
+
Claude loads it through `CLAUDE.md`; Codex loads it directly.
|
|
34
|
+
|
|
35
|
+
## Hard rules
|
|
36
|
+
|
|
37
|
+
- **One bundle at a time.** If a question spans two bundles, follow the collation protocol
|
|
38
|
+
in `skills/query/SKILL.md`: resolve in the first bundle, follow its `refs.md` to the
|
|
39
|
+
second, and enter the second **through its own index.md**. Never deep-link from one bundle's notes into
|
|
40
|
+
another's. Join answers in your response, not in the files.
|
|
41
|
+
- **Indexes are routing only.** Never add knowledge content to `outer.index.md` or any
|
|
42
|
+
`index.md` — add a concept doc and link it from the index.
|
|
43
|
+
- **Writing (OKF v0.1):** a concept = one markdown file, anywhere in the bundle except
|
|
44
|
+
reserved names (`index.md`, `log.md`, `refs.md`) and `raw/`. Group concepts in
|
|
45
|
+
subdirectories that fit the domain (`tables/`, `metrics/`, `notes/`, …). Every
|
|
46
|
+
concept doc starts with front matter — `type` is required:
|
|
47
|
+
|
|
48
|
+
```yaml
|
|
49
|
+
---
|
|
50
|
+
type: Metric # free-form: Table, Playbook, Decision Log, Reference, …
|
|
51
|
+
title: Display name
|
|
52
|
+
description: One-line summary (reused by index generators)
|
|
53
|
+
resource: <canonical URI of the underlying asset, if any>
|
|
54
|
+
tags: [a, b]
|
|
55
|
+
timestamp: 2026-07-19T00:00:00Z
|
|
56
|
+
---
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
List every concept in an `index.md` (`* [Title](path.md) - description`). Link
|
|
60
|
+
concepts to each other with plain markdown links, `/path/from/bundle/root.md`
|
|
61
|
+
preferred; relationship meaning goes in the surrounding prose.
|
|
62
|
+
- **Cross-bundle relationships** go in `refs.md` only — never inline links across bundles.
|
|
63
|
+
- **Decisions:** if this hub has a `meta` bundle, any change to how the hub itself works
|
|
64
|
+
gets a dated line in `bundles/meta/notes/decisions.md`.
|
|
65
|
+
- **`raw/` is not canonical.** It's ingested source material awaiting curation. Cite
|
|
66
|
+
concept docs; use `raw/` only when curating.
|
|
67
|
+
- **New bundle:** `khb new-bundle <name>` — never hand-copy `.bundle_template`.
|
|
68
|
+
- After structural edits run `khb lint` and fix what it reports.
|
|
69
|
+
|
|
70
|
+
## Division of labor — khb vs agent
|
|
71
|
+
|
|
72
|
+
One hard boundary governs every workflow and every future change to the tooling:
|
|
73
|
+
|
|
74
|
+
- **`khb` converts bytes to text.** Hashing, caching, file plumbing, ledger-keeping, and
|
|
75
|
+
*every* local extractor: PDF/DOCX/ODT/XLSX/PPTX libraries, tesseract OCR for scans and
|
|
76
|
+
images, whisper for audio and video. All of it deterministic, offline, and free of
|
|
77
|
+
charge. It **never contacts a model** — not directly, not by shelling out.
|
|
78
|
+
- **The agent decides what the text means.** Splitting a document into concepts, labeling
|
|
79
|
+
and linking them, curating `raw/` into the wiki, escalating a bad OCR to a vision read,
|
|
80
|
+
and judging when a query has produced a new concept worth keeping.
|
|
81
|
+
|
|
82
|
+
The line is *conversion vs. interpretation*, not cheap vs. expensive: tesseract and whisper
|
|
83
|
+
belong in `khb` despite costing real CPU, because their output is reproducible and needs no
|
|
84
|
+
judgement. The split exists so every intelligent decision is auditable — it leaves a
|
|
85
|
+
provenance header and a `log.md` row. When extending khb, keep conversion in the CLI and
|
|
86
|
+
interpretation in an agent pass. Never add a `khb … --auto-label` / `--summarize` flag that
|
|
87
|
+
calls a model; that rots the boundary.
|
|
88
|
+
|
|
89
|
+
## Where skills live — one canonical file, thin adapters
|
|
90
|
+
|
|
91
|
+
A skill exists **once**, in `skills/<name>/SKILL.md`, and that one file carries the entire
|
|
92
|
+
protocol. Runtimes that have a notion of "skills" discover it through a parallel adapter
|
|
93
|
+
that only points at it:
|
|
94
|
+
|
|
95
|
+
```
|
|
96
|
+
skills/<name>/SKILL.md ← canonical: the whole protocol
|
|
97
|
+
.claude/skills/<name>/SKILL.md ← adapter: frontmatter + a link
|
|
98
|
+
.agents/skills/<name>/SKILL.md ← adapter: frontmatter + a link
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
Each adapter is the same handful of lines — nothing else belongs in one:
|
|
102
|
+
|
|
103
|
+
```markdown
|
|
104
|
+
---
|
|
105
|
+
name: <name>
|
|
106
|
+
description: <one line; this is what the runtime matches on when deciding to load the skill>
|
|
107
|
+
---
|
|
108
|
+
|
|
109
|
+
Read and follow the [canonical workflow](../../../skills/<name>/SKILL.md) in full; treat all
|
|
110
|
+
of its instructions as part of this skill.
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
The `../../../` climbs from `.claude/skills/<name>/` back to the hub root. The two adapters
|
|
114
|
+
are byte-identical to each other; only the canonical file differs from them.
|
|
115
|
+
|
|
116
|
+
**Adding a hub-specific skill** follows the same shape — create all three files, keep
|
|
117
|
+
`<name>` identical across them, and put every instruction in the canonical file only. The
|
|
118
|
+
adapter `description` is a triggering surface, so write it as *when to reach for this*, not
|
|
119
|
+
as a summary of what it does; the canonical file's own frontmatter can be longer and more
|
|
120
|
+
precise, since it is read after the decision to load has been made.
|
|
121
|
+
|
|
122
|
+
Never let protocol text drift into an adapter: two copies of a workflow means an agent can
|
|
123
|
+
load the stale one and have no way to tell. If a runtime needs something the link cannot
|
|
124
|
+
express, extend the canonical file and keep the adapter a pointer.
|
|
125
|
+
|
|
126
|
+
`khb upgrade` refreshes all three trees, so the built-in skills are package-owned — but it
|
|
127
|
+
overwrites paths it ships and does not delete unknown ones, so a skill you add to a hub
|
|
128
|
+
under a name khb does not use survives upgrades intact.
|
|
129
|
+
|
|
130
|
+
## Tooling
|
|
131
|
+
|
|
132
|
+
Run `khb` from anywhere inside the hub — it finds the hub by walking up to `khb.json`.
|
|
133
|
+
From outside, pass `--hub <dir>` or set `$KHB_HUB`.
|
|
134
|
+
|
|
135
|
+
| Command | Purpose |
|
|
136
|
+
|---|---|
|
|
137
|
+
| `khb lint` | validate structure against `skills/lint/SKILL.md` |
|
|
138
|
+
| `khb upgrade` | refresh this hub's package-owned contract docs |
|
|
139
|
+
| `khb visualize` | regenerate `visualizer/graph.html` |
|
|
140
|
+
| `khb new-bundle <name>` | scaffold + register a bundle |
|
|
141
|
+
| `khb ingest [bundle]` | acquire + extract every source in `sources.yaml` → `raw/`; maintains `log.md`. No bundle named → `default`, created if absent |
|
|
142
|
+
| `khb export <bundle> [dest]` | standalone copy: bundle + common patterns, shareable alone |
|
|
143
|
+
|
|
144
|
+
There is no `khb catalog` command — cataloging is entirely a judgement pass.
|
|
145
|
+
|
|
146
|
+
## Ingest, then catalog — two steps, in that order
|
|
147
|
+
|
|
148
|
+
**Ingest** (`skills/ingest/SKILL.md`) is mechanical and flat: `khb ingest <bundle>` pulls
|
|
149
|
+
every declared source into `raw/` as markdown with a provenance header, extracting
|
|
150
|
+
everything it can locally — into the named bundle, or into `default` when none is named — text, PDF, DOCX, ODT, XLSX, PPTX, images by OCR, audio and
|
|
151
|
+
video by whisper. Sources behind an authenticated API (Confluence, ADO, git hosts) you pull
|
|
152
|
+
yourself via MCP/CLI into the same `raw/` shape. Ingest never interprets content.
|
|
153
|
+
|
|
154
|
+
**Catalog** (`skills/catalog/SKILL.md`) is the judgement half, one bundle at a time: read
|
|
155
|
+
each `raw/` file, split it into concepts, give each OKF frontmatter, link them, register
|
|
156
|
+
them in `index.md`. When the runtime supports parallel agents, fan them out over the raw
|
|
157
|
+
files; only the orchestrating agent writes `index.md`, `log.md` and `refs.md`.
|
|
158
|
+
|
|
159
|
+
Each bundle's `log.md` is the durable ledger across both steps — rows with an empty
|
|
160
|
+
`curated` column are the catalog backlog, so keep it current. A raw file carrying
|
|
161
|
+
`quality: low` came from OCR or a transcript: distrust it, and re-read the original named in
|
|
162
|
+
its `source:` header when the text looks wrong.
|
|
163
|
+
|
|
164
|
+
## Parallel work
|
|
165
|
+
|
|
166
|
+
Two agents may work two different bundles concurrently — bundles are independent by
|
|
167
|
+
construction. Do not have two agents write the same bundle simultaneously.
|
package/CLAUDE.md
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
# CLAUDE.md
|
|
2
|
+
|
|
3
|
+
KHB is agent-agnostic; this file is only a Claude-specific shim.
|
|
4
|
+
|
|
5
|
+
@AGENTS.md
|
|
6
|
+
|
|
7
|
+
Claude discovers the workflow adapters in `.claude/skills/`. Each adapter points to the
|
|
8
|
+
canonical protocol in `skills/<name>/SKILL.md`; use them for querying, ingesting,
|
|
9
|
+
cataloging, linting, creating, visualizing, and exporting bundles.
|
|
10
|
+
Note the split: `ingest` acquires and extracts (mechanical), `catalog` turns `raw/` into
|
|
11
|
+
concept docs (judgement, no CLI command).
|
|
12
|
+
Each canonical `skills/<name>/SKILL.md` is self-contained: the whole protocol is in that
|
|
13
|
+
one file, with no root-level companion doc to chase.
|
package/README.md
ADDED
|
@@ -0,0 +1,289 @@
|
|
|
1
|
+
# knowledge-hub-builder
|
|
2
|
+
|
|
3
|
+
<p align="center">
|
|
4
|
+
<img src="images/demo.gif" alt="KHB creates a hub, ingests sources, routes a question to a concept, and validates the result" width="544">
|
|
5
|
+
</p>
|
|
6
|
+
|
|
7
|
+
**KHB (Knowledge Hub Builder)** is a local, markdown-based knowledge base maintained with
|
|
8
|
+
Claude Code, Codex, or another coding agent.
|
|
9
|
+
|
|
10
|
+
Knowledge is divided into **bundles** owned by a person, team, project, or client. Each
|
|
11
|
+
bundle can contain many topics. A small outer index routes the agent to one bundle, and the
|
|
12
|
+
bundle index routes it to the relevant concept documents.
|
|
13
|
+
|
|
14
|
+
KHB keeps two jobs separate:
|
|
15
|
+
|
|
16
|
+
- `khb ingest` converts source files into markdown. It is mechanical and does not call a
|
|
17
|
+
model.
|
|
18
|
+
- The agent catalogs that material into concepts, links, and indexes. This requires
|
|
19
|
+
judgement and has no CLI command.
|
|
20
|
+
|
|
21
|
+
Full design: [SPEC.md](SPEC.md). Agent contract: [AGENTS.md](AGENTS.md). Common workflow
|
|
22
|
+
questions: [FAQ](document/faq.md).
|
|
23
|
+
|
|
24
|
+
## Quick Start
|
|
25
|
+
|
|
26
|
+
KHB requires [Bun](https://bun.sh).
|
|
27
|
+
|
|
28
|
+
```bash
|
|
29
|
+
bun install -g @msareen/knowledge-hub-builder
|
|
30
|
+
|
|
31
|
+
khb init ~/my-knowledge
|
|
32
|
+
cd ~/my-knowledge
|
|
33
|
+
|
|
34
|
+
khb new-bundle personal "My accounts, plans, records, and reference material"
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
Add sources to `bundles/personal/sources.yaml`:
|
|
38
|
+
|
|
39
|
+
```yaml
|
|
40
|
+
sources:
|
|
41
|
+
- type: folder
|
|
42
|
+
path: /absolute/path/to/documents
|
|
43
|
+
- type: files
|
|
44
|
+
paths:
|
|
45
|
+
- /absolute/path/to/one.pdf
|
|
46
|
+
- /absolute/path/to/two.xlsx
|
|
47
|
+
- type: web
|
|
48
|
+
urls:
|
|
49
|
+
- https://example.com/reference
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Then either run the workflow directly:
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
khb ingest personal
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
or ask the agent:
|
|
59
|
+
|
|
60
|
+
> Ingest and catalog the personal bundle, then run the KHB lint.
|
|
61
|
+
|
|
62
|
+
The agent discovers the KHB workflow skills from the hub and performs the catalog pass
|
|
63
|
+
after ingestion.
|
|
64
|
+
|
|
65
|
+
## How It Works
|
|
66
|
+
|
|
67
|
+
### 1. Ingest
|
|
68
|
+
|
|
69
|
+
`khb ingest <bundle>` reads `sources.yaml` and writes extracted markdown under
|
|
70
|
+
`bundles/<bundle>/raw/`.
|
|
71
|
+
|
|
72
|
+
Supported without additional system tools:
|
|
73
|
+
|
|
74
|
+
- text, markdown, CSV, JSON, and YAML
|
|
75
|
+
- PDF and DOCX
|
|
76
|
+
- ODT, XLSX, and PPTX
|
|
77
|
+
|
|
78
|
+
Optional local tools add:
|
|
79
|
+
|
|
80
|
+
- OCR for images and scanned PDFs: `@hyzyla/pdfium`, `sharp`, and `tesseract.js`
|
|
81
|
+
- audio and video transcription: a `whisper` or `faster-whisper` executable on `PATH`
|
|
82
|
+
|
|
83
|
+
If an optional extractor is missing, KHB leaves a pending row in `log.md` and prints the
|
|
84
|
+
required setup. OCR packages must be installed where the KHB package resolves modules; the
|
|
85
|
+
CLI prints that location.
|
|
86
|
+
|
|
87
|
+
Every raw markdown file carries provenance:
|
|
88
|
+
|
|
89
|
+
```yaml
|
|
90
|
+
---
|
|
91
|
+
source: "/absolute/path/to/statement.pdf"
|
|
92
|
+
fetched: 2026-07-23T09:14:02Z
|
|
93
|
+
sha256: db2ee470c95d
|
|
94
|
+
extract_tool: "tesseract.js"
|
|
95
|
+
quality: low
|
|
96
|
+
---
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
`quality: low` means OCR or transcription may be inaccurate. The agent can return to the
|
|
100
|
+
original `source` during cataloging.
|
|
101
|
+
|
|
102
|
+
KHB also maintains `bundles/<bundle>/log.md`:
|
|
103
|
+
|
|
104
|
+
- unchanged source hashes are skipped on later runs;
|
|
105
|
+
- a source that moved or was renamed is recognised by its hash and its existing row is
|
|
106
|
+
re-pointed at the new path, keeping its `raw` file and its `curated` value — it does not
|
|
107
|
+
come back as a second row to catalog again;
|
|
108
|
+
- `raw` identifies the extracted file;
|
|
109
|
+
- an empty `raw` means extraction is pending;
|
|
110
|
+
- an empty `curated` means the raw file has not been cataloged.
|
|
111
|
+
|
|
112
|
+
Authenticated systems such as Confluence, Azure DevOps, or private git hosts remain an
|
|
113
|
+
agent integration boundary. Declare them in `sources.yaml`; the agent acquires them through
|
|
114
|
+
the available MCP server or CLI and preserves the same raw-file provenance shape.
|
|
115
|
+
|
|
116
|
+
### 2. Catalog
|
|
117
|
+
|
|
118
|
+
Ask the agent to catalog one bundle. It reads uncataloged rows from `log.md`, turns the raw
|
|
119
|
+
material into concept documents, and updates the bundle index.
|
|
120
|
+
|
|
121
|
+
A concept is one markdown file with OKF frontmatter:
|
|
122
|
+
|
|
123
|
+
```markdown
|
|
124
|
+
---
|
|
125
|
+
type: Playbook
|
|
126
|
+
title: Quarterly tax filing
|
|
127
|
+
description: Steps and deadlines for estimated quarterly tax.
|
|
128
|
+
tags: [tax, recurring]
|
|
129
|
+
---
|
|
130
|
+
|
|
131
|
+
# Steps
|
|
132
|
+
|
|
133
|
+
...
|
|
134
|
+
|
|
135
|
+
# Citations
|
|
136
|
+
|
|
137
|
+
- raw/folder/tax-notes.md
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
Only `type` is required. `title` and `description` are recommended. `tags`, when present,
|
|
141
|
+
must be a YAML list of strings.
|
|
142
|
+
|
|
143
|
+
The catalog workflow:
|
|
144
|
+
|
|
145
|
+
1. Reuses the bundle's existing vocabulary and directories.
|
|
146
|
+
2. Splits or merges source material by concept, not by source filename.
|
|
147
|
+
3. Links related concepts inside the same bundle.
|
|
148
|
+
4. Lists every concept in an `index.md`.
|
|
149
|
+
5. Fills the source row's `curated` value in `log.md`.
|
|
150
|
+
6. Runs `khb lint`.
|
|
151
|
+
|
|
152
|
+
When the runtime supports subagents, cataloging may process independent raw files in
|
|
153
|
+
parallel. The orchestrating agent remains the only writer of shared index and ledger files.
|
|
154
|
+
|
|
155
|
+
### 3. Query
|
|
156
|
+
|
|
157
|
+
Ask a question in natural language. The query workflow follows:
|
|
158
|
+
|
|
159
|
+
```text
|
|
160
|
+
outer.index.md -> bundles/<name>/index.md -> concept documents
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
The agent answers from curated concepts, not `raw/`. For a question that genuinely spans
|
|
164
|
+
bundles, it follows `refs.md` and enters the second bundle through its own index.
|
|
165
|
+
|
|
166
|
+
Indexes contain routing only. Knowledge belongs in concept documents.
|
|
167
|
+
|
|
168
|
+
## Bundles
|
|
169
|
+
|
|
170
|
+
A bundle is a unit of ownership, not a subject category. Examples:
|
|
171
|
+
|
|
172
|
+
- `personal`
|
|
173
|
+
- `team-payments`
|
|
174
|
+
- `client-acme`
|
|
175
|
+
- `project-atlas`
|
|
176
|
+
|
|
177
|
+
Create one only when you intend to:
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
khb new-bundle team-payments "Payments team roadmap, incidents, and vendor decisions"
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
KHB never creates, splits, or merges named bundles based on their contents. When
|
|
184
|
+
`khb ingest` is run without a bundle name, it uses a `default` landing bundle and creates it
|
|
185
|
+
if needed.
|
|
186
|
+
|
|
187
|
+
Cross-bundle relationships belong in `refs.md`; concept documents must not link directly
|
|
188
|
+
into another bundle.
|
|
189
|
+
|
|
190
|
+
## Claude And Codex
|
|
191
|
+
|
|
192
|
+
KHB has one source of truth for agent behavior:
|
|
193
|
+
|
|
194
|
+
- `AGENTS.md` is the common contract. Codex loads it directly.
|
|
195
|
+
- `CLAUDE.md` imports `AGENTS.md` for Claude Code.
|
|
196
|
+
- `skills/<name>/SKILL.md` contains each canonical workflow.
|
|
197
|
+
- `.agents/skills/` contains Codex discovery adapters.
|
|
198
|
+
- `.claude/skills/` contains Claude discovery adapters.
|
|
199
|
+
|
|
200
|
+
The adapter files are small pointers, not copies of the workflows. Edit only the canonical
|
|
201
|
+
files under `skills/` when developing KHB.
|
|
202
|
+
|
|
203
|
+
`khb init` copies this managed contract into a hub. `khb upgrade` refreshes it without
|
|
204
|
+
changing `bundles/` or `outer.index.md`. `khb export` includes the same compatibility
|
|
205
|
+
layout in a standalone bundle export.
|
|
206
|
+
|
|
207
|
+
## Commands
|
|
208
|
+
|
|
209
|
+
Commands can be run directly or requested through the matching agent skill.
|
|
210
|
+
|
|
211
|
+
| Command | Purpose |
|
|
212
|
+
|---|---|
|
|
213
|
+
| `khb init [dir]` | Create a hub |
|
|
214
|
+
| `khb upgrade` | Refresh package-owned contracts and skills |
|
|
215
|
+
| `khb new-bundle <name> ["scope"]` | Create and register a bundle |
|
|
216
|
+
| `khb ingest [bundle] [--force]` | Acquire and extract declared sources |
|
|
217
|
+
| `khb lint` | Validate routing, bundle structure, and OKF metadata |
|
|
218
|
+
| `khb visualize [--port N] [--no-open]` (aliases: `vis`, `viz`) | Serve the live bundle graph on a random free port and open it in your default browser — pan/zoomable cross-bundle map, drill into a bundle for its folder-clustered concepts, rebuild-on-refresh, exits when you close the tab |
|
|
219
|
+
| `khb export <bundle> [dest]` | Export one standalone bundle |
|
|
220
|
+
|
|
221
|
+
Additional ingest flags:
|
|
222
|
+
|
|
223
|
+
- `--skip-ocr`
|
|
224
|
+
- `--skip-audio`
|
|
225
|
+
|
|
226
|
+
Run against a hub outside the current directory with `--hub <dir>` or `$KHB_HUB`.
|
|
227
|
+
|
|
228
|
+
To update the installed package and then refresh a hub:
|
|
229
|
+
|
|
230
|
+
```bash
|
|
231
|
+
bun update -g @msareen/knowledge-hub-builder
|
|
232
|
+
cd ~/my-knowledge
|
|
233
|
+
khb upgrade
|
|
234
|
+
```
|
|
235
|
+
|
|
236
|
+
## Hub Layout
|
|
237
|
+
|
|
238
|
+
```text
|
|
239
|
+
khb.json
|
|
240
|
+
outer.index.md
|
|
241
|
+
AGENTS.md
|
|
242
|
+
CLAUDE.md
|
|
243
|
+
SPEC.md
|
|
244
|
+
skills/<name>/SKILL.md
|
|
245
|
+
.agents/skills/<name>/SKILL.md
|
|
246
|
+
.claude/skills/<name>/SKILL.md
|
|
247
|
+
bundles/<name>/
|
|
248
|
+
index.md
|
|
249
|
+
refs.md
|
|
250
|
+
sources.yaml
|
|
251
|
+
log.md
|
|
252
|
+
<group>/<concept>.md
|
|
253
|
+
raw/
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
The package owns the contract files and skill directories. You own `outer.index.md` and
|
|
257
|
+
everything under `bundles/`.
|
|
258
|
+
|
|
259
|
+
## Privacy
|
|
260
|
+
|
|
261
|
+
`raw/` and the extraction cache under `inbox/` are gitignored. `log.md` is committed and
|
|
262
|
+
records source paths, which may be absolute. Ignore `log.md` before the first commit if
|
|
263
|
+
those paths are sensitive.
|
|
264
|
+
|
|
265
|
+
`khb export` copies the complete bundle, including `log.md`.
|
|
266
|
+
|
|
267
|
+
KHB extraction runs locally and does not call a model. Agent cataloging and querying use
|
|
268
|
+
the model provider configured in Claude Code, Codex, or the active agent runtime.
|
|
269
|
+
|
|
270
|
+
## Development
|
|
271
|
+
|
|
272
|
+
This repository is both the npm package and a small KHB hub used to exercise the tooling.
|
|
273
|
+
|
|
274
|
+
```bash
|
|
275
|
+
bun install
|
|
276
|
+
bun run lint
|
|
277
|
+
bun scripts/cli.ts help
|
|
278
|
+
bun scripts/cli.ts init /tmp/scratch-hub
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
`khb.json`, `bundles/`, and `outer.index.md` are excluded from the published package by the
|
|
282
|
+
`files` allowlist in `package.json`.
|
|
283
|
+
|
|
284
|
+
## Lineage
|
|
285
|
+
|
|
286
|
+
KHB combines [Karpathy's LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f)
|
|
287
|
+
with [Google's Open Knowledge Format](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf).
|
|
288
|
+
KHB adds the bundle router, cross-bundle reference rules, local ingestion tooling, and
|
|
289
|
+
standalone bundle export.
|