@msareen/knowledge-hub-builder 0.1.3 → 0.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +10 -3
- package/README.md +18 -7
- package/SPEC.md +24 -7
- package/document/faq.md +6 -2
- package/package.json +1 -1
- package/scripts/cli.ts +24 -2
- package/scripts/ingest/index.ts +18 -5
- package/scripts/init.ts +10 -50
- package/scripts/lib/paths.ts +21 -1
- package/scripts/lib/scaffold.ts +30 -17
- package/scripts/lib/upgrade.ts +93 -0
- package/scripts/lib/util.ts +6 -14
- package/skills/ingest/SKILL.md +59 -42
package/AGENTS.md
CHANGED
|
@@ -135,10 +135,10 @@ From outside, pass `--hub <dir>` or set `$KHB_HUB`.
|
|
|
135
135
|
| Command | Purpose |
|
|
136
136
|
|---|---|
|
|
137
137
|
| `khb lint` | validate structure against `skills/lint/SKILL.md` |
|
|
138
|
-
| `khb upgrade` | refresh this hub's package-owned contract docs |
|
|
138
|
+
| `khb upgrade` | refresh this hub's package-owned contract docs — runs by itself whenever `khb.json`'s stamped version differs from the installed khb, so these docs always match the CLI |
|
|
139
139
|
| `khb visualize` | regenerate `visualizer/graph.html` |
|
|
140
140
|
| `khb new-bundle <name>` | scaffold + register a bundle |
|
|
141
|
-
| `khb ingest
|
|
141
|
+
| `khb ingest <bundle>` | acquire + extract every source in `sources.yaml` → `raw/`; maintains `log.md`. Name the bundle — with none it lists the hub's bundles and stops, unless there is nothing to choose between (no bundles, or only `default`), where it uses `default` |
|
|
142
142
|
| `khb export <bundle> [dest]` | standalone copy: bundle + common patterns, shareable alone |
|
|
143
143
|
|
|
144
144
|
There is no `khb catalog` command — cataloging is entirely a judgement pass.
|
|
@@ -147,10 +147,17 @@ There is no `khb catalog` command — cataloging is entirely a judgement pass.
|
|
|
147
147
|
|
|
148
148
|
**Ingest** (`skills/ingest/SKILL.md`) is mechanical and flat: `khb ingest <bundle>` pulls
|
|
149
149
|
every declared source into `raw/` as markdown with a provenance header, extracting
|
|
150
|
-
everything it can locally —
|
|
150
|
+
everything it can locally — text, PDF, DOCX, ODT, XLSX, PPTX, images by OCR, audio and
|
|
151
151
|
video by whisper. Sources behind an authenticated API (Confluence, ADO, git hosts) you pull
|
|
152
152
|
yourself via MCP/CLI into the same `raw/` shape. Ingest never interprets content.
|
|
153
153
|
|
|
154
|
+
When the user has not named a bundle, **ask** which existing bundle owns the material or
|
|
155
|
+
whether to start a new one, and for an existing bundle ask whether to re-ingest what its
|
|
156
|
+
`sources.yaml` already declares or to take a new path. Never choose a destination silently.
|
|
157
|
+
The single exception is a hub with nothing to choose between: with no bundles at all an
|
|
158
|
+
unnamed ingest lands in `default`, created on the spot, and where `default` is the only
|
|
159
|
+
bundle it lands there as it stands.
|
|
160
|
+
|
|
154
161
|
**Catalog** (`skills/catalog/SKILL.md`) is the judgement half, one bundle at a time: read
|
|
155
162
|
each `raw/` file, split it into concepts, give each OKF frontmatter, link them, register
|
|
156
163
|
them in `index.md`. When the runtime supports parallel agents, fan them out over the raw
|
package/README.md
CHANGED
|
@@ -180,9 +180,13 @@ Create one only when you intend to:
|
|
|
180
180
|
khb new-bundle team-payments "Payments team roadmap, incidents, and vendor decisions"
|
|
181
181
|
```
|
|
182
182
|
|
|
183
|
-
KHB never creates, splits, or merges
|
|
184
|
-
`khb ingest`
|
|
185
|
-
|
|
183
|
+
KHB never creates, splits, or merges bundles based on their contents. Name the bundle when
|
|
184
|
+
you ingest: run `khb ingest` without one and it lists the hub's bundles and stops. Ask an
|
|
185
|
+
agent to ingest without naming a bundle and it asks you — existing bundle or new one, and for
|
|
186
|
+
an existing one whether to reuse the paths in its `sources.yaml` or take a new path. An
|
|
187
|
+
unnamed ingest lands somewhere by itself only when there is nothing to choose between: a hub
|
|
188
|
+
with no bundles gets a `default` bundle created on the spot, and a hub whose only bundle is
|
|
189
|
+
`default` uses it. Material moves out of `default` when you say who owns it.
|
|
186
190
|
|
|
187
191
|
Cross-bundle relationships belong in `refs.md`; concept documents must not link directly
|
|
188
192
|
into another bundle.
|
|
@@ -204,6 +208,12 @@ files under `skills/` when developing KHB.
|
|
|
204
208
|
changing `bundles/` or `outer.index.md`. `khb export` includes the same compatibility
|
|
205
209
|
layout in a standalone bundle export.
|
|
206
210
|
|
|
211
|
+
The refresh is also automatic: every command that works on a hub first compares the `khb`
|
|
212
|
+
version stamped in `khb.json` against the installed package, and upgrades the hub in place
|
|
213
|
+
if they differ, printing one line to stderr. So updating the package is enough — the hub's
|
|
214
|
+
contract docs can never be a version behind the CLI acting on them. Set
|
|
215
|
+
`KHB_NO_AUTO_UPGRADE=1` to suppress the check and leave the hub as it is.
|
|
216
|
+
|
|
207
217
|
## Commands
|
|
208
218
|
|
|
209
219
|
Commands can be run directly or requested through the matching agent skill.
|
|
@@ -211,7 +221,7 @@ Commands can be run directly or requested through the matching agent skill.
|
|
|
211
221
|
| Command | Purpose |
|
|
212
222
|
|---|---|
|
|
213
223
|
| `khb init [dir]` | Create a hub |
|
|
214
|
-
| `khb upgrade` | Refresh package-owned contracts and skills |
|
|
224
|
+
| `khb upgrade` | Refresh package-owned contracts and skills (also runs automatically on version drift) |
|
|
215
225
|
| `khb new-bundle <name> ["scope"]` | Create and register a bundle |
|
|
216
226
|
| `khb ingest [bundle] [--force]` | Acquire and extract declared sources |
|
|
217
227
|
| `khb lint` | Validate routing, bundle structure, and OKF metadata |
|
|
@@ -225,14 +235,15 @@ Additional ingest flags:
|
|
|
225
235
|
|
|
226
236
|
Run against a hub outside the current directory with `--hub <dir>` or `$KHB_HUB`.
|
|
227
237
|
|
|
228
|
-
To update the installed package
|
|
238
|
+
To update the installed package:
|
|
229
239
|
|
|
230
240
|
```bash
|
|
231
241
|
bun update -g @msareen/knowledge-hub-builder
|
|
232
|
-
cd ~/my-knowledge
|
|
233
|
-
khb upgrade
|
|
234
242
|
```
|
|
235
243
|
|
|
244
|
+
Each hub refreshes itself the next time you run any command in it. `khb upgrade` does the
|
|
245
|
+
same thing on demand, with a fuller report.
|
|
246
|
+
|
|
236
247
|
## Hub Layout
|
|
237
248
|
|
|
238
249
|
```text
|
package/SPEC.md
CHANGED
|
@@ -92,6 +92,14 @@ agent is opened on the hub folder and must find its rules there, without knowing
|
|
|
92
92
|
`khb` is installed. They are package-owned: `khb upgrade` overwrites them in place and
|
|
93
93
|
leaves `bundles/` and `outer.index.md` alone.
|
|
94
94
|
|
|
95
|
+
Copies drift, so the copy is kept honest by the CLI rather than by the user's memory.
|
|
96
|
+
`khb.json` stamps the version that wrote the hub's copies; before running any command that
|
|
97
|
+
touches a hub, `khb` compares that stamp to the installed package version and, if they
|
|
98
|
+
differ, performs the upgrade in place and says so on stderr. Only `init` (no hub yet) and
|
|
99
|
+
`upgrade` (which is the operation) skip the check, and `KHB_NO_AUTO_UPGRADE=1` disables it.
|
|
100
|
+
The invariant it buys: **a hub's contract docs always state the same version as the `khb`
|
|
101
|
+
acting on them** — an agent can never read a protocol the CLI no longer implements.
|
|
102
|
+
|
|
95
103
|
### 2b. The package — `@msareen/knowledge-hub-builder`, installed once
|
|
96
104
|
|
|
97
105
|
> **None of the following is in your hub.** This section describes the *installed tool*,
|
|
@@ -105,7 +113,7 @@ leaves `bundles/` and `outer.index.md` alone.
|
|
|
105
113
|
@msareen/knowledge-hub-builder/ # installed once, globally — NOT part of a hub
|
|
106
114
|
├── package.json # bin: khb → scripts/cli.ts
|
|
107
115
|
├── scripts/
|
|
108
|
-
│ ├── cli.ts # subcommand dispatch;
|
|
116
|
+
│ ├── cli.ts # subcommand dispatch; --hub flag; the version drift check
|
|
109
117
|
│ ├── init.ts # khb init / khb upgrade
|
|
110
118
|
│ ├── new-bundle.ts # scaffold from .bundle_template, register in outer.index.md
|
|
111
119
|
│ ├── export.ts # bundle + common patterns → standalone shareable folder
|
|
@@ -116,6 +124,7 @@ leaves `bundles/` and `outer.index.md` alone.
|
|
|
116
124
|
│ ├── extract.ts # every local extractor + the content-hash cache
|
|
117
125
|
│ ├── ledger.ts # log.md read/write
|
|
118
126
|
│ ├── paths.ts # package-side paths — importing it never needs a hub
|
|
127
|
+
│ ├── upgrade.ts # the refresh itself: `khb upgrade` and the drift check
|
|
119
128
|
│ └── util.ts # hub resolution + shared helpers
|
|
120
129
|
├── .bundle_template/ # copied by `khb new-bundle`
|
|
121
130
|
├── templates/hub/ # copied by `khb init`
|
|
@@ -217,12 +226,20 @@ Two look-alikes are deliberately excluded, since both would rewire provenance on
|
|
|
217
226
|
**copy** (the twin's path still exists — two real sources, both ingested) and an **ambiguous**
|
|
218
227
|
match (several vanished rows share the hash). Each is reported and left to judgement.
|
|
219
228
|
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
`
|
|
224
|
-
|
|
225
|
-
|
|
229
|
+
A bundle is a logical unit its owner defines, so the destination is a human decision. An
|
|
230
|
+
explicit name that doesn't resolve is an error, not a scaffold request; a bare `khb ingest`
|
|
231
|
+
in a hub that has real bundles prints them and stops, because the CLI cannot ask. The agent
|
|
232
|
+
asks instead (`skills/ingest/SKILL.md` §1–2): which existing bundle, or a new one; then, for
|
|
233
|
+
an existing bundle, whether to re-ingest its declared `sources.yaml` paths or take a new path.
|
|
234
|
+
|
|
235
|
+
`default` survives as the fallback for a hub with nothing to choose between: a bare
|
|
236
|
+
`khb ingest` scaffolds it and lands there when there are **no bundles at all**, and uses it
|
|
237
|
+
when it is the **only** bundle — a one-option question is not a choice, and the first ingest
|
|
238
|
+
anyone runs should not fail for want of a destination. It is not an option once a real bundle
|
|
239
|
+
exists — that was the earlier design, and it bought a pile of material whose ownership
|
|
240
|
+
nobody had decided, which is exactly the decision cataloging then has to make blind.
|
|
241
|
+
`default` is a holding area, not a tier: its contents are cataloged like any bundle's, and
|
|
242
|
+
they move only when a human says which bundle should own them.
|
|
226
243
|
|
|
227
244
|
Every raw file carries its origin, so a lossy extraction is always recoverable:
|
|
228
245
|
|
package/document/faq.md
CHANGED
|
@@ -58,8 +58,12 @@ source, extracts it into markdown under `raw/`, and updates `log.md`.
|
|
|
58
58
|
It does not summarize, label, organize, or create concepts. Unchanged content hashes are
|
|
59
59
|
skipped by default; `--force` reacquires everything.
|
|
60
60
|
|
|
61
|
-
|
|
62
|
-
|
|
61
|
+
A named bundle must already exist. Run without a name and the command lists the hub's bundles
|
|
62
|
+
and stops — unless there is nothing to choose between: a hub with no bundles gets a `default`
|
|
63
|
+
landing bundle created so a first ingest still works, and a hub whose only bundle is `default`
|
|
64
|
+
uses it. An agent asked to ingest without a named bundle asks you which
|
|
65
|
+
existing bundle owns the material or whether to create a new one, and for an existing bundle
|
|
66
|
+
whether to re-ingest the paths already in its `sources.yaml` or take a new path.
|
|
63
67
|
|
|
64
68
|
## Should I run ingestion manually or ask an agent?
|
|
65
69
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@msareen/knowledge-hub-builder",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.5",
|
|
4
4
|
"description": "Knowledge Hub Builder — a bundle-of-bundles knowledge base you build with an agent",
|
|
5
5
|
"keywords": ["knowledge-base", "okf", "agent", "wiki", "bundles", "claude", "codex", "rag"],
|
|
6
6
|
"license": "MIT",
|
package/scripts/cli.ts
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
#!/usr/bin/env bun
|
|
2
2
|
// khb — the CLI. Subcommands are loaded lazily: `init` must run before a hub exists,
|
|
3
3
|
// so nothing that resolves a hub may be imported at module scope.
|
|
4
|
-
import { version } from "./lib/paths";
|
|
4
|
+
import { version, findHub, markerIn, MARKER } from "./lib/paths";
|
|
5
5
|
|
|
6
6
|
const COMMANDS: Record<string, { load: () => Promise<unknown>; help: string }> = {
|
|
7
7
|
init: { load: () => import("./init"), help: "khb init [dir] create a hub here (or in dir)" },
|
|
8
8
|
upgrade: { load: () => import("./init"), help: "khb upgrade refresh this hub's contract docs" },
|
|
9
9
|
"new-bundle": { load: () => import("./new-bundle"), help: 'khb new-bundle <name> ["scope"] scaffold a bundle + register it' },
|
|
10
|
-
ingest: { load: () => import("./ingest/index"), help: "khb ingest
|
|
10
|
+
ingest: { load: () => import("./ingest/index"), help: "khb ingest <bundle> [--force] acquire + extract declared sources → raw/ (name required once the hub has a bundle other than 'default')" },
|
|
11
11
|
lint: { load: () => import("./lint"), help: "khb lint validate the hub against skills/lint/SKILL.md" },
|
|
12
12
|
visualize: { load: () => import("./visualize"), help: "khb visualize [--port N] [--no-open] serve the live bundle graph in your browser; aliases: vis, viz" },
|
|
13
13
|
export: { load: () => import("./export"), help: "khb export <bundle> [dest] standalone copy of one bundle" },
|
|
@@ -59,6 +59,28 @@ if (!entry) {
|
|
|
59
59
|
process.exit(1);
|
|
60
60
|
}
|
|
61
61
|
|
|
62
|
+
// Version drift: a hub carries package-owned copies of the agent contract, and a hub
|
|
63
|
+
// stamped at an older version than the installed khb is stating an older contract than
|
|
64
|
+
// the one the CLI now implements. Rather than let the two disagree, refresh the hub in
|
|
65
|
+
// place before running the command — `khb upgrade` touches nothing the user wrote.
|
|
66
|
+
// `init` has no hub yet, `upgrade` does this itself, and $KHB_NO_AUTO_UPGRADE opts out.
|
|
67
|
+
if (cmd !== "init" && cmd !== "upgrade" && !process.env.KHB_NO_AUTO_UPGRADE) {
|
|
68
|
+
const hub = findHub();
|
|
69
|
+
if (hub) {
|
|
70
|
+
const { hubVersion, upgradeHub } = await import("./lib/upgrade");
|
|
71
|
+
// A marker under a pre-rename name is drift too, even at a matching version.
|
|
72
|
+
if (hubVersion(hub) !== version() || markerIn(hub) !== MARKER) {
|
|
73
|
+
const { from, to, pruned, renamed } = upgradeHub(hub);
|
|
74
|
+
// stderr, so a command's own output stays pipeable.
|
|
75
|
+
console.error(
|
|
76
|
+
`khb: hub was built by ${from ?? "an unknown version"}, khb is ${to} — refreshed its contract docs.`,
|
|
77
|
+
);
|
|
78
|
+
if (renamed) console.error(`khb: renamed ${renamed} -> khb.json`);
|
|
79
|
+
if (pruned.length) console.error(`khb: removed (no longer part of the contract): ${pruned.join(", ")}`);
|
|
80
|
+
}
|
|
81
|
+
}
|
|
82
|
+
}
|
|
83
|
+
|
|
62
84
|
// Subcommand modules parse process.argv.slice(2) themselves — reshape it so they see
|
|
63
85
|
// their own arguments and not the subcommand name.
|
|
64
86
|
process.argv = [process.argv[0], process.argv[1], ...argv];
|
package/scripts/ingest/index.ts
CHANGED
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
import { parse } from "yaml";
|
|
13
13
|
import { read, join, HUB } from "../lib/util";
|
|
14
14
|
import { detail, section, totalElapsed } from "../lib/log";
|
|
15
|
-
import { bundleForIngest, DEFAULT_BUNDLE } from "../lib/scaffold";
|
|
15
|
+
import { bundleForIngest, listBundles, DEFAULT_BUNDLE } from "../lib/scaffold";
|
|
16
16
|
import { readLedger, writeLedger } from "../lib/ledger";
|
|
17
17
|
import { takeFlag } from "../lib/args";
|
|
18
18
|
import { ingestFolder } from "./folder";
|
|
@@ -43,10 +43,23 @@ if (positional.length > 1) {
|
|
|
43
43
|
console.error("Usage: khb ingest [bundle] [--force] [--skip-ocr] [--skip-audio]");
|
|
44
44
|
process.exit(1);
|
|
45
45
|
}
|
|
46
|
-
// No bundle named
|
|
47
|
-
//
|
|
48
|
-
//
|
|
49
|
-
|
|
46
|
+
// No bundle named: which one owns the material is a human decision (AGENTS.md) and a CLI
|
|
47
|
+
// cannot ask, so it stops and shows what the hub has, leaving the choice to whoever is
|
|
48
|
+
// driving. The exception is a hub with nothing to choose between — no bundles at all, or
|
|
49
|
+
// only the landing bundle — where bytes go to `default` rather than the ingest failing.
|
|
50
|
+
let bundle = positional[0];
|
|
51
|
+
if (!bundle) {
|
|
52
|
+
const have = listBundles();
|
|
53
|
+
const onlyLanding = have.length === 1 && have[0] === DEFAULT_BUNDLE;
|
|
54
|
+
if (have.length && !onlyLanding) {
|
|
55
|
+
console.error("Usage: khb ingest [bundle] [--force] [--skip-ocr] [--skip-audio]");
|
|
56
|
+
console.error(`\nBundles in this hub: ${have.join(", ")}`);
|
|
57
|
+
console.error(`Name the one that owns this material, or start a new one:`);
|
|
58
|
+
console.error(` khb new-bundle <name> "<scope>"`);
|
|
59
|
+
process.exit(1);
|
|
60
|
+
}
|
|
61
|
+
bundle = DEFAULT_BUNDLE;
|
|
62
|
+
}
|
|
50
63
|
|
|
51
64
|
const dir = bundleForIngest(bundle);
|
|
52
65
|
let cfg: unknown;
|
package/scripts/init.ts
CHANGED
|
@@ -5,63 +5,23 @@
|
|
|
5
5
|
// copied INTO the hub — an agent opened on the hub folder must be able to read them
|
|
6
6
|
// without knowing where khb is installed. Those copies are package-owned: `upgrade`
|
|
7
7
|
// overwrites them.
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
8
|
+
//
|
|
9
|
+
// The mechanism itself lives in lib/upgrade.ts, because cli.ts also runs it on version
|
|
10
|
+
// drift before any hub command.
|
|
11
|
+
import { cpSync, mkdirSync, existsSync } from "node:fs";
|
|
12
|
+
import { join, resolve, basename } from "node:path";
|
|
13
|
+
import { HUB_TEMPLATE, MARKER, markerIn } from "./lib/paths";
|
|
14
|
+
import { upgradeHub, syncManaged, stamp } from "./lib/upgrade";
|
|
11
15
|
|
|
12
16
|
const upgrading = process.env.KHB_SUBCOMMAND === "upgrade";
|
|
13
17
|
const [dirArg] = process.argv.slice(2);
|
|
14
18
|
|
|
15
|
-
/** Copy every package-owned contract file into the hub, replacing what is there. */
|
|
16
|
-
function syncManaged(hub: string): string[] {
|
|
17
|
-
const done: string[] = [];
|
|
18
|
-
for (const f of MANAGED) {
|
|
19
|
-
const src = join(PKG, f);
|
|
20
|
-
if (!existsSync(src)) continue;
|
|
21
|
-
const dest = join(hub, f);
|
|
22
|
-
mkdirSync(dirname(dest), { recursive: true });
|
|
23
|
-
cpSync(src, dest, { recursive: true, force: true });
|
|
24
|
-
done.push(statSync(src).isDirectory() ? `${f}/` : f);
|
|
25
|
-
}
|
|
26
|
-
return done;
|
|
27
|
-
}
|
|
28
|
-
|
|
29
|
-
/** Drop package-owned files that later versions stopped shipping. */
|
|
30
|
-
function pruneRetired(hub: string): string[] {
|
|
31
|
-
const gone: string[] = [];
|
|
32
|
-
for (const f of RETIRED) {
|
|
33
|
-
const p = join(hub, f);
|
|
34
|
-
if (!existsSync(p)) continue;
|
|
35
|
-
rmSync(p, { recursive: true, force: true });
|
|
36
|
-
gone.push(f);
|
|
37
|
-
}
|
|
38
|
-
return gone;
|
|
39
|
-
}
|
|
40
|
-
|
|
41
|
-
function stamp(hub: string, created?: string) {
|
|
42
|
-
writeFileSync(
|
|
43
|
-
join(hub, MARKER),
|
|
44
|
-
JSON.stringify(
|
|
45
|
-
{ khb: version(), created: created ?? new Date().toISOString(), upgraded: new Date().toISOString() },
|
|
46
|
-
null,
|
|
47
|
-
2,
|
|
48
|
-
) + "\n",
|
|
49
|
-
);
|
|
50
|
-
}
|
|
51
|
-
|
|
52
19
|
if (upgrading) {
|
|
53
20
|
const { HUB } = await import("./lib/util"); // resolves the hub, or exits with guidance
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
const found = markerIn(HUB)!;
|
|
57
|
-
const before = JSON.parse(await Bun.file(join(HUB, found)).text());
|
|
58
|
-
if (found !== MARKER) rmSync(join(HUB, found));
|
|
59
|
-
const synced = syncManaged(HUB);
|
|
60
|
-
const pruned = pruneRetired(HUB);
|
|
61
|
-
stamp(HUB, before.created);
|
|
62
|
-
console.log(`Upgraded ${HUB}: ${before.khb ?? before.bkr ?? "?"} -> ${version()}`);
|
|
21
|
+
const { from, to, synced, pruned, renamed } = upgradeHub(HUB);
|
|
22
|
+
console.log(`Upgraded ${HUB}: ${from ?? "?"} -> ${to}`);
|
|
63
23
|
console.log(` refreshed: ${synced.join(", ")}`);
|
|
64
|
-
if (
|
|
24
|
+
if (renamed) console.log(` renamed: ${renamed} -> ${MARKER}`);
|
|
65
25
|
if (pruned.length) console.log(` removed (no longer part of the contract): ${pruned.join(", ")}`);
|
|
66
26
|
console.log(`Your bundles/ and outer.index.md were not touched. Next: khb lint`);
|
|
67
27
|
} else {
|
package/scripts/lib/paths.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
// Package-side paths. Importing this must never require a hub to exist — `khb init`
|
|
2
2
|
// runs before there is one. Hub-side paths live in util.ts.
|
|
3
3
|
import { fileURLToPath } from "node:url";
|
|
4
|
-
import { join } from "node:path";
|
|
4
|
+
import { join, dirname, resolve } from "node:path";
|
|
5
5
|
import { readFileSync, existsSync } from "node:fs";
|
|
6
6
|
|
|
7
7
|
/** Root of the installed @msareen/knowledge-hub-builder package (NOT the user's hub). */
|
|
@@ -29,6 +29,26 @@ export const LEGACY_MARKERS = ["bkr.json"];
|
|
|
29
29
|
export const markerIn = (dir: string): string | undefined =>
|
|
30
30
|
[MARKER, ...LEGACY_MARKERS].find((m) => existsSync(join(dir, m)));
|
|
31
31
|
|
|
32
|
+
/**
|
|
33
|
+
* The hub this invocation acts on, or undefined if there is none. Precedence:
|
|
34
|
+
* $KHB_HUB (set from --hub by cli.ts) > nearest ancestor of cwd holding a marker.
|
|
35
|
+
* Soft by design — util.ts turns "none" into an error with guidance, while the version
|
|
36
|
+
* drift check in cli.ts must stay silent for commands that run outside a hub.
|
|
37
|
+
*/
|
|
38
|
+
export function findHub(): string | undefined {
|
|
39
|
+
const explicit = process.env.KHB_HUB;
|
|
40
|
+
if (explicit) {
|
|
41
|
+
const dir = resolve(explicit);
|
|
42
|
+
return markerIn(dir) ? dir : undefined;
|
|
43
|
+
}
|
|
44
|
+
for (let dir = process.cwd(); ; ) {
|
|
45
|
+
if (markerIn(dir)) return dir;
|
|
46
|
+
const up = dirname(dir);
|
|
47
|
+
if (up === dir) return undefined;
|
|
48
|
+
dir = up;
|
|
49
|
+
}
|
|
50
|
+
}
|
|
51
|
+
|
|
32
52
|
/**
|
|
33
53
|
* Package-owned files copied into every hub by `khb init` and refreshed by
|
|
34
54
|
* `khb upgrade`. These are the agent contract — the hub needs its own copies so an
|
package/scripts/lib/scaffold.ts
CHANGED
|
@@ -1,9 +1,12 @@
|
|
|
1
|
-
// Bundle creation, shared by `khb new-bundle` and
|
|
2
|
-
// One implementation so
|
|
3
|
-
//
|
|
1
|
+
// Bundle creation and lookup, shared by `khb new-bundle` and `khb ingest`.
|
|
2
|
+
// One implementation so every bundle is born the same way: same template, same {{name}}
|
|
3
|
+
// substitution, same outer.index.md row — and nothing ever conjures one implicitly.
|
|
4
4
|
import { cpSync, readFileSync, writeFileSync, readdirSync, statSync, existsSync } from "node:fs";
|
|
5
5
|
import { HUB, BUNDLES, TEMPLATE, join } from "./util";
|
|
6
6
|
|
|
7
|
+
// The landing bundle: where a bare `khb ingest` goes in a hub with nothing to choose between
|
|
8
|
+
// — conjured when the hub has no bundles at all, reused when it is the only one. With a real
|
|
9
|
+
// bundle present the destination is a choice and the user makes it, so nothing is auto-created.
|
|
7
10
|
export const DEFAULT_BUNDLE = "default";
|
|
8
11
|
|
|
9
12
|
export const VALID_NAME = /^[a-z0-9][a-z0-9-]*$/;
|
|
@@ -33,24 +36,34 @@ export function createBundle(name: string, scope: string): string {
|
|
|
33
36
|
return dest;
|
|
34
37
|
}
|
|
35
38
|
|
|
39
|
+
/** Names of the bundles that exist in this hub, alphabetically. */
|
|
40
|
+
export function listBundles(): string[] {
|
|
41
|
+
if (!existsSync(BUNDLES)) return [];
|
|
42
|
+
return readdirSync(BUNDLES)
|
|
43
|
+
.filter((n) => statSync(join(BUNDLES, n)).isDirectory())
|
|
44
|
+
.sort();
|
|
45
|
+
}
|
|
46
|
+
|
|
36
47
|
/**
|
|
37
|
-
* Resolve the bundle to ingest into
|
|
38
|
-
*
|
|
39
|
-
*
|
|
40
|
-
*
|
|
48
|
+
* Resolve the bundle to ingest into. A name the user gave must already exist — which bundle
|
|
49
|
+
* owns material is their decision, so an unresolvable name is a typo, not a scaffold request.
|
|
50
|
+
* The one exception is `default` in a hub with no bundles: the first ingest anywhere must
|
|
51
|
+
* still have somewhere to land, and there is no choice to put to the user yet.
|
|
41
52
|
*/
|
|
42
53
|
export function bundleForIngest(name: string): string {
|
|
43
54
|
const dir = join(BUNDLES, name);
|
|
44
55
|
if (existsSync(dir)) return dir;
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
56
|
+
const have = listBundles();
|
|
57
|
+
if (name === DEFAULT_BUNDLE && !have.length) {
|
|
58
|
+
// The scope line lands in outer.index.md, where every agent reads it — so it must not
|
|
59
|
+
// read as an instruction to reorganize the hub. Splitting `default` into real bundles is
|
|
60
|
+
// the user's call, exactly like any other bundle decision.
|
|
61
|
+
createBundle(DEFAULT_BUNDLE, "Unsorted material — where a first ingest lands before any bundle exists; moves out when you say which bundle owns it");
|
|
62
|
+
console.log(`Created bundles/${DEFAULT_BUNDLE}/ — this hub had no bundles to land in.`);
|
|
63
|
+
return dir;
|
|
49
64
|
}
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
console.log(`Created bundles/${DEFAULT_BUNDLE}/ — the landing bundle for unrouted material.`);
|
|
55
|
-
return dir;
|
|
65
|
+
console.error(`No such bundle: ${name}`);
|
|
66
|
+
if (have.length) console.error(`This hub has: ${have.join(", ")}`);
|
|
67
|
+
console.error(`Create it: khb new-bundle ${name} "<scope>"`);
|
|
68
|
+
process.exit(1);
|
|
56
69
|
}
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
// The upgrade mechanism: refresh a hub's package-owned contract docs to match the
|
|
2
|
+
// installed khb. Lives here rather than in init.ts because two callers need it — the
|
|
3
|
+
// explicit `khb upgrade`, and the drift check cli.ts runs before every hub command.
|
|
4
|
+
//
|
|
5
|
+
// Nothing here may import util.ts: the drift check runs before a hub is resolved, and
|
|
6
|
+
// util.ts resolves one or exits.
|
|
7
|
+
import { cpSync, mkdirSync, writeFileSync, readFileSync, existsSync, statSync, rmSync } from "node:fs";
|
|
8
|
+
import { join, dirname } from "node:path";
|
|
9
|
+
import { PKG, MANAGED, RETIRED, MARKER, markerIn, version } from "./paths";
|
|
10
|
+
|
|
11
|
+
export type UpgradeResult = {
|
|
12
|
+
/** Version recorded in the hub's marker before the upgrade, if it recorded one. */
|
|
13
|
+
from?: string;
|
|
14
|
+
to: string;
|
|
15
|
+
synced: string[];
|
|
16
|
+
pruned: string[];
|
|
17
|
+
/** Set when a legacy marker name was renamed to khb.json. */
|
|
18
|
+
renamed?: string;
|
|
19
|
+
};
|
|
20
|
+
|
|
21
|
+
/** Read the khb version a hub was last stamped with. Undefined if it records none. */
|
|
22
|
+
export function hubVersion(hub: string): string | undefined {
|
|
23
|
+
const found = markerIn(hub);
|
|
24
|
+
if (!found) return undefined;
|
|
25
|
+
try {
|
|
26
|
+
const marker = JSON.parse(readFileSync(join(hub, found), "utf8"));
|
|
27
|
+
return marker.khb ?? marker.bkr; // bkr: the pre-rename field name
|
|
28
|
+
} catch {
|
|
29
|
+
return undefined; // unreadable marker — treat as drifted, upgrade will restamp it
|
|
30
|
+
}
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
/** Copy every package-owned contract file into the hub, replacing what is there. */
|
|
34
|
+
export function syncManaged(hub: string): string[] {
|
|
35
|
+
const done: string[] = [];
|
|
36
|
+
for (const f of MANAGED) {
|
|
37
|
+
const src = join(PKG, f);
|
|
38
|
+
if (!existsSync(src)) continue;
|
|
39
|
+
const dest = join(hub, f);
|
|
40
|
+
mkdirSync(dirname(dest), { recursive: true });
|
|
41
|
+
cpSync(src, dest, { recursive: true, force: true });
|
|
42
|
+
done.push(statSync(src).isDirectory() ? `${f}/` : f);
|
|
43
|
+
}
|
|
44
|
+
return done;
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
/** Drop package-owned files that later versions stopped shipping. */
|
|
48
|
+
function pruneRetired(hub: string): string[] {
|
|
49
|
+
const gone: string[] = [];
|
|
50
|
+
for (const f of RETIRED) {
|
|
51
|
+
const p = join(hub, f);
|
|
52
|
+
if (!existsSync(p)) continue;
|
|
53
|
+
rmSync(p, { recursive: true, force: true });
|
|
54
|
+
gone.push(f);
|
|
55
|
+
}
|
|
56
|
+
return gone;
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
/** Write khb.json with the installed version, preserving the hub's creation date. */
|
|
60
|
+
export function stamp(hub: string, created?: string) {
|
|
61
|
+
writeFileSync(
|
|
62
|
+
join(hub, MARKER),
|
|
63
|
+
JSON.stringify(
|
|
64
|
+
{ khb: version(), created: created ?? new Date().toISOString(), upgraded: new Date().toISOString() },
|
|
65
|
+
null,
|
|
66
|
+
2,
|
|
67
|
+
) + "\n",
|
|
68
|
+
);
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
/**
|
|
72
|
+
* Bring `hub`'s package-owned files up to the installed version. Touches only what the
|
|
73
|
+
* package owns: bundles/, outer.index.md and anything else the user wrote are untouched.
|
|
74
|
+
*/
|
|
75
|
+
export function upgradeHub(hub: string): UpgradeResult {
|
|
76
|
+
// The hub may still carry a marker name from an older version; stamp() writes MARKER,
|
|
77
|
+
// so drop the old file rather than leaving the hub with two.
|
|
78
|
+
const found = markerIn(hub)!;
|
|
79
|
+
let created: string | undefined;
|
|
80
|
+
let from: string | undefined;
|
|
81
|
+
try {
|
|
82
|
+
const before = JSON.parse(readFileSync(join(hub, found), "utf8"));
|
|
83
|
+
created = before.created;
|
|
84
|
+
from = before.khb ?? before.bkr;
|
|
85
|
+
} catch {
|
|
86
|
+
/* unreadable marker: rewritten below with today's date */
|
|
87
|
+
}
|
|
88
|
+
if (found !== MARKER) rmSync(join(hub, found));
|
|
89
|
+
const synced = syncManaged(hub);
|
|
90
|
+
const pruned = pruneRetired(hub);
|
|
91
|
+
stamp(hub, created);
|
|
92
|
+
return { from, to: version(), synced, pruned, renamed: found === MARKER ? undefined : found };
|
|
93
|
+
}
|
package/scripts/lib/util.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { readdirSync, readFileSync, existsSync, statSync, mkdirSync, writeFileSync } from "node:fs";
|
|
2
2
|
import { createHash } from "node:crypto";
|
|
3
|
-
import { join, basename,
|
|
4
|
-
import { MARKER, markerIn } from "./paths";
|
|
3
|
+
import { join, basename, resolve } from "node:path";
|
|
4
|
+
import { MARKER, markerIn, findHub } from "./paths";
|
|
5
5
|
|
|
6
6
|
/**
|
|
7
7
|
* Find the hub root — the folder holding khb.json, outer.index.md and bundles/.
|
|
@@ -10,20 +10,12 @@ import { MARKER, markerIn } from "./paths";
|
|
|
10
10
|
* ancestor of cwd containing the marker.
|
|
11
11
|
*/
|
|
12
12
|
function resolveHub(): string {
|
|
13
|
+
const found = findHub();
|
|
14
|
+
if (found) return found;
|
|
13
15
|
const explicit = process.env.KHB_HUB;
|
|
14
16
|
if (explicit) {
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
console.error(`Not a KHB hub (no ${MARKER}): ${dir}`);
|
|
18
|
-
process.exit(1);
|
|
19
|
-
}
|
|
20
|
-
return dir;
|
|
21
|
-
}
|
|
22
|
-
for (let dir = process.cwd(); ; ) {
|
|
23
|
-
if (markerIn(dir)) return dir;
|
|
24
|
-
const up = dirname(dir);
|
|
25
|
-
if (up === dir) break;
|
|
26
|
-
dir = up;
|
|
17
|
+
console.error(`Not a KHB hub (no ${MARKER}): ${resolve(explicit)}`);
|
|
18
|
+
process.exit(1);
|
|
27
19
|
}
|
|
28
20
|
console.error(`No KHB hub found in ${process.cwd()} or any parent directory.`);
|
|
29
21
|
console.error(`Create one: khb init <dir>`);
|
package/skills/ingest/SKILL.md
CHANGED
|
@@ -11,14 +11,60 @@ text exists. Deciding what the text *means* — splitting it into concepts, titl
|
|
|
11
11
|
tagging, linking, indexing — is the [catalog skill](../catalog/SKILL.md), a separate step
|
|
12
12
|
you run afterwards.
|
|
13
13
|
|
|
14
|
-
Do not curate here. Do not
|
|
15
|
-
|
|
16
|
-
to understand it, you have left this skill.
|
|
17
|
-
|
|
18
|
-
## 1.
|
|
19
|
-
|
|
20
|
-
Ingest is bundle-first: material lands in one bundle, and
|
|
21
|
-
|
|
14
|
+
Do not curate here. Do not split or merge bundles here, and create one only as the user's
|
|
15
|
+
answer to the question in step 1 — never on your own initiative. If you find yourself reading
|
|
16
|
+
a document to understand it, you have left this skill.
|
|
17
|
+
|
|
18
|
+
## 1. Settle the bundle — ask, never assume
|
|
19
|
+
|
|
20
|
+
Ingest is bundle-first: material lands in exactly one bundle, and which bundle owns it is the
|
|
21
|
+
user's decision, not yours. Whenever there is a choice to make, put it to them.
|
|
22
|
+
|
|
23
|
+
**Take the first of these that applies:**
|
|
24
|
+
|
|
25
|
+
1. **The user named a bundle** — "re-ingest the real-estate bundle" — → use it, no question
|
|
26
|
+
asked. If the name does not resolve to a bundle in `bundles/`, say so and ask whether to
|
|
27
|
+
create it; never scaffold on a guess, and never silently fall back to a similar name.
|
|
28
|
+
2. **No bundle named, and the hub has real bundles** → ask, always, offering the choice
|
|
29
|
+
explicitly: **an existing bundle** (list them, from `outer.index.md`, with their scope
|
|
30
|
+
lines so the user can tell them apart) **or a new bundle**. Do not pick for the user,
|
|
31
|
+
however plainly one bundle seems to own the material — say which you would pick if you
|
|
32
|
+
have a view, then wait for the answer. Do not offer `default` here and do not mention it;
|
|
33
|
+
with real bundles on the table it is not one of the options.
|
|
34
|
+
3. **The user answered "a new bundle"** → they name it and give its scope; you run
|
|
35
|
+
`khb new-bundle <name> "<scope>"`. Creating a bundle is a human decision (`AGENTS.md`), so
|
|
36
|
+
this branch only ever runs on an explicit answer to the question above.
|
|
37
|
+
4. **No bundle named and nothing to choose between** → `default`, without asking. Two shapes
|
|
38
|
+
of hub qualify:
|
|
39
|
+
- **no bundles at all** → `default` is created on the spot; a first ingest should not fail
|
|
40
|
+
for want of a destination.
|
|
41
|
+
- **`default` is the only bundle** → it is used as it stands. A one-option question is not
|
|
42
|
+
a choice, so do not put it to the user.
|
|
43
|
+
|
|
44
|
+
**The `default` bundle** is only that last case — a landing place in a hub that has no other,
|
|
45
|
+
not an option to fall back on once a real bundle exists. The moment one does, `default` stops
|
|
46
|
+
being a destination for unnamed ingests and case 2 applies. What lands there is ordinary
|
|
47
|
+
bundle content: catalog it like any other. Do **not** graduate it into new bundles on your
|
|
48
|
+
own; material leaves `default` when the user says which bundle owns it.
|
|
49
|
+
|
|
50
|
+
## 2. Declare the sources — reuse or replace
|
|
51
|
+
|
|
52
|
+
Once the bundle is settled, settle where the material comes from. If the user already named
|
|
53
|
+
the files, folders or URLs, that is the answer — you still ask the add-or-replace question
|
|
54
|
+
below when the bundle has declarations of its own. **Otherwise, for an existing bundle, read
|
|
55
|
+
its `sources.yaml` first and ask which you are doing:**
|
|
56
|
+
|
|
57
|
+
- **use what's declared** — re-ingest the paths already in `sources.yaml` (this is what
|
|
58
|
+
"re-ingest the real-estate bundle" usually means), or
|
|
59
|
+
- **a new path** — the user gives files, folders or URLs; ask whether they are *added* to
|
|
60
|
+
the declarations or *replace* them before you edit the file.
|
|
61
|
+
|
|
62
|
+
Quote the current declarations in the question so the answer is informed. A bundle with an
|
|
63
|
+
empty `sources.yaml` has nothing to re-ingest, so there the only answer is a new path — ask
|
|
64
|
+
for it. Do not infer sources from nearby files, do not edit `sources.yaml`, and do not run
|
|
65
|
+
`khb ingest` until the user has answered.
|
|
66
|
+
|
|
67
|
+
Sources live in `bundles/<bundle>/sources.yaml`:
|
|
22
68
|
|
|
23
69
|
```yaml
|
|
24
70
|
sources:
|
|
@@ -32,46 +78,17 @@ sources:
|
|
|
32
78
|
urls:
|
|
33
79
|
- https://example.com/design-doc
|
|
34
80
|
# Types with no scripted ingester are still declared here, for the record —
|
|
35
|
-
# you pull them yourself in step
|
|
81
|
+
# you pull them yourself in step 4.
|
|
36
82
|
- type: confluence
|
|
37
83
|
space: PROJX
|
|
38
84
|
```
|
|
39
85
|
|
|
40
|
-
If the user has not explicitly named the source locations, inspect the bundle's current
|
|
41
|
-
`sources.yaml`, then ask which files, folders, URLs, or services to ingest. Include any
|
|
42
|
-
existing declarations in the question so the user can confirm or replace them. Do not
|
|
43
|
-
infer sources from nearby files, edit `sources.yaml`, or run `khb ingest` until the user
|
|
44
|
-
answers.
|
|
45
|
-
|
|
46
86
|
Nothing is copied by declaring a source.
|
|
47
87
|
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
1. **The user named a bundle** → use it. A named bundle that does not exist is an error,
|
|
51
|
-
not an invitation to create one.
|
|
52
|
-
2. **The hub has bundles and exactly one plainly owns the material** → use it, and say
|
|
53
|
-
which you picked. If several could own it, ask which — this is the only bundle question
|
|
54
|
-
ingest ever asks.
|
|
55
|
-
3. **Anything else** — no bundle named, or the hub has no bundles at all → `default`,
|
|
56
|
-
created on the spot, without asking.
|
|
57
|
-
|
|
58
|
-
Never ask the user to name or create a bundle *for the ingest to land in*. `default` exists
|
|
59
|
-
so that question never has to be asked at this stage: bytes always have somewhere to go, and
|
|
60
|
-
which bundle owns them is a cheaper decision later, once the text exists and the user can see
|
|
61
|
-
what they actually have.
|
|
62
|
-
|
|
63
|
-
**The `default` bundle.** When no bundle is named, ingest targets `default` and creates it
|
|
64
|
-
if the hub has none — a first `khb ingest` never fails for want of a destination. It is not a
|
|
65
|
-
way around step 2: when a bundle in the hub plainly owns the material, that bundle wins. What
|
|
66
|
-
lands there is ordinary bundle content: catalog it like any other. Do **not** graduate it into new
|
|
67
|
-
bundles on your own — a bundle is a logical unit the user defines (a person, a team, a
|
|
68
|
-
project), so material leaves `default` only when the user says which bundle owns it. An
|
|
69
|
-
explicitly named bundle that doesn't exist is still an error — only `default` is conjured.
|
|
70
|
-
|
|
71
|
-
## 2. Run it
|
|
88
|
+
## 3. Run it
|
|
72
89
|
|
|
73
90
|
```
|
|
74
|
-
khb ingest #
|
|
91
|
+
khb ingest # only where 'default' is the sole bundle, or none is
|
|
75
92
|
khb ingest <bundle> # incremental: unchanged content hashes are skipped
|
|
76
93
|
khb ingest <bundle> --force # re-acquire everything
|
|
77
94
|
khb ingest <bundle> --skip-ocr # leave scans and images unread
|
|
@@ -135,7 +152,7 @@ pip install -U openai-whisper # transcription (faster-whisper als
|
|
|
135
152
|
Install them where `khb` resolves modules from — for a global install that is the khb
|
|
136
153
|
package directory, not your hub. khb prints the exact `cd … && bun add …` to use.
|
|
137
154
|
|
|
138
|
-
##
|
|
155
|
+
## 4. Sources khb cannot reach
|
|
139
156
|
|
|
140
157
|
Anything behind an authenticated API has no scripted ingester, because maintaining API
|
|
141
158
|
wrappers is not what this tool is for. Pull those yourself with the site's MCP server or
|
|
@@ -175,7 +192,7 @@ and the text reads thin, garbled, or contradictory, **open the `source:` file an
|
|
|
175
192
|
directly** — a vision pass over a chart or a scanned table recovers what OCR drops. Rewrite
|
|
176
193
|
the `raw/` file with `extract_tool: claude-vision` and `quality: high` when you do.
|
|
177
194
|
|
|
178
|
-
##
|
|
195
|
+
## 5. The ledger — `log.md`
|
|
179
196
|
|
|
180
197
|
Every bundle keeps its ingest ledger in `log.md` (OKF-reserved, so it is never mistaken for
|
|
181
198
|
a concept doc, and committed, so it survives `raw/` being deleted and re-derived).
|