@msareen/knowledge-hub-builder 0.1.8 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +9 -5
- package/README.md +11 -3
- package/SPEC.md +17 -4
- package/package.json +1 -1
- package/scripts/cli.ts +35 -10
- package/scripts/export.ts +6 -1
- package/scripts/hubs.ts +6 -13
- package/scripts/ingest/acquire.ts +56 -13
- package/scripts/ingest/exts.ts +82 -1
- package/scripts/ingest/files.ts +7 -2
- package/scripts/ingest/folder.ts +7 -2
- package/scripts/ingest/index.ts +6 -9
- package/scripts/init.ts +27 -14
- package/scripts/lib/args.ts +37 -0
- package/scripts/lib/extract.ts +269 -28
- package/scripts/lib/util.ts +7 -0
- package/scripts/lint.ts +3 -0
- package/scripts/new-bundle.ts +4 -1
- package/scripts/visualize.ts +15 -10
- package/skills/ingest/SKILL.md +62 -4
- package/templates/hub/gitignore +4 -3
package/AGENTS.md
CHANGED
|
@@ -73,8 +73,10 @@ One hard boundary governs every workflow and every future change to the tooling:
|
|
|
73
73
|
|
|
74
74
|
- **`khb` converts bytes to text.** Hashing, caching, file plumbing, ledger-keeping, and
|
|
75
75
|
*every* local extractor: PDF/DOCX/ODT/XLSX/PPTX libraries, tesseract OCR for scans and
|
|
76
|
-
images, whisper for audio and video
|
|
77
|
-
|
|
76
|
+
images, whisper.cpp (via `vno`) or whisper for audio and video, and the caption reader
|
|
77
|
+
that spares them the job whenever a `.vtt`/`.srt` sits beside the recording. All of it
|
|
78
|
+
deterministic, offline, and free of charge. It **never contacts a model** — not directly,
|
|
79
|
+
not by shelling out.
|
|
78
80
|
- **The agent decides what the text means.** Splitting a document into concepts, labeling
|
|
79
81
|
and linking them, curating `raw/` into the wiki, escalating a bad OCR to a vision read,
|
|
80
82
|
and judging when a query has produced a new concept worth keeping.
|
|
@@ -136,7 +138,7 @@ From outside, pass `--hub <dir>` or set `$KHB_HUB`.
|
|
|
136
138
|
|---|---|
|
|
137
139
|
| `khb lint` | validate structure against `skills/lint/SKILL.md` |
|
|
138
140
|
| `khb upgrade` | refresh this hub's package-owned contract docs — runs by itself whenever `khb.json`'s stamped version differs from the installed khb, so these docs always match the CLI |
|
|
139
|
-
| `khb visualize` |
|
|
141
|
+
| `khb visualize [--port N] [--no-open]` | serve the live bundle graph in your browser; aliases `vis`, `viz` |
|
|
140
142
|
| `khb new-bundle <name>` | scaffold + register a bundle |
|
|
141
143
|
| `khb ingest <bundle>` | acquire + extract every source in `sources.yaml` → `raw/`; maintains `log.md`. Name the bundle — with none it lists the hub's bundles and stops, unless there is nothing to choose between (no bundles, or only `default`), where it uses `default` |
|
|
142
144
|
| `khb export <bundle> [dest]` | standalone copy: bundle + common patterns, shareable alone |
|
|
@@ -164,8 +166,10 @@ There is no `khb catalog` command — cataloging is entirely a judgement pass.
|
|
|
164
166
|
**Ingest** (`skills/ingest/SKILL.md`) is mechanical and flat: `khb ingest <bundle>` pulls
|
|
165
167
|
every declared source into `raw/` as markdown with a provenance header, extracting
|
|
166
168
|
everything it can locally — text, PDF, DOCX, ODT, XLSX, PPTX, images by OCR, audio and
|
|
167
|
-
video by whisper
|
|
168
|
-
|
|
169
|
+
video by whisper — or, where a recording has a `.vtt`/`.srt` beside it, from those captions,
|
|
170
|
+
the pair acquired as a single source. Sources behind an authenticated API (Confluence, ADO,
|
|
171
|
+
git hosts) you pull yourself via MCP/CLI into the same `raw/` shape. Ingest never interprets
|
|
172
|
+
content.
|
|
169
173
|
|
|
170
174
|
When the user has not named a bundle, **ask** which existing bundle owns the material or
|
|
171
175
|
whether to start a new one, and for an existing bundle ask whether to re-ingest what its
|
package/README.md
CHANGED
|
@@ -88,6 +88,8 @@ Supported out of the box, with no system tools and no further installation:
|
|
|
88
88
|
- PDF and DOCX
|
|
89
89
|
- ODT, XLSX, and PPTX
|
|
90
90
|
- OCR for images and scanned PDFs, applied automatically when a PDF has no text layer
|
|
91
|
+
- subtitle files (`.vtt`, `.srt`) — and a video or audio file with one beside it is read
|
|
92
|
+
from the captions rather than transcribed, as one source rather than two
|
|
91
93
|
|
|
92
94
|
OCR ships with KHB: `@hyzyla/pdfium`, `sharp`, and `tesseract.js` are ordinary dependencies,
|
|
93
95
|
so installing KHB pulls them down whether or not you ever ingest a scan. That costs roughly
|
|
@@ -96,8 +98,14 @@ waiting for a setup step.
|
|
|
96
98
|
|
|
97
99
|
One extractor is genuinely optional:
|
|
98
100
|
|
|
99
|
-
- audio and video transcription needs a
|
|
100
|
-
(
|
|
101
|
+
- audio and video transcription needs a transcriber on `PATH`. KHB prefers
|
|
102
|
+
[`vno`](https://www.npmjs.com/package/@msareen/voice-notes-organizer)
|
|
103
|
+
(`npm install -g @msareen/voice-notes-organizer`), which wraps whisper.cpp and installs
|
|
104
|
+
its own ffmpeg and model, and falls back to `whisper` / `faster-whisper`
|
|
105
|
+
(`pip install -U openai-whisper`). KHB checks `vno status` first: a vno that is installed
|
|
106
|
+
but not yet set up leaves recordings pending with `run: vno setup` and holds up nothing
|
|
107
|
+
else in the run. A recording that already has a `.vtt` or `.srt` beside
|
|
108
|
+
it needs neither — those captions are read instead
|
|
101
109
|
|
|
102
110
|
Without it, KHB leaves a pending row in `log.md` and prints the required setup rather than
|
|
103
111
|
failing the run.
|
|
@@ -241,7 +249,7 @@ Commands can be run directly or requested through the matching agent skill.
|
|
|
241
249
|
| `khb init [dir] [--name N] [--description "…"]` | Create a hub |
|
|
242
250
|
| `khb upgrade` | Refresh package-owned contracts and skills (also runs automatically on version drift) |
|
|
243
251
|
| `khb new-bundle <name> ["scope"]` | Create and register a bundle |
|
|
244
|
-
| `khb ingest [bundle] [--force]` | Acquire and extract declared sources |
|
|
252
|
+
| `khb ingest [bundle] [--force] [--skip-ocr] [--skip-audio]` | Acquire and extract declared sources |
|
|
245
253
|
| `khb lint` | Validate routing, bundle structure, and OKF metadata |
|
|
246
254
|
| `khb visualize [--port N] [--no-open]` (aliases: `vis`, `viz`) | Serve the live bundle graph on a random free port and open it in your default browser — pan/zoomable cross-bundle map, drill into a bundle for its folder-clustered concepts, rebuild-on-refresh, exits when you close the tab |
|
|
247
255
|
| `khb export <bundle> [dest]` | Export one standalone bundle |
|
package/SPEC.md
CHANGED
|
@@ -45,7 +45,8 @@ KHB's own contribution is the bundle-of-bundles layer over both — see §1.
|
|
|
45
45
|
every concept traces back through a provenance header to the original file.
|
|
46
46
|
6. **Extract to markdown, locally** — binary/opaque formats (PDF, DOCX, XLSX, images,
|
|
47
47
|
audio) are converted to markdown so every bundle's knowledge is plain, greppable text.
|
|
48
|
-
Every extractor is local and deterministic: pure-JS libraries, tesseract WASM,
|
|
48
|
+
Every extractor is local and deterministic: pure-JS libraries, tesseract WASM,
|
|
49
|
+
whisper.cpp.
|
|
49
50
|
Lossy routes (OCR, ASR) are marked `quality: low` rather than hidden.
|
|
50
51
|
7. **Bun** is the scripting language for all tooling and third-party interfaces.
|
|
51
52
|
|
|
@@ -494,11 +495,23 @@ puts them on the CLI side of the §Division-of-labor line.
|
|
|
494
495
|
| XLSX | `fflate` → one markdown table per sheet | bundled | high |
|
|
495
496
|
| scanned PDF | `pdfium` + `tesseract.js` (WASM) | bundled, ~75 MB | low |
|
|
496
497
|
| Images (png/jpg/webp/tif) | `tesseract.js` | bundled, ~75 MB | low |
|
|
497
|
-
| Audio, video | `whisper` / `faster-whisper` | opt-in, pip | low |
|
|
498
|
+
| Audio, video | `vno` (whisper.cpp), else `whisper` / `faster-whisper` | opt-in, npm or pip | low |
|
|
499
|
+
| Captions (vtt/srt) | built-in reader | bundled | high |
|
|
500
|
+
|
|
501
|
+
A recording that has a caption sidecar beside it — `talk.vtt`, `talk.en.vtt`, `talk.srt` —
|
|
502
|
+
is read from the sidecar rather than transcribed, and the two are acquired as one source:
|
|
503
|
+
one ledger row under the recording, no row and no `raw/` file for the sidecar. The pair's
|
|
504
|
+
content hash covers both files, so correcting a caption re-ingests the recording. Where two
|
|
505
|
+
sidecars disagree about language khb transcribes instead: choosing an audience is not a
|
|
506
|
+
conversion decision, and §Division-of-labor puts choices on the agent's side of the line.
|
|
498
507
|
|
|
499
508
|
The OCR stack is bundled rather than opt-in: an ingest that stops to ask for an install is
|
|
500
|
-
worse than an install that carries WASM nobody uses. Transcription stays opt-in because it
|
|
501
|
-
|
|
509
|
+
worse than an install that carries WASM nobody uses. Transcription stays opt-in because it
|
|
510
|
+
is an external executable, not something khb's own dependency tree can carry: `vno`
|
|
511
|
+
(@msareen/voice-notes-organizer) where `vno status` reports it ready, since whisper.cpp is
|
|
512
|
+
faster than the Python whisper and hands back WebVTT the caption reader can anchor, else
|
|
513
|
+
`whisper` / `faster-whisper`. A vno that is installed but not set up degrades to the
|
|
514
|
+
fallback, or to pending rows, and never to a failed run.
|
|
502
515
|
|
|
503
516
|
A missing dep degrades to a ledger row with an empty `raw` and a printed install
|
|
504
517
|
hint — never to a failed run. `quality: low` output is a standing invitation for the catalog
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@msareen/knowledge-hub-builder",
|
|
3
|
-
"version": "0.1
|
|
3
|
+
"version": "0.2.1",
|
|
4
4
|
"description": "Knowledge Hub Builder — a bundle-of-bundles knowledge base you build with an agent",
|
|
5
5
|
"keywords": ["knowledge-base", "okf", "agent", "wiki", "bundles", "claude", "codex", "rag"],
|
|
6
6
|
"license": "MIT",
|
package/scripts/cli.ts
CHANGED
|
@@ -9,8 +9,8 @@ const COMMANDS: Record<string, { load: () => Promise<unknown>; usage: string; de
|
|
|
9
9
|
"new-bundle": { load: () => import("./new-bundle"), usage: 'khb new-bundle <name> ["scope"]', desc: "scaffold a bundle + register it" },
|
|
10
10
|
ingest: {
|
|
11
11
|
load: () => import("./ingest/index"),
|
|
12
|
-
usage: "khb ingest
|
|
13
|
-
desc: "acquire + extract declared sources → raw/
|
|
12
|
+
usage: "khb ingest [bundle] [--force] [--skip-ocr] [--skip-audio]",
|
|
13
|
+
desc: "acquire + extract declared sources → raw/",
|
|
14
14
|
},
|
|
15
15
|
lint: { load: () => import("./lint"), usage: "khb lint", desc: "validate the hub against skills/lint/SKILL.md" },
|
|
16
16
|
visualize: {
|
|
@@ -20,14 +20,22 @@ const COMMANDS: Record<string, { load: () => Promise<unknown>; usage: string; de
|
|
|
20
20
|
},
|
|
21
21
|
export: { load: () => import("./export"), usage: "khb export <bundle> [dest]", desc: "standalone copy of one bundle" },
|
|
22
22
|
list: { load: () => import("./hubs"), usage: "khb list [--json]", desc: "every hub on this machine" },
|
|
23
|
-
go: {
|
|
24
|
-
|
|
23
|
+
go: {
|
|
24
|
+
load: () => import("./hubs"),
|
|
25
|
+
usage: "khb go [name|N] [--path] [--no-agent] [--agent X]",
|
|
26
|
+
desc: "open a hub with your agent (bare 'khb' picks one)",
|
|
27
|
+
},
|
|
28
|
+
agent: {
|
|
29
|
+
load: () => import("./hubs"),
|
|
30
|
+
usage: 'khb agent [name|none] [--command X] [--args "…"]',
|
|
31
|
+
desc: "which agent 'khb go' launches",
|
|
32
|
+
},
|
|
25
33
|
update: {
|
|
26
34
|
load: () => import("./hubs"),
|
|
27
|
-
usage: "khb update [new] [--path|-p] [--schema|-s] [--dry-run]",
|
|
28
|
-
desc: "repair a moved hub's paths and/or backfill sources.yaml
|
|
35
|
+
usage: "khb update [new-path] [--path|-p] [--schema|-s] [--from <old>] [--dry-run]",
|
|
36
|
+
desc: "repair a moved hub's paths, and/or backfill sources.yaml",
|
|
29
37
|
},
|
|
30
|
-
forget: { load: () => import("./hubs"), usage: "khb forget <name>", desc: "drop a hub from the list (folder untouched)" },
|
|
38
|
+
forget: { load: () => import("./hubs"), usage: "khb forget <name|path>", desc: "drop a hub from the list (folder untouched)" },
|
|
31
39
|
};
|
|
32
40
|
|
|
33
41
|
/**
|
|
@@ -62,12 +70,23 @@ const cmd0 = argv.shift();
|
|
|
62
70
|
const cmd = !cmd0 ? "go" : (ALIASES[cmd0] ?? cmd0);
|
|
63
71
|
|
|
64
72
|
if (cmd === "help" || cmd === "--help" || cmd === "-h") {
|
|
65
|
-
|
|
73
|
+
// Descriptions align to the longest usage — but only among those that fit. One long
|
|
74
|
+
// entry should not push every other description off an 80-column terminal: past the
|
|
75
|
+
// cap, that entry gets its own line for the usage and an indented line for the
|
|
76
|
+
// description, instead of dragging the whole column wide.
|
|
77
|
+
const CAP = 60;
|
|
78
|
+
const usages = Object.values(COMMANDS).map((c) => c.usage.length);
|
|
79
|
+
const width = Math.max(...usages.filter((n) => n <= CAP));
|
|
66
80
|
const printSection = (title: string, names: string[]) => {
|
|
67
81
|
console.log(title);
|
|
68
82
|
for (const name of names) {
|
|
69
83
|
const c = COMMANDS[name];
|
|
70
|
-
|
|
84
|
+
if (c.usage.length > width) {
|
|
85
|
+
console.log(` ${c.usage}`);
|
|
86
|
+
console.log(` ${" ".repeat(width)} ${c.desc}`);
|
|
87
|
+
} else {
|
|
88
|
+
console.log(` ${c.usage.padEnd(width)} ${c.desc}`);
|
|
89
|
+
}
|
|
71
90
|
}
|
|
72
91
|
};
|
|
73
92
|
|
|
@@ -89,7 +108,13 @@ if (cmd === "help" || cmd === "--help" || cmd === "-h") {
|
|
|
89
108
|
);
|
|
90
109
|
console.log();
|
|
91
110
|
|
|
92
|
-
console.log(`Global: --hub <dir>
|
|
111
|
+
console.log(`Global: --hub <dir> operate on that hub instead of searching upward from cwd`);
|
|
112
|
+
console.log(` help | --help | -h this help --version | -v version`);
|
|
113
|
+
console.log(`Env: KHB_HUB same as --hub`);
|
|
114
|
+
console.log(` KHB_HOME where the hub list lives (default ~/.khb)`);
|
|
115
|
+
console.log(` KHB_NO_AUTO_UPGRADE don't refresh a hub's contract docs on version drift`);
|
|
116
|
+
console.log(`Exit: 0 on success, 1 on a usage error or a failure. An unknown option is an error;`);
|
|
117
|
+
console.log(` a source khb cannot extract is not — it becomes a pending row in log.md.`);
|
|
93
118
|
console.log(`Docs: https://github.com/msareen/knowledge-hub-builder`);
|
|
94
119
|
process.exit(0);
|
|
95
120
|
}
|
package/scripts/export.ts
CHANGED
|
@@ -4,8 +4,13 @@
|
|
|
4
4
|
import { cpSync, writeFileSync, mkdirSync, existsSync, readFileSync } from "node:fs";
|
|
5
5
|
import { HUB, bundleDir, join } from "./lib/util";
|
|
6
6
|
import { detail, totalElapsed } from "./lib/log";
|
|
7
|
+
import { rejectUnknownFlags } from "./lib/args";
|
|
7
8
|
|
|
8
|
-
const
|
|
9
|
+
const argv = process.argv.slice(2);
|
|
10
|
+
// Before reading positionals: an unrecognized flag would otherwise become the destination,
|
|
11
|
+
// and `khb export mybundle --force` would export into a folder named `--force`.
|
|
12
|
+
rejectUnknownFlags(argv, "khb export <bundle> [dest]");
|
|
13
|
+
const [name, destArg] = argv;
|
|
9
14
|
if (!name) { console.error("Usage: khb export <bundle> [dest]"); process.exit(1); }
|
|
10
15
|
|
|
11
16
|
const src = bundleDir(name);
|
package/scripts/hubs.ts
CHANGED
|
@@ -29,24 +29,12 @@ import {
|
|
|
29
29
|
touchHub,
|
|
30
30
|
type HubEntry,
|
|
31
31
|
} from "./lib/registry";
|
|
32
|
-
import { takeFlag } from "./lib/args";
|
|
32
|
+
import { takeFlag, takeOpt, rejectUnknownFlags } from "./lib/args";
|
|
33
33
|
import { detail, section, totalElapsed } from "./lib/log";
|
|
34
34
|
|
|
35
35
|
const cmd = process.env.KHB_SUBCOMMAND ?? "go";
|
|
36
36
|
const argv = process.argv.slice(2);
|
|
37
37
|
|
|
38
|
-
/** Take `--name value`, removing both. */
|
|
39
|
-
function takeOpt(args: string[], name: string): string | undefined {
|
|
40
|
-
const i = args.indexOf(name);
|
|
41
|
-
if (i < 0) return undefined;
|
|
42
|
-
const v = args[i + 1];
|
|
43
|
-
if (v === undefined) {
|
|
44
|
-
console.error(`${name} needs a value`);
|
|
45
|
-
process.exit(1);
|
|
46
|
-
}
|
|
47
|
-
args.splice(i, 2);
|
|
48
|
-
return v;
|
|
49
|
-
}
|
|
50
38
|
|
|
51
39
|
const ago = (iso?: string): string => {
|
|
52
40
|
if (!iso) return "";
|
|
@@ -352,6 +340,7 @@ async function wizard(): Promise<never> {
|
|
|
352
340
|
|
|
353
341
|
if (cmd === "list") {
|
|
354
342
|
const asJson = takeFlag(argv, "--json");
|
|
343
|
+
rejectUnknownFlags(argv, "khb list [--json]");
|
|
355
344
|
const hubs = listHubs();
|
|
356
345
|
if (asJson) {
|
|
357
346
|
console.log(JSON.stringify({ config: CONFIG, ...loadConfig(), hubs }, null, 2));
|
|
@@ -527,6 +516,7 @@ if (cmd === "update") {
|
|
|
527
516
|
const doSchema = takeFlag(argv, "--schema", "-s");
|
|
528
517
|
const dryRun = takeFlag(argv, "--dry-run");
|
|
529
518
|
const fromOpt = takeOpt(argv, "--from");
|
|
519
|
+
rejectUnknownFlags(argv, "khb update [new-path] [--path|-p] [--schema|-s] [--from <old>] [--dry-run]");
|
|
530
520
|
const both = !doPath && !doSchema;
|
|
531
521
|
const [dest] = argv;
|
|
532
522
|
|
|
@@ -561,6 +551,7 @@ if (cmd === "update") {
|
|
|
561
551
|
// --------------------------------------------------------------------------- forget
|
|
562
552
|
|
|
563
553
|
if (cmd === "forget") {
|
|
554
|
+
rejectUnknownFlags(argv, "khb forget <name|path>");
|
|
564
555
|
const [what] = argv;
|
|
565
556
|
if (!what) {
|
|
566
557
|
console.error(`Usage: khb forget <name|path>`);
|
|
@@ -581,6 +572,7 @@ if (cmd === "forget") {
|
|
|
581
572
|
if (cmd === "agent") {
|
|
582
573
|
const command = takeOpt(argv, "--command");
|
|
583
574
|
const argsOpt = takeOpt(argv, "--args");
|
|
575
|
+
rejectUnknownFlags(argv, 'khb agent [name|none] [--command X] [--args "…"]');
|
|
584
576
|
const [name] = argv;
|
|
585
577
|
const cfg = loadConfig();
|
|
586
578
|
|
|
@@ -622,6 +614,7 @@ if (cmd === "agent") {
|
|
|
622
614
|
const wantPath = takeFlag(argv, "--path");
|
|
623
615
|
const noAgent = takeFlag(argv, "--no-agent") || wantPath;
|
|
624
616
|
const agentName = takeOpt(argv, "--agent");
|
|
617
|
+
rejectUnknownFlags(argv, "khb go [name|N] [--path] [--no-agent] [--agent X]");
|
|
625
618
|
const [what] = argv;
|
|
626
619
|
|
|
627
620
|
// Named target: resolve through the registry, then fall back to any path that is a hub —
|
|
@@ -9,13 +9,13 @@
|
|
|
9
9
|
// what the text *says* is the catalog pass (skills/catalog/SKILL.md).
|
|
10
10
|
import { readFileSync, existsSync } from "node:fs";
|
|
11
11
|
import { basename } from "node:path";
|
|
12
|
-
import { writeRaw, sha256File, rawNameFor, retargetRaw } from "../lib/util";
|
|
12
|
+
import { writeRaw, sha256, sha256File, rawNameFor, retargetRaw, normPath } from "../lib/util";
|
|
13
13
|
import { record, isFresh, identify, adopt, type Entry } from "../lib/ledger";
|
|
14
14
|
import {
|
|
15
15
|
extractCached, ocrCached, ocrImageCached, transcribeCached,
|
|
16
16
|
extractedBody, extractedPath, type Extraction,
|
|
17
17
|
} from "../lib/extract";
|
|
18
|
-
import { kindOf, extOf, mdName } from "./exts";
|
|
18
|
+
import { captionFor, kindOf, mediaFor, extOf, mdName } from "./exts";
|
|
19
19
|
import { detectPasswordProtected, PROTECTABLE } from "./protect";
|
|
20
20
|
import { item, note, outcome } from "../lib/log";
|
|
21
21
|
|
|
@@ -23,14 +23,22 @@ export type Options = {
|
|
|
23
23
|
force: boolean; // re-acquire even when the content hash is unchanged
|
|
24
24
|
ocr: boolean; // OCR scanned PDFs and images (default on; ~seconds/page)
|
|
25
25
|
audio: boolean; // transcribe audio/video (default on; ~minutes/file)
|
|
26
|
+
// Every path this source will visit, normalized. Only the caption/media pairing reads it
|
|
27
|
+
// — a sidecar may only be folded into a recording that is itself being acquired, or a
|
|
28
|
+
// `files:` source naming just the `.vtt` would acquire nothing at all.
|
|
29
|
+
scope?: ReadonlySet<string>;
|
|
26
30
|
};
|
|
27
31
|
|
|
32
|
+
/** Will this source reach that file? An unset scope has no opinion, so: yes. */
|
|
33
|
+
const willVisit = (opts: Options, path: string) => !opts.scope || opts.scope.has(normPath(path));
|
|
34
|
+
|
|
28
35
|
export type Counters = {
|
|
29
36
|
copied: number; // text files, taken verbatim
|
|
30
37
|
extracted: number; // converted this run
|
|
31
38
|
fromCache: number; // converted by an earlier run or another bundle
|
|
32
39
|
ocrd: number;
|
|
33
40
|
transcribed: number;
|
|
41
|
+
captioned: number; // read from a caption sidecar — the whisper run it saved
|
|
34
42
|
lowQuality: number; // OCR/ASR output — worth re-reading from source during curation
|
|
35
43
|
skipped: number; // unchanged since last ingest
|
|
36
44
|
moved: number; // same bytes at a new path — row re-pointed, nothing re-extracted
|
|
@@ -41,7 +49,7 @@ export type Counters = {
|
|
|
41
49
|
};
|
|
42
50
|
|
|
43
51
|
export const newCounters = (): Counters => ({
|
|
44
|
-
copied: 0, extracted: 0, fromCache: 0, ocrd: 0, transcribed: 0,
|
|
52
|
+
copied: 0, extracted: 0, fromCache: 0, ocrd: 0, transcribed: 0, captioned: 0,
|
|
45
53
|
lowQuality: 0, skipped: 0, moved: 0, pending: 0,
|
|
46
54
|
});
|
|
47
55
|
|
|
@@ -67,7 +75,26 @@ export async function acquireFile(
|
|
|
67
75
|
// real time, and a run that printed only successes left the slow file unnamed.
|
|
68
76
|
item(at, path);
|
|
69
77
|
const kind = kindOf(path);
|
|
70
|
-
|
|
78
|
+
|
|
79
|
+
// A caption sidecar is not a source of its own: `talk.vtt` beside `talk.mp4` is that
|
|
80
|
+
// recording's words, and the recording's row claims them below. Acquiring it here as well
|
|
81
|
+
// would spend a second raw/ file, a second uncurated ledger row and eventually a second
|
|
82
|
+
// concept on the same sentences. A caption with no recording beside it — or one whose
|
|
83
|
+
// recording this source will not visit — is a source like any other and falls through.
|
|
84
|
+
if (kind === "caption") {
|
|
85
|
+
const media = mediaFor(path);
|
|
86
|
+
if (media && willVisit(opts, media)) {
|
|
87
|
+
outcome(`captions for ${basename(media)} — acquired with the recording`);
|
|
88
|
+
return;
|
|
89
|
+
}
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
// Identity of a captioned recording is the pair's, not the file's. Hashing the media
|
|
93
|
+
// alone would let a corrected transcript sit next to an "unchanged, skipped" row forever.
|
|
94
|
+
const captions = kind === "av" ? captionFor(path) : undefined;
|
|
95
|
+
const own = await sha256File(path);
|
|
96
|
+
const capHash = captions ? await sha256File(captions) : kind === "caption" ? own : undefined;
|
|
97
|
+
const hash = captions ? sha256(`${own}:${capHash}`) : own;
|
|
71
98
|
if (kind === "skip") {
|
|
72
99
|
pend(entries, path, hash, c, "no extractor for this format");
|
|
73
100
|
return;
|
|
@@ -123,12 +150,16 @@ export async function acquireFile(
|
|
|
123
150
|
return;
|
|
124
151
|
}
|
|
125
152
|
|
|
126
|
-
|
|
153
|
+
// The extraction cache is keyed on the bytes that actually get converted, which for a
|
|
154
|
+
// captioned recording is the sidecar — so the same captions beside a re-encoded copy of
|
|
155
|
+
// the video, or ingested alone into another bundle, hit the same entry.
|
|
156
|
+
const key = capHash ?? hash;
|
|
157
|
+
const hit = existsSync(extractedPath(key));
|
|
127
158
|
let res: Extraction;
|
|
128
159
|
|
|
129
160
|
if (kind === "doc") {
|
|
130
161
|
note(hit ? `${ext.slice(1)} — reusing cached extraction` : `extracting ${ext.slice(1)} …`);
|
|
131
|
-
res = await extractCached(path,
|
|
162
|
+
res = await extractCached(path, key, ext);
|
|
132
163
|
// Pages but no text layer: the file is fine, the reader was wrong. OCR is the remedy,
|
|
133
164
|
// and running it here is what keeps ingest a single pass instead of a hunt afterwards.
|
|
134
165
|
if (res.status === "needs-ocr") {
|
|
@@ -137,7 +168,7 @@ export async function acquireFile(
|
|
|
137
168
|
return;
|
|
138
169
|
}
|
|
139
170
|
note(`no text layer, ${res.pages}p — scanned, running OCR (seconds per page)`);
|
|
140
|
-
res = await ocrCached(path,
|
|
171
|
+
res = await ocrCached(path, key);
|
|
141
172
|
if (res.status === "ok") c.ocrd++;
|
|
142
173
|
}
|
|
143
174
|
} else if (kind === "image") {
|
|
@@ -146,15 +177,22 @@ export async function acquireFile(
|
|
|
146
177
|
return;
|
|
147
178
|
}
|
|
148
179
|
note(hit ? "image — reusing cached OCR" : "image — running OCR …");
|
|
149
|
-
res = await ocrImageCached(path,
|
|
180
|
+
res = await ocrImageCached(path, key);
|
|
150
181
|
if (res.status === "ok" && !hit) c.ocrd++;
|
|
182
|
+
} else if (kind === "caption" || captions) {
|
|
183
|
+
// Ahead of the --skip-audio check on purpose: that flag exists to skip minutes of CPU
|
|
184
|
+
// per file, and reading a sidecar costs none. A pair is acquired even on a fast run.
|
|
185
|
+
const src = captions ?? path;
|
|
186
|
+
note(hit ? "captions — reusing cached extraction" : `reading captions from ${basename(src)} …`);
|
|
187
|
+
res = await extractCached(src, key, extOf(src));
|
|
188
|
+
if (res.status === "ok" && captions) c.captioned++;
|
|
151
189
|
} else {
|
|
152
190
|
if (!opts.audio) {
|
|
153
191
|
pend(entries, path, hash, c, "audio/video (--skip-audio)");
|
|
154
192
|
return;
|
|
155
193
|
}
|
|
156
|
-
note(hit ? "audio/video — reusing cached transcript" : "
|
|
157
|
-
res = await transcribeCached(path,
|
|
194
|
+
note(hit ? "audio/video — reusing cached transcript" : "no captions beside it — transcribing (minutes per file) …");
|
|
195
|
+
res = await transcribeCached(path, key);
|
|
158
196
|
if (res.status === "ok" && !hit) c.transcribed++;
|
|
159
197
|
}
|
|
160
198
|
|
|
@@ -165,14 +203,18 @@ export async function acquireFile(
|
|
|
165
203
|
|
|
166
204
|
// Copy out of the hash-keyed cache rather than moving: raw/ stays derived and the cache
|
|
167
205
|
// stays reusable by any other bundle that holds the same content.
|
|
168
|
-
|
|
206
|
+
// Name the sidecar in the provenance header, not just in this run's output: the recording
|
|
207
|
+
// is the source, but which file the words were read from is what a curator needs to know
|
|
208
|
+
// when the transcript and the audio disagree.
|
|
209
|
+
const tool = captions ? `${res.tool} (sidecar: ${basename(captions)})` : res.tool;
|
|
210
|
+
const raw = writeRaw(rawDir, file, { source: path, sha256: hash.slice(0, 12), tool, quality: res.quality }, extractedBody(res.path));
|
|
169
211
|
stamp(raw);
|
|
170
212
|
if (hit) c.fromCache++;
|
|
171
|
-
else if (kind === "doc") c.extracted++;
|
|
213
|
+
else if (kind === "doc" || kind === "caption") c.extracted++;
|
|
172
214
|
if (res.quality === "low") c.lowQuality++;
|
|
173
215
|
// Name the tool and the quality on the line: `quality: low` is the flag that tells
|
|
174
216
|
// curation to re-read the original, and burying it in the file made it easy to miss.
|
|
175
|
-
outcome(`${hit ? "cached" : "extracted"} → ${raw} [${
|
|
217
|
+
outcome(`${hit ? "cached" : "extracted"} → ${raw} [${tool}, quality: ${res.quality}]`);
|
|
176
218
|
}
|
|
177
219
|
|
|
178
220
|
export function report(c: Counters) {
|
|
@@ -184,6 +226,7 @@ export function report(c: Counters) {
|
|
|
184
226
|
line(c.fromCache, "reused from the extraction cache (inbox/extracted/)");
|
|
185
227
|
line(c.ocrd, "read by OCR");
|
|
186
228
|
line(c.transcribed, "transcribed");
|
|
229
|
+
line(c.captioned, "read from a caption sidecar (no transcription needed)");
|
|
187
230
|
line(c.lowQuality, "marked `quality: low` — verify against the source when curating");
|
|
188
231
|
line(c.pending, "not extracted (empty `raw` in log.md)");
|
|
189
232
|
}
|
package/scripts/ingest/exts.ts
CHANGED
|
@@ -2,6 +2,10 @@
|
|
|
2
2
|
//
|
|
3
3
|
// The kind decides which extractor runs, and every kind here has one — ingest is a single
|
|
4
4
|
// flat pass that converts everything it can, locally, with no agent turn in the middle.
|
|
5
|
+
import { readdirSync } from "node:fs";
|
|
6
|
+
import { dirname, basename, join } from "node:path";
|
|
7
|
+
import { normPath } from "../lib/util";
|
|
8
|
+
|
|
5
9
|
export const TEXT = [".md", ".txt", ".rst", ".adoc", ".html", ".csv", ".json", ".yaml", ".yml"];
|
|
6
10
|
/** Born-digital documents: khb's own pure-JS libraries read these, no system install. */
|
|
7
11
|
export const DOC = [".pdf", ".docx", ".odt", ".xlsx", ".pptx"];
|
|
@@ -9,6 +13,12 @@ export const DOC = [".pdf", ".docx", ".odt", ".xlsx", ".pptx"];
|
|
|
9
13
|
export const IMAGE = [".png", ".jpg", ".jpeg", ".webp", ".bmp", ".tif", ".tiff", ".gif"];
|
|
10
14
|
/** Audio and video: local whisper. Minutes of CPU per file, hence `--skip-audio`. */
|
|
11
15
|
export const AV = [".mp3", ".wav", ".m4a", ".flac", ".ogg", ".mp4", ".mov", ".mkv", ".webm"];
|
|
16
|
+
/**
|
|
17
|
+
* Subtitle sidecars: the words of a recording, already written down by someone who could
|
|
18
|
+
* hear it. Ordered by preference — the same captions often exist in both containers, and
|
|
19
|
+
* `.vtt` carries speaker names where `.srt` does not.
|
|
20
|
+
*/
|
|
21
|
+
export const CAPTION = [".vtt", ".srt"];
|
|
12
22
|
|
|
13
23
|
export const extOf = (p: string) => {
|
|
14
24
|
const i = p.lastIndexOf(".");
|
|
@@ -18,12 +28,83 @@ export const extOf = (p: string) => {
|
|
|
18
28
|
/** raw/ files are always markdown — don't produce "budget.md.md". */
|
|
19
29
|
export const mdName = (n: string) => (n.toLowerCase().endsWith(".md") ? n : n + ".md");
|
|
20
30
|
|
|
21
|
-
export type Kind = "text" | "doc" | "image" | "av" | "skip";
|
|
31
|
+
export type Kind = "text" | "doc" | "image" | "av" | "caption" | "skip";
|
|
22
32
|
export function kindOf(path: string): Kind {
|
|
23
33
|
const e = extOf(path);
|
|
24
34
|
if (TEXT.includes(e)) return "text";
|
|
25
35
|
if (DOC.includes(e)) return "doc";
|
|
26
36
|
if (IMAGE.includes(e)) return "image";
|
|
27
37
|
if (AV.includes(e)) return "av";
|
|
38
|
+
if (CAPTION.includes(e)) return "caption";
|
|
28
39
|
return "skip";
|
|
29
40
|
}
|
|
41
|
+
|
|
42
|
+
// --- sidecar pairing -------------------------------------------------------------------
|
|
43
|
+
//
|
|
44
|
+
// `talk.vtt` next to `talk.mp4` is not a second source, it is that recording's transcript.
|
|
45
|
+
// Pairing them is what lets ingest read the words instead of guessing at them with whisper,
|
|
46
|
+
// for no CPU and at higher fidelity. Both directions of the pairing are decided here so the
|
|
47
|
+
// two sides can never disagree about which file belongs to which.
|
|
48
|
+
|
|
49
|
+
/** One readdir per directory per run — a folder walk asks about the same siblings a lot. */
|
|
50
|
+
const listing = new Map<string, string[]>();
|
|
51
|
+
function siblings(path: string): { dir: string; names: string[] } {
|
|
52
|
+
const dir = dirname(path) || ".";
|
|
53
|
+
let names = listing.get(dir);
|
|
54
|
+
if (!names) {
|
|
55
|
+
try {
|
|
56
|
+
names = readdirSync(dir);
|
|
57
|
+
} catch {
|
|
58
|
+
names = []; // unreadable directory: no sidecar, and the file itself will say why
|
|
59
|
+
}
|
|
60
|
+
listing.set(dir, names);
|
|
61
|
+
}
|
|
62
|
+
return { dir, names };
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
const stemOf = (name: string) => name.slice(0, name.length - extOf(name).length);
|
|
66
|
+
|
|
67
|
+
/**
|
|
68
|
+
* A caption file's stem usually carries a language tag — `talk.en.vtt`, `talk.pt-BR.vtt`,
|
|
69
|
+
* which is what yt-dlp and every "download the subtitles" button writes — beside a plain
|
|
70
|
+
* `talk.mp4`. Strip one tag so the pair still matches.
|
|
71
|
+
*/
|
|
72
|
+
const untag = (stem: string) => stem.replace(/\.[a-z]{2,3}(?:-[A-Za-z]{2,4})?$/, "");
|
|
73
|
+
|
|
74
|
+
/**
|
|
75
|
+
* The caption sidecar belonging to a media file, when there is one obvious candidate.
|
|
76
|
+
*
|
|
77
|
+
* Two languages on disk is a choice about audience, and khb does not make choices: it
|
|
78
|
+
* transcribes instead and leaves the sidecars for a human to point at. The same captions in
|
|
79
|
+
* two containers is not a choice — they are the same words — so CAPTION order settles it.
|
|
80
|
+
*/
|
|
81
|
+
export function captionFor(media: string): string | undefined {
|
|
82
|
+
if (kindOf(media) !== "av") return undefined;
|
|
83
|
+
const { dir, names } = siblings(media);
|
|
84
|
+
const stem = stemOf(basename(media));
|
|
85
|
+
const captions = names.filter((n) => kindOf(n) === "caption");
|
|
86
|
+
const exact = captions.filter((n) => stemOf(n) === stem);
|
|
87
|
+
const pool = exact.length ? exact : captions.filter((n) => untag(stemOf(n)) === stem);
|
|
88
|
+
if (!pool.length) return undefined;
|
|
89
|
+
if (!exact.length && new Set(pool.map(stemOf)).size > 1) return undefined; // several languages
|
|
90
|
+
const best = [...pool].sort((a, b) => CAPTION.indexOf(extOf(a)) - CAPTION.indexOf(extOf(b)))[0];
|
|
91
|
+
return join(dir, best);
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
/**
|
|
95
|
+
* The media file a caption sidecar belongs to, if any — deliberately defined in terms of
|
|
96
|
+
* `captionFor`, so a caption is only somebody's sidecar when that somebody would actually
|
|
97
|
+
* claim it. A caption with no recording beside it is a source in its own right.
|
|
98
|
+
*/
|
|
99
|
+
export function mediaFor(caption: string): string | undefined {
|
|
100
|
+
if (kindOf(caption) !== "caption") return undefined;
|
|
101
|
+
const { dir, names } = siblings(caption);
|
|
102
|
+
const stems = new Set([stemOf(basename(caption)), untag(stemOf(basename(caption)))]);
|
|
103
|
+
for (const n of names) {
|
|
104
|
+
if (kindOf(n) !== "av" || !stems.has(stemOf(n))) continue;
|
|
105
|
+
const media = join(dir, n);
|
|
106
|
+
const found = captionFor(media);
|
|
107
|
+
if (found && normPath(found) === normPath(caption)) return media;
|
|
108
|
+
}
|
|
109
|
+
return undefined;
|
|
110
|
+
}
|
package/scripts/ingest/files.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
// Explicit file list → raw/files/. Same acquisition as a folder source, for the case where
|
|
2
2
|
// the interesting files are scattered and naming them is easier than naming a root.
|
|
3
3
|
import { existsSync } from "node:fs";
|
|
4
|
-
import { basename } from "../lib/util";
|
|
4
|
+
import { basename, normPath } from "../lib/util";
|
|
5
5
|
import type { Entry } from "../lib/ledger";
|
|
6
6
|
import { acquireFile, newCounters, report, type Options } from "./acquire";
|
|
7
7
|
import { detail, item, outcome, pos } from "../lib/log";
|
|
@@ -21,6 +21,11 @@ export async function ingestFiles(
|
|
|
21
21
|
const skippedCount = s.paths.length - paths.length;
|
|
22
22
|
if (skippedCount) detail(`${skippedCount} excluded by 'exclude' rule(s), ${paths.length} remain`);
|
|
23
23
|
|
|
24
|
+
// What this source will visit, for the caption/media pairing: a `.vtt` is only folded
|
|
25
|
+
// into a recording that is itself on the list, so naming just the sidecar still acquires
|
|
26
|
+
// it on its own.
|
|
27
|
+
const scoped = { ...opts, scope: new Set(paths.map((p) => normPath(p))) };
|
|
28
|
+
|
|
24
29
|
const c = newCounters();
|
|
25
30
|
for (const [i, p] of paths.entries()) {
|
|
26
31
|
const at = pos(i + 1, paths.length);
|
|
@@ -29,7 +34,7 @@ export async function ingestFiles(
|
|
|
29
34
|
outcome("missing, skipped");
|
|
30
35
|
continue;
|
|
31
36
|
}
|
|
32
|
-
await acquireFile(at, p, basename(p), rawDir, bundleDir, entries, c,
|
|
37
|
+
await acquireFile(at, p, basename(p), rawDir, bundleDir, entries, c, scoped);
|
|
33
38
|
}
|
|
34
39
|
report(c);
|
|
35
40
|
}
|
package/scripts/ingest/folder.ts
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
// owns the extraction decisions. Unchanged files (same content hash, raw/ copy still
|
|
3
3
|
// present) are skipped.
|
|
4
4
|
import { readdirSync, statSync, existsSync } from "node:fs";
|
|
5
|
-
import { join } from "../lib/util";
|
|
5
|
+
import { join, normPath } from "../lib/util";
|
|
6
6
|
import type { Entry } from "../lib/ledger";
|
|
7
7
|
import { acquireFile, newCounters, report, type Options } from "./acquire";
|
|
8
8
|
import { detail, pos } from "../lib/log";
|
|
@@ -43,12 +43,17 @@ export async function ingestFolder(
|
|
|
43
43
|
const skippedCount = all.length - files.length;
|
|
44
44
|
if (skippedCount) detail(`${skippedCount} excluded by 'exclude' rule(s), ${files.length} remain`);
|
|
45
45
|
|
|
46
|
+
// What this walk will visit, for the caption/media pairing: a `.vtt` whose recording is
|
|
47
|
+
// excluded (or simply absent) is acquired on its own rather than folded into a row that
|
|
48
|
+
// would never appear.
|
|
49
|
+
const scoped = { ...opts, scope: new Set(files.map((p) => normPath(p))) };
|
|
50
|
+
|
|
46
51
|
const c = newCounters();
|
|
47
52
|
for (const [i, p] of files.entries()) {
|
|
48
53
|
// Flatten the subtree into the filename so two `notes.md` in sibling folders don't
|
|
49
54
|
// collide in raw/, and so the origin stays legible without opening the file.
|
|
50
55
|
const rel = relOf(p).replaceAll("/", "__");
|
|
51
|
-
await acquireFile(pos(i + 1, files.length), p, rel, rawDir, bundleDir, entries, c,
|
|
56
|
+
await acquireFile(pos(i + 1, files.length), p, rel, rawDir, bundleDir, entries, c, scoped);
|
|
52
57
|
}
|
|
53
58
|
report(c);
|
|
54
59
|
}
|
package/scripts/ingest/index.ts
CHANGED
|
@@ -14,7 +14,7 @@ import { read, join, HUB } from "../lib/util";
|
|
|
14
14
|
import { detail, section, totalElapsed } from "../lib/log";
|
|
15
15
|
import { bundleForIngest, listBundles, DEFAULT_BUNDLE } from "../lib/scaffold";
|
|
16
16
|
import { readLedger, writeLedger } from "../lib/ledger";
|
|
17
|
-
import { takeFlag } from "../lib/args";
|
|
17
|
+
import { takeFlag, rejectUnknownFlags } from "../lib/args";
|
|
18
18
|
import { ingestFolder } from "./folder";
|
|
19
19
|
import { ingestFiles } from "./files";
|
|
20
20
|
import { ingestWeb } from "./web";
|
|
@@ -25,6 +25,7 @@ export type Source =
|
|
|
25
25
|
| { type: "files"; paths: string[]; exclude?: string[] }
|
|
26
26
|
| { type: "web"; urls: string[] };
|
|
27
27
|
|
|
28
|
+
const USAGE = "khb ingest [bundle] [--force] [--skip-ocr] [--skip-audio]";
|
|
28
29
|
const argv = process.argv.slice(2);
|
|
29
30
|
const opts: Options = {
|
|
30
31
|
force: takeFlag(argv, "--force"),
|
|
@@ -33,14 +34,10 @@ const opts: Options = {
|
|
|
33
34
|
ocr: !takeFlag(argv, "--skip-ocr"),
|
|
34
35
|
audio: !takeFlag(argv, "--skip-audio"),
|
|
35
36
|
};
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
console.error(`Unknown ingest flag: ${unknownFlag}`);
|
|
39
|
-
process.exit(1);
|
|
40
|
-
}
|
|
41
|
-
const positional = argv.filter((a) => !a.startsWith("--"));
|
|
37
|
+
rejectUnknownFlags(argv, USAGE);
|
|
38
|
+
const positional = argv;
|
|
42
39
|
if (positional.length > 1) {
|
|
43
|
-
console.error(
|
|
40
|
+
console.error(`Usage: ${USAGE}`);
|
|
44
41
|
process.exit(1);
|
|
45
42
|
}
|
|
46
43
|
// No bundle named: which one owns the material is a human decision (AGENTS.md) and a CLI
|
|
@@ -52,7 +49,7 @@ if (!bundle) {
|
|
|
52
49
|
const have = listBundles();
|
|
53
50
|
const onlyLanding = have.length === 1 && have[0] === DEFAULT_BUNDLE;
|
|
54
51
|
if (have.length && !onlyLanding) {
|
|
55
|
-
console.error(
|
|
52
|
+
console.error(`Usage: ${USAGE}`);
|
|
56
53
|
console.error(`\nBundles in this hub: ${have.join(", ")}`);
|
|
57
54
|
console.error(`Name the one that owns this material, or start a new one:`);
|
|
58
55
|
console.error(` khb new-bundle <name> "<scope>"`);
|
package/scripts/init.ts
CHANGED
|
@@ -10,7 +10,8 @@
|
|
|
10
10
|
// drift before any hub command.
|
|
11
11
|
import { resolve, basename } from "node:path";
|
|
12
12
|
import { MARKER, markerIn } from "./lib/paths";
|
|
13
|
-
import { upgradeHub, updateHint } from "./lib/upgrade";
|
|
13
|
+
import { recordLocation, upgradeHub, updateHint } from "./lib/upgrade";
|
|
14
|
+
import { takeOpt, rejectUnknownFlags } from "./lib/args";
|
|
14
15
|
|
|
15
16
|
const upgrading = process.env.KHB_SUBCOMMAND === "upgrade";
|
|
16
17
|
const argv = process.argv.slice(2);
|
|
@@ -18,23 +19,35 @@ const argv = process.argv.slice(2);
|
|
|
18
19
|
// A hub describes itself in its own marker, and the machine-level registry reads those
|
|
19
20
|
// two fields from there — so the label follows the hub when it is moved or cloned onto
|
|
20
21
|
// another machine, instead of living only in one laptop's shortcut list.
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
process.exit(1);
|
|
28
|
-
}
|
|
29
|
-
argv.splice(i, 2);
|
|
30
|
-
return v;
|
|
31
|
-
}
|
|
32
|
-
const nameOpt = takeOpt("--name");
|
|
33
|
-
const descOpt = takeOpt("--description");
|
|
22
|
+
// Both describe a hub being created, so `upgrade` must not consume them: left in argv they
|
|
23
|
+
// are refused below like any other unknown option, instead of being silently swallowed by a
|
|
24
|
+
// command whose help says it takes no flags at all.
|
|
25
|
+
const nameOpt = upgrading ? undefined : takeOpt(argv, "--name");
|
|
26
|
+
const descOpt = upgrading ? undefined : takeOpt(argv, "--description");
|
|
27
|
+
rejectUnknownFlags(argv, upgrading ? "khb upgrade" : 'khb init [dir] [--name N] [--description "…"]');
|
|
34
28
|
const [dirArg] = argv;
|
|
35
29
|
|
|
36
30
|
if (upgrading) {
|
|
37
31
|
const { HUB } = await import("./lib/util"); // resolves the hub, or exits with guidance
|
|
32
|
+
|
|
33
|
+
// cli.ts runs these two before every *other* in-hub command and skips `upgrade`, on the
|
|
34
|
+
// grounds that upgrade does the refresh itself. Neither of these is the refresh:
|
|
35
|
+
//
|
|
36
|
+
// - unregistered, a hub you only ever upgrade never appears in `khb list` or `khb go`;
|
|
37
|
+
// - unrecorded, a hub upgraded right after a move loses the move. `upgradeHub` stamps
|
|
38
|
+
// the marker with wherever the hub is now, so the old path has to be read — and
|
|
39
|
+
// appended to `movedFrom` — before that happens, or `khb update --path` is left with
|
|
40
|
+
// nothing to repair from and the arguments it exists to avoid.
|
|
41
|
+
const { registerHub, touchHub } = await import("./lib/registry");
|
|
42
|
+
registerHub(HUB);
|
|
43
|
+
touchHub(HUB);
|
|
44
|
+
const { moved } = recordLocation(HUB);
|
|
45
|
+
if (moved) {
|
|
46
|
+
console.log(`This hub was at ${moved} and is now at ${HUB}.`);
|
|
47
|
+
console.log(` absolute paths recorded inside it still name the old location.`);
|
|
48
|
+
console.log(` repair them: khb update --path (--dry-run to preview)`);
|
|
49
|
+
}
|
|
50
|
+
|
|
38
51
|
const { from, to, synced, pruned, renamed } = upgradeHub(HUB);
|
|
39
52
|
console.log(`Upgraded ${HUB}: ${from ?? "?"} -> ${to}`);
|
|
40
53
|
console.log(` refreshed: ${synced.join(", ")}`);
|
package/scripts/lib/args.ts
CHANGED
|
@@ -10,3 +10,40 @@ export function takeFlag(args: string[], ...names: string[]): boolean {
|
|
|
10
10
|
}
|
|
11
11
|
return false;
|
|
12
12
|
}
|
|
13
|
+
|
|
14
|
+
/**
|
|
15
|
+
* Take `--name value`, removing both. Also accepts `--name=value`, since that is the form
|
|
16
|
+
* fingers produce when the docs show the spaced one.
|
|
17
|
+
*/
|
|
18
|
+
export function takeOpt(args: string[], name: string): string | undefined {
|
|
19
|
+
const eq = args.findIndex((a) => a.startsWith(`${name}=`));
|
|
20
|
+
if (eq >= 0) {
|
|
21
|
+
const [v] = args.splice(eq, 1);
|
|
22
|
+
return v.slice(name.length + 1);
|
|
23
|
+
}
|
|
24
|
+
const i = args.indexOf(name);
|
|
25
|
+
if (i < 0) return undefined;
|
|
26
|
+
const v = args[i + 1];
|
|
27
|
+
if (v === undefined) {
|
|
28
|
+
console.error(`${name} needs a value`);
|
|
29
|
+
process.exit(1);
|
|
30
|
+
}
|
|
31
|
+
args.splice(i, 2);
|
|
32
|
+
return v;
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
/**
|
|
36
|
+
* Refuse anything flag-shaped this command does not understand. Call it once every
|
|
37
|
+
* `takeFlag`/`takeOpt` has removed what it consumed, so whatever remains is positional.
|
|
38
|
+
*
|
|
39
|
+
* Silence is the wrong default here: an unrecognized flag left in the array becomes a
|
|
40
|
+
* positional argument, and `khb export mybundle --force` used to quietly export into a
|
|
41
|
+
* directory named `--force`. A typo should cost an error message, not a mystery folder.
|
|
42
|
+
*/
|
|
43
|
+
export function rejectUnknownFlags(args: string[], usage: string): void {
|
|
44
|
+
const bad = args.find((a) => a.length > 1 && a.startsWith("-"));
|
|
45
|
+
if (!bad) return;
|
|
46
|
+
console.error(`Unknown option: ${bad}`);
|
|
47
|
+
console.error(`Usage: ${usage}`);
|
|
48
|
+
process.exit(1);
|
|
49
|
+
}
|
package/scripts/lib/extract.ts
CHANGED
|
@@ -77,8 +77,134 @@ function colIndex(ref: string): number {
|
|
|
77
77
|
return n - 1;
|
|
78
78
|
}
|
|
79
79
|
|
|
80
|
+
// --- captions --------------------------------------------------------------------------
|
|
81
|
+
//
|
|
82
|
+
// WebVTT and SRT are the same document in two dialects: an optional cue index, a timecode
|
|
83
|
+
// line, then the words. Everything but the words is scaffolding for a player, and reaches
|
|
84
|
+
// curation as noise — a transcript diced into three-second lines, every one of them
|
|
85
|
+
// preceded by a timestamp nobody will read.
|
|
86
|
+
//
|
|
87
|
+
// Two things are worth keeping. *When*, coarsely: a transcript nobody can point into is
|
|
88
|
+
// hard to cite, so anything longer than a chapter gets a heading per interval, enough to
|
|
89
|
+
// find the passage in the recording. And *who*, where the file says so: WebVTT's `<v Name>`
|
|
90
|
+
// is the only speaker attribution that survives from the original, and it is exactly what a
|
|
91
|
+
// meeting or an interview is read for.
|
|
92
|
+
//
|
|
93
|
+
// One thing is worth removing beyond the scaffolding. Auto-generated captions scroll: each
|
|
94
|
+
// cue repeats the tail of the one before it so the viewer sees a stable two-line window.
|
|
95
|
+
// Written down verbatim that doubles the transcript and reads as a stutter, so an
|
|
96
|
+
// overlapping head is trimmed instead of appended twice.
|
|
97
|
+
|
|
98
|
+
/** Seconds per `## h:mm:ss` heading, and the length below which a transcript gets none. */
|
|
99
|
+
const CHAPTER = 300;
|
|
100
|
+
|
|
101
|
+
/**
|
|
102
|
+
* Paragraph lengths, in characters. Captions have no paragraphs of their own — five minutes
|
|
103
|
+
* of speech arrives as one unbroken line — so they get made here: break at the first
|
|
104
|
+
* sentence end past SOFT, and give up and break anywhere past HARD, which is what
|
|
105
|
+
* auto-generated captions need, since they carry no punctuation to break on.
|
|
106
|
+
*/
|
|
107
|
+
const SOFT = 600;
|
|
108
|
+
const HARD = 1600;
|
|
109
|
+
|
|
110
|
+
/** "01:02:03.456" or "02:03.456" → seconds. */
|
|
111
|
+
function cueSeconds(t: string): number {
|
|
112
|
+
const m = t.match(/(?:(\d+):)?(\d{1,2}):(\d{2})[.,](\d{1,3})/);
|
|
113
|
+
if (!m) return 0;
|
|
114
|
+
return Number(m[1] ?? 0) * 3600 + Number(m[2]) * 60 + Number(m[3]) + Number(m[4]) / 1000;
|
|
115
|
+
}
|
|
116
|
+
|
|
117
|
+
/** A closing `.`, `?` or `!`, with the quote or bracket that may follow it. */
|
|
118
|
+
const SENTENCE_END = /[.!?]["')\]]?$/;
|
|
119
|
+
|
|
120
|
+
function stamp(s: number): string {
|
|
121
|
+
const pad = (n: number) => String(Math.floor(n)).padStart(2, "0");
|
|
122
|
+
return `${Math.floor(s / 3600)}:${pad((s % 3600) / 60)}:${pad(s % 60)}`;
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
/** A cue's payload: its speaker, if the file names one, and its words minus the markup. */
|
|
126
|
+
function cueText(lines: string[]): { speaker?: string; text: string } {
|
|
127
|
+
let speaker: string | undefined;
|
|
128
|
+
const text = lines
|
|
129
|
+
.join(" ")
|
|
130
|
+
// <v Roger Bingham> / <v.loud Esme> — the voice span names who is talking.
|
|
131
|
+
.replace(/<v[^\s>]*\s+([^>]*)>/g, (_, name: string) => {
|
|
132
|
+
speaker ??= name.trim();
|
|
133
|
+
return "";
|
|
134
|
+
})
|
|
135
|
+
// Everything else is presentation: <c> classes, </v>, and the inline <00:00:01.000>
|
|
136
|
+
// stamps that karaoke-style captions put between words.
|
|
137
|
+
.replace(/<[^>]*>/g, "")
|
|
138
|
+
.replace(/ /g, " ");
|
|
139
|
+
return { speaker, text: unxml(text).replace(/\s+/g, " ").trim() };
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
/**
|
|
143
|
+
* Append `next` to `text`, dropping the head the two share. Only a whole-word overlap of
|
|
144
|
+
* some length counts: trimming on a coincidental few characters would splice two different
|
|
145
|
+
* words into one, which is worse than the repetition it avoids.
|
|
146
|
+
*/
|
|
147
|
+
function joinOverlap(text: string, next: string): string {
|
|
148
|
+
for (let k = Math.min(next.length, 400); k >= 4; k--) {
|
|
149
|
+
if (!text.endsWith(next.slice(0, k))) continue;
|
|
150
|
+
const before = text[text.length - k - 1];
|
|
151
|
+
const after = next[k];
|
|
152
|
+
if ((before !== undefined && before !== " ") || (after !== undefined && after !== " ")) continue;
|
|
153
|
+
const rest = next.slice(k).trim();
|
|
154
|
+
return rest ? `${text} ${rest}` : text;
|
|
155
|
+
}
|
|
156
|
+
return `${text} ${next}`;
|
|
157
|
+
}
|
|
158
|
+
|
|
159
|
+
/** WebVTT/SRT → prose. Pure text in, pure text out; no file access, so it is easy to test. */
|
|
160
|
+
export function captionText(raw: string): string {
|
|
161
|
+
type Para = { at: number; speaker?: string; text: string };
|
|
162
|
+
const paras: Para[] = [];
|
|
163
|
+
let open: Para | undefined;
|
|
164
|
+
|
|
165
|
+
for (const block of raw.replace(/\r\n?/g, "\n").split(/\n{2,}/)) {
|
|
166
|
+
const lines = block.split("\n").map((l) => l.trim()).filter(Boolean);
|
|
167
|
+
// WEBVTT is the file signature; NOTE, STYLE and REGION blocks are metadata and CSS.
|
|
168
|
+
if (!lines.length || /^(WEBVTT|NOTE|STYLE|REGION)\b/.test(lines[0])) continue;
|
|
169
|
+
const timed = lines.findIndex((l) => l.includes("-->"));
|
|
170
|
+
if (timed === -1) continue; // no timecode: not a cue
|
|
171
|
+
const at = cueSeconds(lines[timed].split("-->")[0]);
|
|
172
|
+
const { speaker, text } = cueText(lines.slice(timed + 1));
|
|
173
|
+
if (!text) continue;
|
|
174
|
+
// A cue break means nothing — captions are cut to fit a screen, mid-sentence — so cues
|
|
175
|
+
// run together, and only a change of speaker or of chapter forces a new paragraph.
|
|
176
|
+
const newPara =
|
|
177
|
+
!open || open.speaker !== speaker || Math.floor(at / CHAPTER) !== Math.floor(open.at / CHAPTER);
|
|
178
|
+
if (newPara) paras.push((open = { at, speaker, text }));
|
|
179
|
+
else open.text = joinOverlap(open.text, text);
|
|
180
|
+
if (open.text.length >= (SENTENCE_END.test(open.text) ? SOFT : HARD)) open = undefined;
|
|
181
|
+
}
|
|
182
|
+
|
|
183
|
+
const out: string[] = [];
|
|
184
|
+
let chapter = -1;
|
|
185
|
+
let voice: string | undefined;
|
|
186
|
+
const long = paras.length > 0 && paras[paras.length - 1].at >= CHAPTER;
|
|
187
|
+
for (const p of paras) {
|
|
188
|
+
if (long && Math.floor(p.at / CHAPTER) !== chapter) {
|
|
189
|
+
chapter = Math.floor(p.at / CHAPTER);
|
|
190
|
+
voice = undefined; // name the speaker again under a new heading
|
|
191
|
+
out.push(`## ${stamp(chapter * CHAPTER)}`);
|
|
192
|
+
}
|
|
193
|
+
// Label a turn, not every paragraph of one: a monologue broken for length is still one
|
|
194
|
+
// person talking, and repeating their name down the page reads as a new speaker.
|
|
195
|
+
out.push(p.speaker && p.speaker !== voice ? `**${p.speaker}:** ${p.text}` : p.text);
|
|
196
|
+
voice = p.speaker;
|
|
197
|
+
}
|
|
198
|
+
return out.join("\n\n");
|
|
199
|
+
}
|
|
200
|
+
|
|
80
201
|
/** Built-in, pure-JS extractors. Loaded lazily so `khb init` never pays for them. */
|
|
81
202
|
const LIBRARY: Record<string, (file: string) => Promise<LibResult>> = {
|
|
203
|
+
// Subtitle sidecars. Text already, but wrapped in cue indices and timecodes, so they get
|
|
204
|
+
// an extractor rather than a verbatim copy — and the extraction cache with it, since the
|
|
205
|
+
// same captions beside a re-encoded copy of a video are the same words.
|
|
206
|
+
".vtt": async (file) => ({ text: captionText(readFileSync(file, "utf8")) }),
|
|
207
|
+
".srt": async (file) => ({ text: captionText(readFileSync(file, "utf8")) }),
|
|
82
208
|
".pdf": async (file) => {
|
|
83
209
|
const { extractText, getDocumentProxy } = await import("unpdf");
|
|
84
210
|
const doc = await getDocumentProxy(new Uint8Array(readFileSync(file)));
|
|
@@ -334,48 +460,163 @@ export async function ocrImageCached(path: string, hash: string): Promise<Extrac
|
|
|
334
460
|
}
|
|
335
461
|
}
|
|
336
462
|
|
|
337
|
-
/**
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
463
|
+
/**
|
|
464
|
+
* Is `vno` installed, and is it actually ready to run?
|
|
465
|
+
*
|
|
466
|
+
* `vno status` exists for exactly this: it reports and installs nothing, and exits non-zero
|
|
467
|
+
* when ffmpeg or whisper.cpp or a model is missing — so a vno that is installed but never
|
|
468
|
+
* set up is caught here rather than discovered one silent per-file failure at a time.
|
|
469
|
+
* `--json` turns the same answer into the blocker list that makes khb's message actionable.
|
|
470
|
+
*
|
|
471
|
+
* An older vno predating `status` would fail this check for the wrong reason, so a
|
|
472
|
+
* response that is not JSON at all falls back to the presence test it used to get. That
|
|
473
|
+
* costs a second spawn only on the rare path.
|
|
474
|
+
*/
|
|
475
|
+
type VnoState =
|
|
476
|
+
| { state: "ready" }
|
|
477
|
+
| { state: "unset-up"; blockers: string[] } // installed, but ffmpeg/whisper.cpp/model missing
|
|
478
|
+
| { state: "absent" };
|
|
479
|
+
|
|
480
|
+
async function vnoStatus(): Promise<VnoState> {
|
|
481
|
+
let out: string;
|
|
482
|
+
let code: number;
|
|
483
|
+
try {
|
|
484
|
+
const proc = Bun.spawn(["vno", "status", "--json"], { stdout: "pipe", stderr: "ignore", stdin: "ignore" });
|
|
485
|
+
out = await new Response(proc.stdout).text();
|
|
486
|
+
code = await proc.exited;
|
|
487
|
+
} catch {
|
|
488
|
+
return { state: "absent" }; // not on PATH — expected, and not worth a warning
|
|
489
|
+
}
|
|
490
|
+
if (code === 0) return { state: "ready" };
|
|
491
|
+
try {
|
|
492
|
+
const blockers = JSON.parse(out).blockers;
|
|
493
|
+
return { state: "unset-up", blockers: Array.isArray(blockers) ? blockers.map(String) : [] };
|
|
494
|
+
} catch {
|
|
495
|
+
return (await runCli(["vno", "--version"])) ? { state: "ready" } : { state: "absent" };
|
|
496
|
+
}
|
|
497
|
+
}
|
|
498
|
+
|
|
499
|
+
/**
|
|
500
|
+
* The local speech-to-text engine, probed once per process — spawning a status check per
|
|
501
|
+
* file would cost more than it saves.
|
|
502
|
+
*
|
|
503
|
+
* `vno` (@msareen/voice-notes-organizer) is preferred where it is set up. It is whisper.cpp
|
|
504
|
+
* under a wrapper, so it is markedly faster than the Python whisper on the same audio and
|
|
505
|
+
* uses whatever acceleration the machine has; it installs its own ffmpeg and model; and it
|
|
506
|
+
* emits WebVTT, which the caption reader above turns into a transcript with `## h:mm:ss`
|
|
507
|
+
* anchors instead of an undifferentiated wall of text. Same division of labor either way: a
|
|
508
|
+
* local binary doing a reproducible conversion, contacting no model.
|
|
509
|
+
*
|
|
510
|
+
* A vno that is installed but not set up is an amber gate, never a red one. Nothing about
|
|
511
|
+
* the run stops: whisper takes over if it is there, and if it is not, the recordings pend
|
|
512
|
+
* with an empty `raw` like every other unavailable extractor and the rest of the corpus is
|
|
513
|
+
* ingested regardless. All khb owes the user is an accurate reason and the one command that
|
|
514
|
+
* fixes it — running `vno setup` on their behalf would be khb installing software nobody
|
|
515
|
+
* asked it to install.
|
|
516
|
+
*/
|
|
517
|
+
type Engine = { bin: string; kind: "vno" | "whisper" };
|
|
518
|
+
let engine: Engine | null | undefined;
|
|
519
|
+
let vnoAmber: string | undefined; // set when vno is installed but not set up
|
|
520
|
+
|
|
521
|
+
async function asrEngine(): Promise<Engine | null> {
|
|
522
|
+
if (engine !== undefined) return engine;
|
|
523
|
+
|
|
524
|
+
const vno = await vnoStatus();
|
|
525
|
+
if (vno.state === "unset-up") {
|
|
526
|
+
const missing = vno.blockers.length ? `: ${vno.blockers.join(", ")}` : "";
|
|
527
|
+
vnoAmber = `vno is installed but not set up${missing} — run: vno setup`;
|
|
528
|
+
}
|
|
529
|
+
|
|
530
|
+
engine =
|
|
531
|
+
vno.state === "ready" ? { bin: "vno", kind: "vno" }
|
|
532
|
+
: (await runCli(["whisper", "--help"])) ? { bin: "whisper", kind: "whisper" }
|
|
533
|
+
: (await runCli(["faster-whisper", "--help"])) ? { bin: "faster-whisper", kind: "whisper" }
|
|
534
|
+
: null;
|
|
535
|
+
|
|
536
|
+
// Amber: say it once, then carry on with whatever else is available.
|
|
537
|
+
if (vnoAmber) console.warn(` ${vnoAmber}${engine ? ` (using ${engine.bin} instead)` : ""}`);
|
|
538
|
+
if (!engine) {
|
|
539
|
+
console.warn(` no transcriber ready — audio and video skipped, everything else proceeds. Install either:`);
|
|
540
|
+
if (!vnoAmber) console.warn(` npm install -g @msareen/voice-notes-organizer (whisper.cpp; installs its own deps)`);
|
|
541
|
+
console.warn(` pip install -U openai-whisper`);
|
|
542
|
+
}
|
|
543
|
+
return engine;
|
|
544
|
+
}
|
|
545
|
+
|
|
546
|
+
/**
|
|
547
|
+
* `vno t <file> -o <out> --no-open` — the one-shot path: no picker, no model prompt, and no
|
|
548
|
+
* requirement that the file live in vno's own library. The model is left to the user's vno
|
|
549
|
+
* settings rather than pinned here, since those are also where their acceleration lives.
|
|
550
|
+
*
|
|
551
|
+
* stdin is closed deliberately. vno offers to install a missing ffmpeg or whisper.cpp, and
|
|
552
|
+
* checks `isTTY` before asking — so a closed stdin turns that offer into printed
|
|
553
|
+
* instructions rather than a prompt with nobody there to answer it.
|
|
554
|
+
*/
|
|
555
|
+
async function transcribeVno(path: string, dest: string, out: string): Promise<Extraction> {
|
|
556
|
+
const tool = "vno (whisper.cpp)";
|
|
557
|
+
const vtt = join(out, "transcript.vtt");
|
|
558
|
+
const proc = Bun.spawn(["vno", "t", path, "-o", vtt, "--no-open"], {
|
|
559
|
+
stdout: "ignore",
|
|
560
|
+
stderr: "ignore",
|
|
561
|
+
stdin: "ignore",
|
|
562
|
+
});
|
|
563
|
+
await proc.exited;
|
|
564
|
+
// vno reports a missing file or an unusable dependency on stdout and still exits 0, so
|
|
565
|
+
// the transcript existing is the only signal worth trusting here.
|
|
566
|
+
if (!existsSync(vtt)) return { status: "failed", reason: "vno produced no transcript" };
|
|
567
|
+
const text = captionText(readFileSync(vtt, "utf8"));
|
|
568
|
+
if (!text) return { status: "failed", reason: "empty transcript" };
|
|
569
|
+
return { status: "ok", path: writeCache(dest, path, tool, "low", text), tool, quality: "low" };
|
|
570
|
+
}
|
|
571
|
+
|
|
572
|
+
/** OpenAI whisper (or faster-whisper): plain text, written into a scratch --output_dir. */
|
|
573
|
+
async function transcribeWhisper(
|
|
574
|
+
path: string,
|
|
575
|
+
dest: string,
|
|
576
|
+
out: string,
|
|
577
|
+
bin: string,
|
|
578
|
+
model: string,
|
|
579
|
+
): Promise<Extraction> {
|
|
580
|
+
const tool = `${bin} (${model})`;
|
|
581
|
+
const proc = Bun.spawn([bin, path, "--model", model, "--output_format", "txt", "--output_dir", out], {
|
|
582
|
+
stdout: "ignore",
|
|
583
|
+
stderr: "ignore",
|
|
584
|
+
stdin: "ignore",
|
|
585
|
+
});
|
|
586
|
+
if ((await proc.exited) !== 0) return { status: "failed", reason: `${bin} exited non-zero` };
|
|
587
|
+
const txt = readdirSync(out).find((f) => f.endsWith(".txt"));
|
|
588
|
+
if (!txt) return { status: "failed", reason: `${bin} produced no transcript` };
|
|
589
|
+
const text = readFileSync(join(out, txt), "utf8").trim();
|
|
590
|
+
if (!text) return { status: "failed", reason: "empty transcript" };
|
|
591
|
+
return { status: "ok", path: writeCache(dest, path, tool, "low", text), tool, quality: "low" };
|
|
346
592
|
}
|
|
347
593
|
|
|
348
594
|
/**
|
|
349
|
-
* Transcribe audio or video
|
|
350
|
-
*
|
|
595
|
+
* Transcribe audio or video locally. Video needs no demux step — both engines read the
|
|
596
|
+
* audio track directly.
|
|
351
597
|
*
|
|
352
598
|
* Minutes of CPU per file, so this is the one extractor worth interrupting: `khb ingest
|
|
353
|
-
* --skip-audio` leaves the rows pending and everything else proceeds.
|
|
354
|
-
*
|
|
599
|
+
* --skip-audio` leaves the rows pending and everything else proceeds. A recording that
|
|
600
|
+
* arrived with captions beside it never gets here at all; ingest reads those instead.
|
|
355
601
|
*/
|
|
356
602
|
export async function transcribeCached(path: string, hash: string, model = "base"): Promise<Extraction> {
|
|
357
603
|
const dest = extractedPath(hash);
|
|
358
604
|
if (existsSync(dest)) return cacheHit(dest);
|
|
359
605
|
|
|
360
|
-
const
|
|
361
|
-
|
|
606
|
+
const eng = await asrEngine();
|
|
607
|
+
// The pending row carries the reason a person can act on: "not set up" and "not installed"
|
|
608
|
+
// have different fixes, and the ledger is where this is read back weeks later.
|
|
609
|
+
if (!eng) return { status: "failed", reason: vnoAmber ?? "no transcriber installed" };
|
|
610
|
+
note(`transcribing with ${eng.bin}`);
|
|
362
611
|
|
|
363
|
-
//
|
|
364
|
-
//
|
|
612
|
+
// Both engines write files rather than to stdout; give each run a scratch directory of
|
|
613
|
+
// its own so a stray sibling transcript is never mistaken for this one's.
|
|
365
614
|
const out = join(INBOX, "tmp", hash.slice(0, 12));
|
|
366
615
|
mkdirSync(out, { recursive: true });
|
|
367
|
-
const tool = `${bin} (${model})`;
|
|
368
616
|
try {
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
});
|
|
373
|
-
if ((await proc.exited) !== 0) return { status: "failed", reason: `${bin} exited non-zero` };
|
|
374
|
-
const txt = readdirSync(out).find((f) => f.endsWith(".txt"));
|
|
375
|
-
if (!txt) return { status: "failed", reason: `${bin} produced no transcript` };
|
|
376
|
-
const text = readFileSync(join(out, txt), "utf8").trim();
|
|
377
|
-
if (!text) return { status: "failed", reason: "empty transcript" };
|
|
378
|
-
return { status: "ok", path: writeCache(dest, path, tool, "low", text), tool, quality: "low" };
|
|
617
|
+
return eng.kind === "vno"
|
|
618
|
+
? await transcribeVno(path, dest, out)
|
|
619
|
+
: await transcribeWhisper(path, dest, out, eng.bin, model);
|
|
379
620
|
} catch (e) {
|
|
380
621
|
return { status: "failed", reason: `transcription failed: ${e}` };
|
|
381
622
|
} finally {
|
package/scripts/lib/util.ts
CHANGED
|
@@ -124,6 +124,13 @@ export function retargetRaw(bundleDir: string, rawRel: string, source: string):
|
|
|
124
124
|
return true;
|
|
125
125
|
}
|
|
126
126
|
|
|
127
|
+
/**
|
|
128
|
+
* One spelling of a path, for comparing two of them. Absolute, and case-folded on Windows,
|
|
129
|
+
* where `D:\Corpus\Talk.mp4` and `d:/corpus/talk.mp4` are the same file.
|
|
130
|
+
*/
|
|
131
|
+
export const normPath = (p: string) =>
|
|
132
|
+
process.platform === "win32" ? resolve(p).toLowerCase() : resolve(p);
|
|
133
|
+
|
|
127
134
|
export const sha256 = (buf: Buffer | string) => createHash("sha256").update(buf).digest("hex");
|
|
128
135
|
|
|
129
136
|
/** Hash a file in chunks — corpora contain multi-GB binaries we must not slurp. */
|
package/scripts/lint.ts
CHANGED
|
@@ -4,6 +4,9 @@ import { detail, section, totalElapsed } from "./lib/log";
|
|
|
4
4
|
import { readdirSync, statSync } from "node:fs";
|
|
5
5
|
import { dirname, relative } from "node:path";
|
|
6
6
|
import { parse as parseYaml } from "yaml";
|
|
7
|
+
import { rejectUnknownFlags } from "./lib/args";
|
|
8
|
+
|
|
9
|
+
rejectUnknownFlags(process.argv.slice(2), "khb lint");
|
|
7
10
|
|
|
8
11
|
/** OKF v0.1 concept frontmatter. Unknown keys are warned, not rejected — OKF is permissive,
|
|
9
12
|
* but a `titel:` typo silently loses the field, so it is worth one line of noise. */
|
package/scripts/new-bundle.ts
CHANGED
|
@@ -3,8 +3,11 @@
|
|
|
3
3
|
import { existsSync } from "node:fs";
|
|
4
4
|
import { BUNDLES, join } from "./lib/util";
|
|
5
5
|
import { createBundle, VALID_NAME } from "./lib/scaffold";
|
|
6
|
+
import { rejectUnknownFlags } from "./lib/args";
|
|
6
7
|
|
|
7
|
-
const
|
|
8
|
+
const argv = process.argv.slice(2);
|
|
9
|
+
rejectUnknownFlags(argv, 'khb new-bundle <name> ["scope"]');
|
|
10
|
+
const [name, scope = "TODO scope"] = argv;
|
|
8
11
|
if (!name || !VALID_NAME.test(name)) {
|
|
9
12
|
console.error("Usage: khb new-bundle <name> [scope] (lowercase, digits, hyphens)");
|
|
10
13
|
process.exit(1);
|
package/scripts/visualize.ts
CHANGED
|
@@ -7,19 +7,24 @@
|
|
|
7
7
|
// process. `--port N` pins a specific port.
|
|
8
8
|
import { buildGraphData, readConceptFile } from "./lib/graph";
|
|
9
9
|
import { renderGraphPage } from "./lib/graph-page";
|
|
10
|
+
import { takeFlag, takeOpt, rejectUnknownFlags } from "./lib/args";
|
|
10
11
|
|
|
12
|
+
const USAGE = "khb visualize [--port N] [--no-open]";
|
|
11
13
|
const argv = process.argv.slice(2);
|
|
12
14
|
// port 0 asks the OS for any free port — no fixed default to collide with something else
|
|
13
|
-
// already running on the machine.
|
|
14
|
-
|
|
15
|
-
const
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
15
|
+
// already running on the machine. `--port N` and `--port=N` both pin it.
|
|
16
|
+
const portArg = takeOpt(argv, "--port");
|
|
17
|
+
const noOpen = takeFlag(argv, "--no-open");
|
|
18
|
+
rejectUnknownFlags(argv, USAGE);
|
|
19
|
+
|
|
20
|
+
// A port that is not a port used to become NaN and silently serve on a random one, which
|
|
21
|
+
// looks like the flag working right up until nothing is listening where you expected.
|
|
22
|
+
const requestedPort = portArg === undefined ? 0 : Number(portArg);
|
|
23
|
+
if (!Number.isInteger(requestedPort) || requestedPort < 0 || requestedPort > 65535) {
|
|
24
|
+
console.error(`--port must be a number between 0 and 65535, not: ${portArg}`);
|
|
25
|
+
console.error(`Usage: ${USAGE}`);
|
|
26
|
+
process.exit(1);
|
|
27
|
+
}
|
|
23
28
|
|
|
24
29
|
function summarize(data: ReturnType<typeof buildGraphData>) {
|
|
25
30
|
const concepts = Object.values(data.bundleGraphs).reduce((n, g) => n + g.concepts.length, 0);
|
package/skills/ingest/SKILL.md
CHANGED
|
@@ -117,6 +117,7 @@ locally. Read the summary it prints — the counts are the state of the world:
|
|
|
117
117
|
| `unchanged, skipped` | already acquired at this exact content hash |
|
|
118
118
|
| `extracted` / `reused from the extraction cache` | converted now / converted by an earlier run or another bundle |
|
|
119
119
|
| `read by OCR` / `transcribed` | lossy routes — see quality, below |
|
|
120
|
+
| `read from a caption sidecar` | a recording whose words were read off its `.vtt`/`.srt` instead |
|
|
120
121
|
| `marked quality: low` | verify these against the source when cataloging |
|
|
121
122
|
| `not extracted` | got a ledger row with an empty `raw`; the per-file line says why |
|
|
122
123
|
|
|
@@ -150,21 +151,78 @@ curation, not transcription.
|
|
|
150
151
|
| `.xlsx` | `fflate` → one markdown table per sheet | high |
|
|
151
152
|
| `.pdf` (scanned, no text layer) | `pdfium` + `tesseract.js`, automatically | **low** |
|
|
152
153
|
| `.png .jpg .webp .tif .gif` | `tesseract.js`, automatically | **low** |
|
|
153
|
-
| `.mp3 .wav .m4a .mp4 .mov .mkv` | local `whisper` / `faster-whisper` | **low** |
|
|
154
|
+
| `.mp3 .wav .m4a .mp4 .mov .mkv` | local `vno` (whisper.cpp), else `whisper` / `faster-whisper` | **low** |
|
|
155
|
+
| `.vtt .srt` | built-in caption reader | high |
|
|
154
156
|
|
|
155
157
|
Extracted text is cached hub-wide by content hash at `inbox/extracted/<sha256>.md`, so the
|
|
156
158
|
same file appearing in two bundles converts once.
|
|
157
159
|
|
|
160
|
+
**A recording next to its captions is one source, not two.** `talk.vtt` (or `talk.en.vtt`,
|
|
161
|
+
or `talk.srt`) beside `talk.mp4` is that recording's words, already written down by someone
|
|
162
|
+
who could hear it — so khb reads them instead of guessing at them with whisper. The pair
|
|
163
|
+
gets one `log.md` row, under the recording; the sidecar earns no row and no `raw/` file of
|
|
164
|
+
its own, and the recording's `extract_tool` names the file the text came from. It is both
|
|
165
|
+
free and better than transcription, so it happens even under `--skip-audio`.
|
|
166
|
+
|
|
167
|
+
Two things follow. The pair's identity is *both* files, so correcting a caption re-ingests
|
|
168
|
+
the recording rather than leaving a stale row marked unchanged. And khb never picks between
|
|
169
|
+
sidecars: `talk.en.vtt` next to `talk.fr.vtt` is a choice about audience, so it transcribes
|
|
170
|
+
instead and leaves both files to be pointed at explicitly. A caption with no recording
|
|
171
|
+
beside it — or one whose recording this source does not visit, because it is excluded or
|
|
172
|
+
simply not listed — is an ordinary source and gets its own row. That is also the lever:
|
|
173
|
+
excluding a sidecar does not unpair it, since `exclude` governs what earns a `raw/` file and
|
|
174
|
+
a paired sidecar never earns one; exclude the *recording* to have its captions ingested
|
|
175
|
+
alone.
|
|
176
|
+
|
|
177
|
+
The caption reader drops what belongs to the player and keeps what belongs to the
|
|
178
|
+
transcript: cue indices and timecodes go, `<v Name>` becomes a speaker label, the rolling
|
|
179
|
+
repetition auto-generated captions leave behind is collapsed, and anything longer than five
|
|
180
|
+
minutes gets a coarse `## h:mm:ss` heading per interval so a passage can be found in the
|
|
181
|
+
source recording. Quality is `high` — the words are what the file says, not what an
|
|
182
|
+
extractor guessed — but auto-generated captions are still ASR underneath, so treat a
|
|
183
|
+
transcript that reads like a machine wrote it the way you would treat one.
|
|
184
|
+
|
|
158
185
|
**OCR needs no setup.** `@hyzyla/pdfium`, `sharp` and `tesseract.js` are dependencies of khb
|
|
159
186
|
itself, so a scanned PDF or a photographed page is read on the first run, in any hub, without
|
|
160
187
|
asking the user to install anything.
|
|
161
188
|
|
|
162
|
-
Transcription is the one route that can be absent
|
|
163
|
-
|
|
189
|
+
Transcription is the one route that can be absent. It wants a transcriber on `PATH`, and
|
|
190
|
+
takes the first of these it finds:
|
|
164
191
|
|
|
165
192
|
```
|
|
166
|
-
|
|
193
|
+
npm install -g @msareen/voice-notes-organizer # vno — whisper.cpp, preferred
|
|
194
|
+
pip install -U openai-whisper # whisper (faster-whisper also works)
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
`vno` is preferred where both are set up: it is whisper.cpp rather than the Python
|
|
198
|
+
whisper, so it is markedly faster on the same audio and uses whatever acceleration the
|
|
199
|
+
machine has, it installs its own ffmpeg and model, and it emits WebVTT — which means a
|
|
200
|
+
transcript with `## h:mm:ss` anchors instead of an undifferentiated wall of text. khb runs
|
|
201
|
+
it as `vno t <file> -o <cache path> --no-open` with stdin closed, so nothing is written
|
|
202
|
+
beside your recordings and vno's setup offers degrade to printed instructions instead of
|
|
203
|
+
prompts.
|
|
204
|
+
|
|
205
|
+
khb gates on `vno status` before using it, because installed and ready are different things
|
|
206
|
+
— vno needs ffmpeg, whisper.cpp and a model, and reports on all three. **A vno that is not
|
|
207
|
+
set up is an amber gate, never a red one.** The run does not stop and nothing else is
|
|
208
|
+
affected: whisper takes over if you have it, and if you don't, the recordings pend with an
|
|
209
|
+
empty `raw` exactly like any other unavailable extractor while the rest of the corpus is
|
|
210
|
+
ingested normally. What you get is the reason and the fix, on the file's own line and again
|
|
211
|
+
in `log.md`:
|
|
212
|
+
|
|
167
213
|
```
|
|
214
|
+
[ 3/12] D:\corpus\standup.m4a
|
|
215
|
+
no captions beside it — transcribing (minutes per file) …
|
|
216
|
+
vno is installed but not set up: ffmpeg, whisper.cpp — run: vno setup
|
|
217
|
+
pending — vno is installed but not set up: ffmpeg, whisper.cpp — run: vno setup
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
Run `vno setup` yourself and re-run the ingest; the pending rows fill in. khb will not run
|
|
221
|
+
it for you — installing software nobody asked it to install is not a conversion step.
|
|
222
|
+
|
|
223
|
+
Either engine is a local binary doing a reproducible conversion, and its output is
|
|
224
|
+
`quality: low` all the same: it is a machine's guess at audio, and the recording is still
|
|
225
|
+
the thing to re-read when a passage looks wrong.
|
|
168
226
|
|
|
169
227
|
When any extractor is unavailable khb says so once and records the affected files as pending
|
|
170
228
|
rather than failing the run — a `log.md` row with an empty `raw`, waiting for the dependency.
|
package/templates/hub/gitignore
CHANGED
|
@@ -5,8 +5,9 @@ bundles/*/raw/
|
|
|
5
5
|
.claude/settings.local.json
|
|
6
6
|
|
|
7
7
|
# Extraction cache (inbox/extracted/<sha256>.md) — re-derivable from the sources.
|
|
8
|
-
inbox/
|
|
8
|
+
/inbox/
|
|
9
9
|
|
|
10
|
-
# Generated
|
|
10
|
+
# Generated. The leading slash matters: a bare `export/` also matches the `skills/export/`
|
|
11
|
+
# these files ship with, and silently drops the export skill from the hub's own history.
|
|
11
12
|
visualizer/graph.html
|
|
12
|
-
export/
|
|
13
|
+
/export/
|