@dzhechkov/skills-book-digitizer 0.2.0 → 0.2.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -12,7 +12,7 @@ Knowledge Base behind them. dz-original; design & rationale in
12
12
  methods at the decisions it's about — e.g. DDIA → `ddia-replication-topology-choice`,
13
13
  `ddia-partitioning-strategy`, `ddia-consistency-model-selection`.
14
14
 
15
- ## The 6 skills (a checkpointed pipeline)
15
+ ## The skills (a checkpointed pipeline + one sibling)
16
16
 
17
17
  | Skill | Stage |
18
18
  |-------|-------|
@@ -20,8 +20,9 @@ methods at the decisions it's about — e.g. DDIA → `ddia-replication-topology
20
20
  | `book-ingest` | PDF → chunked, page-anchored corpus + manifest (deterministic: poppler + heuristics, no LLM) |
21
21
  | `book-knowledge-extract` | corpus → **verified** Knowledge Units (map-reduce, dedup, KU-verify entailment, FACTS-verbatim/PROSE-paraphrase regime) |
22
22
  | `book-skill-distill` | verified KUs → **decision-moment** skills (slug-prefixed, routing evals + activation gate, KU→skill traceability) |
23
- | `book-pack-assemble` | skills → installable pack (`references/` deep-lookup shipped inside, **private-by-default** distribution, collision gate, Copilot policy) |
23
+ | `book-pack-assemble` | skills → installable pack (`references/` deep-lookup + the per-book `brain/<slug>.sqlite` KB slice shipped inside, **private-by-default** distribution, collision gate, Copilot policy) |
24
24
  | `book-kb-index` | KUs → the memory layer: `.dz/memory/books.sqlite` (lexical) + shared agentdb (vector) |
25
+ | `source-brain-ingest` | **Sibling (not a book stage, ADR §6)** — deep-walk a **repo/dir** → the *same* Knowledge Units → the *same* brain via `dz brain add --from-kus … --kind repo`. One brain, two ingest front-ends (books + code). |
25
26
 
26
27
  ## Install & use
27
28
 
@@ -33,6 +34,83 @@ dz init --target claude-code --select digitize-book
33
34
  Prerequisites: `poppler-utils` (ingest) and, for the KB, `dz setup --memory agentdb`
34
35
  (installs `agentdb` + `better-sqlite3`).
35
36
 
37
+ ## Usage scenarios
38
+
39
+ The digitizer produces three durable artifacts from one book: (1) an installable **skill pack**, (2) a
40
+ per-project **two-store KB** (`book-kb-index` → `.dz/memory/books.sqlite` lexical + shared agentdb
41
+ vector), and (3) a promote path into a durable cross-project **Knowledge Brain** (`~/.dz/brain/`).
42
+ Full design + the RU version: [ADR-001 Knowledge Brain](https://github.com/djd1m/dz-harness-hub/blob/main/features/book-brain/ADR-001-knowledge-brain.md).
43
+
44
+ ### 1. Digitize a book → skills + KB
45
+
46
+ The core pipeline. Install the orchestrator, then hand it a PDF; it runs `book-ingest → extract →
47
+ distill → assemble → book-kb-index` through 5 human checkpoints.
48
+
49
+ ```bash
50
+ dz init --target claude-code --select digitize-book
51
+ # then, in your agent: «оцифруй вот эту книгу — хочу, чтобы агент проектировал по её методикам»
52
+ ```
53
+
54
+ Output: a `skills-book-<slug>` pack (decision-moment skills) **and** the two-store KB behind it. Live
55
+ run: DDIA → «Высоконагруженные приложения» digitized to **116 Knowledge Units**.
56
+
57
+ ### 2. Query one book's knowledge
58
+
59
+ Lexical (FTS5) recall over a single digitized book's KB, in the project that ingested it. `--book`
60
+ narrows the search to one book slug:
61
+
62
+ ```bash
63
+ dz recall --books --book <slug> "replication topology"
64
+ ```
65
+
66
+ Pull-based: you ask, you get KU hits back with source + page anchors. No hook, no injection.
67
+
68
+ ### 3. Build a cross-project brain from N books
69
+
70
+ Promote each project's digitized-book KUs into **one shared brain** (`~/.dz/brain/` — lexical
71
+ `books.sqlite` + vector `agentdb.db`), then query across every ingested book at once:
72
+
73
+ ```bash
74
+ dz brain add --source <slug> # promote this project's KUs into the brain (idempotent)
75
+ dz brain list # what's in the brain: slug, kind, KU count, corpus version
76
+ dz brain query "replication topology" # cross-source recall over ALL books
77
+ dz brain query "replication topology" --source <slug> # optional: narrow to one book
78
+ ```
79
+
80
+ One shared store, not per-book packages: you do **not** install anything to query the brain — it
81
+ accretes on `dz brain add`. The *right* book surfaces by **relevance** (KUs rank on one common scale,
82
+ each hit labelled with its source book + page); `--source` is a label filter, not a manual routing
83
+ pick. Cross-source is the default — multiple books scoring on a query is a feature (richer grounding),
84
+ not a bug.
85
+
86
+ ### 4. Grounded answering (the hook)
87
+
88
+ ```bash
89
+ dz brain init # opt-in: wire the grounding UserPromptSubmit hook (once)
90
+ dz brain ground "<prompt>" # what the hook runs: retrieve + emit citations (or nothing)
91
+ ```
92
+
93
+ `dz brain init` (opt-in) wires a single `UserPromptSubmit` hook that runs `dz brain ground` on each
94
+ prompt. On a *relevant* prompt (relevance-gated, so it stays quiet on unrelated turns) the hook
95
+ mechanically retrieves top-K brain KUs and injects them — with source + page citations — into that
96
+ turn's context, so the agent answers **from the book, not from training data**. Off by default; a
97
+ prompt with no lexical overlap (e.g. "what's the weather") injects nothing.
98
+
99
+ Honesty boundary: the injection is **mechanical** (the hook forces the grounding block into context and
100
+ the agent literally cannot start the turn without it); "answering from it" is **agent discipline** — the
101
+ hook can force the context but not the completion. The hook is **generic and wired once** (one hook over
102
+ the whole brain across all sources — *not* per book), and needs **no edits to `CLAUDE.md` or governance
103
+ shards** — the injection channel is the hook's stdout (`hookSpecificOutput.additionalContext`).
104
+
105
+ ### 5. Sharing a digitized book
106
+
107
+ The `skills-book-<slug>` **pack** (via GitHub / `npx`) carries the behavior — the decision-moment
108
+ skills and `references/` deep-lookup. The design (ADR §8.1, P2) also ships the book's **KB slice inside
109
+ the pack** under the same CP5 licence decision that gates the pack itself: copyrighted books stay
110
+ `private: true` (structural, mechanically enforced — no auto-publish, no preset entry), permissive
111
+ sources are publishable. Your whole personal **brain is never published** — it is a local accretion you
112
+ back up, not a distributable.
113
+
36
114
  ## Design guarantees — structural vs. protocol (stated honestly)
37
115
 
38
116
  Two kinds of guarantee, and the difference matters:
@@ -34,6 +34,7 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
34
34
  <slug>-<decision>/SKILL.md
35
35
  <slug>-<decision>/references/*.md ← the deep-lookup, shipped IN the pack
36
36
  <slug>-<decision>/evals/routing.yaml
37
+ brain/<slug>.sqlite ← the per-book KB slice, shipped IN the pack (step 9)
37
38
  package.json sources.json README.md LICENSE
38
39
  ```
39
40
  2. **`references/` ships the deep-lookup inside the pack** — each skill's consumed KUs in full
@@ -41,8 +42,8 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
41
42
  every target/machine, not a call to a KB the recipient doesn't have. `sources.json` declares
42
43
  which lookup tiers exist per skill: `references/` = always, corpus = owner-local, agentdb =
43
44
  owner-local — the skill degrades honestly.
44
- 3. **`files[]` whitelist** every skill dir + `references`/`evals` + `sources.json` + `README.md` +
45
- `LICENSE` (forget one → npm silently drops it). Verify the tarball after.
45
+ 3. **`files[]` whitelist** every skill dir + `references`/`evals` + `brain/` (the KB slice, step 9) +
46
+ `sources.json` + `README.md` + `LICENSE` (forget one → npm silently drops it). Verify the tarball after.
46
47
  4. **`package.json` distribution guardrails (structural, not policy):**
47
48
  `"private": true` + `"publishConfig": { "access": "restricted" }`. **No automatic
48
49
  `harness-presets` entry** (presets are the public install surface). Flipping to publishable
@@ -68,6 +69,19 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
68
69
  (re-paraphrase) and re-run. Cited ≤25-word quotes with a page anchor are exempt. Only once it
69
70
  passes: `dz benchmark <pack> --all` (honest L0 grade per skill — never fabricate schemas/evals to
70
71
  game it) → `dz init --target claude-code --select <skills>` install smoke → **CP4** report.
72
+ 9. **Ship the per-book KB slice inside the pack (ADR §8.1)** — once skills + `references/` are
73
+ assembled, export the book's KUs as a standalone slice:
74
+ `dz brain export --source <slug> --out brain/<slug>.sqlite` (this book's rows only —
75
+ `WHERE book=<slug>`, **lexical-only** for portability — the recipient's `dz brain add --from-pack`
76
+ re-embeds the vectors into their own brain on import). Whitelist `brain/` in `files[]` (step 3)
77
+ and pass the slice **through the same shingling IP gate** (step 8) as the rest of the pack — a
78
+ verbatim leak in the KB slice fails the pack just like one in a SKILL.md. The slice rides the pack
79
+ under the **same CP5 licence decision**: a copyrighted book stays `private: true` (slice never
80
+ auto-published); a permissive source is publishable (slice ships). A recipient loads it with
81
+ `dz brain add --from-pack @dzhechkov/skills-book-<slug>` — an idempotent upsert on
82
+ `(book, ku_id, corpus_version)` into their `~/.dz/brain/` — so one `npx`/github install carries
83
+ BOTH the behavior (skills) AND the knowledge (the brain slice), inseparably, under one decision.
84
+ Document the `--from-pack` load in the pack README.
71
85
 
72
86
  ## Anti-patterns
73
87
 
@@ -79,6 +93,8 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
79
93
  | `origin` block in sources.json | sync-upstream would chase a non-repo | documentation-only, no origin |
80
94
  | Fat SKILL.md bodies on Copilot | floods every request | lean body + `references/` |
81
95
  | Shipping a pack without the shingling gate | uncited verbatim runs = IP violation | run `shingling-check.mjs` before the L0 benchmark; re-paraphrase any hit |
96
+ | Publishing the KB slice separately from the pack's CP5 decision | book knowledge leaks out from under the licence gate | slice rides the pack: `private` book → private slice, permissive → publishable |
97
+ | KB slice skipping the shingling gate | a verbatim leak in the slice = same IP violation | pass `brain/<slug>.sqlite` through the shingling gate like the rest of the pack |
82
98
 
83
99
  ## Self-check
84
100
 
@@ -88,6 +104,7 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
88
104
  - [ ] Collision gate clean; slug-prefixed unique ids?
89
105
  - [ ] Faithfulness gate run (each shipped skill's claims supported by its `derived_from` KUs)?
90
106
  - [ ] Shingling gate passed (no uncited verbatim run ≥8 words; cited ≤25-word quotes anchored)?
107
+ - [ ] KB slice `brain/<slug>.sqlite` exported, whitelisted in `files[]`, passed the shingling gate, and follows the pack's CP5 licence (private book → private slice); `--from-pack` load documented in README?
91
108
  - [ ] L0 grades honest; install smoke passed; Copilot bytes shown if relevant?
92
109
 
93
110
  ## Examples
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@dzhechkov/skills-book-digitizer",
3
- "version": "0.2.0",
3
+ "version": "0.2.4",
4
4
  "description": "Canonical book-digitizer skill pack — 6 skills that turn an engineering book/textbook/monograph into an installable skill pack whose skills make AI coders work within the book's methodologies: digitize-book (orchestrator), book-ingest, book-knowledge-extract, book-skill-distill, book-pack-assemble, book-kb-index.",
5
5
  "license": "MIT",
6
6
  "author": "dzhechko",
@@ -22,6 +22,7 @@
22
22
  "book-skill-distill",
23
23
  "book-pack-assemble",
24
24
  "book-kb-index",
25
+ "source-brain-ingest",
25
26
  "scripts",
26
27
  "sources.json",
27
28
  "README.md",
@@ -36,4 +37,4 @@
36
37
  "directory": "packages/@dzhechkov/skills-book-digitizer"
37
38
  },
38
39
  "homepage": "https://github.com/djd1m/dz-harness-hub/tree/main/packages/@dzhechkov/skills-book-digitizer#readme"
39
- }
40
+ }
File without changes
@@ -0,0 +1,129 @@
1
+ ---
2
+ name: source-brain-ingest
3
+ description: >
4
+ The repo/dir SIBLING of the book pipeline (ADR-001 §6): deep-walk a codebase or directory
5
+ (respect .gitignore, skip binaries/node_modules/dist, cap size), then distill DECISION-GRADE
6
+ Knowledge Units in the SAME BookKU shape the book pipeline emits — and feed them into the SAME
7
+ shared brain register via `dz brain add --from-kus`, WITHOUT the book-only stages (no pdftotext
8
+ ingest, no skill distillation). Same KU schema = free reuse of the whole brain (query, primer,
9
+ grounding). Records the source's SPDX license and gates on it. Triggers on: "ingest a repo into
10
+ the brain", "source-brain-ingest", "add this codebase to my knowledge brain".
11
+ trust_tier: 1
12
+ trust_tier_label: "Structured"
13
+ trust_tier_path: "Run /bto-test to promote to Tier 2"
14
+ ---
15
+
16
+ # source-brain-ingest — repo/dir → brain KUs (the non-book front-end)
17
+
18
+ One brain, **two ingest front-ends** (ADR §6): the book pipeline turns PDFs into KUs; this turns a
19
+ **repo or directory** into KUs — the *same* KU shape, the *same* register stage, the *same* query /
20
+ primer / grounding surface. A repo plugs into everything the book path already built, for free. This
21
+ is a **sibling** skill — it does not touch, and must not contort, the intact book pipeline.
22
+
23
+ ## When to use / NOT
24
+
25
+ - **Use** to distill a codebase/dir into decision-grade Knowledge Units and promote them into the
26
+ durable brain (`~/.dz/brain/`) so `dz brain query` / primers / the grounding hook cover repos too.
27
+ - **NOT for books/PDFs** → use `digitize-book` (pdftotext ingest is book-shaped and would be
28
+ contorted by repo logic; keeping it pure protects the proven book path).
29
+ - **NOT to author skills** → that's `book-skill-distill`. This stage emits KUs into the brain only;
30
+ it does not produce a `skills-<slug>` pack (repos may later return whole files, §8/G5, where books
31
+ return distilled KUs for IP reasons).
32
+ - **NOT to extract/verify book KUs** → that's `book-knowledge-extract`.
33
+
34
+ ## Prerequisites
35
+
36
+ - A local repo or directory (a checkout, or a `git clone` you control).
37
+ - `git` (to honour `.gitignore` and capture the commit for provenance) — degrade gracefully on a
38
+ non-git dir (walk the tree; `commit: null`).
39
+ - The brain stores (`dz setup --memory agentdb`) — the same `~/.dz/brain/` the book path promotes into.
40
+ - The source's **SPDX license identifier** (read `LICENSE`/`package.json`; pass `--license <spdx>`).
41
+
42
+ ## The Knowledge Unit schema (identical to the book pipeline — that's the whole point)
43
+
44
+ Emit the **same BookKU shape** `book-knowledge-extract` emits, so a repo is just another `source`:
45
+
46
+ ```yaml
47
+ id: <repo-slug>-<path-hash>-kuNN # GLOBALLY UNIQUE: <path-hash> discriminates the file/chunk
48
+ # (short hash of the repo-relative path) so two files never
49
+ # collide on kuNN — the brain upsert keys on (source, ku_id,
50
+ # corpus_version) and silently drops id collisions otherwise.
51
+ type: methodology | decision-framework | formula | heuristic | checklist |
52
+ tradeoff-table | case-pattern | definition
53
+ name: "Retry-with-jitter backoff policy"
54
+ problem: the decision/task this knowledge serves (when/why it applies)
55
+ content: the paraphrased procedure / framework / restructured table / verbatim signature-or-formula
56
+ applicability: preconditions, scale ranges, contexts where it holds
57
+ limits: tradeoffs, failure modes, when NOT to apply
58
+ chapter: src/net/backoff.ts # PATH takes the book's `chapter` slot
59
+ pages: [40, 88] # LINE-RANGE takes the book's `pages` slot
60
+ sources: [{ repo: <repo-slug>, path: src/net/backoff.ts, lines: [40, 88], commit: <sha> }]
61
+ metadata: { license: <spdx>, corpus_version: <sha-or-ts>, lang: <detected> }
62
+ verified: true | false | partial # entailment against the cited lines
63
+ ```
64
+
65
+ Because the shape is identical, the brain treats books and repos on one common scale (query, primer,
66
+ grounding all work unchanged). `kind: repo` in the registry lets the brain treat repos **distinctly**
67
+ where it matters (e.g. return whole files, §8/G5) without a second schema.
68
+
69
+ ## Protocol
70
+
71
+ 1. **Walk** — deep-walk the repo/dir. **Respect `.gitignore`** (use `git ls-files` when it's a git
72
+ repo — free `.gitignore` honouring + tracked-only). **Skip** `node_modules/`, `dist/`, `build/`,
73
+ `.git/`, vendored trees, lockfiles, and **binaries** (detect by extension + a NUL-byte sniff).
74
+ **Cap** per-file size (skip/point at oversized generated files) and total bytes scanned; log what
75
+ was skipped so coverage is honest.
76
+ 2. **Chunk** — split each kept file into page-analogue units (a function/class/section, or a
77
+ line-window for prose docs). Anchor every chunk to `path` + `[startLine–endLine]` — the analogue
78
+ of the book's `[p.N]` anchor; these anchors make the verify step a cheap lookup.
79
+ 3. **Distill KUs** — emit **decision-grade** Knowledge Units in the schema above (a decision the code
80
+ encodes, a policy, a tradeoff, a reusable pattern) — NOT a file summary. **Fidelity regime** (same
81
+ as the book path): FACTS (signatures, constants, config values, exact code) **verbatim** with a
82
+ `path:line` anchor; PROSE (rationale, README narrative) **paraphrased** — no uncited verbatim run
83
+ ≥ 8 words (this is what a downstream shingling gate checks). Then a fresh checker re-opens each
84
+ KU's cited `path:line` range and sets `verified` (entailment gate; verify all, lookup ≪ generation).
85
+ 4. **License-check (the IP gate)** — record the source's SPDX license in every KU's `metadata.license`
86
+ and in what you pass to the register. The register **refuses** an unknown / incompatible license
87
+ without an explicit `--override` — so an unlicensed or GPL-incompatible repo cannot silently land
88
+ in the brain. Repo license travels with the KUs; if the brain ever returns whole files (§8/G5),
89
+ that license binds any downstream reuse.
90
+ 5. **Promote** — write the verified KUs to a JSON file in the BookKU shape and hand them to the SAME
91
+ register the book path uses:
92
+ ```bash
93
+ dz brain add --from-kus <file.json> --slug <repo-slug> --kind repo --license <spdx>
94
+ ```
95
+ This idempotently upserts on `(source, ku_id, corpus_version)` into `~/.dz/brain/` (lexical
96
+ `books.sqlite` + vector `agentdb.db`), writes the `brain.json` registry entry (`kind: repo`,
97
+ license, corpus_version, ku_count), and generates the primer — **no book-ingest, no
98
+ book-skill-distill in the path**. Verify: `dz brain query "<a KU name>"` returns it labeled with
99
+ the repo source.
100
+
101
+ ## Anti-patterns
102
+
103
+ | Anti-pattern | Why it fails | Instead |
104
+ |--------------|--------------|---------|
105
+ | Inventing a repo-specific KU schema | breaks the shared brain (query/primer/grounding assume one shape) | emit the exact BookKU schema — path→chapter, lines→pages |
106
+ | Ignoring `.gitignore` / walking `node_modules` | ingests junk + secrets, blows the token budget | `git ls-files`; skip binaries/vendored/dist; cap size |
107
+ | `kuNN` id without a path discriminator | two files collide → upsert-by-id drops KUs | `<repo-slug>-<path-hash>-kuNN` |
108
+ | Routing a repo through `book-ingest`/`digitize-book` | pdftotext + book stages don't fit code | this sibling → `dz brain add --from-kus` |
109
+ | Distilling repo KUs into a skill pack | that's the book path's job; repos may return whole files (§8) | KUs into the brain only |
110
+ | Promoting an unknown/incompatible license | leaks IP-encumbered code into the durable brain | license-check gate; `--override` is an explicit decision |
111
+ | File-summary "KUs" | RAG noise, not decisions | decision-grade KUs (policy/tradeoff/pattern) |
112
+
113
+ ## Self-check
114
+
115
+ - [ ] Walk respected `.gitignore`, skipped binaries/`node_modules`/`dist`, capped size — skips logged?
116
+ - [ ] Every KU in the BookKU schema (id `<repo-slug>-<path-hash>-kuNN`; path→chapter; lines→pages; `sources` carry repo+path+commit)?
117
+ - [ ] FACTS verbatim + `path:line`, PROSE paraphrased (no uncited ≥8-word run); all KUs entailment-verified?
118
+ - [ ] Source SPDX license recorded in `metadata.license`; license-check gate passed (or explicit `--override`)?
119
+ - [ ] Promoted via `dz brain add --from-kus … --kind repo --license <spdx>` (NOT book-ingest/distill); registry shows `kind: repo`; `dz brain query` returns a KU labeled with the repo?
120
+
121
+ ## Examples
122
+
123
+ - «Add this backoff library to my brain» → deep-walk → KUs like `retry-lib-a1b2c3-ku02`
124
+ (decision-framework «Retry-with-jitter backoff policy», `src/net/backoff.ts:40–88`, verified,
125
+ license MIT) → `dz brain add --from-kus retry-lib.kus.json --slug retry-lib --kind repo --license MIT`.
126
+ - «Почему этот репозиторий не попал в brain?» → license unknown → the register refused it; re-run with
127
+ a recorded `--override` decision, or add a `LICENSE`.
128
+ - «Ground my design questions on our infra repo AND the DDIA book» → both are just `source`s in the one
129
+ brain; `dz brain query "replication topology"` ranks KUs from both, each labeled with its source.