@dzhechkov/skills-book-digitizer 0.2.2 → 0.2.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -12,16 +12,18 @@ Knowledge Base behind them. dz-original; design & rationale in
12
12
  methods at the decisions it's about — e.g. DDIA → `ddia-replication-topology-choice`,
13
13
  `ddia-partitioning-strategy`, `ddia-consistency-model-selection`.
14
14
 
15
- ## The 6 skills (a checkpointed pipeline)
15
+ ## The skills (a checkpointed pipeline + a terminal promote + one sibling)
16
16
 
17
17
  | Skill | Stage |
18
18
  |-------|-------|
19
- | `digitize-book` | **Orchestrator** — runs the pipeline through 5 human checkpoints, resumable across sessions |
19
+ | `digitize-book` | **Orchestrator** — runs the pipeline through up to 6 human checkpoints (CP6 promote-to-brain optional), resumable across sessions |
20
20
  | `book-ingest` | PDF → chunked, page-anchored corpus + manifest (deterministic: poppler + heuristics, no LLM) |
21
21
  | `book-knowledge-extract` | corpus → **verified** Knowledge Units (map-reduce, dedup, KU-verify entailment, FACTS-verbatim/PROSE-paraphrase regime) |
22
22
  | `book-skill-distill` | verified KUs → **decision-moment** skills (slug-prefixed, routing evals + activation gate, KU→skill traceability) |
23
- | `book-pack-assemble` | skills → installable pack (`references/` deep-lookup shipped inside, **private-by-default** distribution, collision gate, Copilot policy) |
23
+ | `book-pack-assemble` | skills → installable pack (`references/` deep-lookup + the per-book `brain/<slug>.sqlite` KB slice shipped inside, **private-by-default** distribution, collision gate, Copilot policy) |
24
24
  | `book-kb-index` | KUs → the memory layer: `.dz/memory/books.sqlite` (lexical) + shared agentdb (vector) |
25
+ | `book-brain-register` | **Terminal promote (optional, CP6, ADR §5.1)** — promote this project's KUs into the durable **cross-project brain** (`~/.dz/brain/`) via `dz brain add --project . --source <slug>`. Default project-local only; an explicit, IP-relevant opt-in like CP5. The in-pipeline promote (vs the manual `dz brain add`). |
26
+ | `source-brain-ingest` | **Sibling (not a book stage, ADR §6)** — deep-walk a **repo/dir** → the *same* Knowledge Units → the *same* brain via `dz brain add --from-kus … --kind repo`. One brain, two ingest front-ends (books + code). |
25
27
 
26
28
  ## Install & use
27
29
 
@@ -43,7 +45,8 @@ Full design + the RU version: [ADR-001 Knowledge Brain](https://github.com/djd1m
43
45
  ### 1. Digitize a book → skills + KB
44
46
 
45
47
  The core pipeline. Install the orchestrator, then hand it a PDF; it runs `book-ingest → extract →
46
- distill → assemble → book-kb-index` through 5 human checkpoints.
48
+ distill → assemble → book-kb-index` through 5 human checkpoints (CP1–CP5), plus an optional CP6
49
+ `book-brain-register` to promote the book into the cross-project brain (scenario 3).
47
50
 
48
51
  ```bash
49
52
  dz init --target claude-code --select digitize-book
@@ -67,20 +70,29 @@ Pull-based: you ask, you get KU hits back with source + page anchors. No hook, n
67
70
  ### 3. Build a cross-project brain from N books
68
71
 
69
72
  Promote each project's digitized-book KUs into **one shared brain** (`~/.dz/brain/` — lexical
70
- `books.sqlite` + vector `agentdb.db`), then query across every ingested book at once:
73
+ `books.sqlite` + vector `agentdb.db`), then query across every ingested book at once. There are two
74
+ ways to promote — **in-pipeline** and **manual** — both drive the same `dz brain add`:
75
+
76
+ - **In-pipeline (preferred): `book-brain-register`** — the optional terminal stage of `digitize-book`,
77
+ gated at **CP6 (promote?)**. After the pack + project KB are built, the orchestrator offers to promote
78
+ the book into the brain; the default is *project-local only* and promotion is an explicit, IP-relevant
79
+ opt-in (like CP5's distribution decision). It runs `dz brain add --project . --source <slug>` for you.
80
+ - **Manual:** run the CLI yourself against a project whose book is already indexed.
71
81
 
72
82
  ```bash
73
- dz brain add --source <slug> # promote this project's KUs into the brain (idempotent)
74
- dz brain list # what's in the brain: slug, kind, KU count, corpus version
75
- dz brain query "replication topology" # cross-source recall over ALL books
83
+ dz brain add --project . --source <slug> # promote this project's KUs into the brain (idempotent)
84
+ dz brain list # what's in the brain: slug, kind, KU count, corpus version
85
+ dz brain query "replication topology" # cross-source recall over ALL books
76
86
  dz brain query "replication topology" --source <slug> # optional: narrow to one book
87
+ dz brain update <slug> --project . # non-clobbering refresh of a re-digitized book
77
88
  ```
78
89
 
79
90
  One shared store, not per-book packages: you do **not** install anything to query the brain — it
80
91
  accretes on `dz brain add`. The *right* book surfaces by **relevance** (KUs rank on one common scale,
81
92
  each hit labelled with its source book + page); `--source` is a label filter, not a manual routing
82
93
  pick. Cross-source is the default — multiple books scoring on a query is a feature (richer grounding),
83
- not a bug.
94
+ not a bug. The brain is **private by default and never distributed** (ADR §8): CP6 promotes a
95
+ copyrighted book into your *local* brain only.
84
96
 
85
97
  ### 4. Grounded answering (the hook)
86
98
 
@@ -0,0 +1,108 @@
1
+ ---
2
+ name: book-brain-register
3
+ description: >
4
+ Optional TERMINAL stage of the book digitizer (ADR-001 §5.1): after book-kb-index has loaded this
5
+ project's verified Knowledge Units into the project-local KB, PROMOTE them into the durable,
6
+ cross-project Knowledge Brain (`~/.dz/brain/`, `DZ_BRAIN_HOME`) via `dz brain add --project . --source
7
+ <slug>` — so the book's know-how is recallable in OTHER projects, not just the one that ingested it.
8
+ Gated at CP6 (promote?): default is project-local only; promotion is an explicit, IP-relevant
9
+ decision. Non-clobbering (upsert on (book, ku_id, corpus_version); refresh via `dz brain update`).
10
+ Triggers on: "promote the book into the brain", "book-brain-register", "dz brain add this book",
11
+ "add this digitized book to my knowledge brain". Invoked by `digitize-book` at CP6.
12
+ trust_tier: 1
13
+ trust_tier_label: "Structured"
14
+ trust_tier_path: "Run /bto-test to promote to Tier 2"
15
+ ---
16
+
17
+ # book-brain-register — promote a digitized book into the durable brain
18
+
19
+ The **terminal** stage, after `book-kb-index`. `book-kb-index` owns the **project** stores (the
20
+ source of truth for the pack — it always runs). This stage does one more, **optional** thing: it
21
+ **promotes** this project's verified KUs into the **durable cross-project brain** at `~/.dz/brain/`
22
+ (override `DZ_BRAIN_HOME`) so the book's know-how is considered *later, in other projects* — not just
23
+ the one that digitized it. It does not re-extract, re-index the project KB, or reimplement mirroring:
24
+ it drives the already-built `dz brain add`, which mirrors both stores (lexical + vector), writes the
25
+ registry entry, and generates the primer card.
26
+
27
+ ## When to use / NOT
28
+
29
+ - **Use** as the OPTIONAL final stage of `digitize-book`, at **CP6**, when the owner wants this book's
30
+ knowledge available across all their projects (the durable `~/.dz/brain/`).
31
+ - **NOT** to build the project KB — that is `book-kb-index`, which still runs regardless of CP6. This
32
+ stage promotes an *already-indexed* project's KUs upward; it is additive and opt-in.
33
+ - **NOT for repos/dirs** → those take the SIBLING path `source-brain-ingest` (deep-walk → same KU
34
+ shape → `dz brain add --from-kus … --kind repo --license <spdx>`, ADR §6). Same brain, different
35
+ front-end. This stage is the **book** path.
36
+ - **NOT** to publish or share the brain — the brain is **private by default and never distributed**
37
+ (ADR §8). A copyrighted book's promotion stays local; the shareable unit is the per-book KB slice
38
+ riding inside the `skills-book-<slug>` pack under the CP5 licence decision (that's `book-pack-assemble`).
39
+
40
+ ## Prerequisites
41
+
42
+ - `book-kb-index` has run: this project's verified KUs are in the project-local stores
43
+ (`.dz/memory/books.sqlite` + `.dz/agentdb.db`) with a settled `corpus_version`.
44
+ - The brain stores (`dz setup --memory agentdb`) — the same `agentdb` + `better-sqlite3` primitives,
45
+ the `dz brain add` command just points `dbPath` at the brain home instead of the project.
46
+ - A **CP6 promote decision** from the owner (see the checkpoint below) — this stage does nothing
47
+ without it; default is project-local only.
48
+
49
+ ## Protocol
50
+
51
+ 1. **CP6 (promote?)** — offer promotion explicitly. DEFAULT is *project-local only* (do nothing): the
52
+ book's KB already exists in this project. Promotion moves book-derived knowledge into a long-lived,
53
+ cross-project location, so — like CP5's distribution decision — it is an IP-relevant call the owner
54
+ makes deliberately, never auto-approved. If the owner declines, stop here (the project KB stands).
55
+ 2. **Promote** — on an explicit yes, drive the existing CLI (do NOT reimplement the mirroring):
56
+ ```bash
57
+ dz brain add --project . --source <slug>
58
+ ```
59
+ This resolves the brain home (`DZ_BRAIN_HOME` → `~/.dz/brain/`, created if absent), reads this
60
+ project's verified KUs for `<slug>`, and **idempotently upserts** them into BOTH brain stores
61
+ (lexical `books.sqlite` + vector `agentdb.db`) keyed on `(book, ku_id, corpus_version)`, then writes
62
+ the `brain.json` registry entry (slug, kind `book`, isbn/license, corpus_version, ku_count, lang,
63
+ primer path, added_ts) and generates `primers/<slug>.md`. All reused primitives — nothing new here.
64
+ 3. **Non-clobbering by construction** — the upsert evicts stale-corpus rows for **this source only**;
65
+ adding this book never touches other sources already in the brain. A re-digitized / updated book
66
+ does NOT duplicate: refresh it with
67
+ ```bash
68
+ dz brain update <slug> --project .
69
+ ```
70
+ which re-mirrors this source at its new `corpus_version` (stale-corpus rows evicted, other sources
71
+ left intact). Never `dz brain add` a second time hoping to "append" — the refresh verb is the path.
72
+ 4. **Verify** — confirm the promotion landed in the brain, labeled from the brain (not the project):
73
+ ```bash
74
+ dz brain query "<a KU name>" --source <slug> # returns the KU, labeled with its source book + page
75
+ dz brain primer <slug> # shows the capability card (synopsis + KU-type histogram)
76
+ ```
77
+ Also `dz brain list` should show the source with its ku_count and corpus_version.
78
+
79
+ ## Anti-patterns
80
+
81
+ | Anti-pattern | Why it fails | Instead |
82
+ |--------------|--------------|---------|
83
+ | Promoting by default / auto-approving CP6 | moves book-derived IP into a durable cross-project home silently | CP6 is explicit, owner-decided, like CP5 |
84
+ | Skipping `book-kb-index` and promoting straight to the brain | there are no settled project KUs to mirror; provenance/corpus_version unset | index the project first; promote after |
85
+ | Re-running `dz brain add` to update a book | INSERT path can leave stale-corpus rows | `dz brain update <slug>` (eviction + upsert) |
86
+ | Reimplementing the store mirroring in this skill | the CLI already mirrors lexical+vector, writes the registry, builds the primer | drive `dz brain add` — reuse, don't rebuild |
87
+ | Routing a repo/dir through this stage | book path assumes book KUs + slug provenance | sibling `source-brain-ingest` → `dz brain add --from-kus … --kind repo --license` |
88
+ | Treating the brain as publishable | the accreted personal brain is private, never distributed (§8) | share the per-book pack slice (CP5), not the brain |
89
+
90
+ ## Self-check
91
+
92
+ - [ ] `book-kb-index` ran first; this project's verified KUs + corpus_version are settled?
93
+ - [ ] CP6 honored — an EXPLICIT owner promote decision (never auto-approved), default project-local?
94
+ - [ ] Promoted via `dz brain add --project . --source <slug>` (reused CLI; both brain stores mirrored, registry + primer written)?
95
+ - [ ] Non-clobbering: upsert on (book, ku_id, corpus_version); a re-run uses `dz brain update <slug>`, not a second add?
96
+ - [ ] Verified: `dz brain query "<KU>" --source <slug>` returns it labeled from the brain; `dz brain primer <slug>` shows the card?
97
+ - [ ] Brain kept private (never distributed); a copyrighted book's promotion stays local?
98
+
99
+ ## Examples
100
+
101
+ - «Оцифровал DDIA, теперь хочу, чтобы эти знания были доступны и в других проектах» → at CP6 the owner
102
+ says promote → `dz brain add --project . --source ddia` → 116 KU mirrored into `~/.dz/brain/`;
103
+ `dz brain query "репликация" --source ddia` returns them labeled from the brain; `dz brain primer ddia`
104
+ shows the card. In a later project, `dz brain query` recalls DDIA KUs with no re-ingest.
105
+ - «Обновил книгу, переоцифровал — не хочу дублей в brain» → `dz brain update ddia --project .` re-mirrors
106
+ the new corpus_version, evicting stale rows for ddia only; other sources untouched.
107
+ - «Промоутить только КБ, пак не публиковать» → CP6 (brain) and CP5 (pack distribution) are separate
108
+ decisions: promote to the private brain, keep the pack `private: true`.
@@ -34,6 +34,7 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
34
34
  <slug>-<decision>/SKILL.md
35
35
  <slug>-<decision>/references/*.md ← the deep-lookup, shipped IN the pack
36
36
  <slug>-<decision>/evals/routing.yaml
37
+ brain/<slug>.sqlite ← the per-book KB slice, shipped IN the pack (step 9)
37
38
  package.json sources.json README.md LICENSE
38
39
  ```
39
40
  2. **`references/` ships the deep-lookup inside the pack** — each skill's consumed KUs in full
@@ -41,8 +42,8 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
41
42
  every target/machine, not a call to a KB the recipient doesn't have. `sources.json` declares
42
43
  which lookup tiers exist per skill: `references/` = always, corpus = owner-local, agentdb =
43
44
  owner-local — the skill degrades honestly.
44
- 3. **`files[]` whitelist** every skill dir + `references`/`evals` + `sources.json` + `README.md` +
45
- `LICENSE` (forget one → npm silently drops it). Verify the tarball after.
45
+ 3. **`files[]` whitelist** every skill dir + `references`/`evals` + `brain/` (the KB slice, step 9) +
46
+ `sources.json` + `README.md` + `LICENSE` (forget one → npm silently drops it). Verify the tarball after.
46
47
  4. **`package.json` distribution guardrails (structural, not policy):**
47
48
  `"private": true` + `"publishConfig": { "access": "restricted" }`. **No automatic
48
49
  `harness-presets` entry** (presets are the public install surface). Flipping to publishable
@@ -68,6 +69,19 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
68
69
  (re-paraphrase) and re-run. Cited ≤25-word quotes with a page anchor are exempt. Only once it
69
70
  passes: `dz benchmark <pack> --all` (honest L0 grade per skill — never fabricate schemas/evals to
70
71
  game it) → `dz init --target claude-code --select <skills>` install smoke → **CP4** report.
72
+ 9. **Ship the per-book KB slice inside the pack (ADR §8.1)** — once skills + `references/` are
73
+ assembled, export the book's KUs as a standalone slice:
74
+ `dz brain export --source <slug> --out brain/<slug>.sqlite` (this book's rows only —
75
+ `WHERE book=<slug>`, **lexical-only** for portability — the recipient's `dz brain add --from-pack`
76
+ re-embeds the vectors into their own brain on import). Whitelist `brain/` in `files[]` (step 3)
77
+ and pass the slice **through the same shingling IP gate** (step 8) as the rest of the pack — a
78
+ verbatim leak in the KB slice fails the pack just like one in a SKILL.md. The slice rides the pack
79
+ under the **same CP5 licence decision**: a copyrighted book stays `private: true` (slice never
80
+ auto-published); a permissive source is publishable (slice ships). A recipient loads it with
81
+ `dz brain add --from-pack @dzhechkov/skills-book-<slug>` — an idempotent upsert on
82
+ `(book, ku_id, corpus_version)` into their `~/.dz/brain/` — so one `npx`/github install carries
83
+ BOTH the behavior (skills) AND the knowledge (the brain slice), inseparably, under one decision.
84
+ Document the `--from-pack` load in the pack README.
71
85
 
72
86
  ## Anti-patterns
73
87
 
@@ -79,6 +93,8 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
79
93
  | `origin` block in sources.json | sync-upstream would chase a non-repo | documentation-only, no origin |
80
94
  | Fat SKILL.md bodies on Copilot | floods every request | lean body + `references/` |
81
95
  | Shipping a pack without the shingling gate | uncited verbatim runs = IP violation | run `shingling-check.mjs` before the L0 benchmark; re-paraphrase any hit |
96
+ | Publishing the KB slice separately from the pack's CP5 decision | book knowledge leaks out from under the licence gate | slice rides the pack: `private` book → private slice, permissive → publishable |
97
+ | KB slice skipping the shingling gate | a verbatim leak in the slice = same IP violation | pass `brain/<slug>.sqlite` through the shingling gate like the rest of the pack |
82
98
 
83
99
  ## Self-check
84
100
 
@@ -88,6 +104,7 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
88
104
  - [ ] Collision gate clean; slug-prefixed unique ids?
89
105
  - [ ] Faithfulness gate run (each shipped skill's claims supported by its `derived_from` KUs)?
90
106
  - [ ] Shingling gate passed (no uncited verbatim run ≥8 words; cited ≤25-word quotes anchored)?
107
+ - [ ] KB slice `brain/<slug>.sqlite` exported, whitelisted in `files[]`, passed the shingling gate, and follows the pack's CP5 licence (private book → private slice); `--from-pack` load documented in README?
91
108
  - [ ] L0 grades honest; install smoke passed; Copilot bytes shown if relevant?
92
109
 
93
110
  ## Examples
@@ -4,8 +4,8 @@ description: >
4
4
  Orchestrator: turn an engineering book/textbook/monograph (PDF, 100–1000+ pages) into an
5
5
  INSTALLABLE skill pack whose skills make AI coders (Claude Code, Codex, …) work within the
6
6
  book's methodologies at real decision moments — plus a searchable Knowledge Base behind them.
7
- Drives the 5-stage pipeline (ingest → extract → distill → pack → index) through 5 human
8
- checkpoints, resumable across sessions. Triggers on: "оцифруй книгу", "digitize this book",
7
+ Drives the pipeline (ingest → extract → distill → pack → index, then an optional promote-to-brain)
8
+ through up to 6 human checkpoints, resumable across sessions. Triggers on: "оцифруй книгу", "digitize this book",
9
9
  "сделай скилл-пак из книги", "turn this textbook into skills", "book to skill pack".
10
10
  trust_tier: 1
11
11
  trust_tier_label: "Structured"
@@ -19,9 +19,10 @@ agent **apply the book's methodologies** when it hits the decisions the book is
19
19
  `ddia-replication-topology-choice`, `ddia-partitioning-strategy`). Design & rationale:
20
20
  [ADR-001 v2](https://github.com/djd1m/dz-harness-hub/blob/main/features/book-knowledge-digitizer/03_adr/001-book-to-skillpack-pipeline.md).
21
21
 
22
- > **This is an orchestrator.** It sequences five stage-skills and stops at five checkpoints for
23
- > your judgment. It never runs the whole book unattended — book digitization is a steered
24
- > operation, and the panel-confirmed cost is **~2.5–4M agent-tokens per 640-page book**.
22
+ > **This is an orchestrator.** It sequences the stage-skills and stops at up to six checkpoints for
23
+ > your judgment (CP6, promote-to-brain, is optional). It never runs the whole book unattended — book
24
+ > digitization is a steered operation, and the panel-confirmed cost is **~2.5–4M agent-tokens per
25
+ > 640-page book**.
25
26
 
26
27
  ## When to use / NOT
27
28
 
@@ -33,8 +34,11 @@ agent **apply the book's methodologies** when it hits the decisions the book is
33
34
 
34
35
  - `poppler-utils` (`pdftotext`, `pdfinfo`, `pdfimages`) for ingest.
35
36
  - For the KB layers: `dz setup --memory agentdb` (installs `agentdb` + `better-sqlite3`).
36
- - The five stage skills of this pack installed (they auto-install together):
37
- `book-ingest`, `book-knowledge-extract`, `book-skill-distill`, `book-pack-assemble`, `book-kb-index`.
37
+ - The stage skills of this pack installed (they auto-install together):
38
+ `book-ingest`, `book-knowledge-extract`, `book-skill-distill`, `book-pack-assemble`, `book-kb-index`,
39
+ and the optional terminal `book-brain-register` (CP6 promote-to-brain).
40
+ - For CP6 promotion into the durable cross-project brain (`~/.dz/brain/`, `DZ_BRAIN_HOME`): the same
41
+ `agentdb` stores (`dz setup --memory agentdb`).
38
42
 
39
43
  ## Pipeline & checkpoints
40
44
 
@@ -42,6 +46,7 @@ agent **apply the book's methodologies** when it hits the decisions the book is
42
46
  book.pdf ─[book-ingest]─► corpus/ + manifest ─[book-knowledge-extract]─► Knowledge Units (+KU-verify)
43
47
  ─[book-skill-distill]─► SKILL.md's ─[book-pack-assemble]─► skills-book-<slug> ─[book-kb-index]─► KB
44
48
  CP1 scope CP2 sample KUs CP3 skill list CP3.5 routing CP4 pack CP5 distribute
49
+ ····optional terminal····► ─[book-brain-register]─► ~/.dz/brain (CP6 promote?)
45
50
  ```
46
51
 
47
52
  | CP | After | You decide |
@@ -51,7 +56,8 @@ book.pdf ─[book-ingest]─► corpus/ + manifest ─[book-knowledge-extract]
51
56
  | **CP3** | distill triage | approve the candidate skill list (decision moments); flat vs gateway layout |
52
57
  | **CP3.5** | routing evals | the routing gate report (≥80% correct activation / ≤10% sibling-steal) — fix or proceed |
53
58
  | **CP4** | pack assembled | shingling-gate result (no uncited verbatim run ≥8 words) + L0 report + install smoke (+ Copilot instruction-byte total if that target) |
54
- | **CP5** | — | distribution: keep `private` (default) or record an explicit publish decision |
59
+ | **CP5** | pack + KB indexed | distribution: keep `private` (default) or record an explicit publish decision |
60
+ | **CP6** | `book-brain-register` | **promote** the book's KUs into the durable **cross-project brain** (`~/.dz/brain/`) — **default: project-local only**; an explicit opt-in, IP-relevant like CP5 (moves book-derived knowledge into a long-lived home). Optional terminal stage |
55
61
 
56
62
  ## Protocol
57
63
 
@@ -68,13 +74,22 @@ book.pdf ─[book-ingest]─► corpus/ + manifest ─[book-knowledge-extract]
68
74
  5. **Distill** → `book-skill-distill` proposes decision-moment skills; **CP3**; then **CP3.5** on
69
75
  the routing evals.
70
76
  6. **Assemble** → `book-pack-assemble`; **CP4**.
71
- 7. **Index** → `book-kb-index` loads KUs into the KB (lexical `books.sqlite` + agentdb vector).
77
+ 7. **Index** → `book-kb-index` loads KUs into the project KB (lexical `books.sqlite` + agentdb vector).
78
+ This always runs — it is the project's KB, independent of any brain decision.
72
79
  8. **Distribute** → **CP5**; then hand over: `dz init --select <the new skills>` and how to use.
80
+ 9. **Promote (optional)** → `book-brain-register`: at **CP6**, offer to `dz brain add --project . --source
81
+ <slug>` this book's KUs into the durable **cross-project brain** (`~/.dz/brain/`, `DZ_BRAIN_HOME`).
82
+ **Default is project-local only** — promotion is an explicit, IP-relevant opt-in (like CP5). A
83
+ re-digitized book refreshes non-clobbering via `dz brain update <slug>`; verify with
84
+ `dz brain query "<a KU name>" --source <slug>` + `dz brain primer <slug>`. The brain is private,
85
+ never distributed (§IP). Sibling: repos/dirs take `source-brain-ingest` → the same brain instead.
73
86
 
74
87
  ## `--trusted` mode
75
88
 
76
- After you've approved 2–3 books, `--trusted` makes CP2 an async spot-check; **CP1, CP3, CP5 stay
77
- mandatory** (scope, the skill list, and distribution are the irreducible judgment calls).
89
+ After you've approved 2–3 books, `--trusted` makes CP2 an async spot-check; **CP1, CP3, CP5, and CP6
90
+ stay interactive** (scope, the skill list, distribution, and brain-promotion are the irreducible
91
+ judgment calls). **CP6 is never auto-approved** — promoting book-derived knowledge into the durable
92
+ cross-project brain is an IP decision, exactly like CP5, so `--trusted` never green-lights it.
78
93
 
79
94
  ## Anti-patterns
80
95
 
@@ -89,8 +104,9 @@ mandatory** (scope, the skill list, and distribution are the irreducible judgmen
89
104
 
90
105
  - [ ] Resumed from the manifest (no completed phase re-run)?
91
106
  - [ ] CP1–CP5 honored (scope, samples, skill list, routing gate, distribution)?
107
+ - [ ] CP6 honored — brain promotion was an explicit opt-in (default project-local; never auto-approved, even under `--trusted`)?
92
108
  - [ ] Only verified KUs fed distill; every skill claim traces to a KU?
93
- - [ ] Pack is `private` unless a CP5 decision says otherwise?
109
+ - [ ] Pack is `private` unless a CP5 decision says otherwise; brain kept private (never distributed)?
94
110
  - [ ] User got: install command + how the new skills trigger?
95
111
 
96
112
  ## Examples
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@dzhechkov/skills-book-digitizer",
3
- "version": "0.2.2",
4
- "description": "Canonical book-digitizer skill pack — 6 skills that turn an engineering book/textbook/monograph into an installable skill pack whose skills make AI coders work within the book's methodologies: digitize-book (orchestrator), book-ingest, book-knowledge-extract, book-skill-distill, book-pack-assemble, book-kb-index.",
3
+ "version": "0.2.6",
4
+ "description": "Canonical book-digitizer skill pack — 8 skills that turn an engineering book/textbook/monograph into an installable skill pack whose skills make AI coders work within the book's methodologies: digitize-book (orchestrator), book-ingest, book-knowledge-extract, book-skill-distill, book-pack-assemble, book-kb-index, book-brain-register (terminal promote-to-brain, CP6), source-brain-ingest (repo/dir sibling).",
5
5
  "license": "MIT",
6
6
  "author": "dzhechko",
7
7
  "keywords": [
@@ -22,6 +22,8 @@
22
22
  "book-skill-distill",
23
23
  "book-pack-assemble",
24
24
  "book-kb-index",
25
+ "book-brain-register",
26
+ "source-brain-ingest",
25
27
  "scripts",
26
28
  "sources.json",
27
29
  "README.md",
@@ -0,0 +1,129 @@
1
+ ---
2
+ name: source-brain-ingest
3
+ description: >
4
+ The repo/dir SIBLING of the book pipeline (ADR-001 §6): deep-walk a codebase or directory
5
+ (respect .gitignore, skip binaries/node_modules/dist, cap size), then distill DECISION-GRADE
6
+ Knowledge Units in the SAME BookKU shape the book pipeline emits — and feed them into the SAME
7
+ shared brain register via `dz brain add --from-kus`, WITHOUT the book-only stages (no pdftotext
8
+ ingest, no skill distillation). Same KU schema = free reuse of the whole brain (query, primer,
9
+ grounding). Records the source's SPDX license and gates on it. Triggers on: "ingest a repo into
10
+ the brain", "source-brain-ingest", "add this codebase to my knowledge brain".
11
+ trust_tier: 1
12
+ trust_tier_label: "Structured"
13
+ trust_tier_path: "Run /bto-test to promote to Tier 2"
14
+ ---
15
+
16
+ # source-brain-ingest — repo/dir → brain KUs (the non-book front-end)
17
+
18
+ One brain, **two ingest front-ends** (ADR §6): the book pipeline turns PDFs into KUs; this turns a
19
+ **repo or directory** into KUs — the *same* KU shape, the *same* register stage, the *same* query /
20
+ primer / grounding surface. A repo plugs into everything the book path already built, for free. This
21
+ is a **sibling** skill — it does not touch, and must not contort, the intact book pipeline.
22
+
23
+ ## When to use / NOT
24
+
25
+ - **Use** to distill a codebase/dir into decision-grade Knowledge Units and promote them into the
26
+ durable brain (`~/.dz/brain/`) so `dz brain query` / primers / the grounding hook cover repos too.
27
+ - **NOT for books/PDFs** → use `digitize-book` (pdftotext ingest is book-shaped and would be
28
+ contorted by repo logic; keeping it pure protects the proven book path).
29
+ - **NOT to author skills** → that's `book-skill-distill`. This stage emits KUs into the brain only;
30
+ it does not produce a `skills-<slug>` pack (repos may later return whole files, §8/G5, where books
31
+ return distilled KUs for IP reasons).
32
+ - **NOT to extract/verify book KUs** → that's `book-knowledge-extract`.
33
+
34
+ ## Prerequisites
35
+
36
+ - A local repo or directory (a checkout, or a `git clone` you control).
37
+ - `git` (to honour `.gitignore` and capture the commit for provenance) — degrade gracefully on a
38
+ non-git dir (walk the tree; `commit: null`).
39
+ - The brain stores (`dz setup --memory agentdb`) — the same `~/.dz/brain/` the book path promotes into.
40
+ - The source's **SPDX license identifier** (read `LICENSE`/`package.json`; pass `--license <spdx>`).
41
+
42
+ ## The Knowledge Unit schema (identical to the book pipeline — that's the whole point)
43
+
44
+ Emit the **same BookKU shape** `book-knowledge-extract` emits, so a repo is just another `source`:
45
+
46
+ ```yaml
47
+ id: <repo-slug>-<path-hash>-kuNN # GLOBALLY UNIQUE: <path-hash> discriminates the file/chunk
48
+ # (short hash of the repo-relative path) so two files never
49
+ # collide on kuNN — the brain upsert keys on (source, ku_id,
50
+ # corpus_version) and silently drops id collisions otherwise.
51
+ type: methodology | decision-framework | formula | heuristic | checklist |
52
+ tradeoff-table | case-pattern | definition
53
+ name: "Retry-with-jitter backoff policy"
54
+ problem: the decision/task this knowledge serves (when/why it applies)
55
+ content: the paraphrased procedure / framework / restructured table / verbatim signature-or-formula
56
+ applicability: preconditions, scale ranges, contexts where it holds
57
+ limits: tradeoffs, failure modes, when NOT to apply
58
+ chapter: src/net/backoff.ts # PATH takes the book's `chapter` slot
59
+ pages: [40, 88] # LINE-RANGE takes the book's `pages` slot
60
+ sources: [{ repo: <repo-slug>, path: src/net/backoff.ts, lines: [40, 88], commit: <sha> }]
61
+ metadata: { license: <spdx>, corpus_version: <sha-or-ts>, lang: <detected> }
62
+ verified: true | false | partial # entailment against the cited lines
63
+ ```
64
+
65
+ Because the shape is identical, the brain treats books and repos on one common scale (query, primer,
66
+ grounding all work unchanged). `kind: repo` in the registry lets the brain treat repos **distinctly**
67
+ where it matters (e.g. return whole files, §8/G5) without a second schema.
68
+
69
+ ## Protocol
70
+
71
+ 1. **Walk** — deep-walk the repo/dir. **Respect `.gitignore`** (use `git ls-files` when it's a git
72
+ repo — free `.gitignore` honouring + tracked-only). **Skip** `node_modules/`, `dist/`, `build/`,
73
+ `.git/`, vendored trees, lockfiles, and **binaries** (detect by extension + a NUL-byte sniff).
74
+ **Cap** per-file size (skip/point at oversized generated files) and total bytes scanned; log what
75
+ was skipped so coverage is honest.
76
+ 2. **Chunk** — split each kept file into page-analogue units (a function/class/section, or a
77
+ line-window for prose docs). Anchor every chunk to `path` + `[startLine–endLine]` — the analogue
78
+ of the book's `[p.N]` anchor; these anchors make the verify step a cheap lookup.
79
+ 3. **Distill KUs** — emit **decision-grade** Knowledge Units in the schema above (a decision the code
80
+ encodes, a policy, a tradeoff, a reusable pattern) — NOT a file summary. **Fidelity regime** (same
81
+ as the book path): FACTS (signatures, constants, config values, exact code) **verbatim** with a
82
+ `path:line` anchor; PROSE (rationale, README narrative) **paraphrased** — no uncited verbatim run
83
+ ≥ 8 words (this is what a downstream shingling gate checks). Then a fresh checker re-opens each
84
+ KU's cited `path:line` range and sets `verified` (entailment gate; verify all, lookup ≪ generation).
85
+ 4. **License-check (the IP gate)** — record the source's SPDX license in every KU's `metadata.license`
86
+ and in what you pass to the register. The register **refuses** an unknown / incompatible license
87
+ without an explicit `--override` — so an unlicensed or GPL-incompatible repo cannot silently land
88
+ in the brain. Repo license travels with the KUs; if the brain ever returns whole files (§8/G5),
89
+ that license binds any downstream reuse.
90
+ 5. **Promote** — write the verified KUs to a JSON file in the BookKU shape and hand them to the SAME
91
+ register the book path uses:
92
+ ```bash
93
+ dz brain add --from-kus <file.json> --slug <repo-slug> --kind repo --license <spdx>
94
+ ```
95
+ This idempotently upserts on `(source, ku_id, corpus_version)` into `~/.dz/brain/` (lexical
96
+ `books.sqlite` + vector `agentdb.db`), writes the `brain.json` registry entry (`kind: repo`,
97
+ license, corpus_version, ku_count), and generates the primer — **no book-ingest, no
98
+ book-skill-distill in the path**. Verify: `dz brain query "<a KU name>"` returns it labeled with
99
+ the repo source.
100
+
101
+ ## Anti-patterns
102
+
103
+ | Anti-pattern | Why it fails | Instead |
104
+ |--------------|--------------|---------|
105
+ | Inventing a repo-specific KU schema | breaks the shared brain (query/primer/grounding assume one shape) | emit the exact BookKU schema — path→chapter, lines→pages |
106
+ | Ignoring `.gitignore` / walking `node_modules` | ingests junk + secrets, blows the token budget | `git ls-files`; skip binaries/vendored/dist; cap size |
107
+ | `kuNN` id without a path discriminator | two files collide → upsert-by-id drops KUs | `<repo-slug>-<path-hash>-kuNN` |
108
+ | Routing a repo through `book-ingest`/`digitize-book` | pdftotext + book stages don't fit code | this sibling → `dz brain add --from-kus` |
109
+ | Distilling repo KUs into a skill pack | that's the book path's job; repos may return whole files (§8) | KUs into the brain only |
110
+ | Promoting an unknown/incompatible license | leaks IP-encumbered code into the durable brain | license-check gate; `--override` is an explicit decision |
111
+ | File-summary "KUs" | RAG noise, not decisions | decision-grade KUs (policy/tradeoff/pattern) |
112
+
113
+ ## Self-check
114
+
115
+ - [ ] Walk respected `.gitignore`, skipped binaries/`node_modules`/`dist`, capped size — skips logged?
116
+ - [ ] Every KU in the BookKU schema (id `<repo-slug>-<path-hash>-kuNN`; path→chapter; lines→pages; `sources` carry repo+path+commit)?
117
+ - [ ] FACTS verbatim + `path:line`, PROSE paraphrased (no uncited ≥8-word run); all KUs entailment-verified?
118
+ - [ ] Source SPDX license recorded in `metadata.license`; license-check gate passed (or explicit `--override`)?
119
+ - [ ] Promoted via `dz brain add --from-kus … --kind repo --license <spdx>` (NOT book-ingest/distill); registry shows `kind: repo`; `dz brain query` returns a KU labeled with the repo?
120
+
121
+ ## Examples
122
+
123
+ - «Add this backoff library to my brain» → deep-walk → KUs like `retry-lib-a1b2c3-ku02`
124
+ (decision-framework «Retry-with-jitter backoff policy», `src/net/backoff.ts:40–88`, verified,
125
+ license MIT) → `dz brain add --from-kus retry-lib.kus.json --slug retry-lib --kind repo --license MIT`.
126
+ - «Почему этот репозиторий не попал в brain?» → license unknown → the register refused it; re-run with
127
+ a recorded `--override` decision, or add a `LICENSE`.
128
+ - «Ground my design questions on our infra repo AND the DDIA book» → both are just `source`s in the one
129
+ brain; `dz brain query "replication topology"` ranks KUs from both, each labeled with its source.
package/sources.json CHANGED
@@ -16,6 +16,8 @@
16
16
  "book-knowledge-extract": { "upstream_type": "original", "upstream_path": null, "description": "Corpus → verified Knowledge Units (map-reduce, KU-verify, fidelity regime)" },
17
17
  "book-skill-distill": { "upstream_type": "original", "upstream_path": null, "description": "Verified KUs → decision-moment skills (routing evals, traceability)" },
18
18
  "book-pack-assemble": { "upstream_type": "original", "upstream_path": null, "description": "Skills → installable pack (references/, private-default, collision gate)" },
19
- "book-kb-index": { "upstream_type": "original", "upstream_path": null, "description": "KUs → books.sqlite (lexical) + agentdb (vector) KB, upsert not append" }
19
+ "book-kb-index": { "upstream_type": "original", "upstream_path": null, "description": "KUs → books.sqlite (lexical) + agentdb (vector) KB, upsert not append" },
20
+ "book-brain-register": { "upstream_type": "original", "upstream_path": null, "description": "Optional terminal stage (CP6) — promote a project's KUs into the durable cross-project brain (~/.dz/brain/) via dz brain add" },
21
+ "source-brain-ingest": { "upstream_type": "original", "upstream_path": null, "description": "Sibling (ADR §6) — deep-walk a repo/dir into the same KUs → same brain via dz brain add --from-kus --kind repo" }
20
22
  }
21
23
  }