@dzhechkov/skills-book-digitizer 0.2.2 → 0.2.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -9
- package/book-brain-register/SKILL.md +108 -0
- package/book-pack-assemble/SKILL.md +19 -2
- package/digitize-book/SKILL.md +28 -12
- package/package.json +4 -2
- package/source-brain-ingest/SKILL.md +129 -0
- package/sources.json +3 -1
package/README.md
CHANGED
|
@@ -12,16 +12,18 @@ Knowledge Base behind them. dz-original; design & rationale in
|
|
|
12
12
|
methods at the decisions it's about — e.g. DDIA → `ddia-replication-topology-choice`,
|
|
13
13
|
`ddia-partitioning-strategy`, `ddia-consistency-model-selection`.
|
|
14
14
|
|
|
15
|
-
## The
|
|
15
|
+
## The skills (a checkpointed pipeline + a terminal promote + one sibling)
|
|
16
16
|
|
|
17
17
|
| Skill | Stage |
|
|
18
18
|
|-------|-------|
|
|
19
|
-
| `digitize-book` | **Orchestrator** — runs the pipeline through
|
|
19
|
+
| `digitize-book` | **Orchestrator** — runs the pipeline through up to 6 human checkpoints (CP6 promote-to-brain optional), resumable across sessions |
|
|
20
20
|
| `book-ingest` | PDF → chunked, page-anchored corpus + manifest (deterministic: poppler + heuristics, no LLM) |
|
|
21
21
|
| `book-knowledge-extract` | corpus → **verified** Knowledge Units (map-reduce, dedup, KU-verify entailment, FACTS-verbatim/PROSE-paraphrase regime) |
|
|
22
22
|
| `book-skill-distill` | verified KUs → **decision-moment** skills (slug-prefixed, routing evals + activation gate, KU→skill traceability) |
|
|
23
|
-
| `book-pack-assemble` | skills → installable pack (`references/` deep-lookup shipped inside, **private-by-default** distribution, collision gate, Copilot policy) |
|
|
23
|
+
| `book-pack-assemble` | skills → installable pack (`references/` deep-lookup + the per-book `brain/<slug>.sqlite` KB slice shipped inside, **private-by-default** distribution, collision gate, Copilot policy) |
|
|
24
24
|
| `book-kb-index` | KUs → the memory layer: `.dz/memory/books.sqlite` (lexical) + shared agentdb (vector) |
|
|
25
|
+
| `book-brain-register` | **Terminal promote (optional, CP6, ADR §5.1)** — promote this project's KUs into the durable **cross-project brain** (`~/.dz/brain/`) via `dz brain add --project . --source <slug>`. Default project-local only; an explicit, IP-relevant opt-in like CP5. The in-pipeline promote (vs the manual `dz brain add`). |
|
|
26
|
+
| `source-brain-ingest` | **Sibling (not a book stage, ADR §6)** — deep-walk a **repo/dir** → the *same* Knowledge Units → the *same* brain via `dz brain add --from-kus … --kind repo`. One brain, two ingest front-ends (books + code). |
|
|
25
27
|
|
|
26
28
|
## Install & use
|
|
27
29
|
|
|
@@ -43,7 +45,8 @@ Full design + the RU version: [ADR-001 Knowledge Brain](https://github.com/djd1m
|
|
|
43
45
|
### 1. Digitize a book → skills + KB
|
|
44
46
|
|
|
45
47
|
The core pipeline. Install the orchestrator, then hand it a PDF; it runs `book-ingest → extract →
|
|
46
|
-
distill → assemble → book-kb-index` through 5 human checkpoints
|
|
48
|
+
distill → assemble → book-kb-index` through 5 human checkpoints (CP1–CP5), plus an optional CP6
|
|
49
|
+
`book-brain-register` to promote the book into the cross-project brain (scenario 3).
|
|
47
50
|
|
|
48
51
|
```bash
|
|
49
52
|
dz init --target claude-code --select digitize-book
|
|
@@ -67,20 +70,29 @@ Pull-based: you ask, you get KU hits back with source + page anchors. No hook, n
|
|
|
67
70
|
### 3. Build a cross-project brain from N books
|
|
68
71
|
|
|
69
72
|
Promote each project's digitized-book KUs into **one shared brain** (`~/.dz/brain/` — lexical
|
|
70
|
-
`books.sqlite` + vector `agentdb.db`), then query across every ingested book at once
|
|
73
|
+
`books.sqlite` + vector `agentdb.db`), then query across every ingested book at once. There are two
|
|
74
|
+
ways to promote — **in-pipeline** and **manual** — both drive the same `dz brain add`:
|
|
75
|
+
|
|
76
|
+
- **In-pipeline (preferred): `book-brain-register`** — the optional terminal stage of `digitize-book`,
|
|
77
|
+
gated at **CP6 (promote?)**. After the pack + project KB are built, the orchestrator offers to promote
|
|
78
|
+
the book into the brain; the default is *project-local only* and promotion is an explicit, IP-relevant
|
|
79
|
+
opt-in (like CP5's distribution decision). It runs `dz brain add --project . --source <slug>` for you.
|
|
80
|
+
- **Manual:** run the CLI yourself against a project whose book is already indexed.
|
|
71
81
|
|
|
72
82
|
```bash
|
|
73
|
-
dz brain add
|
|
74
|
-
dz brain list
|
|
75
|
-
dz brain query "replication topology"
|
|
83
|
+
dz brain add --project . --source <slug> # promote this project's KUs into the brain (idempotent)
|
|
84
|
+
dz brain list # what's in the brain: slug, kind, KU count, corpus version
|
|
85
|
+
dz brain query "replication topology" # cross-source recall over ALL books
|
|
76
86
|
dz brain query "replication topology" --source <slug> # optional: narrow to one book
|
|
87
|
+
dz brain update <slug> --project . # non-clobbering refresh of a re-digitized book
|
|
77
88
|
```
|
|
78
89
|
|
|
79
90
|
One shared store, not per-book packages: you do **not** install anything to query the brain — it
|
|
80
91
|
accretes on `dz brain add`. The *right* book surfaces by **relevance** (KUs rank on one common scale,
|
|
81
92
|
each hit labelled with its source book + page); `--source` is a label filter, not a manual routing
|
|
82
93
|
pick. Cross-source is the default — multiple books scoring on a query is a feature (richer grounding),
|
|
83
|
-
not a bug.
|
|
94
|
+
not a bug. The brain is **private by default and never distributed** (ADR §8): CP6 promotes a
|
|
95
|
+
copyrighted book into your *local* brain only.
|
|
84
96
|
|
|
85
97
|
### 4. Grounded answering (the hook)
|
|
86
98
|
|
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: book-brain-register
|
|
3
|
+
description: >
|
|
4
|
+
Optional TERMINAL stage of the book digitizer (ADR-001 §5.1): after book-kb-index has loaded this
|
|
5
|
+
project's verified Knowledge Units into the project-local KB, PROMOTE them into the durable,
|
|
6
|
+
cross-project Knowledge Brain (`~/.dz/brain/`, `DZ_BRAIN_HOME`) via `dz brain add --project . --source
|
|
7
|
+
<slug>` — so the book's know-how is recallable in OTHER projects, not just the one that ingested it.
|
|
8
|
+
Gated at CP6 (promote?): default is project-local only; promotion is an explicit, IP-relevant
|
|
9
|
+
decision. Non-clobbering (upsert on (book, ku_id, corpus_version); refresh via `dz brain update`).
|
|
10
|
+
Triggers on: "promote the book into the brain", "book-brain-register", "dz brain add this book",
|
|
11
|
+
"add this digitized book to my knowledge brain". Invoked by `digitize-book` at CP6.
|
|
12
|
+
trust_tier: 1
|
|
13
|
+
trust_tier_label: "Structured"
|
|
14
|
+
trust_tier_path: "Run /bto-test to promote to Tier 2"
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
# book-brain-register — promote a digitized book into the durable brain
|
|
18
|
+
|
|
19
|
+
The **terminal** stage, after `book-kb-index`. `book-kb-index` owns the **project** stores (the
|
|
20
|
+
source of truth for the pack — it always runs). This stage does one more, **optional** thing: it
|
|
21
|
+
**promotes** this project's verified KUs into the **durable cross-project brain** at `~/.dz/brain/`
|
|
22
|
+
(override `DZ_BRAIN_HOME`) so the book's know-how is considered *later, in other projects* — not just
|
|
23
|
+
the one that digitized it. It does not re-extract, re-index the project KB, or reimplement mirroring:
|
|
24
|
+
it drives the already-built `dz brain add`, which mirrors both stores (lexical + vector), writes the
|
|
25
|
+
registry entry, and generates the primer card.
|
|
26
|
+
|
|
27
|
+
## When to use / NOT
|
|
28
|
+
|
|
29
|
+
- **Use** as the OPTIONAL final stage of `digitize-book`, at **CP6**, when the owner wants this book's
|
|
30
|
+
knowledge available across all their projects (the durable `~/.dz/brain/`).
|
|
31
|
+
- **NOT** to build the project KB — that is `book-kb-index`, which still runs regardless of CP6. This
|
|
32
|
+
stage promotes an *already-indexed* project's KUs upward; it is additive and opt-in.
|
|
33
|
+
- **NOT for repos/dirs** → those take the SIBLING path `source-brain-ingest` (deep-walk → same KU
|
|
34
|
+
shape → `dz brain add --from-kus … --kind repo --license <spdx>`, ADR §6). Same brain, different
|
|
35
|
+
front-end. This stage is the **book** path.
|
|
36
|
+
- **NOT** to publish or share the brain — the brain is **private by default and never distributed**
|
|
37
|
+
(ADR §8). A copyrighted book's promotion stays local; the shareable unit is the per-book KB slice
|
|
38
|
+
riding inside the `skills-book-<slug>` pack under the CP5 licence decision (that's `book-pack-assemble`).
|
|
39
|
+
|
|
40
|
+
## Prerequisites
|
|
41
|
+
|
|
42
|
+
- `book-kb-index` has run: this project's verified KUs are in the project-local stores
|
|
43
|
+
(`.dz/memory/books.sqlite` + `.dz/agentdb.db`) with a settled `corpus_version`.
|
|
44
|
+
- The brain stores (`dz setup --memory agentdb`) — the same `agentdb` + `better-sqlite3` primitives,
|
|
45
|
+
the `dz brain add` command just points `dbPath` at the brain home instead of the project.
|
|
46
|
+
- A **CP6 promote decision** from the owner (see the checkpoint below) — this stage does nothing
|
|
47
|
+
without it; default is project-local only.
|
|
48
|
+
|
|
49
|
+
## Protocol
|
|
50
|
+
|
|
51
|
+
1. **CP6 (promote?)** — offer promotion explicitly. DEFAULT is *project-local only* (do nothing): the
|
|
52
|
+
book's KB already exists in this project. Promotion moves book-derived knowledge into a long-lived,
|
|
53
|
+
cross-project location, so — like CP5's distribution decision — it is an IP-relevant call the owner
|
|
54
|
+
makes deliberately, never auto-approved. If the owner declines, stop here (the project KB stands).
|
|
55
|
+
2. **Promote** — on an explicit yes, drive the existing CLI (do NOT reimplement the mirroring):
|
|
56
|
+
```bash
|
|
57
|
+
dz brain add --project . --source <slug>
|
|
58
|
+
```
|
|
59
|
+
This resolves the brain home (`DZ_BRAIN_HOME` → `~/.dz/brain/`, created if absent), reads this
|
|
60
|
+
project's verified KUs for `<slug>`, and **idempotently upserts** them into BOTH brain stores
|
|
61
|
+
(lexical `books.sqlite` + vector `agentdb.db`) keyed on `(book, ku_id, corpus_version)`, then writes
|
|
62
|
+
the `brain.json` registry entry (slug, kind `book`, isbn/license, corpus_version, ku_count, lang,
|
|
63
|
+
primer path, added_ts) and generates `primers/<slug>.md`. All reused primitives — nothing new here.
|
|
64
|
+
3. **Non-clobbering by construction** — the upsert evicts stale-corpus rows for **this source only**;
|
|
65
|
+
adding this book never touches other sources already in the brain. A re-digitized / updated book
|
|
66
|
+
does NOT duplicate: refresh it with
|
|
67
|
+
```bash
|
|
68
|
+
dz brain update <slug> --project .
|
|
69
|
+
```
|
|
70
|
+
which re-mirrors this source at its new `corpus_version` (stale-corpus rows evicted, other sources
|
|
71
|
+
left intact). Never `dz brain add` a second time hoping to "append" — the refresh verb is the path.
|
|
72
|
+
4. **Verify** — confirm the promotion landed in the brain, labeled from the brain (not the project):
|
|
73
|
+
```bash
|
|
74
|
+
dz brain query "<a KU name>" --source <slug> # returns the KU, labeled with its source book + page
|
|
75
|
+
dz brain primer <slug> # shows the capability card (synopsis + KU-type histogram)
|
|
76
|
+
```
|
|
77
|
+
Also `dz brain list` should show the source with its ku_count and corpus_version.
|
|
78
|
+
|
|
79
|
+
## Anti-patterns
|
|
80
|
+
|
|
81
|
+
| Anti-pattern | Why it fails | Instead |
|
|
82
|
+
|--------------|--------------|---------|
|
|
83
|
+
| Promoting by default / auto-approving CP6 | moves book-derived IP into a durable cross-project home silently | CP6 is explicit, owner-decided, like CP5 |
|
|
84
|
+
| Skipping `book-kb-index` and promoting straight to the brain | there are no settled project KUs to mirror; provenance/corpus_version unset | index the project first; promote after |
|
|
85
|
+
| Re-running `dz brain add` to update a book | INSERT path can leave stale-corpus rows | `dz brain update <slug>` (eviction + upsert) |
|
|
86
|
+
| Reimplementing the store mirroring in this skill | the CLI already mirrors lexical+vector, writes the registry, builds the primer | drive `dz brain add` — reuse, don't rebuild |
|
|
87
|
+
| Routing a repo/dir through this stage | book path assumes book KUs + slug provenance | sibling `source-brain-ingest` → `dz brain add --from-kus … --kind repo --license` |
|
|
88
|
+
| Treating the brain as publishable | the accreted personal brain is private, never distributed (§8) | share the per-book pack slice (CP5), not the brain |
|
|
89
|
+
|
|
90
|
+
## Self-check
|
|
91
|
+
|
|
92
|
+
- [ ] `book-kb-index` ran first; this project's verified KUs + corpus_version are settled?
|
|
93
|
+
- [ ] CP6 honored — an EXPLICIT owner promote decision (never auto-approved), default project-local?
|
|
94
|
+
- [ ] Promoted via `dz brain add --project . --source <slug>` (reused CLI; both brain stores mirrored, registry + primer written)?
|
|
95
|
+
- [ ] Non-clobbering: upsert on (book, ku_id, corpus_version); a re-run uses `dz brain update <slug>`, not a second add?
|
|
96
|
+
- [ ] Verified: `dz brain query "<KU>" --source <slug>` returns it labeled from the brain; `dz brain primer <slug>` shows the card?
|
|
97
|
+
- [ ] Brain kept private (never distributed); a copyrighted book's promotion stays local?
|
|
98
|
+
|
|
99
|
+
## Examples
|
|
100
|
+
|
|
101
|
+
- «Оцифровал DDIA, теперь хочу, чтобы эти знания были доступны и в других проектах» → at CP6 the owner
|
|
102
|
+
says promote → `dz brain add --project . --source ddia` → 116 KU mirrored into `~/.dz/brain/`;
|
|
103
|
+
`dz brain query "репликация" --source ddia` returns them labeled from the brain; `dz brain primer ddia`
|
|
104
|
+
shows the card. In a later project, `dz brain query` recalls DDIA KUs with no re-ingest.
|
|
105
|
+
- «Обновил книгу, переоцифровал — не хочу дублей в brain» → `dz brain update ddia --project .` re-mirrors
|
|
106
|
+
the new corpus_version, evicting stale rows for ddia only; other sources untouched.
|
|
107
|
+
- «Промоутить только КБ, пак не публиковать» → CP6 (brain) and CP5 (pack distribution) are separate
|
|
108
|
+
decisions: promote to the private brain, keep the pack `private: true`.
|
|
@@ -34,6 +34,7 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
|
|
|
34
34
|
<slug>-<decision>/SKILL.md
|
|
35
35
|
<slug>-<decision>/references/*.md ← the deep-lookup, shipped IN the pack
|
|
36
36
|
<slug>-<decision>/evals/routing.yaml
|
|
37
|
+
brain/<slug>.sqlite ← the per-book KB slice, shipped IN the pack (step 9)
|
|
37
38
|
package.json sources.json README.md LICENSE
|
|
38
39
|
```
|
|
39
40
|
2. **`references/` ships the deep-lookup inside the pack** — each skill's consumed KUs in full
|
|
@@ -41,8 +42,8 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
|
|
|
41
42
|
every target/machine, not a call to a KB the recipient doesn't have. `sources.json` declares
|
|
42
43
|
which lookup tiers exist per skill: `references/` = always, corpus = owner-local, agentdb =
|
|
43
44
|
owner-local — the skill degrades honestly.
|
|
44
|
-
3. **`files[]` whitelist** every skill dir + `references`/`evals` + `
|
|
45
|
-
`LICENSE` (forget one → npm silently drops it). Verify the tarball after.
|
|
45
|
+
3. **`files[]` whitelist** every skill dir + `references`/`evals` + `brain/` (the KB slice, step 9) +
|
|
46
|
+
`sources.json` + `README.md` + `LICENSE` (forget one → npm silently drops it). Verify the tarball after.
|
|
46
47
|
4. **`package.json` distribution guardrails (structural, not policy):**
|
|
47
48
|
`"private": true` + `"publishConfig": { "access": "restricted" }`. **No automatic
|
|
48
49
|
`harness-presets` entry** (presets are the public install surface). Flipping to publishable
|
|
@@ -68,6 +69,19 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
|
|
|
68
69
|
(re-paraphrase) and re-run. Cited ≤25-word quotes with a page anchor are exempt. Only once it
|
|
69
70
|
passes: `dz benchmark <pack> --all` (honest L0 grade per skill — never fabricate schemas/evals to
|
|
70
71
|
game it) → `dz init --target claude-code --select <skills>` install smoke → **CP4** report.
|
|
72
|
+
9. **Ship the per-book KB slice inside the pack (ADR §8.1)** — once skills + `references/` are
|
|
73
|
+
assembled, export the book's KUs as a standalone slice:
|
|
74
|
+
`dz brain export --source <slug> --out brain/<slug>.sqlite` (this book's rows only —
|
|
75
|
+
`WHERE book=<slug>`, **lexical-only** for portability — the recipient's `dz brain add --from-pack`
|
|
76
|
+
re-embeds the vectors into their own brain on import). Whitelist `brain/` in `files[]` (step 3)
|
|
77
|
+
and pass the slice **through the same shingling IP gate** (step 8) as the rest of the pack — a
|
|
78
|
+
verbatim leak in the KB slice fails the pack just like one in a SKILL.md. The slice rides the pack
|
|
79
|
+
under the **same CP5 licence decision**: a copyrighted book stays `private: true` (slice never
|
|
80
|
+
auto-published); a permissive source is publishable (slice ships). A recipient loads it with
|
|
81
|
+
`dz brain add --from-pack @dzhechkov/skills-book-<slug>` — an idempotent upsert on
|
|
82
|
+
`(book, ku_id, corpus_version)` into their `~/.dz/brain/` — so one `npx`/github install carries
|
|
83
|
+
BOTH the behavior (skills) AND the knowledge (the brain slice), inseparably, under one decision.
|
|
84
|
+
Document the `--from-pack` load in the pack README.
|
|
71
85
|
|
|
72
86
|
## Anti-patterns
|
|
73
87
|
|
|
@@ -79,6 +93,8 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
|
|
|
79
93
|
| `origin` block in sources.json | sync-upstream would chase a non-repo | documentation-only, no origin |
|
|
80
94
|
| Fat SKILL.md bodies on Copilot | floods every request | lean body + `references/` |
|
|
81
95
|
| Shipping a pack without the shingling gate | uncited verbatim runs = IP violation | run `shingling-check.mjs` before the L0 benchmark; re-paraphrase any hit |
|
|
96
|
+
| Publishing the KB slice separately from the pack's CP5 decision | book knowledge leaks out from under the licence gate | slice rides the pack: `private` book → private slice, permissive → publishable |
|
|
97
|
+
| KB slice skipping the shingling gate | a verbatim leak in the slice = same IP violation | pass `brain/<slug>.sqlite` through the shingling gate like the rest of the pack |
|
|
82
98
|
|
|
83
99
|
## Self-check
|
|
84
100
|
|
|
@@ -88,6 +104,7 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
|
|
|
88
104
|
- [ ] Collision gate clean; slug-prefixed unique ids?
|
|
89
105
|
- [ ] Faithfulness gate run (each shipped skill's claims supported by its `derived_from` KUs)?
|
|
90
106
|
- [ ] Shingling gate passed (no uncited verbatim run ≥8 words; cited ≤25-word quotes anchored)?
|
|
107
|
+
- [ ] KB slice `brain/<slug>.sqlite` exported, whitelisted in `files[]`, passed the shingling gate, and follows the pack's CP5 licence (private book → private slice); `--from-pack` load documented in README?
|
|
91
108
|
- [ ] L0 grades honest; install smoke passed; Copilot bytes shown if relevant?
|
|
92
109
|
|
|
93
110
|
## Examples
|
package/digitize-book/SKILL.md
CHANGED
|
@@ -4,8 +4,8 @@ description: >
|
|
|
4
4
|
Orchestrator: turn an engineering book/textbook/monograph (PDF, 100–1000+ pages) into an
|
|
5
5
|
INSTALLABLE skill pack whose skills make AI coders (Claude Code, Codex, …) work within the
|
|
6
6
|
book's methodologies at real decision moments — plus a searchable Knowledge Base behind them.
|
|
7
|
-
Drives the
|
|
8
|
-
checkpoints, resumable across sessions. Triggers on: "оцифруй книгу", "digitize this book",
|
|
7
|
+
Drives the pipeline (ingest → extract → distill → pack → index, then an optional promote-to-brain)
|
|
8
|
+
through up to 6 human checkpoints, resumable across sessions. Triggers on: "оцифруй книгу", "digitize this book",
|
|
9
9
|
"сделай скилл-пак из книги", "turn this textbook into skills", "book to skill pack".
|
|
10
10
|
trust_tier: 1
|
|
11
11
|
trust_tier_label: "Structured"
|
|
@@ -19,9 +19,10 @@ agent **apply the book's methodologies** when it hits the decisions the book is
|
|
|
19
19
|
`ddia-replication-topology-choice`, `ddia-partitioning-strategy`). Design & rationale:
|
|
20
20
|
[ADR-001 v2](https://github.com/djd1m/dz-harness-hub/blob/main/features/book-knowledge-digitizer/03_adr/001-book-to-skillpack-pipeline.md).
|
|
21
21
|
|
|
22
|
-
> **This is an orchestrator.** It sequences
|
|
23
|
-
> your judgment. It never runs the whole book unattended — book
|
|
24
|
-
> operation, and the panel-confirmed cost is **~2.5–4M agent-tokens per
|
|
22
|
+
> **This is an orchestrator.** It sequences the stage-skills and stops at up to six checkpoints for
|
|
23
|
+
> your judgment (CP6, promote-to-brain, is optional). It never runs the whole book unattended — book
|
|
24
|
+
> digitization is a steered operation, and the panel-confirmed cost is **~2.5–4M agent-tokens per
|
|
25
|
+
> 640-page book**.
|
|
25
26
|
|
|
26
27
|
## When to use / NOT
|
|
27
28
|
|
|
@@ -33,8 +34,11 @@ agent **apply the book's methodologies** when it hits the decisions the book is
|
|
|
33
34
|
|
|
34
35
|
- `poppler-utils` (`pdftotext`, `pdfinfo`, `pdfimages`) for ingest.
|
|
35
36
|
- For the KB layers: `dz setup --memory agentdb` (installs `agentdb` + `better-sqlite3`).
|
|
36
|
-
- The
|
|
37
|
-
`book-ingest`, `book-knowledge-extract`, `book-skill-distill`, `book-pack-assemble`, `book-kb-index
|
|
37
|
+
- The stage skills of this pack installed (they auto-install together):
|
|
38
|
+
`book-ingest`, `book-knowledge-extract`, `book-skill-distill`, `book-pack-assemble`, `book-kb-index`,
|
|
39
|
+
and the optional terminal `book-brain-register` (CP6 promote-to-brain).
|
|
40
|
+
- For CP6 promotion into the durable cross-project brain (`~/.dz/brain/`, `DZ_BRAIN_HOME`): the same
|
|
41
|
+
`agentdb` stores (`dz setup --memory agentdb`).
|
|
38
42
|
|
|
39
43
|
## Pipeline & checkpoints
|
|
40
44
|
|
|
@@ -42,6 +46,7 @@ agent **apply the book's methodologies** when it hits the decisions the book is
|
|
|
42
46
|
book.pdf ─[book-ingest]─► corpus/ + manifest ─[book-knowledge-extract]─► Knowledge Units (+KU-verify)
|
|
43
47
|
─[book-skill-distill]─► SKILL.md's ─[book-pack-assemble]─► skills-book-<slug> ─[book-kb-index]─► KB
|
|
44
48
|
CP1 scope CP2 sample KUs CP3 skill list CP3.5 routing CP4 pack CP5 distribute
|
|
49
|
+
····optional terminal····► ─[book-brain-register]─► ~/.dz/brain (CP6 promote?)
|
|
45
50
|
```
|
|
46
51
|
|
|
47
52
|
| CP | After | You decide |
|
|
@@ -51,7 +56,8 @@ book.pdf ─[book-ingest]─► corpus/ + manifest ─[book-knowledge-extract]
|
|
|
51
56
|
| **CP3** | distill triage | approve the candidate skill list (decision moments); flat vs gateway layout |
|
|
52
57
|
| **CP3.5** | routing evals | the routing gate report (≥80% correct activation / ≤10% sibling-steal) — fix or proceed |
|
|
53
58
|
| **CP4** | pack assembled | shingling-gate result (no uncited verbatim run ≥8 words) + L0 report + install smoke (+ Copilot instruction-byte total if that target) |
|
|
54
|
-
| **CP5** |
|
|
59
|
+
| **CP5** | pack + KB indexed | distribution: keep `private` (default) or record an explicit publish decision |
|
|
60
|
+
| **CP6** | `book-brain-register` | **promote** the book's KUs into the durable **cross-project brain** (`~/.dz/brain/`) — **default: project-local only**; an explicit opt-in, IP-relevant like CP5 (moves book-derived knowledge into a long-lived home). Optional terminal stage |
|
|
55
61
|
|
|
56
62
|
## Protocol
|
|
57
63
|
|
|
@@ -68,13 +74,22 @@ book.pdf ─[book-ingest]─► corpus/ + manifest ─[book-knowledge-extract]
|
|
|
68
74
|
5. **Distill** → `book-skill-distill` proposes decision-moment skills; **CP3**; then **CP3.5** on
|
|
69
75
|
the routing evals.
|
|
70
76
|
6. **Assemble** → `book-pack-assemble`; **CP4**.
|
|
71
|
-
7. **Index** → `book-kb-index` loads KUs into the KB (lexical `books.sqlite` + agentdb vector).
|
|
77
|
+
7. **Index** → `book-kb-index` loads KUs into the project KB (lexical `books.sqlite` + agentdb vector).
|
|
78
|
+
This always runs — it is the project's KB, independent of any brain decision.
|
|
72
79
|
8. **Distribute** → **CP5**; then hand over: `dz init --select <the new skills>` and how to use.
|
|
80
|
+
9. **Promote (optional)** → `book-brain-register`: at **CP6**, offer to `dz brain add --project . --source
|
|
81
|
+
<slug>` this book's KUs into the durable **cross-project brain** (`~/.dz/brain/`, `DZ_BRAIN_HOME`).
|
|
82
|
+
**Default is project-local only** — promotion is an explicit, IP-relevant opt-in (like CP5). A
|
|
83
|
+
re-digitized book refreshes non-clobbering via `dz brain update <slug>`; verify with
|
|
84
|
+
`dz brain query "<a KU name>" --source <slug>` + `dz brain primer <slug>`. The brain is private,
|
|
85
|
+
never distributed (§IP). Sibling: repos/dirs take `source-brain-ingest` → the same brain instead.
|
|
73
86
|
|
|
74
87
|
## `--trusted` mode
|
|
75
88
|
|
|
76
|
-
After you've approved 2–3 books, `--trusted` makes CP2 an async spot-check; **CP1, CP3, CP5
|
|
77
|
-
|
|
89
|
+
After you've approved 2–3 books, `--trusted` makes CP2 an async spot-check; **CP1, CP3, CP5, and CP6
|
|
90
|
+
stay interactive** (scope, the skill list, distribution, and brain-promotion are the irreducible
|
|
91
|
+
judgment calls). **CP6 is never auto-approved** — promoting book-derived knowledge into the durable
|
|
92
|
+
cross-project brain is an IP decision, exactly like CP5, so `--trusted` never green-lights it.
|
|
78
93
|
|
|
79
94
|
## Anti-patterns
|
|
80
95
|
|
|
@@ -89,8 +104,9 @@ mandatory** (scope, the skill list, and distribution are the irreducible judgmen
|
|
|
89
104
|
|
|
90
105
|
- [ ] Resumed from the manifest (no completed phase re-run)?
|
|
91
106
|
- [ ] CP1–CP5 honored (scope, samples, skill list, routing gate, distribution)?
|
|
107
|
+
- [ ] CP6 honored — brain promotion was an explicit opt-in (default project-local; never auto-approved, even under `--trusted`)?
|
|
92
108
|
- [ ] Only verified KUs fed distill; every skill claim traces to a KU?
|
|
93
|
-
- [ ] Pack is `private` unless a CP5 decision says otherwise?
|
|
109
|
+
- [ ] Pack is `private` unless a CP5 decision says otherwise; brain kept private (never distributed)?
|
|
94
110
|
- [ ] User got: install command + how the new skills trigger?
|
|
95
111
|
|
|
96
112
|
## Examples
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@dzhechkov/skills-book-digitizer",
|
|
3
|
-
"version": "0.2.
|
|
4
|
-
"description": "Canonical book-digitizer skill pack —
|
|
3
|
+
"version": "0.2.6",
|
|
4
|
+
"description": "Canonical book-digitizer skill pack — 8 skills that turn an engineering book/textbook/monograph into an installable skill pack whose skills make AI coders work within the book's methodologies: digitize-book (orchestrator), book-ingest, book-knowledge-extract, book-skill-distill, book-pack-assemble, book-kb-index, book-brain-register (terminal promote-to-brain, CP6), source-brain-ingest (repo/dir sibling).",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"author": "dzhechko",
|
|
7
7
|
"keywords": [
|
|
@@ -22,6 +22,8 @@
|
|
|
22
22
|
"book-skill-distill",
|
|
23
23
|
"book-pack-assemble",
|
|
24
24
|
"book-kb-index",
|
|
25
|
+
"book-brain-register",
|
|
26
|
+
"source-brain-ingest",
|
|
25
27
|
"scripts",
|
|
26
28
|
"sources.json",
|
|
27
29
|
"README.md",
|
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: source-brain-ingest
|
|
3
|
+
description: >
|
|
4
|
+
The repo/dir SIBLING of the book pipeline (ADR-001 §6): deep-walk a codebase or directory
|
|
5
|
+
(respect .gitignore, skip binaries/node_modules/dist, cap size), then distill DECISION-GRADE
|
|
6
|
+
Knowledge Units in the SAME BookKU shape the book pipeline emits — and feed them into the SAME
|
|
7
|
+
shared brain register via `dz brain add --from-kus`, WITHOUT the book-only stages (no pdftotext
|
|
8
|
+
ingest, no skill distillation). Same KU schema = free reuse of the whole brain (query, primer,
|
|
9
|
+
grounding). Records the source's SPDX license and gates on it. Triggers on: "ingest a repo into
|
|
10
|
+
the brain", "source-brain-ingest", "add this codebase to my knowledge brain".
|
|
11
|
+
trust_tier: 1
|
|
12
|
+
trust_tier_label: "Structured"
|
|
13
|
+
trust_tier_path: "Run /bto-test to promote to Tier 2"
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
# source-brain-ingest — repo/dir → brain KUs (the non-book front-end)
|
|
17
|
+
|
|
18
|
+
One brain, **two ingest front-ends** (ADR §6): the book pipeline turns PDFs into KUs; this turns a
|
|
19
|
+
**repo or directory** into KUs — the *same* KU shape, the *same* register stage, the *same* query /
|
|
20
|
+
primer / grounding surface. A repo plugs into everything the book path already built, for free. This
|
|
21
|
+
is a **sibling** skill — it does not touch, and must not contort, the intact book pipeline.
|
|
22
|
+
|
|
23
|
+
## When to use / NOT
|
|
24
|
+
|
|
25
|
+
- **Use** to distill a codebase/dir into decision-grade Knowledge Units and promote them into the
|
|
26
|
+
durable brain (`~/.dz/brain/`) so `dz brain query` / primers / the grounding hook cover repos too.
|
|
27
|
+
- **NOT for books/PDFs** → use `digitize-book` (pdftotext ingest is book-shaped and would be
|
|
28
|
+
contorted by repo logic; keeping it pure protects the proven book path).
|
|
29
|
+
- **NOT to author skills** → that's `book-skill-distill`. This stage emits KUs into the brain only;
|
|
30
|
+
it does not produce a `skills-<slug>` pack (repos may later return whole files, §8/G5, where books
|
|
31
|
+
return distilled KUs for IP reasons).
|
|
32
|
+
- **NOT to extract/verify book KUs** → that's `book-knowledge-extract`.
|
|
33
|
+
|
|
34
|
+
## Prerequisites
|
|
35
|
+
|
|
36
|
+
- A local repo or directory (a checkout, or a `git clone` you control).
|
|
37
|
+
- `git` (to honour `.gitignore` and capture the commit for provenance) — degrade gracefully on a
|
|
38
|
+
non-git dir (walk the tree; `commit: null`).
|
|
39
|
+
- The brain stores (`dz setup --memory agentdb`) — the same `~/.dz/brain/` the book path promotes into.
|
|
40
|
+
- The source's **SPDX license identifier** (read `LICENSE`/`package.json`; pass `--license <spdx>`).
|
|
41
|
+
|
|
42
|
+
## The Knowledge Unit schema (identical to the book pipeline — that's the whole point)
|
|
43
|
+
|
|
44
|
+
Emit the **same BookKU shape** `book-knowledge-extract` emits, so a repo is just another `source`:
|
|
45
|
+
|
|
46
|
+
```yaml
|
|
47
|
+
id: <repo-slug>-<path-hash>-kuNN # GLOBALLY UNIQUE: <path-hash> discriminates the file/chunk
|
|
48
|
+
# (short hash of the repo-relative path) so two files never
|
|
49
|
+
# collide on kuNN — the brain upsert keys on (source, ku_id,
|
|
50
|
+
# corpus_version) and silently drops id collisions otherwise.
|
|
51
|
+
type: methodology | decision-framework | formula | heuristic | checklist |
|
|
52
|
+
tradeoff-table | case-pattern | definition
|
|
53
|
+
name: "Retry-with-jitter backoff policy"
|
|
54
|
+
problem: the decision/task this knowledge serves (when/why it applies)
|
|
55
|
+
content: the paraphrased procedure / framework / restructured table / verbatim signature-or-formula
|
|
56
|
+
applicability: preconditions, scale ranges, contexts where it holds
|
|
57
|
+
limits: tradeoffs, failure modes, when NOT to apply
|
|
58
|
+
chapter: src/net/backoff.ts # PATH takes the book's `chapter` slot
|
|
59
|
+
pages: [40, 88] # LINE-RANGE takes the book's `pages` slot
|
|
60
|
+
sources: [{ repo: <repo-slug>, path: src/net/backoff.ts, lines: [40, 88], commit: <sha> }]
|
|
61
|
+
metadata: { license: <spdx>, corpus_version: <sha-or-ts>, lang: <detected> }
|
|
62
|
+
verified: true | false | partial # entailment against the cited lines
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Because the shape is identical, the brain treats books and repos on one common scale (query, primer,
|
|
66
|
+
grounding all work unchanged). `kind: repo` in the registry lets the brain treat repos **distinctly**
|
|
67
|
+
where it matters (e.g. return whole files, §8/G5) without a second schema.
|
|
68
|
+
|
|
69
|
+
## Protocol
|
|
70
|
+
|
|
71
|
+
1. **Walk** — deep-walk the repo/dir. **Respect `.gitignore`** (use `git ls-files` when it's a git
|
|
72
|
+
repo — free `.gitignore` honouring + tracked-only). **Skip** `node_modules/`, `dist/`, `build/`,
|
|
73
|
+
`.git/`, vendored trees, lockfiles, and **binaries** (detect by extension + a NUL-byte sniff).
|
|
74
|
+
**Cap** per-file size (skip/point at oversized generated files) and total bytes scanned; log what
|
|
75
|
+
was skipped so coverage is honest.
|
|
76
|
+
2. **Chunk** — split each kept file into page-analogue units (a function/class/section, or a
|
|
77
|
+
line-window for prose docs). Anchor every chunk to `path` + `[startLine–endLine]` — the analogue
|
|
78
|
+
of the book's `[p.N]` anchor; these anchors make the verify step a cheap lookup.
|
|
79
|
+
3. **Distill KUs** — emit **decision-grade** Knowledge Units in the schema above (a decision the code
|
|
80
|
+
encodes, a policy, a tradeoff, a reusable pattern) — NOT a file summary. **Fidelity regime** (same
|
|
81
|
+
as the book path): FACTS (signatures, constants, config values, exact code) **verbatim** with a
|
|
82
|
+
`path:line` anchor; PROSE (rationale, README narrative) **paraphrased** — no uncited verbatim run
|
|
83
|
+
≥ 8 words (this is what a downstream shingling gate checks). Then a fresh checker re-opens each
|
|
84
|
+
KU's cited `path:line` range and sets `verified` (entailment gate; verify all, lookup ≪ generation).
|
|
85
|
+
4. **License-check (the IP gate)** — record the source's SPDX license in every KU's `metadata.license`
|
|
86
|
+
and in what you pass to the register. The register **refuses** an unknown / incompatible license
|
|
87
|
+
without an explicit `--override` — so an unlicensed or GPL-incompatible repo cannot silently land
|
|
88
|
+
in the brain. Repo license travels with the KUs; if the brain ever returns whole files (§8/G5),
|
|
89
|
+
that license binds any downstream reuse.
|
|
90
|
+
5. **Promote** — write the verified KUs to a JSON file in the BookKU shape and hand them to the SAME
|
|
91
|
+
register the book path uses:
|
|
92
|
+
```bash
|
|
93
|
+
dz brain add --from-kus <file.json> --slug <repo-slug> --kind repo --license <spdx>
|
|
94
|
+
```
|
|
95
|
+
This idempotently upserts on `(source, ku_id, corpus_version)` into `~/.dz/brain/` (lexical
|
|
96
|
+
`books.sqlite` + vector `agentdb.db`), writes the `brain.json` registry entry (`kind: repo`,
|
|
97
|
+
license, corpus_version, ku_count), and generates the primer — **no book-ingest, no
|
|
98
|
+
book-skill-distill in the path**. Verify: `dz brain query "<a KU name>"` returns it labeled with
|
|
99
|
+
the repo source.
|
|
100
|
+
|
|
101
|
+
## Anti-patterns
|
|
102
|
+
|
|
103
|
+
| Anti-pattern | Why it fails | Instead |
|
|
104
|
+
|--------------|--------------|---------|
|
|
105
|
+
| Inventing a repo-specific KU schema | breaks the shared brain (query/primer/grounding assume one shape) | emit the exact BookKU schema — path→chapter, lines→pages |
|
|
106
|
+
| Ignoring `.gitignore` / walking `node_modules` | ingests junk + secrets, blows the token budget | `git ls-files`; skip binaries/vendored/dist; cap size |
|
|
107
|
+
| `kuNN` id without a path discriminator | two files collide → upsert-by-id drops KUs | `<repo-slug>-<path-hash>-kuNN` |
|
|
108
|
+
| Routing a repo through `book-ingest`/`digitize-book` | pdftotext + book stages don't fit code | this sibling → `dz brain add --from-kus` |
|
|
109
|
+
| Distilling repo KUs into a skill pack | that's the book path's job; repos may return whole files (§8) | KUs into the brain only |
|
|
110
|
+
| Promoting an unknown/incompatible license | leaks IP-encumbered code into the durable brain | license-check gate; `--override` is an explicit decision |
|
|
111
|
+
| File-summary "KUs" | RAG noise, not decisions | decision-grade KUs (policy/tradeoff/pattern) |
|
|
112
|
+
|
|
113
|
+
## Self-check
|
|
114
|
+
|
|
115
|
+
- [ ] Walk respected `.gitignore`, skipped binaries/`node_modules`/`dist`, capped size — skips logged?
|
|
116
|
+
- [ ] Every KU in the BookKU schema (id `<repo-slug>-<path-hash>-kuNN`; path→chapter; lines→pages; `sources` carry repo+path+commit)?
|
|
117
|
+
- [ ] FACTS verbatim + `path:line`, PROSE paraphrased (no uncited ≥8-word run); all KUs entailment-verified?
|
|
118
|
+
- [ ] Source SPDX license recorded in `metadata.license`; license-check gate passed (or explicit `--override`)?
|
|
119
|
+
- [ ] Promoted via `dz brain add --from-kus … --kind repo --license <spdx>` (NOT book-ingest/distill); registry shows `kind: repo`; `dz brain query` returns a KU labeled with the repo?
|
|
120
|
+
|
|
121
|
+
## Examples
|
|
122
|
+
|
|
123
|
+
- «Add this backoff library to my brain» → deep-walk → KUs like `retry-lib-a1b2c3-ku02`
|
|
124
|
+
(decision-framework «Retry-with-jitter backoff policy», `src/net/backoff.ts:40–88`, verified,
|
|
125
|
+
license MIT) → `dz brain add --from-kus retry-lib.kus.json --slug retry-lib --kind repo --license MIT`.
|
|
126
|
+
- «Почему этот репозиторий не попал в brain?» → license unknown → the register refused it; re-run with
|
|
127
|
+
a recorded `--override` decision, or add a `LICENSE`.
|
|
128
|
+
- «Ground my design questions on our infra repo AND the DDIA book» → both are just `source`s in the one
|
|
129
|
+
brain; `dz brain query "replication topology"` ranks KUs from both, each labeled with its source.
|
package/sources.json
CHANGED
|
@@ -16,6 +16,8 @@
|
|
|
16
16
|
"book-knowledge-extract": { "upstream_type": "original", "upstream_path": null, "description": "Corpus → verified Knowledge Units (map-reduce, KU-verify, fidelity regime)" },
|
|
17
17
|
"book-skill-distill": { "upstream_type": "original", "upstream_path": null, "description": "Verified KUs → decision-moment skills (routing evals, traceability)" },
|
|
18
18
|
"book-pack-assemble": { "upstream_type": "original", "upstream_path": null, "description": "Skills → installable pack (references/, private-default, collision gate)" },
|
|
19
|
-
"book-kb-index": { "upstream_type": "original", "upstream_path": null, "description": "KUs → books.sqlite (lexical) + agentdb (vector) KB, upsert not append" }
|
|
19
|
+
"book-kb-index": { "upstream_type": "original", "upstream_path": null, "description": "KUs → books.sqlite (lexical) + agentdb (vector) KB, upsert not append" },
|
|
20
|
+
"book-brain-register": { "upstream_type": "original", "upstream_path": null, "description": "Optional terminal stage (CP6) — promote a project's KUs into the durable cross-project brain (~/.dz/brain/) via dz brain add" },
|
|
21
|
+
"source-brain-ingest": { "upstream_type": "original", "upstream_path": null, "description": "Sibling (ADR §6) — deep-walk a repo/dir into the same KUs → same brain via dz brain add --from-kus --kind repo" }
|
|
20
22
|
}
|
|
21
23
|
}
|