@dzhechkov/skills-book-digitizer 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 dzhechko
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,64 @@
1
+ # @dzhechkov/skills-book-digitizer
2
+
3
+ Turn an engineering **book / textbook / monograph** into an **installable skill pack** whose skills
4
+ make AI coders (Claude Code, Codex, …) *work within the book's methodologies* — plus a searchable
5
+ Knowledge Base behind them. dz-original; design & rationale in
6
+ [ADR-001 v2](https://github.com/djd1m/dz-harness-hub/blob/main/features/book-knowledge-digitizer/03_adr/001-book-to-skillpack-pipeline.md)
7
+ (passed a 4-lens adversarial design panel: 28 findings, all folded in).
8
+
9
+ ## What it produces
10
+
11
+ `book.pdf` → `skills-book-<slug>`: a pack you `dz init --select …` so the agent applies the book's
12
+ methods at the decisions it's about — e.g. DDIA → `ddia-replication-topology-choice`,
13
+ `ddia-partitioning-strategy`, `ddia-consistency-model-selection`.
14
+
15
+ ## The 6 skills (a checkpointed pipeline)
16
+
17
+ | Skill | Stage |
18
+ |-------|-------|
19
+ | `digitize-book` | **Orchestrator** — runs the pipeline through 5 human checkpoints, resumable across sessions |
20
+ | `book-ingest` | PDF → chunked, page-anchored corpus + manifest (deterministic: poppler + heuristics, no LLM) |
21
+ | `book-knowledge-extract` | corpus → **verified** Knowledge Units (map-reduce, dedup, KU-verify entailment, FACTS-verbatim/PROSE-paraphrase regime) |
22
+ | `book-skill-distill` | verified KUs → **decision-moment** skills (slug-prefixed, routing evals + activation gate, KU→skill traceability) |
23
+ | `book-pack-assemble` | skills → installable pack (`references/` deep-lookup shipped inside, **private-by-default** distribution, collision gate, Copilot policy) |
24
+ | `book-kb-index` | KUs → the memory layer: `.dz/memory/books.sqlite` (lexical) + shared agentdb (vector) |
25
+
26
+ ## Install & use
27
+
28
+ ```bash
29
+ dz init --target claude-code --select digitize-book
30
+ # then, in your agent: «оцифруй вот эту книгу — хочу, чтобы агент проектировал по её методикам»
31
+ ```
32
+
33
+ Prerequisites: `poppler-utils` (ingest) and, for the KB, `dz setup --memory agentdb`
34
+ (installs `agentdb` + `better-sqlite3`).
35
+
36
+ ## Design guarantees — structural vs. protocol (stated honestly)
37
+
38
+ Two kinds of guarantee, and the difference matters:
39
+
40
+ - **Structural (mechanically enforced by code/packaging):** generated packs ship `private: true` +
41
+ restricted `publishConfig` and get **no automatic preset entry** — a book-derived pack cannot be
42
+ auto-published; distribution is your explicit CP5 decision. `files[]` whitelist + no-`origin`
43
+ `sources.json`. The ingest is deterministic. The book KB is a separate namespace (book KUs never
44
+ pollute `dz recommend`).
45
+ - **Protocol (agent-enforced, fail-open — NOT mechanically proven):** KU provenance is
46
+ *agent-verified* (an entailment pass re-reads the cited pages) but not cryptographically proven;
47
+ the FACTS-verbatim / PROSE-paraphrase fidelity regime and the KU→skill traceability are
48
+ discipline the distill/extract agents follow. Generated skills therefore ship at **`trust_tier`
49
+ 0/1 ("machine-distilled, unreviewed")** with a documented promote path (human review against the
50
+ cited pages, or passing the routing evals). A scripted word-shingling verbatim-check is the
51
+ intended hard gate; it is **specified in the ADR but not yet shipped as a script** — until then
52
+ the paraphrase rule is agent-enforced, not automatic.
53
+
54
+ - **Honest cost** — a 640-page book ≈ 2.5–4M agent-tokens end-to-end; the orchestrator shows the
55
+ estimate at CP1 and hard-gates the bulk map behind your sample review at CP2.
56
+
57
+ ## Provenance
58
+
59
+ dz-original (ADR-001). Delegates by reference to `skill-crystallizer` (skill scaffolding/benchmark)
60
+ and `knowledge-extractor` (quality-gate discipline) — see `sources.json`.
61
+
62
+ ## License
63
+
64
+ MIT © dzhechko.
@@ -0,0 +1,110 @@
1
+ ---
2
+ name: book-ingest
3
+ description: >
4
+ Stage 1 of the book digitizer: turn a book PDF into a chunked, page-anchored corpus + a
5
+ manifest, DETERMINISTICALLY (poppler + heuristics, no LLM). Token-budget chunker, per-page
6
+ quality matrix (empty/low-coherence/math-dense/two-column), figure/table inventory,
7
+ front-matter listed-not-skipped, deterministic corpus_version + per-chunk source_hash and
8
+ watermarks. Triggers on: "ingest this book", "book-ingest", "разбей книгу на главы",
9
+ "prepare book corpus". Usually invoked by `digitize-book`, not directly.
10
+ trust_tier: 1
11
+ trust_tier_label: "Structured"
12
+ trust_tier_path: "Run /bto-test to promote to Tier 2"
13
+ ---
14
+
15
+ # book-ingest — PDF → corpus + manifest (deterministic)
16
+
17
+ The reproducible foundation of the pipeline: everything downstream trusts the manifest. **No LLM
18
+ here** — a book must ingest the same way every time, or resume and idempotency break.
19
+
20
+ ## When to use / NOT
21
+
22
+ - **Use** as the first stage of `digitize-book`, or standalone to inspect a book's structure.
23
+ - **NOT** for extracting knowledge (that's `book-knowledge-extract`) — ingest only structures text.
24
+
25
+ ## Prerequisites
26
+
27
+ ```bash
28
+ pdftotext -v && pdfinfo -v # poppler-utils; on Debian/Ubuntu: apt-get install poppler-utils
29
+ ```
30
+
31
+ ## Protocol
32
+
33
+ Run the bundled deterministic script (it lives in this skill's pack under `scripts/`):
34
+
35
+ ```bash
36
+ node <pack>/scripts/book-ingest.mjs <book.pdf> --out <workspace> \
37
+ --book <slug> --isbn <isbn> [--max-chunk-tokens 70000]
38
+ ```
39
+
40
+ What it produces:
41
+
42
+ - `<workspace>/corpus/NNN-<chunk-slug>.md` — each chunk's text with **`[p.N]` page anchors** before
43
+ every page (the anchors are what make provenance verifiable downstream).
44
+ - `<workspace>/manifest.json` — the contract for the whole run:
45
+
46
+ ```jsonc
47
+ {
48
+ "book": { "slug", "isbn", "title", "pages", "source_file" },
49
+ "structure_type": "chaptered | monolithic", // "collection" (papers) is a planned enhancement
50
+ "corpus_version": "<hash of chunk page-ranges + settings>", // bumps on any re-ingest
51
+ "chapters": [{ "num", "title", "pages": [a, b] }],
52
+ "front_matter": [{ "page", "reason": "empty-text | pre-first-chapter" }], // LISTED, never dropped
53
+ "chunks": [{
54
+ "id", "file", "parent_chapter", "pages": [a, b], "token_estimate",
55
+ "figure_table_mentions", "page_flags": { "<page>": ["math_dense", "two_column"] },
56
+ "source_hash": "<sha of chunk text>",
57
+ "watermark": { "extracted": false, "reduced": false, "indexed": false }
58
+ }],
59
+ "page_quality_summary": { "flag_counts", "total_figure_table_mentions" },
60
+ "phase_state": { "extracted": 0, "reduced": false, "distilled": false, "packed": false }
61
+ }
62
+ ```
63
+
64
+ Key design points (all in the script, ADR-001 v2):
65
+
66
+ - **Token-budget chunker**, not chapter=unit: a chapter over `max-chunk-tokens` splits at
67
+ sub-headings → page windows, so every chunk is agent-sized. `parent_chapter` links split chunks.
68
+ - **Per-page quality matrix**: `empty` (cover/blank → OCR candidate), `low_coherence` (garbled),
69
+ `math_dense` (formula-heavy → the extract stage must quarantine, not reproduce, mangled formulas),
70
+ `two_column` (layout divergence). Flags are advisory signals surfaced at CP1 — nothing silent.
71
+ - **Figure/table inventory**: counts «Рис. N.M» / «Таблица N.M» / «Figure/Table» *mentions* (text
72
+ regex) per chunk, so the extract coverage report can flag uncaptured visuals. (`pdfimages` is an
73
+ optional deeper check, not required by the current script.)
74
+ - **Front matter is LISTED**, never silently skipped (the "no silent caps" rule).
75
+ - **Determinism**: `corpus_version` and per-chunk `source_hash` are content hashes — a re-ingest
76
+ with the same PDF + settings yields identical values (verified on a 640-page book).
77
+
78
+ Then relay the manifest summary to the user for **CP1** (structure_type, chapter count, chunk
79
+ count, front-matter pages, quality flags, figure/table total, and the token-cost estimate).
80
+
81
+ ## OCR / bad-layout fallback (when quality flags are heavy)
82
+
83
+ - Many `empty` pages on a scanned book → run `tesseract` per page image (`pdftoppm` → PNG) and
84
+ re-ingest from the OCR text; mark those pages' KUs `ocr: true`.
85
+ - Many `math_dense` pages → those chunks carry the flag; the extract stage MUST quarantine
86
+ formulas from mangled text (cite «формула на с. X, см. источник») rather than reproduce them.
87
+ Better path (enhancement): render flagged pages via `pdftoppm` + a vision model, set
88
+ `formula_verified: true` only when transcribed from the page image.
89
+
90
+ ## Anti-patterns
91
+
92
+ | Anti-pattern | Why it fails | Instead |
93
+ |--------------|--------------|---------|
94
+ | Hand-editing the corpus/manifest | breaks corpus_version → resume + idempotency | re-run ingest |
95
+ | Using an LLM to chunk | non-deterministic → duplicate/re-pointed KUs on resume | the script is deterministic |
96
+ | Trusting `math_dense` chunk text for formulas | pdftotext mangles math | quarantine + cite / vision fallback |
97
+ | Silently dropping front matter | violates no-silent-caps | it's listed for CP1 |
98
+
99
+ ## Self-check
100
+
101
+ - [ ] `pdftotext`/`pdfinfo` present; PDF not encrypted?
102
+ - [ ] manifest chapter page-ranges look right (spot-check 2 chapters vs the book)?
103
+ - [ ] corpus_version stable on a re-run (determinism)?
104
+ - [ ] Heavy quality flags triaged (OCR / formula policy) before extraction?
105
+
106
+ ## Examples
107
+
108
+ - «book-ingest ddia.pdf» → 640 p. → 12 chapters, ~19 chunks (oversized chapters auto-split), 24 front-matter pages listed.
109
+ - «Почему пропущены страницы 1–24?» → they're `front_matter` in the manifest (pre-first-chapter),
110
+ listed not dropped.
@@ -0,0 +1,87 @@
1
+ ---
2
+ name: book-kb-index
3
+ description: >
4
+ Stage 5 of the book digitizer (supporting): load verified Knowledge Units into the memory layer —
5
+ the lexical books.sqlite namespace (dz recall --books) AND the shared agentdb vector index
6
+ (agentdb_pattern_search), so book know-how is recallable lexically and semantically across all
7
+ digitized books. Upsert by (book, ku_id, corpus_version), never append-only. Triggers on:
8
+ "index the book KB", "book-kb-index", "load KUs into memory". Invoked by `digitize-book`.
9
+ trust_tier: 1
10
+ trust_tier_label: "Structured"
11
+ trust_tier_path: "Run /bto-test to promote to Tier 2"
12
+ ---
13
+
14
+ # book-kb-index — KUs → the searchable Knowledge Base
15
+
16
+ The layer BEHIND the distilled skills: when a skill says «см. источник, гл.5», or when you want to
17
+ recall book knowledge that never became a standalone skill, this is what answers. Two stores, one
18
+ purpose.
19
+
20
+ ## When to use / NOT
21
+
22
+ - **Use** as stage 5 of `digitize-book`, to load verified KUs into the searchable memory layer.
23
+ - **NOT** to extract or verify KUs (that's `book-knowledge-extract`) — this stage only indexes them into the two stores.
24
+
25
+ ## Prerequisites
26
+
27
+ - Verified KUs from `book-knowledge-extract`.
28
+ - `agentdb` + `better-sqlite3` in the project (`dz setup --memory agentdb`) for both stores.
29
+
30
+ ## The two stores (why two)
31
+
32
+ | Store | Backend | Recall | Reached via |
33
+ |-------|---------|--------|-------------|
34
+ | **Lexical** | `.dz/memory/books.sqlite` (FTS5) | keyword | `dz recall --books` |
35
+ | **Semantic** | shared `.dz/agentdb.db` (vector) | similarity | `agentdb_pattern_search` |
36
+
37
+ The lexical store is a **separate namespace by construction** — `computePatternBoost` (the
38
+ `dz recommend` reward loop) never sees book KUs, and they never expire under default retention.
39
+
40
+ **Lexical matching (FTS5, know its limits):** `books.sqlite` now does **prefix** matching
41
+ (AND-of-terms) — morphology-tolerant, so `репликаци` matches `репликация`/`репликации`. But it has
42
+ **no stemming and no synonymy**, and it is single-language. So route **cross-lingual, fuzzy, or
43
+ semantic** queries through the VECTOR store (agentdb, multilingual embeddings) via
44
+ `agentdb_pattern_search` — lexical prefix-AND is for exact-ish keyword recall only.
45
+
46
+ ## Protocol
47
+
48
+ 1. **Idempotent upsert** — for each verified KU, write to BOTH stores keyed on
49
+ `(book, ku_id, corpus_version)`. A re-ingest bumps `corpus_version`; indexing then **evicts**
50
+ the book's rows from any stale corpus_version before inserting (upsert, never append) — so the
51
+ KB mirrors exactly the current ingest and re-runs don't duplicate.
52
+ - Lexical: `putBookKnowledge(projectRoot, kus)` (harness-core) — FTS5 over name/problem/content.
53
+ - Semantic: `indexPatternsToAgentdb(projectRoot, rows)` (harness-core) — native better-sqlite3
54
+ insert into ReasoningBank's schema (agentdb only for embeddings; never its sql.js
55
+ `createDatabase`). This primitive is **INSERT-only**, so to stay idempotent the skill FIRST
56
+ deletes this book's rows from a stale corpus_version (a better-sqlite3
57
+ `DELETE FROM reasoning_patterns WHERE tags LIKE '%"'||<slug>||'"%' AND metadata NOT LIKE
58
+ '%'||<corpus_version>||'%'` — mirror the lexical store's eviction) BEFORE calling it. Map each
59
+ KU → `{ taskType: 'book-knowledge', text: \`${name}: ${problem}\`, score: <documented
60
+ confidence, NOT a fabricated 1.0>, tags: ['book', <slug>, type], metadata: { isbn, chapter,
61
+ pages, ku_id, corpus_version } }`. (The lexical `putBookKnowledge` upserts automatically.)
62
+ 2. **Honest score** — `score` = the KU's extraction/verify confidence (documented mapping), never a
63
+ fake 1.0 (the pollution lesson from the session-hook telemetry).
64
+ 3. **Mark** each KU's `watermark.indexed = true` in the manifest (resume skips it next run).
65
+ 4. **Verify recall** — a lexical probe (`dz recall --books "<a KU name>"`) and, if the MCP server is
66
+ up, a semantic probe (`agentdb_pattern_search`) return the KU with its page provenance.
67
+
68
+ ## Anti-patterns
69
+
70
+ | Anti-pattern | Why it fails | Instead |
71
+ |--------------|--------------|---------|
72
+ | Append-only indexing | re-ingest duplicates KUs | lexical auto-upserts; for the vector store, DELETE stale-corpus rows before insert |
73
+ | Book KUs in the pattern store | pollutes `dz recommend` boost | separate books.sqlite namespace |
74
+ | Fabricated score 1.0 | biases vector ranking | honest extraction confidence |
75
+ | Using agentdb's createDatabase to write | sql.js corrupts the shared DB | native better-sqlite3 insert |
76
+
77
+ ## Self-check
78
+
79
+ - [ ] Both stores written; keyed on (book, ku_id, corpus_version)?
80
+ - [ ] Lexical upserted; vector stale-corpus rows explicitly deleted before insert (no dup vectors)?
81
+ - [ ] Scores honest; book KUs excluded from `computePatternBoost`?
82
+ - [ ] `watermark.indexed` set; recall probe returns KU + provenance?
83
+
84
+ ## Examples
85
+
86
+ - «Index the DDIA KUs» → 180 KUs → lexical + vector; `dz recall --books "репликация"` returns them.
87
+ - «Найди в книге про кворумы» → `agentdb_pattern_search` surfaces the quorum KU with gл.5 pages.
@@ -0,0 +1,117 @@
1
+ ---
2
+ name: book-knowledge-extract
3
+ description: >
4
+ Stage 2 of the book digitizer: map-reduce the ingested corpus into page-anchored Knowledge Units
5
+ (methodologies, decision-frameworks, formulas, heuristics, checklists, tradeoff-tables,
6
+ case-patterns, definitions), then dedup/merge and run a KU-VERIFY entailment pass so only
7
+ KUs actually supported by their cited pages proceed. Enforces the FACTS-verbatim /
8
+ PROSE-paraphrase fidelity regime. Triggers on: "extract knowledge from the corpus",
9
+ "book-knowledge-extract", "harvest KUs". Invoked by `digitize-book`.
10
+ trust_tier: 1
11
+ trust_tier_label: "Structured"
12
+ trust_tier_path: "Run /bto-test to promote to Tier 2"
13
+ ---
14
+
15
+ # book-knowledge-extract — corpus → verified Knowledge Units
16
+
17
+ Turn structured text into **actionable know-how**, page-anchored and verified. This is where the
18
+ book's expertise becomes machine-usable — and where the IP and honesty rules bite.
19
+
20
+ ## When to use / NOT
21
+
22
+ - **Use** as stage 2 of `digitize-book`, after `book-ingest` produced a manifest + corpus.
23
+ - **NOT** to author skills (that's `book-skill-distill`) or to structure text (that's ingest) — this stage produces verified Knowledge Units only.
24
+
25
+ ## Prerequisites
26
+
27
+ A manifest + corpus from `book-ingest`. Reuse `knowledge-extractor`'s quality-gate discipline by
28
+ reference (`Read` its SKILL.md) — do not restate it; ADD only the book-specific gates below.
29
+
30
+ ## The Knowledge Unit schema (the shared contract for the whole pipeline)
31
+
32
+ ```yaml
33
+ id: <book>-<chunk>-<seq> # assigned within ONE atomic chunk run (see idempotency)
34
+ type: methodology | decision-framework | formula | heuristic | checklist |
35
+ tradeoff-table | case-pattern | definition
36
+ name: "Выбор топологии репликации"
37
+ problem: when/why this knowledge applies (the decision or task it serves)
38
+ content: the paraphrased procedure / framework / restructured table / verbatim formula
39
+ applicability: preconditions, scale ranges, contexts where it holds
40
+ limits: tradeoffs, failure modes, when NOT to apply
41
+ sources: [{ isbn, title, chapter, pages: [169, 214], unit_title? }] # LIST (survives merges)
42
+ aliases: [] # ids merged into this KU at reduce
43
+ skill_worthiness: high | medium | low # HINT — reduce re-scores with global view
44
+ verified: true | false | partial # set by the KU-verify pass; distill needs true
45
+ ```
46
+
47
+ ## Protocol
48
+
49
+ ### Map — one agent per chunk (parallelizable, resumable)
50
+
51
+ For each chunk whose `watermark.extracted == false`: read `corpus/<chunk>.md`, emit KUs in the
52
+ schema. **Idempotency**: `<seq>` is assigned only within one complete chunk run — write to a temp
53
+ file, then atomically rename + set `watermark.extracted = true` together; on retry, delete partial
54
+ output first. A changed `source_hash` invalidates that chunk's watermark AND all downstream
55
+ phase watermarks.
56
+
57
+ **Fidelity regime (the IP contract — non-negotiable):**
58
+ - **OUTPUT LANGUAGE (pin it — one language per book):** the book's language by default — detect from
59
+ the corpus (Cyrillic-dominant → Russian, Latin → English) — OR an explicit `--lang <code>` override
60
+ the orchestrator passes. ALL KU fields (`name`, `problem`, `content`, `applicability`, `limits`)
61
+ must be in that ONE language — **never mix**. Rationale: a consistent language is required for
62
+ lexical recall (`dz recall --books`) and for coherent distilled skills; a Russian book digitized
63
+ into English KUs breaks RU search and reads incoherently.
64
+ - **FACTS** (formulas, numeric thresholds, table DATA, algorithm names, code): preserved
65
+ **verbatim** with a page anchor. A `formula` KU's `content` must byte-match the source
66
+ expression — UNLESS the chunk flags the page `math_dense` (mangled), in which case the KU may
67
+ only POINT («формула на с. X, см. источник»), never reproduce a mangled formula.
68
+ - **PROSE** (explanations, narrative): **paraphrased** — no verbatim run ≥ 8–10 words outside a
69
+ cited quote (quotes ≤ ~25 words each, with an immediate page citation). **If the skills are
70
+ authored in a DIFFERENT language than the source** (e.g. RU book → EN skills), n-gram checking is
71
+ insufficient — run a sampled LLM-judge paraphrase audit and gate on it.
72
+ - **TABLES**: RESTRUCTURED, never transcribed — re-derive axes from the decision the KU serves,
73
+ reword cells, reorder/filter rows, cite «по мотивам табл. X, гл. Y, с. Z».
74
+ - Every KU accounts for the chunk's inventoried figures/tables — reconstruct (mermaid/markdown) or
75
+ emit `not_captured`. The coverage report counts capture rate, not just chunks.
76
+
77
+ ### Reduce — single-threaded, after a barrier on all map watermarks
78
+
79
+ 1. **Blocking** — embed every KU (via `book-kb-index`'s embedding path) and cluster by cosine
80
+ similarity within the same `type` (deterministic, near-free).
81
+ 2. **Adjudication** — one small LLM call per above-threshold cluster: merge / link / keep-separate.
82
+ Merged KUs get a new id + `aliases` for the constituents (no dangling references); `sources`
83
+ accumulates. Cross-links between related KUs are recorded (distill uses them for "Related
84
+ decisions").
85
+ 3. Re-score `skill_worthiness` with the coverage report in view (map scores were hints).
86
+
87
+ ### KU-verify — the provenance gate (adversarial-verifier pattern)
88
+
89
+ A fresh checker agent re-opens each KU's cited page range in the corpus (the `[p.N]` anchors make
90
+ this a cheap lookup) and scores entailment: *is this KU actually supported by these pages?* Set
91
+ `verified: true|false|partial`. **Verify ALL KUs — do not sample** (lookup ≪ generation cost).
92
+ Only `verified` KUs proceed to distill. This is what stops hallucinated page anchors.
93
+
94
+ ## Anti-patterns
95
+
96
+ | Anti-pattern | Why it fails | Instead |
97
+ |--------------|--------------|---------|
98
+ | Paraphrasing a formula/threshold | corrupts the highest-value content | FACTS verbatim + anchor |
99
+ | Transcribing a book table | IP + it mirrors the book's framing | restructure around the decision |
100
+ | Sampling KU-verify | hallucinated anchors slip through | verify all — it's cheap |
101
+ | Extracting raw summary instead of know-how | RAG, not behavior | KUs are decisions/procedures |
102
+ | Reproducing a formula from a `math_dense` page | pdftotext mangled it | point + cite, or vision-verify |
103
+ | Mixed-language KUs from a single book | breaks lexical recall — half the corpus is unsearchable | pin the language (book default or `--lang`) |
104
+
105
+ ## Self-check
106
+
107
+ - [ ] Every KU page-anchored to a real `[p.N]` in the corpus?
108
+ - [ ] FACTS verbatim, PROSE paraphrased, TABLES restructured — regime respected?
109
+ - [ ] Figures/tables accounted for (reconstructed or `not_captured`)?
110
+ - [ ] Reduce merged duplicates with `aliases`; `skill_worthiness` re-scored globally?
111
+ - [ ] KU-verify run on ALL KUs; only `verified` proceed?
112
+
113
+ ## Examples
114
+
115
+ - «Extract chapter 5» → KUs like `ddia-ch05-ku03` (decision-framework «Выбор топологии
116
+ репликации», pp.169–214, verified).
117
+ - «Почему этот KU не попал в дистилляцию?» → `verified: false` — the cited pages didn't support it.
@@ -0,0 +1,92 @@
1
+ ---
2
+ name: book-pack-assemble
3
+ description: >
4
+ Stage 4 of the book digitizer: assemble the distilled skills into an installable pack
5
+ skills-book-<slug> (ADR-0001 library layout) — files[] whitelist, references/ deep-lookup
6
+ shipped INSIDE the pack, private-by-default distribution + CP5 decision, catalog-collision gate,
7
+ Copilot lean-body policy, no-origin sources.json (upstream_type: book), L0 benchmark + install
8
+ smoke. Triggers on: "assemble the book pack", "book-pack-assemble", "build skills-book-<slug>".
9
+ Invoked by `digitize-book`.
10
+ trust_tier: 1
11
+ trust_tier_label: "Structured"
12
+ trust_tier_path: "Run /bto-test to promote to Tier 2"
13
+ ---
14
+
15
+ # book-pack-assemble — skills → installable pack
16
+
17
+ Package the distilled skills so `dz init --select …` drops the book's methodology into any of the
18
+ 6 targets — with the distribution and quality guardrails the panel demanded.
19
+
20
+ ## When to use / NOT
21
+
22
+ - **Use** as stage 4 of `digitize-book`, once skills are distilled and routing-gated.
23
+ - **NOT** to author skills (that's `book-skill-distill`) or to index the KB (that's `book-kb-index`) — this stage builds the installable pack.
24
+
25
+ ## Prerequisites
26
+
27
+ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benchmark` for the gate.
28
+
29
+ ## Protocol
30
+
31
+ 1. **Scaffold `skills-book-<slug>/`** (ADR-0001 library layout — skill dirs at pack root):
32
+ ```
33
+ skills-book-<slug>/
34
+ <slug>-<decision>/SKILL.md
35
+ <slug>-<decision>/references/*.md ← the deep-lookup, shipped IN the pack
36
+ <slug>-<decision>/evals/routing.yaml
37
+ package.json sources.json README.md LICENSE
38
+ ```
39
+ 2. **`references/` ships the deep-lookup inside the pack** — each skill's consumed KUs in full
40
+ (+ condensed chapter extracts, size permitting). So «см. источник» is a file read that works on
41
+ every target/machine, not a call to a KB the recipient doesn't have. `sources.json` declares
42
+ which lookup tiers exist per skill: `references/` = always, corpus = owner-local, agentdb =
43
+ owner-local — the skill degrades honestly.
44
+ 3. **`files[]` whitelist** every skill dir + `references`/`evals` + `sources.json` + `README.md` +
45
+ `LICENSE` (forget one → npm silently drops it). Verify the tarball after.
46
+ 4. **`package.json` distribution guardrails (structural, not policy):**
47
+ `"private": true` + `"publishConfig": { "access": "restricted" }`. **No automatic
48
+ `harness-presets` entry** (presets are the public install surface). Flipping to publishable
49
+ requires the recorded **CP5** decision (a `--allow-publish` step) written into `sources.json`
50
+ (`distribution`, `decided_by`, `date`). The gate FAILS if `private` is absent without a recorded
51
+ decision.
52
+ 5. **`sources.json`** — documentation-only form, **NO `origin` block, ever** (the book is the
53
+ immutable upstream; `dz sync-upstream` intentionally skips book packs). Per skill: `upstream_type:
54
+ "book"`, isbn/title/edition, `derived_from` KU ids, verified-KU ratio, digitizer version,
55
+ extraction model, date. Declare the cross-pack deps (`skill-crystallizer`, `knowledge-extractor`)
56
+ so `dz bundle`'s cross-skill-dep warning fires honestly.
57
+ 6. **Catalog-collision gate**: embed the new descriptions + all installed descriptions, flag
58
+ high-similarity pairs, require each resolved (differentiated description or explicit «NOT for X —
59
+ use Y» boundary clause). Also run the plain uniqueness check against `dz registry` (belt +
60
+ suspenders over slug-prefixing).
61
+ 7. **Copilot policy**: keep SKILL.md bodies lean (trigger + protocol skeleton); big
62
+ criteria/tradeoff tables live in `references/*.md`. README carries a Copilot note («always-on
63
+ there — install a subset»). At CP4 with `target=copilot`, surface the aggregate instruction-file
64
+ byte count.
65
+ 8. **Gate + smoke**: `dz benchmark <pack> --all` (honest L0 grade per skill — never fabricate
66
+ schemas/evals to game it) → `dz init --target claude-code --select <skills>` install smoke →
67
+ **CP4** report.
68
+
69
+ ## Anti-patterns
70
+
71
+ | Anti-pattern | Why it fails | Instead |
72
+ |--------------|--------------|---------|
73
+ | Referencing an agentdb KB the recipient lacks | broken deep-lookup on their machine | ship `references/` in the pack |
74
+ | `private` absent / auto public publish | book-derived pack leaks | structural `private:true` + CP5 |
75
+ | Forgetting a skill dir in `files[]` | npm silently drops it | whitelist + verify tarball |
76
+ | `origin` block in sources.json | sync-upstream would chase a non-repo | documentation-only, no origin |
77
+ | Fat SKILL.md bodies on Copilot | floods every request | lean body + `references/` |
78
+
79
+ ## Self-check
80
+
81
+ - [ ] `files[]` whitelists every dir; tarball verified?
82
+ - [ ] `references/` deep-lookup ships inside the pack; tiers declared in sources.json?
83
+ - [ ] `private:true` + restricted; no auto preset; CP5 recorded if publishing?
84
+ - [ ] Collision gate clean; slug-prefixed unique ids?
85
+ - [ ] Faithfulness gate run (each shipped skill's claims supported by its `derived_from` KUs)?
86
+ - [ ] L0 grades honest; install smoke passed; Copilot bytes shown if relevant?
87
+
88
+ ## Examples
89
+
90
+ - «Assemble the DDIA pack» → `skills-book-ddia` (8 skills, private, references shipped, L0 B/A) →
91
+ `dz init --select ddia-partitioning-strategy`.
92
+ - «Хочу опубликовать этот пак» → CP5 records the decision, flips `private`, then a manual publish.
@@ -0,0 +1,102 @@
1
+ ---
2
+ name: book-skill-distill
3
+ description: >
4
+ Stage 3 of the book digitizer (the heart): cluster verified Knowledge Units into DECISION-MOMENT
5
+ skills — the choices an AI coder actually faces — and author each as a house-format SKILL.md that
6
+ makes the agent apply the book's method. Slug-prefixed ids, routing evals + a CP3.5 activation
7
+ gate, KU→skill traceability, related-decision links, trust_tier 0/1. Triggers on:
8
+ "distill skills from the book", "book-skill-distill", "turn KUs into skills". Invoked by
9
+ `digitize-book`.
10
+ trust_tier: 1
11
+ trust_tier_label: "Structured"
12
+ trust_tier_path: "Run /bto-test to promote to Tier 2"
13
+ ---
14
+
15
+ # book-skill-distill — verified KUs → agent-behavior skills
16
+
17
+ The stage that decides *which* of the book's knowledge becomes agent behavior, and authors it so
18
+ it actually triggers and is trustworthy. Delegate scaffolding + benchmarking to `skill-crystallizer`
19
+ (`Read` its SKILL.md) — this skill adds the book-specific contract below.
20
+
21
+ ## When to use / NOT
22
+
23
+ - **Use** as stage 3 of `digitize-book`, on the verified KUs from extract.
24
+ - **NOT** to extract KUs (that's `book-knowledge-extract`) or to package skills (that's `book-pack-assemble`) — this stage authors decision-moment SKILL.md's.
25
+
26
+ ## Prerequisites
27
+
28
+ Verified KUs + cross-link graph from `book-knowledge-extract`. The installed catalog (`dz registry`)
29
+ for the collision + routing checks.
30
+
31
+ ## Protocol
32
+
33
+ ### 1. Cluster into decision moments (not chapter mirrors)
34
+
35
+ Group `skill_worthiness: high` (and strong `medium`) KUs by the **decision an AI coder hits**, not
36
+ by chapter. DDIA → `ddia-db-model-selection`, `ddia-storage-engine-tradeoffs`,
37
+ `ddia-encoding-evolution`, `ddia-replication-topology-choice`, `ddia-partitioning-strategy`,
38
+ `ddia-transactions-isolation-choice`, `ddia-consistency-model-selection`,
39
+ `ddia-batch-vs-stream-processing`. Present the candidate list at **CP3** for approval.
40
+
41
+ ### 2. Naming & layout
42
+
43
+ - **Slug-prefix every skill id** with the book slug (`<slug>-<decision>`) — provenance in the id,
44
+ zero-lookup, collision-safe across books. The decision-moment phrasing lives in the description.
45
+ - **Flat vs gateway** (decide at CP3 by catalog pressure): ≤ ~10 skills → flat; more → a per-book
46
+ **gateway skill** (one description routing to protocols shipped as `references/`).
47
+
48
+ ### 3. Author each skill (house L0 format)
49
+
50
+ - frontmatter: name (slug-prefixed), description with **decision-moment triggers** and an explicit
51
+ **boundary clause** («NOT for schema-change review — use `database-review`»), `trust_tier: 0`,
52
+ `trust_tier_label: "Machine-distilled from <book> (unreviewed)"`.
53
+ - **Protocol** = the book's decision criteria as concrete steps + a restructured criteria table.
54
+ - **Anti-patterns** = the book's warnings/failure-modes.
55
+ - **Related decisions** = sibling skills with the one-line coupling constraint (from the cross-link
56
+ graph): «If you chose leaderless replication here, `consistency-model-selection`: quorum math
57
+ changes.»
58
+ - **Источник** = citation COMPUTED from the consumed KUs' `sources` (chapter/pages) + 1–2 short
59
+ verbatim anchor quotes for a human spot-check + a pointer to KB deep-lookup.
60
+ - Self-check + Examples.
61
+
62
+ ### 4. Traceability contract (anti-hallucination — non-negotiable)
63
+
64
+ - frontmatter carries `derived_from: [ku_ids]`; the «Источник» citation is **computed** from those
65
+ KUs, never free-typed by the distill agent.
66
+ - Content not traceable to a listed KU is removed OR explicitly labeled «адаптация для агента, не
67
+ из книги» — the book citation covers only KU-derived material.
68
+ - **Faithfulness check**: an independent judge verifies each Protocol step / Anti-pattern is
69
+ supported by the referenced KUs (KUs are page-anchored → checkable). Unsupported claims are
70
+ flagged and fixed before the pack gate.
71
+
72
+ ### 5. Routing evals (CP3.5 — the value gate)
73
+
74
+ For each skill, author `evals/routing.yaml`: ~10 **positive** prompts (realistic RU+EN coding-session
75
+ phrasings for that decision) + **hard negatives** owned by sibling skills AND by named existing
76
+ arsenal skills. Run them against the FULL installed catalog; require **≥80% correct activation /
77
+ ≤10% sibling-steal** before pack assembly. A skill that can't route is a skill that never fires.
78
+
79
+ ## Anti-patterns
80
+
81
+ | Anti-pattern | Why it fails | Instead |
82
+ |--------------|--------------|---------|
83
+ | One skill per chapter | breaks triggering + context budget | one skill per decision moment |
84
+ | Bare skill ids (`partitioning-strategy`) | collide at book #2 | `<slug>-`prefix |
85
+ | Free-typed page citations | can cite the book for invented text | compute from `derived_from` KUs |
86
+ | Shipping without routing evals | the skill never triggers | evals + CP3.5 gate |
87
+ | Claiming Validated trust | it's machine-distilled, unreviewed | trust_tier 0/1 + promote path |
88
+
89
+ ## Self-check
90
+
91
+ - [ ] Skills = decision moments, slug-prefixed, approved at CP3?
92
+ - [ ] Every skill: `derived_from` KUs + computed citation + faithfulness-checked?
93
+ - [ ] Routing evals shipped; CP3.5 gate ≥80%/≤10% passed?
94
+ - [ ] trust_tier 0/1 with a documented promote path?
95
+ - [ ] Related-decisions links present (the book's decision web)?
96
+
97
+ ## Examples
98
+
99
+ - «Distill DDIA» → CP3 proposes 8 skills → after approval, `ddia-replication-topology-choice`
100
+ (Protocol = the book's leader/leaderless/multi-leader criteria, cites gл.5 pp.169–214).
101
+ - «Этот скилл не срабатывает на моём запросе» → check its `evals/routing.yaml`; CP3.5 gate caught
102
+ weak activation before ship.
@@ -0,0 +1,101 @@
1
+ ---
2
+ name: digitize-book
3
+ description: >
4
+ Orchestrator: turn an engineering book/textbook/monograph (PDF, 100–1000+ pages) into an
5
+ INSTALLABLE skill pack whose skills make AI coders (Claude Code, Codex, …) work within the
6
+ book's methodologies at real decision moments — plus a searchable Knowledge Base behind them.
7
+ Drives the 5-stage pipeline (ingest → extract → distill → pack → index) through 5 human
8
+ checkpoints, resumable across sessions. Triggers on: "оцифруй книгу", "digitize this book",
9
+ "сделай скилл-пак из книги", "turn this textbook into skills", "book to skill pack".
10
+ trust_tier: 1
11
+ trust_tier_label: "Structured"
12
+ trust_tier_path: "Run /bto-test to promote to Tier 2"
13
+ ---
14
+
15
+ # digitize-book — book → installable methodology pack
16
+
17
+ Take a book and produce `skills-book-<slug>`: a pack that, once installed via `dz init`, makes the
18
+ agent **apply the book's methodologies** when it hits the decisions the book is about (e.g. DDIA →
19
+ `ddia-replication-topology-choice`, `ddia-partitioning-strategy`). Design & rationale:
20
+ [ADR-001 v2](https://github.com/djd1m/dz-harness-hub/blob/main/features/book-knowledge-digitizer/03_adr/001-book-to-skillpack-pipeline.md).
21
+
22
+ > **This is an orchestrator.** It sequences five stage-skills and stops at five checkpoints for
23
+ > your judgment. It never runs the whole book unattended — book digitization is a steered
24
+ > operation, and the panel-confirmed cost is **~2.5–4M agent-tokens per 640-page book**.
25
+
26
+ ## When to use / NOT
27
+
28
+ - **Use** for a legally-obtained engineering book you want your agents to *work by*.
29
+ - **NOT** for: summarizing a book (that's not a pack), a scanned book with no text layer *and* no
30
+ OCR available, or republishing the book's text (this produces paraphrased know-how, not the book).
31
+
32
+ ## Prerequisites
33
+
34
+ - `poppler-utils` (`pdftotext`, `pdfinfo`, `pdfimages`) for ingest.
35
+ - For the KB layers: `dz setup --memory agentdb` (installs `agentdb` + `better-sqlite3`).
36
+ - The five stage skills of this pack installed (they auto-install together):
37
+ `book-ingest`, `book-knowledge-extract`, `book-skill-distill`, `book-pack-assemble`, `book-kb-index`.
38
+
39
+ ## Pipeline & checkpoints
40
+
41
+ ```
42
+ book.pdf ─[book-ingest]─► corpus/ + manifest ─[book-knowledge-extract]─► Knowledge Units (+KU-verify)
43
+ ─[book-skill-distill]─► SKILL.md's ─[book-pack-assemble]─► skills-book-<slug> ─[book-kb-index]─► KB
44
+ CP1 scope CP2 sample KUs CP3 skill list CP3.5 routing CP4 pack CP5 distribute
45
+ ```
46
+
47
+ | CP | After | You decide |
48
+ |----|-------|-----------|
49
+ | **CP1** | ingest | scope (whole book vs chapters), confirm structure_type + skipped front-matter + **detected corpus language** (`--lang <code>` overrides) + **token-cost estimate** |
50
+ | **CP2** | 1–2 sample chapters extracted | KU quality — steer the extraction before the bulk map runs (bulk is hard-gated here) |
51
+ | **CP3** | distill triage | approve the candidate skill list (decision moments); flat vs gateway layout |
52
+ | **CP3.5** | routing evals | the routing gate report (≥80% correct activation / ≤10% sibling-steal) — fix or proceed |
53
+ | **CP4** | pack assembled | L0 report + install smoke (+ Copilot instruction-byte total if that target) |
54
+ | **CP5** | — | distribution: keep `private` (default) or record an explicit publish decision |
55
+
56
+ ## Protocol
57
+
58
+ 1. **Resume check** — if `<workspace>/manifest.json` exists, read `phase_state` + per-chunk
59
+ watermarks and continue from the first unfinished phase; else start at ingest. Never re-run a
60
+ completed phase (the manifest is the single source of truth).
61
+ 2. **Ingest** → invoke `book-ingest`. Relay its summary — including the **detected corpus language**
62
+ (Cyrillic-dominant → Russian, Latin → English); **CP1**. A `--lang <code>` flag overrides the
63
+ detection; thread the resolved language to every `book-knowledge-extract` call so all KUs are
64
+ emitted in that ONE language (no RU/EN mix).
65
+ 3. **Sample extract** → run `book-knowledge-extract` on 1–2 chapters only; show sample KUs; **CP2**.
66
+ 4. **Bulk extract + reduce + KU-verify** → the rest of the chapters (parallelizable per chunk),
67
+ dedup/merge at reduce, then the entailment verify pass (verified KUs only proceed).
68
+ 5. **Distill** → `book-skill-distill` proposes decision-moment skills; **CP3**; then **CP3.5** on
69
+ the routing evals.
70
+ 6. **Assemble** → `book-pack-assemble`; **CP4**.
71
+ 7. **Index** → `book-kb-index` loads KUs into the KB (lexical `books.sqlite` + agentdb vector).
72
+ 8. **Distribute** → **CP5**; then hand over: `dz init --select <the new skills>` and how to use.
73
+
74
+ ## `--trusted` mode
75
+
76
+ After you've approved 2–3 books, `--trusted` makes CP2 an async spot-check; **CP1, CP3, CP5 stay
77
+ mandatory** (scope, the skill list, and distribution are the irreducible judgment calls).
78
+
79
+ ## Anti-patterns
80
+
81
+ | Anti-pattern | Why it fails | Instead |
82
+ |--------------|--------------|---------|
83
+ | Run the whole book unattended | 2.5–4M tokens; distill triage is judgment | steer at CP1/CP3 |
84
+ | Skip CP2 and bulk-extract first | a bad extraction prompt wastes the whole map | sample first, tune, then bulk |
85
+ | Mirror chapters 1:1 as skills | breaks triggering + context | skills = decision moments (CP3) |
86
+ | Publish the generated pack by default | it's book-derived; distribution is the owner's | `private` default + CP5 |
87
+
88
+ ## Self-check
89
+
90
+ - [ ] Resumed from the manifest (no completed phase re-run)?
91
+ - [ ] CP1–CP5 honored (scope, samples, skill list, routing gate, distribution)?
92
+ - [ ] Only verified KUs fed distill; every skill claim traces to a KU?
93
+ - [ ] Pack is `private` unless a CP5 decision says otherwise?
94
+ - [ ] User got: install command + how the new skills trigger?
95
+
96
+ ## Examples
97
+
98
+ - «Оцифруй вот эту книгу по распределённым системам — хочу, чтобы агент проектировал по ней» →
99
+ full pipeline → `skills-book-ddia` with ~8 decision-moment skills.
100
+ - «Продолжи оцифровку, я вчера остановился на 5 главе» → resume from the manifest watermark.
101
+ - «Оцифруй только главы про репликацию и партиционирование» → CP1 scope = chapters 5–6.
package/package.json ADDED
@@ -0,0 +1,39 @@
1
+ {
2
+ "name": "@dzhechkov/skills-book-digitizer",
3
+ "version": "0.1.0",
4
+ "description": "Canonical book-digitizer skill pack — 6 skills that turn an engineering book/textbook/monograph into an installable skill pack whose skills make AI coders work within the book's methodologies: digitize-book (orchestrator), book-ingest, book-knowledge-extract, book-skill-distill, book-pack-assemble, book-kb-index.",
5
+ "license": "MIT",
6
+ "author": "dzhechko",
7
+ "keywords": [
8
+ "agent-skills",
9
+ "agentskills.io",
10
+ "skill-pack",
11
+ "claude-code",
12
+ "knowledge-extraction",
13
+ "book",
14
+ "pdf",
15
+ "methodology",
16
+ "agentdb"
17
+ ],
18
+ "files": [
19
+ "digitize-book",
20
+ "book-ingest",
21
+ "book-knowledge-extract",
22
+ "book-skill-distill",
23
+ "book-pack-assemble",
24
+ "book-kb-index",
25
+ "scripts",
26
+ "sources.json",
27
+ "README.md",
28
+ "LICENSE"
29
+ ],
30
+ "publishConfig": {
31
+ "access": "public"
32
+ },
33
+ "repository": {
34
+ "type": "git",
35
+ "url": "https://github.com/djd1m/dz-harness-hub.git",
36
+ "directory": "packages/@dzhechkov/skills-book-digitizer"
37
+ },
38
+ "homepage": "https://github.com/djd1m/dz-harness-hub/tree/main/packages/@dzhechkov/skills-book-digitizer#readme"
39
+ }
@@ -0,0 +1,249 @@
1
+ #!/usr/bin/env node
2
+ // book-ingest — deterministic PDF → chunked corpus + manifest (ADR-001 v2, book-knowledge-digitizer).
3
+ // NO LLM: poppler (pdfinfo/pdftotext/pdfimages) + heuristics only, so ingest is reproducible.
4
+ // node book-ingest.mjs <book.pdf> --out <dir> [--book <slug>] [--isbn <isbn>] [--max-chunk-tokens N]
5
+ // Emits <out>/corpus/NN-<slug>.md (with [p.N] page anchors) + <out>/manifest.json.
6
+ // Design: token-budget chunker (not structure-only), per-page quality matrix, figure/table
7
+ // inventory, front-matter listed-not-skipped, corpus_version + per-chunk source_hash/watermarks.
8
+
9
+ import { execFileSync, execSync } from 'node:child_process';
10
+ import { mkdirSync, writeFileSync, existsSync, readFileSync } from 'node:fs';
11
+ import { join, basename } from 'node:path';
12
+ import { createHash } from 'node:crypto';
13
+
14
+ // ---- args -------------------------------------------------------------------
15
+ const argv = process.argv.slice(2);
16
+ const pdf = argv[0];
17
+ const opt = (name, def) => { const i = argv.indexOf(`--${name}`); return i >= 0 ? argv[i + 1] : def; };
18
+ if (!pdf || !existsSync(pdf)) { console.error('usage: book-ingest <book.pdf> --out <dir> [--book <slug>] [--isbn <isbn>] [--max-chunk-tokens N]'); process.exit(2); }
19
+ const outDir = opt('out', join(process.cwd(), 'book-corpus'));
20
+ const MAX_CHUNK_TOKENS = parseInt(opt('max-chunk-tokens', '70000'), 10);
21
+ const CHUNK_BUDGET = MAX_CHUNK_TOKENS * 0.8; // QE P3: safety margin so estimator error can't overflow the extract agent
22
+ const slugify = (s) => s.toLowerCase().normalize('NFKD').replace(/[^\p{L}\p{N}]+/gu, '-').replace(/^-+|-+$/g, '').slice(0, 40);
23
+
24
+ // ---- poppler helpers --------------------------------------------------------
25
+ // `strict` rethrows on a non-zero exit (used for the PRIMARY extraction — QE P2: a poppler crash
26
+ // must NOT masquerade as an empty-but-successful book). Non-strict tolerates partial stdout.
27
+ const run = (cmd, args, strict) => {
28
+ try { return execFileSync(cmd, args, { encoding: 'utf-8', maxBuffer: 512 * 1024 * 1024 }); }
29
+ catch (e) { if (strict) throw new Error(`${cmd} failed (exit ${e.status ?? '?'}): ${String(e.stderr || e.message).slice(0, 200)}`); return e.stdout ? String(e.stdout) : ''; }
30
+ };
31
+ function popplerVersion() { // folded into corpus_version so cross-machine drift is attributable (QE P3)
32
+ // `pdftotext -v` prints to STDERR and may exit non-zero → merge streams to capture it reliably.
33
+ try { return (execSync('pdftotext -v 2>&1', { encoding: 'utf-8' }).match(/version\s+([\d.]+)/i)?.[1]) ?? 'unknown'; }
34
+ catch (e) { return (String(e.stdout || e.stderr || '').match(/version\s+([\d.]+)/i)?.[1]) ?? 'unknown'; }
35
+ }
36
+ function pdfInfo(file) {
37
+ const out = run('pdfinfo', [file]);
38
+ const g = (k) => { const m = new RegExp(`^${k}:\\s*(.+)$`, 'm').exec(out); return m ? m[1].trim() : ''; };
39
+ return { pages: parseInt(g('Pages') || '0', 10), title: g('Title'), encrypted: g('Encrypted') };
40
+ }
41
+ // Whole-book text with per-page form-feed (\f) separators → array indexed by page-1.
42
+ function pagesOf(file, layout) {
43
+ const args = layout ? ['-layout', file, '-'] : [file, '-'];
44
+ const txt = run('pdftotext', args, !layout); // primary (non-layout) run is strict
45
+ const parts = txt.split('\f');
46
+ if (parts.length && parts[parts.length - 1].trim() === '') parts.pop();
47
+ return parts;
48
+ }
49
+
50
+ // ---- per-page quality matrix (cheap, deterministic) -------------------------
51
+ const CYR_LAT = /[\p{Script=Cyrillic}\p{Script=Latin}]/u;
52
+ function pageQuality(text, layoutText, layoutAvailable) {
53
+ const chars = text.length;
54
+ const words = (text.match(/[\p{L}]{2,}/gu) || []).length;
55
+ const letters = (text.match(/[\p{L}]/gu) || []).length;
56
+ const mathish = (text.match(/[=≤≥±∑∏∫√∞·×÷→∈∉⊂∪∩αβγδλμσΣΩθπ∂∇]/gu) || []).length;
57
+ const mathDensity = chars > 0 ? mathish / chars : 0;
58
+ const coherence = chars > 0 ? letters / chars : 0;
59
+ // two-column signal (QE P2): only when the -layout counterpart is actually available and this
60
+ // page has real text; require a strong divergence. This alone is advisory — a run of consecutive
61
+ // flagged pages (computed after) is what the summary should trust.
62
+ const lines = text.split('\n').length, lLines = (layoutText || '').split('\n').length;
63
+ const columnSignal = (layoutAvailable && chars > 200 && lines > 0) ? Math.abs(lLines - lines) / lines : 0;
64
+ const flags = [];
65
+ if (chars < 40) flags.push('empty');
66
+ else if (coherence < 0.55 && chars > 200) flags.push('low_coherence');
67
+ if (mathDensity > 0.02) flags.push('math_dense');
68
+ if (columnSignal > 0.6) flags.push('two_column_maybe'); // renamed: raw per-page hint, confirmed by runs
69
+ return { chars, words, mathDensity: +mathDensity.toFixed(4), coherence: +coherence.toFixed(3), flags };
70
+ }
71
+
72
+ // ---- structure detection ----------------------------------------------------
73
+ // Chapter opener (QE P2 — broadened beyond "Глава/Chapter + digit"): arabic OR roman OR
74
+ // spelled-out numeral, plus «Раздел» and «§». The capture is normalized to an index afterwards.
75
+ const NUM_WORD = { один: 1, первая: 1, два: 2, вторая: 2, три: 3, третья: 3, четыре: 4, четвертая: 4, пять: 5, пятая: 5, one: 1, first: 1, two: 2, second: 2, three: 3, third: 3 };
76
+ const CHAPTER_RE = /^\s*(?:Глава|ГЛАВА|Chapter|CHAPTER|Раздел|РАЗДЕЛ|§)\s+([IVXLC]+|\d+|[A-Za-zА-Яа-я]+)\b/;
77
+ const HEADING_RE = /^\s*(?:\d+\.\d+\.?|§\s*\d)\s+\S/; // sub-section like "5.3 Репликация" or "§ 5.3"
78
+ const romanToInt = (r) => ({ I: 1, V: 5, X: 10, L: 50, C: 100 } && r.split('').reduce((a, c, i, s) => { const v = { I: 1, V: 5, X: 10, L: 50, C: 100 }[c] || 0; const n = { I: 1, V: 5, X: 10, L: 50, C: 100 }[s[i + 1]] || 0; return a + (v < n ? -v : v); }, 0));
79
+ function chapterIndex(token, fallback) {
80
+ if (/^\d+$/.test(token)) return parseInt(token, 10);
81
+ if (/^[IVXLC]+$/.test(token)) return romanToInt(token) || fallback;
82
+ return NUM_WORD[token.toLowerCase()] ?? fallback;
83
+ }
84
+ const FIG_TAB_RE = /(?:Рис(?:\.|унок)|Таблиц[аы]|Табл\.|Figure|Table)\s*\.?\s*\d+([.\-]\d+)?/gu;
85
+
86
+ function detectChapters(pages) {
87
+ const chapters = [];
88
+ const DOT_LEADER = /\.{4,}\s*\d+\s*$/; // "…title.......... 53" — a TOC/contents line
89
+ pages.forEach((text, i) => {
90
+ const lines = text.split('\n');
91
+ // TOC/contents pages list many chapter refs with dot leaders — never a chapter START.
92
+ if (lines.filter((l) => DOT_LEADER.test(l)).length >= 3) return;
93
+ // A real chapter opener has "Глава N" in the first 3 non-empty lines.
94
+ const topNonEmpty = lines.filter((l) => l.trim()).slice(0, 3);
95
+ const hLine = topNonEmpty.find((l) => CHAPTER_RE.test(l));
96
+ if (!hLine) return;
97
+ const num = chapterIndex(CHAPTER_RE.exec(hLine)[1], chapters.length + 1);
98
+ // Descriptive name: first substantial line after the heading, not a dot-leader/bullet/number,
99
+ // trailing page number stripped ("… языки запросов 55" → "… языки запросов").
100
+ let name = '';
101
+ const after = lines.slice(lines.indexOf(hLine) + 1, lines.indexOf(hLine) + 8);
102
+ for (const l of after) {
103
+ if (DOT_LEADER.test(l)) continue;
104
+ const s = l.replace(/\s+\d+\s*$/, '').replace(/\s{2,}\d+$/, '').replace(/^[•·▪\-\s]+/, '').trim();
105
+ const isCaption = /^(?:Рис|Табл|Figure|Table|Листинг|Listing)\b/.test(s) || /^\d+\.\d/.test(s); // figure/subsection, not a title
106
+ if (s.length >= 6 && (s.match(/[\p{L}]{2,}/gu) || []).length >= 2 && !/^[•·\-\d.\s]+$/.test(s) && !CHAPTER_RE.test(s) && !isCaption) { name = s; break; }
107
+ }
108
+ // Sanity: a good title is a short-ish noun phrase. Reject long/garbled captures (extraction
109
+ // reading-order sometimes surfaces a body line first) → fall back to the bare "Глава N".
110
+ const good = name && name.length <= 70 && !/^\d/.test(name) && (name.match(/\s/g) || []).length <= 8;
111
+ chapters.push({ num, startPage: i + 1, title: good ? `Глава ${num}. ${name}` : `Глава ${num}` });
112
+ });
113
+ // de-dup consecutive same-number hits (running headers); keep first occurrence per number
114
+ const seen = new Set(); const uniq = [];
115
+ for (const c of chapters) { if (!seen.has(c.num)) { seen.add(c.num); uniq.push(c); } }
116
+ uniq.sort((a, b) => a.startPage - b.startPage);
117
+ for (let k = 0; k < uniq.length; k++) uniq[k].endPage = (k + 1 < uniq.length ? uniq[k + 1].startPage - 1 : pages.length);
118
+ return uniq;
119
+ }
120
+
121
+ // ---- token-budget chunker within a chapter ----------------------------------
122
+ // Cyrillic tokenizes ~2x denser than Latin (QE P3): compute chars/token from the actual script
123
+ // mix so a "70k-token" chunk isn't really 100k and overflowing the extract agent's context.
124
+ function charsPerToken(text) {
125
+ const cyr = (text.match(/\p{Script=Cyrillic}/gu) || []).length;
126
+ const lat = (text.match(/\p{Script=Latin}/gu) || []).length;
127
+ const tot = cyr + lat;
128
+ if (tot === 0) return 3.0;
129
+ const cyrShare = cyr / tot; // Cyrillic ≈ 1.9 chars/token, Latin ≈ 4.0
130
+ return 1.9 * cyrShare + 4.0 * (1 - cyrShare);
131
+ }
132
+ let CPT = 2.5; // set per-book after extraction
133
+ const estTokens = (s) => Math.ceil(s.length / CPT);
134
+ function chunkPages(pages, startPage, endPage) {
135
+ const chunks = []; let cur = []; let curTokens = 0;
136
+ for (let p = startPage; p <= endPage; p++) {
137
+ const t = estTokens(pages[p - 1] || '');
138
+ if (t > CHUNK_BUDGET) process.stderr.write(` ⚠ page ${p} alone ≈${t} tokens > budget ${Math.round(CHUNK_BUDGET)} — extract this page carefully\n`);
139
+ const startsHeading = HEADING_RE.test((pages[p - 1] || '').split('\n')[0] || '');
140
+ if (cur.length && (curTokens + t > CHUNK_BUDGET || (startsHeading && curTokens > CHUNK_BUDGET * 0.6))) {
141
+ chunks.push([cur[0], cur[cur.length - 1]]); cur = []; curTokens = 0;
142
+ }
143
+ cur.push(p); curTokens += t;
144
+ }
145
+ if (cur.length) chunks.push([cur[0], cur[cur.length - 1]]);
146
+ return chunks;
147
+ }
148
+
149
+ // ---- main -------------------------------------------------------------------
150
+ const info = pdfInfo(pdf);
151
+ if (!info.pages) { console.error('pdfinfo: 0 pages (encrypted? not a PDF?)'); process.exit(1); }
152
+ const bookSlug = opt('book', slugify(info.title || basename(pdf, '.pdf')));
153
+ const isbn = opt('isbn', '');
154
+ const pages = pagesOf(pdf, false);
155
+ const layoutPages = pagesOf(pdf, true);
156
+ // QE P2: only trust the two-column signal when the -layout run aligns page-for-page.
157
+ const layoutAligned = layoutPages.length === pages.length;
158
+ CPT = charsPerToken(pages.join(''));
159
+ const quality = pages.map((t, i) => pageQuality(t, layoutPages[i], layoutAligned));
160
+
161
+ // QE P2: fail LOUD instead of writing a green manifest for an empty/broken extraction.
162
+ const emptyShare = quality.filter((q) => q.flags.includes('empty')).length / Math.max(1, pages.length);
163
+ if (pages.length === 0) { console.error('ERROR: 0 pages of text extracted. Encrypted or no text layer? Run OCR (pdftoppm + tesseract) and re-ingest.'); process.exit(1); }
164
+ if (info.encrypted && info.encrypted !== 'no' && emptyShare > 0.5) { console.error(`ERROR: PDF is encrypted (${info.encrypted}) and ${Math.round(emptyShare * 100)}% pages have no text — decrypt or OCR first.`); process.exit(1); }
165
+ if (emptyShare > 0.9) { console.error(`ERROR: ${Math.round(emptyShare * 100)}% of pages have no extractable text — this looks like a scanned book. OCR (pdftoppm + tesseract) and re-ingest.`); process.exit(1); }
166
+
167
+ const chapters = detectChapters(pages);
168
+ const structureType = chapters.length >= 3 ? 'chaptered' : (chapters.length ? 'monolithic' : 'monolithic');
169
+ const firstContentPage = chapters.length ? chapters[0].startPage : 1;
170
+
171
+ // front matter = pages before the first chapter (LISTED, never silently dropped)
172
+ const frontMatter = [];
173
+ for (let p = 1; p < firstContentPage; p++) {
174
+ const q = quality[p - 1];
175
+ frontMatter.push({ page: p, reason: q.chars < 40 ? 'empty-text' : 'pre-first-chapter' });
176
+ }
177
+
178
+ // build chunks
179
+ mkdirSync(join(outDir, 'corpus'), { recursive: true });
180
+ const ranges = chapters.length
181
+ ? chapters.flatMap((c) => chunkPages(pages, c.startPage, c.endPage).map((r) => ({ r, chapter: c })))
182
+ : chunkPages(pages, firstContentPage, pages.length).map((r) => ({ r, chapter: { num: 1, title: bookSlug, startPage: firstContentPage, endPage: pages.length } }));
183
+
184
+ const manifestChunks = [];
185
+ ranges.forEach(({ r, chapter }, i) => {
186
+ const [a, b] = r;
187
+ let body = '';
188
+ let figTab = 0;
189
+ for (let p = a; p <= b; p++) {
190
+ const pt = pages[p - 1] || '';
191
+ body += `\n\n[p.${p}]\n${pt}`;
192
+ figTab += (pt.match(FIG_TAB_RE) || []).length;
193
+ }
194
+ const seq = String(i + 1).padStart(3, '0');
195
+ const chunkSlug = `${seq}-${slugify(chapter.title || `chapter-${chapter.num}`)}`;
196
+ const file = join('corpus', `${chunkSlug}.md`);
197
+ const header = `# ${chapter.title || `Chapter ${chapter.num}`} [pp.${a}-${b}]\n`;
198
+ const content = header + body;
199
+ writeFileSync(join(outDir, file), content);
200
+ const pageFlags = {};
201
+ for (let p = a; p <= b; p++) if (quality[p - 1].flags.length) pageFlags[p] = quality[p - 1].flags;
202
+ manifestChunks.push({
203
+ id: chunkSlug,
204
+ file,
205
+ parent_chapter: chapter.num,
206
+ chapter_title: chapter.title || null,
207
+ pages: [a, b],
208
+ token_estimate: estTokens(body),
209
+ figure_table_mentions: figTab,
210
+ page_flags: pageFlags,
211
+ source_hash: createHash('sha256').update(content).digest('hex').slice(0, 16),
212
+ watermark: { extracted: false, reduced: false, indexed: false },
213
+ });
214
+ });
215
+
216
+ const poppler = popplerVersion();
217
+ const corpusVersion = createHash('sha256')
218
+ .update(JSON.stringify(manifestChunks.map((c) => [c.pages, c.source_hash])) + `|maxchunk=${MAX_CHUNK_TOKENS}|poppler=${poppler}`)
219
+ .digest('hex').slice(0, 16);
220
+
221
+ const flagCounts = quality.flatMap((q) => q.flags).reduce((m, f) => ((m[f] = (m[f] || 0) + 1), m), {});
222
+ const manifest = {
223
+ book: { slug: bookSlug, isbn, title: info.title || basename(pdf), pages: info.pages, source_file: basename(pdf) },
224
+ structure_type: structureType,
225
+ structure_note: chapters.length ? null : 'no chapter markers detected — single monolithic chunk-set; confirm scope at CP1',
226
+ corpus_version: corpusVersion,
227
+ ingested_settings: { max_chunk_tokens: MAX_CHUNK_TOKENS, chunk_budget: Math.round(CHUNK_BUDGET), chars_per_token: +CPT.toFixed(2), poppler_version: poppler },
228
+ chapters: chapters.map((c) => ({ num: c.num, title: c.title, pages: [c.startPage, c.endPage] })),
229
+ front_matter: frontMatter, // listed for CP1 confirmation — never silently skipped
230
+ chunks: manifestChunks,
231
+ page_quality_summary: {
232
+ total_pages: pages.length,
233
+ pages_declared: info.pages, // QE P2: vs extracted, so a mismatch is visible at CP1
234
+ pages_extracted: pages.length,
235
+ layout_aligned: layoutAligned,
236
+ flag_counts: flagCounts, // empty/low_coherence/math_dense/two_column_maybe tallies
237
+ total_figure_table_mentions: manifestChunks.reduce((s, c) => s + c.figure_table_mentions, 0),
238
+ },
239
+ phase_state: { extracted: 0, reduced: false, distilled: false, packed: false },
240
+ };
241
+ writeFileSync(join(outDir, 'manifest.json'), JSON.stringify(manifest, null, 2));
242
+
243
+ // human summary to stdout (the skill relays this at CP1)
244
+ console.log(`book-ingest: "${manifest.book.title}"`);
245
+ console.log(` ${info.pages} pages · structure=${structureType} · ${chapters.length} chapters · ${manifestChunks.length} chunks · corpus_version=${corpusVersion}`);
246
+ console.log(` front matter: ${frontMatter.length} pages listed (not dropped)`);
247
+ console.log(` quality flags: ${Object.entries(flagCounts).map(([k, v]) => `${k}=${v}`).join(' ') || 'none'}`);
248
+ console.log(` figure/table mentions: ${manifest.page_quality_summary.total_figure_table_mentions}`);
249
+ console.log(` → ${join(outDir, 'manifest.json')}`);
package/sources.json ADDED
@@ -0,0 +1,21 @@
1
+ {
2
+ "upstream_note": "dz-original skill pack (ADR-001, features/book-knowledge-digitizer). Turns a book into an installable skill pack. Declares cross-pack dependencies so dz bundle's cross-skill-dep warning fires honestly.",
3
+ "cross_pack_dependencies": {
4
+ "@dzhechkov/skills-meta": [
5
+ "skill-crystallizer (book-skill-distill delegates scaffold + benchmark by reference)",
6
+ "knowledge-extractor (book-knowledge-extract reuses its quality-gate discipline by reference)"
7
+ ]
8
+ },
9
+ "runtime_dependencies": {
10
+ "system": ["poppler-utils (pdftotext, pdfinfo, pdfimages) for book-ingest; tesseract optional for scanned books"],
11
+ "harness-core": ["indexPatternsToAgentdb + putBookKnowledge/queryBookKnowledge (book-kb-index); skillPackBaseDirs discovery.skillScopes/skillDirs (generated packs)"]
12
+ },
13
+ "skills": {
14
+ "digitize-book": { "upstream_type": "original", "upstream_path": null, "description": "Orchestrator — book → installable pack through 5 checkpoints, resumable" },
15
+ "book-ingest": { "upstream_type": "original", "upstream_path": null, "description": "Deterministic PDF → chunked corpus + manifest (poppler + heuristics, no LLM)" },
16
+ "book-knowledge-extract": { "upstream_type": "original", "upstream_path": null, "description": "Corpus → verified Knowledge Units (map-reduce, KU-verify, fidelity regime)" },
17
+ "book-skill-distill": { "upstream_type": "original", "upstream_path": null, "description": "Verified KUs → decision-moment skills (routing evals, traceability)" },
18
+ "book-pack-assemble": { "upstream_type": "original", "upstream_path": null, "description": "Skills → installable pack (references/, private-default, collision gate)" },
19
+ "book-kb-index": { "upstream_type": "original", "upstream_path": null, "description": "KUs → books.sqlite (lexical) + agentdb (vector) KB, upsert not append" }
20
+ }
21
+ }