@dzhechkov/skills-book-digitizer 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +64 -0
- package/book-ingest/SKILL.md +110 -0
- package/book-kb-index/SKILL.md +87 -0
- package/book-knowledge-extract/SKILL.md +117 -0
- package/book-pack-assemble/SKILL.md +92 -0
- package/book-skill-distill/SKILL.md +102 -0
- package/digitize-book/SKILL.md +101 -0
- package/package.json +39 -0
- package/scripts/book-ingest.mjs +249 -0
- package/sources.json +21 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 dzhechko
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
# @dzhechkov/skills-book-digitizer
|
|
2
|
+
|
|
3
|
+
Turn an engineering **book / textbook / monograph** into an **installable skill pack** whose skills
|
|
4
|
+
make AI coders (Claude Code, Codex, …) *work within the book's methodologies* — plus a searchable
|
|
5
|
+
Knowledge Base behind them. dz-original; design & rationale in
|
|
6
|
+
[ADR-001 v2](https://github.com/djd1m/dz-harness-hub/blob/main/features/book-knowledge-digitizer/03_adr/001-book-to-skillpack-pipeline.md)
|
|
7
|
+
(passed a 4-lens adversarial design panel: 28 findings, all folded in).
|
|
8
|
+
|
|
9
|
+
## What it produces
|
|
10
|
+
|
|
11
|
+
`book.pdf` → `skills-book-<slug>`: a pack you `dz init --select …` so the agent applies the book's
|
|
12
|
+
methods at the decisions it's about — e.g. DDIA → `ddia-replication-topology-choice`,
|
|
13
|
+
`ddia-partitioning-strategy`, `ddia-consistency-model-selection`.
|
|
14
|
+
|
|
15
|
+
## The 6 skills (a checkpointed pipeline)
|
|
16
|
+
|
|
17
|
+
| Skill | Stage |
|
|
18
|
+
|-------|-------|
|
|
19
|
+
| `digitize-book` | **Orchestrator** — runs the pipeline through 5 human checkpoints, resumable across sessions |
|
|
20
|
+
| `book-ingest` | PDF → chunked, page-anchored corpus + manifest (deterministic: poppler + heuristics, no LLM) |
|
|
21
|
+
| `book-knowledge-extract` | corpus → **verified** Knowledge Units (map-reduce, dedup, KU-verify entailment, FACTS-verbatim/PROSE-paraphrase regime) |
|
|
22
|
+
| `book-skill-distill` | verified KUs → **decision-moment** skills (slug-prefixed, routing evals + activation gate, KU→skill traceability) |
|
|
23
|
+
| `book-pack-assemble` | skills → installable pack (`references/` deep-lookup shipped inside, **private-by-default** distribution, collision gate, Copilot policy) |
|
|
24
|
+
| `book-kb-index` | KUs → the memory layer: `.dz/memory/books.sqlite` (lexical) + shared agentdb (vector) |
|
|
25
|
+
|
|
26
|
+
## Install & use
|
|
27
|
+
|
|
28
|
+
```bash
|
|
29
|
+
dz init --target claude-code --select digitize-book
|
|
30
|
+
# then, in your agent: «оцифруй вот эту книгу — хочу, чтобы агент проектировал по её методикам»
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Prerequisites: `poppler-utils` (ingest) and, for the KB, `dz setup --memory agentdb`
|
|
34
|
+
(installs `agentdb` + `better-sqlite3`).
|
|
35
|
+
|
|
36
|
+
## Design guarantees — structural vs. protocol (stated honestly)
|
|
37
|
+
|
|
38
|
+
Two kinds of guarantee, and the difference matters:
|
|
39
|
+
|
|
40
|
+
- **Structural (mechanically enforced by code/packaging):** generated packs ship `private: true` +
|
|
41
|
+
restricted `publishConfig` and get **no automatic preset entry** — a book-derived pack cannot be
|
|
42
|
+
auto-published; distribution is your explicit CP5 decision. `files[]` whitelist + no-`origin`
|
|
43
|
+
`sources.json`. The ingest is deterministic. The book KB is a separate namespace (book KUs never
|
|
44
|
+
pollute `dz recommend`).
|
|
45
|
+
- **Protocol (agent-enforced, fail-open — NOT mechanically proven):** KU provenance is
|
|
46
|
+
*agent-verified* (an entailment pass re-reads the cited pages) but not cryptographically proven;
|
|
47
|
+
the FACTS-verbatim / PROSE-paraphrase fidelity regime and the KU→skill traceability are
|
|
48
|
+
discipline the distill/extract agents follow. Generated skills therefore ship at **`trust_tier`
|
|
49
|
+
0/1 ("machine-distilled, unreviewed")** with a documented promote path (human review against the
|
|
50
|
+
cited pages, or passing the routing evals). A scripted word-shingling verbatim-check is the
|
|
51
|
+
intended hard gate; it is **specified in the ADR but not yet shipped as a script** — until then
|
|
52
|
+
the paraphrase rule is agent-enforced, not automatic.
|
|
53
|
+
|
|
54
|
+
- **Honest cost** — a 640-page book ≈ 2.5–4M agent-tokens end-to-end; the orchestrator shows the
|
|
55
|
+
estimate at CP1 and hard-gates the bulk map behind your sample review at CP2.
|
|
56
|
+
|
|
57
|
+
## Provenance
|
|
58
|
+
|
|
59
|
+
dz-original (ADR-001). Delegates by reference to `skill-crystallizer` (skill scaffolding/benchmark)
|
|
60
|
+
and `knowledge-extractor` (quality-gate discipline) — see `sources.json`.
|
|
61
|
+
|
|
62
|
+
## License
|
|
63
|
+
|
|
64
|
+
MIT © dzhechko.
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: book-ingest
|
|
3
|
+
description: >
|
|
4
|
+
Stage 1 of the book digitizer: turn a book PDF into a chunked, page-anchored corpus + a
|
|
5
|
+
manifest, DETERMINISTICALLY (poppler + heuristics, no LLM). Token-budget chunker, per-page
|
|
6
|
+
quality matrix (empty/low-coherence/math-dense/two-column), figure/table inventory,
|
|
7
|
+
front-matter listed-not-skipped, deterministic corpus_version + per-chunk source_hash and
|
|
8
|
+
watermarks. Triggers on: "ingest this book", "book-ingest", "разбей книгу на главы",
|
|
9
|
+
"prepare book corpus". Usually invoked by `digitize-book`, not directly.
|
|
10
|
+
trust_tier: 1
|
|
11
|
+
trust_tier_label: "Structured"
|
|
12
|
+
trust_tier_path: "Run /bto-test to promote to Tier 2"
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# book-ingest — PDF → corpus + manifest (deterministic)
|
|
16
|
+
|
|
17
|
+
The reproducible foundation of the pipeline: everything downstream trusts the manifest. **No LLM
|
|
18
|
+
here** — a book must ingest the same way every time, or resume and idempotency break.
|
|
19
|
+
|
|
20
|
+
## When to use / NOT
|
|
21
|
+
|
|
22
|
+
- **Use** as the first stage of `digitize-book`, or standalone to inspect a book's structure.
|
|
23
|
+
- **NOT** for extracting knowledge (that's `book-knowledge-extract`) — ingest only structures text.
|
|
24
|
+
|
|
25
|
+
## Prerequisites
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
pdftotext -v && pdfinfo -v # poppler-utils; on Debian/Ubuntu: apt-get install poppler-utils
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
## Protocol
|
|
32
|
+
|
|
33
|
+
Run the bundled deterministic script (it lives in this skill's pack under `scripts/`):
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
node <pack>/scripts/book-ingest.mjs <book.pdf> --out <workspace> \
|
|
37
|
+
--book <slug> --isbn <isbn> [--max-chunk-tokens 70000]
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
What it produces:
|
|
41
|
+
|
|
42
|
+
- `<workspace>/corpus/NNN-<chunk-slug>.md` — each chunk's text with **`[p.N]` page anchors** before
|
|
43
|
+
every page (the anchors are what make provenance verifiable downstream).
|
|
44
|
+
- `<workspace>/manifest.json` — the contract for the whole run:
|
|
45
|
+
|
|
46
|
+
```jsonc
|
|
47
|
+
{
|
|
48
|
+
"book": { "slug", "isbn", "title", "pages", "source_file" },
|
|
49
|
+
"structure_type": "chaptered | monolithic", // "collection" (papers) is a planned enhancement
|
|
50
|
+
"corpus_version": "<hash of chunk page-ranges + settings>", // bumps on any re-ingest
|
|
51
|
+
"chapters": [{ "num", "title", "pages": [a, b] }],
|
|
52
|
+
"front_matter": [{ "page", "reason": "empty-text | pre-first-chapter" }], // LISTED, never dropped
|
|
53
|
+
"chunks": [{
|
|
54
|
+
"id", "file", "parent_chapter", "pages": [a, b], "token_estimate",
|
|
55
|
+
"figure_table_mentions", "page_flags": { "<page>": ["math_dense", "two_column"] },
|
|
56
|
+
"source_hash": "<sha of chunk text>",
|
|
57
|
+
"watermark": { "extracted": false, "reduced": false, "indexed": false }
|
|
58
|
+
}],
|
|
59
|
+
"page_quality_summary": { "flag_counts", "total_figure_table_mentions" },
|
|
60
|
+
"phase_state": { "extracted": 0, "reduced": false, "distilled": false, "packed": false }
|
|
61
|
+
}
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Key design points (all in the script, ADR-001 v2):
|
|
65
|
+
|
|
66
|
+
- **Token-budget chunker**, not chapter=unit: a chapter over `max-chunk-tokens` splits at
|
|
67
|
+
sub-headings → page windows, so every chunk is agent-sized. `parent_chapter` links split chunks.
|
|
68
|
+
- **Per-page quality matrix**: `empty` (cover/blank → OCR candidate), `low_coherence` (garbled),
|
|
69
|
+
`math_dense` (formula-heavy → the extract stage must quarantine, not reproduce, mangled formulas),
|
|
70
|
+
`two_column` (layout divergence). Flags are advisory signals surfaced at CP1 — nothing silent.
|
|
71
|
+
- **Figure/table inventory**: counts «Рис. N.M» / «Таблица N.M» / «Figure/Table» *mentions* (text
|
|
72
|
+
regex) per chunk, so the extract coverage report can flag uncaptured visuals. (`pdfimages` is an
|
|
73
|
+
optional deeper check, not required by the current script.)
|
|
74
|
+
- **Front matter is LISTED**, never silently skipped (the "no silent caps" rule).
|
|
75
|
+
- **Determinism**: `corpus_version` and per-chunk `source_hash` are content hashes — a re-ingest
|
|
76
|
+
with the same PDF + settings yields identical values (verified on a 640-page book).
|
|
77
|
+
|
|
78
|
+
Then relay the manifest summary to the user for **CP1** (structure_type, chapter count, chunk
|
|
79
|
+
count, front-matter pages, quality flags, figure/table total, and the token-cost estimate).
|
|
80
|
+
|
|
81
|
+
## OCR / bad-layout fallback (when quality flags are heavy)
|
|
82
|
+
|
|
83
|
+
- Many `empty` pages on a scanned book → run `tesseract` per page image (`pdftoppm` → PNG) and
|
|
84
|
+
re-ingest from the OCR text; mark those pages' KUs `ocr: true`.
|
|
85
|
+
- Many `math_dense` pages → those chunks carry the flag; the extract stage MUST quarantine
|
|
86
|
+
formulas from mangled text (cite «формула на с. X, см. источник») rather than reproduce them.
|
|
87
|
+
Better path (enhancement): render flagged pages via `pdftoppm` + a vision model, set
|
|
88
|
+
`formula_verified: true` only when transcribed from the page image.
|
|
89
|
+
|
|
90
|
+
## Anti-patterns
|
|
91
|
+
|
|
92
|
+
| Anti-pattern | Why it fails | Instead |
|
|
93
|
+
|--------------|--------------|---------|
|
|
94
|
+
| Hand-editing the corpus/manifest | breaks corpus_version → resume + idempotency | re-run ingest |
|
|
95
|
+
| Using an LLM to chunk | non-deterministic → duplicate/re-pointed KUs on resume | the script is deterministic |
|
|
96
|
+
| Trusting `math_dense` chunk text for formulas | pdftotext mangles math | quarantine + cite / vision fallback |
|
|
97
|
+
| Silently dropping front matter | violates no-silent-caps | it's listed for CP1 |
|
|
98
|
+
|
|
99
|
+
## Self-check
|
|
100
|
+
|
|
101
|
+
- [ ] `pdftotext`/`pdfinfo` present; PDF not encrypted?
|
|
102
|
+
- [ ] manifest chapter page-ranges look right (spot-check 2 chapters vs the book)?
|
|
103
|
+
- [ ] corpus_version stable on a re-run (determinism)?
|
|
104
|
+
- [ ] Heavy quality flags triaged (OCR / formula policy) before extraction?
|
|
105
|
+
|
|
106
|
+
## Examples
|
|
107
|
+
|
|
108
|
+
- «book-ingest ddia.pdf» → 640 p. → 12 chapters, ~19 chunks (oversized chapters auto-split), 24 front-matter pages listed.
|
|
109
|
+
- «Почему пропущены страницы 1–24?» → they're `front_matter` in the manifest (pre-first-chapter),
|
|
110
|
+
listed not dropped.
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: book-kb-index
|
|
3
|
+
description: >
|
|
4
|
+
Stage 5 of the book digitizer (supporting): load verified Knowledge Units into the memory layer —
|
|
5
|
+
the lexical books.sqlite namespace (dz recall --books) AND the shared agentdb vector index
|
|
6
|
+
(agentdb_pattern_search), so book know-how is recallable lexically and semantically across all
|
|
7
|
+
digitized books. Upsert by (book, ku_id, corpus_version), never append-only. Triggers on:
|
|
8
|
+
"index the book KB", "book-kb-index", "load KUs into memory". Invoked by `digitize-book`.
|
|
9
|
+
trust_tier: 1
|
|
10
|
+
trust_tier_label: "Structured"
|
|
11
|
+
trust_tier_path: "Run /bto-test to promote to Tier 2"
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# book-kb-index — KUs → the searchable Knowledge Base
|
|
15
|
+
|
|
16
|
+
The layer BEHIND the distilled skills: when a skill says «см. источник, гл.5», or when you want to
|
|
17
|
+
recall book knowledge that never became a standalone skill, this is what answers. Two stores, one
|
|
18
|
+
purpose.
|
|
19
|
+
|
|
20
|
+
## When to use / NOT
|
|
21
|
+
|
|
22
|
+
- **Use** as stage 5 of `digitize-book`, to load verified KUs into the searchable memory layer.
|
|
23
|
+
- **NOT** to extract or verify KUs (that's `book-knowledge-extract`) — this stage only indexes them into the two stores.
|
|
24
|
+
|
|
25
|
+
## Prerequisites
|
|
26
|
+
|
|
27
|
+
- Verified KUs from `book-knowledge-extract`.
|
|
28
|
+
- `agentdb` + `better-sqlite3` in the project (`dz setup --memory agentdb`) for both stores.
|
|
29
|
+
|
|
30
|
+
## The two stores (why two)
|
|
31
|
+
|
|
32
|
+
| Store | Backend | Recall | Reached via |
|
|
33
|
+
|-------|---------|--------|-------------|
|
|
34
|
+
| **Lexical** | `.dz/memory/books.sqlite` (FTS5) | keyword | `dz recall --books` |
|
|
35
|
+
| **Semantic** | shared `.dz/agentdb.db` (vector) | similarity | `agentdb_pattern_search` |
|
|
36
|
+
|
|
37
|
+
The lexical store is a **separate namespace by construction** — `computePatternBoost` (the
|
|
38
|
+
`dz recommend` reward loop) never sees book KUs, and they never expire under default retention.
|
|
39
|
+
|
|
40
|
+
**Lexical matching (FTS5, know its limits):** `books.sqlite` now does **prefix** matching
|
|
41
|
+
(AND-of-terms) — morphology-tolerant, so `репликаци` matches `репликация`/`репликации`. But it has
|
|
42
|
+
**no stemming and no synonymy**, and it is single-language. So route **cross-lingual, fuzzy, or
|
|
43
|
+
semantic** queries through the VECTOR store (agentdb, multilingual embeddings) via
|
|
44
|
+
`agentdb_pattern_search` — lexical prefix-AND is for exact-ish keyword recall only.
|
|
45
|
+
|
|
46
|
+
## Protocol
|
|
47
|
+
|
|
48
|
+
1. **Idempotent upsert** — for each verified KU, write to BOTH stores keyed on
|
|
49
|
+
`(book, ku_id, corpus_version)`. A re-ingest bumps `corpus_version`; indexing then **evicts**
|
|
50
|
+
the book's rows from any stale corpus_version before inserting (upsert, never append) — so the
|
|
51
|
+
KB mirrors exactly the current ingest and re-runs don't duplicate.
|
|
52
|
+
- Lexical: `putBookKnowledge(projectRoot, kus)` (harness-core) — FTS5 over name/problem/content.
|
|
53
|
+
- Semantic: `indexPatternsToAgentdb(projectRoot, rows)` (harness-core) — native better-sqlite3
|
|
54
|
+
insert into ReasoningBank's schema (agentdb only for embeddings; never its sql.js
|
|
55
|
+
`createDatabase`). This primitive is **INSERT-only**, so to stay idempotent the skill FIRST
|
|
56
|
+
deletes this book's rows from a stale corpus_version (a better-sqlite3
|
|
57
|
+
`DELETE FROM reasoning_patterns WHERE tags LIKE '%"'||<slug>||'"%' AND metadata NOT LIKE
|
|
58
|
+
'%'||<corpus_version>||'%'` — mirror the lexical store's eviction) BEFORE calling it. Map each
|
|
59
|
+
KU → `{ taskType: 'book-knowledge', text: \`${name}: ${problem}\`, score: <documented
|
|
60
|
+
confidence, NOT a fabricated 1.0>, tags: ['book', <slug>, type], metadata: { isbn, chapter,
|
|
61
|
+
pages, ku_id, corpus_version } }`. (The lexical `putBookKnowledge` upserts automatically.)
|
|
62
|
+
2. **Honest score** — `score` = the KU's extraction/verify confidence (documented mapping), never a
|
|
63
|
+
fake 1.0 (the pollution lesson from the session-hook telemetry).
|
|
64
|
+
3. **Mark** each KU's `watermark.indexed = true` in the manifest (resume skips it next run).
|
|
65
|
+
4. **Verify recall** — a lexical probe (`dz recall --books "<a KU name>"`) and, if the MCP server is
|
|
66
|
+
up, a semantic probe (`agentdb_pattern_search`) return the KU with its page provenance.
|
|
67
|
+
|
|
68
|
+
## Anti-patterns
|
|
69
|
+
|
|
70
|
+
| Anti-pattern | Why it fails | Instead |
|
|
71
|
+
|--------------|--------------|---------|
|
|
72
|
+
| Append-only indexing | re-ingest duplicates KUs | lexical auto-upserts; for the vector store, DELETE stale-corpus rows before insert |
|
|
73
|
+
| Book KUs in the pattern store | pollutes `dz recommend` boost | separate books.sqlite namespace |
|
|
74
|
+
| Fabricated score 1.0 | biases vector ranking | honest extraction confidence |
|
|
75
|
+
| Using agentdb's createDatabase to write | sql.js corrupts the shared DB | native better-sqlite3 insert |
|
|
76
|
+
|
|
77
|
+
## Self-check
|
|
78
|
+
|
|
79
|
+
- [ ] Both stores written; keyed on (book, ku_id, corpus_version)?
|
|
80
|
+
- [ ] Lexical upserted; vector stale-corpus rows explicitly deleted before insert (no dup vectors)?
|
|
81
|
+
- [ ] Scores honest; book KUs excluded from `computePatternBoost`?
|
|
82
|
+
- [ ] `watermark.indexed` set; recall probe returns KU + provenance?
|
|
83
|
+
|
|
84
|
+
## Examples
|
|
85
|
+
|
|
86
|
+
- «Index the DDIA KUs» → 180 KUs → lexical + vector; `dz recall --books "репликация"` returns them.
|
|
87
|
+
- «Найди в книге про кворумы» → `agentdb_pattern_search` surfaces the quorum KU with gл.5 pages.
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: book-knowledge-extract
|
|
3
|
+
description: >
|
|
4
|
+
Stage 2 of the book digitizer: map-reduce the ingested corpus into page-anchored Knowledge Units
|
|
5
|
+
(methodologies, decision-frameworks, formulas, heuristics, checklists, tradeoff-tables,
|
|
6
|
+
case-patterns, definitions), then dedup/merge and run a KU-VERIFY entailment pass so only
|
|
7
|
+
KUs actually supported by their cited pages proceed. Enforces the FACTS-verbatim /
|
|
8
|
+
PROSE-paraphrase fidelity regime. Triggers on: "extract knowledge from the corpus",
|
|
9
|
+
"book-knowledge-extract", "harvest KUs". Invoked by `digitize-book`.
|
|
10
|
+
trust_tier: 1
|
|
11
|
+
trust_tier_label: "Structured"
|
|
12
|
+
trust_tier_path: "Run /bto-test to promote to Tier 2"
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# book-knowledge-extract — corpus → verified Knowledge Units
|
|
16
|
+
|
|
17
|
+
Turn structured text into **actionable know-how**, page-anchored and verified. This is where the
|
|
18
|
+
book's expertise becomes machine-usable — and where the IP and honesty rules bite.
|
|
19
|
+
|
|
20
|
+
## When to use / NOT
|
|
21
|
+
|
|
22
|
+
- **Use** as stage 2 of `digitize-book`, after `book-ingest` produced a manifest + corpus.
|
|
23
|
+
- **NOT** to author skills (that's `book-skill-distill`) or to structure text (that's ingest) — this stage produces verified Knowledge Units only.
|
|
24
|
+
|
|
25
|
+
## Prerequisites
|
|
26
|
+
|
|
27
|
+
A manifest + corpus from `book-ingest`. Reuse `knowledge-extractor`'s quality-gate discipline by
|
|
28
|
+
reference (`Read` its SKILL.md) — do not restate it; ADD only the book-specific gates below.
|
|
29
|
+
|
|
30
|
+
## The Knowledge Unit schema (the shared contract for the whole pipeline)
|
|
31
|
+
|
|
32
|
+
```yaml
|
|
33
|
+
id: <book>-<chunk>-<seq> # assigned within ONE atomic chunk run (see idempotency)
|
|
34
|
+
type: methodology | decision-framework | formula | heuristic | checklist |
|
|
35
|
+
tradeoff-table | case-pattern | definition
|
|
36
|
+
name: "Выбор топологии репликации"
|
|
37
|
+
problem: when/why this knowledge applies (the decision or task it serves)
|
|
38
|
+
content: the paraphrased procedure / framework / restructured table / verbatim formula
|
|
39
|
+
applicability: preconditions, scale ranges, contexts where it holds
|
|
40
|
+
limits: tradeoffs, failure modes, when NOT to apply
|
|
41
|
+
sources: [{ isbn, title, chapter, pages: [169, 214], unit_title? }] # LIST (survives merges)
|
|
42
|
+
aliases: [] # ids merged into this KU at reduce
|
|
43
|
+
skill_worthiness: high | medium | low # HINT — reduce re-scores with global view
|
|
44
|
+
verified: true | false | partial # set by the KU-verify pass; distill needs true
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## Protocol
|
|
48
|
+
|
|
49
|
+
### Map — one agent per chunk (parallelizable, resumable)
|
|
50
|
+
|
|
51
|
+
For each chunk whose `watermark.extracted == false`: read `corpus/<chunk>.md`, emit KUs in the
|
|
52
|
+
schema. **Idempotency**: `<seq>` is assigned only within one complete chunk run — write to a temp
|
|
53
|
+
file, then atomically rename + set `watermark.extracted = true` together; on retry, delete partial
|
|
54
|
+
output first. A changed `source_hash` invalidates that chunk's watermark AND all downstream
|
|
55
|
+
phase watermarks.
|
|
56
|
+
|
|
57
|
+
**Fidelity regime (the IP contract — non-negotiable):**
|
|
58
|
+
- **OUTPUT LANGUAGE (pin it — one language per book):** the book's language by default — detect from
|
|
59
|
+
the corpus (Cyrillic-dominant → Russian, Latin → English) — OR an explicit `--lang <code>` override
|
|
60
|
+
the orchestrator passes. ALL KU fields (`name`, `problem`, `content`, `applicability`, `limits`)
|
|
61
|
+
must be in that ONE language — **never mix**. Rationale: a consistent language is required for
|
|
62
|
+
lexical recall (`dz recall --books`) and for coherent distilled skills; a Russian book digitized
|
|
63
|
+
into English KUs breaks RU search and reads incoherently.
|
|
64
|
+
- **FACTS** (formulas, numeric thresholds, table DATA, algorithm names, code): preserved
|
|
65
|
+
**verbatim** with a page anchor. A `formula` KU's `content` must byte-match the source
|
|
66
|
+
expression — UNLESS the chunk flags the page `math_dense` (mangled), in which case the KU may
|
|
67
|
+
only POINT («формула на с. X, см. источник»), never reproduce a mangled formula.
|
|
68
|
+
- **PROSE** (explanations, narrative): **paraphrased** — no verbatim run ≥ 8–10 words outside a
|
|
69
|
+
cited quote (quotes ≤ ~25 words each, with an immediate page citation). **If the skills are
|
|
70
|
+
authored in a DIFFERENT language than the source** (e.g. RU book → EN skills), n-gram checking is
|
|
71
|
+
insufficient — run a sampled LLM-judge paraphrase audit and gate on it.
|
|
72
|
+
- **TABLES**: RESTRUCTURED, never transcribed — re-derive axes from the decision the KU serves,
|
|
73
|
+
reword cells, reorder/filter rows, cite «по мотивам табл. X, гл. Y, с. Z».
|
|
74
|
+
- Every KU accounts for the chunk's inventoried figures/tables — reconstruct (mermaid/markdown) or
|
|
75
|
+
emit `not_captured`. The coverage report counts capture rate, not just chunks.
|
|
76
|
+
|
|
77
|
+
### Reduce — single-threaded, after a barrier on all map watermarks
|
|
78
|
+
|
|
79
|
+
1. **Blocking** — embed every KU (via `book-kb-index`'s embedding path) and cluster by cosine
|
|
80
|
+
similarity within the same `type` (deterministic, near-free).
|
|
81
|
+
2. **Adjudication** — one small LLM call per above-threshold cluster: merge / link / keep-separate.
|
|
82
|
+
Merged KUs get a new id + `aliases` for the constituents (no dangling references); `sources`
|
|
83
|
+
accumulates. Cross-links between related KUs are recorded (distill uses them for "Related
|
|
84
|
+
decisions").
|
|
85
|
+
3. Re-score `skill_worthiness` with the coverage report in view (map scores were hints).
|
|
86
|
+
|
|
87
|
+
### KU-verify — the provenance gate (adversarial-verifier pattern)
|
|
88
|
+
|
|
89
|
+
A fresh checker agent re-opens each KU's cited page range in the corpus (the `[p.N]` anchors make
|
|
90
|
+
this a cheap lookup) and scores entailment: *is this KU actually supported by these pages?* Set
|
|
91
|
+
`verified: true|false|partial`. **Verify ALL KUs — do not sample** (lookup ≪ generation cost).
|
|
92
|
+
Only `verified` KUs proceed to distill. This is what stops hallucinated page anchors.
|
|
93
|
+
|
|
94
|
+
## Anti-patterns
|
|
95
|
+
|
|
96
|
+
| Anti-pattern | Why it fails | Instead |
|
|
97
|
+
|--------------|--------------|---------|
|
|
98
|
+
| Paraphrasing a formula/threshold | corrupts the highest-value content | FACTS verbatim + anchor |
|
|
99
|
+
| Transcribing a book table | IP + it mirrors the book's framing | restructure around the decision |
|
|
100
|
+
| Sampling KU-verify | hallucinated anchors slip through | verify all — it's cheap |
|
|
101
|
+
| Extracting raw summary instead of know-how | RAG, not behavior | KUs are decisions/procedures |
|
|
102
|
+
| Reproducing a formula from a `math_dense` page | pdftotext mangled it | point + cite, or vision-verify |
|
|
103
|
+
| Mixed-language KUs from a single book | breaks lexical recall — half the corpus is unsearchable | pin the language (book default or `--lang`) |
|
|
104
|
+
|
|
105
|
+
## Self-check
|
|
106
|
+
|
|
107
|
+
- [ ] Every KU page-anchored to a real `[p.N]` in the corpus?
|
|
108
|
+
- [ ] FACTS verbatim, PROSE paraphrased, TABLES restructured — regime respected?
|
|
109
|
+
- [ ] Figures/tables accounted for (reconstructed or `not_captured`)?
|
|
110
|
+
- [ ] Reduce merged duplicates with `aliases`; `skill_worthiness` re-scored globally?
|
|
111
|
+
- [ ] KU-verify run on ALL KUs; only `verified` proceed?
|
|
112
|
+
|
|
113
|
+
## Examples
|
|
114
|
+
|
|
115
|
+
- «Extract chapter 5» → KUs like `ddia-ch05-ku03` (decision-framework «Выбор топологии
|
|
116
|
+
репликации», pp.169–214, verified).
|
|
117
|
+
- «Почему этот KU не попал в дистилляцию?» → `verified: false` — the cited pages didn't support it.
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: book-pack-assemble
|
|
3
|
+
description: >
|
|
4
|
+
Stage 4 of the book digitizer: assemble the distilled skills into an installable pack
|
|
5
|
+
skills-book-<slug> (ADR-0001 library layout) — files[] whitelist, references/ deep-lookup
|
|
6
|
+
shipped INSIDE the pack, private-by-default distribution + CP5 decision, catalog-collision gate,
|
|
7
|
+
Copilot lean-body policy, no-origin sources.json (upstream_type: book), L0 benchmark + install
|
|
8
|
+
smoke. Triggers on: "assemble the book pack", "book-pack-assemble", "build skills-book-<slug>".
|
|
9
|
+
Invoked by `digitize-book`.
|
|
10
|
+
trust_tier: 1
|
|
11
|
+
trust_tier_label: "Structured"
|
|
12
|
+
trust_tier_path: "Run /bto-test to promote to Tier 2"
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# book-pack-assemble — skills → installable pack
|
|
16
|
+
|
|
17
|
+
Package the distilled skills so `dz init --select …` drops the book's methodology into any of the
|
|
18
|
+
6 targets — with the distribution and quality guardrails the panel demanded.
|
|
19
|
+
|
|
20
|
+
## When to use / NOT
|
|
21
|
+
|
|
22
|
+
- **Use** as stage 4 of `digitize-book`, once skills are distilled and routing-gated.
|
|
23
|
+
- **NOT** to author skills (that's `book-skill-distill`) or to index the KB (that's `book-kb-index`) — this stage builds the installable pack.
|
|
24
|
+
|
|
25
|
+
## Prerequisites
|
|
26
|
+
|
|
27
|
+
Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benchmark` for the gate.
|
|
28
|
+
|
|
29
|
+
## Protocol
|
|
30
|
+
|
|
31
|
+
1. **Scaffold `skills-book-<slug>/`** (ADR-0001 library layout — skill dirs at pack root):
|
|
32
|
+
```
|
|
33
|
+
skills-book-<slug>/
|
|
34
|
+
<slug>-<decision>/SKILL.md
|
|
35
|
+
<slug>-<decision>/references/*.md ← the deep-lookup, shipped IN the pack
|
|
36
|
+
<slug>-<decision>/evals/routing.yaml
|
|
37
|
+
package.json sources.json README.md LICENSE
|
|
38
|
+
```
|
|
39
|
+
2. **`references/` ships the deep-lookup inside the pack** — each skill's consumed KUs in full
|
|
40
|
+
(+ condensed chapter extracts, size permitting). So «см. источник» is a file read that works on
|
|
41
|
+
every target/machine, not a call to a KB the recipient doesn't have. `sources.json` declares
|
|
42
|
+
which lookup tiers exist per skill: `references/` = always, corpus = owner-local, agentdb =
|
|
43
|
+
owner-local — the skill degrades honestly.
|
|
44
|
+
3. **`files[]` whitelist** every skill dir + `references`/`evals` + `sources.json` + `README.md` +
|
|
45
|
+
`LICENSE` (forget one → npm silently drops it). Verify the tarball after.
|
|
46
|
+
4. **`package.json` distribution guardrails (structural, not policy):**
|
|
47
|
+
`"private": true` + `"publishConfig": { "access": "restricted" }`. **No automatic
|
|
48
|
+
`harness-presets` entry** (presets are the public install surface). Flipping to publishable
|
|
49
|
+
requires the recorded **CP5** decision (a `--allow-publish` step) written into `sources.json`
|
|
50
|
+
(`distribution`, `decided_by`, `date`). The gate FAILS if `private` is absent without a recorded
|
|
51
|
+
decision.
|
|
52
|
+
5. **`sources.json`** — documentation-only form, **NO `origin` block, ever** (the book is the
|
|
53
|
+
immutable upstream; `dz sync-upstream` intentionally skips book packs). Per skill: `upstream_type:
|
|
54
|
+
"book"`, isbn/title/edition, `derived_from` KU ids, verified-KU ratio, digitizer version,
|
|
55
|
+
extraction model, date. Declare the cross-pack deps (`skill-crystallizer`, `knowledge-extractor`)
|
|
56
|
+
so `dz bundle`'s cross-skill-dep warning fires honestly.
|
|
57
|
+
6. **Catalog-collision gate**: embed the new descriptions + all installed descriptions, flag
|
|
58
|
+
high-similarity pairs, require each resolved (differentiated description or explicit «NOT for X —
|
|
59
|
+
use Y» boundary clause). Also run the plain uniqueness check against `dz registry` (belt +
|
|
60
|
+
suspenders over slug-prefixing).
|
|
61
|
+
7. **Copilot policy**: keep SKILL.md bodies lean (trigger + protocol skeleton); big
|
|
62
|
+
criteria/tradeoff tables live in `references/*.md`. README carries a Copilot note («always-on
|
|
63
|
+
there — install a subset»). At CP4 with `target=copilot`, surface the aggregate instruction-file
|
|
64
|
+
byte count.
|
|
65
|
+
8. **Gate + smoke**: `dz benchmark <pack> --all` (honest L0 grade per skill — never fabricate
|
|
66
|
+
schemas/evals to game it) → `dz init --target claude-code --select <skills>` install smoke →
|
|
67
|
+
**CP4** report.
|
|
68
|
+
|
|
69
|
+
## Anti-patterns
|
|
70
|
+
|
|
71
|
+
| Anti-pattern | Why it fails | Instead |
|
|
72
|
+
|--------------|--------------|---------|
|
|
73
|
+
| Referencing an agentdb KB the recipient lacks | broken deep-lookup on their machine | ship `references/` in the pack |
|
|
74
|
+
| `private` absent / auto public publish | book-derived pack leaks | structural `private:true` + CP5 |
|
|
75
|
+
| Forgetting a skill dir in `files[]` | npm silently drops it | whitelist + verify tarball |
|
|
76
|
+
| `origin` block in sources.json | sync-upstream would chase a non-repo | documentation-only, no origin |
|
|
77
|
+
| Fat SKILL.md bodies on Copilot | floods every request | lean body + `references/` |
|
|
78
|
+
|
|
79
|
+
## Self-check
|
|
80
|
+
|
|
81
|
+
- [ ] `files[]` whitelists every dir; tarball verified?
|
|
82
|
+
- [ ] `references/` deep-lookup ships inside the pack; tiers declared in sources.json?
|
|
83
|
+
- [ ] `private:true` + restricted; no auto preset; CP5 recorded if publishing?
|
|
84
|
+
- [ ] Collision gate clean; slug-prefixed unique ids?
|
|
85
|
+
- [ ] Faithfulness gate run (each shipped skill's claims supported by its `derived_from` KUs)?
|
|
86
|
+
- [ ] L0 grades honest; install smoke passed; Copilot bytes shown if relevant?
|
|
87
|
+
|
|
88
|
+
## Examples
|
|
89
|
+
|
|
90
|
+
- «Assemble the DDIA pack» → `skills-book-ddia` (8 skills, private, references shipped, L0 B/A) →
|
|
91
|
+
`dz init --select ddia-partitioning-strategy`.
|
|
92
|
+
- «Хочу опубликовать этот пак» → CP5 records the decision, flips `private`, then a manual publish.
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: book-skill-distill
|
|
3
|
+
description: >
|
|
4
|
+
Stage 3 of the book digitizer (the heart): cluster verified Knowledge Units into DECISION-MOMENT
|
|
5
|
+
skills — the choices an AI coder actually faces — and author each as a house-format SKILL.md that
|
|
6
|
+
makes the agent apply the book's method. Slug-prefixed ids, routing evals + a CP3.5 activation
|
|
7
|
+
gate, KU→skill traceability, related-decision links, trust_tier 0/1. Triggers on:
|
|
8
|
+
"distill skills from the book", "book-skill-distill", "turn KUs into skills". Invoked by
|
|
9
|
+
`digitize-book`.
|
|
10
|
+
trust_tier: 1
|
|
11
|
+
trust_tier_label: "Structured"
|
|
12
|
+
trust_tier_path: "Run /bto-test to promote to Tier 2"
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# book-skill-distill — verified KUs → agent-behavior skills
|
|
16
|
+
|
|
17
|
+
The stage that decides *which* of the book's knowledge becomes agent behavior, and authors it so
|
|
18
|
+
it actually triggers and is trustworthy. Delegate scaffolding + benchmarking to `skill-crystallizer`
|
|
19
|
+
(`Read` its SKILL.md) — this skill adds the book-specific contract below.
|
|
20
|
+
|
|
21
|
+
## When to use / NOT
|
|
22
|
+
|
|
23
|
+
- **Use** as stage 3 of `digitize-book`, on the verified KUs from extract.
|
|
24
|
+
- **NOT** to extract KUs (that's `book-knowledge-extract`) or to package skills (that's `book-pack-assemble`) — this stage authors decision-moment SKILL.md's.
|
|
25
|
+
|
|
26
|
+
## Prerequisites
|
|
27
|
+
|
|
28
|
+
Verified KUs + cross-link graph from `book-knowledge-extract`. The installed catalog (`dz registry`)
|
|
29
|
+
for the collision + routing checks.
|
|
30
|
+
|
|
31
|
+
## Protocol
|
|
32
|
+
|
|
33
|
+
### 1. Cluster into decision moments (not chapter mirrors)
|
|
34
|
+
|
|
35
|
+
Group `skill_worthiness: high` (and strong `medium`) KUs by the **decision an AI coder hits**, not
|
|
36
|
+
by chapter. DDIA → `ddia-db-model-selection`, `ddia-storage-engine-tradeoffs`,
|
|
37
|
+
`ddia-encoding-evolution`, `ddia-replication-topology-choice`, `ddia-partitioning-strategy`,
|
|
38
|
+
`ddia-transactions-isolation-choice`, `ddia-consistency-model-selection`,
|
|
39
|
+
`ddia-batch-vs-stream-processing`. Present the candidate list at **CP3** for approval.
|
|
40
|
+
|
|
41
|
+
### 2. Naming & layout
|
|
42
|
+
|
|
43
|
+
- **Slug-prefix every skill id** with the book slug (`<slug>-<decision>`) — provenance in the id,
|
|
44
|
+
zero-lookup, collision-safe across books. The decision-moment phrasing lives in the description.
|
|
45
|
+
- **Flat vs gateway** (decide at CP3 by catalog pressure): ≤ ~10 skills → flat; more → a per-book
|
|
46
|
+
**gateway skill** (one description routing to protocols shipped as `references/`).
|
|
47
|
+
|
|
48
|
+
### 3. Author each skill (house L0 format)
|
|
49
|
+
|
|
50
|
+
- frontmatter: name (slug-prefixed), description with **decision-moment triggers** and an explicit
|
|
51
|
+
**boundary clause** («NOT for schema-change review — use `database-review`»), `trust_tier: 0`,
|
|
52
|
+
`trust_tier_label: "Machine-distilled from <book> (unreviewed)"`.
|
|
53
|
+
- **Protocol** = the book's decision criteria as concrete steps + a restructured criteria table.
|
|
54
|
+
- **Anti-patterns** = the book's warnings/failure-modes.
|
|
55
|
+
- **Related decisions** = sibling skills with the one-line coupling constraint (from the cross-link
|
|
56
|
+
graph): «If you chose leaderless replication here, `consistency-model-selection`: quorum math
|
|
57
|
+
changes.»
|
|
58
|
+
- **Источник** = citation COMPUTED from the consumed KUs' `sources` (chapter/pages) + 1–2 short
|
|
59
|
+
verbatim anchor quotes for a human spot-check + a pointer to KB deep-lookup.
|
|
60
|
+
- Self-check + Examples.
|
|
61
|
+
|
|
62
|
+
### 4. Traceability contract (anti-hallucination — non-negotiable)
|
|
63
|
+
|
|
64
|
+
- frontmatter carries `derived_from: [ku_ids]`; the «Источник» citation is **computed** from those
|
|
65
|
+
KUs, never free-typed by the distill agent.
|
|
66
|
+
- Content not traceable to a listed KU is removed OR explicitly labeled «адаптация для агента, не
|
|
67
|
+
из книги» — the book citation covers only KU-derived material.
|
|
68
|
+
- **Faithfulness check**: an independent judge verifies each Protocol step / Anti-pattern is
|
|
69
|
+
supported by the referenced KUs (KUs are page-anchored → checkable). Unsupported claims are
|
|
70
|
+
flagged and fixed before the pack gate.
|
|
71
|
+
|
|
72
|
+
### 5. Routing evals (CP3.5 — the value gate)
|
|
73
|
+
|
|
74
|
+
For each skill, author `evals/routing.yaml`: ~10 **positive** prompts (realistic RU+EN coding-session
|
|
75
|
+
phrasings for that decision) + **hard negatives** owned by sibling skills AND by named existing
|
|
76
|
+
arsenal skills. Run them against the FULL installed catalog; require **≥80% correct activation /
|
|
77
|
+
≤10% sibling-steal** before pack assembly. A skill that can't route is a skill that never fires.
|
|
78
|
+
|
|
79
|
+
## Anti-patterns
|
|
80
|
+
|
|
81
|
+
| Anti-pattern | Why it fails | Instead |
|
|
82
|
+
|--------------|--------------|---------|
|
|
83
|
+
| One skill per chapter | breaks triggering + context budget | one skill per decision moment |
|
|
84
|
+
| Bare skill ids (`partitioning-strategy`) | collide at book #2 | `<slug>-`prefix |
|
|
85
|
+
| Free-typed page citations | can cite the book for invented text | compute from `derived_from` KUs |
|
|
86
|
+
| Shipping without routing evals | the skill never triggers | evals + CP3.5 gate |
|
|
87
|
+
| Claiming Validated trust | it's machine-distilled, unreviewed | trust_tier 0/1 + promote path |
|
|
88
|
+
|
|
89
|
+
## Self-check
|
|
90
|
+
|
|
91
|
+
- [ ] Skills = decision moments, slug-prefixed, approved at CP3?
|
|
92
|
+
- [ ] Every skill: `derived_from` KUs + computed citation + faithfulness-checked?
|
|
93
|
+
- [ ] Routing evals shipped; CP3.5 gate ≥80%/≤10% passed?
|
|
94
|
+
- [ ] trust_tier 0/1 with a documented promote path?
|
|
95
|
+
- [ ] Related-decisions links present (the book's decision web)?
|
|
96
|
+
|
|
97
|
+
## Examples
|
|
98
|
+
|
|
99
|
+
- «Distill DDIA» → CP3 proposes 8 skills → after approval, `ddia-replication-topology-choice`
|
|
100
|
+
(Protocol = the book's leader/leaderless/multi-leader criteria, cites gл.5 pp.169–214).
|
|
101
|
+
- «Этот скилл не срабатывает на моём запросе» → check its `evals/routing.yaml`; CP3.5 gate caught
|
|
102
|
+
weak activation before ship.
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: digitize-book
|
|
3
|
+
description: >
|
|
4
|
+
Orchestrator: turn an engineering book/textbook/monograph (PDF, 100–1000+ pages) into an
|
|
5
|
+
INSTALLABLE skill pack whose skills make AI coders (Claude Code, Codex, …) work within the
|
|
6
|
+
book's methodologies at real decision moments — plus a searchable Knowledge Base behind them.
|
|
7
|
+
Drives the 5-stage pipeline (ingest → extract → distill → pack → index) through 5 human
|
|
8
|
+
checkpoints, resumable across sessions. Triggers on: "оцифруй книгу", "digitize this book",
|
|
9
|
+
"сделай скилл-пак из книги", "turn this textbook into skills", "book to skill pack".
|
|
10
|
+
trust_tier: 1
|
|
11
|
+
trust_tier_label: "Structured"
|
|
12
|
+
trust_tier_path: "Run /bto-test to promote to Tier 2"
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# digitize-book — book → installable methodology pack
|
|
16
|
+
|
|
17
|
+
Take a book and produce `skills-book-<slug>`: a pack that, once installed via `dz init`, makes the
|
|
18
|
+
agent **apply the book's methodologies** when it hits the decisions the book is about (e.g. DDIA →
|
|
19
|
+
`ddia-replication-topology-choice`, `ddia-partitioning-strategy`). Design & rationale:
|
|
20
|
+
[ADR-001 v2](https://github.com/djd1m/dz-harness-hub/blob/main/features/book-knowledge-digitizer/03_adr/001-book-to-skillpack-pipeline.md).
|
|
21
|
+
|
|
22
|
+
> **This is an orchestrator.** It sequences five stage-skills and stops at five checkpoints for
|
|
23
|
+
> your judgment. It never runs the whole book unattended — book digitization is a steered
|
|
24
|
+
> operation, and the panel-confirmed cost is **~2.5–4M agent-tokens per 640-page book**.
|
|
25
|
+
|
|
26
|
+
## When to use / NOT
|
|
27
|
+
|
|
28
|
+
- **Use** for a legally-obtained engineering book you want your agents to *work by*.
|
|
29
|
+
- **NOT** for: summarizing a book (that's not a pack), a scanned book with no text layer *and* no
|
|
30
|
+
OCR available, or republishing the book's text (this produces paraphrased know-how, not the book).
|
|
31
|
+
|
|
32
|
+
## Prerequisites
|
|
33
|
+
|
|
34
|
+
- `poppler-utils` (`pdftotext`, `pdfinfo`, `pdfimages`) for ingest.
|
|
35
|
+
- For the KB layers: `dz setup --memory agentdb` (installs `agentdb` + `better-sqlite3`).
|
|
36
|
+
- The five stage skills of this pack installed (they auto-install together):
|
|
37
|
+
`book-ingest`, `book-knowledge-extract`, `book-skill-distill`, `book-pack-assemble`, `book-kb-index`.
|
|
38
|
+
|
|
39
|
+
## Pipeline & checkpoints
|
|
40
|
+
|
|
41
|
+
```
|
|
42
|
+
book.pdf ─[book-ingest]─► corpus/ + manifest ─[book-knowledge-extract]─► Knowledge Units (+KU-verify)
|
|
43
|
+
─[book-skill-distill]─► SKILL.md's ─[book-pack-assemble]─► skills-book-<slug> ─[book-kb-index]─► KB
|
|
44
|
+
CP1 scope CP2 sample KUs CP3 skill list CP3.5 routing CP4 pack CP5 distribute
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
| CP | After | You decide |
|
|
48
|
+
|----|-------|-----------|
|
|
49
|
+
| **CP1** | ingest | scope (whole book vs chapters), confirm structure_type + skipped front-matter + **detected corpus language** (`--lang <code>` overrides) + **token-cost estimate** |
|
|
50
|
+
| **CP2** | 1–2 sample chapters extracted | KU quality — steer the extraction before the bulk map runs (bulk is hard-gated here) |
|
|
51
|
+
| **CP3** | distill triage | approve the candidate skill list (decision moments); flat vs gateway layout |
|
|
52
|
+
| **CP3.5** | routing evals | the routing gate report (≥80% correct activation / ≤10% sibling-steal) — fix or proceed |
|
|
53
|
+
| **CP4** | pack assembled | L0 report + install smoke (+ Copilot instruction-byte total if that target) |
|
|
54
|
+
| **CP5** | — | distribution: keep `private` (default) or record an explicit publish decision |
|
|
55
|
+
|
|
56
|
+
## Protocol
|
|
57
|
+
|
|
58
|
+
1. **Resume check** — if `<workspace>/manifest.json` exists, read `phase_state` + per-chunk
|
|
59
|
+
watermarks and continue from the first unfinished phase; else start at ingest. Never re-run a
|
|
60
|
+
completed phase (the manifest is the single source of truth).
|
|
61
|
+
2. **Ingest** → invoke `book-ingest`. Relay its summary — including the **detected corpus language**
|
|
62
|
+
(Cyrillic-dominant → Russian, Latin → English); **CP1**. A `--lang <code>` flag overrides the
|
|
63
|
+
detection; thread the resolved language to every `book-knowledge-extract` call so all KUs are
|
|
64
|
+
emitted in that ONE language (no RU/EN mix).
|
|
65
|
+
3. **Sample extract** → run `book-knowledge-extract` on 1–2 chapters only; show sample KUs; **CP2**.
|
|
66
|
+
4. **Bulk extract + reduce + KU-verify** → the rest of the chapters (parallelizable per chunk),
|
|
67
|
+
dedup/merge at reduce, then the entailment verify pass (verified KUs only proceed).
|
|
68
|
+
5. **Distill** → `book-skill-distill` proposes decision-moment skills; **CP3**; then **CP3.5** on
|
|
69
|
+
the routing evals.
|
|
70
|
+
6. **Assemble** → `book-pack-assemble`; **CP4**.
|
|
71
|
+
7. **Index** → `book-kb-index` loads KUs into the KB (lexical `books.sqlite` + agentdb vector).
|
|
72
|
+
8. **Distribute** → **CP5**; then hand over: `dz init --select <the new skills>` and how to use.
|
|
73
|
+
|
|
74
|
+
## `--trusted` mode
|
|
75
|
+
|
|
76
|
+
After you've approved 2–3 books, `--trusted` makes CP2 an async spot-check; **CP1, CP3, CP5 stay
|
|
77
|
+
mandatory** (scope, the skill list, and distribution are the irreducible judgment calls).
|
|
78
|
+
|
|
79
|
+
## Anti-patterns
|
|
80
|
+
|
|
81
|
+
| Anti-pattern | Why it fails | Instead |
|
|
82
|
+
|--------------|--------------|---------|
|
|
83
|
+
| Run the whole book unattended | 2.5–4M tokens; distill triage is judgment | steer at CP1/CP3 |
|
|
84
|
+
| Skip CP2 and bulk-extract first | a bad extraction prompt wastes the whole map | sample first, tune, then bulk |
|
|
85
|
+
| Mirror chapters 1:1 as skills | breaks triggering + context | skills = decision moments (CP3) |
|
|
86
|
+
| Publish the generated pack by default | it's book-derived; distribution is the owner's | `private` default + CP5 |
|
|
87
|
+
|
|
88
|
+
## Self-check
|
|
89
|
+
|
|
90
|
+
- [ ] Resumed from the manifest (no completed phase re-run)?
|
|
91
|
+
- [ ] CP1–CP5 honored (scope, samples, skill list, routing gate, distribution)?
|
|
92
|
+
- [ ] Only verified KUs fed distill; every skill claim traces to a KU?
|
|
93
|
+
- [ ] Pack is `private` unless a CP5 decision says otherwise?
|
|
94
|
+
- [ ] User got: install command + how the new skills trigger?
|
|
95
|
+
|
|
96
|
+
## Examples
|
|
97
|
+
|
|
98
|
+
- «Оцифруй вот эту книгу по распределённым системам — хочу, чтобы агент проектировал по ней» →
|
|
99
|
+
full pipeline → `skills-book-ddia` with ~8 decision-moment skills.
|
|
100
|
+
- «Продолжи оцифровку, я вчера остановился на 5 главе» → resume from the manifest watermark.
|
|
101
|
+
- «Оцифруй только главы про репликацию и партиционирование» → CP1 scope = chapters 5–6.
|
package/package.json
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@dzhechkov/skills-book-digitizer",
|
|
3
|
+
"version": "0.1.0",
|
|
4
|
+
"description": "Canonical book-digitizer skill pack — 6 skills that turn an engineering book/textbook/monograph into an installable skill pack whose skills make AI coders work within the book's methodologies: digitize-book (orchestrator), book-ingest, book-knowledge-extract, book-skill-distill, book-pack-assemble, book-kb-index.",
|
|
5
|
+
"license": "MIT",
|
|
6
|
+
"author": "dzhechko",
|
|
7
|
+
"keywords": [
|
|
8
|
+
"agent-skills",
|
|
9
|
+
"agentskills.io",
|
|
10
|
+
"skill-pack",
|
|
11
|
+
"claude-code",
|
|
12
|
+
"knowledge-extraction",
|
|
13
|
+
"book",
|
|
14
|
+
"pdf",
|
|
15
|
+
"methodology",
|
|
16
|
+
"agentdb"
|
|
17
|
+
],
|
|
18
|
+
"files": [
|
|
19
|
+
"digitize-book",
|
|
20
|
+
"book-ingest",
|
|
21
|
+
"book-knowledge-extract",
|
|
22
|
+
"book-skill-distill",
|
|
23
|
+
"book-pack-assemble",
|
|
24
|
+
"book-kb-index",
|
|
25
|
+
"scripts",
|
|
26
|
+
"sources.json",
|
|
27
|
+
"README.md",
|
|
28
|
+
"LICENSE"
|
|
29
|
+
],
|
|
30
|
+
"publishConfig": {
|
|
31
|
+
"access": "public"
|
|
32
|
+
},
|
|
33
|
+
"repository": {
|
|
34
|
+
"type": "git",
|
|
35
|
+
"url": "https://github.com/djd1m/dz-harness-hub.git",
|
|
36
|
+
"directory": "packages/@dzhechkov/skills-book-digitizer"
|
|
37
|
+
},
|
|
38
|
+
"homepage": "https://github.com/djd1m/dz-harness-hub/tree/main/packages/@dzhechkov/skills-book-digitizer#readme"
|
|
39
|
+
}
|
|
@@ -0,0 +1,249 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// book-ingest — deterministic PDF → chunked corpus + manifest (ADR-001 v2, book-knowledge-digitizer).
|
|
3
|
+
// NO LLM: poppler (pdfinfo/pdftotext/pdfimages) + heuristics only, so ingest is reproducible.
|
|
4
|
+
// node book-ingest.mjs <book.pdf> --out <dir> [--book <slug>] [--isbn <isbn>] [--max-chunk-tokens N]
|
|
5
|
+
// Emits <out>/corpus/NN-<slug>.md (with [p.N] page anchors) + <out>/manifest.json.
|
|
6
|
+
// Design: token-budget chunker (not structure-only), per-page quality matrix, figure/table
|
|
7
|
+
// inventory, front-matter listed-not-skipped, corpus_version + per-chunk source_hash/watermarks.
|
|
8
|
+
|
|
9
|
+
import { execFileSync, execSync } from 'node:child_process';
|
|
10
|
+
import { mkdirSync, writeFileSync, existsSync, readFileSync } from 'node:fs';
|
|
11
|
+
import { join, basename } from 'node:path';
|
|
12
|
+
import { createHash } from 'node:crypto';
|
|
13
|
+
|
|
14
|
+
// ---- args -------------------------------------------------------------------
|
|
15
|
+
const argv = process.argv.slice(2);
|
|
16
|
+
const pdf = argv[0];
|
|
17
|
+
const opt = (name, def) => { const i = argv.indexOf(`--${name}`); return i >= 0 ? argv[i + 1] : def; };
|
|
18
|
+
if (!pdf || !existsSync(pdf)) { console.error('usage: book-ingest <book.pdf> --out <dir> [--book <slug>] [--isbn <isbn>] [--max-chunk-tokens N]'); process.exit(2); }
|
|
19
|
+
const outDir = opt('out', join(process.cwd(), 'book-corpus'));
|
|
20
|
+
const MAX_CHUNK_TOKENS = parseInt(opt('max-chunk-tokens', '70000'), 10);
|
|
21
|
+
const CHUNK_BUDGET = MAX_CHUNK_TOKENS * 0.8; // QE P3: safety margin so estimator error can't overflow the extract agent
|
|
22
|
+
const slugify = (s) => s.toLowerCase().normalize('NFKD').replace(/[^\p{L}\p{N}]+/gu, '-').replace(/^-+|-+$/g, '').slice(0, 40);
|
|
23
|
+
|
|
24
|
+
// ---- poppler helpers --------------------------------------------------------
|
|
25
|
+
// `strict` rethrows on a non-zero exit (used for the PRIMARY extraction — QE P2: a poppler crash
|
|
26
|
+
// must NOT masquerade as an empty-but-successful book). Non-strict tolerates partial stdout.
|
|
27
|
+
const run = (cmd, args, strict) => {
|
|
28
|
+
try { return execFileSync(cmd, args, { encoding: 'utf-8', maxBuffer: 512 * 1024 * 1024 }); }
|
|
29
|
+
catch (e) { if (strict) throw new Error(`${cmd} failed (exit ${e.status ?? '?'}): ${String(e.stderr || e.message).slice(0, 200)}`); return e.stdout ? String(e.stdout) : ''; }
|
|
30
|
+
};
|
|
31
|
+
function popplerVersion() { // folded into corpus_version so cross-machine drift is attributable (QE P3)
|
|
32
|
+
// `pdftotext -v` prints to STDERR and may exit non-zero → merge streams to capture it reliably.
|
|
33
|
+
try { return (execSync('pdftotext -v 2>&1', { encoding: 'utf-8' }).match(/version\s+([\d.]+)/i)?.[1]) ?? 'unknown'; }
|
|
34
|
+
catch (e) { return (String(e.stdout || e.stderr || '').match(/version\s+([\d.]+)/i)?.[1]) ?? 'unknown'; }
|
|
35
|
+
}
|
|
36
|
+
function pdfInfo(file) {
|
|
37
|
+
const out = run('pdfinfo', [file]);
|
|
38
|
+
const g = (k) => { const m = new RegExp(`^${k}:\\s*(.+)$`, 'm').exec(out); return m ? m[1].trim() : ''; };
|
|
39
|
+
return { pages: parseInt(g('Pages') || '0', 10), title: g('Title'), encrypted: g('Encrypted') };
|
|
40
|
+
}
|
|
41
|
+
// Whole-book text with per-page form-feed (\f) separators → array indexed by page-1.
|
|
42
|
+
function pagesOf(file, layout) {
|
|
43
|
+
const args = layout ? ['-layout', file, '-'] : [file, '-'];
|
|
44
|
+
const txt = run('pdftotext', args, !layout); // primary (non-layout) run is strict
|
|
45
|
+
const parts = txt.split('\f');
|
|
46
|
+
if (parts.length && parts[parts.length - 1].trim() === '') parts.pop();
|
|
47
|
+
return parts;
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
// ---- per-page quality matrix (cheap, deterministic) -------------------------
|
|
51
|
+
const CYR_LAT = /[\p{Script=Cyrillic}\p{Script=Latin}]/u;
|
|
52
|
+
function pageQuality(text, layoutText, layoutAvailable) {
|
|
53
|
+
const chars = text.length;
|
|
54
|
+
const words = (text.match(/[\p{L}]{2,}/gu) || []).length;
|
|
55
|
+
const letters = (text.match(/[\p{L}]/gu) || []).length;
|
|
56
|
+
const mathish = (text.match(/[=≤≥±∑∏∫√∞·×÷→∈∉⊂∪∩αβγδλμσΣΩθπ∂∇]/gu) || []).length;
|
|
57
|
+
const mathDensity = chars > 0 ? mathish / chars : 0;
|
|
58
|
+
const coherence = chars > 0 ? letters / chars : 0;
|
|
59
|
+
// two-column signal (QE P2): only when the -layout counterpart is actually available and this
|
|
60
|
+
// page has real text; require a strong divergence. This alone is advisory — a run of consecutive
|
|
61
|
+
// flagged pages (computed after) is what the summary should trust.
|
|
62
|
+
const lines = text.split('\n').length, lLines = (layoutText || '').split('\n').length;
|
|
63
|
+
const columnSignal = (layoutAvailable && chars > 200 && lines > 0) ? Math.abs(lLines - lines) / lines : 0;
|
|
64
|
+
const flags = [];
|
|
65
|
+
if (chars < 40) flags.push('empty');
|
|
66
|
+
else if (coherence < 0.55 && chars > 200) flags.push('low_coherence');
|
|
67
|
+
if (mathDensity > 0.02) flags.push('math_dense');
|
|
68
|
+
if (columnSignal > 0.6) flags.push('two_column_maybe'); // renamed: raw per-page hint, confirmed by runs
|
|
69
|
+
return { chars, words, mathDensity: +mathDensity.toFixed(4), coherence: +coherence.toFixed(3), flags };
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
// ---- structure detection ----------------------------------------------------
|
|
73
|
+
// Chapter opener (QE P2 — broadened beyond "Глава/Chapter + digit"): arabic OR roman OR
|
|
74
|
+
// spelled-out numeral, plus «Раздел» and «§». The capture is normalized to an index afterwards.
|
|
75
|
+
const NUM_WORD = { один: 1, первая: 1, два: 2, вторая: 2, три: 3, третья: 3, четыре: 4, четвертая: 4, пять: 5, пятая: 5, one: 1, first: 1, two: 2, second: 2, three: 3, third: 3 };
|
|
76
|
+
const CHAPTER_RE = /^\s*(?:Глава|ГЛАВА|Chapter|CHAPTER|Раздел|РАЗДЕЛ|§)\s+([IVXLC]+|\d+|[A-Za-zА-Яа-я]+)\b/;
|
|
77
|
+
const HEADING_RE = /^\s*(?:\d+\.\d+\.?|§\s*\d)\s+\S/; // sub-section like "5.3 Репликация" or "§ 5.3"
|
|
78
|
+
const romanToInt = (r) => ({ I: 1, V: 5, X: 10, L: 50, C: 100 } && r.split('').reduce((a, c, i, s) => { const v = { I: 1, V: 5, X: 10, L: 50, C: 100 }[c] || 0; const n = { I: 1, V: 5, X: 10, L: 50, C: 100 }[s[i + 1]] || 0; return a + (v < n ? -v : v); }, 0));
|
|
79
|
+
function chapterIndex(token, fallback) {
|
|
80
|
+
if (/^\d+$/.test(token)) return parseInt(token, 10);
|
|
81
|
+
if (/^[IVXLC]+$/.test(token)) return romanToInt(token) || fallback;
|
|
82
|
+
return NUM_WORD[token.toLowerCase()] ?? fallback;
|
|
83
|
+
}
|
|
84
|
+
const FIG_TAB_RE = /(?:Рис(?:\.|унок)|Таблиц[аы]|Табл\.|Figure|Table)\s*\.?\s*\d+([.\-]\d+)?/gu;
|
|
85
|
+
|
|
86
|
+
function detectChapters(pages) {
|
|
87
|
+
const chapters = [];
|
|
88
|
+
const DOT_LEADER = /\.{4,}\s*\d+\s*$/; // "…title.......... 53" — a TOC/contents line
|
|
89
|
+
pages.forEach((text, i) => {
|
|
90
|
+
const lines = text.split('\n');
|
|
91
|
+
// TOC/contents pages list many chapter refs with dot leaders — never a chapter START.
|
|
92
|
+
if (lines.filter((l) => DOT_LEADER.test(l)).length >= 3) return;
|
|
93
|
+
// A real chapter opener has "Глава N" in the first 3 non-empty lines.
|
|
94
|
+
const topNonEmpty = lines.filter((l) => l.trim()).slice(0, 3);
|
|
95
|
+
const hLine = topNonEmpty.find((l) => CHAPTER_RE.test(l));
|
|
96
|
+
if (!hLine) return;
|
|
97
|
+
const num = chapterIndex(CHAPTER_RE.exec(hLine)[1], chapters.length + 1);
|
|
98
|
+
// Descriptive name: first substantial line after the heading, not a dot-leader/bullet/number,
|
|
99
|
+
// trailing page number stripped ("… языки запросов 55" → "… языки запросов").
|
|
100
|
+
let name = '';
|
|
101
|
+
const after = lines.slice(lines.indexOf(hLine) + 1, lines.indexOf(hLine) + 8);
|
|
102
|
+
for (const l of after) {
|
|
103
|
+
if (DOT_LEADER.test(l)) continue;
|
|
104
|
+
const s = l.replace(/\s+\d+\s*$/, '').replace(/\s{2,}\d+$/, '').replace(/^[•·▪\-\s]+/, '').trim();
|
|
105
|
+
const isCaption = /^(?:Рис|Табл|Figure|Table|Листинг|Listing)\b/.test(s) || /^\d+\.\d/.test(s); // figure/subsection, not a title
|
|
106
|
+
if (s.length >= 6 && (s.match(/[\p{L}]{2,}/gu) || []).length >= 2 && !/^[•·\-\d.\s]+$/.test(s) && !CHAPTER_RE.test(s) && !isCaption) { name = s; break; }
|
|
107
|
+
}
|
|
108
|
+
// Sanity: a good title is a short-ish noun phrase. Reject long/garbled captures (extraction
|
|
109
|
+
// reading-order sometimes surfaces a body line first) → fall back to the bare "Глава N".
|
|
110
|
+
const good = name && name.length <= 70 && !/^\d/.test(name) && (name.match(/\s/g) || []).length <= 8;
|
|
111
|
+
chapters.push({ num, startPage: i + 1, title: good ? `Глава ${num}. ${name}` : `Глава ${num}` });
|
|
112
|
+
});
|
|
113
|
+
// de-dup consecutive same-number hits (running headers); keep first occurrence per number
|
|
114
|
+
const seen = new Set(); const uniq = [];
|
|
115
|
+
for (const c of chapters) { if (!seen.has(c.num)) { seen.add(c.num); uniq.push(c); } }
|
|
116
|
+
uniq.sort((a, b) => a.startPage - b.startPage);
|
|
117
|
+
for (let k = 0; k < uniq.length; k++) uniq[k].endPage = (k + 1 < uniq.length ? uniq[k + 1].startPage - 1 : pages.length);
|
|
118
|
+
return uniq;
|
|
119
|
+
}
|
|
120
|
+
|
|
121
|
+
// ---- token-budget chunker within a chapter ----------------------------------
|
|
122
|
+
// Cyrillic tokenizes ~2x denser than Latin (QE P3): compute chars/token from the actual script
|
|
123
|
+
// mix so a "70k-token" chunk isn't really 100k and overflowing the extract agent's context.
|
|
124
|
+
function charsPerToken(text) {
|
|
125
|
+
const cyr = (text.match(/\p{Script=Cyrillic}/gu) || []).length;
|
|
126
|
+
const lat = (text.match(/\p{Script=Latin}/gu) || []).length;
|
|
127
|
+
const tot = cyr + lat;
|
|
128
|
+
if (tot === 0) return 3.0;
|
|
129
|
+
const cyrShare = cyr / tot; // Cyrillic ≈ 1.9 chars/token, Latin ≈ 4.0
|
|
130
|
+
return 1.9 * cyrShare + 4.0 * (1 - cyrShare);
|
|
131
|
+
}
|
|
132
|
+
let CPT = 2.5; // set per-book after extraction
|
|
133
|
+
const estTokens = (s) => Math.ceil(s.length / CPT);
|
|
134
|
+
function chunkPages(pages, startPage, endPage) {
|
|
135
|
+
const chunks = []; let cur = []; let curTokens = 0;
|
|
136
|
+
for (let p = startPage; p <= endPage; p++) {
|
|
137
|
+
const t = estTokens(pages[p - 1] || '');
|
|
138
|
+
if (t > CHUNK_BUDGET) process.stderr.write(` ⚠ page ${p} alone ≈${t} tokens > budget ${Math.round(CHUNK_BUDGET)} — extract this page carefully\n`);
|
|
139
|
+
const startsHeading = HEADING_RE.test((pages[p - 1] || '').split('\n')[0] || '');
|
|
140
|
+
if (cur.length && (curTokens + t > CHUNK_BUDGET || (startsHeading && curTokens > CHUNK_BUDGET * 0.6))) {
|
|
141
|
+
chunks.push([cur[0], cur[cur.length - 1]]); cur = []; curTokens = 0;
|
|
142
|
+
}
|
|
143
|
+
cur.push(p); curTokens += t;
|
|
144
|
+
}
|
|
145
|
+
if (cur.length) chunks.push([cur[0], cur[cur.length - 1]]);
|
|
146
|
+
return chunks;
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
// ---- main -------------------------------------------------------------------
|
|
150
|
+
const info = pdfInfo(pdf);
|
|
151
|
+
if (!info.pages) { console.error('pdfinfo: 0 pages (encrypted? not a PDF?)'); process.exit(1); }
|
|
152
|
+
const bookSlug = opt('book', slugify(info.title || basename(pdf, '.pdf')));
|
|
153
|
+
const isbn = opt('isbn', '');
|
|
154
|
+
const pages = pagesOf(pdf, false);
|
|
155
|
+
const layoutPages = pagesOf(pdf, true);
|
|
156
|
+
// QE P2: only trust the two-column signal when the -layout run aligns page-for-page.
|
|
157
|
+
const layoutAligned = layoutPages.length === pages.length;
|
|
158
|
+
CPT = charsPerToken(pages.join(''));
|
|
159
|
+
const quality = pages.map((t, i) => pageQuality(t, layoutPages[i], layoutAligned));
|
|
160
|
+
|
|
161
|
+
// QE P2: fail LOUD instead of writing a green manifest for an empty/broken extraction.
|
|
162
|
+
const emptyShare = quality.filter((q) => q.flags.includes('empty')).length / Math.max(1, pages.length);
|
|
163
|
+
if (pages.length === 0) { console.error('ERROR: 0 pages of text extracted. Encrypted or no text layer? Run OCR (pdftoppm + tesseract) and re-ingest.'); process.exit(1); }
|
|
164
|
+
if (info.encrypted && info.encrypted !== 'no' && emptyShare > 0.5) { console.error(`ERROR: PDF is encrypted (${info.encrypted}) and ${Math.round(emptyShare * 100)}% pages have no text — decrypt or OCR first.`); process.exit(1); }
|
|
165
|
+
if (emptyShare > 0.9) { console.error(`ERROR: ${Math.round(emptyShare * 100)}% of pages have no extractable text — this looks like a scanned book. OCR (pdftoppm + tesseract) and re-ingest.`); process.exit(1); }
|
|
166
|
+
|
|
167
|
+
const chapters = detectChapters(pages);
|
|
168
|
+
const structureType = chapters.length >= 3 ? 'chaptered' : (chapters.length ? 'monolithic' : 'monolithic');
|
|
169
|
+
const firstContentPage = chapters.length ? chapters[0].startPage : 1;
|
|
170
|
+
|
|
171
|
+
// front matter = pages before the first chapter (LISTED, never silently dropped)
|
|
172
|
+
const frontMatter = [];
|
|
173
|
+
for (let p = 1; p < firstContentPage; p++) {
|
|
174
|
+
const q = quality[p - 1];
|
|
175
|
+
frontMatter.push({ page: p, reason: q.chars < 40 ? 'empty-text' : 'pre-first-chapter' });
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
// build chunks
|
|
179
|
+
mkdirSync(join(outDir, 'corpus'), { recursive: true });
|
|
180
|
+
const ranges = chapters.length
|
|
181
|
+
? chapters.flatMap((c) => chunkPages(pages, c.startPage, c.endPage).map((r) => ({ r, chapter: c })))
|
|
182
|
+
: chunkPages(pages, firstContentPage, pages.length).map((r) => ({ r, chapter: { num: 1, title: bookSlug, startPage: firstContentPage, endPage: pages.length } }));
|
|
183
|
+
|
|
184
|
+
const manifestChunks = [];
|
|
185
|
+
ranges.forEach(({ r, chapter }, i) => {
|
|
186
|
+
const [a, b] = r;
|
|
187
|
+
let body = '';
|
|
188
|
+
let figTab = 0;
|
|
189
|
+
for (let p = a; p <= b; p++) {
|
|
190
|
+
const pt = pages[p - 1] || '';
|
|
191
|
+
body += `\n\n[p.${p}]\n${pt}`;
|
|
192
|
+
figTab += (pt.match(FIG_TAB_RE) || []).length;
|
|
193
|
+
}
|
|
194
|
+
const seq = String(i + 1).padStart(3, '0');
|
|
195
|
+
const chunkSlug = `${seq}-${slugify(chapter.title || `chapter-${chapter.num}`)}`;
|
|
196
|
+
const file = join('corpus', `${chunkSlug}.md`);
|
|
197
|
+
const header = `# ${chapter.title || `Chapter ${chapter.num}`} [pp.${a}-${b}]\n`;
|
|
198
|
+
const content = header + body;
|
|
199
|
+
writeFileSync(join(outDir, file), content);
|
|
200
|
+
const pageFlags = {};
|
|
201
|
+
for (let p = a; p <= b; p++) if (quality[p - 1].flags.length) pageFlags[p] = quality[p - 1].flags;
|
|
202
|
+
manifestChunks.push({
|
|
203
|
+
id: chunkSlug,
|
|
204
|
+
file,
|
|
205
|
+
parent_chapter: chapter.num,
|
|
206
|
+
chapter_title: chapter.title || null,
|
|
207
|
+
pages: [a, b],
|
|
208
|
+
token_estimate: estTokens(body),
|
|
209
|
+
figure_table_mentions: figTab,
|
|
210
|
+
page_flags: pageFlags,
|
|
211
|
+
source_hash: createHash('sha256').update(content).digest('hex').slice(0, 16),
|
|
212
|
+
watermark: { extracted: false, reduced: false, indexed: false },
|
|
213
|
+
});
|
|
214
|
+
});
|
|
215
|
+
|
|
216
|
+
const poppler = popplerVersion();
|
|
217
|
+
const corpusVersion = createHash('sha256')
|
|
218
|
+
.update(JSON.stringify(manifestChunks.map((c) => [c.pages, c.source_hash])) + `|maxchunk=${MAX_CHUNK_TOKENS}|poppler=${poppler}`)
|
|
219
|
+
.digest('hex').slice(0, 16);
|
|
220
|
+
|
|
221
|
+
const flagCounts = quality.flatMap((q) => q.flags).reduce((m, f) => ((m[f] = (m[f] || 0) + 1), m), {});
|
|
222
|
+
const manifest = {
|
|
223
|
+
book: { slug: bookSlug, isbn, title: info.title || basename(pdf), pages: info.pages, source_file: basename(pdf) },
|
|
224
|
+
structure_type: structureType,
|
|
225
|
+
structure_note: chapters.length ? null : 'no chapter markers detected — single monolithic chunk-set; confirm scope at CP1',
|
|
226
|
+
corpus_version: corpusVersion,
|
|
227
|
+
ingested_settings: { max_chunk_tokens: MAX_CHUNK_TOKENS, chunk_budget: Math.round(CHUNK_BUDGET), chars_per_token: +CPT.toFixed(2), poppler_version: poppler },
|
|
228
|
+
chapters: chapters.map((c) => ({ num: c.num, title: c.title, pages: [c.startPage, c.endPage] })),
|
|
229
|
+
front_matter: frontMatter, // listed for CP1 confirmation — never silently skipped
|
|
230
|
+
chunks: manifestChunks,
|
|
231
|
+
page_quality_summary: {
|
|
232
|
+
total_pages: pages.length,
|
|
233
|
+
pages_declared: info.pages, // QE P2: vs extracted, so a mismatch is visible at CP1
|
|
234
|
+
pages_extracted: pages.length,
|
|
235
|
+
layout_aligned: layoutAligned,
|
|
236
|
+
flag_counts: flagCounts, // empty/low_coherence/math_dense/two_column_maybe tallies
|
|
237
|
+
total_figure_table_mentions: manifestChunks.reduce((s, c) => s + c.figure_table_mentions, 0),
|
|
238
|
+
},
|
|
239
|
+
phase_state: { extracted: 0, reduced: false, distilled: false, packed: false },
|
|
240
|
+
};
|
|
241
|
+
writeFileSync(join(outDir, 'manifest.json'), JSON.stringify(manifest, null, 2));
|
|
242
|
+
|
|
243
|
+
// human summary to stdout (the skill relays this at CP1)
|
|
244
|
+
console.log(`book-ingest: "${manifest.book.title}"`);
|
|
245
|
+
console.log(` ${info.pages} pages · structure=${structureType} · ${chapters.length} chapters · ${manifestChunks.length} chunks · corpus_version=${corpusVersion}`);
|
|
246
|
+
console.log(` front matter: ${frontMatter.length} pages listed (not dropped)`);
|
|
247
|
+
console.log(` quality flags: ${Object.entries(flagCounts).map(([k, v]) => `${k}=${v}`).join(' ') || 'none'}`);
|
|
248
|
+
console.log(` figure/table mentions: ${manifest.page_quality_summary.total_figure_table_mentions}`);
|
|
249
|
+
console.log(` → ${join(outDir, 'manifest.json')}`);
|
package/sources.json
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
{
|
|
2
|
+
"upstream_note": "dz-original skill pack (ADR-001, features/book-knowledge-digitizer). Turns a book into an installable skill pack. Declares cross-pack dependencies so dz bundle's cross-skill-dep warning fires honestly.",
|
|
3
|
+
"cross_pack_dependencies": {
|
|
4
|
+
"@dzhechkov/skills-meta": [
|
|
5
|
+
"skill-crystallizer (book-skill-distill delegates scaffold + benchmark by reference)",
|
|
6
|
+
"knowledge-extractor (book-knowledge-extract reuses its quality-gate discipline by reference)"
|
|
7
|
+
]
|
|
8
|
+
},
|
|
9
|
+
"runtime_dependencies": {
|
|
10
|
+
"system": ["poppler-utils (pdftotext, pdfinfo, pdfimages) for book-ingest; tesseract optional for scanned books"],
|
|
11
|
+
"harness-core": ["indexPatternsToAgentdb + putBookKnowledge/queryBookKnowledge (book-kb-index); skillPackBaseDirs discovery.skillScopes/skillDirs (generated packs)"]
|
|
12
|
+
},
|
|
13
|
+
"skills": {
|
|
14
|
+
"digitize-book": { "upstream_type": "original", "upstream_path": null, "description": "Orchestrator — book → installable pack through 5 checkpoints, resumable" },
|
|
15
|
+
"book-ingest": { "upstream_type": "original", "upstream_path": null, "description": "Deterministic PDF → chunked corpus + manifest (poppler + heuristics, no LLM)" },
|
|
16
|
+
"book-knowledge-extract": { "upstream_type": "original", "upstream_path": null, "description": "Corpus → verified Knowledge Units (map-reduce, KU-verify, fidelity regime)" },
|
|
17
|
+
"book-skill-distill": { "upstream_type": "original", "upstream_path": null, "description": "Verified KUs → decision-moment skills (routing evals, traceability)" },
|
|
18
|
+
"book-pack-assemble": { "upstream_type": "original", "upstream_path": null, "description": "Skills → installable pack (references/, private-default, collision gate)" },
|
|
19
|
+
"book-kb-index": { "upstream_type": "original", "upstream_path": null, "description": "KUs → books.sqlite (lexical) + agentdb (vector) KB, upsert not append" }
|
|
20
|
+
}
|
|
21
|
+
}
|