@dzhechkov/skills-book-digitizer 0.1.1 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -41,15 +41,16 @@ Two kinds of guarantee, and the difference matters:
41
41
  restricted `publishConfig` and get **no automatic preset entry** — a book-derived pack cannot be
42
42
  auto-published; distribution is your explicit CP5 decision. `files[]` whitelist + no-`origin`
43
43
  `sources.json`. The ingest is deterministic. The book KB is a separate namespace (book KUs never
44
- pollute `dz recommend`).
44
+ pollute `dz recommend`). The **word-shingling verbatim-check is now a shipped deterministic gate**
45
+ (`scripts/shingling-check.mjs`) run at pack assembly: any uncited verbatim run ≥8 words fails the
46
+ pack (cited ≤25-word quotes with a page anchor are exempt) — the paraphrase rule is mechanically
47
+ enforced, no longer merely agent-enforced.
45
48
  - **Protocol (agent-enforced, fail-open — NOT mechanically proven):** KU provenance is
46
49
  *agent-verified* (an entailment pass re-reads the cited pages) but not cryptographically proven;
47
50
  the FACTS-verbatim / PROSE-paraphrase fidelity regime and the KU→skill traceability are
48
51
  discipline the distill/extract agents follow. Generated skills therefore ship at **`trust_tier`
49
52
  0/1 ("machine-distilled, unreviewed")** with a documented promote path (human review against the
50
- cited pages, or passing the routing evals). A scripted word-shingling verbatim-check is the
51
- intended hard gate; it is **specified in the ADR but not yet shipped as a script** — until then
52
- the paraphrase rule is agent-enforced, not automatic.
53
+ cited pages, or passing the routing evals).
53
54
 
54
55
  - **Honest cost** — a 640-page book ≈ 2.5–4M agent-tokens end-to-end; the orchestrator shows the
55
56
  estimate at CP1 and hard-gates the bulk map behind your sample review at CP2.
@@ -62,9 +62,12 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
62
62
  criteria/tradeoff tables live in `references/*.md`. README carries a Copilot note («always-on
63
63
  there — install a subset»). At CP4 with `target=copilot`, surface the aggregate instruction-file
64
64
  byte count.
65
- 8. **Gate + smoke**: `dz benchmark <pack> --all` (honest L0 grade per skill — never fabricate
66
- schemas/evals to game it) → `dz init --target claude-code --select <skills>` install smoke →
67
- **CP4** report.
65
+ 8. **Gate + smoke**: first the **shingling gate** —
66
+ `node <pack>/scripts/shingling-check.mjs --source <corpus> --output <pack>` (`--shingle 8`).
67
+ ANY uncited verbatim run ≥8 words FAILS the pack (IP violation) — fix the offending KU
68
+ (re-paraphrase) and re-run. Cited ≤25-word quotes with a page anchor are exempt. Only once it
69
+ passes: `dz benchmark <pack> --all` (honest L0 grade per skill — never fabricate schemas/evals to
70
+ game it) → `dz init --target claude-code --select <skills>` install smoke → **CP4** report.
68
71
 
69
72
  ## Anti-patterns
70
73
 
@@ -75,6 +78,7 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
75
78
  | Forgetting a skill dir in `files[]` | npm silently drops it | whitelist + verify tarball |
76
79
  | `origin` block in sources.json | sync-upstream would chase a non-repo | documentation-only, no origin |
77
80
  | Fat SKILL.md bodies on Copilot | floods every request | lean body + `references/` |
81
+ | Shipping a pack without the shingling gate | uncited verbatim runs = IP violation | run `shingling-check.mjs` before the L0 benchmark; re-paraphrase any hit |
78
82
 
79
83
  ## Self-check
80
84
 
@@ -83,6 +87,7 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
83
87
  - [ ] `private:true` + restricted; no auto preset; CP5 recorded if publishing?
84
88
  - [ ] Collision gate clean; slug-prefixed unique ids?
85
89
  - [ ] Faithfulness gate run (each shipped skill's claims supported by its `derived_from` KUs)?
90
+ - [ ] Shingling gate passed (no uncited verbatim run ≥8 words; cited ≤25-word quotes anchored)?
86
91
  - [ ] L0 grades honest; install smoke passed; Copilot bytes shown if relevant?
87
92
 
88
93
  ## Examples
@@ -50,7 +50,7 @@ book.pdf ─[book-ingest]─► corpus/ + manifest ─[book-knowledge-extract]
50
50
  | **CP2** | 1–2 sample chapters extracted | KU quality — steer the extraction before the bulk map runs (bulk is hard-gated here) |
51
51
  | **CP3** | distill triage | approve the candidate skill list (decision moments); flat vs gateway layout |
52
52
  | **CP3.5** | routing evals | the routing gate report (≥80% correct activation / ≤10% sibling-steal) — fix or proceed |
53
- | **CP4** | pack assembled | L0 report + install smoke (+ Copilot instruction-byte total if that target) |
53
+ | **CP4** | pack assembled | shingling-gate result (no uncited verbatim run ≥8 words) + L0 report + install smoke (+ Copilot instruction-byte total if that target) |
54
54
  | **CP5** | — | distribution: keep `private` (default) or record an explicit publish decision |
55
55
 
56
56
  ## Protocol
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@dzhechkov/skills-book-digitizer",
3
- "version": "0.1.1",
3
+ "version": "0.2.0",
4
4
  "description": "Canonical book-digitizer skill pack — 6 skills that turn an engineering book/textbook/monograph into an installable skill pack whose skills make AI coders work within the book's methodologies: digitize-book (orchestrator), book-ingest, book-knowledge-extract, book-skill-distill, book-pack-assemble, book-kb-index.",
5
5
  "license": "MIT",
6
6
  "author": "dzhechko",
@@ -0,0 +1,271 @@
1
+ #!/usr/bin/env node
2
+ // shingling-check — DETERMINISTIC word-shingling verbatim-overlap IP gate for the book-digitizer.
3
+ // NO deps, NO Date/random: node builtins only, so the gate is reproducible in CI and on any machine.
4
+ //
5
+ // node shingling-check.mjs --source <dir-or-file> --output <dir-or-file> [--shingle 8] [--max-quote 25] [--json report.json]
6
+ //
7
+ // WHY: the digitizer distills copyrighted books into PARAPHRASED KUs/skills. The IP rule is that
8
+ // prose must be paraphrased — no verbatim run of >= N words (default 8) copied from the source —
9
+ // EXCEPT short, explicitly-cited quotes. This mechanically enforces that discipline (previously it
10
+ // was only agent-enforced). A verbatim run of N words is detected as an N-word shingle that exists
11
+ // in BOTH the source shingle-set and the output; consecutive overlapping shingles collapse into one
12
+ // contiguous run for reporting. Cited short quotes («…» / "…" followed by a page citation) are
13
+ // removed from the OUTPUT before shingling, so a properly-cited <= max-quote quote MUST pass.
14
+ //
15
+ // Design: deterministic (no Date/random); Unicode-aware (RU + EN, keep \p{L}\p{N}); the cited-quote
16
+ // exemption is CONSERVATIVE — it only exempts a quoted run that is both short (<= max-quote words)
17
+ // AND immediately followed by a page citation, so a borderline copy is flagged rather than silently
18
+ // passed. Exit 0 iff zero violations remain after exemptions, else 1.
19
+
20
+ import { readFileSync, writeFileSync, statSync, readdirSync } from 'node:fs';
21
+ import { join, relative, extname } from 'node:path';
22
+
23
+ // ---- args -------------------------------------------------------------------
24
+ const argv = process.argv.slice(2);
25
+ const opt = (name, def) => { const i = argv.indexOf(`--${name}`); return i >= 0 ? argv[i + 1] : def; };
26
+ const has = (name) => argv.includes(`--${name}`);
27
+ const sourceArg = opt('source', null);
28
+ const outputArg = opt('output', null);
29
+ const W = Math.max(1, parseInt(opt('shingle', '8'), 10) || 8);
30
+ const MAX_QUOTE = Math.max(1, parseInt(opt('max-quote', '25'), 10) || 25);
31
+ const jsonOut = opt('json', null);
32
+
33
+ if (!sourceArg || !outputArg || has('help') || has('h')) {
34
+ console.error('usage: shingling-check --source <dir-or-file> --output <dir-or-file> [--shingle 8] [--max-quote 25] [--json report.json]');
35
+ process.exit(2);
36
+ }
37
+
38
+ // ---- file collection --------------------------------------------------------
39
+ // A dir is recursed for *.md (SKILL.md + references/*.md + corpus chunks); a file is taken as-is.
40
+ function collect(path) {
41
+ let st;
42
+ try { st = statSync(path); }
43
+ catch { console.error(`shingling-check: path not found: ${path}`); process.exit(2); }
44
+ if (st.isFile()) return [path];
45
+ const out = [];
46
+ const walk = (dir) => {
47
+ for (const name of readdirSync(dir, { withFileTypes: true }).sort((a, b) => a.name < b.name ? -1 : 1)) {
48
+ const p = join(dir, name.name);
49
+ if (name.isDirectory()) walk(p);
50
+ else if (name.isFile() && extname(name.name).toLowerCase() === '.md') out.push(p);
51
+ }
52
+ };
53
+ walk(path);
54
+ return out.sort();
55
+ }
56
+
57
+ // ---- normalization → tokens with original-offset mapping --------------------
58
+ // Strip markdown scaffolding + [p.N] anchors + YAML frontmatter, lowercase, and map every
59
+ // non-(letter|digit) run to a single space. We keep, per emitted token, the [start,end) offsets in
60
+ // the STRIPPED string so an offending run can be reconstructed as readable text for the report.
61
+ const CITE_NEAR = /(?:(?:с|p|стр|page|гл|pp|pg)\.?\s*\d+|\[[^\]]*\d+[^\]]*\])/i; // page-citation signal
62
+
63
+ function stripMarkdown(raw) {
64
+ let s = raw.replace(/\r\n?/g, '\n');
65
+ // YAML frontmatter (only if the very first line is a --- fence)
66
+ if (s.startsWith('---\n')) {
67
+ const end = s.indexOf('\n---', 4);
68
+ if (end >= 0) { const after = s.indexOf('\n', end + 1); s = s.slice(after >= 0 ? after + 1 : s.length); }
69
+ }
70
+ // fenced code blocks ```…``` (and ~~~) — replace with spaces of equal length to preserve offsets
71
+ s = s.replace(/```[\s\S]*?```/g, (m) => ' '.repeat(m.length));
72
+ s = s.replace(/~~~[\s\S]*?~~~/g, (m) => ' '.repeat(m.length));
73
+ // inline code `…`
74
+ s = s.replace(/`[^`\n]*`/g, (m) => ' '.repeat(m.length));
75
+ // markdown links [text](url) / images ![alt](url) → keep the visible text, blank the target
76
+ s = s.replace(/!?\[([^\]]*)\]\(([^)]*)\)/g, (m, txt) => ' ' + txt + ' '.repeat(m.length - txt.length - 1));
77
+ return s;
78
+ }
79
+
80
+ // Blank [p.N] / [с.5] style bracketed page anchors (never part of prose). Run this AFTER the
81
+ // cited-quote exemption on the OUTPUT — the brackets are the exemption's citation signal, so they
82
+ // must still be present when blankCitedQuotes inspects a quote's tail.
83
+ function blankAnchors(s) {
84
+ return s.replace(/\[[^\]\n]*\d[^\]\n]*\]/g, (m) => ' '.repeat(m.length));
85
+ }
86
+
87
+ // Tokenize a stripped string into { tokens:[lc words], starts:[], ends:[] } over that string.
88
+ function tokenize(stripped) {
89
+ const tokens = [], starts = [], ends = [];
90
+ const re = /[\p{L}\p{N}]+/gu;
91
+ let m;
92
+ while ((m = re.exec(stripped)) !== null) {
93
+ tokens.push(m[0].toLowerCase());
94
+ starts.push(m.index);
95
+ ends.push(m.index + m[0].length);
96
+ }
97
+ return { tokens, starts, ends, stripped };
98
+ }
99
+
100
+ // Reconstruct a human-readable snippet spanning output token range [i, j] (inclusive).
101
+ function reconstruct(tk, i, j) {
102
+ const raw = tk.stripped.slice(tk.starts[i], tk.ends[j]);
103
+ return raw.replace(/\s+/g, ' ').trim();
104
+ }
105
+
106
+ // ---- cited-quote exemption (OUTPUT only) ------------------------------------
107
+ // CONSERVATIVE: blank out any quoted run of <= MAX_QUOTE words whose closing quote is immediately
108
+ // (within ~40 chars) followed by a page citation. Done on the STRIPPED string before tokenizing so
109
+ // the exempted words never enter the output shingle set. Quote styles: «…», "…", "…"(curly), '…'.
110
+ const QUOTE_PAIRS = [['«', '»'], ['“', '”'], ['"', '"'], ['‘', '’']];
111
+ function blankCitedQuotes(stripped) {
112
+ let s = stripped;
113
+ for (const [open, close] of QUOTE_PAIRS) {
114
+ let out = '', idx = 0;
115
+ while (idx < s.length) {
116
+ const o = s.indexOf(open, idx);
117
+ if (o < 0) { out += s.slice(idx); break; }
118
+ // find matching close (same char for straight quotes → search after open)
119
+ const c = s.indexOf(close, o + 1);
120
+ if (c < 0) { out += s.slice(idx); break; }
121
+ const inner = s.slice(o + 1, c);
122
+ const words = inner.match(/[\p{L}\p{N}]+/gu) || [];
123
+ const tail = s.slice(c + 1, c + 1 + 40);
124
+ const cited = CITE_NEAR.test(tail);
125
+ out += s.slice(idx, o); // text before the quote is kept verbatim
126
+ if (words.length <= MAX_QUOTE && words.length >= 1 && cited) {
127
+ // exempt: blank the ENTIRE quote span (open..close) so its shingles never form
128
+ out += ' '.repeat(c - o + 1);
129
+ } else {
130
+ // not exempt: keep the quote content so a long/uncited copy is still shingled & flaggable
131
+ out += s.slice(o, c + 1);
132
+ }
133
+ idx = c + 1;
134
+ }
135
+ s = out;
136
+ }
137
+ return s;
138
+ }
139
+
140
+ // ---- shingling --------------------------------------------------------------
141
+ // A shingle = W consecutive tokens joined by a single space. We store the joined string itself in a
142
+ // Set (exact, collision-free — no hashing needed for correctness); source contributes only the set.
143
+ function shinglesOf(tokens) {
144
+ const out = [];
145
+ for (let i = 0; i + W <= tokens.length; i++) out.push(tokens.slice(i, i + W).join(' '));
146
+ return out;
147
+ }
148
+
149
+ // ---- table header extraction (light copy heuristic) -------------------------
150
+ // Collect the normalized header-cell SET of every markdown table. A table = a header row of pipe
151
+ // cells immediately followed by a separator row (|---|---|). Best-effort, cheap.
152
+ function tableHeaderSets(raw) {
153
+ const lines = raw.replace(/\r\n?/g, '\n').split('\n');
154
+ const sets = [];
155
+ for (let i = 0; i + 1 < lines.length; i++) {
156
+ const sep = lines[i + 1];
157
+ if (!/^\s*\|?\s*:?-{2,}.*\|/.test(sep) || !sep.includes('-')) continue;
158
+ if (!lines[i].includes('|')) continue;
159
+ const cells = lines[i].split('|').map((c) => c.trim()).filter(Boolean)
160
+ .map((c) => (c.match(/[\p{L}\p{N}]+/gu) || []).join(' ').toLowerCase()).filter(Boolean);
161
+ if (cells.length >= 2) sets.push(new Set(cells));
162
+ }
163
+ return sets;
164
+ }
165
+ const setEq = (a, b) => a.size === b.size && [...a].every((x) => b.has(x));
166
+
167
+ // ---- build SOURCE ----------------------------------------------------------
168
+ const sourceFiles = collect(sourceArg);
169
+ const sourceSet = new Set();
170
+ const sourceTableSets = [];
171
+ for (const f of sourceFiles) {
172
+ const raw = readFileSync(f, 'utf-8');
173
+ const stripped = blankAnchors(stripMarkdown(raw));
174
+ const { tokens } = tokenize(stripped);
175
+ for (const sh of shinglesOf(tokens)) sourceSet.add(sh);
176
+ for (const t of tableHeaderSets(raw)) sourceTableSets.push(t);
177
+ }
178
+
179
+ // ---- scan OUTPUT -----------------------------------------------------------
180
+ const outputFiles = collect(outputArg);
181
+ const violations = []; // { file, words, text, startTok, endTok }
182
+ const tableWarnings = []; // { file, header:[...] }
183
+ let outputShingleCount = 0;
184
+
185
+ for (const f of outputFiles) {
186
+ const raw = readFileSync(f, 'utf-8');
187
+ let stripped = stripMarkdown(raw); // anchors kept — needed as the citation signal
188
+ stripped = blankCitedQuotes(stripped); // exemption uses the still-present [с.N] anchors
189
+ stripped = blankAnchors(stripped); // now remove any remaining anchors before shingling
190
+ const tk = tokenize(stripped);
191
+ const shingles = shinglesOf(tk.tokens);
192
+ outputShingleCount += shingles.length;
193
+
194
+ // mark which output shingles overlap the source, then coalesce consecutive hits into runs.
195
+ const hit = shingles.map((sh) => sourceSet.has(sh));
196
+ let i = 0;
197
+ while (i < hit.length) {
198
+ if (!hit[i]) { i++; continue; }
199
+ let j = i;
200
+ while (j + 1 < hit.length && hit[j + 1]) j++;
201
+ // shingles [i..j] overlap → token span [i .. j + W - 1] is one contiguous verbatim run
202
+ const startTok = i, endTok = j + W - 1;
203
+ violations.push({
204
+ file: f,
205
+ words: endTok - startTok + 1,
206
+ text: reconstruct(tk, startTok, endTok),
207
+ startTok, endTok,
208
+ });
209
+ i = j + 1;
210
+ }
211
+
212
+ // table-copy heuristic
213
+ for (const oh of tableHeaderSets(raw)) {
214
+ if (oh.size >= 2 && sourceTableSets.some((sh) => setEq(oh, sh))) {
215
+ tableWarnings.push({ file: f, header: [...oh] });
216
+ }
217
+ }
218
+ }
219
+
220
+ // ---- report ----------------------------------------------------------------
221
+ const rel = (p) => { try { return relative(process.cwd(), p) || p; } catch { return p; } };
222
+ const perFile = {};
223
+ for (const v of violations) perFile[v.file] = (perFile[v.file] || 0) + 1;
224
+
225
+ const report = {
226
+ tool: 'shingling-check',
227
+ config: { shingle: W, maxQuote: MAX_QUOTE, source: sourceArg, output: outputArg },
228
+ sourceFiles: sourceFiles.length,
229
+ outputFiles: outputFiles.length,
230
+ sourceShingles: sourceSet.size,
231
+ outputShingles: outputShingleCount,
232
+ totalViolations: violations.length,
233
+ perFile: Object.fromEntries(Object.entries(perFile).map(([k, v]) => [rel(k), v])),
234
+ samples: violations.slice(0, 15).map((v) => ({ file: rel(v.file), words: v.words, text: v.text })),
235
+ tableWarnings: tableWarnings.map((t) => ({ file: rel(t.file), header: t.header })),
236
+ pass: violations.length === 0,
237
+ };
238
+
239
+ // human summary
240
+ const line = '─'.repeat(60);
241
+ console.log(line);
242
+ console.log(`shingling-check — verbatim-overlap IP gate (w=${W}, max-quote=${MAX_QUOTE})`);
243
+ console.log(line);
244
+ console.log(`source: ${sourceFiles.length} file(s), ${sourceSet.size} unique ${W}-word shingles`);
245
+ console.log(`output: ${outputFiles.length} file(s), ${outputShingleCount} shingles scanned`);
246
+ console.log(`verbatim runs (violations): ${violations.length}`);
247
+ if (violations.length) {
248
+ console.log('\nper-file:');
249
+ for (const [f, n] of Object.entries(report.perFile)) console.log(` ${n} ${f}`);
250
+ console.log(`\nsample runs (up to 15):`);
251
+ for (const s of report.samples) {
252
+ const snip = s.text.length > 120 ? s.text.slice(0, 117) + '…' : s.text;
253
+ console.log(` [${s.words}w] ${rel(s.file)}\n "${snip}"`);
254
+ }
255
+ }
256
+ if (tableWarnings.length) {
257
+ console.log(`\ntable-copy warnings (RESTRUCTURE, don't transcribe): ${tableWarnings.length}`);
258
+ for (const t of report.tableWarnings.slice(0, 10)) console.log(` ${t.file}: [${t.header.join(' | ')}]`);
259
+ }
260
+ console.log(line);
261
+ console.log(violations.length === 0
262
+ ? 'PASS — no uncited verbatim run >= ' + W + ' words.'
263
+ : `FAIL — ${violations.length} uncited verbatim run(s) >= ${W} words. Paraphrase or cite as a short quote.`);
264
+ console.log(line);
265
+
266
+ if (jsonOut) {
267
+ writeFileSync(jsonOut, JSON.stringify(report, null, 2));
268
+ console.log(`report → ${rel(jsonOut)}`);
269
+ }
270
+
271
+ process.exit(violations.length === 0 ? 0 : 1);