@dzhechkov/skills-book-digitizer 0.1.1 → 0.2.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -33,6 +33,83 @@ dz init --target claude-code --select digitize-book
33
33
  Prerequisites: `poppler-utils` (ingest) and, for the KB, `dz setup --memory agentdb`
34
34
  (installs `agentdb` + `better-sqlite3`).
35
35
 
36
+ ## Usage scenarios
37
+
38
+ The digitizer produces three durable artifacts from one book: (1) an installable **skill pack**, (2) a
39
+ per-project **two-store KB** (`book-kb-index` → `.dz/memory/books.sqlite` lexical + shared agentdb
40
+ vector), and (3) a promote path into a durable cross-project **Knowledge Brain** (`~/.dz/brain/`).
41
+ Full design + the RU version: [ADR-001 Knowledge Brain](https://github.com/djd1m/dz-harness-hub/blob/main/features/book-brain/ADR-001-knowledge-brain.md).
42
+
43
+ ### 1. Digitize a book → skills + KB
44
+
45
+ The core pipeline. Install the orchestrator, then hand it a PDF; it runs `book-ingest → extract →
46
+ distill → assemble → book-kb-index` through 5 human checkpoints.
47
+
48
+ ```bash
49
+ dz init --target claude-code --select digitize-book
50
+ # then, in your agent: «оцифруй вот эту книгу — хочу, чтобы агент проектировал по её методикам»
51
+ ```
52
+
53
+ Output: a `skills-book-<slug>` pack (decision-moment skills) **and** the two-store KB behind it. Live
54
+ run: DDIA → «Высоконагруженные приложения» digitized to **116 Knowledge Units**.
55
+
56
+ ### 2. Query one book's knowledge
57
+
58
+ Lexical (FTS5) recall over a single digitized book's KB, in the project that ingested it. `--book`
59
+ narrows the search to one book slug:
60
+
61
+ ```bash
62
+ dz recall --books --book <slug> "replication topology"
63
+ ```
64
+
65
+ Pull-based: you ask, you get KU hits back with source + page anchors. No hook, no injection.
66
+
67
+ ### 3. Build a cross-project brain from N books
68
+
69
+ Promote each project's digitized-book KUs into **one shared brain** (`~/.dz/brain/` — lexical
70
+ `books.sqlite` + vector `agentdb.db`), then query across every ingested book at once:
71
+
72
+ ```bash
73
+ dz brain add --source <slug> # promote this project's KUs into the brain (idempotent)
74
+ dz brain list # what's in the brain: slug, kind, KU count, corpus version
75
+ dz brain query "replication topology" # cross-source recall over ALL books
76
+ dz brain query "replication topology" --source <slug> # optional: narrow to one book
77
+ ```
78
+
79
+ One shared store, not per-book packages: you do **not** install anything to query the brain — it
80
+ accretes on `dz brain add`. The *right* book surfaces by **relevance** (KUs rank on one common scale,
81
+ each hit labelled with its source book + page); `--source` is a label filter, not a manual routing
82
+ pick. Cross-source is the default — multiple books scoring on a query is a feature (richer grounding),
83
+ not a bug.
84
+
85
+ ### 4. Grounded answering (the hook)
86
+
87
+ ```bash
88
+ dz brain init # opt-in: wire the grounding UserPromptSubmit hook (once)
89
+ dz brain ground "<prompt>" # what the hook runs: retrieve + emit citations (or nothing)
90
+ ```
91
+
92
+ `dz brain init` (opt-in) wires a single `UserPromptSubmit` hook that runs `dz brain ground` on each
93
+ prompt. On a *relevant* prompt (relevance-gated, so it stays quiet on unrelated turns) the hook
94
+ mechanically retrieves top-K brain KUs and injects them — with source + page citations — into that
95
+ turn's context, so the agent answers **from the book, not from training data**. Off by default; a
96
+ prompt with no lexical overlap (e.g. "what's the weather") injects nothing.
97
+
98
+ Honesty boundary: the injection is **mechanical** (the hook forces the grounding block into context and
99
+ the agent literally cannot start the turn without it); "answering from it" is **agent discipline** — the
100
+ hook can force the context but not the completion. The hook is **generic and wired once** (one hook over
101
+ the whole brain across all sources — *not* per book), and needs **no edits to `CLAUDE.md` or governance
102
+ shards** — the injection channel is the hook's stdout (`hookSpecificOutput.additionalContext`).
103
+
104
+ ### 5. Sharing a digitized book
105
+
106
+ The `skills-book-<slug>` **pack** (via GitHub / `npx`) carries the behavior — the decision-moment
107
+ skills and `references/` deep-lookup. The design (ADR §8.1, P2) also ships the book's **KB slice inside
108
+ the pack** under the same CP5 licence decision that gates the pack itself: copyrighted books stay
109
+ `private: true` (structural, mechanically enforced — no auto-publish, no preset entry), permissive
110
+ sources are publishable. Your whole personal **brain is never published** — it is a local accretion you
111
+ back up, not a distributable.
112
+
36
113
  ## Design guarantees — structural vs. protocol (stated honestly)
37
114
 
38
115
  Two kinds of guarantee, and the difference matters:
@@ -41,15 +118,16 @@ Two kinds of guarantee, and the difference matters:
41
118
  restricted `publishConfig` and get **no automatic preset entry** — a book-derived pack cannot be
42
119
  auto-published; distribution is your explicit CP5 decision. `files[]` whitelist + no-`origin`
43
120
  `sources.json`. The ingest is deterministic. The book KB is a separate namespace (book KUs never
44
- pollute `dz recommend`).
121
+ pollute `dz recommend`). The **word-shingling verbatim-check is now a shipped deterministic gate**
122
+ (`scripts/shingling-check.mjs`) run at pack assembly: any uncited verbatim run ≥8 words fails the
123
+ pack (cited ≤25-word quotes with a page anchor are exempt) — the paraphrase rule is mechanically
124
+ enforced, no longer merely agent-enforced.
45
125
  - **Protocol (agent-enforced, fail-open — NOT mechanically proven):** KU provenance is
46
126
  *agent-verified* (an entailment pass re-reads the cited pages) but not cryptographically proven;
47
127
  the FACTS-verbatim / PROSE-paraphrase fidelity regime and the KU→skill traceability are
48
128
  discipline the distill/extract agents follow. Generated skills therefore ship at **`trust_tier`
49
129
  0/1 ("machine-distilled, unreviewed")** with a documented promote path (human review against the
50
- cited pages, or passing the routing evals). A scripted word-shingling verbatim-check is the
51
- intended hard gate; it is **specified in the ADR but not yet shipped as a script** — until then
52
- the paraphrase rule is agent-enforced, not automatic.
130
+ cited pages, or passing the routing evals).
53
131
 
54
132
  - **Honest cost** — a 640-page book ≈ 2.5–4M agent-tokens end-to-end; the orchestrator shows the
55
133
  estimate at CP1 and hard-gates the bulk map behind your sample review at CP2.
@@ -62,9 +62,12 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
62
62
  criteria/tradeoff tables live in `references/*.md`. README carries a Copilot note («always-on
63
63
  there — install a subset»). At CP4 with `target=copilot`, surface the aggregate instruction-file
64
64
  byte count.
65
- 8. **Gate + smoke**: `dz benchmark <pack> --all` (honest L0 grade per skill — never fabricate
66
- schemas/evals to game it) → `dz init --target claude-code --select <skills>` install smoke →
67
- **CP4** report.
65
+ 8. **Gate + smoke**: first the **shingling gate** —
66
+ `node <pack>/scripts/shingling-check.mjs --source <corpus> --output <pack>` (`--shingle 8`).
67
+ ANY uncited verbatim run ≥8 words FAILS the pack (IP violation) — fix the offending KU
68
+ (re-paraphrase) and re-run. Cited ≤25-word quotes with a page anchor are exempt. Only once it
69
+ passes: `dz benchmark <pack> --all` (honest L0 grade per skill — never fabricate schemas/evals to
70
+ game it) → `dz init --target claude-code --select <skills>` install smoke → **CP4** report.
68
71
 
69
72
  ## Anti-patterns
70
73
 
@@ -75,6 +78,7 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
75
78
  | Forgetting a skill dir in `files[]` | npm silently drops it | whitelist + verify tarball |
76
79
  | `origin` block in sources.json | sync-upstream would chase a non-repo | documentation-only, no origin |
77
80
  | Fat SKILL.md bodies on Copilot | floods every request | lean body + `references/` |
81
+ | Shipping a pack without the shingling gate | uncited verbatim runs = IP violation | run `shingling-check.mjs` before the L0 benchmark; re-paraphrase any hit |
78
82
 
79
83
  ## Self-check
80
84
 
@@ -83,6 +87,7 @@ Distilled skills + their KUs + routing evals from `book-skill-distill`. `dz benc
83
87
  - [ ] `private:true` + restricted; no auto preset; CP5 recorded if publishing?
84
88
  - [ ] Collision gate clean; slug-prefixed unique ids?
85
89
  - [ ] Faithfulness gate run (each shipped skill's claims supported by its `derived_from` KUs)?
90
+ - [ ] Shingling gate passed (no uncited verbatim run ≥8 words; cited ≤25-word quotes anchored)?
86
91
  - [ ] L0 grades honest; install smoke passed; Copilot bytes shown if relevant?
87
92
 
88
93
  ## Examples
@@ -50,7 +50,7 @@ book.pdf ─[book-ingest]─► corpus/ + manifest ─[book-knowledge-extract]
50
50
  | **CP2** | 1–2 sample chapters extracted | KU quality — steer the extraction before the bulk map runs (bulk is hard-gated here) |
51
51
  | **CP3** | distill triage | approve the candidate skill list (decision moments); flat vs gateway layout |
52
52
  | **CP3.5** | routing evals | the routing gate report (≥80% correct activation / ≤10% sibling-steal) — fix or proceed |
53
- | **CP4** | pack assembled | L0 report + install smoke (+ Copilot instruction-byte total if that target) |
53
+ | **CP4** | pack assembled | shingling-gate result (no uncited verbatim run ≥8 words) + L0 report + install smoke (+ Copilot instruction-byte total if that target) |
54
54
  | **CP5** | — | distribution: keep `private` (default) or record an explicit publish decision |
55
55
 
56
56
  ## Protocol
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@dzhechkov/skills-book-digitizer",
3
- "version": "0.1.1",
3
+ "version": "0.2.2",
4
4
  "description": "Canonical book-digitizer skill pack — 6 skills that turn an engineering book/textbook/monograph into an installable skill pack whose skills make AI coders work within the book's methodologies: digitize-book (orchestrator), book-ingest, book-knowledge-extract, book-skill-distill, book-pack-assemble, book-kb-index.",
5
5
  "license": "MIT",
6
6
  "author": "dzhechko",
@@ -36,4 +36,4 @@
36
36
  "directory": "packages/@dzhechkov/skills-book-digitizer"
37
37
  },
38
38
  "homepage": "https://github.com/djd1m/dz-harness-hub/tree/main/packages/@dzhechkov/skills-book-digitizer#readme"
39
- }
39
+ }
File without changes
@@ -0,0 +1,271 @@
1
+ #!/usr/bin/env node
2
+ // shingling-check — DETERMINISTIC word-shingling verbatim-overlap IP gate for the book-digitizer.
3
+ // NO deps, NO Date/random: node builtins only, so the gate is reproducible in CI and on any machine.
4
+ //
5
+ // node shingling-check.mjs --source <dir-or-file> --output <dir-or-file> [--shingle 8] [--max-quote 25] [--json report.json]
6
+ //
7
+ // WHY: the digitizer distills copyrighted books into PARAPHRASED KUs/skills. The IP rule is that
8
+ // prose must be paraphrased — no verbatim run of >= N words (default 8) copied from the source —
9
+ // EXCEPT short, explicitly-cited quotes. This mechanically enforces that discipline (previously it
10
+ // was only agent-enforced). A verbatim run of N words is detected as an N-word shingle that exists
11
+ // in BOTH the source shingle-set and the output; consecutive overlapping shingles collapse into one
12
+ // contiguous run for reporting. Cited short quotes («…» / "…" followed by a page citation) are
13
+ // removed from the OUTPUT before shingling, so a properly-cited <= max-quote quote MUST pass.
14
+ //
15
+ // Design: deterministic (no Date/random); Unicode-aware (RU + EN, keep \p{L}\p{N}); the cited-quote
16
+ // exemption is CONSERVATIVE — it only exempts a quoted run that is both short (<= max-quote words)
17
+ // AND immediately followed by a page citation, so a borderline copy is flagged rather than silently
18
+ // passed. Exit 0 iff zero violations remain after exemptions, else 1.
19
+
20
+ import { readFileSync, writeFileSync, statSync, readdirSync } from 'node:fs';
21
+ import { join, relative, extname } from 'node:path';
22
+
23
+ // ---- args -------------------------------------------------------------------
24
+ const argv = process.argv.slice(2);
25
+ const opt = (name, def) => { const i = argv.indexOf(`--${name}`); return i >= 0 ? argv[i + 1] : def; };
26
+ const has = (name) => argv.includes(`--${name}`);
27
+ const sourceArg = opt('source', null);
28
+ const outputArg = opt('output', null);
29
+ const W = Math.max(1, parseInt(opt('shingle', '8'), 10) || 8);
30
+ const MAX_QUOTE = Math.max(1, parseInt(opt('max-quote', '25'), 10) || 25);
31
+ const jsonOut = opt('json', null);
32
+
33
+ if (!sourceArg || !outputArg || has('help') || has('h')) {
34
+ console.error('usage: shingling-check --source <dir-or-file> --output <dir-or-file> [--shingle 8] [--max-quote 25] [--json report.json]');
35
+ process.exit(2);
36
+ }
37
+
38
+ // ---- file collection --------------------------------------------------------
39
+ // A dir is recursed for *.md (SKILL.md + references/*.md + corpus chunks); a file is taken as-is.
40
+ function collect(path) {
41
+ let st;
42
+ try { st = statSync(path); }
43
+ catch { console.error(`shingling-check: path not found: ${path}`); process.exit(2); }
44
+ if (st.isFile()) return [path];
45
+ const out = [];
46
+ const walk = (dir) => {
47
+ for (const name of readdirSync(dir, { withFileTypes: true }).sort((a, b) => a.name < b.name ? -1 : 1)) {
48
+ const p = join(dir, name.name);
49
+ if (name.isDirectory()) walk(p);
50
+ else if (name.isFile() && extname(name.name).toLowerCase() === '.md') out.push(p);
51
+ }
52
+ };
53
+ walk(path);
54
+ return out.sort();
55
+ }
56
+
57
+ // ---- normalization → tokens with original-offset mapping --------------------
58
+ // Strip markdown scaffolding + [p.N] anchors + YAML frontmatter, lowercase, and map every
59
+ // non-(letter|digit) run to a single space. We keep, per emitted token, the [start,end) offsets in
60
+ // the STRIPPED string so an offending run can be reconstructed as readable text for the report.
61
+ const CITE_NEAR = /(?:(?:с|p|стр|page|гл|pp|pg)\.?\s*\d+|\[[^\]]*\d+[^\]]*\])/i; // page-citation signal
62
+
63
+ function stripMarkdown(raw) {
64
+ let s = raw.replace(/\r\n?/g, '\n');
65
+ // YAML frontmatter (only if the very first line is a --- fence)
66
+ if (s.startsWith('---\n')) {
67
+ const end = s.indexOf('\n---', 4);
68
+ if (end >= 0) { const after = s.indexOf('\n', end + 1); s = s.slice(after >= 0 ? after + 1 : s.length); }
69
+ }
70
+ // fenced code blocks ```…``` (and ~~~) — replace with spaces of equal length to preserve offsets
71
+ s = s.replace(/```[\s\S]*?```/g, (m) => ' '.repeat(m.length));
72
+ s = s.replace(/~~~[\s\S]*?~~~/g, (m) => ' '.repeat(m.length));
73
+ // inline code `…`
74
+ s = s.replace(/`[^`\n]*`/g, (m) => ' '.repeat(m.length));
75
+ // markdown links [text](url) / images ![alt](url) → keep the visible text, blank the target
76
+ s = s.replace(/!?\[([^\]]*)\]\(([^)]*)\)/g, (m, txt) => ' ' + txt + ' '.repeat(m.length - txt.length - 1));
77
+ return s;
78
+ }
79
+
80
+ // Blank [p.N] / [с.5] style bracketed page anchors (never part of prose). Run this AFTER the
81
+ // cited-quote exemption on the OUTPUT — the brackets are the exemption's citation signal, so they
82
+ // must still be present when blankCitedQuotes inspects a quote's tail.
83
+ function blankAnchors(s) {
84
+ return s.replace(/\[[^\]\n]*\d[^\]\n]*\]/g, (m) => ' '.repeat(m.length));
85
+ }
86
+
87
+ // Tokenize a stripped string into { tokens:[lc words], starts:[], ends:[] } over that string.
88
+ function tokenize(stripped) {
89
+ const tokens = [], starts = [], ends = [];
90
+ const re = /[\p{L}\p{N}]+/gu;
91
+ let m;
92
+ while ((m = re.exec(stripped)) !== null) {
93
+ tokens.push(m[0].toLowerCase());
94
+ starts.push(m.index);
95
+ ends.push(m.index + m[0].length);
96
+ }
97
+ return { tokens, starts, ends, stripped };
98
+ }
99
+
100
+ // Reconstruct a human-readable snippet spanning output token range [i, j] (inclusive).
101
+ function reconstruct(tk, i, j) {
102
+ const raw = tk.stripped.slice(tk.starts[i], tk.ends[j]);
103
+ return raw.replace(/\s+/g, ' ').trim();
104
+ }
105
+
106
+ // ---- cited-quote exemption (OUTPUT only) ------------------------------------
107
+ // CONSERVATIVE: blank out any quoted run of <= MAX_QUOTE words whose closing quote is immediately
108
+ // (within ~40 chars) followed by a page citation. Done on the STRIPPED string before tokenizing so
109
+ // the exempted words never enter the output shingle set. Quote styles: «…», "…", "…"(curly), '…'.
110
+ const QUOTE_PAIRS = [['«', '»'], ['“', '”'], ['"', '"'], ['‘', '’']];
111
+ function blankCitedQuotes(stripped) {
112
+ let s = stripped;
113
+ for (const [open, close] of QUOTE_PAIRS) {
114
+ let out = '', idx = 0;
115
+ while (idx < s.length) {
116
+ const o = s.indexOf(open, idx);
117
+ if (o < 0) { out += s.slice(idx); break; }
118
+ // find matching close (same char for straight quotes → search after open)
119
+ const c = s.indexOf(close, o + 1);
120
+ if (c < 0) { out += s.slice(idx); break; }
121
+ const inner = s.slice(o + 1, c);
122
+ const words = inner.match(/[\p{L}\p{N}]+/gu) || [];
123
+ const tail = s.slice(c + 1, c + 1 + 40);
124
+ const cited = CITE_NEAR.test(tail);
125
+ out += s.slice(idx, o); // text before the quote is kept verbatim
126
+ if (words.length <= MAX_QUOTE && words.length >= 1 && cited) {
127
+ // exempt: blank the ENTIRE quote span (open..close) so its shingles never form
128
+ out += ' '.repeat(c - o + 1);
129
+ } else {
130
+ // not exempt: keep the quote content so a long/uncited copy is still shingled & flaggable
131
+ out += s.slice(o, c + 1);
132
+ }
133
+ idx = c + 1;
134
+ }
135
+ s = out;
136
+ }
137
+ return s;
138
+ }
139
+
140
+ // ---- shingling --------------------------------------------------------------
141
+ // A shingle = W consecutive tokens joined by a single space. We store the joined string itself in a
142
+ // Set (exact, collision-free — no hashing needed for correctness); source contributes only the set.
143
+ function shinglesOf(tokens) {
144
+ const out = [];
145
+ for (let i = 0; i + W <= tokens.length; i++) out.push(tokens.slice(i, i + W).join(' '));
146
+ return out;
147
+ }
148
+
149
+ // ---- table header extraction (light copy heuristic) -------------------------
150
+ // Collect the normalized header-cell SET of every markdown table. A table = a header row of pipe
151
+ // cells immediately followed by a separator row (|---|---|). Best-effort, cheap.
152
+ function tableHeaderSets(raw) {
153
+ const lines = raw.replace(/\r\n?/g, '\n').split('\n');
154
+ const sets = [];
155
+ for (let i = 0; i + 1 < lines.length; i++) {
156
+ const sep = lines[i + 1];
157
+ if (!/^\s*\|?\s*:?-{2,}.*\|/.test(sep) || !sep.includes('-')) continue;
158
+ if (!lines[i].includes('|')) continue;
159
+ const cells = lines[i].split('|').map((c) => c.trim()).filter(Boolean)
160
+ .map((c) => (c.match(/[\p{L}\p{N}]+/gu) || []).join(' ').toLowerCase()).filter(Boolean);
161
+ if (cells.length >= 2) sets.push(new Set(cells));
162
+ }
163
+ return sets;
164
+ }
165
+ const setEq = (a, b) => a.size === b.size && [...a].every((x) => b.has(x));
166
+
167
+ // ---- build SOURCE ----------------------------------------------------------
168
+ const sourceFiles = collect(sourceArg);
169
+ const sourceSet = new Set();
170
+ const sourceTableSets = [];
171
+ for (const f of sourceFiles) {
172
+ const raw = readFileSync(f, 'utf-8');
173
+ const stripped = blankAnchors(stripMarkdown(raw));
174
+ const { tokens } = tokenize(stripped);
175
+ for (const sh of shinglesOf(tokens)) sourceSet.add(sh);
176
+ for (const t of tableHeaderSets(raw)) sourceTableSets.push(t);
177
+ }
178
+
179
+ // ---- scan OUTPUT -----------------------------------------------------------
180
+ const outputFiles = collect(outputArg);
181
+ const violations = []; // { file, words, text, startTok, endTok }
182
+ const tableWarnings = []; // { file, header:[...] }
183
+ let outputShingleCount = 0;
184
+
185
+ for (const f of outputFiles) {
186
+ const raw = readFileSync(f, 'utf-8');
187
+ let stripped = stripMarkdown(raw); // anchors kept — needed as the citation signal
188
+ stripped = blankCitedQuotes(stripped); // exemption uses the still-present [с.N] anchors
189
+ stripped = blankAnchors(stripped); // now remove any remaining anchors before shingling
190
+ const tk = tokenize(stripped);
191
+ const shingles = shinglesOf(tk.tokens);
192
+ outputShingleCount += shingles.length;
193
+
194
+ // mark which output shingles overlap the source, then coalesce consecutive hits into runs.
195
+ const hit = shingles.map((sh) => sourceSet.has(sh));
196
+ let i = 0;
197
+ while (i < hit.length) {
198
+ if (!hit[i]) { i++; continue; }
199
+ let j = i;
200
+ while (j + 1 < hit.length && hit[j + 1]) j++;
201
+ // shingles [i..j] overlap → token span [i .. j + W - 1] is one contiguous verbatim run
202
+ const startTok = i, endTok = j + W - 1;
203
+ violations.push({
204
+ file: f,
205
+ words: endTok - startTok + 1,
206
+ text: reconstruct(tk, startTok, endTok),
207
+ startTok, endTok,
208
+ });
209
+ i = j + 1;
210
+ }
211
+
212
+ // table-copy heuristic
213
+ for (const oh of tableHeaderSets(raw)) {
214
+ if (oh.size >= 2 && sourceTableSets.some((sh) => setEq(oh, sh))) {
215
+ tableWarnings.push({ file: f, header: [...oh] });
216
+ }
217
+ }
218
+ }
219
+
220
+ // ---- report ----------------------------------------------------------------
221
+ const rel = (p) => { try { return relative(process.cwd(), p) || p; } catch { return p; } };
222
+ const perFile = {};
223
+ for (const v of violations) perFile[v.file] = (perFile[v.file] || 0) + 1;
224
+
225
+ const report = {
226
+ tool: 'shingling-check',
227
+ config: { shingle: W, maxQuote: MAX_QUOTE, source: sourceArg, output: outputArg },
228
+ sourceFiles: sourceFiles.length,
229
+ outputFiles: outputFiles.length,
230
+ sourceShingles: sourceSet.size,
231
+ outputShingles: outputShingleCount,
232
+ totalViolations: violations.length,
233
+ perFile: Object.fromEntries(Object.entries(perFile).map(([k, v]) => [rel(k), v])),
234
+ samples: violations.slice(0, 15).map((v) => ({ file: rel(v.file), words: v.words, text: v.text })),
235
+ tableWarnings: tableWarnings.map((t) => ({ file: rel(t.file), header: t.header })),
236
+ pass: violations.length === 0,
237
+ };
238
+
239
+ // human summary
240
+ const line = '─'.repeat(60);
241
+ console.log(line);
242
+ console.log(`shingling-check — verbatim-overlap IP gate (w=${W}, max-quote=${MAX_QUOTE})`);
243
+ console.log(line);
244
+ console.log(`source: ${sourceFiles.length} file(s), ${sourceSet.size} unique ${W}-word shingles`);
245
+ console.log(`output: ${outputFiles.length} file(s), ${outputShingleCount} shingles scanned`);
246
+ console.log(`verbatim runs (violations): ${violations.length}`);
247
+ if (violations.length) {
248
+ console.log('\nper-file:');
249
+ for (const [f, n] of Object.entries(report.perFile)) console.log(` ${n} ${f}`);
250
+ console.log(`\nsample runs (up to 15):`);
251
+ for (const s of report.samples) {
252
+ const snip = s.text.length > 120 ? s.text.slice(0, 117) + '…' : s.text;
253
+ console.log(` [${s.words}w] ${rel(s.file)}\n "${snip}"`);
254
+ }
255
+ }
256
+ if (tableWarnings.length) {
257
+ console.log(`\ntable-copy warnings (RESTRUCTURE, don't transcribe): ${tableWarnings.length}`);
258
+ for (const t of report.tableWarnings.slice(0, 10)) console.log(` ${t.file}: [${t.header.join(' | ')}]`);
259
+ }
260
+ console.log(line);
261
+ console.log(violations.length === 0
262
+ ? 'PASS — no uncited verbatim run >= ' + W + ' words.'
263
+ : `FAIL — ${violations.length} uncited verbatim run(s) >= ${W} words. Paraphrase or cite as a short quote.`);
264
+ console.log(line);
265
+
266
+ if (jsonOut) {
267
+ writeFileSync(jsonOut, JSON.stringify(report, null, 2));
268
+ console.log(`report → ${rel(jsonOut)}`);
269
+ }
270
+
271
+ process.exit(violations.length === 0 ? 0 : 1);