deep-read-summarize 0.5.0 → 0.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/.env.example CHANGED
@@ -4,7 +4,7 @@
4
4
  # 请在启动 DSH 的那个环境里直接设置(shell 的 export / Windows 的 setx)。
5
5
  #
6
6
  # 其余配置不是环境变量,而是插件 config 或 workflow 的 options:
7
- # outputDir / tempDir / minWords / maxChunks / fastMode / transcribe /
7
+ # outputDir / tempDir / minWords / maxChunks / readConcurrency / parallelQuality / readCache / readCacheIdentity / fastMode / transcribe /
8
8
  # requireCitations / includeTimestamps / whisperModel / language / device / maxRetries
9
9
 
10
10
  # uv 拉取 CPython 时使用的 GitHub 镜像前缀(只在需要下载 Python 3.12 时才用得上)。
package/CHANGELOG.md CHANGED
@@ -5,6 +5,48 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/).
5
5
 
6
6
  ## [Unreleased]
7
7
 
8
+ ## [0.6.1] — 2026-11-10
9
+
10
+ 修复普通执行路径的结果序列化错误,避免已生成的精读笔记在返回宿主时被丢弃。
11
+
12
+ ### Fixed
13
+
14
+ - 无缓存复用的成功分支不再返回 `__cacheReads: undefined`:该字段无法通过宿主 workflow 引擎的 JSON 序列化检查(「is not JSON data」),会连已成稿的 `note` 一起被整体丢弃。改为按需展开(仅缓存复用时携带),普通路径的结果不再含内部缓存字段;缓存复用路径返回数组不变。回归测试见 `tests/workflow-regressions.test.js`。
15
+
16
+ ### Verification
17
+
18
+ - PR 的七项 CI 检查通过,覆盖 Node.js 18、20、22、24 和 Windows、macOS、Linux。使用 DSH 0.2.2-rc.1 的 workflow 执行器与模拟代理复现原错误,并验证修复后普通路径和缓存路径的返回结果;未进行真实模型端到端调用。
19
+
20
+ ### Compatibility
21
+
22
+ - 升级无需更改配置;缓存复用路径保持原有行为。
23
+
24
+ ### 贡献者与致谢
25
+
26
+ - 感谢 @Zyx-A 报告、定位并修复结果序列化问题,以及补充回归测试([PR #8](https://github.com/PensiveFei/deep-read-summarize/pull/8))。
27
+
28
+ ## [0.6.0] — 2026-11-05
29
+
30
+ 新增可配置并发、字符均衡分块与可选的并行原文核验和跨次精读缓存,保留原有默认核验路径。
31
+
32
+ ### Added
33
+
34
+ - **精读并发与计时**:`readConcurrency` 可设置 1–12,默认仍为 3;`readTimings` 返回各块耗时、成功与缓存命中状态。提高并发前应观察宿主限流和失败情况。
35
+ - **字符均衡分块**:获取代理可调用随包脚本计算整行字符负载并优先选择接近的段落边界;统计不可用或计划无效时回退按行均衡,均保留全文行号覆盖。字符数只是 token 的近似,超长单行不会拆开。
36
+ - **可选并行独立核验**:`parallelQuality: true` 时多块分别读取原文、检查全部显式引用,另检查全局术语和语言;证据缺失或核验失败不通过。默认使用原有核验路径。
37
+ - **可选跨次精读缓存**:仅插件工具入口支持,默认关闭。开启时必须由调用者声明包含供应商、固定模型版本及推理设置的 `readCacheIdentity`;源文件内容、提示词和身份参与缓存键,仅质量通过后写入,命中后仍进行独立核验。身份无法由宿主自动认证。缓存为本地明文,可能含原文摘录。
38
+
39
+ ### Verification and limitations
40
+
41
+ - 回归测试覆盖分块完整性、并发调度、引用证据、缓存失效和失败回退;调度与插件入口测试使用模拟宿主,未测量真实 DSH 模型的延迟或质量,不能据此声称加速比例。
42
+ - 字符统计依赖获取代理执行和返回辅助脚本结果;模型身份由用户保证。源文件在任务前后重新哈希,但无法检测期间短暂发生后恢复的变化。Windows 当前测试环境没有创建符号链接的权限,相关平台测试需由支持的环境补测。
43
+
44
+ ### Compatibility
45
+
46
+ - `parallelQuality` 与 `readCache` 默认关闭;`readConcurrency` 默认为 3。旧 workflow 入口不支持跨次精读缓存,其他能力保持可用。
47
+ - 贡献者与致谢:本版无外部 PR、报告或复测贡献需要署名。
48
+
49
+
8
50
  ## [0.5.0] — 2026-11-01
9
51
 
10
52
  无字幕视频新增逐字稿内容缓存、转写阶段计时和 CPU 线程配置,保留默认精读与原文核验门槛。
package/README.en.md CHANGED
@@ -7,396 +7,152 @@
7
7
  [![CI](https://img.shields.io/github/actions/workflow/status/PensiveFei/deep-read-summarize/ci.yml)](https://github.com/PensiveFei/deep-read-summarize/actions/workflows/ci.yml)
8
8
  [![npm total downloads](https://badgen.net/npm/dt/deep-read-summarize)](https://www.npmjs.com/package/deep-read-summarize)
9
9
 
10
- > **Disclaimer**: This is an **unofficial third-party tool**. It is not affiliated with, endorsed by, or sponsored by DeepSeek or the DeepSeek Harness project. "DeepSeek" and "DeepSeek Harness" are trademarks of their respective owners. This project only orchestrates the workflow tool available in your DSH environment; it does not redistribute any DeepSeek software.
10
+ Turn a paper, book, video, or web page into a **Markdown reading note you can keep, edit, and check against the source**.
11
11
 
12
- > **⚠️ Most importantly: this is an "assisted deep-reading" tool, not a substitute for your own learning.** It helps you quickly extract, summarize, and structure what you read, but **real understanding, thinking, and critique must still come from you**. Always treat the original source as authoritative — do not mistake the generated notes for the learning itself.
12
+ This third-party plugin runs in [DeepSeek Harness (DSH)](https://github.com/deepseek-ai/deepseek-harness). It attempts to retrieve the source text, reads long material in chunks, and combines the results into a structured note with YAML frontmatter. You can use the note in Obsidian, Logseq, or another Markdown tool.
13
13
 
14
- A deep-reading workflow for DSH (DeepSeek Harness): feed it a book, a paper, a video link, or a web page, and it produces a structured Markdown note (YAML frontmatter — ready for Obsidian, Logseq, static-site content folders, or any Markdown editor).
14
+ [See real notes](#see-the-notes) · [Get started](#get-started) · [Quality and limitations](#quality-and-limitations) · [Common options](#common-options)
15
15
 
16
- Content is split into chunks, read in parallel by multiple sub-agents, then merged into a Markdown file with YAML frontmatter. Key conclusions carry citations to the original source, and a quality check runs before the final draft.
16
+ > AI-generated notes can omit or misinterpret information. Use them as a starting point for reading, and check important claims and quotations against the source.
17
17
 
18
- > If you run into problems, or see how it could be better, **feel free to open a GitHub issue**.
18
+ ## See the notes
19
19
 
20
- > Note: DSH is currently a developer preview and its interfaces may change. This repo targets a specific version of the workflow-tool semantics — see [Compatibility](#compatibility).
20
+ - 📄 [Paper: Emotional Intelligence of Large Language Models](https://github.com/PensiveFei/deep-read-summarize/blob/main/examples/emotional-intelligence.md)
21
+ - 📄 [Paper: Chain-of-Thought Prompting](https://github.com/PensiveFei/deep-read-summarize/blob/main/examples/chain-of-thought.md)
22
+ - 🎬 [Video: Andrej Karpathy's introduction to LLMs](https://github.com/PensiveFei/deep-read-summarize/blob/main/examples/intro-to-llms-video.md)
21
23
 
22
- > **Platforms**: Windows / macOS / Linux are equally supported (since 0.3.9) — see [Platform support](#platform-support) below.
24
+ These are historical notes supplied by the maintainer, in Chinese. The paper notes have records of early project runs; how the video note was generated has not been confirmed. Their model, project version, and runtime were not recorded, so they are not performance claims for the current release. Some quotations and timestamps have been spot-checked, and edits to the public copies are documented in the [examples guide](https://github.com/PensiveFei/deep-read-summarize/blob/main/examples/README.md).
23
25
 
24
- ---
25
-
26
- ## What it looks like
27
-
28
- (Images use absolute GitHub URLs so they render on npm too; the PNGs ship with the repository, not inside the npm package.)
29
-
30
- | Deep-read paper note | Video note metadata |
31
- | --- | --- |
32
- | ![Paper note](https://raw.githubusercontent.com/PensiveFei/deep-read-summarize/main/docs/showcase/showcase-paper-note-cot.png) | ![Video note metadata](https://raw.githubusercontent.com/PensiveFei/deep-read-summarize/main/docs/showcase/showcase-video-note-metadata.png) |
33
-
34
- | Video note action items | In-chat digest |
35
- | --- | --- |
36
- | ![Video note action items](https://raw.githubusercontent.com/PensiveFei/deep-read-summarize/main/docs/showcase/showcase-video-note-actions.png) | ![In-chat digest](https://raw.githubusercontent.com/PensiveFei/deep-read-summarize/main/docs/showcase/showcase-in-chat-digest.png) |
37
-
38
- ---
26
+ ![A structured reading note for a paper](https://raw.githubusercontent.com/PensiveFei/deep-read-summarize/main/docs/showcase/showcase-paper-note-cot.png)
39
27
 
40
28
  ## What it does
41
29
 
42
- - Books (PDF/EPUB/MOBI), papers (arXiv/PDF/HTML), videos (**subtitles first, auto-transcription when there are none** — produces a full transcript for deep reading), web pages
43
- - Long content is chunked and read in parallel by sub-agents, then merged
44
- - The fetch/chunk plan and the quality check are constrained by JSON Schema, so their output is stable and machine-readable
45
- - Chunk count — and therefore sub-agent count — is enforced by `maxChunks` (a runaway plan blows the per-call time limit)
46
- - Key citations must carry page/chapter/paragraph references to reduce fabrication
47
- - Configuration errors abort immediately; a chunk that fails to parse is skipped and marked as a gap
48
- - Output is **standard Markdown + YAML frontmatter**: the body is plain CommonMark (no wikilinks, no callouts), so Obsidian (with Dataview), Logseq, Jekyll / Hugo / Astro content folders and any Markdown editor can consume it directly
49
-
50
- ---
51
-
52
- ## Structure
53
-
54
- ```
55
- Input (link or file path)
56
- │
57
- ▼
58
- Parser registry ── book / paper / video / web
59
- │
60
- ▼
61
- Wave 1 Fetch content → write temp file → build chunk plan
62
- │
63
- ▼
64
- Wave 2 N sub-agents deep-read chunks in parallel (Map)
65
- │
66
- ▼
67
- Wave 3 Merge draft + quality check (Reduce)
68
- │
69
- ▼
70
- Markdown note
71
- ```
72
-
73
- Three waves, normally N+3 sub-agent calls (fetch, N chunk reads, draft, and quality check). In `auto` mode, ambiguous PDFs, unfamiliar web pages, and text inputs require one additional lightweight classification call. An explicit `type` skips classification.
74
-
75
- `auto` routes recognized video links, arXiv paper links, and ebook extensions directly to their parsers; columns, channels, and help pages on those sites still require classification. PDFs are classified as books or papers from their content. Failed classification returns `stage: "detect"` and asks for an explicit type. `options.tempDir` is a temporary root: each call creates its own `run-*` subdirectory, keeping fetched text, audio, and transcripts separate from other tasks.
76
-
77
- Chunk plans are sorted, duplicate ranges trimmed, and gaps filled. Plans above `maxChunks` merge adjacent ranges instead of discarding the document tail. The returned `coverageComplete` indicates coverage of all declared lines based on a valid `totalLines`; without a line count, known ranges are retained, `qualityPassed` is set to `false`, and the coverage uncertainty is reported.
78
-
79
- Quality checks combine code and source review. Code validates the supplied scalar/list frontmatter template, required sections, and length (CJK characters plus other words, excluding metadata, code blocks, HTML comments, and link URLs). A separate reviewer reads the current source file and returns per-chunk excerpts and line-bound evidence for verbatim quotations in section 4. A bare `pass: true` cannot override these checks. `qualityChecks` exposes the gates, and `qualityEvidence` exposes the source path and evidence. Source authenticity still depends on the reviewer's actual source reading; uncertainty fails the gate. Empty, rejected, or structurally degraded repairs retain the preceding usable note.
80
-
81
- Direct workflow options are strict: `minWords` is an integer in 1–100000, `maxChunks` is a positive integer (values above 12 are capped), and `maxRetries` is an integer in 0–3. Invalid values and unknown keys fail before work begins. Plugin profile configuration instead warns per invalid field and uses its default, preserving host startup.
82
-
83
- ```
84
- parsers/ Parsers for each input type, dispatched by type
85
- book.js Books: PDF/EPUB/MOBI text extraction, chapter chunking
86
- paper.js Papers: arXiv/PDF/HTML structure detection
87
- video.js Video: full transcript (subtitles first; auto-transcribe via scripts/transcribe.js when none)
88
- web.js Web pages: main-text extraction
89
- index.js Registry: parser discovery and fallback
90
- schemas/ JSON Schema for the output contract (injected into the script by workflow.js - single source of truth)
91
- scripts/ lint (incl. line-ending/whitespace hygiene), security checks, cross-platform transcription bootstrap
92
- lib/ Small standalone helpers
93
- cache.js input fingerprint and idempotency records (for callers driving the workflow themselves -- the
94
- workflow sandbox has no filesystem, so it cannot use this)
95
- workflow-script.txt the workflow script body (plain text: read and edit it directly, no escaping involved)
96
- tests/ tests and validation scripts (**not published to npm**, repo only)
97
- workflow.js the workflow meta and assembly logic (parser inlining, schema injection)
98
- ```
99
-
100
- > The script body lives in `lib/workflow-script.txt` on purpose: this file was corrupted twice by "escaping on top of escaping". It is plain text now, so reading and editing it needs no encoding step.
101
-
102
- To override or add a parser, use the package's `custom-parsers/` directory (included in npm packaging), or set the plugin-level `config.customParsersDir` to your own directory. This is not a workflow option. Per-profile directories remain isolated, and rebuilding skill content uses the current registry, including programmatic registrations. Parsers expose `name`, `types`, and `buildPrompt(input, opts)`; registered types are inlined into the workflow and its schema.
103
-
104
- File parsers bundle their CommonJS modules and literal relative `.js`, `.cjs`, and `.json` dependencies, preserving module constants, helpers, method shorthand, and cycles. Acorn parses JavaScript during script construction and is installed automatically by npm; it does not run in the workflow sandbox. Node built-ins, external packages, and dynamic `require`/`import` are unsupported in sandbox parsers and fail at build time. Programmatically registered functions must be self-contained; use a file parser for module closures.
105
-
106
- ---
107
-
108
- ## Platform support
109
-
110
- **Windows / macOS / Linux are equally supported** (since 0.3.9). The plugin itself only needs Node (which DSH ships); the extra tools each input type may use are installed per platform:
111
-
112
- - **Book / paper PDF** → `pdftotext` (poppler): macOS `brew install poppler`, Windows `winget install poppler`, Debian/Ubuntu `apt install poppler-utils`
113
- - **Bilibili subtitles / audio** → `curl`: already present on Windows 10+, macOS and Linux — nothing to install
114
- - **YouTube subtitles** → `yt-dlp`: macOS `brew install yt-dlp`, Windows `winget install yt-dlp.yt-dlp`, any platform `pip install -U yt-dlp`
115
- - **Transcription when there are no subtitles** → `uv` (the script bootstraps a Python 3.12 env and installs faster-whisper): macOS `brew install uv`, Windows `winget install astral-sh.uv`, or the official installer script on macOS / Linux (hand `curl -LsSf https://astral.sh/uv/install.sh` to `sh`)
116
-
117
- Further notes:
118
-
119
- - Transcription is the **only** part that needs an extra toolchain, and all three platforms share one Node script and one command: `node scripts/transcribe.js --audio <file> --out <txt>`.
120
- - Transcription runs on **CPU** by default, so Apple Silicon / Intel Macs and Linux need no CUDA; GPU support needs a CUDA runtime and `options.device: "auto"` will try it and fall back to CPU on failure.
121
- - Cache location per platform: macOS `~/Library/Caches/deep-read-summarize`, Linux `$XDG_CACHE_HOME/deep-read-summarize`, Windows `%LOCALAPPDATA%\deep-read-summarize`.
122
- - A **full transcription has been run successfully on real macOS hardware** (since 0.3.12): `uv` bootstraps Python 3.12, installs faster-whisper, downloads the model and writes the transcript — exit code 0. CI's `macos-latest` job separately covers platform path resolution and the whole test suite. If you hit an error on a Mac, please open an issue.
123
-
124
- ---
125
- ## Installation (DSH plugin)
126
-
127
- deep-read-summarize is a DSH plugin installable via npm or locally:
128
-
129
- ```bash
130
- # npm install (includes Acorn for parser construction)
131
- npm install deep-read-summarize
132
-
133
- # or local install (dsh profile directory)
134
- pnpm add ./deep-read-summarize-0.3.10.tgz
135
- # then append to the dsh config's dsh.profile.bundles:
136
- # - deep-read-summarize
137
- # restart dsh web (POST /dsh-market/restart)
138
- ```
139
-
140
- After installation it registers:
141
- - the `deep_read_summarize` tool when the host provides tools and workflowEngine services; models send only input/type/options
142
- - the `deep-read-summarize` workflow definition (meta + script), with the existing workflow-tool route retained for older hosts
143
- - the `deep-read-summarize` skill (`skills/deep-read-summarize/SKILL.md`)
144
- - four parsers (`parsers/`) and JSON Schemas (`schemas/`)
145
-
146
- ---
147
- ## Quick start (~5 minutes)
30
+ | You provide | The workflow attempts to retrieve | You receive |
31
+ | --- | --- | --- |
32
+ | A book: PDF, EPUB, MOBI | Text and chapters | A reading note organized by topic |
33
+ | A paper: arXiv, PDF, HTML | Paper text and structure | Research question, method, findings, and limitations |
34
+ | A video link | Subtitles first; local transcription when needed and available | A breakdown based on the text obtained |
35
+ | A web page | Page text | Key points, concepts, and source information |
148
36
 
149
- ### 1. Install
37
+ Extraction depends on source accessibility, available tools, the model, and host permissions. Scanned PDFs generally need OCR first; the workflow does not provide automatic OCR. Complex formulas, diagrams, and poor-quality captions or audio may also need manual handling.
150
38
 
151
- ```bash
152
- # Option A: npm install directly (no clone needed)
153
- npm install deep-read-summarize
39
+ Notes typically include an overview, structural breakdown, glossary, key quotations, critical analysis, and applications. Their contents are model-generated and may differ between inputs and configurations.
154
40
 
155
- # Option B: develop from source
156
- git clone https://github.com/PensiveFei/deep-read-summarize.git
157
- cd deep-read-summarize
158
- npm ci # install locked dependencies
159
- ```
41
+ ## Get started
160
42
 
161
- ### 2. Verify installation (offline, no API key needed)
43
+ ### 1. Install the plugin
162
44
 
163
- ```bash
164
- node -e "console.log(require('deep-read-summarize').name)"
165
- ```
166
-
167
- Seeing `deep-read-summarize` confirms the package loads. Source checkouts can additionally run `npm test` and `npm run test:node`; tests are not included in the npm package.
168
-
169
- ### 3. Feed a demo file with one command
45
+ Install and start DSH Desktop once to initialize its desktop profile. **Quit the application completely**, then use the DSH command bundled with Desktop to install the plugin:
170
46
 
171
47
  ```bash
172
- node -e "const wf = require('./workflow.js'); console.log('meta:', wf.meta.name); console.log('parsers:', wf.parsers.list().map(p => p.name).join(', '));"
48
+ dsh plugin --profile desktop add deep-read-summarize
173
49
  ```
174
50
 
175
- Output looks like:
51
+ Reopen DSH Desktop when installation finishes. The bundled command may not be on your system PATH; use `resources/runtime/cli/bin/dsh` inside the installation directory (`dsh.cmd` on Windows). Interfaces and commands can change between host versions; see the [official DSH Desktop instructions](https://github.com/deepseek-ai/deepseek-harness/blob/master/apps/desktop/README.md). Do not run `npm install` or `npm update` directly inside a DSH profile directory.
176
52
 
177
- ```
178
- meta: deep-read-summarize
179
- parsers: book, paper, video, web
180
- ```
53
+ Deep reading also requires a working model and suitable permissions in DSH. Some sources need additional tools: PDF extraction may need `pdftotext`, YouTube captions may need `yt-dlp`, and local transcription of a video without subtitles may require `uv` and a transcription model.
181
54
 
182
- ### 4. Real run (requires DSH)
55
+ ### 2. Request a reading note
183
56
 
184
- Prefer passing the JSON below directly to `deep_read_summarize`. On older hosts, use the skill's meta + script with the existing `workflow` tool:
57
+ In a DSH conversation, for example:
185
58
 
186
- ```jsonc
187
- { "input": "https://arxiv.org/abs/2307.09042", "type": "paper", "options": { "maxChunks": 4, "fastMode": true } }
59
+ ```text
60
+ Please read this paper in depth and save a Markdown reading note:
61
+ https://arxiv.org/abs/2307.09042
188
62
  ```
189
63
 
190
- The workflow returns a structured result: `{ ok, kind, title, filePath, qualityPassed, note }`, where `note` is the final Markdown note.
64
+ You can also provide a local file, web page, or video link. If needed, specify an output location, sections to retain, or whether you want timestamps.
191
65
 
192
- > ⚠️ **The caller persists the note.** DSH's workflow contract gives the script no filesystem access — it only coordinates subagents — so the script returns the note body plus a suggested `filePath` and never writes the file itself. The bundled SKILL.md tells the model to write `note` to `filePath` with the `write` tool after the run; do the same when you invoke the workflow by hand, or the note exists only in the return value.
66
+ Hosts with plugin-tool support register `deep_read_summarize`; older hosts use the bundled skill and workflow entry point. The tool returns the note and a suggested path; **the main agent must write the file**. Check that the file was saved and whether the result reports gaps or quality issues.
193
67
 
194
- ---
195
- ## Usage
68
+ <details>
69
+ <summary>Direct tool-call example</summary>
196
70
 
197
- The `deep_read_summarize` tool takes these parameters directly. On older hosts, use them as workflow args with the skill's meta + script:
198
-
199
- ```jsonc
71
+ ```json
200
72
  {
201
- "input": "https://arxiv.org/abs/2307.09042", // link or file path
202
- "type": "auto", // auto | book | paper | video | web
73
+ "input": "https://arxiv.org/abs/2307.09042",
74
+ "type": "paper",
203
75
  "options": {
204
- "minWords": 2500,
205
- "fastMode": false, // true skips sections 5-7, faster
206
- "maxChunks": 4, // chunk limit, 1-12 (default lowered to control sub-agent count and time)
207
- "transcribe": true, // video: auto local transcription (faster-whisper) when no subtitles; false skips it — see Video
208
- "whisperModel": "small", // video ASR tier: small | base | medium
209
- "language": "zh", // video ASR language: use "en" for English-only videos
210
- "device": "cpu", // ASR device: cpu (default) | auto | cuda; auto/cuda needs a CUDA runtime and falls back to cpu
211
- "cpuThreads": 0, // 0 keeps library defaults; 1-64 allows hardware-specific tuning
212
- "transcriptCache": true, // reuse a verified local transcript for matching audio and settings
213
- "maxRetries": 0, // quality-check retries (default 0 = no retry; retrying is noticeably slower)
214
- "frontmatter": "obsidian", // key set: obsidian (default, for Dataview) | generic (drops the keys Hugo reserves) | minimal
215
- "requireCitations": true, // whether key conclusions must carry citations
216
- "includeTimestamps": false,
217
- "outputDir": "./output", // note output directory (can point at your Obsidian vault or a static-site content folder)
218
- "tempDir": "./.tmp" // temp file directory
76
+ "maxChunks": 4,
77
+ "outputDir": "./output"
219
78
  }
220
79
  }
221
80
  ```
222
81
 
223
- Examples:
224
-
225
- | Type | input |
226
- |------|-------|
227
- | Paper | `https://arxiv.org/abs/2307.09042` |
228
- | Book | local path, e.g. `~/books/xxx.pdf` |
229
- | Video | `https://youtube.com/watch?v=xxx` |
230
- | Web page | `https://example.com/article` |
231
-
232
- Notes are written to `options.outputDir` (default `./output`, can point at your Obsidian vault or a static-site content folder), with the filename taken from the content title.
233
-
234
- ### Frontmatter profiles (`options.frontmatter`)
235
-
236
- The default is `obsidian`, with the same keys as earlier versions and safely quoted scalar/list values. Profiles exist because `type`, `aliases` and `url` are **reserved keys in Hugo** with different meanings -- dropping an Obsidian note into Hugo changes how the page renders, rather than adding a few harmless fields.
237
-
238
- | Key | `obsidian` (default) | `generic` | `minimal` |
239
- |---|---|---|---|
240
- | `title` | yes | yes | yes |
241
- | `aliases` | yes | -- | -- |
242
- | `author` / `year` | yes | yes | -- |
243
- | `type` | yes | -- | -- |
244
- | `docType` | -- | yes | -- |
245
- | `url` | yes | -- | -- |
246
- | `sourceUrl` | -- | yes | yes |
247
- | `tags` | yes | yes | yes |
248
- | `created` | yes | -- | -- |
249
- | `date` | -- | yes | yes |
250
- | `status` | yes | yes | -- |
251
-
252
- Why `generic` renames things:
253
-
254
- - `type` in Hugo selects the **layout** (`type: video` makes it look for `layouts/video/`)
255
- - `aliases` in Hugo is **a list of old URLs that redirect to this page**
256
- - `url` in Hugo **overrides the page's own path** -- the source link would become the article's permalink
257
- - `date` is the time key both Hugo and Jekyll understand; `created` is just a custom field
258
-
259
- An unknown value is fatal rather than silently ignored, matching how `type` is handled.
260
-
261
- ### Video: full transcript (subtitles first, auto-transcription when none)
262
-
263
- Video text extraction is a **unified pipeline** (the old "three tiers" are gone): the goal is to get the **full transcript**, then deep-read it.
264
-
265
- > **⚠️ To preserve deep-reading quality, processing may take noticeably longer (a deliberate trade-off, not a hang)**: a **subtitle-less** video is transcribed first (roughly 0.5–2× the video length on CPU) before deep reading, so such videos are **much slower than subtitled videos**; in exchange you get a **full transcript + real deep reading**. **Subtitled** videos remain seconds-fast. Please do not mistake the slower transcription for a hang.
266
-
267
- 1. **Platform subtitles available** (Bilibili's official API AI subtitles / YouTube CC) → **use them directly** (= full text, fastest, zero dependencies).
268
- 2. **No public subtitles** → use the bundled transcription `scripts/transcribe.js` (faster-whisper small / int8 / VAD / Chinese) to get the full text.
269
- 3. **Neither works** → **degrade gracefully**: prompt the user to supply a transcript manually, or fall back to `desc` as background; **never block**.
270
-
271
- `options.transcribe`: **default `true`** (auto-transcribe when no subtitles); set `false` to skip transcription (use subtitles/desc or degrade).
272
-
273
- **Transcription toolchain**: `uv` provisions **Python 3.12** and installs **`faster-whisper==1.2.1`** from the Tsinghua mirror; models use `HF_ENDPOINT=https://hf-mirror.com` and the local cache. Healthy cached environments run directly without requiring uv or a separate Python import probe. Actual interpreter and dependency versions are recorded in the cache's `runtime.json`; Python patch releases, uv, and all transitive dependencies are not fully pinned. PyAV decodes audio without a separate ffmpeg. All three platforms use `node scripts/transcribe.js --audio <file> --out <txt>`. CPU is the default; `options.device: "auto"` can try a supplied CUDA runtime and falls back to CPU. Cache roots are `%LOCALAPPDATA%\deep-read-summarize` on Windows, `~/Library/Caches/deep-read-summarize` on macOS, and `$XDG_CACHE_HOME/deep-read-summarize` on Linux (falling back to `~/.cache`). Python provisioning tries a mirror and then direct GitHub; `DRS_UV_PYTHON_MIRROR` overrides it, with an empty value selecting direct downloads.
274
-
275
- > ⚠️ **The first transcription downloads the model (small ≈ 484 MB, via hf-mirror, not GitHub) and may take a few minutes**. Later runs reuse model files; new audio still requires full inference. Only a verified transcript cache hit skips inference. **Only subtitle-less videos trigger transcription** (subtitled videos use their subtitles directly).
276
-
277
- **Transcript cache and threads**: `options.transcriptCache` defaults to true. Keys include the audio SHA-256, managed local model snapshots, installed dependencies, and decoding settings. Canonical segments retain real start/end times, so changing timestamp presentation does not repeat inference. Changed content/configuration/models/dependencies, custom local model directories, unknown environments, corrupt entries and expired entries bypass reuse. Cache lookup uses the existing local snapshot and does not check for upstream model updates online. Entries live under `transcripts-v1` in the cache root: 8 MiB per entry, 128 MiB total, 30-day validity, pruned on writes. Audio is not stored, and reading/source review still runs.
278
-
279
- `options.cpuThreads` accepts integers 0–64. The default `0` preserves the library's CPU thread policy. Compare 4/6/8 on the same audio; more threads are not necessarily faster. CLI controls:
280
-
281
- ```bash
282
- node scripts/transcribe.js --audio sample.m4a --out transcript.txt --cpu-threads 8
283
- node scripts/transcribe.js --audio sample.m4a --out transcript.txt --refresh-cache
284
- node scripts/transcribe.js --audio sample.m4a --out transcript.txt --no-cache
285
- node scripts/transcribe.js --clear-cache
286
- ```
287
-
288
- `--clear-cache` removes only transcripts, retaining models and Python. `--refresh-cache` forces inference and updates the cache; `--no-cache` disables reads and writes. Successful calls print one `metrics` JSON line with `cacheHit`, actual device, audio duration, and milliseconds for hashing, environment preparation, dependency installation, Python, publication and total time. Misses also include import, model loading, preparation (including decoding/VAD), and complete lazy-segment iteration. Hits do not reuse previous inference timings.
82
+ `type` can be `auto`, `book`, `paper`, `video`, or `web`. Specifying it can skip classification for ambiguous inputs. The tool-call interface depends on the DSH host; asking in conversation is also supported.
289
83
 
290
- The note's `qualityPassed` checks the note against acquired text; it does not establish ASR accuracy. ASR changes still need audio/reference comparisons for errors, omissions, numbers, negation and timestamps.
84
+ </details>
291
85
 
292
- **Quality/speed**: `small` + `int8` quantization + `VAD` silence filtering gives usable Chinese quality and is CPU-friendly. **Subtitled videos are not transcribed (fast)**; subtitle-less videos are transcribed at roughly 0.5–2× the video length on CPU.
86
+ ## Quality and limitations
293
87
 
294
- **Failure handling**: install/download failures **report an error and degrade** (prompt for a manual transcript), never silently and never blocking for long. Use `options.transcribe: false` to skip transcription.
88
+ The workflow normalizes chunk ranges and checks coverage of known source line numbers. After drafting, code checks format and length, and an independent review agent reads the obtained text to check chunk coverage and key quotations. Results are available through `qualityPassed`, `qualityChecks`, and `qualityEvidence`.
295
89
 
296
- **How to know what source was used**: source info is **not written into the note** (to keep it clean); it surfaces in: ① the run log's `[source] <value> -> <label>` line; ② the `textSource` field in the returned result (`subtitle / transcription / desc / manual`).
90
+ These checks have limits:
297
91
 
298
- ---
92
+ - **`qualityPassed` does not prove factual accuracy.** Coverage and source authenticity still depend on the reviewing agent reading and evaluating the source. Check important claims, numbers, and quotations yourself.
93
+ - **Retrieving the full text does not guarantee understanding it.** Complex tables, formulas, images, or extraction errors can affect the result; incomplete sources or failed chunks are reported as gaps.
94
+ - **Captions and transcripts can differ from audio.** Automatic captions and speech recognition can omit words or mishear numbers and negations; checking the note against a transcript does not prove the transcript is accurate.
95
+ - **Video timestamps cannot be inferred reliably.** Locations depend on whether the obtained subtitles or transcript contain time information.
299
96
 
300
- ## Variance and influencing factors (why length and content may differ)
97
+ ## Common options
301
98
 
302
- **Output length and content vary noticeably with the model and the input; this is normal.**
303
-
304
- ### Model
305
- - **ASR model tier**: `small` (default, usable for Chinese) → `base` (faster, slightly lower quality) → `medium` (better, slower). Adjust with `options.whisperModel`; download size and speed grow with the model.
306
- - **Language/accent**: Chinese/English/mixed language, dialect, accent, stress → **different transcription accuracy and time**; for pure English video you can set `language: "en"`.
307
- - **Deep-reading LLM**: uses the host DSH model; different context/reasoning models → **different quality, length, and speed**.
308
- - **Non-determinism**: both the LLM and ASR are stochastic, so **re-running the same input may produce slightly different results** (content/length).
309
-
310
- ### Input
311
- - **Video**:
312
- - Whether **public subtitles** exist is the biggest differentiator — public subtitles = seconds-fast; login-only AI subtitles or no subtitles = triggers transcription, **noticeably slower**.
313
- - **Length**: longer video = longer transcription (≈ 0.5–2× the video length on CPU).
314
- - **Audio quality/noise/low volume/pure BGM/no voice**: transcription is error-prone or **empty**; supply a transcript manually in that case.
315
- - **Paper/book/web page**:
316
- - Format: PDF/EPUB/MOBI/HTML/arXiv; **scanned OCR, broken PDFs, dense formulas/figures** → different extraction quality and time.
317
- - **Term density/length** → different chunk count and deep-reading depth.
318
- - **Chunking structure**: different types chunk differently (chapter/topic/paragraph), affecting **output structure** and coverage.
319
-
320
- ### Hardware
321
- - **GPU vs CPU**: gains depend on hardware, runtime libraries, model and audio. A visible GPU does not prove CUDA inference works; compare actual time and transcript accuracy.
322
-
323
- ### Other
324
- - **First run vs reuse**: the first run downloads the model (≈ 484 MB); later runs reuse the cache.
325
- - **Token/cost**: chunk count, length, and citations drive token usage; `fastMode`/`maxChunks` save tokens.
326
- - **Citation/fabrication**: transcription or OCR errors may be quoted as "original text"; **re-check key citations**.
327
- - **Boundaries**: pure BGM, distorted audio → empty/poor transcription; no subtitles and transcription unavailable → degrade to a prompt for a manual transcript.
328
-
329
- ---
330
-
331
- ## How failures are handled
332
-
333
- Two classes:
334
-
335
- - **Configuration errors** (missing input, invalid type, malformed options): throw immediately and abort — no half-baked output.
336
- - **Content problems** (fetch failed, a chunk's deep-read failed): return `{ ok: false, stage, fatal: false }` and let the caller decide. A failed chunk is skipped and marked as a gap in the draft; the whole run is not aborted.
337
-
338
- ---
339
-
340
- ## Faster runs with the same quality gates
341
-
342
- - Prefer `deep_read_summarize` on supported hosts: the model submits parameters instead of reproducing the entire workflow script. Older hosts retain the existing route.
343
- - All known source lines remain covered. Up to three reading tasks run concurrently, and each free slot immediately starts another task. Line-balanced chunks avoid one oversized task holding up a batch.
344
- - Independent source review remains enabled, with duplicate full Map notes removed from review contexts. Missing line counts, failed reads, and description-only video sources cannot be fixed by rewriting the draft, so those states no longer consume full repair loops.
345
- - Video `textSource: "desc"` is explicitly degraded and cannot pass as a complete deep read. Bilibili `p` selects the matching CID, part title, and duration.
346
- - Returned `timings` reports stage durations in milliseconds. Compare the same sources, model, output size, and quality gates; scheduling simulations are not real LLM speedup measurements.
99
+ | Option | Default | Purpose |
100
+ | --- | --- | --- |
101
+ | `minWords` | `2500` | Minimum length; the check counts Chinese characters and words in other text |
102
+ | `fastMode` | `false` | Omit the critical analysis, personal applications, and further-reading sections |
103
+ | `maxChunks` | `4` | Chunk-count limit, capped at 12; this is not the concurrency setting |
104
+ | `readConcurrency` | `3` | Maximum concurrent chunk reads; higher concurrency is not always faster |
105
+ | `maxRetries` | `0` | Repair attempts after a failed quality check; by default, a successfully drafted note receives one review and no repair |
106
+ | `outputDir` | `./output` | Suggested note directory; the main agent performs the write |
107
+ | `requireCitations` | `true` | Require verifiable source information for key quotations |
108
+ | `includeTimestamps` | `false` | Request timestamps for video notes |
109
+ | `transcribe` | `true` | Attempt local transcription when a video has no usable subtitles |
110
+ | `whisperModel` / `language` | `small` / `zh` | Choose a transcription model and language; set `en` for English when appropriate |
111
+ | `device` / `cpuThreads` | `cpu` / `0` | Select the transcription device and CPU thread policy |
112
+ | `transcriptCache` | `true` | Reuse validated transcripts when the audio and configuration match |
113
+ | `parallelQuality` | `false` | Opt in to parallel source review; the original review path remains the default |
114
+ | `readCache` | `false` | Optional cross-run chunk cache; supported by the plugin tool only |
347
115
 
348
- Defaults preserve small/int8, existing decoding settings, output length and independent source review. Transcript caching speeds up repeated audio; new audio can be profiled and tuned with CPU thread settings. Persistent ASR workers, batch inference and hierarchical review require separate benchmarks and are not enabled automatically.
116
+ To enable `readCache`, also set `readCacheIdentity` to include the provider, pinned model version, and output-affecting settings; change it whenever those change. The caller supplies this identity, and the host cannot verify it automatically. Cache entries are local plaintext and may contain source excerpts; cache hits still receive independent source review.
349
117
 
350
- ## Cost
118
+ Other options include `frontmatter` (`obsidian`, `generic`, or `minimal`) and `tempDir`. See the [advanced reference](https://github.com/PensiveFei/deep-read-summarize/blob/main/docs/README-reference.en.md) for detailed options, supported formats, and transcription and cache behavior.
351
119
 
352
- Token use depends on document length, chunk count, and output size. Source review reads the document again, and each repair round adds a repair and a review call. `fastMode` reduces output sections; reducing `maxChunks` limits agent count while preserving the complete source.
120
+ ## Time, cost, and privacy
353
121
 
354
- Programmatic `lib/cache` callers: URL trailing slashes, explicit empty queries, and fragments are preserved because they may select different content. New fingerprints are isolated from old records, so an upgrade can cause one reprocessing pass. Callers must still verify source freshness and configuration before reusing results.
122
+ - **Time** depends on content length, retrieval, chunk count, model, and host concurrency. A video without subtitles needs extra transcription; the first run may also download a transcription model.
123
+ - **Cost** depends on your model and input/output usage; the project cannot promise fixed time or cost.
124
+ - **Privacy:** speech transcription runs locally, but deep reading uses your configured DSH model. With a cloud model, source text, subtitles, notes, or related context may be sent to its provider; check its data-handling terms first.
125
+ - **Local files:** temporary source text, transcripts, and optional caches may remain on your device. Check directory permissions and cleanup before processing sensitive material.
355
126
 
356
- ---
127
+ ## Troubleshooting
357
128
 
358
- ## Security & copyright
129
+ - **Plugin unavailable:** check the DSH profile, restart Desktop after installation, and check host-version compatibility.
130
+ - **Fetch failure or missing sections:** check source accessibility and required tools; inspect the returned stage, source status, and failed chunks.
131
+ - **Only a video description is available:** a complete subtitle or transcript has not been confirmed. Check subtitle and transcription availability, or provide a transcript.
132
+ - **Quality check failed:** inspect `qualityIssues` and `qualityEvidence` and review the source as needed. Do not treat an unverified note as a complete account.
359
133
 
360
- - Fetching videos/web pages makes external requests, executed by sub-agents under DSH's sandbox and approval policy; don't let a model run arbitrary scripts unattended.
361
- - Sub-agents need appropriate permissions to write files; if permissions are insufficient, the workflow returns the content and the main agent handles persistence.
362
- - **Local transcription**: video transcription (faster-whisper) runs **locally**; audio/subtitles **do not leave your machine**; network requests to the corresponding platforms are made only when fetching video metadata/subtitles/audio.
363
- - This repo contains only the pipeline itself: workflow definitions, parser code, prompt templates, schemas. **It contains no extracted copyrighted content.** Test fixtures are self-written public-domain fables. When you process copyrighted material with this workflow, responsibility for the output is yours.
134
+ For further help, [open a GitHub issue](https://github.com/PensiveFei/deep-read-summarize/issues) with your operating system, DSH and plugin versions, input type, and error details. Remove secrets, private links, and personal data first.
364
135
 
365
- ---
136
+ ## Development and contributing
366
137
 
367
- ## Development
138
+ Work in a **separate project directory**; do not run npm inside a DSH profile:
368
139
 
369
140
  ```bash
370
- npm install
371
- npm run lint # node --check on all JS
372
- npm test # fixture tests (35, all offline)
373
- npm run validate # pre-release validation (including security checks)
141
+ git clone https://github.com/PensiveFei/deep-read-summarize.git
142
+ cd deep-read-summarize
143
+ npm ci
144
+ npm test
145
+ npm run test:node
146
+ npm run lint
147
+ npm run validate
374
148
  ```
375
149
 
376
- `npm publish` automatically runs `prepublishOnly` (tests + lint + security checks) first; any failure blocks the publish.
377
-
378
- See CHANGELOG.md for changes and CONTRIBUTING.md for contribution guidelines.
379
-
380
- ---
381
-
382
- ## Compatibility
383
-
384
- **Verified host: DSH 0.2.0-rc.2** (Node 24). The dependency surface is exactly two things:
385
-
386
- | Surface | Current contract | How this repo uses it |
387
- | --- | --- | --- |
388
- | workflow script hooks | `agent()` / `pipeline()` / `parallel()` / `phase()` / `log()` / `args` | only `agent/parallel/phase/log/args`; `agent()` options limited to `label/phase/schema` (the engine accepts only `label/phase/schema/provider/model` and treats anything else as fatal) |
389
- | JSON Schema subset | `type / properties / required / additionalProperties / items / enum / const / oneOf` plus annotation keys | both `agent()` schemas verified against the host's real `assertObjectJsonSchema` |
390
-
391
- Two contract details that bite:
392
-
393
- - **The script sandbox has no filesystem, network, timers or `require`** — it coordinates subagents; the caller persists the note (see ⚠️ above).
394
- - **Breaking the schema subset terminates the run** rather than degrading: the engine raises `UNSUPPORTED_SCHEMA` and the whole workflow fails. `npm test` carries a regression for this, including a reverse assertion so the checker cannot pass vacuously.
150
+ The project uses parsers for different input types and supports custom parsers. See the [repository](https://github.com/PensiveFei/deep-read-summarize), [CHANGELOG](https://github.com/PensiveFei/deep-read-summarize/blob/main/CHANGELOG.md), and [CONTRIBUTING](https://github.com/PensiveFei/deep-read-summarize/blob/main/CONTRIBUTING.md).
395
151
 
396
- After upgrading DSH, run `npm test` and `npm run test:node`; if something breaks, cross-reference CHANGELOG.md.
152
+ The relevant host contracts were previously verified with DSH `0.2.0-rc.2`. DSH interfaces can change; verify other versions against the actual host and compatibility information.
397
153
 
398
- ---
154
+ ## License and disclaimer
399
155
 
400
- ## License
156
+ Project code is available under the [MIT License](https://github.com/PensiveFei/deep-read-summarize/blob/main/LICENSE). Papers, video, and short quotations cited in examples remain subject to their respective rights and permissions.
401
157
 
402
- MIT — see [LICENSE](LICENSE).
158
+ deep-read-summarize is an independent third-party project. It is not affiliated with, endorsed by, or sponsored by DeepSeek or DeepSeek Harness.