deep-read-summarize 0.5.0 → 0.6.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.env.example +1 -1
- package/CHANGELOG.md +42 -0
- package/README.en.md +91 -335
- package/README.md +93 -336
- package/cordis.patch.yml +4 -0
- package/docs/RELEASE.md +3 -1
- package/index.js +36 -8
- package/lib/options.js +12 -4
- package/lib/read-cache.js +115 -0
- package/lib/workflow-script.txt +150 -17
- package/package.json +1 -1
- package/parsers/_prompt.js +1 -0
- package/schemas/index.js +12 -0
- package/scripts/chunk-balance.js +71 -0
- package/skills/deep-read-summarize/SKILL.md +7 -5
- package/workflow.js +1 -1
package/.env.example
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
# 请在启动 DSH 的那个环境里直接设置(shell 的 export / Windows 的 setx)。
|
|
5
5
|
#
|
|
6
6
|
# 其余配置不是环境变量,而是插件 config 或 workflow 的 options:
|
|
7
|
-
# outputDir / tempDir / minWords / maxChunks / fastMode / transcribe /
|
|
7
|
+
# outputDir / tempDir / minWords / maxChunks / readConcurrency / parallelQuality / readCache / readCacheIdentity / fastMode / transcribe /
|
|
8
8
|
# requireCitations / includeTimestamps / whisperModel / language / device / maxRetries
|
|
9
9
|
|
|
10
10
|
# uv 拉取 CPython 时使用的 GitHub 镜像前缀(只在需要下载 Python 3.12 时才用得上)。
|
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,48 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/).
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [0.6.1] — 2026-11-10
|
|
9
|
+
|
|
10
|
+
修复普通执行路径的结果序列化错误,避免已生成的精读笔记在返回宿主时被丢弃。
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
|
|
14
|
+
- 无缓存复用的成功分支不再返回 `__cacheReads: undefined`:该字段无法通过宿主 workflow 引擎的 JSON 序列化检查(「is not JSON data」),会连已成稿的 `note` 一起被整体丢弃。改为按需展开(仅缓存复用时携带),普通路径的结果不再含内部缓存字段;缓存复用路径返回数组不变。回归测试见 `tests/workflow-regressions.test.js`。
|
|
15
|
+
|
|
16
|
+
### Verification
|
|
17
|
+
|
|
18
|
+
- PR 的七项 CI 检查通过,覆盖 Node.js 18、20、22、24 和 Windows、macOS、Linux。使用 DSH 0.2.2-rc.1 的 workflow 执行器与模拟代理复现原错误,并验证修复后普通路径和缓存路径的返回结果;未进行真实模型端到端调用。
|
|
19
|
+
|
|
20
|
+
### Compatibility
|
|
21
|
+
|
|
22
|
+
- 升级无需更改配置;缓存复用路径保持原有行为。
|
|
23
|
+
|
|
24
|
+
### 贡献者与致谢
|
|
25
|
+
|
|
26
|
+
- 感谢 @Zyx-A 报告、定位并修复结果序列化问题,以及补充回归测试([PR #8](https://github.com/PensiveFei/deep-read-summarize/pull/8))。
|
|
27
|
+
|
|
28
|
+
## [0.6.0] — 2026-11-05
|
|
29
|
+
|
|
30
|
+
新增可配置并发、字符均衡分块与可选的并行原文核验和跨次精读缓存,保留原有默认核验路径。
|
|
31
|
+
|
|
32
|
+
### Added
|
|
33
|
+
|
|
34
|
+
- **精读并发与计时**:`readConcurrency` 可设置 1–12,默认仍为 3;`readTimings` 返回各块耗时、成功与缓存命中状态。提高并发前应观察宿主限流和失败情况。
|
|
35
|
+
- **字符均衡分块**:获取代理可调用随包脚本计算整行字符负载并优先选择接近的段落边界;统计不可用或计划无效时回退按行均衡,均保留全文行号覆盖。字符数只是 token 的近似,超长单行不会拆开。
|
|
36
|
+
- **可选并行独立核验**:`parallelQuality: true` 时多块分别读取原文、检查全部显式引用,另检查全局术语和语言;证据缺失或核验失败不通过。默认使用原有核验路径。
|
|
37
|
+
- **可选跨次精读缓存**:仅插件工具入口支持,默认关闭。开启时必须由调用者声明包含供应商、固定模型版本及推理设置的 `readCacheIdentity`;源文件内容、提示词和身份参与缓存键,仅质量通过后写入,命中后仍进行独立核验。身份无法由宿主自动认证。缓存为本地明文,可能含原文摘录。
|
|
38
|
+
|
|
39
|
+
### Verification and limitations
|
|
40
|
+
|
|
41
|
+
- 回归测试覆盖分块完整性、并发调度、引用证据、缓存失效和失败回退;调度与插件入口测试使用模拟宿主,未测量真实 DSH 模型的延迟或质量,不能据此声称加速比例。
|
|
42
|
+
- 字符统计依赖获取代理执行和返回辅助脚本结果;模型身份由用户保证。源文件在任务前后重新哈希,但无法检测期间短暂发生后恢复的变化。Windows 当前测试环境没有创建符号链接的权限,相关平台测试需由支持的环境补测。
|
|
43
|
+
|
|
44
|
+
### Compatibility
|
|
45
|
+
|
|
46
|
+
- `parallelQuality` 与 `readCache` 默认关闭;`readConcurrency` 默认为 3。旧 workflow 入口不支持跨次精读缓存,其他能力保持可用。
|
|
47
|
+
- 贡献者与致谢:本版无外部 PR、报告或复测贡献需要署名。
|
|
48
|
+
|
|
49
|
+
|
|
8
50
|
## [0.5.0] — 2026-11-01
|
|
9
51
|
|
|
10
52
|
无字幕视频新增逐字稿内容缓存、转写阶段计时和 CPU 线程配置,保留默认精读与原文核验门槛。
|
package/README.en.md
CHANGED
|
@@ -7,396 +7,152 @@
|
|
|
7
7
|
[](https://github.com/PensiveFei/deep-read-summarize/actions/workflows/ci.yml)
|
|
8
8
|
[](https://www.npmjs.com/package/deep-read-summarize)
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
Turn a paper, book, video, or web page into a **Markdown reading note you can keep, edit, and check against the source**.
|
|
11
11
|
|
|
12
|
-
|
|
12
|
+
This third-party plugin runs in [DeepSeek Harness (DSH)](https://github.com/deepseek-ai/deepseek-harness). It attempts to retrieve the source text, reads long material in chunks, and combines the results into a structured note with YAML frontmatter. You can use the note in Obsidian, Logseq, or another Markdown tool.
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
[See real notes](#see-the-notes) · [Get started](#get-started) · [Quality and limitations](#quality-and-limitations) · [Common options](#common-options)
|
|
15
15
|
|
|
16
|
-
|
|
16
|
+
> AI-generated notes can omit or misinterpret information. Use them as a starting point for reading, and check important claims and quotations against the source.
|
|
17
17
|
|
|
18
|
-
|
|
18
|
+
## See the notes
|
|
19
19
|
|
|
20
|
-
|
|
20
|
+
- 📄 [Paper: Emotional Intelligence of Large Language Models](https://github.com/PensiveFei/deep-read-summarize/blob/main/examples/emotional-intelligence.md)
|
|
21
|
+
- 📄 [Paper: Chain-of-Thought Prompting](https://github.com/PensiveFei/deep-read-summarize/blob/main/examples/chain-of-thought.md)
|
|
22
|
+
- 🎬 [Video: Andrej Karpathy's introduction to LLMs](https://github.com/PensiveFei/deep-read-summarize/blob/main/examples/intro-to-llms-video.md)
|
|
21
23
|
|
|
22
|
-
|
|
24
|
+
These are historical notes supplied by the maintainer, in Chinese. The paper notes have records of early project runs; how the video note was generated has not been confirmed. Their model, project version, and runtime were not recorded, so they are not performance claims for the current release. Some quotations and timestamps have been spot-checked, and edits to the public copies are documented in the [examples guide](https://github.com/PensiveFei/deep-read-summarize/blob/main/examples/README.md).
|
|
23
25
|
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
## What it looks like
|
|
27
|
-
|
|
28
|
-
(Images use absolute GitHub URLs so they render on npm too; the PNGs ship with the repository, not inside the npm package.)
|
|
29
|
-
|
|
30
|
-
| Deep-read paper note | Video note metadata |
|
|
31
|
-
| --- | --- |
|
|
32
|
-
|  |  |
|
|
33
|
-
|
|
34
|
-
| Video note action items | In-chat digest |
|
|
35
|
-
| --- | --- |
|
|
36
|
-
|  |  |
|
|
37
|
-
|
|
38
|
-
---
|
|
26
|
+

|
|
39
27
|
|
|
40
28
|
## What it does
|
|
41
29
|
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
- Output is **standard Markdown + YAML frontmatter**: the body is plain CommonMark (no wikilinks, no callouts), so Obsidian (with Dataview), Logseq, Jekyll / Hugo / Astro content folders and any Markdown editor can consume it directly
|
|
49
|
-
|
|
50
|
-
---
|
|
51
|
-
|
|
52
|
-
## Structure
|
|
53
|
-
|
|
54
|
-
```
|
|
55
|
-
Input (link or file path)
|
|
56
|
-
│
|
|
57
|
-
▼
|
|
58
|
-
Parser registry ── book / paper / video / web
|
|
59
|
-
│
|
|
60
|
-
▼
|
|
61
|
-
Wave 1 Fetch content → write temp file → build chunk plan
|
|
62
|
-
│
|
|
63
|
-
▼
|
|
64
|
-
Wave 2 N sub-agents deep-read chunks in parallel (Map)
|
|
65
|
-
│
|
|
66
|
-
▼
|
|
67
|
-
Wave 3 Merge draft + quality check (Reduce)
|
|
68
|
-
│
|
|
69
|
-
▼
|
|
70
|
-
Markdown note
|
|
71
|
-
```
|
|
72
|
-
|
|
73
|
-
Three waves, normally N+3 sub-agent calls (fetch, N chunk reads, draft, and quality check). In `auto` mode, ambiguous PDFs, unfamiliar web pages, and text inputs require one additional lightweight classification call. An explicit `type` skips classification.
|
|
74
|
-
|
|
75
|
-
`auto` routes recognized video links, arXiv paper links, and ebook extensions directly to their parsers; columns, channels, and help pages on those sites still require classification. PDFs are classified as books or papers from their content. Failed classification returns `stage: "detect"` and asks for an explicit type. `options.tempDir` is a temporary root: each call creates its own `run-*` subdirectory, keeping fetched text, audio, and transcripts separate from other tasks.
|
|
76
|
-
|
|
77
|
-
Chunk plans are sorted, duplicate ranges trimmed, and gaps filled. Plans above `maxChunks` merge adjacent ranges instead of discarding the document tail. The returned `coverageComplete` indicates coverage of all declared lines based on a valid `totalLines`; without a line count, known ranges are retained, `qualityPassed` is set to `false`, and the coverage uncertainty is reported.
|
|
78
|
-
|
|
79
|
-
Quality checks combine code and source review. Code validates the supplied scalar/list frontmatter template, required sections, and length (CJK characters plus other words, excluding metadata, code blocks, HTML comments, and link URLs). A separate reviewer reads the current source file and returns per-chunk excerpts and line-bound evidence for verbatim quotations in section 4. A bare `pass: true` cannot override these checks. `qualityChecks` exposes the gates, and `qualityEvidence` exposes the source path and evidence. Source authenticity still depends on the reviewer's actual source reading; uncertainty fails the gate. Empty, rejected, or structurally degraded repairs retain the preceding usable note.
|
|
80
|
-
|
|
81
|
-
Direct workflow options are strict: `minWords` is an integer in 1–100000, `maxChunks` is a positive integer (values above 12 are capped), and `maxRetries` is an integer in 0–3. Invalid values and unknown keys fail before work begins. Plugin profile configuration instead warns per invalid field and uses its default, preserving host startup.
|
|
82
|
-
|
|
83
|
-
```
|
|
84
|
-
parsers/ Parsers for each input type, dispatched by type
|
|
85
|
-
book.js Books: PDF/EPUB/MOBI text extraction, chapter chunking
|
|
86
|
-
paper.js Papers: arXiv/PDF/HTML structure detection
|
|
87
|
-
video.js Video: full transcript (subtitles first; auto-transcribe via scripts/transcribe.js when none)
|
|
88
|
-
web.js Web pages: main-text extraction
|
|
89
|
-
index.js Registry: parser discovery and fallback
|
|
90
|
-
schemas/ JSON Schema for the output contract (injected into the script by workflow.js - single source of truth)
|
|
91
|
-
scripts/ lint (incl. line-ending/whitespace hygiene), security checks, cross-platform transcription bootstrap
|
|
92
|
-
lib/ Small standalone helpers
|
|
93
|
-
cache.js input fingerprint and idempotency records (for callers driving the workflow themselves -- the
|
|
94
|
-
workflow sandbox has no filesystem, so it cannot use this)
|
|
95
|
-
workflow-script.txt the workflow script body (plain text: read and edit it directly, no escaping involved)
|
|
96
|
-
tests/ tests and validation scripts (**not published to npm**, repo only)
|
|
97
|
-
workflow.js the workflow meta and assembly logic (parser inlining, schema injection)
|
|
98
|
-
```
|
|
99
|
-
|
|
100
|
-
> The script body lives in `lib/workflow-script.txt` on purpose: this file was corrupted twice by "escaping on top of escaping". It is plain text now, so reading and editing it needs no encoding step.
|
|
101
|
-
|
|
102
|
-
To override or add a parser, use the package's `custom-parsers/` directory (included in npm packaging), or set the plugin-level `config.customParsersDir` to your own directory. This is not a workflow option. Per-profile directories remain isolated, and rebuilding skill content uses the current registry, including programmatic registrations. Parsers expose `name`, `types`, and `buildPrompt(input, opts)`; registered types are inlined into the workflow and its schema.
|
|
103
|
-
|
|
104
|
-
File parsers bundle their CommonJS modules and literal relative `.js`, `.cjs`, and `.json` dependencies, preserving module constants, helpers, method shorthand, and cycles. Acorn parses JavaScript during script construction and is installed automatically by npm; it does not run in the workflow sandbox. Node built-ins, external packages, and dynamic `require`/`import` are unsupported in sandbox parsers and fail at build time. Programmatically registered functions must be self-contained; use a file parser for module closures.
|
|
105
|
-
|
|
106
|
-
---
|
|
107
|
-
|
|
108
|
-
## Platform support
|
|
109
|
-
|
|
110
|
-
**Windows / macOS / Linux are equally supported** (since 0.3.9). The plugin itself only needs Node (which DSH ships); the extra tools each input type may use are installed per platform:
|
|
111
|
-
|
|
112
|
-
- **Book / paper PDF** → `pdftotext` (poppler): macOS `brew install poppler`, Windows `winget install poppler`, Debian/Ubuntu `apt install poppler-utils`
|
|
113
|
-
- **Bilibili subtitles / audio** → `curl`: already present on Windows 10+, macOS and Linux — nothing to install
|
|
114
|
-
- **YouTube subtitles** → `yt-dlp`: macOS `brew install yt-dlp`, Windows `winget install yt-dlp.yt-dlp`, any platform `pip install -U yt-dlp`
|
|
115
|
-
- **Transcription when there are no subtitles** → `uv` (the script bootstraps a Python 3.12 env and installs faster-whisper): macOS `brew install uv`, Windows `winget install astral-sh.uv`, or the official installer script on macOS / Linux (hand `curl -LsSf https://astral.sh/uv/install.sh` to `sh`)
|
|
116
|
-
|
|
117
|
-
Further notes:
|
|
118
|
-
|
|
119
|
-
- Transcription is the **only** part that needs an extra toolchain, and all three platforms share one Node script and one command: `node scripts/transcribe.js --audio <file> --out <txt>`.
|
|
120
|
-
- Transcription runs on **CPU** by default, so Apple Silicon / Intel Macs and Linux need no CUDA; GPU support needs a CUDA runtime and `options.device: "auto"` will try it and fall back to CPU on failure.
|
|
121
|
-
- Cache location per platform: macOS `~/Library/Caches/deep-read-summarize`, Linux `$XDG_CACHE_HOME/deep-read-summarize`, Windows `%LOCALAPPDATA%\deep-read-summarize`.
|
|
122
|
-
- A **full transcription has been run successfully on real macOS hardware** (since 0.3.12): `uv` bootstraps Python 3.12, installs faster-whisper, downloads the model and writes the transcript — exit code 0. CI's `macos-latest` job separately covers platform path resolution and the whole test suite. If you hit an error on a Mac, please open an issue.
|
|
123
|
-
|
|
124
|
-
---
|
|
125
|
-
## Installation (DSH plugin)
|
|
126
|
-
|
|
127
|
-
deep-read-summarize is a DSH plugin installable via npm or locally:
|
|
128
|
-
|
|
129
|
-
```bash
|
|
130
|
-
# npm install (includes Acorn for parser construction)
|
|
131
|
-
npm install deep-read-summarize
|
|
132
|
-
|
|
133
|
-
# or local install (dsh profile directory)
|
|
134
|
-
pnpm add ./deep-read-summarize-0.3.10.tgz
|
|
135
|
-
# then append to the dsh config's dsh.profile.bundles:
|
|
136
|
-
# - deep-read-summarize
|
|
137
|
-
# restart dsh web (POST /dsh-market/restart)
|
|
138
|
-
```
|
|
139
|
-
|
|
140
|
-
After installation it registers:
|
|
141
|
-
- the `deep_read_summarize` tool when the host provides tools and workflowEngine services; models send only input/type/options
|
|
142
|
-
- the `deep-read-summarize` workflow definition (meta + script), with the existing workflow-tool route retained for older hosts
|
|
143
|
-
- the `deep-read-summarize` skill (`skills/deep-read-summarize/SKILL.md`)
|
|
144
|
-
- four parsers (`parsers/`) and JSON Schemas (`schemas/`)
|
|
145
|
-
|
|
146
|
-
---
|
|
147
|
-
## Quick start (~5 minutes)
|
|
30
|
+
| You provide | The workflow attempts to retrieve | You receive |
|
|
31
|
+
| --- | --- | --- |
|
|
32
|
+
| A book: PDF, EPUB, MOBI | Text and chapters | A reading note organized by topic |
|
|
33
|
+
| A paper: arXiv, PDF, HTML | Paper text and structure | Research question, method, findings, and limitations |
|
|
34
|
+
| A video link | Subtitles first; local transcription when needed and available | A breakdown based on the text obtained |
|
|
35
|
+
| A web page | Page text | Key points, concepts, and source information |
|
|
148
36
|
|
|
149
|
-
|
|
37
|
+
Extraction depends on source accessibility, available tools, the model, and host permissions. Scanned PDFs generally need OCR first; the workflow does not provide automatic OCR. Complex formulas, diagrams, and poor-quality captions or audio may also need manual handling.
|
|
150
38
|
|
|
151
|
-
|
|
152
|
-
# Option A: npm install directly (no clone needed)
|
|
153
|
-
npm install deep-read-summarize
|
|
39
|
+
Notes typically include an overview, structural breakdown, glossary, key quotations, critical analysis, and applications. Their contents are model-generated and may differ between inputs and configurations.
|
|
154
40
|
|
|
155
|
-
|
|
156
|
-
git clone https://github.com/PensiveFei/deep-read-summarize.git
|
|
157
|
-
cd deep-read-summarize
|
|
158
|
-
npm ci # install locked dependencies
|
|
159
|
-
```
|
|
41
|
+
## Get started
|
|
160
42
|
|
|
161
|
-
###
|
|
43
|
+
### 1. Install the plugin
|
|
162
44
|
|
|
163
|
-
|
|
164
|
-
node -e "console.log(require('deep-read-summarize').name)"
|
|
165
|
-
```
|
|
166
|
-
|
|
167
|
-
Seeing `deep-read-summarize` confirms the package loads. Source checkouts can additionally run `npm test` and `npm run test:node`; tests are not included in the npm package.
|
|
168
|
-
|
|
169
|
-
### 3. Feed a demo file with one command
|
|
45
|
+
Install and start DSH Desktop once to initialize its desktop profile. **Quit the application completely**, then use the DSH command bundled with Desktop to install the plugin:
|
|
170
46
|
|
|
171
47
|
```bash
|
|
172
|
-
|
|
48
|
+
dsh plugin --profile desktop add deep-read-summarize
|
|
173
49
|
```
|
|
174
50
|
|
|
175
|
-
|
|
51
|
+
Reopen DSH Desktop when installation finishes. The bundled command may not be on your system PATH; use `resources/runtime/cli/bin/dsh` inside the installation directory (`dsh.cmd` on Windows). Interfaces and commands can change between host versions; see the [official DSH Desktop instructions](https://github.com/deepseek-ai/deepseek-harness/blob/master/apps/desktop/README.md). Do not run `npm install` or `npm update` directly inside a DSH profile directory.
|
|
176
52
|
|
|
177
|
-
|
|
178
|
-
meta: deep-read-summarize
|
|
179
|
-
parsers: book, paper, video, web
|
|
180
|
-
```
|
|
53
|
+
Deep reading also requires a working model and suitable permissions in DSH. Some sources need additional tools: PDF extraction may need `pdftotext`, YouTube captions may need `yt-dlp`, and local transcription of a video without subtitles may require `uv` and a transcription model.
|
|
181
54
|
|
|
182
|
-
###
|
|
55
|
+
### 2. Request a reading note
|
|
183
56
|
|
|
184
|
-
|
|
57
|
+
In a DSH conversation, for example:
|
|
185
58
|
|
|
186
|
-
```
|
|
187
|
-
|
|
59
|
+
```text
|
|
60
|
+
Please read this paper in depth and save a Markdown reading note:
|
|
61
|
+
https://arxiv.org/abs/2307.09042
|
|
188
62
|
```
|
|
189
63
|
|
|
190
|
-
|
|
64
|
+
You can also provide a local file, web page, or video link. If needed, specify an output location, sections to retain, or whether you want timestamps.
|
|
191
65
|
|
|
192
|
-
|
|
66
|
+
Hosts with plugin-tool support register `deep_read_summarize`; older hosts use the bundled skill and workflow entry point. The tool returns the note and a suggested path; **the main agent must write the file**. Check that the file was saved and whether the result reports gaps or quality issues.
|
|
193
67
|
|
|
194
|
-
|
|
195
|
-
|
|
68
|
+
<details>
|
|
69
|
+
<summary>Direct tool-call example</summary>
|
|
196
70
|
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
```jsonc
|
|
71
|
+
```json
|
|
200
72
|
{
|
|
201
|
-
"input": "https://arxiv.org/abs/2307.09042",
|
|
202
|
-
"type": "
|
|
73
|
+
"input": "https://arxiv.org/abs/2307.09042",
|
|
74
|
+
"type": "paper",
|
|
203
75
|
"options": {
|
|
204
|
-
"
|
|
205
|
-
"
|
|
206
|
-
"maxChunks": 4, // chunk limit, 1-12 (default lowered to control sub-agent count and time)
|
|
207
|
-
"transcribe": true, // video: auto local transcription (faster-whisper) when no subtitles; false skips it — see Video
|
|
208
|
-
"whisperModel": "small", // video ASR tier: small | base | medium
|
|
209
|
-
"language": "zh", // video ASR language: use "en" for English-only videos
|
|
210
|
-
"device": "cpu", // ASR device: cpu (default) | auto | cuda; auto/cuda needs a CUDA runtime and falls back to cpu
|
|
211
|
-
"cpuThreads": 0, // 0 keeps library defaults; 1-64 allows hardware-specific tuning
|
|
212
|
-
"transcriptCache": true, // reuse a verified local transcript for matching audio and settings
|
|
213
|
-
"maxRetries": 0, // quality-check retries (default 0 = no retry; retrying is noticeably slower)
|
|
214
|
-
"frontmatter": "obsidian", // key set: obsidian (default, for Dataview) | generic (drops the keys Hugo reserves) | minimal
|
|
215
|
-
"requireCitations": true, // whether key conclusions must carry citations
|
|
216
|
-
"includeTimestamps": false,
|
|
217
|
-
"outputDir": "./output", // note output directory (can point at your Obsidian vault or a static-site content folder)
|
|
218
|
-
"tempDir": "./.tmp" // temp file directory
|
|
76
|
+
"maxChunks": 4,
|
|
77
|
+
"outputDir": "./output"
|
|
219
78
|
}
|
|
220
79
|
}
|
|
221
80
|
```
|
|
222
81
|
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
| Type | input |
|
|
226
|
-
|------|-------|
|
|
227
|
-
| Paper | `https://arxiv.org/abs/2307.09042` |
|
|
228
|
-
| Book | local path, e.g. `~/books/xxx.pdf` |
|
|
229
|
-
| Video | `https://youtube.com/watch?v=xxx` |
|
|
230
|
-
| Web page | `https://example.com/article` |
|
|
231
|
-
|
|
232
|
-
Notes are written to `options.outputDir` (default `./output`, can point at your Obsidian vault or a static-site content folder), with the filename taken from the content title.
|
|
233
|
-
|
|
234
|
-
### Frontmatter profiles (`options.frontmatter`)
|
|
235
|
-
|
|
236
|
-
The default is `obsidian`, with the same keys as earlier versions and safely quoted scalar/list values. Profiles exist because `type`, `aliases` and `url` are **reserved keys in Hugo** with different meanings -- dropping an Obsidian note into Hugo changes how the page renders, rather than adding a few harmless fields.
|
|
237
|
-
|
|
238
|
-
| Key | `obsidian` (default) | `generic` | `minimal` |
|
|
239
|
-
|---|---|---|---|
|
|
240
|
-
| `title` | yes | yes | yes |
|
|
241
|
-
| `aliases` | yes | -- | -- |
|
|
242
|
-
| `author` / `year` | yes | yes | -- |
|
|
243
|
-
| `type` | yes | -- | -- |
|
|
244
|
-
| `docType` | -- | yes | -- |
|
|
245
|
-
| `url` | yes | -- | -- |
|
|
246
|
-
| `sourceUrl` | -- | yes | yes |
|
|
247
|
-
| `tags` | yes | yes | yes |
|
|
248
|
-
| `created` | yes | -- | -- |
|
|
249
|
-
| `date` | -- | yes | yes |
|
|
250
|
-
| `status` | yes | yes | -- |
|
|
251
|
-
|
|
252
|
-
Why `generic` renames things:
|
|
253
|
-
|
|
254
|
-
- `type` in Hugo selects the **layout** (`type: video` makes it look for `layouts/video/`)
|
|
255
|
-
- `aliases` in Hugo is **a list of old URLs that redirect to this page**
|
|
256
|
-
- `url` in Hugo **overrides the page's own path** -- the source link would become the article's permalink
|
|
257
|
-
- `date` is the time key both Hugo and Jekyll understand; `created` is just a custom field
|
|
258
|
-
|
|
259
|
-
An unknown value is fatal rather than silently ignored, matching how `type` is handled.
|
|
260
|
-
|
|
261
|
-
### Video: full transcript (subtitles first, auto-transcription when none)
|
|
262
|
-
|
|
263
|
-
Video text extraction is a **unified pipeline** (the old "three tiers" are gone): the goal is to get the **full transcript**, then deep-read it.
|
|
264
|
-
|
|
265
|
-
> **⚠️ To preserve deep-reading quality, processing may take noticeably longer (a deliberate trade-off, not a hang)**: a **subtitle-less** video is transcribed first (roughly 0.5–2× the video length on CPU) before deep reading, so such videos are **much slower than subtitled videos**; in exchange you get a **full transcript + real deep reading**. **Subtitled** videos remain seconds-fast. Please do not mistake the slower transcription for a hang.
|
|
266
|
-
|
|
267
|
-
1. **Platform subtitles available** (Bilibili's official API AI subtitles / YouTube CC) → **use them directly** (= full text, fastest, zero dependencies).
|
|
268
|
-
2. **No public subtitles** → use the bundled transcription `scripts/transcribe.js` (faster-whisper small / int8 / VAD / Chinese) to get the full text.
|
|
269
|
-
3. **Neither works** → **degrade gracefully**: prompt the user to supply a transcript manually, or fall back to `desc` as background; **never block**.
|
|
270
|
-
|
|
271
|
-
`options.transcribe`: **default `true`** (auto-transcribe when no subtitles); set `false` to skip transcription (use subtitles/desc or degrade).
|
|
272
|
-
|
|
273
|
-
**Transcription toolchain**: `uv` provisions **Python 3.12** and installs **`faster-whisper==1.2.1`** from the Tsinghua mirror; models use `HF_ENDPOINT=https://hf-mirror.com` and the local cache. Healthy cached environments run directly without requiring uv or a separate Python import probe. Actual interpreter and dependency versions are recorded in the cache's `runtime.json`; Python patch releases, uv, and all transitive dependencies are not fully pinned. PyAV decodes audio without a separate ffmpeg. All three platforms use `node scripts/transcribe.js --audio <file> --out <txt>`. CPU is the default; `options.device: "auto"` can try a supplied CUDA runtime and falls back to CPU. Cache roots are `%LOCALAPPDATA%\deep-read-summarize` on Windows, `~/Library/Caches/deep-read-summarize` on macOS, and `$XDG_CACHE_HOME/deep-read-summarize` on Linux (falling back to `~/.cache`). Python provisioning tries a mirror and then direct GitHub; `DRS_UV_PYTHON_MIRROR` overrides it, with an empty value selecting direct downloads.
|
|
274
|
-
|
|
275
|
-
> ⚠️ **The first transcription downloads the model (small ≈ 484 MB, via hf-mirror, not GitHub) and may take a few minutes**. Later runs reuse model files; new audio still requires full inference. Only a verified transcript cache hit skips inference. **Only subtitle-less videos trigger transcription** (subtitled videos use their subtitles directly).
|
|
276
|
-
|
|
277
|
-
**Transcript cache and threads**: `options.transcriptCache` defaults to true. Keys include the audio SHA-256, managed local model snapshots, installed dependencies, and decoding settings. Canonical segments retain real start/end times, so changing timestamp presentation does not repeat inference. Changed content/configuration/models/dependencies, custom local model directories, unknown environments, corrupt entries and expired entries bypass reuse. Cache lookup uses the existing local snapshot and does not check for upstream model updates online. Entries live under `transcripts-v1` in the cache root: 8 MiB per entry, 128 MiB total, 30-day validity, pruned on writes. Audio is not stored, and reading/source review still runs.
|
|
278
|
-
|
|
279
|
-
`options.cpuThreads` accepts integers 0–64. The default `0` preserves the library's CPU thread policy. Compare 4/6/8 on the same audio; more threads are not necessarily faster. CLI controls:
|
|
280
|
-
|
|
281
|
-
```bash
|
|
282
|
-
node scripts/transcribe.js --audio sample.m4a --out transcript.txt --cpu-threads 8
|
|
283
|
-
node scripts/transcribe.js --audio sample.m4a --out transcript.txt --refresh-cache
|
|
284
|
-
node scripts/transcribe.js --audio sample.m4a --out transcript.txt --no-cache
|
|
285
|
-
node scripts/transcribe.js --clear-cache
|
|
286
|
-
```
|
|
287
|
-
|
|
288
|
-
`--clear-cache` removes only transcripts, retaining models and Python. `--refresh-cache` forces inference and updates the cache; `--no-cache` disables reads and writes. Successful calls print one `metrics` JSON line with `cacheHit`, actual device, audio duration, and milliseconds for hashing, environment preparation, dependency installation, Python, publication and total time. Misses also include import, model loading, preparation (including decoding/VAD), and complete lazy-segment iteration. Hits do not reuse previous inference timings.
|
|
82
|
+
`type` can be `auto`, `book`, `paper`, `video`, or `web`. Specifying it can skip classification for ambiguous inputs. The tool-call interface depends on the DSH host; asking in conversation is also supported.
|
|
289
83
|
|
|
290
|
-
|
|
84
|
+
</details>
|
|
291
85
|
|
|
292
|
-
|
|
86
|
+
## Quality and limitations
|
|
293
87
|
|
|
294
|
-
|
|
88
|
+
The workflow normalizes chunk ranges and checks coverage of known source line numbers. After drafting, code checks format and length, and an independent review agent reads the obtained text to check chunk coverage and key quotations. Results are available through `qualityPassed`, `qualityChecks`, and `qualityEvidence`.
|
|
295
89
|
|
|
296
|
-
|
|
90
|
+
These checks have limits:
|
|
297
91
|
|
|
298
|
-
|
|
92
|
+
- **`qualityPassed` does not prove factual accuracy.** Coverage and source authenticity still depend on the reviewing agent reading and evaluating the source. Check important claims, numbers, and quotations yourself.
|
|
93
|
+
- **Retrieving the full text does not guarantee understanding it.** Complex tables, formulas, images, or extraction errors can affect the result; incomplete sources or failed chunks are reported as gaps.
|
|
94
|
+
- **Captions and transcripts can differ from audio.** Automatic captions and speech recognition can omit words or mishear numbers and negations; checking the note against a transcript does not prove the transcript is accurate.
|
|
95
|
+
- **Video timestamps cannot be inferred reliably.** Locations depend on whether the obtained subtitles or transcript contain time information.
|
|
299
96
|
|
|
300
|
-
##
|
|
97
|
+
## Common options
|
|
301
98
|
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
- **Chunking structure**: different types chunk differently (chapter/topic/paragraph), affecting **output structure** and coverage.
|
|
319
|
-
|
|
320
|
-
### Hardware
|
|
321
|
-
- **GPU vs CPU**: gains depend on hardware, runtime libraries, model and audio. A visible GPU does not prove CUDA inference works; compare actual time and transcript accuracy.
|
|
322
|
-
|
|
323
|
-
### Other
|
|
324
|
-
- **First run vs reuse**: the first run downloads the model (≈ 484 MB); later runs reuse the cache.
|
|
325
|
-
- **Token/cost**: chunk count, length, and citations drive token usage; `fastMode`/`maxChunks` save tokens.
|
|
326
|
-
- **Citation/fabrication**: transcription or OCR errors may be quoted as "original text"; **re-check key citations**.
|
|
327
|
-
- **Boundaries**: pure BGM, distorted audio → empty/poor transcription; no subtitles and transcription unavailable → degrade to a prompt for a manual transcript.
|
|
328
|
-
|
|
329
|
-
---
|
|
330
|
-
|
|
331
|
-
## How failures are handled
|
|
332
|
-
|
|
333
|
-
Two classes:
|
|
334
|
-
|
|
335
|
-
- **Configuration errors** (missing input, invalid type, malformed options): throw immediately and abort — no half-baked output.
|
|
336
|
-
- **Content problems** (fetch failed, a chunk's deep-read failed): return `{ ok: false, stage, fatal: false }` and let the caller decide. A failed chunk is skipped and marked as a gap in the draft; the whole run is not aborted.
|
|
337
|
-
|
|
338
|
-
---
|
|
339
|
-
|
|
340
|
-
## Faster runs with the same quality gates
|
|
341
|
-
|
|
342
|
-
- Prefer `deep_read_summarize` on supported hosts: the model submits parameters instead of reproducing the entire workflow script. Older hosts retain the existing route.
|
|
343
|
-
- All known source lines remain covered. Up to three reading tasks run concurrently, and each free slot immediately starts another task. Line-balanced chunks avoid one oversized task holding up a batch.
|
|
344
|
-
- Independent source review remains enabled, with duplicate full Map notes removed from review contexts. Missing line counts, failed reads, and description-only video sources cannot be fixed by rewriting the draft, so those states no longer consume full repair loops.
|
|
345
|
-
- Video `textSource: "desc"` is explicitly degraded and cannot pass as a complete deep read. Bilibili `p` selects the matching CID, part title, and duration.
|
|
346
|
-
- Returned `timings` reports stage durations in milliseconds. Compare the same sources, model, output size, and quality gates; scheduling simulations are not real LLM speedup measurements.
|
|
99
|
+
| Option | Default | Purpose |
|
|
100
|
+
| --- | --- | --- |
|
|
101
|
+
| `minWords` | `2500` | Minimum length; the check counts Chinese characters and words in other text |
|
|
102
|
+
| `fastMode` | `false` | Omit the critical analysis, personal applications, and further-reading sections |
|
|
103
|
+
| `maxChunks` | `4` | Chunk-count limit, capped at 12; this is not the concurrency setting |
|
|
104
|
+
| `readConcurrency` | `3` | Maximum concurrent chunk reads; higher concurrency is not always faster |
|
|
105
|
+
| `maxRetries` | `0` | Repair attempts after a failed quality check; by default, a successfully drafted note receives one review and no repair |
|
|
106
|
+
| `outputDir` | `./output` | Suggested note directory; the main agent performs the write |
|
|
107
|
+
| `requireCitations` | `true` | Require verifiable source information for key quotations |
|
|
108
|
+
| `includeTimestamps` | `false` | Request timestamps for video notes |
|
|
109
|
+
| `transcribe` | `true` | Attempt local transcription when a video has no usable subtitles |
|
|
110
|
+
| `whisperModel` / `language` | `small` / `zh` | Choose a transcription model and language; set `en` for English when appropriate |
|
|
111
|
+
| `device` / `cpuThreads` | `cpu` / `0` | Select the transcription device and CPU thread policy |
|
|
112
|
+
| `transcriptCache` | `true` | Reuse validated transcripts when the audio and configuration match |
|
|
113
|
+
| `parallelQuality` | `false` | Opt in to parallel source review; the original review path remains the default |
|
|
114
|
+
| `readCache` | `false` | Optional cross-run chunk cache; supported by the plugin tool only |
|
|
347
115
|
|
|
348
|
-
|
|
116
|
+
To enable `readCache`, also set `readCacheIdentity` to include the provider, pinned model version, and output-affecting settings; change it whenever those change. The caller supplies this identity, and the host cannot verify it automatically. Cache entries are local plaintext and may contain source excerpts; cache hits still receive independent source review.
|
|
349
117
|
|
|
350
|
-
|
|
118
|
+
Other options include `frontmatter` (`obsidian`, `generic`, or `minimal`) and `tempDir`. See the [advanced reference](https://github.com/PensiveFei/deep-read-summarize/blob/main/docs/README-reference.en.md) for detailed options, supported formats, and transcription and cache behavior.
|
|
351
119
|
|
|
352
|
-
|
|
120
|
+
## Time, cost, and privacy
|
|
353
121
|
|
|
354
|
-
|
|
122
|
+
- **Time** depends on content length, retrieval, chunk count, model, and host concurrency. A video without subtitles needs extra transcription; the first run may also download a transcription model.
|
|
123
|
+
- **Cost** depends on your model and input/output usage; the project cannot promise fixed time or cost.
|
|
124
|
+
- **Privacy:** speech transcription runs locally, but deep reading uses your configured DSH model. With a cloud model, source text, subtitles, notes, or related context may be sent to its provider; check its data-handling terms first.
|
|
125
|
+
- **Local files:** temporary source text, transcripts, and optional caches may remain on your device. Check directory permissions and cleanup before processing sensitive material.
|
|
355
126
|
|
|
356
|
-
|
|
127
|
+
## Troubleshooting
|
|
357
128
|
|
|
358
|
-
|
|
129
|
+
- **Plugin unavailable:** check the DSH profile, restart Desktop after installation, and check host-version compatibility.
|
|
130
|
+
- **Fetch failure or missing sections:** check source accessibility and required tools; inspect the returned stage, source status, and failed chunks.
|
|
131
|
+
- **Only a video description is available:** a complete subtitle or transcript has not been confirmed. Check subtitle and transcription availability, or provide a transcript.
|
|
132
|
+
- **Quality check failed:** inspect `qualityIssues` and `qualityEvidence` and review the source as needed. Do not treat an unverified note as a complete account.
|
|
359
133
|
|
|
360
|
-
|
|
361
|
-
- Sub-agents need appropriate permissions to write files; if permissions are insufficient, the workflow returns the content and the main agent handles persistence.
|
|
362
|
-
- **Local transcription**: video transcription (faster-whisper) runs **locally**; audio/subtitles **do not leave your machine**; network requests to the corresponding platforms are made only when fetching video metadata/subtitles/audio.
|
|
363
|
-
- This repo contains only the pipeline itself: workflow definitions, parser code, prompt templates, schemas. **It contains no extracted copyrighted content.** Test fixtures are self-written public-domain fables. When you process copyrighted material with this workflow, responsibility for the output is yours.
|
|
134
|
+
For further help, [open a GitHub issue](https://github.com/PensiveFei/deep-read-summarize/issues) with your operating system, DSH and plugin versions, input type, and error details. Remove secrets, private links, and personal data first.
|
|
364
135
|
|
|
365
|
-
|
|
136
|
+
## Development and contributing
|
|
366
137
|
|
|
367
|
-
|
|
138
|
+
Work in a **separate project directory**; do not run npm inside a DSH profile:
|
|
368
139
|
|
|
369
140
|
```bash
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
npm
|
|
373
|
-
npm
|
|
141
|
+
git clone https://github.com/PensiveFei/deep-read-summarize.git
|
|
142
|
+
cd deep-read-summarize
|
|
143
|
+
npm ci
|
|
144
|
+
npm test
|
|
145
|
+
npm run test:node
|
|
146
|
+
npm run lint
|
|
147
|
+
npm run validate
|
|
374
148
|
```
|
|
375
149
|
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
See CHANGELOG.md for changes and CONTRIBUTING.md for contribution guidelines.
|
|
379
|
-
|
|
380
|
-
---
|
|
381
|
-
|
|
382
|
-
## Compatibility
|
|
383
|
-
|
|
384
|
-
**Verified host: DSH 0.2.0-rc.2** (Node 24). The dependency surface is exactly two things:
|
|
385
|
-
|
|
386
|
-
| Surface | Current contract | How this repo uses it |
|
|
387
|
-
| --- | --- | --- |
|
|
388
|
-
| workflow script hooks | `agent()` / `pipeline()` / `parallel()` / `phase()` / `log()` / `args` | only `agent/parallel/phase/log/args`; `agent()` options limited to `label/phase/schema` (the engine accepts only `label/phase/schema/provider/model` and treats anything else as fatal) |
|
|
389
|
-
| JSON Schema subset | `type / properties / required / additionalProperties / items / enum / const / oneOf` plus annotation keys | both `agent()` schemas verified against the host's real `assertObjectJsonSchema` |
|
|
390
|
-
|
|
391
|
-
Two contract details that bite:
|
|
392
|
-
|
|
393
|
-
- **The script sandbox has no filesystem, network, timers or `require`** — it coordinates subagents; the caller persists the note (see ⚠️ above).
|
|
394
|
-
- **Breaking the schema subset terminates the run** rather than degrading: the engine raises `UNSUPPORTED_SCHEMA` and the whole workflow fails. `npm test` carries a regression for this, including a reverse assertion so the checker cannot pass vacuously.
|
|
150
|
+
The project uses parsers for different input types and supports custom parsers. See the [repository](https://github.com/PensiveFei/deep-read-summarize), [CHANGELOG](https://github.com/PensiveFei/deep-read-summarize/blob/main/CHANGELOG.md), and [CONTRIBUTING](https://github.com/PensiveFei/deep-read-summarize/blob/main/CONTRIBUTING.md).
|
|
395
151
|
|
|
396
|
-
|
|
152
|
+
The relevant host contracts were previously verified with DSH `0.2.0-rc.2`. DSH interfaces can change; verify other versions against the actual host and compatibility information.
|
|
397
153
|
|
|
398
|
-
|
|
154
|
+
## License and disclaimer
|
|
399
155
|
|
|
400
|
-
|
|
156
|
+
Project code is available under the [MIT License](https://github.com/PensiveFei/deep-read-summarize/blob/main/LICENSE). Papers, video, and short quotations cited in examples remain subject to their respective rights and permissions.
|
|
401
157
|
|
|
402
|
-
|
|
158
|
+
deep-read-summarize is an independent third-party project. It is not affiliated with, endorsed by, or sponsored by DeepSeek or DeepSeek Harness.
|