deep-read-summarize 0.3.4 → 0.3.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +40 -1
- package/README.en.md +310 -0
- package/README.md +90 -10
- package/index.js +5 -3
- package/package.json +4 -3
- package/parsers/video.js +35 -12
- package/scripts/transcribe.ps1 +68 -0
- package/skills/deep-read-summarize/SKILL.md +16 -4
- package/tests/run-tests.js +162 -0
- package/workflow.js +2 -2
package/CHANGELOG.md
CHANGED
|
@@ -3,7 +3,46 @@
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
The format is based on [Keep a Changelog](https://keepachangelog.com/).
|
|
5
5
|
|
|
6
|
-
## [0.3.
|
|
6
|
+
## [0.3.7] — 2026-09-09
|
|
7
|
+
|
|
8
|
+
针对 **DSH 0.1.2-rc.1** 的一轮兼容性核查与打磨(对使用者无破坏性变更)。
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
|
|
12
|
+
- **笔记落盘步骤缺失(最容易丢数据的一条)**:SKILL.md 写「默认写到 `./output/`」,但 DSH 的 workflow 契约规定脚本只能协调子代理——**沙箱内没有文件系统**,脚本实际只返回 `{ filePath, note }`。照原文档执行,模型会以为文件已经写好,笔记只存在于返回值里。现在 SKILL.md 新增明确的第 8 步「用 `write` 工具把 `note` 写入 `filePath`」,两版 README 同步说明。
|
|
13
|
+
- **`npm run test:node` 在 Node 24 上直接失败**:`node --test tests/` 在该版本会把 `tests/` 当模块解析(`MODULE_NOT_FOUND`)。改为 `node --test`(自动发现)。
|
|
14
|
+
- **CI 从不运行 `tests/index.test.js`**:里面正是 0.3.3 事故(入口缺 `apply()` → 宿主拒绝启动)的 Cordis 契约回归测试,却因为只在 `node --test` 下跑而从未被执行。CI 新增该步骤,矩阵补上 **Node 24**(宿主实际运行版本)。
|
|
15
|
+
|
|
16
|
+
### Changed
|
|
17
|
+
|
|
18
|
+
- **npm 包不再携带 docs 截图**:`files` 由整个 `docs/` 收窄为两份 RELEASE 文档。0.3.6 发布之后仓库才提交的 4 张 showcase 图(合计约 1.14MB)会让下次发版的 tarball 从 99KB 涨到 1.1MB;现在回到约 100KB。图片仍随仓库分发,README 改用 GitHub 绝对地址引用(npm 页面照常显示),顺带把一直没有被任何文档引用的演示图补进 README。
|
|
19
|
+
- **技能 `source` 改为 `bundled`**:该字段是技能的「来源桶」(`runtime`/`bundled`/`user-dsh`…),此前填的是自由描述 `plugin:deep-read-summarize`。
|
|
20
|
+
- README/README.en.md 的「兼容性」节改写为:已验证宿主(DSH 0.1.2-rc.1)+ 依赖面表格 + 两条契约细节(沙箱无文件系统;违反 schema 子集是**终止**而非降级)。
|
|
21
|
+
- `.gitignore` 增加 `*.tgz`(本地 pack 产物不再出现在 `git status`)。
|
|
22
|
+
|
|
23
|
+
### Added
|
|
24
|
+
|
|
25
|
+
- **workflow 契约回归测试(零依赖)**:schema 子集递归校验(含反向断言,确保校验器不会空放行)、`agent()` 选项白名单、沙箱禁用 API(`require`/`fs`/`process`/`fetch`/定时器)扫描、脚本四解析器自包含、技能内容包含 meta + script + args 与落盘步骤、插件 Cordis 契约 + 技能登记行为。测试 28 → **35**。
|
|
26
|
+
|
|
27
|
+
### Compatibility
|
|
28
|
+
|
|
29
|
+
- 实测证据(0.1.2-rc.1):两个 `agent()` schema 用宿主真实的 `assertObjectJsonSchema` 校验**通过**,对照用例 `minLength` 被拒(证明校验器有效、不是空放行);脚本只使用 `agent/parallel/phase/log/args`;`agent()` 失败返回 `null` 已被脚本正确处理。
|
|
30
|
+
- 未改动 workflow 脚本本身与解析器行为;本次修复集中在「落盘契约、测试与 CI、打包体积、文档准确性」。
|
|
31
|
+
|
|
32
|
+
## [0.3.6] — 2026-08-28
|
|
33
|
+
|
|
34
|
+
### Added
|
|
35
|
+
- **统一视频转写工具 `scripts/transcribe.ps1`(关键)**:自举 faster-whisper(`uv` 建 Python 3.11 环境 + 清华镜像安装 + `HF_ENDPOINT=https://hf-mirror.com` 下模型并缓存),small/int8/VAD;**本机与用户使用一致**(只依赖环境变量与标准路径),无需单独 ffmpeg(faster-whisper 内置 PyAV 解码)。
|
|
36
|
+
|
|
37
|
+
### Fixed
|
|
38
|
+
- **视频解析器 yt-dlp 防卡死**:未装 yt-dlp 时**绝不下载 exe 二进制**(GitHub 直连易卡死),改用 `winget install yt-dlp.yt-dlp` 或 `pip install -U yt-dlp -i 镜像`;并加 `--socket-timeout 15 --retries 3` 防超时。
|
|
39
|
+
- **workflow 精读防超时中止**:`maxChunks` 默认 6→4、`maxRetries` 默认 1→0;波次2「并行精读」改为**分批并行**(每批 3 个块),降低子代理数与单次调用时长。
|
|
40
|
+
|
|
41
|
+
### Changed
|
|
42
|
+
- **`parsers/video.js` 统一流水线(去三档)**:目标 = 拿到**完整逐字稿**再精读。① 有平台字幕(B站 AI 字幕 / YouTube CC / yt-dlp CC)→ 直接用字幕(快、零依赖);② 无公开字幕 → 用 `scripts/transcribe.ps1` 转写得到全文;③ 都不行 → 降级提示人工转写文本,或退回 desc 作背景;**绝不阻塞、绝不自动装重依赖**。
|
|
43
|
+
- **`options.transcribe` 默认 true**(无字幕自动走本地转写),`false` 则跳过转写。
|
|
44
|
+
- **textSource** 标注实际来源:`subtitle / transcription / desc / manual`;来源信息**透出在运行日志 `[source] …` 与返回结果 `textSource`,不写进笔记**(保持笔记美观)。
|
|
45
|
+
- `SKILL.md` / `README.md` 同步为「完整逐字稿(字幕优先,无字幕自动转写)」策略与效果预估。
|
|
7
46
|
|
|
8
47
|
### Fixed
|
|
9
48
|
- **DSH rc7(Cordis)兼容修复**:插件入口改为标准 Cordis 插件契约(导出 `name` + `inject: ['skills']` + `apply(ctx)`),apply 时将自带 SKILL.md 注册为运行时技能(含 workflow meta + 自包含 script + args 示例),修复 0.3.3 在 rc7 上「invalid plugin / 插件树加载失败 / harness 无法启动」的问题(旧入口只导出 load/workflow/parsers/schemas,无 apply)。
|
package/README.en.md
ADDED
|
@@ -0,0 +1,310 @@
|
|
|
1
|
+
# deep-read-summarize
|
|
2
|
+
|
|
3
|
+
[](https://github.com/PensiveFei/deep-read-summarize/blob/main/LICENSE)
|
|
4
|
+
[](https://github.com/PensiveFei/deep-read-summarize/releases)
|
|
5
|
+
[](https://github.com/PensiveFei/deep-read-summarize/actions/workflows/ci.yml)
|
|
6
|
+
[](https://www.npmjs.com/package/deep-read-summarize)
|
|
7
|
+
|
|
8
|
+
> **Disclaimer**: This is an **unofficial third-party tool**. It is not affiliated with, endorsed by, or sponsored by DeepSeek or the DeepSeek Harness project. "DeepSeek" and "DeepSeek Harness" are trademarks of their respective owners. This project only orchestrates the workflow tool available in your DSH environment; it does not redistribute any DeepSeek software.
|
|
9
|
+
|
|
10
|
+
> **⚠️ Most importantly: this is an "assisted deep-reading" tool, not a substitute for your own learning.** It helps you quickly extract, summarize, and structure what you read, but **real understanding, thinking, and critique must still come from you**. Always treat the original source as authoritative — do not mistake the generated notes for the learning itself.
|
|
11
|
+
|
|
12
|
+
A deep-reading workflow for DSH (DeepSeek Harness): feed it a book, a paper, a video link, or a web page, and it produces a structured Obsidian note.
|
|
13
|
+
|
|
14
|
+
Content is split into chunks, read in parallel by multiple sub-agents, then merged into a Markdown file with YAML frontmatter. Key conclusions carry citations to the original source, and a quality check runs before the final draft.
|
|
15
|
+
|
|
16
|
+
> If you run into problems, or see how it could be better, **feel free to open a GitHub issue**.
|
|
17
|
+
|
|
18
|
+
> Note: DSH is currently a developer preview and its interfaces may change. This repo targets a specific version of the workflow-tool semantics — see [Compatibility](#compatibility).
|
|
19
|
+
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
## What it looks like
|
|
23
|
+
|
|
24
|
+
(Images use absolute GitHub URLs so they render on npm too; the PNGs ship with the repository, not inside the npm package.)
|
|
25
|
+
|
|
26
|
+
| Deep-read paper note | Video note metadata |
|
|
27
|
+
| --- | --- |
|
|
28
|
+
|  |  |
|
|
29
|
+
|
|
30
|
+
| Video note action items | In-chat digest |
|
|
31
|
+
| --- | --- |
|
|
32
|
+
|  |  |
|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## What it does
|
|
37
|
+
|
|
38
|
+
- Books (PDF/EPUB/MOBI), papers (arXiv/PDF/HTML), videos (**subtitles first, auto-transcription when there are none** — produces a full transcript for deep reading), web pages
|
|
39
|
+
- Long content is chunked and read in parallel by sub-agents, then merged
|
|
40
|
+
- Sub-task outputs are constrained by JSON Schema; invalid outputs are retried automatically
|
|
41
|
+
- Key citations must carry page/chapter/paragraph references to reduce fabrication
|
|
42
|
+
- Configuration errors abort immediately; a chunk that fails to parse is skipped and marked as a gap
|
|
43
|
+
- Output drops straight into Obsidian and works with Dataview
|
|
44
|
+
|
|
45
|
+
---
|
|
46
|
+
|
|
47
|
+
## Structure
|
|
48
|
+
|
|
49
|
+
```
|
|
50
|
+
Input (link or file path)
|
|
51
|
+
│
|
|
52
|
+
▼
|
|
53
|
+
Parser registry ── book / paper / video / web
|
|
54
|
+
│
|
|
55
|
+
▼
|
|
56
|
+
Wave 1 Fetch content → write temp file → build chunk plan
|
|
57
|
+
│
|
|
58
|
+
▼
|
|
59
|
+
Wave 2 N sub-agents deep-read chunks in parallel (Map)
|
|
60
|
+
│
|
|
61
|
+
▼
|
|
62
|
+
Wave 3 Merge draft + quality check (Reduce)
|
|
63
|
+
│
|
|
64
|
+
▼
|
|
65
|
+
Obsidian note
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Three waves, roughly N+2 sub-agents in total, where N is the number of chunks.
|
|
69
|
+
|
|
70
|
+
```
|
|
71
|
+
parsers/ Parsers for each input type, dispatched by type
|
|
72
|
+
book.js Books: PDF/EPUB/MOBI text extraction, chapter chunking
|
|
73
|
+
paper.js Papers: arXiv/PDF/HTML structure detection
|
|
74
|
+
video.js Video: full transcript (subtitles first; auto-transcribe via scripts/transcribe.ps1 when none)
|
|
75
|
+
web.js Web pages: main-text extraction
|
|
76
|
+
index.js Registry: parser discovery and fallback
|
|
77
|
+
schemas/ JSON Schema for sub-task outputs
|
|
78
|
+
scripts/ lint, security checks
|
|
79
|
+
tests/ fixture tests and validation scripts
|
|
80
|
+
workflow.js The workflow script itself (meta + script)
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
To change how an input type is handled, drop a parser with the same interface into `custom-parsers/`; a type with the same name overrides the built-in implementation. The interface has only three fields: `name`, `types`, `buildPrompt(input, opts)`.
|
|
84
|
+
|
|
85
|
+
---
|
|
86
|
+
|
|
87
|
+
## dsh.so ecosystem
|
|
88
|
+
|
|
89
|
+
This repo carries the `dsh-plugin` topic and the `dsh` field in `package.json`, so it can be indexed by the dsh.so registry.
|
|
90
|
+
|
|
91
|
+
dsh.so listing requirements: a public repo, the `dsh-plugin` topic (or the dsh field), README install instructions, and an SPDX license (MIT).
|
|
92
|
+
After submission it is marked **Declared** (self-declared compatibility); if you actually use it and verify compatibility, report your results in the official Discussions — that is the only way to be promoted to **Verified**.
|
|
93
|
+
|
|
94
|
+
---
|
|
95
|
+
## Installation (DSH plugin)
|
|
96
|
+
|
|
97
|
+
deep-read-summarize is a DSH plugin installable via npm, the dsh.so ecosystem, or locally:
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
# npm install (published to the registry, no third-party dependencies)
|
|
101
|
+
npm install deep-read-summarize
|
|
102
|
+
|
|
103
|
+
# or local install (dsh profile directory)
|
|
104
|
+
pnpm add ./deep-read-summarize-0.3.6.tgz
|
|
105
|
+
# then append to the dsh config's dsh.profile.bundles:
|
|
106
|
+
# - deep-read-summarize
|
|
107
|
+
# restart dsh web (POST /dsh-market/restart)
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
After installation it registers:
|
|
111
|
+
- the `deep-read-summarize` workflow (meta + script)
|
|
112
|
+
- the `deep-read-summarize` skill (`skills/deep-read-summarize/SKILL.md`)
|
|
113
|
+
- four parsers (`parsers/`) and JSON Schemas (`schemas/`)
|
|
114
|
+
|
|
115
|
+
---
|
|
116
|
+
## Quick start (~5 minutes)
|
|
117
|
+
|
|
118
|
+
### 1. Install
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
# Option A: npm install directly (no clone needed)
|
|
122
|
+
npm install deep-read-summarize
|
|
123
|
+
|
|
124
|
+
# Option B: develop from source
|
|
125
|
+
git clone https://github.com/<your-org>/deep-read-summarize.git
|
|
126
|
+
cd deep-read-summarize
|
|
127
|
+
npm install # no third-party dependencies, just initialization
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
### 2. Verify the environment (offline, no API key needed)
|
|
131
|
+
|
|
132
|
+
```bash
|
|
133
|
+
npm test # 25 fixture tests, all run offline
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Seeing `TOTAL: 25 passed, 0 failed` means the environment is ready.
|
|
137
|
+
|
|
138
|
+
### 3. Feed a demo file with one command
|
|
139
|
+
|
|
140
|
+
```bash
|
|
141
|
+
node -e "const wf = require('./workflow.js'); console.log('meta:', wf.meta.name); console.log('parsers:', wf.parsers.list().map(p => p.name).join(', '));"
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
Output looks like:
|
|
145
|
+
|
|
146
|
+
```
|
|
147
|
+
meta: deep-read-summarize
|
|
148
|
+
parsers: book, paper, video, web
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
### 4. Real run (requires the DSH workflow tool)
|
|
152
|
+
|
|
153
|
+
Pass the JSON below to DSH's workflow tool (see Usage):
|
|
154
|
+
|
|
155
|
+
```jsonc
|
|
156
|
+
{ "input": "https://arxiv.org/abs/2307.09042", "type": "paper", "options": { "maxChunks": 4, "fastMode": true } }
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
The workflow returns a structured result: `{ ok, kind, title, filePath, qualityPassed, note }`, where `note` is the final Markdown note.
|
|
160
|
+
|
|
161
|
+
> ⚠️ **The caller persists the note.** DSH's workflow contract gives the script no filesystem access — it only coordinates subagents — so the script returns the note body plus a suggested `filePath` and never writes the file itself. The bundled SKILL.md tells the model to write `note` to `filePath` with the `write` tool after the run; do the same when you invoke the workflow by hand, or the note exists only in the return value.
|
|
162
|
+
|
|
163
|
+
---
|
|
164
|
+
## Usage
|
|
165
|
+
|
|
166
|
+
Pass this JSON to DSH's workflow tool:
|
|
167
|
+
|
|
168
|
+
```jsonc
|
|
169
|
+
{
|
|
170
|
+
"input": "https://arxiv.org/abs/2307.09042", // link or file path
|
|
171
|
+
"type": "auto", // auto | book | paper | video | web
|
|
172
|
+
"options": {
|
|
173
|
+
"minWords": 2500,
|
|
174
|
+
"fastMode": false, // true skips sections 5-7, faster
|
|
175
|
+
"maxChunks": 4, // chunk limit, 1-12 (default lowered to control sub-agent count and time)
|
|
176
|
+
"transcribe": true, // video: auto local transcription (faster-whisper) when no subtitles; false skips it — see Video
|
|
177
|
+
"requireCitations": true, // whether key conclusions must carry citations
|
|
178
|
+
"includeTimestamps": false,
|
|
179
|
+
"outputDir": "./output", // note output directory (can point at your Obsidian vault)
|
|
180
|
+
"tempDir": "./.tmp" // temp file directory
|
|
181
|
+
}
|
|
182
|
+
}
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
Examples:
|
|
186
|
+
|
|
187
|
+
| Type | input |
|
|
188
|
+
|------|-------|
|
|
189
|
+
| Paper | `https://arxiv.org/abs/2307.09042` |
|
|
190
|
+
| Book | local path, e.g. `~/books/xxx.pdf` |
|
|
191
|
+
| Video | `https://youtube.com/watch?v=xxx` |
|
|
192
|
+
| Web page | `https://example.com/article` |
|
|
193
|
+
|
|
194
|
+
Notes are written to `options.outputDir` (default `./output`, can point at your Obsidian vault), with the filename taken from the content title.
|
|
195
|
+
|
|
196
|
+
### Video: full transcript (subtitles first, auto-transcription when none)
|
|
197
|
+
|
|
198
|
+
Video text extraction is a **unified pipeline** (the old "three tiers" are gone): the goal is to get the **full transcript**, then deep-read it.
|
|
199
|
+
|
|
200
|
+
> **⚠️ To preserve deep-reading quality, processing may take noticeably longer (a deliberate trade-off, not a hang)**: a **subtitle-less** video is transcribed first (roughly 0.5–2× the video length on CPU) before deep reading, so such videos are **much slower than subtitled videos**; in exchange you get a **full transcript + real deep reading**. **Subtitled** videos remain seconds-fast. Please do not mistake the slower transcription for a hang.
|
|
201
|
+
|
|
202
|
+
1. **Platform subtitles available** (Bilibili's official API AI subtitles / YouTube CC) → **use them directly** (= full text, fastest, zero dependencies).
|
|
203
|
+
2. **No public subtitles** → use the bundled transcription `scripts/transcribe.ps1` (faster-whisper small / int8 / VAD / Chinese) to get the full text.
|
|
204
|
+
3. **Neither works** → **degrade gracefully**: prompt the user to supply a transcript manually, or fall back to `desc` as background; **never block**.
|
|
205
|
+
|
|
206
|
+
`options.transcribe`: **default `true`** (auto-transcribe when no subtitles); set `false` to skip transcription (use subtitles/desc or degrade).
|
|
207
|
+
|
|
208
|
+
**Transcription toolchain (identical for you and the author)**: the script **bootstraps itself** — it uses `uv` to create a **Python 3.12** environment + **Tsinghua mirror** to install `faster-whisper`, downloads the model via `HF_ENDPOINT=https://hf-mirror.com` and **caches it for reuse**; faster-whisper bundles PyAV to decode audio, so **no separate ffmpeg** is needed; **versions are pinned** (Python 3.12 / faster-whisper / uv) so the author's environment matches yours.
|
|
209
|
+
|
|
210
|
+
> ⚠️ **The first transcription downloads the model first (small ≈ 484 MB, via hf-mirror, not GitHub) and may take a few minutes** — please be patient; **it downloads only once** on your machine, then every later transcription uses the cache and starts instantly. **Only subtitle-less videos trigger transcription** (subtitled videos use their subtitles directly — no trigger, no download).
|
|
211
|
+
|
|
212
|
+
**Quality/speed**: `small` + `int8` quantization + `VAD` silence filtering gives usable Chinese quality and is CPU-friendly. **Subtitled videos are not transcribed (fast)**; subtitle-less videos are transcribed at roughly 0.5–2× the video length on CPU.
|
|
213
|
+
|
|
214
|
+
**Failure handling**: install/download failures **report an error and degrade** (prompt for a manual transcript), never silently and never blocking for long. Use `options.transcribe: false` to skip transcription.
|
|
215
|
+
|
|
216
|
+
**How to know what source was used**: source info is **not written into the note** (to keep it clean); it surfaces in: ① the run log's `[source] <value> -> <label>` line; ② the `textSource` field in the returned result (`subtitle / transcription / desc / manual`).
|
|
217
|
+
|
|
218
|
+
---
|
|
219
|
+
|
|
220
|
+
## Variance and influencing factors (why length and content may differ)
|
|
221
|
+
|
|
222
|
+
**Output length and content vary noticeably with the model and the input; this is normal.**
|
|
223
|
+
|
|
224
|
+
### Model
|
|
225
|
+
- **ASR model tier**: `small` (default, usable for Chinese) → `base` (faster, slightly lower quality) → `medium` (better, slower). Adjust with `options.whisperModel`; download size and speed grow with the model.
|
|
226
|
+
- **Language/accent**: Chinese/English/mixed language, dialect, accent, stress → **different transcription accuracy and time**; for pure English video you can set `language: "en"`.
|
|
227
|
+
- **Deep-reading LLM**: uses the host DSH model; different context/reasoning models → **different quality, length, and speed**.
|
|
228
|
+
- **Non-determinism**: both the LLM and ASR are stochastic, so **re-running the same input may produce slightly different results** (content/length).
|
|
229
|
+
|
|
230
|
+
### Input
|
|
231
|
+
- **Video**:
|
|
232
|
+
- Whether **public subtitles** exist is the biggest differentiator — public subtitles = seconds-fast; login-only AI subtitles or no subtitles = triggers transcription, **noticeably slower**.
|
|
233
|
+
- **Length**: longer video = longer transcription (≈ 0.5–2× the video length on CPU).
|
|
234
|
+
- **Audio quality/noise/low volume/pure BGM/no voice**: transcription is error-prone or **empty**; supply a transcript manually in that case.
|
|
235
|
+
- **Paper/book/web page**:
|
|
236
|
+
- Format: PDF/EPUB/MOBI/HTML/arXiv; **scanned OCR, broken PDFs, dense formulas/figures** → different extraction quality and time.
|
|
237
|
+
- **Term density/length** → different chunk count and deep-reading depth.
|
|
238
|
+
- **Chunking structure**: different types chunk differently (chapter/topic/paragraph), affecting **output structure** and coverage.
|
|
239
|
+
|
|
240
|
+
### Hardware
|
|
241
|
+
- **GPU vs CPU**: GPU transcription can be an order of magnitude faster than CPU (`device='auto'` picks automatically). Subtitle-less videos are ≈ 0.5–2× the length on CPU.
|
|
242
|
+
|
|
243
|
+
### Other
|
|
244
|
+
- **First run vs reuse**: the first run downloads the model (≈ 484 MB); later runs reuse the cache.
|
|
245
|
+
- **Token/cost**: chunk count, length, and citations drive token usage; `fastMode`/`maxChunks` save tokens.
|
|
246
|
+
- **Citation/fabrication**: transcription or OCR errors may be quoted as "original text"; **re-check key citations**.
|
|
247
|
+
- **Boundaries**: pure BGM, distorted audio → empty/poor transcription; no subtitles and transcription unavailable → degrade to a prompt for a manual transcript.
|
|
248
|
+
|
|
249
|
+
---
|
|
250
|
+
|
|
251
|
+
## How failures are handled
|
|
252
|
+
|
|
253
|
+
Two classes:
|
|
254
|
+
|
|
255
|
+
- **Configuration errors** (missing input, invalid type, malformed options): throw immediately and abort — no half-baked output.
|
|
256
|
+
- **Content problems** (fetch failed, a chunk's deep-read failed): return `{ ok: false, stage, fatal: false }` and let the caller decide. A failed chunk is skipped and marked as a gap in the draft; the whole run is not aborted.
|
|
257
|
+
|
|
258
|
+
---
|
|
259
|
+
|
|
260
|
+
## Cost
|
|
261
|
+
|
|
262
|
+
A paper, 6 chunks, full mode: roughly 15–25k tokens. `fastMode` with a lower `maxChunks` saves about 40%.
|
|
263
|
+
|
|
264
|
+
---
|
|
265
|
+
|
|
266
|
+
## Security & copyright
|
|
267
|
+
|
|
268
|
+
- Fetching videos/web pages makes external requests, executed by sub-agents under DSH's sandbox and approval policy; don't let a model run arbitrary scripts unattended.
|
|
269
|
+
- Sub-agents need appropriate permissions to write files; if permissions are insufficient, the workflow returns the content and the main agent handles persistence.
|
|
270
|
+
- **Local transcription**: video transcription (faster-whisper) runs **locally**; audio/subtitles **do not leave your machine**; network requests to the corresponding platforms are made only when fetching video metadata/subtitles/audio.
|
|
271
|
+
- This repo contains only the pipeline itself: workflow definitions, parser code, prompt templates, schemas. **It contains no extracted copyrighted content.** Test fixtures are self-written public-domain fables. When you process copyrighted material with this workflow, responsibility for the output is yours.
|
|
272
|
+
|
|
273
|
+
---
|
|
274
|
+
|
|
275
|
+
## Development
|
|
276
|
+
|
|
277
|
+
```bash
|
|
278
|
+
npm install
|
|
279
|
+
npm run lint # node --check on all JS
|
|
280
|
+
npm test # fixture tests (25, all offline)
|
|
281
|
+
npm run validate # pre-release validation (including security checks)
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
`npm publish` automatically runs `prepublishOnly` (tests + lint + security checks) first; any failure blocks the publish.
|
|
285
|
+
|
|
286
|
+
See CHANGELOG.md for changes and CONTRIBUTING.md for contribution guidelines.
|
|
287
|
+
|
|
288
|
+
---
|
|
289
|
+
|
|
290
|
+
## Compatibility
|
|
291
|
+
|
|
292
|
+
**Verified host: DSH 0.1.2-rc.1** (Node 24). The dependency surface is exactly two things:
|
|
293
|
+
|
|
294
|
+
| Surface | Current contract | How this repo uses it |
|
|
295
|
+
| --- | --- | --- |
|
|
296
|
+
| workflow script hooks | `agent()` / `pipeline()` / `parallel()` / `phase()` / `log()` / `args` | only `agent/parallel/phase/log/args`; `agent()` options limited to `label/phase/schema` (the engine accepts only `label/phase/schema/provider/model` and treats anything else as fatal) |
|
|
297
|
+
| JSON Schema subset | `type / properties / required / additionalProperties / items / enum / const / oneOf` plus annotation keys | both `agent()` schemas verified against the host's real `assertObjectJsonSchema` |
|
|
298
|
+
|
|
299
|
+
Two contract details that bite:
|
|
300
|
+
|
|
301
|
+
- **The script sandbox has no filesystem, network, timers or `require`** — it coordinates subagents; the caller persists the note (see ⚠️ above).
|
|
302
|
+
- **Breaking the schema subset terminates the run** rather than degrading: the engine raises `UNSUPPORTED_SCHEMA` and the whole workflow fails. `npm test` carries a regression for this, including a reverse assertion so the checker cannot pass vacuously.
|
|
303
|
+
|
|
304
|
+
After upgrading DSH, run `npm test` and `npm run test:node`; if something breaks, cross-reference CHANGELOG.md.
|
|
305
|
+
|
|
306
|
+
---
|
|
307
|
+
|
|
308
|
+
## License
|
|
309
|
+
|
|
310
|
+
MIT — see [LICENSE](LICENSE).
|
package/README.md
CHANGED
|
@@ -3,21 +3,39 @@
|
|
|
3
3
|
[](https://github.com/PensiveFei/deep-read-summarize/blob/main/LICENSE)
|
|
4
4
|
[](https://github.com/PensiveFei/deep-read-summarize/releases)
|
|
5
5
|
[](https://github.com/PensiveFei/deep-read-summarize/actions/workflows/ci.yml)
|
|
6
|
-
[](https://www.npmjs.com/package/deep-read-summarize)
|
|
7
7
|
|
|
8
8
|
> **Disclaimer**: This is an **unofficial third-party tool**. It is not affiliated with, endorsed by, or sponsored by DeepSeek or the DeepSeek Harness project. "DeepSeek" and "DeepSeek Harness" are trademarks of their respective owners. This project only orchestrates the workflow tool available in your DSH environment; it does not redistribute any DeepSeek software.
|
|
9
9
|
|
|
10
|
+
> **⚠️ 最重要:本工具是「辅助精读」工具,不能代替自主学习。** 它帮你快速提取、归纳、结构化阅读内容,但**真正的理解、思考与批判仍需你自己完成**;请始终以原文为准,不要把它当作学习成果本身。
|
|
11
|
+
|
|
10
12
|
给 DSH(DeepSeek Harness)写的一个精读工作流:输入一本书、一篇论文、一个视频链接或网页,输出一份结构化的 Obsidian 笔记。
|
|
11
13
|
|
|
12
14
|
内容会被拆成若干块,由多个子代理并行精读,再合并成一篇带 YAML frontmatter 的 Markdown。关键结论附原文出处,成稿前有一道质量校验。
|
|
13
15
|
|
|
16
|
+
> 使用中遇到问题、或觉得哪里能更好,**欢迎随时在 GitHub issue 反馈**。
|
|
17
|
+
|
|
14
18
|
> 注意:DSH 目前是 developer preview,接口可能变化。本仓库针对特定版本的 workflow 工具语义编写,见下文[兼容性](#兼容性)。
|
|
15
19
|
|
|
16
20
|
---
|
|
17
21
|
|
|
22
|
+
## 实际效果
|
|
23
|
+
|
|
24
|
+
(图用 GitHub 绝对地址,npm 页面同样能显示;图片本身不打进 npm 包,只随仓库分发)
|
|
25
|
+
|
|
26
|
+
| 精读论文 | 视频笔记(元信息) |
|
|
27
|
+
| --- | --- |
|
|
28
|
+
|  |  |
|
|
29
|
+
|
|
30
|
+
| 视频笔记(行动要点) | 会话内摘要 |
|
|
31
|
+
| --- | --- |
|
|
32
|
+
|  |  |
|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
18
36
|
## 能做什么
|
|
19
37
|
|
|
20
|
-
- 书籍(PDF/EPUB/MOBI)、论文(arXiv/PDF/HTML
|
|
38
|
+
- 书籍(PDF/EPUB/MOBI)、论文(arXiv/PDF/HTML)、视频(**字幕优先,无字幕自动转写**,出品完整逐字稿精读)、网页
|
|
21
39
|
- 长内容分块后由并行子代理精读,再合并
|
|
22
40
|
- 各子任务的输出用 JSON Schema 约束,不合格自动重试
|
|
23
41
|
- 关键引用必须标注页码/章节/段落,降低编造风险
|
|
@@ -53,7 +71,7 @@ Obsidian 笔记
|
|
|
53
71
|
parsers/ 各输入类型的解析器,按类型分发
|
|
54
72
|
book.js 书籍:PDF/EPUB/MOBI 文本提取、章节分块
|
|
55
73
|
paper.js 论文:arXiv/PDF/HTML 结构识别
|
|
56
|
-
video.js
|
|
74
|
+
video.js 视频:完整逐字稿(字幕优先,无字幕自动转写,走 scripts/transcribe.ps1)
|
|
57
75
|
web.js 网页:正文提取
|
|
58
76
|
index.js 注册表:解析器发现与回退
|
|
59
77
|
schemas/ 子任务输出的 JSON Schema
|
|
@@ -83,7 +101,7 @@ deep-read-summarize 是 DSH 插件,可通过 npm、dsh.so 生态或本地安
|
|
|
83
101
|
npm install deep-read-summarize
|
|
84
102
|
|
|
85
103
|
# 或本地安装(dsh profile 目录)
|
|
86
|
-
pnpm add ./deep-read-summarize-0.3.
|
|
104
|
+
pnpm add ./deep-read-summarize-0.3.6.tgz
|
|
87
105
|
# 然后在 dsh 配置的 dsh.profile.bundles 追加:
|
|
88
106
|
# - deep-read-summarize
|
|
89
107
|
# 重启 dsh web 即可(POST /dsh-market/restart)
|
|
@@ -140,6 +158,8 @@ parsers: book, paper, video, web
|
|
|
140
158
|
|
|
141
159
|
工作流会返回结构化结果:`{ ok, kind, title, filePath, qualityPassed, note }`,`note` 即最终 Markdown 笔记。
|
|
142
160
|
|
|
161
|
+
> ⚠️ **笔记由主代理落盘**:DSH 的 workflow 契约规定脚本只能协调子代理(沙箱内没有文件系统),所以脚本只返回 `note` 正文与建议路径 `filePath`,**不会自己写文件**。插件自带的 SKILL.md 已明确要求模型在拿到结果后用 `write` 工具写入 `filePath`——自己拼 workflow 调用时也要记得这一步,否则笔记只存在于返回值里。
|
|
162
|
+
|
|
143
163
|
---
|
|
144
164
|
## 用法
|
|
145
165
|
|
|
@@ -152,7 +172,8 @@ parsers: book, paper, video, web
|
|
|
152
172
|
"options": {
|
|
153
173
|
"minWords": 2500,
|
|
154
174
|
"fastMode": false, // true 时跳过 5-7 节,速度快一些
|
|
155
|
-
"maxChunks":
|
|
175
|
+
"maxChunks": 4, // 分块上限,1-12(默认调低,控制子代理数与耗时)
|
|
176
|
+
"transcribe": true, // 视频:无字幕时自动本地转写(faster-whisper);false 则跳过转写,见「视频」节
|
|
156
177
|
"requireCitations": true, // 关键结论是否必须标注出处
|
|
157
178
|
"includeTimestamps": false,
|
|
158
179
|
"outputDir": "./output", // 笔记输出目录(可指向 Obsidian 仓库)
|
|
@@ -172,7 +193,58 @@ parsers: book, paper, video, web
|
|
|
172
193
|
|
|
173
194
|
笔记写到 `options.outputDir` 指定的目录(默认 `./output`,可指向你的 Obsidian 仓库),文件名取自内容标题。
|
|
174
195
|
|
|
175
|
-
|
|
196
|
+
### 视频:完整逐字稿(字幕优先,无字幕自动转写)
|
|
197
|
+
|
|
198
|
+
视频取文本是**统一流水线**(已去掉「三档」):目标 = 拿到**完整逐字稿**再精读。
|
|
199
|
+
|
|
200
|
+
> **⚠️ 为保证精读质量,处理时间可能略长(刻意取舍,不是卡死)**:**无字幕**视频会先做一次转写(CPU 上约为视频时长的 0.5–2×)再精读,所以这类视频整体会**明显比有字幕视频慢**;换来的是**完整逐字稿 + 真正的精读**。**有字幕**视频仍是秒级。请勿把转写的较慢误判为卡死。
|
|
201
|
+
|
|
202
|
+
1. **有平台字幕**(B站官方 API 的 AI 字幕 / YouTube CC)→ **直接用字幕**(=全文,最快、零依赖)。
|
|
203
|
+
2. **无公开字幕** → 用插件自带转写 `scripts/transcribe.ps1`(faster-whisper small / int8 / VAD / 中文)得到全文。
|
|
204
|
+
3. 都不行 → **降级**:提示手动提供转写文本,或退回 `desc` 作背景;**绝不阻塞**。
|
|
205
|
+
|
|
206
|
+
`options.transcribe`:**默认 `true`**(无字幕自动转写);设为 `false` 则跳过转写(只用字幕/desc 或降级)。
|
|
207
|
+
|
|
208
|
+
**转写工具链(本机/用户一致)**:脚本**自举**——用 `uv` 建 **Python 3.12** 环境 + **清华镜像**装 `faster-whisper` + `HF_ENDPOINT=https://hf-mirror.com` 下模型并**缓存复用**;faster-whisper 内置 PyAV 解码音频,**无需单独 ffmpeg**;**版本已锁定**(Python 3.12 / faster-whisper / uv),保证本机与用户环境一致。
|
|
209
|
+
|
|
210
|
+
> ⚠️ **首次转写会先下载模型(small 约 484MB,走 hf-mirror,非 GitHub),可能需要几分钟**,请耐心等待;**本机只下这一次**,之后所有视频转写用缓存、秒开。**只有「无字幕」视频才会触发转写**(有字幕的视频直接用字幕,不触发、不下载)。
|
|
211
|
+
|
|
212
|
+
**质量/速度**:`small` + `int8` 量化 + `VAD` 静音过滤,中文质量可用且 CPU 友好。**有字幕的视频不转写(快)**;无字幕视频转写,CPU 上约为视频时长的 0.5–2×。
|
|
213
|
+
|
|
214
|
+
**失败处理**:安装/下载失败则**明确报错并降级**(提示手动提供转写文本),不静默、不长期阻塞。此时可用 `options.transcribe: false` 跳过转写。
|
|
215
|
+
|
|
216
|
+
**怎么知道本次用了什么来源**:来源信息**不写进笔记**(保证笔记干净),透出在:① 运行日志的 `[source] <值> -> <标签>` 行;② 返回结果里的 `textSource` 字段(`subtitle / transcription / desc / manual`)。
|
|
217
|
+
|
|
218
|
+
---
|
|
219
|
+
|
|
220
|
+
## 差异与影响因素(时长与内容为何可能不同)
|
|
221
|
+
|
|
222
|
+
**输出时长与内容会因「模型」和「输入」不同而有明显差异,属正常现象**,请知悉:
|
|
223
|
+
|
|
224
|
+
### 模型
|
|
225
|
+
- **ASR 模型档位**:`small`(默认,中文可用)→ `base`(更快、质量略降)→ `medium`(更好、更慢)。可用 `options.whisperModel` 调整;下载量与速度随模型变大而增长。
|
|
226
|
+
- **语言/口音**:中/英/混合语言、方言、口音、重音 → 转写**准确度与耗时不同**;纯英文视频可设 `language: "en"`。
|
|
227
|
+
- **精读 LLM**:用宿主 DSH 的模型,不同上下文/推理模型 → 输出**质量、篇幅、速度**不同。
|
|
228
|
+
- **非确定性**:LLM 与 ASR 均带随机性,**同一输入重复跑可能得到略有差异**的结果(内容/篇幅)。
|
|
229
|
+
|
|
230
|
+
### 输入
|
|
231
|
+
- **视频**:
|
|
232
|
+
- 有无**公开字幕**是最大差异点——有公开字幕 = 秒级;只有登录态 AI 字幕/无字幕 = 触发转写,**明显变慢**。
|
|
233
|
+
- **时长**:越长转写越久(CPU 约 0.5–2× 视频时长)。
|
|
234
|
+
- **音质/噪杂/低音量/纯 BGM/无人声**:转写易错或**结果空**,此时请手动提供转写文本。
|
|
235
|
+
- **论文/书籍/网页**:
|
|
236
|
+
- 格式:PDF/EPUB/MOBI/HTML/arXiv;**扫描版 OCR、坏 PDF、公式/图表密集** → 提取质量与耗时不同。
|
|
237
|
+
- **术语密度/篇幅** → 分块数与精读深度不同。
|
|
238
|
+
- **分块结构**:不同类型分块方式不同(章节/主题/段落),影响**输出结构**与覆盖。
|
|
239
|
+
|
|
240
|
+
### 硬件
|
|
241
|
+
- **GPU vs CPU**:GPU 转写可比 CPU 快一个数量级(`device='auto'` 自动选)。CPU 上无字幕视频约 0.5–2× 时长。
|
|
242
|
+
|
|
243
|
+
### 其他
|
|
244
|
+
- **首次 vs 复用**:首次要下模型(约 484MB),之后缓存复用。
|
|
245
|
+
- **Token/成本**:分块数/篇幅/引证决定 token 消耗;`fastMode`/`maxChunks` 可省。
|
|
246
|
+
- **引用/编造**:转写或 OCR 的错词可能被当作「原文」引用,**关键引用请复核**。
|
|
247
|
+
- **边界**:纯 BGM、失真音频 → 转写空/差;无字幕且转写不可用 → 降级提示人工提供转写文本。
|
|
176
248
|
|
|
177
249
|
---
|
|
178
250
|
|
|
@@ -195,6 +267,7 @@ parsers: book, paper, video, web
|
|
|
195
267
|
|
|
196
268
|
- 抓取视频/网页会发起外部请求,由子代理在 DSH 的沙箱和审批策略下执行;不要让模型无人值守地跑任意脚本。
|
|
197
269
|
- 子代理写文件需要相应权限;如果权限不够,workflow 会把内容返回,由主代理负责落盘。
|
|
270
|
+
- **本地转写**:视频转写(faster-whisper)在**本机**运行,音频/字幕**不出本地**;抓取视频元数据/字幕/音频时才会向对应平台发起网络请求。
|
|
198
271
|
- 这个仓库只有流程本身:workflow 定义、解析器代码、提示词模板、schema。**不含任何受版权保护内容的提取结果。** 测试用的 fixture 是自写的公共领域寓言。用本工作流处理有版权的材料时,产出物的使用责任在你。
|
|
199
272
|
|
|
200
273
|
---
|
|
@@ -216,12 +289,19 @@ npm publish 会自动先跑 prepublishOnly(测试 + lint + 安全检查),
|
|
|
216
289
|
|
|
217
290
|
## 兼容性
|
|
218
291
|
|
|
219
|
-
DSH
|
|
292
|
+
**已验证的宿主:DSH 0.1.2-rc.1**(Node 24)。本仓库的依赖面只有两处:
|
|
293
|
+
|
|
294
|
+
| 依赖面 | 现行契约 | 本仓库的用法 |
|
|
295
|
+
| --- | --- | --- |
|
|
296
|
+
| workflow 脚本钩子 | `agent()` / `pipeline()` / `parallel()` / `phase()` / `log()` / `args` | 只用了 `agent/parallel/phase/log/args`;`agent()` 选项只用 `label/phase/schema`(引擎只接受 `label/phase/schema/provider/model`,其它选项会被判为 fatal) |
|
|
297
|
+
| JSON Schema 子集 | `type / properties / required / additionalProperties / items / enum / const / oneOf` + 注解键 | 两个 `agent()` schema 已用宿主真实的 `assertObjectJsonSchema` 实测通过 |
|
|
298
|
+
|
|
299
|
+
两点容易踩的契约细节:
|
|
220
300
|
|
|
221
|
-
-
|
|
222
|
-
-
|
|
301
|
+
- **脚本沙箱没有文件系统/网络/定时器/require**——只能协调子代理;笔记落盘由主代理完成(见上文 ⚠️)。
|
|
302
|
+
- **违反上述子集不是降级而是终止**:引擎抛 `UNSUPPORTED_SCHEMA`,整条 workflow 直接失败。仓库的 `npm test` 里有对应回归(含反向断言,确保校验器不是空放行)。
|
|
223
303
|
|
|
224
|
-
升级 DSH
|
|
304
|
+
升级 DSH 后先跑 `npm test` + `npm run test:node`;坏了对照 CHANGELOG.md 排查。
|
|
225
305
|
|
|
226
306
|
---
|
|
227
307
|
|
package/index.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
// ============================================================================
|
|
2
2
|
// deep-read-summarize — DSH plugin entry
|
|
3
3
|
// ============================================================================
|
|
4
|
-
// 标准 Cordis 插件(DSH 0.1.x
|
|
4
|
+
// 标准 Cordis 插件(DSH 0.1.x 契约,已在 0.1.2-rc.1 实测):导出 name + inject + apply(ctx)。
|
|
5
5
|
// apply() 将插件自带的 SKILL.md 注册为运行时技能,并把 workflow 的
|
|
6
6
|
// meta + 自包含 script + args 示例嵌入技能内容,使模型可直接用 workflow
|
|
7
7
|
// 工具执行精读流程。
|
|
@@ -47,7 +47,7 @@ function buildSkillContent() {
|
|
|
47
47
|
].join('\n');
|
|
48
48
|
}
|
|
49
49
|
|
|
50
|
-
/**
|
|
50
|
+
/** Cordis 插件入口:注册运行时技能,使模型可直接执行精读 workflow。 */
|
|
51
51
|
function apply(ctx, config = {}) {
|
|
52
52
|
const skillDir = path.join(__dirname, 'skills', 'deep-read-summarize');
|
|
53
53
|
return ctx.skills.register({
|
|
@@ -56,7 +56,9 @@ function apply(ctx, config = {}) {
|
|
|
56
56
|
whenToUse: workflow.meta.whenToUse,
|
|
57
57
|
content: buildSkillContent(),
|
|
58
58
|
resourceBase: { kind: 'directory', path: skillDir },
|
|
59
|
-
source
|
|
59
|
+
// source 是技能的「来源桶」(runtime / bundled / user-dsh …),不是自由描述:
|
|
60
|
+
// 本技能随插件包分发,因此登记为 bundled。
|
|
61
|
+
source: 'bundled'
|
|
60
62
|
});
|
|
61
63
|
}
|
|
62
64
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "deep-read-summarize",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.7",
|
|
4
4
|
"description": "Deep reading & summarization for DSH: books/papers/videos/web → structured Obsidian notes. Plugin parsers, MapReduce deep-read, JSON Schema output, idempotent cache.",
|
|
5
5
|
"type": "commonjs",
|
|
6
6
|
"main": "index.js",
|
|
@@ -20,7 +20,8 @@
|
|
|
20
20
|
"skills",
|
|
21
21
|
"scripts",
|
|
22
22
|
"tests",
|
|
23
|
-
"docs",
|
|
23
|
+
"docs/RELEASE.md",
|
|
24
|
+
"docs/RELEASE-v0.1.0.md",
|
|
24
25
|
"cordis.patch.yml",
|
|
25
26
|
"CHANGELOG.md",
|
|
26
27
|
"README.md",
|
|
@@ -33,7 +34,7 @@
|
|
|
33
34
|
"lint": "node scripts/lint.js",
|
|
34
35
|
"validate": "node tests/quick_validate.js && node scripts/security-check.js",
|
|
35
36
|
"security": "node scripts/security-check.js",
|
|
36
|
-
"test:node": "node --test
|
|
37
|
+
"test:node": "node --test",
|
|
37
38
|
"prepublishOnly": "node tests/run-tests.js && node scripts/lint.js && node scripts/security-check.js"
|
|
38
39
|
},
|
|
39
40
|
"dsh": {
|
package/parsers/video.js
CHANGED
|
@@ -1,8 +1,12 @@
|
|
|
1
1
|
// ============================================================================
|
|
2
|
-
// Parser plugin: video (YouTube
|
|
2
|
+
// Parser plugin: video (Bilibili/Douyin/YouTube → transcript & metadata)
|
|
3
3
|
// ============================================================================
|
|
4
|
-
//
|
|
5
|
-
//
|
|
4
|
+
// 统一流水线(去掉三档):目标 = 拿到完整逐字稿再精读。
|
|
5
|
+
// ① 有平台字幕(B站 AI 字幕 / YouTube CC)→ 直接用字幕(=全文,最快、零依赖)
|
|
6
|
+
// ② 无公开字幕 → 用插件自带转写(faster-whisper small/int8/VAD,镜像+缓存)得到全文
|
|
7
|
+
// ③ 都不行 → 降级(提示人工提供转写文本,或退回 desc 作背景)。
|
|
8
|
+
// 本机/用户使用一致:只依赖环境变量 + 相对/标准路径,不写死本机绝对路径。
|
|
9
|
+
// 提示词骨架复用 parsers/_prompt.js,此处只提供差异化配置。
|
|
6
10
|
// ============================================================================
|
|
7
11
|
|
|
8
12
|
const prompt = require("./_prompt");
|
|
@@ -10,22 +14,41 @@ const prompt = require("./_prompt");
|
|
|
10
14
|
module.exports = {
|
|
11
15
|
name: "video",
|
|
12
16
|
types: ["video"],
|
|
13
|
-
description: "视频解析器(YouTube
|
|
17
|
+
description: "视频解析器(B站/抖音/YouTube → 完整逐字稿,字幕优先,无字幕自动转写)",
|
|
14
18
|
|
|
15
19
|
buildPrompt: function (input, opts) {
|
|
16
20
|
const tempDir = (opts && opts.tempDir) || "./.tmp";
|
|
21
|
+
const wantTranscribe = opts.transcribe !== false; // 默认 true:无字幕时自动走本地转写
|
|
22
|
+
const degradeMsg = "需要人工提供转写文本或视频文案";
|
|
23
|
+
const steps = [
|
|
24
|
+
"记录文本来源(metadata.textSource):subtitle(平台字幕)/ transcription(本地转写)/ desc(仅简介,作为降级/背景)/ manual(用户提供)。",
|
|
25
|
+
"统一流程:① 先取视频字幕(B站 AI 字幕 / YouTube yt-dlp CC);② 若**无公开字幕**,用插件自带 faster-whisper 转写得到全文;③ 仍不可行才降级(提示人工提供转写文本 或 退回 desc 作背景)。",
|
|
26
|
+
"若输入是 B 站链接(bilibili.com / b23.tv,短链先用 curl -L 跟随重定向提取 BV 号):用 curl(Windows 自带,零安装)调官方公开 API:\n" +
|
|
27
|
+
" 1) 元数据+简介:curl 'https://api.bilibili.com/x/web-interface/view?bvid=<BV>' → 取 title / desc(简介仅作背景/补充,不满一篇精读)/ owner.name / duration / aid / cid\n" +
|
|
28
|
+
" 2) 字幕(**优先,最快**):curl 'https://api.bilibili.com/x/player/v2?aid=<aid>&cid=<cid>' 取 subtitle 列表;若有则下载转文本(textSource=subtitle)——这已是完整逐字稿,直接使用\n" +
|
|
29
|
+
" 3) 无公开字幕时(textSource=transcription):按下方「转写执行」用 faster-whisper 转写整段音频,得到完整全文。",
|
|
30
|
+
|
|
31
|
+
"若输入是抖音链接/分享口令(douyin.com / v.douyin.com):用户分享时复制的文字本身就是该视频的文案(desc)。desc 较完整时可直接作为文本(textSource=desc);若 desc 太短,仍按下方「转写执行」转写(textSource=transcription)。",
|
|
32
|
+
"若输入是 YouTube 等其他平台:先探测是否已装 yt-dlp(pwsh 运行 `yt-dlp --version`)。若未安装:绝对不要去下载 yt-dlp 二进制(GitHub 直连在部分网络下极慢且易中断,会卡死整个流程);先试 `winget install yt-dlp.yt-dlp`,再试 `pip install -U yt-dlp -i https://pypi.tuna.tsinghua.edu.cn/simple`;仍不可用则按下方「转写执行」直接转写或降级。已安装时用 yt-dlp 抓字幕(CC)转文本;无 CC 则转写。",
|
|
33
|
+
|
|
34
|
+
"转写执行(faster-whisper small/int8/中文,镜像+缓存,保证本机与用户一致):",
|
|
35
|
+
" ① 先取音频:B站用 playurl(curl 'https://api.bilibili.com/x/player/playurl?avid=<aid>&cid=<cid>&fnval=16&fourk=1' 的 dash.audio[0].baseUrl)或 yt-dlp,下载存到 '" + tempDir + "/audio.m4a';\n" +
|
|
36
|
+
" ② 定位插件转写脚本:用 Get-ChildItem -Path $env:USERPROFILE/.dsh -Recurse -Filter transcribe.ps1(或 glob 搜 *transcribe.ps1)找到 <脚本路径>;\n" +
|
|
37
|
+
" ③ 运行:pwsh <脚本路径> -Audio '" + tempDir + "/audio.m4a' -Out '" + tempDir + "/transcript.txt' -Model small -Language zh;\n" +
|
|
38
|
+
" ④ 转写可能较慢:用 pwsh(run_in_background: true)启动后,轮询输出文件 '" + tempDir + "/transcript.txt' 是否写出内容(非空且稳定),或日志出现 DONE;设合理超时,超时/失败则降级;\n" +
|
|
39
|
+
" ⑤ 成功后读取 transcript.txt 作为全文(textSource=transcription)。脚本自举:uv 建 Python 3.12 环境 + 镜像装 faster-whisper + hf-mirror 下模型并缓存。**⚠️ 首次转写会先下载 small 模型(约 484MB,hf-mirror)并较慢——先向用户说明这是正常的一次性下载,勿当成卡死;之后缓存复用、秒开。**",
|
|
40
|
+
|
|
41
|
+
"若以上均不可用或抓取失败:输出 { \"saved\": false, \"message\": \"" + degradeMsg + "\" }。"
|
|
42
|
+
];
|
|
43
|
+
|
|
17
44
|
return prompt.buildFetchPrompt({
|
|
18
45
|
input: input,
|
|
19
46
|
opts: opts,
|
|
20
|
-
fetchTarget: "
|
|
47
|
+
fetchTarget: "视频完整逐字稿(字幕优先,无字幕自动转写)",
|
|
21
48
|
kindLabel: "视频(video)",
|
|
22
|
-
steps:
|
|
23
|
-
"用 pwsh 运行 yt-dlp 抓字幕,命令示例:\n yt-dlp --skip-download --write-auto-sub --write-subs --sub-langs 'zh,en' --sub-format vtt/srt -o '" + tempDir + "/sub.%(ext)s' <URL>",
|
|
24
|
-
"读取字幕文件(vtt/srt),去除时间戳与 HTML 标签,转为纯文本写入目标文件。",
|
|
25
|
-
"若 yt-dlp 不可用或抓取失败:输出 { \"saved\": false, \"message\": \"需要人工提供转写文本\" }。"
|
|
26
|
-
],
|
|
49
|
+
steps: steps,
|
|
27
50
|
chunkRule: "主题脉络(而非时间)",
|
|
28
|
-
jsonExample: '{ "kind": "video", "saved": true, "totalLines": N, "metadata": { "title": "", "author": "", "year": "", "language": "zh|en" }, "chunkPlan": [ { "id": 1, "topic": "主题段", "startLine": 1, "endLine": 200 } ] }'
|
|
51
|
+
jsonExample: '{ "kind": "video", "saved": true, "totalLines": N, "metadata": { "title": "", "author": "", "year": "", "language": "zh|en", "textSource": "subtitle|transcription|desc|manual" }, "chunkPlan": [ { "id": 1, "topic": "主题段", "startLine": 1, "endLine": 200 } ] }'
|
|
29
52
|
});
|
|
30
53
|
}
|
|
31
|
-
};
|
|
54
|
+
};
|
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
# transcribe.ps1 - faster-whisper transcription bootstrap (plugin-internal, self-bootstrapping)
|
|
2
|
+
# Usage: pwsh transcribe.ps1 -Audio <file> -Out <txt> [-Model small] [-Language zh]
|
|
3
|
+
# Deterministic per-user cache so behavior is identical on dev & user machines.
|
|
4
|
+
param(
|
|
5
|
+
[Parameter(Mandatory=$true)][string]$Audio,
|
|
6
|
+
[Parameter(Mandatory=$true)][string]$Out,
|
|
7
|
+
[string]$Model = 'small',
|
|
8
|
+
[string]$Language = 'zh'
|
|
9
|
+
)
|
|
10
|
+
$ErrorActionPreference = 'Stop'
|
|
11
|
+
|
|
12
|
+
$cacheRoot = Join-Path $env:LOCALAPPDATA 'deep-read-summarize'
|
|
13
|
+
$venvDir = Join-Path $cacheRoot 'venv'
|
|
14
|
+
$venvPy = Join-Path $venvDir 'Scripts/python.exe'
|
|
15
|
+
$hfHome = Join-Path $cacheRoot 'hf'
|
|
16
|
+
$pyScript = Join-Path $cacheRoot 'transcribe_run.py'
|
|
17
|
+
|
|
18
|
+
New-Item -ItemType Directory -Force -Path $cacheRoot | Out-Null
|
|
19
|
+
|
|
20
|
+
$uv = $null
|
|
21
|
+
$uvCmd = Get-Command uv -ErrorAction SilentlyContinue
|
|
22
|
+
if ($uvCmd) { $uv = $uvCmd.Source }
|
|
23
|
+
else { $alt = Join-Path $HOME '.local/bin/uv.exe'; if (Test-Path $alt) { $uv = $alt } }
|
|
24
|
+
if (-not $uv) { throw 'uv not found. Please install uv (winget install astral-sh.uv) or ensure it is on PATH.' }
|
|
25
|
+
|
|
26
|
+
# Use a China mirror for CPython download (uv pulls python-build-standalone from GitHub, slow in CN),
|
|
27
|
+
# and prefer an already-installed 3.12 (avoids the download entirely on many machines).
|
|
28
|
+
$env:UV_PYTHON_INSTALL_MIRROR = 'https://ghproxy.com/https://github.com/astral-sh/python-build-standalone/releases/download'
|
|
29
|
+
|
|
30
|
+
if (-not (Test-Path $venvPy)) {
|
|
31
|
+
# reuse an existing python 3.12 if present; else uv installs 3.12 (via mirror)
|
|
32
|
+
$usePy = (py -0p 2>$null | Select-String '3.12' | Select-Object -First 1)
|
|
33
|
+
if ($usePy) { $pyArg = '3.12' } else { $pyArg = '3.12' }
|
|
34
|
+
& $uv venv --python $pyArg $venvDir
|
|
35
|
+
if ($LASTEXITCODE -ne 0) { throw 'uv venv creation failed.' }
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
& $venvPy -c 'import faster_whisper' 2>$null
|
|
39
|
+
if ($LASTEXITCODE -ne 0) {
|
|
40
|
+
& $uv pip install --python $venvPy faster-whisper -i https://pypi.tuna.tsinghua.edu.cn/simple
|
|
41
|
+
if ($LASTEXITCODE -ne 0) { throw 'faster-whisper install failed (mirror unreachable?).' }
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
$env:HF_ENDPOINT = 'https://hf-mirror.com'
|
|
45
|
+
$env:HF_HOME = $hfHome
|
|
46
|
+
# Disable the hf-xet backend: it bypasses the mirror (cas-server.xethub.hf.co) and 401s; the mirror works over plain HTTP.
|
|
47
|
+
$env:HF_HUB_DISABLE_XET = '1'
|
|
48
|
+
$env:HF_HUB_DISABLE_SYMLINKS_WARNING = '1'
|
|
49
|
+
$env:DRS_AUDIO = $Audio
|
|
50
|
+
$env:DRS_OUT = $Out
|
|
51
|
+
$env:DRS_MODEL = $Model
|
|
52
|
+
$env:DRS_LANG = $Language
|
|
53
|
+
|
|
54
|
+
$py = @'
|
|
55
|
+
import os
|
|
56
|
+
from faster_whisper import WhisperModel
|
|
57
|
+
m = WhisperModel(os.environ['DRS_MODEL'], device='auto', compute_type='int8')
|
|
58
|
+
segs, info = m.transcribe(os.environ['DRS_AUDIO'], vad_filter=True, language=(os.environ.get('DRS_LANG') or None))
|
|
59
|
+
with open(os.environ['DRS_OUT'], 'w', encoding='utf-8') as f:
|
|
60
|
+
for s in segs:
|
|
61
|
+
f.write(s.text.strip() + ' ')
|
|
62
|
+
print('DONE', info.language, round(info.duration, 1))
|
|
63
|
+
'@
|
|
64
|
+
[System.IO.File]::WriteAllText($pyScript, $py, (New-Object System.Text.UTF8Encoding($false)))
|
|
65
|
+
|
|
66
|
+
& $venvPy $pyScript
|
|
67
|
+
if ($LASTEXITCODE -ne 0) { throw 'transcription failed.' }
|
|
68
|
+
Write-Output ('TRANSCRIBED ' + (Get-Item $Out).Length + ' bytes to ' + $Out)
|
|
@@ -20,7 +20,7 @@ whenToUse: 用户提供一本书、论文(arXiv/PDF)、视频链接(YouTub
|
|
|
20
20
|
## 输入
|
|
21
21
|
|
|
22
22
|
用户提供:内容链接或本地文件路径。
|
|
23
|
-
可选参数:`type`(auto/book/paper/video/web)、`options`(minWords/fastMode/maxChunks/requireCitations/includeTimestamps/outputDir)。
|
|
23
|
+
可选参数:`type`(auto/book/paper/video/web)、`options`(minWords / fastMode / maxChunks / maxRetries / requireCitations / includeTimestamps / transcribe / cache / outputDir / tempDir)。
|
|
24
24
|
|
|
25
25
|
## 执行步骤
|
|
26
26
|
|
|
@@ -31,10 +31,13 @@ whenToUse: 用户提供一本书、论文(arXiv/PDF)、视频链接(YouTub
|
|
|
31
31
|
5. **波次2 并行精读**:每个分块一个子代理深度精读(Map)
|
|
32
32
|
6. **波次3 合并成稿**:整合为完整笔记(Reduce),内嵌质量自检
|
|
33
33
|
7. **质量校验**:可选重试(maxRetries),检查覆盖度/引用真实性/术语一致/格式完整/篇幅/语言
|
|
34
|
+
8. **落盘(必须,别跳过)**:workflow 工具只返回 `note`(笔记正文)与 `filePath`(建议路径)——**脚本本身没有文件系统权限**。拿到结果后要用 `write` 工具把 `note` 原样写入 `filePath`(目录不存在就先创建);不写就等于笔记丢了,用户只会看到一段返回文本
|
|
34
35
|
|
|
35
36
|
## 输出
|
|
36
37
|
|
|
37
|
-
|
|
38
|
+
workflow 返回 `{ ok, filePath, note, qualityPassed, qualityIssues, failedChunks, textSource }`。
|
|
39
|
+
|
|
40
|
+
> ⚠️ **`note` 就是完整笔记正文,`filePath` 只是建议路径**:脚本运行在无文件系统的沙箱里(DSH 的 workflow 契约:脚本只负责协调子代理),**它不会替你写文件**。必须由你(主代理)在拿到结果后用 `write` 工具把它写到 `filePath`——默认 `./output/`,可用 `options.outputDir` 指向 Obsidian 仓库。
|
|
38
41
|
|
|
39
42
|
```markdown
|
|
40
43
|
---
|
|
@@ -59,8 +62,17 @@ status: 已完成
|
|
|
59
62
|
|
|
60
63
|
## 视频说明
|
|
61
64
|
|
|
62
|
-
-
|
|
63
|
-
-
|
|
65
|
+
- **统一流水线(去三档)**:目标 = 拿到完整逐字稿再精读。
|
|
66
|
+
1. **有平台字幕**(B站 AI 字幕 / YouTube CC / yt-dlp CC)→ 直接用字幕(=全文,最快、零依赖)。
|
|
67
|
+
2. **无公开字幕** → 用插件自带转写(`scripts/transcribe.ps1`:faster-whisper small / int8 / VAD)得到全文。
|
|
68
|
+
3. 都不行 → 降级提示用户手动提供转写文本,或退回 desc 作背景;**绝不阻塞、绝不自动装重依赖**。
|
|
69
|
+
- `options.transcribe`:**默认 true**(无字幕自动转写);设为 `false` 则跳过转写(只用字幕/desc 或降级)。
|
|
70
|
+
- **转写工具链**:脚本自举,本机/用户一致——`uv` 建 Python 3.12 环境 + 清华镜像装 `faster-whisper` + `HF_ENDPOINT=https://hf-mirror.com` 下模型并缓存;faster-whisper 内置 PyAV 解码音频,**无需单独 ffmpeg**。
|
|
71
|
+
- **⚠️ 首次转写会先下载 small 模型(约 484MB,hf-mirror,非 GitHub),可能较慢**——执行转写前**先向用户说明**这是正常的一次性下载,之后缓存复用、秒开;不要当成卡死而中断。
|
|
72
|
+
- **质量/速度**:small/int8 + VAD 静音过滤,中文质量可用且 CPU 友好;有字幕视频不转写(快)。装不上/失败则降级,不静默。
|
|
73
|
+
- YouTube 用 yt-dlp 抓 CC;未装时**绝不下载 exe 二进制**(GitHub 直连易卡死),改用 winget/pip;仍不可用则转写或降级。
|
|
74
|
+
- 运行 yt-dlp 加 `--socket-timeout 15 --retries 3` 防超时。
|
|
75
|
+
- **平台限制**:无字幕自动转写依赖 `scripts/transcribe.ps1`(PowerShell + `uv` + Python 3.12),**目前仅在 Windows 可用**;macOS/Linux 请让用户提供转写文本,或用平台字幕/desc 降级。
|
|
64
76
|
- 默认不标时间戳(`options.includeTimestamps` 可开)
|
|
65
77
|
|
|
66
78
|
## 参考
|
package/tests/run-tests.js
CHANGED
|
@@ -79,6 +79,34 @@ test('fixture fable.txt exists and is public-domain self-authored', () => {
|
|
|
79
79
|
assert.ok(!fx.includes('Copyright ©'));
|
|
80
80
|
});
|
|
81
81
|
|
|
82
|
+
// ---------- Test: plugin entry contract (Cordis) ----------
|
|
83
|
+
// Regression for 0.3.3: the entry exported only { load, workflow, parsers,
|
|
84
|
+
// schemas } - no apply method - so dsh rejected the plugin at startup
|
|
85
|
+
// ("invalid plugin, expect function or object with an \"apply\" method"),
|
|
86
|
+
// the web server exited immediately, and the launcher just sat waiting on a
|
|
87
|
+
// dead port. The entry must satisfy the Cordis contract, always.
|
|
88
|
+
const plugin = require('../index');
|
|
89
|
+
|
|
90
|
+
test('plugin entry satisfies the Cordis contract (function or object with apply)', () => {
|
|
91
|
+
const ok = typeof plugin === 'function' ||
|
|
92
|
+
(plugin && typeof plugin === 'object' && typeof plugin.apply === 'function');
|
|
93
|
+
assert.ok(ok, 'entry must be a function or an object with an apply method');
|
|
94
|
+
assert.ok(typeof plugin.apply === 'function', 'apply must be callable');
|
|
95
|
+
});
|
|
96
|
+
|
|
97
|
+
test('plugin entry carries name + inject for the bundle row', () => {
|
|
98
|
+
assert.ok(typeof plugin.name === 'string' && plugin.name !== '', 'name must be a non-empty string');
|
|
99
|
+
assert.ok(Array.isArray(plugin.inject) && plugin.inject.every((s) => typeof s === 'string'),
|
|
100
|
+
'inject must be an array of strings');
|
|
101
|
+
});
|
|
102
|
+
|
|
103
|
+
test('plugin entry keeps the legacy load/workflow/parsers exports', () => {
|
|
104
|
+
assert.ok(typeof plugin.load === 'function', 'legacy load() export missing');
|
|
105
|
+
assert.ok(plugin.workflow && plugin.workflow.meta, 'workflow export missing');
|
|
106
|
+
assert.ok(plugin.parsers && typeof plugin.parsers.resolve === 'function', 'parsers export missing');
|
|
107
|
+
assert.ok(plugin.schemas, 'schemas export missing');
|
|
108
|
+
});
|
|
109
|
+
|
|
82
110
|
// ---------- Test: security scanner (negative/positive samples) ----------
|
|
83
111
|
const { scanDir } = require('../scripts/security-check');
|
|
84
112
|
test('security scanner flags fake secrets and skips clean/ignored files', () => {
|
|
@@ -98,6 +126,140 @@ test('security scanner flags fake secrets and skips clean/ignored files', () =>
|
|
|
98
126
|
}
|
|
99
127
|
});
|
|
100
128
|
|
|
129
|
+
// ---------- Test: workflow script contract(DSH workflow 工具)----------
|
|
130
|
+
// 现行契约(0.1.2-rc.1):agent/parallel/pipeline/phase/log/args 六个钩子;agent() 选项只允许
|
|
131
|
+
// label/phase/schema/provider/model(其它选项会被引擎判为 fatal);schema 只允许
|
|
132
|
+
// type/properties/required/additionalProperties/items/enum/const/oneOf + 注解键;
|
|
133
|
+
// 脚本沙箱没有文件系统/网络/定时器/require。任何一条违反都会在真机上直接终止整条 workflow。
|
|
134
|
+
const workflow = require('../workflow');
|
|
135
|
+
const wfScript = workflow.script;
|
|
136
|
+
|
|
137
|
+
const ALLOWED_SCHEMA_KEYWORDS = [
|
|
138
|
+
'type', 'properties', 'required', 'additionalProperties', 'items', 'enum', 'const', 'oneOf',
|
|
139
|
+
'description', 'title', 'default', 'examples',
|
|
140
|
+
];
|
|
141
|
+
const ALLOWED_AGENT_OPTIONS = ['label', 'phase', 'schema', 'provider', 'model'];
|
|
142
|
+
|
|
143
|
+
/** 从脚本里取出 \`const XSchema = { ... }\` 的字面量并求值(脚本里的 schema 只是对象字面量)。 */
|
|
144
|
+
function extractSchemaLiterals(text) {
|
|
145
|
+
const out = [];
|
|
146
|
+
const re = /const (\w*[Ss]chema) = \{/g;
|
|
147
|
+
let m;
|
|
148
|
+
while ((m = re.exec(text)) !== null) {
|
|
149
|
+
const from = text.indexOf('{', m.index);
|
|
150
|
+
let depth = 0;
|
|
151
|
+
let i = from;
|
|
152
|
+
for (; i < text.length; i += 1) {
|
|
153
|
+
if (text[i] === '{') depth += 1;
|
|
154
|
+
else if (text[i] === '}') { depth -= 1; if (depth === 0) break; }
|
|
155
|
+
}
|
|
156
|
+
out.push({ name: m[1], value: eval('(' + text.slice(from, i + 1) + ')') }); // eslint-disable-line no-eval
|
|
157
|
+
}
|
|
158
|
+
return out;
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
/**
|
|
162
|
+
* 递归校验一个 JSON Schema 节点:只允许 DSH 子集的关键字。
|
|
163
|
+
* 只有 schema 位置的关键字受限——\`properties\` 下的键名是数据字段名,不参与校验。
|
|
164
|
+
* @returns {string[]} 违规关键字(含路径)
|
|
165
|
+
*/
|
|
166
|
+
function unsupportedSchemaKeywords(node, path = '$') {
|
|
167
|
+
if (node === null || typeof node !== 'object' || Array.isArray(node)) return [];
|
|
168
|
+
const bad = [];
|
|
169
|
+
for (const key of Object.keys(node)) {
|
|
170
|
+
if (!ALLOWED_SCHEMA_KEYWORDS.includes(key)) bad.push(path + '.' + key);
|
|
171
|
+
}
|
|
172
|
+
if (node.properties && typeof node.properties === 'object') {
|
|
173
|
+
for (const [name, child] of Object.entries(node.properties)) {
|
|
174
|
+
bad.push(...unsupportedSchemaKeywords(child, path + '.properties.' + name));
|
|
175
|
+
}
|
|
176
|
+
}
|
|
177
|
+
if (node.items) bad.push(...unsupportedSchemaKeywords(node.items, path + '.items'));
|
|
178
|
+
if (Array.isArray(node.oneOf)) node.oneOf.forEach((child, i) => bad.push(...unsupportedSchemaKeywords(child, path + '.oneOf[' + i + ']')));
|
|
179
|
+
return bad;
|
|
180
|
+
}
|
|
181
|
+
|
|
182
|
+
test('workflow: agent() schemas stay inside the DSH-supported subset', () => {
|
|
183
|
+
// 违反子集 = 引擎抛 UNSUPPORTED_SCHEMA = 整条 workflow 直接终止(不是降级)。
|
|
184
|
+
const schemas = extractSchemaLiterals(wfScript);
|
|
185
|
+
assert.ok(schemas.length >= 2, 'expected the fetch/check schemas, found ' + schemas.length);
|
|
186
|
+
for (const s of schemas) {
|
|
187
|
+
assert.deepStrictEqual(unsupportedSchemaKeywords(s.value), [],
|
|
188
|
+
s.name + ' uses unsupported schema keyword(s)');
|
|
189
|
+
}
|
|
190
|
+
});
|
|
191
|
+
|
|
192
|
+
test('workflow: the schema-subset checker actually rejects an unsupported keyword', () => {
|
|
193
|
+
// 反向断言:确保上面的“通过”不是因为校验器写空了。
|
|
194
|
+
const bad = unsupportedSchemaKeywords({ type: 'object', properties: { a: { type: 'string', minLength: 3 } } });
|
|
195
|
+
assert.deepStrictEqual(bad, ['$.properties.a.minLength']);
|
|
196
|
+
});
|
|
197
|
+
|
|
198
|
+
test('workflow: agent() options stay inside the engine whitelist', () => {
|
|
199
|
+
const re = /agent\([^,]*,\s*\{([^}]*)\}/g;
|
|
200
|
+
let m;
|
|
201
|
+
let calls = 0;
|
|
202
|
+
while ((m = re.exec(wfScript)) !== null) {
|
|
203
|
+
calls += 1;
|
|
204
|
+
const keys = [];
|
|
205
|
+
const keyRe = /([A-Za-z_][A-Za-z0-9_]*)\s*:/g;
|
|
206
|
+
let k;
|
|
207
|
+
while ((k = keyRe.exec(m[1])) !== null) keys.push(k[1]);
|
|
208
|
+
for (const key of keys) {
|
|
209
|
+
assert.ok(ALLOWED_AGENT_OPTIONS.includes(key),
|
|
210
|
+
'agent() option "' + key + '" is not supported by the engine (allowed: ' + ALLOWED_AGENT_OPTIONS.join(', ') + ')');
|
|
211
|
+
}
|
|
212
|
+
}
|
|
213
|
+
assert.ok(calls >= 5, 'expected the script agent() calls, found ' + calls);
|
|
214
|
+
});
|
|
215
|
+
|
|
216
|
+
test('workflow: the script never reaches for APIs the sandbox does not provide', () => {
|
|
217
|
+
const forbidden = [
|
|
218
|
+
[/\brequire\s*\(/, 'require()'],
|
|
219
|
+
[/\bprocess\s*\./, 'process'],
|
|
220
|
+
[/\bfs\s*\./, 'fs'],
|
|
221
|
+
[/\bfetch\s*\(/, 'fetch()'],
|
|
222
|
+
[/\bsetTimeout\s*\(/, 'setTimeout()'],
|
|
223
|
+
[/\bsetInterval\s*\(/, 'setInterval()'],
|
|
224
|
+
[/\b__dirname\b/, '__dirname'],
|
|
225
|
+
[/\bchild_process\b/, 'child_process'],
|
|
226
|
+
];
|
|
227
|
+
for (const [re, label] of forbidden) {
|
|
228
|
+
assert.ok(!re.test(wfScript), 'the workflow sandbox provides no ' + label);
|
|
229
|
+
}
|
|
230
|
+
});
|
|
231
|
+
|
|
232
|
+
test('workflow: the script self-contains all four parsers', () => {
|
|
233
|
+
for (const name of ['book', 'paper', 'video', 'web']) {
|
|
234
|
+
assert.ok(wfScript.includes(name + ': { buildPrompt:'), 'parser "' + name + '" is not inlined');
|
|
235
|
+
}
|
|
236
|
+
assert.ok(wfScript.includes('const parsers = args._parsers || __parsers;'), 'the args._parsers fallback is missing');
|
|
237
|
+
});
|
|
238
|
+
|
|
239
|
+
// ---------- Test: skill content(复用本文件顶部已有的 plugin 引用)----------
|
|
240
|
+
test('skill content carries meta + script + args and the write-back step', () => {
|
|
241
|
+
const content = plugin.buildSkillContent();
|
|
242
|
+
assert.ok(content.includes(workflow.meta.name), 'meta missing from the skill body');
|
|
243
|
+
assert.ok(content.includes(wfScript), 'the workflow script must be embedded verbatim');
|
|
244
|
+
assert.ok(content.includes('"input"'), 'the args example is missing');
|
|
245
|
+
// 脚本没有文件系统权限:技能必须明确要求主代理把 note 落盘,否则笔记会丢。
|
|
246
|
+
assert.ok(/filePath/.test(content) && /write/.test(content), 'the skill must tell the model to persist note -> filePath');
|
|
247
|
+
});
|
|
248
|
+
|
|
249
|
+
test('plugin entry satisfies the Cordis contract and registers one bundled skill', () => {
|
|
250
|
+
assert.strictEqual(plugin.name, 'deep-read-summarize');
|
|
251
|
+
assert.deepStrictEqual(plugin.inject, ['skills']);
|
|
252
|
+
assert.strictEqual(typeof plugin.apply, 'function');
|
|
253
|
+
const registered = [];
|
|
254
|
+
const ctx = { skills: { register: (skill) => { registered.push(skill); return () => {}; } } };
|
|
255
|
+
const dispose = plugin.apply(ctx, {});
|
|
256
|
+
assert.strictEqual(registered.length, 1);
|
|
257
|
+
assert.strictEqual(registered[0].name, 'deep-read-summarize');
|
|
258
|
+
assert.strictEqual(registered[0].source, 'bundled');
|
|
259
|
+
assert.ok(registered[0].content.length > 1000);
|
|
260
|
+
assert.strictEqual(typeof dispose, 'function');
|
|
261
|
+
});
|
|
262
|
+
|
|
101
263
|
// ---------- Run cache tests ----------
|
|
102
264
|
console.log('\n=== cache tests ===');
|
|
103
265
|
const cacheResult = require('./cache.test.js');
|
package/workflow.js
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
// Version: 3.0.0
|
|
4
4
|
// ============================================================================
|
|
5
5
|
// 【用法】meta → workflow meta;script → workflow script;
|
|
6
|
-
// args: { "input": "<url|path>", "type": "auto", "options": { "minWords": 2500, "fastMode": false, "maxChunks":
|
|
6
|
+
// args: { "input": "<url|path>", "type": "auto", "options": { "minWords": 2500, "fastMode": false, "maxChunks": 4, "transcribe": true, "requireCitations": true, "includeTimestamps": false, "cache": true, "outputDir": "./output", "tempDir": "./.tmp" } }
|
|
7
7
|
// ============================================================================
|
|
8
8
|
|
|
9
9
|
const meta = {
|
|
@@ -35,7 +35,7 @@ const meta = {
|
|
|
35
35
|
};
|
|
36
36
|
|
|
37
37
|
// ===== script(复制到 workflow 工具的 script 参数)=====
|
|
38
|
-
const RAW_SCRIPT = "// ========== deep-read-summarize v3.0 ==========\n// 3 波次:解析器获取分块 → 并行精读 → 成稿+质量校验\n// 特性:解析器插件化 / 失败纪律(fatal vs 降级) / 幂等 / 成本控制\n\n// ---------- 0. 配置校验(fatal 级错误:必须终止) ----------\nconst input = (args && args.input || \"\").trim();\nif (!input) throw new Error(\"[FATAL] args.input 缺失:请输入内容链接或文件路径\");\nconst forcedType = ((args && args.type) || \"auto\").toLowerCase();\nif ([\"auto\",\"book\",\"paper\",\"video\",\"web\"].indexOf(forcedType) === -1)\n throw new Error(\"[FATAL] args.type 非法:必须为 auto|book|paper|video|web\");\nconst options = (args && args.options) || {};\nconst minWords = options.minWords || 2500;\nconst fastMode = !!options.fastMode;\nconst maxChunks = Math.max(1, Math.min(options.maxChunks ||
|
|
38
|
+
const RAW_SCRIPT = "// ========== deep-read-summarize v3.0 ==========\n// 3 波次:解析器获取分块 → 并行精读 → 成稿+质量校验\n// 特性:解析器插件化 / 失败纪律(fatal vs 降级) / 幂等 / 成本控制\n\n// ---------- 0. 配置校验(fatal 级错误:必须终止) ----------\nconst input = (args && args.input || \"\").trim();\nif (!input) throw new Error(\"[FATAL] args.input 缺失:请输入内容链接或文件路径\");\nconst forcedType = ((args && args.type) || \"auto\").toLowerCase();\nif ([\"auto\",\"book\",\"paper\",\"video\",\"web\"].indexOf(forcedType) === -1)\n throw new Error(\"[FATAL] args.type 非法:必须为 auto|book|paper|video|web\");\nconst options = (args && args.options) || {};\nconst minWords = options.minWords || 2500;\nconst fastMode = !!options.fastMode;\nconst maxChunks = Math.max(1, Math.min(options.maxChunks || 4, 12));\nconst requireCitations = options.requireCitations !== false;\nconst includeTimestamps = !!options.includeTimestamps;\nconst maxRetries = options.maxRetries || 0;\nconst cacheEnabled = options.cache !== false;\nconst outputDir = options.outputDir || \"./output\";\nconst tempDir = options.tempDir || \"./.tmp\";\nconst tempFile = tempDir.replace(/[\\/]+$/, \"\") + \"/input.txt\";\n\n// ---------- 0.5 幂等检查:输入指纹缓存 ----------\nlet cacheHit = false;\nif (cacheEnabled && Array.isArray(args._processedKeys)) {\n const fpKey = input.trim().slice(0, 300);\n cacheHit = args._processedKeys.indexOf(fpKey) !== -1;\n if (cacheHit) log(\"幂等命中:该输入已处理过,跳过\");\n}\nif (cacheHit) {\n return { ok: true, cached: true, kind: forcedType, title: (args && args.title) || \"cached\", filePath: \"\", chunksCount: 0, fastMode: fastMode, qualityPassed: true, qualityIssues: [], failedChunks: 0, note: \"\" };\n}\n\n// ---------- 1. 解析器选择(由 args._parsers 注入) ----------\nphase(\"解析器选择\");\nconst parsers = args._parsers || {};\nconst kindNames = { book: \"书籍\", paper: \"学术论文\", video: \"视频\", web: \"网页/文章\" };\nconst kindFocus = {\n book: \"按章节主题展开核心论点、论证、关键例子、章节间关系,收集关键概念与可引用原文(含页码/章节)\",\n paper: \"四维精读:1)研究问题与方法 2)结论与贡献 3)局限与批判 4)与同类工作比较;收集术语与可引用原文(含章节/段落)\",\n video: \"按主题脉络组织,提取核心论点、证据、例子、可操作建议,收集关键概念与主讲人原话\" + (includeTimestamps ? \"(含时间戳定位)\" : \"(不标时间戳)\"),\n web: \"提取核心论点、结构拆解、关键概念、可引用原文\"\n};\n\n// ---------- 2. 波次1:解析器获取 + 分块 ----------\nphase(\"获取与分块\");\nconst fetchPrompt = (parsers[forcedType !== \"auto\" ? forcedType : \"web\"] || parsers.web || { buildPrompt: function() { return [\"未知解析器\"]; } })\n .buildPrompt(input, { tempDir: tempDir, tempFile: tempFile, maxChunks: maxChunks, transcribe: options.transcribe !== false });\nconst fetchSchema = {\n type: \"object\",\n properties: {\n kind: { type: \"string\" }, saved: { type: \"boolean\" }, message: { type: \"string\" }, totalLines: { type: \"number\" },\n metadata: { type: \"object\", properties: { title: { type: \"string\" }, author: { type: \"string\" }, year: { type: \"string\" }, language: { type: \"string\" } }, required: [], additionalProperties: true },\n chunkPlan: {\n type: \"array\",\n items: { type: \"object\", properties: { id: { type: \"number\" }, topic: { type: \"string\" }, startLine: { type: \"number\" }, endLine: { type: \"number\" } }, required: [\"id\",\"topic\",\"startLine\",\"endLine\"], additionalProperties: false }\n }\n },\n required: [\"saved\"], additionalProperties: false\n};\n\nconst fetched = await agent(fetchPrompt, { schema: fetchSchema, label: \"获取+分块\", phase: \"获取与分块\" });\n\n// 失败纪律:内容获取失败 = 可降级错误(返回失败结果,不抛 fatal)\nif (!fetched || !fetched.saved) {\n return { ok: false, stage: \"fetch\", fatal: false, message: (fetched && fetched.message) || \"获取失败(可降级:检查输入或手动提供文本)\" };\n}\nlet kind = (fetched.kind && forcedType === \"auto\") ? fetched.kind : forcedType;\nconst chunkPlan = (fetched && fetched.chunkPlan) || [];\nif (chunkPlan.length === 0) return { ok: false, stage: \"fetch\", fatal: false, message: \"分块计划为空\" };\nconst metadata = (fetched && fetched.metadata) || {};\nconst textSource = (metadata && metadata.textSource) || \"unknown\";\nconst srcLabels = { desc: \"视频简介/desc\", subtitle: \"平台字幕\", transcription: \"完整语音转写\", manual: \"用户提供转写文本\" };\nconst srcLabel = srcLabels[textSource] || textSource;\nlog(\"[source] \" + textSource + \" -> \" + srcLabel);\nlog(\"类型:\" + kind + \" 行数:\" + fetched.totalLines + \" 块数:\" + chunkPlan.length);\n\n// ---------- 3. 波次2:并行精读(Map) ----------\nphase(\"并行精读\");\nconst chunkPrompts = chunkPlan.map(function (c) {\n return [\n \"请深度精读以下内容块,输出结构化精读笔记。\",\n \"内容类型:\" + (kindNames[kind] || kind) + \" | 本块主题:\" + c.topic,\n \"精读侧重:\" + (kindFocus[kind] || kindFocus.web),\n \"用 read 工具读取 \" + tempFile + \" 第 \" + c.startLine + \" 到 \" + c.endLine + \" 行(offset=\" + c.startLine + \", limit=\" + (c.endLine - c.startLine + 1) + \"),然后输出:\",\n \"## 本块主题:\" + c.topic,\n \"### 要点(详细展开,保留数据/例子/论证)\",\n \"### 关键概念(术语+解释)\",\n \"### 关键引用(最多2句原文+位置)\" + (requireCitations ? \",必须标注页码/章节/段落位置\" : \"\"),\n \"### 批判观察(如有)\"\n ].join(\"\\n\");\n});\n\nconst reads = [];\nconst readBatch = 3;\nfor (let bi = 0; bi < chunkPrompts.length; bi += readBatch) {\n const batch = chunkPrompts.slice(bi, bi + readBatch);\n const batchReads = await parallel(batch.map(function (p) {\n return function () { return agent(p, { label: \"精读块\", phase: \"并行精读\" }); };\n }));\n for (let k = 0; k < batchReads.length; k++) reads.push(batchReads[k]);\n}\nconst validReads = reads.filter(function (r) { return r && r.trim && r.trim().length > 0; });\n\n// 失败纪律:某块精读失败 = 可降级(跳过该块,标记缺口继续)\nconst failedChunks = chunkPlan.length - validReads.length;\nif (validReads.length === 0) return { ok: false, stage: \"read\", fatal: false, message: \"所有分块精读均失败\" };\nif (failedChunks > 0) log(\"警告:\" + failedChunks + \" 块精读失败,标记缺口继续成稿\");\nlog(\"精读完成:\" + validReads.length + \"/\" + chunkPlan.length);\n\n// ---------- 4. 波次3:成稿 + 内嵌质量自检(Reduce) ----------\nphase(\"汇总成稿\");\nconst today = new Date().toISOString().slice(0, 10);\nconst finalTitle = metadata.title || \"未命名内容\";\nconst safeTitle = finalTitle.replace(/[\\\\/:*?\"<>|]/g, \" \").trim();\nconst mdPath = outputDir.replace(/\\\\$/, \"\") + \"/\" + safeTitle + \".md\";\n\nconst extraSections = fastMode ? [] : [\n \"\", \"## 5. 批判性分析(论证强处/弱处/可质疑之处)\", \"\",\n \"## 6. 个人思考与应用(与已有知识联系、如何应用、行动要点)\", \"\",\n \"## 7. 延伸阅读(相关主题/作者其他作品/参考文献)\", \"\"\n];\n\nconst draftPrompt = [\n \"将以下多块精读结果整合为完整 Obsidian 精读笔记,只输出 Markdown(含 YAML frontmatter)。\",\n \"模板:\", \"---\",\n 'title: \"《' + finalTitle + '》深度精读笔记\"',\n \"aliases: [\" + JSON.stringify(finalTitle) + \"]\",\n \"author: \" + (metadata.author || \"\"),\n \"year: \" + (metadata.year || \"\"),\n \"type: \" + kind,\n \"url: \" + (metadata.url || input),\n \"tags: [精读, \" + (kindNames[kind] || kind) + \"]\",\n \"created: \" + today,\n \"status: 已完成\",\n \"---\",\n \"# 《\" + finalTitle + \"》深度精读笔记\", \"\",\n \"> 一句话核心概括。\", \"\",\n \"## 1. 概述(约500字:主题/背景/核心问题/总体结论)\", \"\",\n \"## 2. 结构拆解(按主题组织,含详细要点;关键结论标注原文出处)\", \"\",\n \"## 3. 关键概念与术语表(表格)\", \"\",\n \"## 4. 关键引用(仅关键结论\" + (requireCitations ? \",必须标注页码/章节/段落\" : \"\") + \")\",\n].concat(extraSections, [\n \"---\",\n \"*笔记整理于 \" + today + \"*\", \"\",\n \"硬性要求:总篇幅≥\" + minWords + \"字;语言与原文一致;术语统一;引用必须真实;视频\" + (includeTimestamps ? \"标注时间戳\" : \"不标注时间戳\") + \"。\",\n \"输出前自检:1)覆盖所有块 2)frontmatter完整 3)各小节齐全 4)引用真实 5)篇幅达标 6)语言一致。\",\n \"\",\n \"各块精读结果:\",\n validReads.map(function (r, i) { return \"\\\\n===== 块 \" + (i + 1) + \" =====\\\\n\" + r; }).join(\"\\\\n\")\n]).join(\"\\n\");\n\nlet draft = await agent(draftPrompt, { label: \"汇总成稿\", phase: \"汇总成稿\" });\nif (!draft || !draft.trim) return { ok: false, stage: \"draft\", fatal: false, message: \"成稿失败\" };\n\n// ---------- 5. 质量校验(轻量重试) ----------\nphase(\"质量校验\");\nlet verified = draft;\nlet passed = false;\nlet issues = [];\nfor (let round = 0; round <= maxRetries; round++) {\n const checkPrompt = [\n \"请对总结笔记做质量校验,逐项检查并输出 JSON:\",\n \"1.coverage 2.citationsReal 3.terminologyConsistent 4.formatComplete 5.lengthMet(≥\" + minWords + \"字) 6.languageConsistent\",\n \"输出:{ \\\"pass\\\": true/false, \\\"issues\\\": [\\\"问题\\\"] },只输出 JSON。\",\n \"笔记:\", verified\n ].join(\"\\n\");\n const checkSchema = { type: \"object\", properties: { pass: { type: \"boolean\" }, issues: { type: \"array\", items: { type: \"string\" } } }, required: [\"pass\",\"issues\"], additionalProperties: false };\n const check = await agent(checkPrompt, { schema: checkSchema, label: \"质量校验\", phase: \"质量校验\" });\n passed = !!(check && check.pass);\n issues = (check && check.issues) || [];\n log(\"校验:\" + (passed ? \"通过\" : \"发现问题 \" + issues.length + \" 条\"));\n if (passed) break;\n if (round < maxRetries) {\n const fixPrompt = [\n \"修正以下笔记以解决校验问题,输出完整修正版:\",\n issues.map(function (s) { return \"- \" + s; }).join(\"\\n\"),\n \"当前笔记:\", verified\n ].join(\"\\n\");\n verified = await agent(fixPrompt, { label: \"修正笔记\", phase: \"质量校验\" });\n }\n}\n\n// ---------- 结果 ----------\nreturn {\n ok: true, kind: kind, title: finalTitle,\n filePath: mdPath, chunksCount: chunkPlan.length, fastMode: fastMode,\n qualityPassed: passed, qualityIssues: issues,\n failedChunks: failedChunks,\n textSource: textSource,\n note: verified\n};";
|
|
39
39
|
|
|
40
40
|
// ===== parsers 注册表(运行时注入,实现插件化)=====
|
|
41
41
|
const parsers = require("./parsers");
|