deep-read-summarize 0.3.19 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +34 -0
- package/README.en.md +23 -10
- package/README.md +23 -10
- package/cordis.patch.yml +2 -1
- package/docs/RELEASE.md +2 -1
- package/index.js +139 -9
- package/lib/cache.js +5 -12
- package/lib/quality.js +25 -5
- package/lib/workflow-script.txt +116 -32
- package/package.json +2 -1
- package/parsers/index.js +23 -7
- package/parsers/video.js +24 -6
- package/scripts/transcribe.js +104 -15
- package/skills/deep-read-summarize/SKILL.md +8 -6
- package/workflow.js +5 -6
package/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,40 @@
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
The format is based on [Keep a Changelog](https://keepachangelog.com/).
|
|
5
5
|
|
|
6
|
+
## [0.4.0] — 2026-10-25
|
|
7
|
+
|
|
8
|
+
保留完整精读与独立原文核验,减少脚本搬运、并发空等和无效修订,并修复来源质量判断与视频分P等问题。
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- **参数直达工具 `deep_read_summarize`**:宿主具有 tools / workflowEngine 服务时,由插件直接提交工作流,模型只传 input/type/options,无需原样重发约 4.6 万字符脚本。转发调用代理和取消信号,完成或失败后释放 workflow run;旧宿主保留原 workflow + skill 入口。
|
|
13
|
+
- **阶段耗时 `timings`**:返回识别、获取、精读、成稿、质量核验和总耗时(毫秒),供同资料、模型与质量门槛下比较。测试调度模拟不代表真实 LLM 加速比例。
|
|
14
|
+
- **显式自定义解析器目录**:插件顶层 `config.customParsersDir` 可加载独立目录,各 profile 隔离;包内 `custom-parsers/` 纳入 npm 分发。
|
|
15
|
+
|
|
16
|
+
### Fixed
|
|
17
|
+
|
|
18
|
+
- **视频来源质量**:只有简介或未确认来源的视频保持质量降级,不能因“简介的所有行都读过”而报告完整精读通过;来源类型进入 QC 上下文,并在笔记中标明限制。
|
|
19
|
+
- **B站多P**:保留短链最终 URL 与 p,按 `pages[p-1]` 选择 CID、标题和时长;字幕、音频使用同一分P,非法或越界参数不能默默回退第1P。
|
|
20
|
+
- **异常与重试**:类型识别和获取的可恢复调用错误返回对应 stage,真正 FATAL 继续终止;缺总行数、失败精读块、来源不完整等不能靠改写解决的问题不再消耗全文修订轮次。
|
|
21
|
+
- **格式与引用**:严格检查模板字符串,拒绝非法 YAML 裸标量和错误字段类型;区分 Markdown 引文的可见原话与强调、行号等展示信息,减少真实引用被误判。
|
|
22
|
+
- **文件名**:过滤控制字符、设备保留名与非法尾缀;对清洗或截短的标题增加稳定后缀,限制 UTF-8 文件名长度,避免 NUL、超长标题落盘失败与不同非法标题折叠到同名路径。
|
|
23
|
+
- **URL 指纹**:保留可选择不同资料的尾斜杠、空 query 与 fragment;新版键与旧缓存隔离,防止旧版碰撞继续跳过资料。
|
|
24
|
+
- **解析器更新**:注册后生成技能和工具调用使用最新注册表,不再引用模块加载时的旧脚本;插件构建阶段定位本版本转写脚本,避免每次递归搜索其他安装目录。
|
|
25
|
+
- **转写热启动**:健康环境直接执行一次 Python runner,移除单独 import 探测;仅明确缺依赖或版本不匹配时才要求 uv 并自举,已有可用环境不再被“找不到 uv”提前拒绝。
|
|
26
|
+
- **转写版本记录**:固定已验证的 `faster-whisper==1.2.1`,记录实际 Python / 依赖版本;纠正文档“uv / Python补丁 / 全部依赖完全锁定”的不实承诺。
|
|
27
|
+
|
|
28
|
+
### Performance
|
|
29
|
+
|
|
30
|
+
- **固定三个并发槽位**:一个任务完成即领取下一块,取消每批三个都结束才启动下一批的屏障,输出仍按原分块顺序合并。
|
|
31
|
+
- **均衡分块工作量**:在既有代理数和 maxChunks 上限内按行数均分已知全文,超长逻辑段可拆分,保持每行恰好覆盖一次。
|
|
32
|
+
- **缩减 QC 上下文**:保留原文路径、行号、缺口与待核验引文,移除重复的完整 Map 笔记;独立核验仍须读取原文。
|
|
33
|
+
|
|
34
|
+
### Compatibility
|
|
35
|
+
|
|
36
|
+
- 默认模型、篇幅、章节与引用门槛不降低;新工具是可选入口,原入口保留。新返回字段为补充元数据。
|
|
37
|
+
- 新版缓存指纹可能导致升级后的首次重处理;清洗过或过长标题的建议文件名会改变,正常标题保持不变。
|
|
38
|
+
- 原文真实性仍取决于核验代理实际读取原文。逐字稿内容缓存、常驻 ASR worker、分层 QC 和共享转写环境的冷启动互斥留待单独验证,未在本次发布中声称完成。
|
|
39
|
+
|
|
6
40
|
## [0.3.19] — 2026-10-24
|
|
7
41
|
|
|
8
42
|
修复 B04–B10,并让质量检查结合程序门禁和可回查的原文证据(O01)。
|
package/README.en.md
CHANGED
|
@@ -97,7 +97,7 @@ workflow.js the workflow meta and assembly logic (parser inlining, schema i
|
|
|
97
97
|
|
|
98
98
|
> The script body lives in `lib/workflow-script.txt` on purpose: this file was corrupted twice by "escaping on top of escaping". It is plain text now, so reading and editing it needs no encoding step.
|
|
99
99
|
|
|
100
|
-
To
|
|
100
|
+
To override or add a parser, use the package's `custom-parsers/` directory (included in npm packaging), or set the plugin-level `config.customParsersDir` to your own directory. This is not a workflow option. Per-profile directories remain isolated, and rebuilding skill content uses the current registry, including programmatic registrations. Parsers expose `name`, `types`, and `buildPrompt(input, opts)`; registered types are inlined into the workflow and its schema.
|
|
101
101
|
|
|
102
102
|
File parsers bundle their CommonJS modules and literal relative `.js`, `.cjs`, and `.json` dependencies, preserving module constants, helpers, method shorthand, and cycles. Acorn parses JavaScript during script construction and is installed automatically by npm; it does not run in the workflow sandbox. Node built-ins, external packages, and dynamic `require`/`import` are unsupported in sandbox parsers and fail at build time. Programmatically registered functions must be self-contained; use a file parser for module closures.
|
|
103
103
|
|
|
@@ -136,7 +136,8 @@ pnpm add ./deep-read-summarize-0.3.10.tgz
|
|
|
136
136
|
```
|
|
137
137
|
|
|
138
138
|
After installation it registers:
|
|
139
|
-
- the `
|
|
139
|
+
- the `deep_read_summarize` tool when the host provides tools and workflowEngine services; models send only input/type/options
|
|
140
|
+
- the `deep-read-summarize` workflow definition (meta + script), with the existing workflow-tool route retained for older hosts
|
|
140
141
|
- the `deep-read-summarize` skill (`skills/deep-read-summarize/SKILL.md`)
|
|
141
142
|
- four parsers (`parsers/`) and JSON Schemas (`schemas/`)
|
|
142
143
|
|
|
@@ -150,18 +151,18 @@ After installation it registers:
|
|
|
150
151
|
npm install deep-read-summarize
|
|
151
152
|
|
|
152
153
|
# Option B: develop from source
|
|
153
|
-
git clone https://github.com
|
|
154
|
+
git clone https://github.com/PensiveFei/deep-read-summarize.git
|
|
154
155
|
cd deep-read-summarize
|
|
155
156
|
npm ci # install locked dependencies
|
|
156
157
|
```
|
|
157
158
|
|
|
158
|
-
### 2. Verify
|
|
159
|
+
### 2. Verify installation (offline, no API key needed)
|
|
159
160
|
|
|
160
161
|
```bash
|
|
161
|
-
|
|
162
|
+
node -e "console.log(require('deep-read-summarize').name)"
|
|
162
163
|
```
|
|
163
164
|
|
|
164
|
-
Seeing `
|
|
165
|
+
Seeing `deep-read-summarize` confirms the package loads. Source checkouts can additionally run `npm test` and `npm run test:node`; tests are not included in the npm package.
|
|
165
166
|
|
|
166
167
|
### 3. Feed a demo file with one command
|
|
167
168
|
|
|
@@ -176,9 +177,9 @@ meta: deep-read-summarize
|
|
|
176
177
|
parsers: book, paper, video, web
|
|
177
178
|
```
|
|
178
179
|
|
|
179
|
-
### 4. Real run (requires
|
|
180
|
+
### 4. Real run (requires DSH)
|
|
180
181
|
|
|
181
|
-
|
|
182
|
+
Prefer passing the JSON below directly to `deep_read_summarize`. On older hosts, use the skill's meta + script with the existing `workflow` tool:
|
|
182
183
|
|
|
183
184
|
```jsonc
|
|
184
185
|
{ "input": "https://arxiv.org/abs/2307.09042", "type": "paper", "options": { "maxChunks": 4, "fastMode": true } }
|
|
@@ -191,7 +192,7 @@ The workflow returns a structured result: `{ ok, kind, title, filePath, qualityP
|
|
|
191
192
|
---
|
|
192
193
|
## Usage
|
|
193
194
|
|
|
194
|
-
|
|
195
|
+
The `deep_read_summarize` tool takes these parameters directly. On older hosts, use them as workflow args with the skill's meta + script:
|
|
195
196
|
|
|
196
197
|
```jsonc
|
|
197
198
|
{
|
|
@@ -265,7 +266,7 @@ Video text extraction is a **unified pipeline** (the old "three tiers" are gone)
|
|
|
265
266
|
|
|
266
267
|
`options.transcribe`: **default `true`** (auto-transcribe when no subtitles); set `false` to skip transcription (use subtitles/desc or degrade).
|
|
267
268
|
|
|
268
|
-
**Transcription toolchain
|
|
269
|
+
**Transcription toolchain**: `uv` provisions **Python 3.12** and installs **`faster-whisper==1.2.1`** from the Tsinghua mirror; models use `HF_ENDPOINT=https://hf-mirror.com` and the local cache. Healthy cached environments run directly without requiring uv or a separate Python import probe. Actual interpreter and dependency versions are recorded in the cache's `runtime.json`; Python patch releases, uv, and all transitive dependencies are not fully pinned. PyAV decodes audio without a separate ffmpeg. All three platforms use `node scripts/transcribe.js --audio <file> --out <txt>`. CPU is the default; `options.device: "auto"` can try a supplied CUDA runtime and falls back to CPU. Cache roots are `%LOCALAPPDATA%\deep-read-summarize` on Windows, `~/Library/Caches/deep-read-summarize` on macOS, and `$XDG_CACHE_HOME/deep-read-summarize` on Linux (falling back to `~/.cache`). Python provisioning tries a mirror and then direct GitHub; `DRS_UV_PYTHON_MIRROR` overrides it, with an empty value selecting direct downloads.
|
|
269
270
|
|
|
270
271
|
> ⚠️ **The first transcription downloads the model first (small ≈ 484 MB, via hf-mirror, not GitHub) and may take a few minutes** — please be patient; **it downloads only once** on your machine, then every later transcription uses the cache and starts instantly. **Only subtitle-less videos trigger transcription** (subtitled videos use their subtitles directly — no trigger, no download).
|
|
271
272
|
|
|
@@ -317,10 +318,22 @@ Two classes:
|
|
|
317
318
|
|
|
318
319
|
---
|
|
319
320
|
|
|
321
|
+
## Faster runs with the same quality gates
|
|
322
|
+
|
|
323
|
+
- Prefer `deep_read_summarize` on supported hosts: the model submits parameters instead of reproducing the entire workflow script. Older hosts retain the existing route.
|
|
324
|
+
- All known source lines remain covered. Up to three reading tasks run concurrently, and each free slot immediately starts another task. Line-balanced chunks avoid one oversized task holding up a batch.
|
|
325
|
+
- Independent source review remains enabled, with duplicate full Map notes removed from review contexts. Missing line counts, failed reads, and description-only video sources cannot be fixed by rewriting the draft, so those states no longer consume full repair loops.
|
|
326
|
+
- Video `textSource: "desc"` is explicitly degraded and cannot pass as a complete deep read. Bilibili `p` selects the matching CID, part title, and duration.
|
|
327
|
+
- Returned `timings` reports stage durations in milliseconds. Compare the same sources, model, output size, and quality gates; scheduling simulations are not real LLM speedup measurements.
|
|
328
|
+
|
|
329
|
+
This release does not lower model size, output length, or source coverage. Persistent ASR workers, transcript caching, and hierarchical review need separate quality and performance benchmarks.
|
|
330
|
+
|
|
320
331
|
## Cost
|
|
321
332
|
|
|
322
333
|
Token use depends on document length, chunk count, and output size. Source review reads the document again, and each repair round adds a repair and a review call. `fastMode` reduces output sections; reducing `maxChunks` limits agent count while preserving the complete source.
|
|
323
334
|
|
|
335
|
+
Programmatic `lib/cache` callers: URL trailing slashes, explicit empty queries, and fragments are preserved because they may select different content. New fingerprints are isolated from old records, so an upgrade can cause one reprocessing pass. Callers must still verify source freshness and configuration before reusing results.
|
|
336
|
+
|
|
324
337
|
---
|
|
325
338
|
|
|
326
339
|
## Security & copyright
|
package/README.md
CHANGED
|
@@ -96,7 +96,7 @@ workflow.js workflow 的 meta 与组装逻辑(解析器内联、schema
|
|
|
96
96
|
|
|
97
97
|
> 脚本正文单独放 `lib/workflow-script.txt`:这个文件曾经因为「转义套转义」被改坏过两次,现在它是普通文本,读改都不需要编码步骤。
|
|
98
98
|
|
|
99
|
-
|
|
99
|
+
想换某个输入类型的处理方式,在插件包的 `custom-parsers/` 放一个同接口的解析器即可,该目录会随 npm 包分发。也可在插件顶层 config 中指定 `customParsersDir`,使用自己的解析器目录;它不是 workflow options。相同类型会覆盖内置实现,也可新增 `podcast` 等类型。程序化注册后重新生成技能会使用最新注册表;各 profile 的专属目录相互隔离。接口为 `name`、`types`、`buildPrompt(input, opts)`。
|
|
100
100
|
|
|
101
101
|
解析器的 CommonJS 模块会连同字面量相对路径的 `.js` / `.cjs` / `.json` 依赖一起打包,保留模块级常量、helper、方法简写和循环依赖。构建时使用 Acorn 解析 JavaScript;它由 npm 自动安装,不进入 workflow 沙箱。沙箱中的解析器只负责生成提示词,不支持 Node 内置模块、外部包或动态 `require` / `import`;这些依赖会在构建时明确报错。程序化注册的函数须自包含,模块闭包请使用文件解析器。
|
|
102
102
|
|
|
@@ -135,7 +135,8 @@ pnpm add ./deep-read-summarize-0.3.10.tgz
|
|
|
135
135
|
```
|
|
136
136
|
|
|
137
137
|
安装后自动注册:
|
|
138
|
-
- `
|
|
138
|
+
- `deep_read_summarize` 工具(宿主提供 tools / workflowEngine 服务时);模型只传 input/type/options,插件直接提交已加载的工作流
|
|
139
|
+
- `deep-read-summarize` 工作流定义(meta + script;旧宿主保留通过 workflow 工具调用的方式)
|
|
139
140
|
- `deep-read-summarize` 技能(`skills/deep-read-summarize/SKILL.md`)
|
|
140
141
|
- 四种解析器(`parsers/`)与 JSON Schema(`schemas/`)
|
|
141
142
|
|
|
@@ -149,18 +150,18 @@ pnpm add ./deep-read-summarize-0.3.10.tgz
|
|
|
149
150
|
npm install deep-read-summarize
|
|
150
151
|
|
|
151
152
|
# 方式 B:从源码开发
|
|
152
|
-
git clone https://github.com
|
|
153
|
+
git clone https://github.com/PensiveFei/deep-read-summarize.git
|
|
153
154
|
cd deep-read-summarize
|
|
154
155
|
npm ci # 安装锁定依赖
|
|
155
156
|
```
|
|
156
157
|
|
|
157
|
-
### 2.
|
|
158
|
+
### 2. 验证安装(离线,不需要任何 API key)
|
|
158
159
|
|
|
159
160
|
```bash
|
|
160
|
-
|
|
161
|
+
node -e "console.log(require('deep-read-summarize').name)"
|
|
161
162
|
```
|
|
162
163
|
|
|
163
|
-
看到 `
|
|
164
|
+
看到 `deep-read-summarize` 即包可正常加载。从源码开发时再运行 `npm test` 和 `npm run test:node`;测试不随 npm 包分发。
|
|
164
165
|
|
|
165
166
|
### 3. 一行命令喂入 demo 文件
|
|
166
167
|
|
|
@@ -175,9 +176,9 @@ meta: deep-read-summarize
|
|
|
175
176
|
parsers: book, paper, video, web
|
|
176
177
|
```
|
|
177
178
|
|
|
178
|
-
### 4. 真实运行(需要 DSH
|
|
179
|
+
### 4. 真实运行(需要 DSH)
|
|
179
180
|
|
|
180
|
-
|
|
181
|
+
优先把下面的 JSON 直接传给 `deep_read_summarize` 工具。旧宿主使用技能提供的 meta + script,通过 `workflow` 工具执行:
|
|
181
182
|
|
|
182
183
|
```jsonc
|
|
183
184
|
{ "input": "https://arxiv.org/abs/2307.09042", "type": "paper", "options": { "maxChunks": 4, "fastMode": true } }
|
|
@@ -190,7 +191,7 @@ parsers: book, paper, video, web
|
|
|
190
191
|
---
|
|
191
192
|
## 用法
|
|
192
193
|
|
|
193
|
-
|
|
194
|
+
`deep_read_summarize` 工具只需要以下参数;旧宿主将其作为 `workflow` 的 args,并使用技能中的 meta + script:
|
|
194
195
|
|
|
195
196
|
```jsonc
|
|
196
197
|
{
|
|
@@ -264,7 +265,7 @@ parsers: book, paper, video, web
|
|
|
264
265
|
|
|
265
266
|
`options.transcribe`:**默认 `true`**(无字幕自动转写);设为 `false` 则跳过转写(只用字幕/desc 或降级)。
|
|
266
267
|
|
|
267
|
-
|
|
268
|
+
**转写工具链**:脚本用 `uv` 创建 **Python 3.12** 环境,从清华镜像安装 **`faster-whisper==1.2.1`**,通过 `HF_ENDPOINT=https://hf-mirror.com` 下载并缓存模型。健康的缓存环境直接运行,不要求 uv 仍在 PATH,也不再启动单独的 Python 导入探测。实际解释器和依赖版本记录在缓存目录的 `runtime.json`;Python 补丁版本、uv 和传递依赖未全部锁定,不能据此声称环境逐字一致。PyAV 可直接解码音频,无需单独 ffmpeg。三平台使用同一条命令 `node scripts/transcribe.js --audio <音频> --out <txt>`,默认 CPU;`options.device: "auto"` 可尝试自备的 CUDA,失败回退 CPU。缓存目录:Windows `%LOCALAPPDATA%\deep-read-summarize`、macOS `~/Library/Caches/deep-read-summarize`、Linux `$XDG_CACHE_HOME/deep-read-summarize`(未设置时用 `~/.cache`)。创建 Python 环境默认先用镜像,失败再直连 GitHub;`DRS_UV_PYTHON_MIRROR` 可覆盖镜像,空字符串表示直连。
|
|
268
269
|
|
|
269
270
|
> ⚠️ **首次转写会先下载模型(small 约 484MB,走 hf-mirror,非 GitHub),可能需要几分钟**,请耐心等待;**本机只下这一次**,之后所有视频转写用缓存、秒开。**只有「无字幕」视频才会触发转写**(有字幕的视频直接用字幕,不触发、不下载)。
|
|
270
271
|
|
|
@@ -316,10 +317,22 @@ parsers: book, paper, video, web
|
|
|
316
317
|
|
|
317
318
|
---
|
|
318
319
|
|
|
320
|
+
## 保持质量的提速
|
|
321
|
+
|
|
322
|
+
- 支持的新宿主优先使用 `deep_read_summarize`,模型只提交参数,省去复制整份工作流脚本。旧宿主自动保留原入口。
|
|
323
|
+
- 精读仍完整覆盖已知原文,最多三个任务并发;空闲槽位立即领取下一块。按行数均衡工作量,避免一个超大块拖住整批。
|
|
324
|
+
- QC 继续独立读取原文;不再在每轮上下文中重复塞入全部 Map 笔记。缺总行数、精读失败或只有视频简介等问题不能靠改写笔记解决,不再浪费全文修订轮次。
|
|
325
|
+
- 视频 `textSource: "desc"` 明确为降级来源,不能取得完整精读的质量通过状态。B站多 P 链接按 `p` 选择对应 CID、分 P 标题和时长。
|
|
326
|
+
- 返回 `timings` 提供各阶段毫秒耗时,用同一资料、模型、篇幅和质量门槛比较;调度测试不能当成真实 LLM 加速比例。
|
|
327
|
+
|
|
328
|
+
本次未改小模型档位、减少字数门槛或抽样跳过原文。常驻 ASR 模型、逐字稿缓存和分层 QC 仍需单独做质量与性能基准。
|
|
329
|
+
|
|
319
330
|
## 成本
|
|
320
331
|
|
|
321
332
|
Token 用量取决于原文长度、分块数和成稿篇幅。原文核验会再次读取资料,每轮修订还会增加一次修订和一次核验调用。`fastMode` 减少输出章节;调低 `maxChunks` 限制代理数,同时保留全文覆盖。
|
|
322
333
|
|
|
334
|
+
使用 `lib/cache` 的程序化调用方请注意:URL 尾斜杠、空 query 与 fragment 可能选择不同资料,现在会保留;新版指纹自动与旧版缓存隔离,升级后可能需要重新处理一次。该工具不负责证明远程内容未更新,复用前仍须由调用方确认来源与配置。
|
|
335
|
+
|
|
323
336
|
---
|
|
324
337
|
|
|
325
338
|
## 安全与版权
|
package/cordis.patch.yml
CHANGED
|
@@ -14,6 +14,7 @@
|
|
|
14
14
|
# = warning + ignore, never a startup failure) and folds them into the default
|
|
15
15
|
# args carried by the skill, so they become this profile's defaults.
|
|
16
16
|
# config:
|
|
17
|
+
# customParsersDir: './custom-parsers' # optional plugin-level parser directory (not a workflow option)
|
|
17
18
|
# outputDir: './output' # where notes are written (default: ./output)
|
|
18
19
|
# tempDir: './.tmp' # temp files for chunked content (default: ./.tmp)
|
|
19
20
|
# minWords: 2500 # minimum output length (default: 2500)
|
|
@@ -33,4 +34,4 @@
|
|
|
33
34
|
# Idempotency is deliberately NOT a plugin config key: the workflow script runs
|
|
34
35
|
# without a filesystem, so it cannot write a fingerprint record. Programmatic
|
|
35
36
|
# callers can use `deep-read-summarize/lib/cache`
|
|
36
|
-
# (fingerprint / hasProcessed / markProcessed).
|
|
37
|
+
# (fingerprint / hasProcessed / markProcessed).
|
package/docs/RELEASE.md
CHANGED
|
@@ -139,7 +139,8 @@ npm view deep-read-summarize --prefer-online # 验证线
|
|
|
139
139
|
| v0.3.16 | npm 已发布 | 新增 `options.frontmatter` 三档(obsidian / generic / minimal),解决 Hugo 保留键冲突;默认档不变 |
|
|
140
140
|
| v0.3.17 | npm 已发布 | 修 `resolve()` 原型链回退与质量清单 required;新增发布脚本 `scripts/release.js` |
|
|
141
141
|
| v0.3.18 | npm 已发布 | 修复自动解析器分流、并发临时文件串稿和分块上限导致的全文遗漏 |
|
|
142
|
-
|
|
|
142
|
+
| v0.3.19 | npm 已发布 | 修复 B04–B10,并让质量检查结合程序门禁和可回查的原文证据(O01)。 |
|
|
143
|
+
| **v0.4.0** | **npm 已发布(当前)** | 保留完整精读与独立原文核验,减少脚本搬运、并发空等和无效修订,并修复来源质量判断与视频分P等问题。 |
|
|
143
144
|
| v1.0.0 | 计划 | API 稳定后发布 |
|
|
144
145
|
|
|
145
146
|
> 早期迭代用 0.x 表达不稳定;API 稳定后再发 1.0.0。
|
package/index.js
CHANGED
|
@@ -31,12 +31,26 @@ module.exports.inject = ['skills'];
|
|
|
31
31
|
* 支持 `config: { outputDir: ... }` 与 `config: { options: { outputDir: ... } }` 两种写法。
|
|
32
32
|
*/
|
|
33
33
|
function normalizeOptions(config) {
|
|
34
|
-
|
|
34
|
+
let raw = config && Object.prototype.hasOwnProperty.call(config, 'options') ? config.options : config;
|
|
35
|
+
if (raw && typeof raw === 'object' && !Array.isArray(raw) && raw === config) {
|
|
36
|
+
raw = Object.assign({}, raw);
|
|
37
|
+
delete raw.customParsersDir; // plugin-only executable extension directory
|
|
38
|
+
}
|
|
35
39
|
return normalizeWorkflowOptions(raw, { mode: 'config', warn: function (message) { console.warn(message); } });
|
|
36
40
|
}
|
|
37
41
|
|
|
42
|
+
function configuredRegistry(config) {
|
|
43
|
+
if (!config || !Object.prototype.hasOwnProperty.call(config, 'customParsersDir')) return parsers.registry;
|
|
44
|
+
const dir = config.customParsersDir;
|
|
45
|
+
if (typeof dir !== 'string' || !dir.trim() || /[\u0000\r\n]/.test(dir)) {
|
|
46
|
+
console.warn('[deep-read-summarize] ignoring config.customParsersDir — expected a non-empty directory path');
|
|
47
|
+
return parsers.registry;
|
|
48
|
+
}
|
|
49
|
+
return new parsers.ParserRegistry({ customParsersDir: dir });
|
|
50
|
+
}
|
|
51
|
+
|
|
38
52
|
/** 组装技能内容:SKILL.md + workflow meta/script/args 示例(script 为自包含版本)。 */
|
|
39
|
-
function buildSkillContent(config = {}) {
|
|
53
|
+
function buildSkillContent(config = {}, runtime = {}) {
|
|
40
54
|
const skillDir = path.join(__dirname, 'skills', 'deep-read-summarize');
|
|
41
55
|
const body = readFileSync(path.join(skillDir, 'SKILL.md'), 'utf8');
|
|
42
56
|
const argsExample = {
|
|
@@ -44,6 +58,12 @@ function buildSkillContent(config = {}) {
|
|
|
44
58
|
type: 'auto',
|
|
45
59
|
options: normalizeOptions(config)
|
|
46
60
|
};
|
|
61
|
+
if (runtime.toolName) {
|
|
62
|
+
return [body.trim(), '', '## 调用精读工具',
|
|
63
|
+
'调用 `' + runtime.toolName + '`,只提交 input / type / options。工具返回完整 note 与建议 filePath;必须按上文执行同名文件询问、落盘和质量报告。',
|
|
64
|
+
'args 示例(options 是本 profile 的默认值,用户另有要求时再覆盖):',
|
|
65
|
+
'```json', JSON.stringify(argsExample, null, 2), '```'].join('\n');
|
|
66
|
+
}
|
|
47
67
|
return [
|
|
48
68
|
body.trim(),
|
|
49
69
|
'',
|
|
@@ -57,7 +77,7 @@ function buildSkillContent(config = {}) {
|
|
|
57
77
|
'',
|
|
58
78
|
'script(完整脚本,原样填入 script 参数;脚本已自包含解析器,无需宿主注入):',
|
|
59
79
|
'```javascript',
|
|
60
|
-
workflow.
|
|
80
|
+
workflow.buildSelfContainedScript(runtime.registry || configuredRegistry(config)),
|
|
61
81
|
'```',
|
|
62
82
|
'',
|
|
63
83
|
'args 示例(options 是本 profile 的默认值,用户另有要求时再覆盖):',
|
|
@@ -67,21 +87,131 @@ function buildSkillContent(config = {}) {
|
|
|
67
87
|
].join('\n');
|
|
68
88
|
}
|
|
69
89
|
|
|
70
|
-
|
|
71
|
-
|
|
90
|
+
// This is the public DSH WorkflowEngine.start / ToolDefinition contract used
|
|
91
|
+
// by dsh-tool-workflow itself. Optional services keep older profiles working.
|
|
92
|
+
function optionalService(ctx, name) {
|
|
93
|
+
if (typeof ctx.get === 'function') return ctx.get(name);
|
|
94
|
+
return ctx[name];
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
function readingTool(engine, defaults, registry) {
|
|
98
|
+
return {
|
|
99
|
+
name: 'deep_read_summarize',
|
|
100
|
+
description: '深度精读书籍、论文、视频或网页,返回完整 Markdown 笔记、建议保存路径及原文质量核验结果。拿到结果后必须检查同名文件并保存 note;质量未通过或有内容缺口时必须报告。',
|
|
101
|
+
parameters: {
|
|
102
|
+
type: 'object', additionalProperties: false,
|
|
103
|
+
properties: {
|
|
104
|
+
input: { type: 'string', description: '内容链接或本地文件路径' },
|
|
105
|
+
type: { type: 'string', description: 'auto 或已注册的解析器类型,默认 auto' },
|
|
106
|
+
options: { type: 'object', additionalProperties: true, description: '覆盖当前 profile 的精读选项' }
|
|
107
|
+
}, required: ['input']
|
|
108
|
+
},
|
|
109
|
+
output: {
|
|
110
|
+
schema: { type: 'object', additionalProperties: true },
|
|
111
|
+
// Never clip the note: the main agent needs its complete value to save it.
|
|
112
|
+
render: function (args, value) { return [{ type: 'text', text: JSON.stringify(value) }]; }
|
|
113
|
+
},
|
|
114
|
+
async execute(args, exec) {
|
|
115
|
+
if (!args || typeof args !== 'object' || Array.isArray(args)) throw new Error('[FATAL] expected an input object');
|
|
116
|
+
if (Object.keys(args).some(function (key) { return ['input', 'type', 'options'].indexOf(key) < 0; })) throw new Error('[FATAL] unknown deep_read_summarize argument');
|
|
117
|
+
if (typeof args.input !== 'string' || !args.input.trim()) throw new Error('[FATAL] args.input 必须是非空字符串');
|
|
118
|
+
if (args.type !== undefined && typeof args.type !== 'string') throw new Error('[FATAL] args.type 必须是字符串');
|
|
119
|
+
if (args.options !== undefined && (!args.options || typeof args.options !== 'object' || Array.isArray(args.options))) throw new Error('[FATAL] options 必须是对象');
|
|
120
|
+
const options = normalizeWorkflowOptions(Object.assign({}, defaults, args.options));
|
|
121
|
+
if (!exec || !exec.agent) throw new Error('deep_read_summarize requires a calling agent');
|
|
122
|
+
if (exec.signal && exec.signal.aborted) throw new Error('deep_read_summarize was cancelled before start');
|
|
123
|
+
const run = engine.start({ script: workflow.buildSelfContainedScript(registry), meta: workflow.meta,
|
|
124
|
+
args: { input: args.input, type: args.type || 'auto', options: options }, parent: exec.agent, signal: exec.signal });
|
|
125
|
+
try {
|
|
126
|
+
const result = await run.result;
|
|
127
|
+
if (result.stopReason !== 'completed') throw new Error('deep_read_summarize ' + result.stopReason + ': ' + (result.error || 'workflow did not complete'));
|
|
128
|
+
if (!result.value || typeof result.value !== 'object' || Array.isArray(result.value)) throw new Error('deep_read_summarize returned an invalid workflow result');
|
|
129
|
+
return result.value;
|
|
130
|
+
} finally { await run.dispose(); }
|
|
131
|
+
}
|
|
132
|
+
};
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
function skillDefinition(config, runtime) {
|
|
72
136
|
const skillDir = path.join(__dirname, 'skills', 'deep-read-summarize');
|
|
73
|
-
return
|
|
137
|
+
return {
|
|
74
138
|
name: 'deep-read-summarize',
|
|
75
139
|
description: workflow.meta.description,
|
|
76
140
|
whenToUse: workflow.meta.whenToUse,
|
|
77
|
-
content: buildSkillContent(config),
|
|
141
|
+
content: buildSkillContent(config, runtime),
|
|
78
142
|
resourceBase: { kind: 'directory', path: skillDir },
|
|
79
|
-
// source 是技能的「来源桶」(runtime / bundled / user-dsh …),不是自由描述:
|
|
80
|
-
// 本技能随插件包分发,因此登记为 bundled。
|
|
81
143
|
source: 'bundled'
|
|
144
|
+
};
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
// Cordis inject() owns a child fiber which activates/restarts as these optional
|
|
148
|
+
// services appear/disappear. Keep a fallback skill while that child is pending.
|
|
149
|
+
function applyWithServiceLifecycle(ctx, config, registry) {
|
|
150
|
+
let stopped = false;
|
|
151
|
+
let releaseSkill = ctx.skills.register(skillDefinition(config, { registry: registry }));
|
|
152
|
+
function replaceSkill(toolName) {
|
|
153
|
+
if (typeof releaseSkill === 'function') releaseSkill();
|
|
154
|
+
releaseSkill = undefined;
|
|
155
|
+
releaseSkill = ctx.skills.register(skillDefinition(config, { registry: registry, toolName: toolName }));
|
|
156
|
+
}
|
|
157
|
+
function parentCanRegister() {
|
|
158
|
+
// Public Cordis FiberState: LOADING=1, ACTIVE=2; never restore while unloading.
|
|
159
|
+
return !stopped && (!ctx.fiber || ctx.fiber.state === 1 || ctx.fiber.state === 2);
|
|
160
|
+
}
|
|
161
|
+
// Cordis treats callbacks with a prototype as constructors. An arrow keeps
|
|
162
|
+
// the returned cleanup function in its effect lifecycle.
|
|
163
|
+
const child = ctx.inject(['tools', 'workflowEngine'], (ready) => {
|
|
164
|
+
const tools = optionalService(ready, 'tools');
|
|
165
|
+
const engine = optionalService(ready, 'workflowEngine');
|
|
166
|
+
const releaseTool = tools.register(readingTool(engine, normalizeOptions(config), registry));
|
|
167
|
+
try { replaceSkill('deep_read_summarize'); }
|
|
168
|
+
catch (error) {
|
|
169
|
+
if (typeof releaseTool === 'function') releaseTool();
|
|
170
|
+
if (parentCanRegister()) replaceSkill();
|
|
171
|
+
throw error;
|
|
172
|
+
}
|
|
173
|
+
return function () {
|
|
174
|
+
try {
|
|
175
|
+
if (typeof releaseSkill === 'function') releaseSkill();
|
|
176
|
+
releaseSkill = undefined;
|
|
177
|
+
} finally {
|
|
178
|
+
if (typeof releaseTool === 'function') releaseTool();
|
|
179
|
+
if (parentCanRegister()) replaceSkill();
|
|
180
|
+
}
|
|
181
|
+
};
|
|
82
182
|
});
|
|
183
|
+
return async function () {
|
|
184
|
+
stopped = true;
|
|
185
|
+
try { if (child && typeof child.dispose === 'function') await child.dispose(); }
|
|
186
|
+
finally {
|
|
187
|
+
if (typeof releaseSkill === 'function') releaseSkill();
|
|
188
|
+
releaseSkill = undefined;
|
|
189
|
+
}
|
|
190
|
+
};
|
|
83
191
|
}
|
|
84
192
|
|
|
193
|
+
/** Cordis 插件入口:注册运行时技能,使模型可直接执行精读 workflow。 */
|
|
194
|
+
const apply = (ctx, config = {}) => {
|
|
195
|
+
const registry = configuredRegistry(config);
|
|
196
|
+
if (typeof ctx.inject === 'function') return applyWithServiceLifecycle(ctx, config, registry);
|
|
197
|
+
const tools = optionalService(ctx, 'tools');
|
|
198
|
+
const engine = optionalService(ctx, 'workflowEngine');
|
|
199
|
+
const canRunDirectly = tools && typeof tools.register === 'function' && engine && typeof engine.start === 'function';
|
|
200
|
+
const unregisterTool = canRunDirectly ? tools.register(readingTool(engine, normalizeOptions(config), registry)) : undefined;
|
|
201
|
+
let unregisterSkill;
|
|
202
|
+
try { unregisterSkill = ctx.skills.register(skillDefinition(config, {
|
|
203
|
+
registry: registry, toolName: canRunDirectly ? 'deep_read_summarize' : undefined
|
|
204
|
+
})); } catch (error) {
|
|
205
|
+
if (typeof unregisterTool === 'function') unregisterTool();
|
|
206
|
+
throw error;
|
|
207
|
+
}
|
|
208
|
+
if (!canRunDirectly) return unregisterSkill;
|
|
209
|
+
return function () {
|
|
210
|
+
try { if (typeof unregisterSkill === 'function') unregisterSkill(); }
|
|
211
|
+
finally { if (typeof unregisterTool === 'function') unregisterTool(); }
|
|
212
|
+
};
|
|
213
|
+
};
|
|
214
|
+
|
|
85
215
|
// 兼容旧式加载(部分工具按 load() 读取插件定义)
|
|
86
216
|
function load() {
|
|
87
217
|
return { name: 'deep-read-summarize', workflow, parsers, schemas };
|
package/lib/cache.js
CHANGED
|
@@ -17,8 +17,8 @@ const path = require("path");
|
|
|
17
17
|
const crypto = require("crypto");
|
|
18
18
|
|
|
19
19
|
/**
|
|
20
|
-
*
|
|
21
|
-
*
|
|
20
|
+
* 规范化输入:只采用 URL 标准的 host、默认端口等归一化。
|
|
21
|
+
* 保留尾斜杠、空 query 与 fragment:它们可能选择不同文档、路由或阅读范围。
|
|
22
22
|
* 非 URL(本地路径/纯文本)原样返回。
|
|
23
23
|
*/
|
|
24
24
|
function normalizeInput(input) {
|
|
@@ -26,19 +26,11 @@ function normalizeInput(input) {
|
|
|
26
26
|
if (!/^https?:\/\//i.test(s)) return s;
|
|
27
27
|
try {
|
|
28
28
|
const url = new URL(s);
|
|
29
|
-
url.hash = "";
|
|
30
29
|
url.hostname = url.hostname.toLowerCase();
|
|
31
30
|
if ((url.protocol === "http:" && url.port === "80") || (url.protocol === "https:" && url.port === "443")) {
|
|
32
31
|
url.port = "";
|
|
33
32
|
}
|
|
34
|
-
|
|
35
|
-
url.pathname = url.pathname.slice(0, -1);
|
|
36
|
-
}
|
|
37
|
-
// 去掉 URL 尾部的空 query "?"(new URL("...doc?").search 返回 "",无法靠 search 判断,
|
|
38
|
-
// 需检查 toString() 输出;同一 URL 的 "doc" 与 "doc?" 两种写法应共享指纹)
|
|
39
|
-
let normalized = url.toString();
|
|
40
|
-
if (normalized.endsWith("?")) normalized = normalized.slice(0, -1);
|
|
41
|
-
return normalized;
|
|
33
|
+
return url.toString();
|
|
42
34
|
} catch (e) {
|
|
43
35
|
return s; // 非法 URL 原样处理
|
|
44
36
|
}
|
|
@@ -51,7 +43,8 @@ function fingerprint(input, fileMeta) {
|
|
|
51
43
|
const metaStr = fileMeta
|
|
52
44
|
? "|" + (fileMeta.size ?? "") + "|" + (fileMeta.mtimeMs ?? "")
|
|
53
45
|
: "";
|
|
54
|
-
|
|
46
|
+
// v2 不复用旧版过度归一化后的记录,避免旧碰撞继续导致错误跳过。
|
|
47
|
+
return crypto.createHash("sha256").update("drs-input-v2|" + base + metaStr).digest("hex").slice(0, 16);
|
|
55
48
|
}
|
|
56
49
|
|
|
57
50
|
/** 从缓存文件读取已处理记录(JSONL) */
|
package/lib/quality.js
CHANGED
|
@@ -12,6 +12,7 @@ function checkNoteStructure(note, options) {
|
|
|
12
12
|
const front = text.match(/^---[ \t]*\n([\s\S]*?)\n---[ \t]*(?:\n|$)/);
|
|
13
13
|
function scalar(raw) {
|
|
14
14
|
const value = raw.trim();
|
|
15
|
+
if (!value || /[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/.test(value)) throw new Error('missing or invalid scalar');
|
|
15
16
|
if (value[0] === '"') {
|
|
16
17
|
const parsed = JSON.parse(value);
|
|
17
18
|
if (typeof parsed !== 'string') throw new Error('expected string');
|
|
@@ -41,8 +42,12 @@ function checkNoteStructure(note, options) {
|
|
|
41
42
|
if (token.trim()) tokens.push(scalar(token));
|
|
42
43
|
return tokens;
|
|
43
44
|
}
|
|
44
|
-
if (
|
|
45
|
-
|
|
45
|
+
if (/:(?:[ \t]|$)|[\[\]{},]|^[!&*|>@`%#]|^[-?:](?:[ \t]|$)/.test(value)) throw new Error('invalid template scalar');
|
|
46
|
+
const plain = value.replace(/[ \t]+#.*$/, '').trim();
|
|
47
|
+
// Template values are strings. YAML null/booleans/numbers must stay quoted
|
|
48
|
+
// instead of silently passing as a different native YAML type.
|
|
49
|
+
if (/^(?:null|~|true|false|[-+]?(?:0o[0-7]+|0x[0-9a-f]+|[0-9]+(?:\.[0-9]*)?(?:e[-+]?[0-9]+)?|\.[0-9]+(?:e[-+]?[0-9]+)?|\.inf)|\.nan)$/i.test(plain)) throw new Error('template strings must be quoted');
|
|
50
|
+
return plain;
|
|
46
51
|
}
|
|
47
52
|
function fields(lines) {
|
|
48
53
|
const values = Object.create(null);
|
|
@@ -110,6 +115,14 @@ function checkNoteStructure(note, options) {
|
|
|
110
115
|
const lengthMet = wordCount >= opts.minWords;
|
|
111
116
|
if (!lengthMet) issues.push('正文篇幅不足:计数 ' + wordCount + ',要求至少 ' + opts.minWords + '(不计 frontmatter、代码块和链接地址)');
|
|
112
117
|
const quotes = [];
|
|
118
|
+
function inlineText(value) {
|
|
119
|
+
// Normalize display syntax only. Source evidence below remains literal so
|
|
120
|
+
// emphasis cannot make an invented quote match a different source phrase.
|
|
121
|
+
return value.replace(/!?\[([^\]\n]+)\]\([^\n)]*\)/g, '$1')
|
|
122
|
+
.replace(/(\*\*|__)(?=\S)([\s\S]*?\S)\1/g, '$2')
|
|
123
|
+
.replace(/(^|[^\w\\])([*_])(?=\S)([^\n]*?\S)\2/g, '$1$3')
|
|
124
|
+
.replace(/(`+)([^`\n]+)\1/g, '$2');
|
|
125
|
+
}
|
|
113
126
|
const quoteStart = prose.findIndex(line => /^##[ \t]+4(?:[.、.][ \t]*|[ \t]+)\S/.test(line));
|
|
114
127
|
if (quoteStart !== -1) {
|
|
115
128
|
const quoteLines = [];
|
|
@@ -124,16 +137,19 @@ function checkNoteStructure(note, options) {
|
|
|
124
137
|
let match, explicit = false;
|
|
125
138
|
while ((match = expression.exec(line))) {
|
|
126
139
|
explicit = true;
|
|
127
|
-
const quote = (match[1] || match[2] || match[3]).trim();
|
|
140
|
+
const quote = inlineText(match[1] || match[2] || match[3]).trim();
|
|
128
141
|
if (quote && quotes.indexOf(quote) === -1) quotes.push(quote);
|
|
129
142
|
}
|
|
130
143
|
if (!explicit && /^>[ \t]+/.test(line)) {
|
|
131
|
-
|
|
144
|
+
// A trailing, explicitly labelled locator belongs to the citation,
|
|
145
|
+
// not to the sentence. Do not remove arbitrary parenthetical prose.
|
|
146
|
+
const quote = inlineText(line.replace(/^(?:>[ \t]*)+/, ''))
|
|
147
|
+
.replace(/[ \t]*[((](?:第[ \t]*\d+[ \t]*(?:[-–][ \t]*\d+[ \t]*)?(?:行|页|章|节|段)|(?:lines?|pages?|chapter|section)[ \t]+\d+(?:[ \t]*[-–][ \t]*\d+)?|\d{1,2}:\d{2}(?::\d{2})?(?:\.\d+)?)[))][ \t]*$/i, '').trim();
|
|
132
148
|
if (quote && quotes.indexOf(quote) === -1) quotes.push(quote);
|
|
133
149
|
}
|
|
134
150
|
}
|
|
135
151
|
}
|
|
136
|
-
return { formatComplete: frontmatterValid && sectionsComplete, lengthMet, wordCount, quotes, visibleText: proseText, issues };
|
|
152
|
+
return { formatComplete: frontmatterValid && sectionsComplete, lengthMet, wordCount, quotes, visibleText: inlineText(proseText), issues };
|
|
137
153
|
}
|
|
138
154
|
|
|
139
155
|
function validateSourceCheck(check, context) {
|
|
@@ -165,6 +181,10 @@ function validateSourceCheck(check, context) {
|
|
|
165
181
|
if (ctx.coverageComplete !== true) {
|
|
166
182
|
results.coverage = false; issues.push('获取结果缺少有效 totalLines,无法确认原文尾部是否完整覆盖');
|
|
167
183
|
}
|
|
184
|
+
if (ctx.sourceComplete === false) {
|
|
185
|
+
results.coverage = false;
|
|
186
|
+
issues.push('来源不完整:' + (ctx.textSource === 'desc' ? '仅有视频简介,不能代替完整字幕或逐字稿' : '视频来源未经确认为完整字幕或逐字稿'));
|
|
187
|
+
}
|
|
168
188
|
if (ctx.failedChunkIds && ctx.failedChunkIds.length) {
|
|
169
189
|
results.coverage = false; issues.push('精读仍有缺口,失败块:' + ctx.failedChunkIds.join(', '));
|
|
170
190
|
}
|