deep-read-summarize 0.4.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.env.example +1 -1
- package/CHANGELOG.md +46 -0
- package/README.en.md +36 -10
- package/README.md +36 -10
- package/SECURITY.md +2 -2
- package/cordis.patch.yml +6 -0
- package/docs/ASR_BENCHMARK.md +46 -0
- package/docs/RELEASE.md +16 -1
- package/index.js +36 -8
- package/lib/options.js +13 -5
- package/lib/parser-bundle.js +11 -1
- package/lib/read-cache.js +115 -0
- package/lib/transcript-cache.js +173 -0
- package/lib/workflow-script.txt +166 -19
- package/package.json +3 -2
- package/parsers/_prompt.js +1 -0
- package/parsers/video.js +5 -3
- package/schemas/index.js +12 -0
- package/scripts/chunk-balance.js +71 -0
- package/scripts/transcribe.js +107 -13
- package/skills/deep-read-summarize/SKILL.md +10 -6
- package/workflow.js +1 -1
package/.env.example
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
# 请在启动 DSH 的那个环境里直接设置(shell 的 export / Windows 的 setx)。
|
|
5
5
|
#
|
|
6
6
|
# 其余配置不是环境变量,而是插件 config 或 workflow 的 options:
|
|
7
|
-
# outputDir / tempDir / minWords / maxChunks / fastMode / transcribe /
|
|
7
|
+
# outputDir / tempDir / minWords / maxChunks / readConcurrency / parallelQuality / readCache / readCacheIdentity / fastMode / transcribe /
|
|
8
8
|
# requireCitations / includeTimestamps / whisperModel / language / device / maxRetries
|
|
9
9
|
|
|
10
10
|
# uv 拉取 CPython 时使用的 GitHub 镜像前缀(只在需要下载 Python 3.12 时才用得上)。
|
package/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,52 @@
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
The format is based on [Keep a Changelog](https://keepachangelog.com/).
|
|
5
5
|
|
|
6
|
+
## [Unreleased]
|
|
7
|
+
|
|
8
|
+
## [0.6.0] — 2026-11-05
|
|
9
|
+
|
|
10
|
+
新增可配置并发、字符均衡分块与可选的并行原文核验和跨次精读缓存,保留原有默认核验路径。
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- **精读并发与计时**:`readConcurrency` 可设置 1–12,默认仍为 3;`readTimings` 返回各块耗时、成功与缓存命中状态。提高并发前应观察宿主限流和失败情况。
|
|
15
|
+
- **字符均衡分块**:获取代理可调用随包脚本计算整行字符负载并优先选择接近的段落边界;统计不可用或计划无效时回退按行均衡,均保留全文行号覆盖。字符数只是 token 的近似,超长单行不会拆开。
|
|
16
|
+
- **可选并行独立核验**:`parallelQuality: true` 时多块分别读取原文、检查全部显式引用,另检查全局术语和语言;证据缺失或核验失败不通过。默认使用原有核验路径。
|
|
17
|
+
- **可选跨次精读缓存**:仅插件工具入口支持,默认关闭。开启时必须由调用者声明包含供应商、固定模型版本及推理设置的 `readCacheIdentity`;源文件内容、提示词和身份参与缓存键,仅质量通过后写入,命中后仍进行独立核验。身份无法由宿主自动认证。缓存为本地明文,可能含原文摘录。
|
|
18
|
+
|
|
19
|
+
### Verification and limitations
|
|
20
|
+
|
|
21
|
+
- 回归测试覆盖分块完整性、并发调度、引用证据、缓存失效和失败回退;调度与插件入口测试使用模拟宿主,未测量真实 DSH 模型的延迟或质量,不能据此声称加速比例。
|
|
22
|
+
- 字符统计依赖获取代理执行和返回辅助脚本结果;模型身份由用户保证。源文件在任务前后重新哈希,但无法检测期间短暂发生后恢复的变化。Windows 当前测试环境没有创建符号链接的权限,相关平台测试需由支持的环境补测。
|
|
23
|
+
|
|
24
|
+
### Compatibility
|
|
25
|
+
|
|
26
|
+
- `parallelQuality` 与 `readCache` 默认关闭;`readConcurrency` 默认为 3。旧 workflow 入口不支持跨次精读缓存,其他能力保持可用。
|
|
27
|
+
- 贡献者与致谢:本版无外部 PR、报告或复测贡献需要署名。
|
|
28
|
+
|
|
29
|
+
|
|
30
|
+
## [0.5.0] — 2026-11-01
|
|
31
|
+
|
|
32
|
+
无字幕视频新增逐字稿内容缓存、转写阶段计时和 CPU 线程配置,保留默认精读与原文核验门槛。
|
|
33
|
+
|
|
34
|
+
### Added
|
|
35
|
+
|
|
36
|
+
- **逐字稿内容缓存**:默认按音频 SHA-256、受管理的本地模型快照、安装环境和固定解码配置复用真实分段;切换时间戳展示无需重复推理。缓存命中仍原子发布本次任务产物,后续精读和独立原文核验不跳过。自定义本地模型、环境不明、损坏/过期记录和失败/空结果不复用。
|
|
37
|
+
- **缓存控制**:`options.transcriptCache`(默认 true);CLI `--no-cache` 禁用读写、`--refresh-cache` 强制推理并更新、`--clear-cache` 仅清理逐字稿。内容保存在本地 `transcripts-v1`,单条最多 8 MiB、总量最多 128 MiB、有效期 30 天;写入时清理。缓存使用本地已有模型快照,不在线探测上游更新,不缓存音频。
|
|
38
|
+
- **CPU 线程配置**:`options.cpuThreads` / `--cpu-threads` 接受 0–64 的整数,默认 0 保持库策略;完整透传到视频转写,默认 small/int8、VAD 与解码参数保持原状。
|
|
39
|
+
- **转写阶段计时**:成功调用输出 `metrics` JSON,分开报告缓存命中、实际设备、音频时长、环境/依赖准备、Python 与发布耗时;未命中时记录导入、模型加载、准备和完整的延迟分段迭代,命中时不复用旧推理计时。
|
|
40
|
+
|
|
41
|
+
### Verification
|
|
42
|
+
|
|
43
|
+
- Windows CPU 对照一段自制 58.439 秒中文语音:原始 runner 24.517 秒,新 runner 强制推理 23.118 秒,命中缓存并开启时间戳 0.098 秒,8 线程且禁用缓存 21.275 秒。默认与 8 线程文本均与原结果逐字一致;缓存切换时间戳后正文一致。单样本结果仅覆盖 ASR,不代表完整精读或其他资料的加速比例。详见 [ASR 基准记录](https://github.com/PensiveFei/deep-read-summarize/blob/v0.5.0/docs/ASR_BENCHMARK.md)。
|
|
44
|
+
- ASR 错词仍可能进入逐字稿;笔记 `qualityPassed` 不等于音轨听写准确。GPU、批量推理和常驻模型未自动启用,仍需独立的音频质量与资源基准。
|
|
45
|
+
|
|
46
|
+
### Fixed
|
|
47
|
+
|
|
48
|
+
- **工作流致命错误**:精读、成稿、原文核验和修订阶段统一传播 `[FATAL]`、`UNSUPPORTED_SCHEMA` 与 `fatal: true` 错误;精读遇到致命错误后停止派发剩余分块。普通临时错误仍保留原有降级行为。
|
|
49
|
+
- **转写环境初始化**:只有 `uv venv` 成功退出且实际生成 Python 解释器文件才继续转写;失败时留下的残缺环境不能跳过直连重试或被误用。
|
|
50
|
+
- **自定义解析器打包**:按 JavaScript 作用域规则分别处理 `switch` 判别式与 `case` 块,避免局部 `require` 隐藏判别式中的模块导入,或局部变量掩盖不可用的宿主 API。
|
|
51
|
+
|
|
6
52
|
## [0.4.0] — 2026-10-25
|
|
7
53
|
|
|
8
54
|
保留完整精读与独立原文核验,减少脚本搬运、并发空等和无效修订,并修复来源质量判断与视频分P等问题。
|
package/README.en.md
CHANGED
|
@@ -1,9 +1,11 @@
|
|
|
1
1
|
# deep-read-summarize
|
|
2
2
|
|
|
3
|
+
[English](https://github.com/PensiveFei/deep-read-summarize/blob/main/README.en.md) | [简体中文](https://github.com/PensiveFei/deep-read-summarize/blob/main/README.md)
|
|
4
|
+
|
|
3
5
|
[](https://github.com/PensiveFei/deep-read-summarize/blob/main/LICENSE)
|
|
4
6
|
[](https://github.com/PensiveFei/deep-read-summarize/releases)
|
|
5
7
|
[](https://github.com/PensiveFei/deep-read-summarize/actions/workflows/ci.yml)
|
|
6
|
-
[](https://www.npmjs.com/package/deep-read-summarize)
|
|
7
9
|
|
|
8
10
|
> **Disclaimer**: This is an **unofficial third-party tool**. It is not affiliated with, endorsed by, or sponsored by DeepSeek or the DeepSeek Harness project. "DeepSeek" and "DeepSeek Harness" are trademarks of their respective owners. This project only orchestrates the workflow tool available in your DSH environment; it does not redistribute any DeepSeek software.
|
|
9
11
|
|
|
@@ -72,11 +74,11 @@ Three waves, normally N+3 sub-agent calls (fetch, N chunk reads, draft, and qual
|
|
|
72
74
|
|
|
73
75
|
`auto` routes recognized video links, arXiv paper links, and ebook extensions directly to their parsers; columns, channels, and help pages on those sites still require classification. PDFs are classified as books or papers from their content. Failed classification returns `stage: "detect"` and asks for an explicit type. `options.tempDir` is a temporary root: each call creates its own `run-*` subdirectory, keeping fetched text, audio, and transcripts separate from other tasks.
|
|
74
76
|
|
|
75
|
-
Chunk plans are sorted, duplicate ranges trimmed, and gaps filled. Plans above `maxChunks` merge adjacent ranges instead of discarding the document tail. The returned `coverageComplete` indicates coverage of all declared lines based on a valid `totalLines`; without a line count, known ranges are retained, `qualityPassed` is set to `false`, and the coverage uncertainty is reported.
|
|
77
|
+
Chunk plans are sorted, duplicate ranges trimmed, and gaps filled. Plans above `maxChunks` merge adjacent ranges instead of discarding the document tail. The fetch agent runs a bundled helper to propose character-balanced whole-line boundaries from the saved source, preferring nearby paragraph boundaries when balance remains close. Missing or invalid results fall back to line balancing. Character counts are a workload approximation, not token counts, and a single long line is not split. The returned `coverageComplete` indicates coverage of all declared lines based on a valid `totalLines`; without a line count, known ranges are retained, `qualityPassed` is set to `false`, and the coverage uncertainty is reported.
|
|
76
78
|
|
|
77
79
|
Quality checks combine code and source review. Code validates the supplied scalar/list frontmatter template, required sections, and length (CJK characters plus other words, excluding metadata, code blocks, HTML comments, and link URLs). A separate reviewer reads the current source file and returns per-chunk excerpts and line-bound evidence for verbatim quotations in section 4. A bare `pass: true` cannot override these checks. `qualityChecks` exposes the gates, and `qualityEvidence` exposes the source path and evidence. Source authenticity still depends on the reviewer's actual source reading; uncertainty fails the gate. Empty, rejected, or structurally degraded repairs retain the preceding usable note.
|
|
78
80
|
|
|
79
|
-
Direct workflow options are strict: `minWords` is an integer in 1–100000, `maxChunks` is a positive integer (values above 12 are capped), and `maxRetries` is an integer in 0–3. Invalid values and unknown keys fail before work begins. Plugin profile configuration instead warns per invalid field and uses its default, preserving host startup.
|
|
81
|
+
Direct workflow options are strict: `minWords` is an integer in 1–100000, `maxChunks` is a positive integer (values above 12 are capped), `readConcurrency` is an integer in 1–12, and `maxRetries` is an integer in 0–3. Invalid values and unknown keys fail before work begins. Plugin profile configuration instead warns per invalid field and uses its default, preserving host startup.
|
|
80
82
|
|
|
81
83
|
```
|
|
82
84
|
parsers/ Parsers for each input type, dispatched by type
|
|
@@ -201,11 +203,17 @@ The `deep_read_summarize` tool takes these parameters directly. On older hosts,
|
|
|
201
203
|
"options": {
|
|
202
204
|
"minWords": 2500,
|
|
203
205
|
"fastMode": false, // true skips sections 5-7, faster
|
|
204
|
-
"maxChunks": 4, // chunk limit, 1-12
|
|
206
|
+
"maxChunks": 4, // chunk limit, 1-12
|
|
207
|
+
"readConcurrency": 3, // concurrent chunk reads and reviews, 1-12
|
|
208
|
+
"parallelQuality": false, // opt in to parallel source review; default retains original review
|
|
209
|
+
"readCache": false, // plugin tool only: opt in to cross-run read caching
|
|
210
|
+
"readCacheIdentity": "", // required when enabled: provider, pinned model version and settings
|
|
205
211
|
"transcribe": true, // video: auto local transcription (faster-whisper) when no subtitles; false skips it — see Video
|
|
206
212
|
"whisperModel": "small", // video ASR tier: small | base | medium
|
|
207
213
|
"language": "zh", // video ASR language: use "en" for English-only videos
|
|
208
214
|
"device": "cpu", // ASR device: cpu (default) | auto | cuda; auto/cuda needs a CUDA runtime and falls back to cpu
|
|
215
|
+
"cpuThreads": 0, // 0 keeps library defaults; 1-64 allows hardware-specific tuning
|
|
216
|
+
"transcriptCache": true, // reuse a verified local transcript for matching audio and settings
|
|
209
217
|
"maxRetries": 0, // quality-check retries (default 0 = no retry; retrying is noticeably slower)
|
|
210
218
|
"frontmatter": "obsidian", // key set: obsidian (default, for Dataview) | generic (drops the keys Hugo reserves) | minimal
|
|
211
219
|
"requireCitations": true, // whether key conclusions must carry citations
|
|
@@ -268,7 +276,22 @@ Video text extraction is a **unified pipeline** (the old "three tiers" are gone)
|
|
|
268
276
|
|
|
269
277
|
**Transcription toolchain**: `uv` provisions **Python 3.12** and installs **`faster-whisper==1.2.1`** from the Tsinghua mirror; models use `HF_ENDPOINT=https://hf-mirror.com` and the local cache. Healthy cached environments run directly without requiring uv or a separate Python import probe. Actual interpreter and dependency versions are recorded in the cache's `runtime.json`; Python patch releases, uv, and all transitive dependencies are not fully pinned. PyAV decodes audio without a separate ffmpeg. All three platforms use `node scripts/transcribe.js --audio <file> --out <txt>`. CPU is the default; `options.device: "auto"` can try a supplied CUDA runtime and falls back to CPU. Cache roots are `%LOCALAPPDATA%\deep-read-summarize` on Windows, `~/Library/Caches/deep-read-summarize` on macOS, and `$XDG_CACHE_HOME/deep-read-summarize` on Linux (falling back to `~/.cache`). Python provisioning tries a mirror and then direct GitHub; `DRS_UV_PYTHON_MIRROR` overrides it, with an empty value selecting direct downloads.
|
|
270
278
|
|
|
271
|
-
> ⚠️ **The first transcription downloads the model
|
|
279
|
+
> ⚠️ **The first transcription downloads the model (small ≈ 484 MB, via hf-mirror, not GitHub) and may take a few minutes**. Later runs reuse model files; new audio still requires full inference. Only a verified transcript cache hit skips inference. **Only subtitle-less videos trigger transcription** (subtitled videos use their subtitles directly).
|
|
280
|
+
|
|
281
|
+
**Transcript cache and threads**: `options.transcriptCache` defaults to true. Keys include the audio SHA-256, managed local model snapshots, installed dependencies, and decoding settings. Canonical segments retain real start/end times, so changing timestamp presentation does not repeat inference. Changed content/configuration/models/dependencies, custom local model directories, unknown environments, corrupt entries and expired entries bypass reuse. Cache lookup uses the existing local snapshot and does not check for upstream model updates online. Entries live under `transcripts-v1` in the cache root: 8 MiB per entry, 128 MiB total, 30-day validity, pruned on writes. Audio is not stored, and reading/source review still runs.
|
|
282
|
+
|
|
283
|
+
`options.cpuThreads` accepts integers 0–64. The default `0` preserves the library's CPU thread policy. Compare 4/6/8 on the same audio; more threads are not necessarily faster. CLI controls:
|
|
284
|
+
|
|
285
|
+
```bash
|
|
286
|
+
node scripts/transcribe.js --audio sample.m4a --out transcript.txt --cpu-threads 8
|
|
287
|
+
node scripts/transcribe.js --audio sample.m4a --out transcript.txt --refresh-cache
|
|
288
|
+
node scripts/transcribe.js --audio sample.m4a --out transcript.txt --no-cache
|
|
289
|
+
node scripts/transcribe.js --clear-cache
|
|
290
|
+
```
|
|
291
|
+
|
|
292
|
+
`--clear-cache` removes only transcripts, retaining models and Python. `--refresh-cache` forces inference and updates the cache; `--no-cache` disables reads and writes. Successful calls print one `metrics` JSON line with `cacheHit`, actual device, audio duration, and milliseconds for hashing, environment preparation, dependency installation, Python, publication and total time. Misses also include import, model loading, preparation (including decoding/VAD), and complete lazy-segment iteration. Hits do not reuse previous inference timings.
|
|
293
|
+
|
|
294
|
+
The note's `qualityPassed` checks the note against acquired text; it does not establish ASR accuracy. ASR changes still need audio/reference comparisons for errors, omissions, numbers, negation and timestamps.
|
|
272
295
|
|
|
273
296
|
**Quality/speed**: `small` + `int8` quantization + `VAD` silence filtering gives usable Chinese quality and is CPU-friendly. **Subtitled videos are not transcribed (fast)**; subtitle-less videos are transcribed at roughly 0.5–2× the video length on CPU.
|
|
274
297
|
|
|
@@ -299,7 +322,7 @@ Video text extraction is a **unified pipeline** (the old "three tiers" are gone)
|
|
|
299
322
|
- **Chunking structure**: different types chunk differently (chapter/topic/paragraph), affecting **output structure** and coverage.
|
|
300
323
|
|
|
301
324
|
### Hardware
|
|
302
|
-
- **GPU vs CPU**:
|
|
325
|
+
- **GPU vs CPU**: gains depend on hardware, runtime libraries, model and audio. A visible GPU does not prove CUDA inference works; compare actual time and transcript accuracy.
|
|
303
326
|
|
|
304
327
|
### Other
|
|
305
328
|
- **First run vs reuse**: the first run downloads the model (≈ 484 MB); later runs reuse the cache.
|
|
@@ -321,12 +344,15 @@ Two classes:
|
|
|
321
344
|
## Faster runs with the same quality gates
|
|
322
345
|
|
|
323
346
|
- Prefer `deep_read_summarize` on supported hosts: the model submits parameters instead of reproducing the entire workflow script. Older hosts retain the existing route.
|
|
324
|
-
- All known source lines remain covered.
|
|
325
|
-
- Independent source review remains enabled, with
|
|
347
|
+
- All known source lines remain covered. By default up to three reading tasks run concurrently, and each free slot immediately starts another task. Character-balanced chunks are used when available; otherwise the workflow balances by line count.
|
|
348
|
+
- Independent source review remains enabled. With `parallelQuality: true`, source chunks and all quotations are checked in parallel, with a separate global terminology and language review; the original review is the default. Missing line counts, failed reads, and description-only video sources cannot be fixed by rewriting the draft, so those states no longer consume full repair loops.
|
|
326
349
|
- Video `textSource: "desc"` is explicitly degraded and cannot pass as a complete deep read. Bilibili `p` selects the matching CID, part title, and duration.
|
|
327
|
-
- Returned `timings` reports stage durations in milliseconds. Compare the same sources, model, output size, and quality gates; scheduling simulations are not real LLM speedup measurements.
|
|
350
|
+
- Returned `timings` reports stage durations in milliseconds; `readTimings` reports each chunk’s line range, duration and success. Compare the same sources, model, output size, and quality gates; scheduling simulations are not real LLM speedup measurements.
|
|
351
|
+
- `readConcurrency` controls concurrent chunk reads without changing the chunk plan or quality review requirements; it also limits concurrent review tasks when `parallelQuality` is enabled. It defaults to 3. When at least four chunks exist and the host supports it, try 4 and compare total time, throttling and failures; higher concurrency is not always faster.
|
|
352
|
+
|
|
353
|
+
**Cross-run read caching** is disabled by default and supported only by the `deep_read_summarize` plugin tool, not direct workflow calls. Set `readCache: true` and provide `readCacheIdentity` containing the provider, pinned model version and all output-affecting settings. The host cannot verify that identity; change it when the model or settings change. The cache matches source content, chunk prompt and identity, writes only after quality passes, and still performs independent source review on hits. `readTimings.cacheHit` reports hits. Entries are plaintext under `tempDir/.read-cache-v1` and may contain source excerpts; they expire after 30 days, are capped at 64 MiB total and 1 MiB per entry, and can be deleted by removing that directory.
|
|
328
354
|
|
|
329
|
-
|
|
355
|
+
Defaults preserve small/int8, existing decoding settings, output length and independent source review. Transcript caching speeds up repeated audio; new audio can be profiled and tuned with CPU thread settings. Persistent ASR workers, batch inference and hierarchical review require separate benchmarks and are not enabled automatically.
|
|
330
356
|
|
|
331
357
|
## Cost
|
|
332
358
|
|
package/README.md
CHANGED
|
@@ -1,9 +1,11 @@
|
|
|
1
1
|
# deep-read-summarize
|
|
2
2
|
|
|
3
|
+
[English](https://github.com/PensiveFei/deep-read-summarize/blob/main/README.en.md) | [简体中文](https://github.com/PensiveFei/deep-read-summarize/blob/main/README.md)
|
|
4
|
+
|
|
3
5
|
[](https://github.com/PensiveFei/deep-read-summarize/blob/main/LICENSE)
|
|
4
6
|
[](https://github.com/PensiveFei/deep-read-summarize/releases)
|
|
5
7
|
[](https://github.com/PensiveFei/deep-read-summarize/actions/workflows/ci.yml)
|
|
6
|
-
[](https://www.npmjs.com/package/deep-read-summarize)
|
|
7
9
|
|
|
8
10
|
> **Disclaimer**: This is an **unofficial third-party tool**. It is not affiliated with, endorsed by, or sponsored by DeepSeek or the DeepSeek Harness project. "DeepSeek" and "DeepSeek Harness" are trademarks of their respective owners. This project only orchestrates the workflow tool available in your DSH environment; it does not redistribute any DeepSeek software.
|
|
9
11
|
|
|
@@ -72,11 +74,11 @@ Markdown 笔记
|
|
|
72
74
|
|
|
73
75
|
`auto` 会将明确的视频链接、arXiv 论文链接和电子书扩展名直接分发到对应解析器;同站的专栏、频道与帮助页仍先分类,PDF 会根据内容区分书籍与论文。无法识别时返回 `stage: "detect"`,请显式指定类型。`options.tempDir` 作为临时根目录,每次调用在其中创建独立的 `run-*` 子目录,获取、音频和转写文件不与其他任务共用。
|
|
74
76
|
|
|
75
|
-
分块计划会排序、裁掉重复区间并补齐缺口,超过 `maxChunks`
|
|
77
|
+
分块计划会排序、裁掉重复区间并补齐缺口,超过 `maxChunks` 时合并相邻块以保留全文。获取代理运行内置脚本,按保存文本的字符数计算整行分界,尽量在不明显牺牲均衡的情况下选择段落边界;若脚本无法运行或分界无效,则回退按行数均衡。字符数只是 token 数和工作量的近似值,超长单行不会拆分。返回的 `coverageComplete` 表示是否依据有效的 `totalLines` 覆盖全部行号;缺少总行数时保留已知范围,同时将 `qualityPassed` 置为 `false` 并说明覆盖无法确认。
|
|
76
78
|
|
|
77
79
|
质量检查分两部分:程序按提供的标量/列表 frontmatter 模板检查字段、必要章节和篇幅(汉字按字、其他文字按词,排除 frontmatter、代码块、HTML 注释及链接地址);独立核验代理读取本次原文文件,逐块检查覆盖,并为第 4 节的逐字引用返回原句和行号。单独的 `pass: true` 无法通过检查。`qualityChecks` 返回检查项,`qualityEvidence` 返回原文路径、分块摘录和引用证据;原文真实性判断仍由读取原文的代理完成,无法核实就报告未通过。修订为空、调用失败或结构退化时保留上一份有效成稿。
|
|
78
80
|
|
|
79
|
-
直接调用 workflow 的 options 必须类型正确:`minWords` 为 1–100000 的整数,`maxChunks` 为正整数(大于 12 仍夹到 12),`maxRetries` 为 0–3 的整数;非法值和未知键在启动时 FATAL。插件 profile 的 config 则逐项告警并回退默认值,避免阻止宿主启动。
|
|
81
|
+
直接调用 workflow 的 options 必须类型正确:`minWords` 为 1–100000 的整数,`maxChunks` 为正整数(大于 12 仍夹到 12),`readConcurrency` 为 1–12 的整数,`maxRetries` 为 0–3 的整数;非法值和未知键在启动时 FATAL。插件 profile 的 config 则逐项告警并回退默认值,避免阻止宿主启动。
|
|
80
82
|
|
|
81
83
|
```
|
|
82
84
|
parsers/ 各输入类型的解析器,按类型分发
|
|
@@ -200,11 +202,17 @@ parsers: book, paper, video, web
|
|
|
200
202
|
"options": {
|
|
201
203
|
"minWords": 2500,
|
|
202
204
|
"fastMode": false, // true 时跳过 5-7 节,速度快一些
|
|
203
|
-
"maxChunks": 4, // 分块上限,1-12
|
|
205
|
+
"maxChunks": 4, // 分块上限,1-12
|
|
206
|
+
"readConcurrency": 3, // 精读和分块核验的并发上限,1-12
|
|
207
|
+
"parallelQuality": false, // 显式启用并行原文核验;默认使用原有核验路径
|
|
208
|
+
"readCache": false, // 仅插件工具支持:启用跨次精读缓存
|
|
209
|
+
"readCacheIdentity": "", // 启用缓存时必填:provider/固定模型版本/影响输出的设置
|
|
204
210
|
"transcribe": true, // 视频:无字幕时自动本地转写(faster-whisper);false 则跳过转写,见「视频」节
|
|
205
211
|
"whisperModel": "small", // 视频转写模型档位:small | base | medium
|
|
206
212
|
"language": "zh", // 视频转写语言:纯英文视频可设 "en"
|
|
207
213
|
"device": "cpu", // 转写设备:cpu(默认)| auto | cuda;auto/cuda 需自备 CUDA 运行库,失败自动回退 cpu
|
|
214
|
+
"cpuThreads": 0, // 0 保持转写库默认值;1-64 可按实际硬件调优
|
|
215
|
+
"transcriptCache": true, // 音频内容与配置一致时复用本地逐字稿;false 强制每次推理
|
|
208
216
|
"maxRetries": 0, // 质量校验重试次数(默认 0 = 不重试,重试会明显变慢)
|
|
209
217
|
"frontmatter": "obsidian", // frontmatter 键集:obsidian(默认,配 Dataview)| generic(换掉 Hugo 保留键)| minimal
|
|
210
218
|
"requireCitations": true, // 关键结论是否必须标注出处
|
|
@@ -267,7 +275,22 @@ parsers: book, paper, video, web
|
|
|
267
275
|
|
|
268
276
|
**转写工具链**:脚本用 `uv` 创建 **Python 3.12** 环境,从清华镜像安装 **`faster-whisper==1.2.1`**,通过 `HF_ENDPOINT=https://hf-mirror.com` 下载并缓存模型。健康的缓存环境直接运行,不要求 uv 仍在 PATH,也不再启动单独的 Python 导入探测。实际解释器和依赖版本记录在缓存目录的 `runtime.json`;Python 补丁版本、uv 和传递依赖未全部锁定,不能据此声称环境逐字一致。PyAV 可直接解码音频,无需单独 ffmpeg。三平台使用同一条命令 `node scripts/transcribe.js --audio <音频> --out <txt>`,默认 CPU;`options.device: "auto"` 可尝试自备的 CUDA,失败回退 CPU。缓存目录:Windows `%LOCALAPPDATA%\deep-read-summarize`、macOS `~/Library/Caches/deep-read-summarize`、Linux `$XDG_CACHE_HOME/deep-read-summarize`(未设置时用 `~/.cache`)。创建 Python 环境默认先用镜像,失败再直连 GitHub;`DRS_UV_PYTHON_MIRROR` 可覆盖镜像,空字符串表示直连。
|
|
269
277
|
|
|
270
|
-
> ⚠️ **首次转写会先下载模型(small 约 484MB,走 hf-mirror,非 GitHub
|
|
278
|
+
> ⚠️ **首次转写会先下载模型(small 约 484MB,走 hf-mirror,非 GitHub),可能需要几分钟**。之后复用模型文件;新音频仍需完整推理。同一音频命中经过校验的逐字稿缓存时,才会跳过推理。**只有「无字幕」视频才会触发转写**(有字幕的视频直接用字幕,不触发、不下载)。
|
|
279
|
+
|
|
280
|
+
**逐字稿缓存与线程**:`options.transcriptCache` 默认开启。缓存键包含音频 SHA-256、受管理的本地模型快照、已安装依赖及解码配置;存储真实分段和起止时间,切换时间戳展示不必重转写。更改内容、配置或本地模型/依赖会失效;自定义本地模型目录、无法确认环境、损坏和过期记录均跳过缓存。缓存复用本地已有快照,不在线探测模型上游是否更新。逐字稿保存在上述缓存根目录的 `transcripts-v1`,单条上限 8 MiB、总量上限 128 MiB、有效期 30 天;清理在写入时执行。它不保存音频,也不会跳过后续精读与原文核验。
|
|
281
|
+
|
|
282
|
+
`options.cpuThreads` 为 0–64 的整数,默认 `0` 保持库的 CPU 线程策略。可用同一音频对比 4/6/8 线程,线程越多不一定越快。CLI 可直接使用:
|
|
283
|
+
|
|
284
|
+
```bash
|
|
285
|
+
node scripts/transcribe.js --audio sample.m4a --out transcript.txt --cpu-threads 8
|
|
286
|
+
node scripts/transcribe.js --audio sample.m4a --out transcript.txt --refresh-cache
|
|
287
|
+
node scripts/transcribe.js --audio sample.m4a --out transcript.txt --no-cache
|
|
288
|
+
node scripts/transcribe.js --clear-cache
|
|
289
|
+
```
|
|
290
|
+
|
|
291
|
+
`--clear-cache` 只清理逐字稿,不删除模型或 Python 环境。`--refresh-cache` 强制推理并更新缓存,`--no-cache` 禁用读写。成功调用输出一行 `metrics` JSON,报告 `cacheHit`、实际设备、音频时长,以及哈希、环境准备、依赖安装、Python、发布和总毫秒耗时;未命中时还报告导入、模型加载、准备(含解码/VAD)与完整分段迭代耗时。缓存命中不会冒用上次推理耗时。
|
|
292
|
+
|
|
293
|
+
笔记的 `qualityPassed` 核验笔记与取得的原文,不证明 ASR 听写准确;调整转写路径仍须对照音频检查误字、漏段、数字、否定句与时间戳。
|
|
271
294
|
|
|
272
295
|
**质量/速度**:`small` + `int8` 量化 + `VAD` 静音过滤,中文质量可用且 CPU 友好。**有字幕的视频不转写(快)**;无字幕视频转写,CPU 上约为视频时长的 0.5–2×。
|
|
273
296
|
|
|
@@ -298,7 +321,7 @@ parsers: book, paper, video, web
|
|
|
298
321
|
- **分块结构**:不同类型分块方式不同(章节/主题/段落),影响**输出结构**与覆盖。
|
|
299
322
|
|
|
300
323
|
### 硬件
|
|
301
|
-
- **GPU vs CPU**:GPU
|
|
324
|
+
- **GPU vs CPU**:GPU 收益依赖显卡、运行库、模型与音频;设备可见不代表 CUDA 推理可用,应以实际耗时与文本对照为准。
|
|
302
325
|
|
|
303
326
|
### 其他
|
|
304
327
|
- **首次 vs 复用**:首次要下模型(约 484MB),之后缓存复用。
|
|
@@ -320,12 +343,15 @@ parsers: book, paper, video, web
|
|
|
320
343
|
## 保持质量的提速
|
|
321
344
|
|
|
322
345
|
- 支持的新宿主优先使用 `deep_read_summarize`,模型只提交参数,省去复制整份工作流脚本。旧宿主自动保留原入口。
|
|
323
|
-
-
|
|
324
|
-
- QC
|
|
346
|
+
- 精读仍完整覆盖已知原文,并发数默认最多为三个任务;空闲槽位立即领取下一块。优先按字符数均衡工作量,统计不可用时回退按行数均衡。
|
|
347
|
+
- QC 继续独立读取原文;可通过 `parallelQuality: true` 让多个分块并行核验原文和全部引文,另行检查整篇术语与语言;默认仍使用原有核验路径。缺总行数、精读失败或只有视频简介等问题不能靠改写笔记解决,不再浪费全文修订轮次。
|
|
325
348
|
- 视频 `textSource: "desc"` 明确为降级来源,不能取得完整精读的质量通过状态。B站多 P 链接按 `p` 选择对应 CID、分 P 标题和时长。
|
|
326
|
-
- 返回 `timings`
|
|
349
|
+
- 返回 `timings` 提供各阶段毫秒耗时,`readTimings` 提供每个精读块的行号、耗时和是否成功。用同一资料、模型、篇幅和质量门槛比较;调度测试不能当成真实 LLM 加速比例。
|
|
350
|
+
- `readConcurrency` 控制精读并发,不改变分块或质量核验的要求;启用 `parallelQuality` 时也限制核验任务数,默认为 3。实际分块数至少为 4 且宿主支持时,可尝试设为 4,比较总耗时、限流及失败情况;并发越高不一定越快。
|
|
351
|
+
|
|
352
|
+
**跨次精读缓存**默认关闭,仅 `deep_read_summarize` 插件工具入口支持,裸 workflow 不启用。设 `readCache: true` 时必须同时设置 `readCacheIdentity`,包含供应商、固定模型版本和所有影响输出的设置;宿主无法验证该身份,模型或设置变化时必须更换。缓存按原文内容、分块提示及身份匹配,仅在质量检查通过后写入;命中后仍执行独立原文核验。`readTimings` 的 `cacheHit` 表示是否命中。缓存是明文,位于 `tempDir/.read-cache-v1`,可能包含原文摘录;超过 30 天、64 MiB 总量或 1 MiB 单条限制会被清理或跳过,不需要时可删除该目录。
|
|
327
353
|
|
|
328
|
-
|
|
354
|
+
默认保持 small/int8、既有解码设置、篇幅与独立原文核验。逐字稿缓存主要加速重复处理,新视频可用线程配置和分阶段计时调优;常驻 ASR 模型、批量推理和分层 QC 仍需独立基准,不自动启用。
|
|
329
355
|
|
|
330
356
|
## 成本
|
|
331
357
|
|
package/SECURITY.md
CHANGED
|
@@ -6,7 +6,7 @@ Please do **not** open a public issue for security vulnerabilities.
|
|
|
6
6
|
|
|
7
7
|
To report a vulnerability privately:
|
|
8
8
|
|
|
9
|
-
1. Open a [private security advisory](https://github.com
|
|
9
|
+
1. Open a [private security advisory](https://github.com/PensiveFei/deep-read-summarize/security/advisories/new) (if available), or
|
|
10
10
|
2. Email the maintainer (see repository profile for contact).
|
|
11
11
|
|
|
12
12
|
## What to include
|
|
@@ -24,4 +24,4 @@ To report a vulnerability privately:
|
|
|
24
24
|
|
|
25
25
|
## Scope
|
|
26
26
|
|
|
27
|
-
This project only ships workflow code, prompt templates, and schemas. It does not contain copyrighted content or secrets. Vulnerabilities of interest include prompt injection in parser prompts, sandbox escape attempts, and unsafe file handling.
|
|
27
|
+
This project only ships workflow code, prompt templates, and schemas. It does not contain copyrighted content or secrets. Vulnerabilities of interest include prompt injection in parser prompts, sandbox escape attempts, and unsafe file handling.
|
package/cordis.patch.yml
CHANGED
|
@@ -20,12 +20,18 @@
|
|
|
20
20
|
# minWords: 2500 # minimum output length (default: 2500)
|
|
21
21
|
# fastMode: false # skip sections 5-7 for speed (default: false)
|
|
22
22
|
# maxChunks: 4 # chunk cap; also caps sub-agent count (1-12, default: 4)
|
|
23
|
+
# readConcurrency: 3 # concurrent chunk reads and reviews (1-12, default: 3)
|
|
24
|
+
# parallelQuality: false # opt in to parallel independent source review
|
|
25
|
+
# readCache: false # plugin tool only; cross-run cache is opt-in
|
|
26
|
+
# readCacheIdentity: "" # required when enabled: provider, pinned model and settings
|
|
23
27
|
# transcribe: true # video: auto-transcribe when no subtitles (default: true)
|
|
24
28
|
# requireCitations: true # key conclusions must cite source location (default: true)
|
|
25
29
|
# includeTimestamps: false # video timestamps (default: false)
|
|
26
30
|
# whisperModel: 'small' # transcription model (default: small)
|
|
27
31
|
# language: 'zh' # transcription language (default: zh)
|
|
28
32
|
# device: 'cpu' # cpu | auto | cuda; GPU needs a CUDA runtime (default: cpu)
|
|
33
|
+
# cpuThreads: 0 # 0 keeps library defaults; 1-64 for measured CPU tuning
|
|
34
|
+
# transcriptCache: true # local content/config-verified transcript reuse (30 days, 128 MiB)
|
|
29
35
|
# maxRetries: 0 # quality-check retry rounds (default: 0)
|
|
30
36
|
# frontmatter: 'obsidian' # obsidian | generic | minimal (default: obsidian)
|
|
31
37
|
# # generic swaps the keys Hugo reserves: type -> docType,
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# ASR performance check — 2026-11-01
|
|
2
|
+
|
|
3
|
+
This check compares the unmodified `6b95206` runner with the transcript-cache/thread/metrics changes. It measures transcription only, excluding video download, deep reading, drafting and independent source review.
|
|
4
|
+
|
|
5
|
+
## Setup
|
|
6
|
+
|
|
7
|
+
- Windows, Intel Core i5-12500H (12 cores, 16 logical processors), 16 GB RAM; CPU execution.
|
|
8
|
+
- Node 24.19.0, faster-whisper 1.2.1, CTranslate2 4.8.1, PyAV 18.1.0.
|
|
9
|
+
- Existing `small` model snapshot `536b0662742c02347bc0e980a01041f333bce120`, int8, Chinese, existing beam/VAD/decoding defaults; model files already downloaded. HF offline mode prevented network/download variation.
|
|
10
|
+
- One self-authored Chinese passage, rendered locally by Microsoft Huihui Desktop to a 58.439-second WAV. The passage includes numeric contrasts and negation. No private audio was used.
|
|
11
|
+
- Each row starts a separate Node process. The first two use the library's default CPU thread policy; the final row explicitly selects 8. A transcript-cache hit still hashes the input, validates cache identity and publishes a new output.
|
|
12
|
+
|
|
13
|
+
## Observations
|
|
14
|
+
|
|
15
|
+
| Run | Wall time | Lazy inference iteration | Text relative to baseline |
|
|
16
|
+
| --- | ---: | ---: | --- |
|
|
17
|
+
| Original runner | 24.517 s | not instrumented | baseline |
|
|
18
|
+
| New runner, forced inference | 23.118 s | 20.184 s | byte-identical |
|
|
19
|
+
| New runner, cached, timestamps enabled | 0.098 s | skipped | byte-identical after removing timestamp prefixes |
|
|
20
|
+
| New runner, 8 threads, cache disabled | 21.275 s | 18.306 s | byte-identical |
|
|
21
|
+
|
|
22
|
+
The new forced-inference run spent 0.549 s importing, 1.378 s loading the model, and 0.334 s preparing audio/VAD before consuming segments. The cache-hit call reported zero Python time. Stored segments were rendered with their real times; enabling timestamps did not call Python again.
|
|
23
|
+
|
|
24
|
+
These are single observations, not a latency distribution or a general speed guarantee. Execution order, OS/model-file caches and machine load can affect the difference between fresh runs. The unchanged default is intentional: this sample does not justify increasing CPU threads for every machine.
|
|
25
|
+
|
|
26
|
+
The normalized reference CER was 9.852% in all four rows. Normalization keeps Unicode letters/numbers, folds case and removes punctuation; it does not equate spoken number words with digits or traditional with simplified Chinese. The result establishes output preservation on this sample, not accurate listening for all videos. Real speech, noise, accents, mixed languages, long silence and long-form omissions need their own reference set. GPU and batch inference were not tested or enabled by this change.
|
|
27
|
+
|
|
28
|
+
## Repeating the check
|
|
29
|
+
|
|
30
|
+
Generate audio from the reference below or use a licensed clip with a reference transcript. Keep the same audio, model snapshot, device, decoding settings and machine load for paired runs. Measure full process wall time and inspect the `metrics` JSON. The `inference` timer consumes the complete lazy segment iterator; timing only the return of `transcribe()` is insufficient.
|
|
31
|
+
|
|
32
|
+
```bash
|
|
33
|
+
# Fresh result and cache population
|
|
34
|
+
node scripts/transcribe.js --audio sample.wav --out fresh.txt --refresh-cache
|
|
35
|
+
# Same audio, different presentation, no inference
|
|
36
|
+
node scripts/transcribe.js --audio sample.wav --out timestamped.txt --timestamps
|
|
37
|
+
# Thread comparisons must bypass the cache
|
|
38
|
+
node scripts/transcribe.js --audio sample.wav --out threads4.txt --cpu-threads 4 --no-cache
|
|
39
|
+
node scripts/transcribe.js --audio sample.wav --out threads8.txt --cpu-threads 8 --no-cache
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Check exact text, omissions, numbers, negation and timestamps as well as aggregate error rates. Keep cold environment/model download, fresh inference and transcript hits separate. Gather multiple representative clips and repeated runs before making a broader performance claim.
|
|
43
|
+
|
|
44
|
+
## Self-authored reference
|
|
45
|
+
|
|
46
|
+
今天讨论怎样检查学习笔记的准确性。第一,完整的音频转写不能只保留开头,也不能把视频简介当成全文。第二,数字和否定句需要特别检查。实验用了五十个样本,不是十五个样本;正确率是百分之九十二,而不是百分之九十七。第三,模型下载到磁盘以后,处理新的音频仍然需要推理。重复处理同一份音频时,可以验证内容哈希并复用逐字稿。第四,缓存命中以后,笔记仍然需要精读和独立核验,不能直接宣布质量通过。最后,请保留每一段的真实时间位置,遇到不确定的术语时明确标记,不要编造。
|
package/docs/RELEASE.md
CHANGED
|
@@ -30,6 +30,19 @@
|
|
|
30
30
|
- <未解决的问题或限制>
|
|
31
31
|
- ...
|
|
32
32
|
|
|
33
|
+
### 贡献者与致谢
|
|
34
|
+
|
|
35
|
+
- @<GitHub 用户名>:<代码、报告、复现、复测或文档等具体贡献>(<对应 Issue/PR 的完整链接>)。
|
|
36
|
+
- <没有外部贡献时如实说明;需要匿名时按贡献者意愿记录,不猜测真实姓名或邮箱。>
|
|
37
|
+
|
|
38
|
+
**归属清单(维护者发布前后各核对一次)**:
|
|
39
|
+
|
|
40
|
+
- [ ] 对照已合并 PR、已解决 Issue 和复测记录,核对用户名与贡献,保留作者/共同作者信息。
|
|
41
|
+
- [ ] 在目标版本的 **CHANGELOG.md 小节**中填好致谢;`npm run release` 从该小节原样生成 Notes,仅修改此模板不会进入发布说明。
|
|
42
|
+
- [ ] GitHub Generate release notes 可用 `.github/release.yml` 分类并列出 PR 作者;人工补齐报告者和复测者,不覆盖已有署名。
|
|
43
|
+
- [ ] 发布后核对公开 Release 的用户名、贡献说明和来源链接;遗漏时补记,保留原有说明、资产和 tag。
|
|
44
|
+
- [ ] 按 [贡献指南](../CONTRIBUTING.md) 在相关讨论中说明结果并及时致谢;敏感报告先确认披露范围。
|
|
45
|
+
|
|
33
46
|
---
|
|
34
47
|
|
|
35
48
|
## npm 发布流程
|
|
@@ -140,7 +153,9 @@ npm view deep-read-summarize --prefer-online # 验证线
|
|
|
140
153
|
| v0.3.17 | npm 已发布 | 修 `resolve()` 原型链回退与质量清单 required;新增发布脚本 `scripts/release.js` |
|
|
141
154
|
| v0.3.18 | npm 已发布 | 修复自动解析器分流、并发临时文件串稿和分块上限导致的全文遗漏 |
|
|
142
155
|
| v0.3.19 | npm 已发布 | 修复 B04–B10,并让质量检查结合程序门禁和可回查的原文证据(O01)。 |
|
|
143
|
-
|
|
|
156
|
+
| v0.4.0 | npm 已发布 | 保留完整精读与独立原文核验,减少脚本搬运、并发空等和无效修订,并修复来源质量判断与视频分P等问题。 |
|
|
157
|
+
| v0.5.0 | npm 已发布 | 无字幕视频新增逐字稿内容缓存、转写阶段计时和 CPU 线程配置,保留默认精读与原文核验门槛。 |
|
|
158
|
+
| **v0.6.0** | **npm 已发布(当前)** | 新增可配置并发、字符均衡分块与可选的并行原文核验和跨次精读缓存,保留原有默认核验路径。 |
|
|
144
159
|
| v1.0.0 | 计划 | API 稳定后发布 |
|
|
145
160
|
|
|
146
161
|
> 早期迭代用 0.x 表达不稳定;API 稳定后再发 1.0.0。
|
package/index.js
CHANGED
|
@@ -21,6 +21,7 @@ const workflow = require('./workflow');
|
|
|
21
21
|
const parsers = require('./parsers');
|
|
22
22
|
const schemas = require('./schemas');
|
|
23
23
|
const { DEFAULT_OPTIONS, normalizeWorkflowOptions } = require('./lib/options');
|
|
24
|
+
const readCache = require('./lib/read-cache');
|
|
24
25
|
|
|
25
26
|
module.exports.name = 'deep-read-summarize';
|
|
26
27
|
module.exports.inject = ['skills'];
|
|
@@ -120,14 +121,41 @@ function readingTool(engine, defaults, registry) {
|
|
|
120
121
|
const options = normalizeWorkflowOptions(Object.assign({}, defaults, args.options));
|
|
121
122
|
if (!exec || !exec.agent) throw new Error('deep_read_summarize requires a calling agent');
|
|
122
123
|
if (exec.signal && exec.signal.aborted) throw new Error('deep_read_summarize was cancelled before start');
|
|
123
|
-
const
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
const
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
124
|
+
const startArgs = { input: args.input, type: args.type || 'auto', options: options };
|
|
125
|
+
const script = workflow.buildSelfContainedScript(registry);
|
|
126
|
+
async function runWorkflow(runArgs) {
|
|
127
|
+
const run = engine.start({ script: script, meta: workflow.meta,
|
|
128
|
+
args: runArgs, parent: exec.agent, signal: exec.signal });
|
|
129
|
+
try {
|
|
130
|
+
const result = await run.result;
|
|
131
|
+
if (result.stopReason !== 'completed') throw new Error('deep_read_summarize ' + result.stopReason + ': ' + (result.error || 'workflow did not complete'));
|
|
132
|
+
if (!result.value || typeof result.value !== 'object' || Array.isArray(result.value)) throw new Error('deep_read_summarize returned an invalid workflow result');
|
|
133
|
+
return result.value;
|
|
134
|
+
} finally { await run.dispose(); }
|
|
135
|
+
}
|
|
136
|
+
if (!options.readCache) return runWorkflow(startArgs);
|
|
137
|
+
const started = Date.now();
|
|
138
|
+
const prepared = await runWorkflow(Object.assign({}, startArgs, { __cachePrepare: true }));
|
|
139
|
+
if (!prepared.__cachePrepared) return prepared;
|
|
140
|
+
if (!Array.isArray(prepared.chunkPrompts) || !Array.isArray(prepared.chunkPlan) || prepared.chunkPrompts.length !== prepared.chunkPlan.length || prepared.chunkPrompts.length > 12 || typeof prepared.tempFile !== 'string' || typeof prepared.tempDir !== 'string' || typeof prepared.selectedType !== 'string' || !prepared.fetched || typeof prepared.fetched !== 'object') throw new Error('deep_read_summarize returned invalid cache preparation');
|
|
141
|
+
const digest = readCache.sourceHash(options.tempDir, prepared.tempDir, prepared.tempFile);
|
|
142
|
+
const keys = prepared.chunkPrompts.map(function (prompt) { return readCache.keyFor(digest, options.readCacheIdentity, prompt, prepared.tempFile); });
|
|
143
|
+
const hits = keys.map(function (key) { return key ? readCache.read(options.tempDir, key) : null; });
|
|
144
|
+
const resume = { tempDir: prepared.tempDir, selectedType: prepared.selectedType, fetched: prepared.fetched, prompts: prepared.chunkPrompts, hits: hits };
|
|
145
|
+
if (exec.signal && exec.signal.aborted) throw new Error('deep_read_summarize was cancelled before cache resume');
|
|
146
|
+
const result = await runWorkflow(Object.assign({}, startArgs, { __cacheResume: resume }));
|
|
147
|
+
// Never use notes matched against a source that changed during the run.
|
|
148
|
+
if (digest && readCache.sourceHash(options.tempDir, prepared.tempDir, prepared.tempFile) !== digest) return runWorkflow(startArgs);
|
|
149
|
+
if (!(exec.signal && exec.signal.aborted) && digest && result.qualityPassed && Array.isArray(result.__cacheReads)) {
|
|
150
|
+
for (let i = 0; i < keys.length; i++) if (keys[i] && !hits[i] && readCache.portable(result.__cacheReads[i], prepared.tempDir)) readCache.write(options.tempDir, keys[i], result.__cacheReads[i]);
|
|
151
|
+
}
|
|
152
|
+
delete result.__cacheReads;
|
|
153
|
+
if (result.timings && prepared.timings) {
|
|
154
|
+
result.timings.detect += prepared.timings.detect || 0;
|
|
155
|
+
result.timings.fetch += prepared.timings.fetch || 0;
|
|
156
|
+
result.timings.total = Date.now() - started;
|
|
157
|
+
}
|
|
158
|
+
return result;
|
|
131
159
|
}
|
|
132
160
|
};
|
|
133
161
|
}
|
package/lib/options.js
CHANGED
|
@@ -3,10 +3,11 @@
|
|
|
3
3
|
/** Kept self-contained: the exact same validator is embedded in the workflow. */
|
|
4
4
|
function normalizeWorkflowOptions(raw, behavior) {
|
|
5
5
|
const defaults = {
|
|
6
|
-
minWords: 2500, fastMode: false, maxChunks: 4, transcribe: true,
|
|
6
|
+
minWords: 2500, fastMode: false, maxChunks: 4, readConcurrency: 3, parallelQuality: false, transcribe: true,
|
|
7
7
|
requireCitations: true, includeTimestamps: false, outputDir: './output',
|
|
8
8
|
tempDir: './.tmp', whisperModel: 'small', language: 'zh', device: 'cpu',
|
|
9
|
-
maxRetries: 0, frontmatter: 'obsidian'
|
|
9
|
+
maxRetries: 0, frontmatter: 'obsidian', cpuThreads: 0, transcriptCache: true,
|
|
10
|
+
readCache: false, readCacheIdentity: ''
|
|
10
11
|
};
|
|
11
12
|
const settings = behavior || {};
|
|
12
13
|
const tolerant = settings.mode === 'config';
|
|
@@ -37,9 +38,9 @@ function normalizeWorkflowOptions(raw, behavior) {
|
|
|
37
38
|
if (!Object.prototype.hasOwnProperty.call(raw, key) || raw[key] === undefined) continue;
|
|
38
39
|
let value = raw[key];
|
|
39
40
|
let reason = '';
|
|
40
|
-
if (key === 'minWords' || key === 'maxChunks' || key === 'maxRetries') {
|
|
41
|
-
const minimum = key === 'maxRetries' ? 0 : 1;
|
|
42
|
-
const maximum = key === 'minWords' ? 100000 : (key === 'maxRetries' ? 3 : Infinity);
|
|
41
|
+
if (key === 'minWords' || key === 'maxChunks' || key === 'readConcurrency' || key === 'maxRetries' || key === 'cpuThreads') {
|
|
42
|
+
const minimum = key === 'maxRetries' || key === 'cpuThreads' ? 0 : 1;
|
|
43
|
+
const maximum = key === 'cpuThreads' ? 64 : key === 'minWords' ? 100000 : key === 'readConcurrency' ? 12 : (key === 'maxRetries' ? 3 : Infinity);
|
|
43
44
|
if (typeof value !== 'number' || !Number.isFinite(value) || !Number.isInteger(value) || value < minimum || value > maximum) {
|
|
44
45
|
reason = 'expected a finite integer in ' + minimum + '..' + (maximum === Infinity ? '12 (larger values are capped)' : maximum);
|
|
45
46
|
} else if (key === 'maxChunks' && value > 12) {
|
|
@@ -48,6 +49,9 @@ function normalizeWorkflowOptions(raw, behavior) {
|
|
|
48
49
|
}
|
|
49
50
|
} else if (typeof defaults[key] === 'boolean') {
|
|
50
51
|
if (typeof value !== 'boolean') reason = 'expected boolean';
|
|
52
|
+
} else if (key === 'readCacheIdentity') {
|
|
53
|
+
if (typeof value !== 'string' || value.length > 512 || /[\u0000\r\n]/.test(value)) reason = 'expected a string up to 512 characters without NUL or line breaks';
|
|
54
|
+
else value = value.trim();
|
|
51
55
|
} else if (typeof value !== 'string' || !value.trim() || /[\u0000\r\n]/.test(value)) {
|
|
52
56
|
reason = 'expected a non-empty string without NUL or line breaks';
|
|
53
57
|
} else if (key !== 'outputDir' && key !== 'tempDir') {
|
|
@@ -60,6 +64,10 @@ function normalizeWorkflowOptions(raw, behavior) {
|
|
|
60
64
|
if (reason) invalid(key, reason);
|
|
61
65
|
else result[key] = value;
|
|
62
66
|
}
|
|
67
|
+
if (result.readCache && !result.readCacheIdentity) {
|
|
68
|
+
invalid('readCacheIdentity', 'required when readCache is true; include provider, immutable model version and generation settings');
|
|
69
|
+
result.readCache = false;
|
|
70
|
+
}
|
|
63
71
|
return result;
|
|
64
72
|
}
|
|
65
73
|
|
package/lib/parser-bundle.js
CHANGED
|
@@ -34,6 +34,16 @@ function analyze(source, filename) {
|
|
|
34
34
|
else if (pattern.type === 'ObjectPattern') pattern.properties.forEach(item => bind(item.type === 'RestElement' ? item.argument : item.value, target));
|
|
35
35
|
}
|
|
36
36
|
function collect(node, active, parent, key) {
|
|
37
|
+
if (node.type === 'SwitchStatement') {
|
|
38
|
+
// The discriminant is evaluated before entering the case block's lexical
|
|
39
|
+
// environment. A case-local `require` must not hide an outer import here.
|
|
40
|
+
scopes.set(node, active);
|
|
41
|
+
nodes.push({ node, parent, key });
|
|
42
|
+
collect(node.discriminant, active, node, 'discriminant');
|
|
43
|
+
const cases = scope(active, 'block');
|
|
44
|
+
node.cases.forEach(item => collect(item, cases, node, 'cases'));
|
|
45
|
+
return;
|
|
46
|
+
}
|
|
37
47
|
if (/^Function/.test(node.type) || node.type === 'ArrowFunctionExpression') {
|
|
38
48
|
if (node.type === 'FunctionDeclaration') bind(node.id, active);
|
|
39
49
|
const parameters = scope(active, 'parameters');
|
|
@@ -49,7 +59,7 @@ function analyze(source, filename) {
|
|
|
49
59
|
if (node.type === 'ClassDeclaration') bind(node.id, active);
|
|
50
60
|
if (node.type === 'ClassDeclaration' || node.type === 'ClassExpression') {
|
|
51
61
|
active = scope(active, 'block'); bind(node.id, active);
|
|
52
|
-
} else if (['BlockStatement', 'CatchClause', 'ForStatement', 'ForInStatement', 'ForOfStatement'
|
|
62
|
+
} else if (['BlockStatement', 'CatchClause', 'ForStatement', 'ForInStatement', 'ForOfStatement'].indexOf(node.type) >= 0) {
|
|
53
63
|
active = scope(active, 'block');
|
|
54
64
|
} else if (node.type === 'StaticBlock') active = scope(active, 'function');
|
|
55
65
|
if (node.type === 'CatchClause') bind(node.param, active);
|