deep-read-summarize 0.3.19 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -3,6 +3,64 @@
3
3
  All notable changes to this project will be documented in this file.
4
4
  The format is based on [Keep a Changelog](https://keepachangelog.com/).
5
5
 
6
+ ## [Unreleased]
7
+
8
+ ## [0.5.0] — 2026-11-01
9
+
10
+ 无字幕视频新增逐字稿内容缓存、转写阶段计时和 CPU 线程配置,保留默认精读与原文核验门槛。
11
+
12
+ ### Added
13
+
14
+ - **逐字稿内容缓存**:默认按音频 SHA-256、受管理的本地模型快照、安装环境和固定解码配置复用真实分段;切换时间戳展示无需重复推理。缓存命中仍原子发布本次任务产物,后续精读和独立原文核验不跳过。自定义本地模型、环境不明、损坏/过期记录和失败/空结果不复用。
15
+ - **缓存控制**:`options.transcriptCache`(默认 true);CLI `--no-cache` 禁用读写、`--refresh-cache` 强制推理并更新、`--clear-cache` 仅清理逐字稿。内容保存在本地 `transcripts-v1`,单条最多 8 MiB、总量最多 128 MiB、有效期 30 天;写入时清理。缓存使用本地已有模型快照,不在线探测上游更新,不缓存音频。
16
+ - **CPU 线程配置**:`options.cpuThreads` / `--cpu-threads` 接受 0–64 的整数,默认 0 保持库策略;完整透传到视频转写,默认 small/int8、VAD 与解码参数保持原状。
17
+ - **转写阶段计时**:成功调用输出 `metrics` JSON,分开报告缓存命中、实际设备、音频时长、环境/依赖准备、Python 与发布耗时;未命中时记录导入、模型加载、准备和完整的延迟分段迭代,命中时不复用旧推理计时。
18
+
19
+ ### Verification
20
+
21
+ - Windows CPU 对照一段自制 58.439 秒中文语音:原始 runner 24.517 秒,新 runner 强制推理 23.118 秒,命中缓存并开启时间戳 0.098 秒,8 线程且禁用缓存 21.275 秒。默认与 8 线程文本均与原结果逐字一致;缓存切换时间戳后正文一致。单样本结果仅覆盖 ASR,不代表完整精读或其他资料的加速比例。详见 [ASR 基准记录](https://github.com/PensiveFei/deep-read-summarize/blob/v0.5.0/docs/ASR_BENCHMARK.md)。
22
+ - ASR 错词仍可能进入逐字稿;笔记 `qualityPassed` 不等于音轨听写准确。GPU、批量推理和常驻模型未自动启用,仍需独立的音频质量与资源基准。
23
+
24
+ ### Fixed
25
+
26
+ - **工作流致命错误**:精读、成稿、原文核验和修订阶段统一传播 `[FATAL]`、`UNSUPPORTED_SCHEMA` 与 `fatal: true` 错误;精读遇到致命错误后停止派发剩余分块。普通临时错误仍保留原有降级行为。
27
+ - **转写环境初始化**:只有 `uv venv` 成功退出且实际生成 Python 解释器文件才继续转写;失败时留下的残缺环境不能跳过直连重试或被误用。
28
+ - **自定义解析器打包**:按 JavaScript 作用域规则分别处理 `switch` 判别式与 `case` 块,避免局部 `require` 隐藏判别式中的模块导入,或局部变量掩盖不可用的宿主 API。
29
+
30
+ ## [0.4.0] — 2026-10-25
31
+
32
+ 保留完整精读与独立原文核验,减少脚本搬运、并发空等和无效修订,并修复来源质量判断与视频分P等问题。
33
+
34
+ ### Added
35
+
36
+ - **参数直达工具 `deep_read_summarize`**:宿主具有 tools / workflowEngine 服务时,由插件直接提交工作流,模型只传 input/type/options,无需原样重发约 4.6 万字符脚本。转发调用代理和取消信号,完成或失败后释放 workflow run;旧宿主保留原 workflow + skill 入口。
37
+ - **阶段耗时 `timings`**:返回识别、获取、精读、成稿、质量核验和总耗时(毫秒),供同资料、模型与质量门槛下比较。测试调度模拟不代表真实 LLM 加速比例。
38
+ - **显式自定义解析器目录**:插件顶层 `config.customParsersDir` 可加载独立目录,各 profile 隔离;包内 `custom-parsers/` 纳入 npm 分发。
39
+
40
+ ### Fixed
41
+
42
+ - **视频来源质量**:只有简介或未确认来源的视频保持质量降级,不能因“简介的所有行都读过”而报告完整精读通过;来源类型进入 QC 上下文,并在笔记中标明限制。
43
+ - **B站多P**:保留短链最终 URL 与 p,按 `pages[p-1]` 选择 CID、标题和时长;字幕、音频使用同一分P,非法或越界参数不能默默回退第1P。
44
+ - **异常与重试**:类型识别和获取的可恢复调用错误返回对应 stage,真正 FATAL 继续终止;缺总行数、失败精读块、来源不完整等不能靠改写解决的问题不再消耗全文修订轮次。
45
+ - **格式与引用**:严格检查模板字符串,拒绝非法 YAML 裸标量和错误字段类型;区分 Markdown 引文的可见原话与强调、行号等展示信息,减少真实引用被误判。
46
+ - **文件名**:过滤控制字符、设备保留名与非法尾缀;对清洗或截短的标题增加稳定后缀,限制 UTF-8 文件名长度,避免 NUL、超长标题落盘失败与不同非法标题折叠到同名路径。
47
+ - **URL 指纹**:保留可选择不同资料的尾斜杠、空 query 与 fragment;新版键与旧缓存隔离,防止旧版碰撞继续跳过资料。
48
+ - **解析器更新**:注册后生成技能和工具调用使用最新注册表,不再引用模块加载时的旧脚本;插件构建阶段定位本版本转写脚本,避免每次递归搜索其他安装目录。
49
+ - **转写热启动**:健康环境直接执行一次 Python runner,移除单独 import 探测;仅明确缺依赖或版本不匹配时才要求 uv 并自举,已有可用环境不再被“找不到 uv”提前拒绝。
50
+ - **转写版本记录**:固定已验证的 `faster-whisper==1.2.1`,记录实际 Python / 依赖版本;纠正文档“uv / Python补丁 / 全部依赖完全锁定”的不实承诺。
51
+
52
+ ### Performance
53
+
54
+ - **固定三个并发槽位**:一个任务完成即领取下一块,取消每批三个都结束才启动下一批的屏障,输出仍按原分块顺序合并。
55
+ - **均衡分块工作量**:在既有代理数和 maxChunks 上限内按行数均分已知全文,超长逻辑段可拆分,保持每行恰好覆盖一次。
56
+ - **缩减 QC 上下文**:保留原文路径、行号、缺口与待核验引文,移除重复的完整 Map 笔记;独立核验仍须读取原文。
57
+
58
+ ### Compatibility
59
+
60
+ - 默认模型、篇幅、章节与引用门槛不降低;新工具是可选入口,原入口保留。新返回字段为补充元数据。
61
+ - 新版缓存指纹可能导致升级后的首次重处理;清洗过或过长标题的建议文件名会改变,正常标题保持不变。
62
+ - 原文真实性仍取决于核验代理实际读取原文。逐字稿内容缓存、常驻 ASR worker、分层 QC 和共享转写环境的冷启动互斥留待单独验证,未在本次发布中声称完成。
63
+
6
64
  ## [0.3.19] — 2026-10-24
7
65
 
8
66
  修复 B04–B10,并让质量检查结合程序门禁和可回查的原文证据(O01)。
package/README.en.md CHANGED
@@ -1,9 +1,11 @@
1
1
  # deep-read-summarize
2
2
 
3
+ [English](https://github.com/PensiveFei/deep-read-summarize/blob/main/README.en.md) | [简体中文](https://github.com/PensiveFei/deep-read-summarize/blob/main/README.md)
4
+
3
5
  [![MIT license](https://img.shields.io/github/license/PensiveFei/deep-read-summarize)](https://github.com/PensiveFei/deep-read-summarize/blob/main/LICENSE)
4
6
  [![release](https://img.shields.io/github/v/release/PensiveFei/deep-read-summarize)](https://github.com/PensiveFei/deep-read-summarize/releases)
5
7
  [![CI](https://img.shields.io/github/actions/workflow/status/PensiveFei/deep-read-summarize/ci.yml)](https://github.com/PensiveFei/deep-read-summarize/actions/workflows/ci.yml)
6
- [![npm downloads](https://badgen.net/npm/dm/deep-read-summarize)](https://www.npmjs.com/package/deep-read-summarize)
8
+ [![npm total downloads](https://badgen.net/npm/dt/deep-read-summarize)](https://www.npmjs.com/package/deep-read-summarize)
7
9
 
8
10
  > **Disclaimer**: This is an **unofficial third-party tool**. It is not affiliated with, endorsed by, or sponsored by DeepSeek or the DeepSeek Harness project. "DeepSeek" and "DeepSeek Harness" are trademarks of their respective owners. This project only orchestrates the workflow tool available in your DSH environment; it does not redistribute any DeepSeek software.
9
11
 
@@ -97,7 +99,7 @@ workflow.js the workflow meta and assembly logic (parser inlining, schema i
97
99
 
98
100
  > The script body lives in `lib/workflow-script.txt` on purpose: this file was corrupted twice by "escaping on top of escaping". It is plain text now, so reading and editing it needs no encoding step.
99
101
 
100
- To change how an input type is handled, drop a parser with the same interface into `custom-parsers/`; a type with the same name overrides the built-in implementation, and you can also add a brand-new type name (e.g. `podcast`) - every registered parser is inlined into the workflow script, and both the `type` whitelist and the schema's `kind` enum follow the registry. The interface has only three fields: `name`, `types`, `buildPrompt(input, opts)`.
102
+ To override or add a parser, use the package's `custom-parsers/` directory (included in npm packaging), or set the plugin-level `config.customParsersDir` to your own directory. This is not a workflow option. Per-profile directories remain isolated, and rebuilding skill content uses the current registry, including programmatic registrations. Parsers expose `name`, `types`, and `buildPrompt(input, opts)`; registered types are inlined into the workflow and its schema.
101
103
 
102
104
  File parsers bundle their CommonJS modules and literal relative `.js`, `.cjs`, and `.json` dependencies, preserving module constants, helpers, method shorthand, and cycles. Acorn parses JavaScript during script construction and is installed automatically by npm; it does not run in the workflow sandbox. Node built-ins, external packages, and dynamic `require`/`import` are unsupported in sandbox parsers and fail at build time. Programmatically registered functions must be self-contained; use a file parser for module closures.
103
105
 
@@ -136,7 +138,8 @@ pnpm add ./deep-read-summarize-0.3.10.tgz
136
138
  ```
137
139
 
138
140
  After installation it registers:
139
- - the `deep-read-summarize` workflow (meta + script)
141
+ - the `deep_read_summarize` tool when the host provides tools and workflowEngine services; models send only input/type/options
142
+ - the `deep-read-summarize` workflow definition (meta + script), with the existing workflow-tool route retained for older hosts
140
143
  - the `deep-read-summarize` skill (`skills/deep-read-summarize/SKILL.md`)
141
144
  - four parsers (`parsers/`) and JSON Schemas (`schemas/`)
142
145
 
@@ -150,18 +153,18 @@ After installation it registers:
150
153
  npm install deep-read-summarize
151
154
 
152
155
  # Option B: develop from source
153
- git clone https://github.com/<your-org>/deep-read-summarize.git
156
+ git clone https://github.com/PensiveFei/deep-read-summarize.git
154
157
  cd deep-read-summarize
155
158
  npm ci # install locked dependencies
156
159
  ```
157
160
 
158
- ### 2. Verify the environment (offline, no API key needed)
161
+ ### 2. Verify installation (offline, no API key needed)
159
162
 
160
163
  ```bash
161
- npm test # 35 fixture tests, all run offline
164
+ node -e "console.log(require('deep-read-summarize').name)"
162
165
  ```
163
166
 
164
- Seeing `TOTAL: 35 passed, 0 failed` means the environment is ready.
167
+ Seeing `deep-read-summarize` confirms the package loads. Source checkouts can additionally run `npm test` and `npm run test:node`; tests are not included in the npm package.
165
168
 
166
169
  ### 3. Feed a demo file with one command
167
170
 
@@ -176,9 +179,9 @@ meta: deep-read-summarize
176
179
  parsers: book, paper, video, web
177
180
  ```
178
181
 
179
- ### 4. Real run (requires the DSH workflow tool)
182
+ ### 4. Real run (requires DSH)
180
183
 
181
- Pass the JSON below to DSH's workflow tool (see Usage):
184
+ Prefer passing the JSON below directly to `deep_read_summarize`. On older hosts, use the skill's meta + script with the existing `workflow` tool:
182
185
 
183
186
  ```jsonc
184
187
  { "input": "https://arxiv.org/abs/2307.09042", "type": "paper", "options": { "maxChunks": 4, "fastMode": true } }
@@ -191,7 +194,7 @@ The workflow returns a structured result: `{ ok, kind, title, filePath, qualityP
191
194
  ---
192
195
  ## Usage
193
196
 
194
- Pass this JSON to DSH's workflow tool:
197
+ The `deep_read_summarize` tool takes these parameters directly. On older hosts, use them as workflow args with the skill's meta + script:
195
198
 
196
199
  ```jsonc
197
200
  {
@@ -205,6 +208,8 @@ Pass this JSON to DSH's workflow tool:
205
208
  "whisperModel": "small", // video ASR tier: small | base | medium
206
209
  "language": "zh", // video ASR language: use "en" for English-only videos
207
210
  "device": "cpu", // ASR device: cpu (default) | auto | cuda; auto/cuda needs a CUDA runtime and falls back to cpu
211
+ "cpuThreads": 0, // 0 keeps library defaults; 1-64 allows hardware-specific tuning
212
+ "transcriptCache": true, // reuse a verified local transcript for matching audio and settings
208
213
  "maxRetries": 0, // quality-check retries (default 0 = no retry; retrying is noticeably slower)
209
214
  "frontmatter": "obsidian", // key set: obsidian (default, for Dataview) | generic (drops the keys Hugo reserves) | minimal
210
215
  "requireCitations": true, // whether key conclusions must carry citations
@@ -265,9 +270,24 @@ Video text extraction is a **unified pipeline** (the old "three tiers" are gone)
265
270
 
266
271
  `options.transcribe`: **default `true`** (auto-transcribe when no subtitles); set `false` to skip transcription (use subtitles/desc or degrade).
267
272
 
268
- **Transcription toolchain (identical for you and the author)**: the script **bootstraps itself** — it uses `uv` to create a **Python 3.12** environment + **Tsinghua mirror** to install `faster-whisper`, downloads the model via `HF_ENDPOINT=https://hf-mirror.com` and **caches it for reuse**; faster-whisper bundles PyAV to decode audio, so **no separate ffmpeg** is needed; **versions are pinned** (Python 3.12 / faster-whisper / uv) so the author's environment matches yours. The script is written in **Node**, so **Windows / macOS / Linux share one implementation and one command** (`node scripts/transcribe.js --audio <file> --out <txt>`). Transcription runs on **CPU** by default (`options.device`); GPU needs a CUDA runtime (cuBLAS/cuDNN), so `options.device: "auto"` is opt-in and falls back to CPU automatically. Cache location per platform: Windows `%LOCALAPPDATA%\deep-read-summarize`, macOS `~/Library/Caches/deep-read-summarize`, Linux `$XDG_CACHE_HOME/deep-read-summarize`. Python provisioning uses a mirror for speed and **falls back to a direct GitHub download when the mirror fails** (a dead mirror must not make the feature uninstallable); override it with the `DRS_UV_PYTHON_MIRROR` environment variable (empty string = never use a mirror).
273
+ **Transcription toolchain**: `uv` provisions **Python 3.12** and installs **`faster-whisper==1.2.1`** from the Tsinghua mirror; models use `HF_ENDPOINT=https://hf-mirror.com` and the local cache. Healthy cached environments run directly without requiring uv or a separate Python import probe. Actual interpreter and dependency versions are recorded in the cache's `runtime.json`; Python patch releases, uv, and all transitive dependencies are not fully pinned. PyAV decodes audio without a separate ffmpeg. All three platforms use `node scripts/transcribe.js --audio <file> --out <txt>`. CPU is the default; `options.device: "auto"` can try a supplied CUDA runtime and falls back to CPU. Cache roots are `%LOCALAPPDATA%\deep-read-summarize` on Windows, `~/Library/Caches/deep-read-summarize` on macOS, and `$XDG_CACHE_HOME/deep-read-summarize` on Linux (falling back to `~/.cache`). Python provisioning tries a mirror and then direct GitHub; `DRS_UV_PYTHON_MIRROR` overrides it, with an empty value selecting direct downloads.
274
+
275
+ > ⚠️ **The first transcription downloads the model (small ≈ 484 MB, via hf-mirror, not GitHub) and may take a few minutes**. Later runs reuse model files; new audio still requires full inference. Only a verified transcript cache hit skips inference. **Only subtitle-less videos trigger transcription** (subtitled videos use their subtitles directly).
276
+
277
+ **Transcript cache and threads**: `options.transcriptCache` defaults to true. Keys include the audio SHA-256, managed local model snapshots, installed dependencies, and decoding settings. Canonical segments retain real start/end times, so changing timestamp presentation does not repeat inference. Changed content/configuration/models/dependencies, custom local model directories, unknown environments, corrupt entries and expired entries bypass reuse. Cache lookup uses the existing local snapshot and does not check for upstream model updates online. Entries live under `transcripts-v1` in the cache root: 8 MiB per entry, 128 MiB total, 30-day validity, pruned on writes. Audio is not stored, and reading/source review still runs.
278
+
279
+ `options.cpuThreads` accepts integers 0–64. The default `0` preserves the library's CPU thread policy. Compare 4/6/8 on the same audio; more threads are not necessarily faster. CLI controls:
280
+
281
+ ```bash
282
+ node scripts/transcribe.js --audio sample.m4a --out transcript.txt --cpu-threads 8
283
+ node scripts/transcribe.js --audio sample.m4a --out transcript.txt --refresh-cache
284
+ node scripts/transcribe.js --audio sample.m4a --out transcript.txt --no-cache
285
+ node scripts/transcribe.js --clear-cache
286
+ ```
287
+
288
+ `--clear-cache` removes only transcripts, retaining models and Python. `--refresh-cache` forces inference and updates the cache; `--no-cache` disables reads and writes. Successful calls print one `metrics` JSON line with `cacheHit`, actual device, audio duration, and milliseconds for hashing, environment preparation, dependency installation, Python, publication and total time. Misses also include import, model loading, preparation (including decoding/VAD), and complete lazy-segment iteration. Hits do not reuse previous inference timings.
269
289
 
270
- > ⚠️ **The first transcription downloads the model first (small ≈ 484 MB, via hf-mirror, not GitHub) and may take a few minutes** — please be patient; **it downloads only once** on your machine, then every later transcription uses the cache and starts instantly. **Only subtitle-less videos trigger transcription** (subtitled videos use their subtitles directly — no trigger, no download).
290
+ The note's `qualityPassed` checks the note against acquired text; it does not establish ASR accuracy. ASR changes still need audio/reference comparisons for errors, omissions, numbers, negation and timestamps.
271
291
 
272
292
  **Quality/speed**: `small` + `int8` quantization + `VAD` silence filtering gives usable Chinese quality and is CPU-friendly. **Subtitled videos are not transcribed (fast)**; subtitle-less videos are transcribed at roughly 0.5–2× the video length on CPU.
273
293
 
@@ -298,7 +318,7 @@ Video text extraction is a **unified pipeline** (the old "three tiers" are gone)
298
318
  - **Chunking structure**: different types chunk differently (chapter/topic/paragraph), affecting **output structure** and coverage.
299
319
 
300
320
  ### Hardware
301
- - **GPU vs CPU**: GPU transcription can be an order of magnitude faster than CPU (`device='auto'` picks automatically). Subtitle-less videos are ≈ 0.5–2× the length on CPU.
321
+ - **GPU vs CPU**: gains depend on hardware, runtime libraries, model and audio. A visible GPU does not prove CUDA inference works; compare actual time and transcript accuracy.
302
322
 
303
323
  ### Other
304
324
  - **First run vs reuse**: the first run downloads the model (≈ 484 MB); later runs reuse the cache.
@@ -317,10 +337,22 @@ Two classes:
317
337
 
318
338
  ---
319
339
 
340
+ ## Faster runs with the same quality gates
341
+
342
+ - Prefer `deep_read_summarize` on supported hosts: the model submits parameters instead of reproducing the entire workflow script. Older hosts retain the existing route.
343
+ - All known source lines remain covered. Up to three reading tasks run concurrently, and each free slot immediately starts another task. Line-balanced chunks avoid one oversized task holding up a batch.
344
+ - Independent source review remains enabled, with duplicate full Map notes removed from review contexts. Missing line counts, failed reads, and description-only video sources cannot be fixed by rewriting the draft, so those states no longer consume full repair loops.
345
+ - Video `textSource: "desc"` is explicitly degraded and cannot pass as a complete deep read. Bilibili `p` selects the matching CID, part title, and duration.
346
+ - Returned `timings` reports stage durations in milliseconds. Compare the same sources, model, output size, and quality gates; scheduling simulations are not real LLM speedup measurements.
347
+
348
+ Defaults preserve small/int8, existing decoding settings, output length and independent source review. Transcript caching speeds up repeated audio; new audio can be profiled and tuned with CPU thread settings. Persistent ASR workers, batch inference and hierarchical review require separate benchmarks and are not enabled automatically.
349
+
320
350
  ## Cost
321
351
 
322
352
  Token use depends on document length, chunk count, and output size. Source review reads the document again, and each repair round adds a repair and a review call. `fastMode` reduces output sections; reducing `maxChunks` limits agent count while preserving the complete source.
323
353
 
354
+ Programmatic `lib/cache` callers: URL trailing slashes, explicit empty queries, and fragments are preserved because they may select different content. New fingerprints are isolated from old records, so an upgrade can cause one reprocessing pass. Callers must still verify source freshness and configuration before reusing results.
355
+
324
356
  ---
325
357
 
326
358
  ## Security & copyright
package/README.md CHANGED
@@ -1,9 +1,11 @@
1
1
  # deep-read-summarize
2
2
 
3
+ [English](https://github.com/PensiveFei/deep-read-summarize/blob/main/README.en.md) | [简体中文](https://github.com/PensiveFei/deep-read-summarize/blob/main/README.md)
4
+
3
5
  [![MIT license](https://img.shields.io/github/license/PensiveFei/deep-read-summarize)](https://github.com/PensiveFei/deep-read-summarize/blob/main/LICENSE)
4
6
  [![release](https://img.shields.io/github/v/release/PensiveFei/deep-read-summarize)](https://github.com/PensiveFei/deep-read-summarize/releases)
5
7
  [![CI](https://img.shields.io/github/actions/workflow/status/PensiveFei/deep-read-summarize/ci.yml)](https://github.com/PensiveFei/deep-read-summarize/actions/workflows/ci.yml)
6
- [![npm downloads](https://badgen.net/npm/dm/deep-read-summarize)](https://www.npmjs.com/package/deep-read-summarize)
8
+ [![npm total downloads](https://badgen.net/npm/dt/deep-read-summarize)](https://www.npmjs.com/package/deep-read-summarize)
7
9
 
8
10
  > **Disclaimer**: This is an **unofficial third-party tool**. It is not affiliated with, endorsed by, or sponsored by DeepSeek or the DeepSeek Harness project. "DeepSeek" and "DeepSeek Harness" are trademarks of their respective owners. This project only orchestrates the workflow tool available in your DSH environment; it does not redistribute any DeepSeek software.
9
11
 
@@ -96,7 +98,7 @@ workflow.js workflow 的 meta 与组装逻辑(解析器内联、schema
96
98
 
97
99
  > 脚本正文单独放 `lib/workflow-script.txt`:这个文件曾经因为「转义套转义」被改坏过两次,现在它是普通文本,读改都不需要编码步骤。
98
100
 
99
- 想换某个输入类型的处理方式,在 `custom-parsers/` 放一个同接口的解析器即可,同名类型会覆盖内置实现;用新类型名(如 `podcast`)也可以——注册表里的解析器会全部内联进 workflow 脚本,`type` 白名单与 schema 的 `kind` 枚举都跟着注册表走。接口只有三个字段:`name`、`types`、`buildPrompt(input, opts)`。
101
+ 想换某个输入类型的处理方式,在插件包的 `custom-parsers/` 放一个同接口的解析器即可,该目录会随 npm 包分发。也可在插件顶层 config 中指定 `customParsersDir`,使用自己的解析器目录;它不是 workflow options。相同类型会覆盖内置实现,也可新增 `podcast` 等类型。程序化注册后重新生成技能会使用最新注册表;各 profile 的专属目录相互隔离。接口为 `name`、`types`、`buildPrompt(input, opts)`。
100
102
 
101
103
  解析器的 CommonJS 模块会连同字面量相对路径的 `.js` / `.cjs` / `.json` 依赖一起打包,保留模块级常量、helper、方法简写和循环依赖。构建时使用 Acorn 解析 JavaScript;它由 npm 自动安装,不进入 workflow 沙箱。沙箱中的解析器只负责生成提示词,不支持 Node 内置模块、外部包或动态 `require` / `import`;这些依赖会在构建时明确报错。程序化注册的函数须自包含,模块闭包请使用文件解析器。
102
104
 
@@ -135,7 +137,8 @@ pnpm add ./deep-read-summarize-0.3.10.tgz
135
137
  ```
136
138
 
137
139
  安装后自动注册:
138
- - `deep-read-summarize` workflow(meta + script)
140
+ - `deep_read_summarize` 工具(宿主提供 tools / workflowEngine 服务时);模型只传 input/type/options,插件直接提交已加载的工作流
141
+ - `deep-read-summarize` 工作流定义(meta + script;旧宿主保留通过 workflow 工具调用的方式)
139
142
  - `deep-read-summarize` 技能(`skills/deep-read-summarize/SKILL.md`)
140
143
  - 四种解析器(`parsers/`)与 JSON Schema(`schemas/`)
141
144
 
@@ -149,18 +152,18 @@ pnpm add ./deep-read-summarize-0.3.10.tgz
149
152
  npm install deep-read-summarize
150
153
 
151
154
  # 方式 B:从源码开发
152
- git clone https://github.com/<your-org>/deep-read-summarize.git
155
+ git clone https://github.com/PensiveFei/deep-read-summarize.git
153
156
  cd deep-read-summarize
154
157
  npm ci # 安装锁定依赖
155
158
  ```
156
159
 
157
- ### 2. 验证环境(离线,不需要任何 API key)
160
+ ### 2. 验证安装(离线,不需要任何 API key)
158
161
 
159
162
  ```bash
160
- npm test # 35 项 fixture 测试,全部离线跑通
163
+ node -e "console.log(require('deep-read-summarize').name)"
161
164
  ```
162
165
 
163
- 看到 `TOTAL: 35 passed, 0 failed` 即环境就绪。
166
+ 看到 `deep-read-summarize` 即包可正常加载。从源码开发时再运行 `npm test` 和 `npm run test:node`;测试不随 npm 包分发。
164
167
 
165
168
  ### 3. 一行命令喂入 demo 文件
166
169
 
@@ -175,9 +178,9 @@ meta: deep-read-summarize
175
178
  parsers: book, paper, video, web
176
179
  ```
177
180
 
178
- ### 4. 真实运行(需要 DSH workflow 工具)
181
+ ### 4. 真实运行(需要 DSH)
179
182
 
180
- 把下面的 JSON 传给 DSH 的 workflow 工具(见下节「用法」):
183
+ 优先把下面的 JSON 直接传给 `deep_read_summarize` 工具。旧宿主使用技能提供的 meta + script,通过 `workflow` 工具执行:
181
184
 
182
185
  ```jsonc
183
186
  { "input": "https://arxiv.org/abs/2307.09042", "type": "paper", "options": { "maxChunks": 4, "fastMode": true } }
@@ -190,7 +193,7 @@ parsers: book, paper, video, web
190
193
  ---
191
194
  ## 用法
192
195
 
193
- 把下面这段 JSON 传给 DSH 的 workflow 工具:
196
+ `deep_read_summarize` 工具只需要以下参数;旧宿主将其作为 `workflow` 的 args,并使用技能中的 meta + script:
194
197
 
195
198
  ```jsonc
196
199
  {
@@ -204,6 +207,8 @@ parsers: book, paper, video, web
204
207
  "whisperModel": "small", // 视频转写模型档位:small | base | medium
205
208
  "language": "zh", // 视频转写语言:纯英文视频可设 "en"
206
209
  "device": "cpu", // 转写设备:cpu(默认)| auto | cuda;auto/cuda 需自备 CUDA 运行库,失败自动回退 cpu
210
+ "cpuThreads": 0, // 0 保持转写库默认值;1-64 可按实际硬件调优
211
+ "transcriptCache": true, // 音频内容与配置一致时复用本地逐字稿;false 强制每次推理
207
212
  "maxRetries": 0, // 质量校验重试次数(默认 0 = 不重试,重试会明显变慢)
208
213
  "frontmatter": "obsidian", // frontmatter 键集:obsidian(默认,配 Dataview)| generic(换掉 Hugo 保留键)| minimal
209
214
  "requireCitations": true, // 关键结论是否必须标注出处
@@ -264,9 +269,24 @@ parsers: book, paper, video, web
264
269
 
265
270
  `options.transcribe`:**默认 `true`**(无字幕自动转写);设为 `false` 则跳过转写(只用字幕/desc 或降级)。
266
271
 
267
- **转写工具链(本机/用户一致)**:脚本**自举**——用 `uv` 建 **Python 3.12** 环境 + **清华镜像**装 `faster-whisper` + `HF_ENDPOINT=https://hf-mirror.com` 下模型并**缓存复用**;faster-whisper 内置 PyAV 解码音频,**无需单独 ffmpeg**;**版本已锁定**(Python 3.12 / faster-whisper / uv),保证本机与用户环境一致。脚本是 **Node** 写的,**Windows / macOS / Linux 同一份实现、同一条命令**(`node scripts/transcribe.js --audio <音频> --out <txt>`);转写默认走 **CPU**(`options.device` 默认 `cpu`),GPU 需自备 CUDA 运行库(cuBLAS/cuDNN),可设 `options.device: "auto"` 尝试,失败会自动回退 CPU。缓存目录按平台取:Windows `%LOCALAPPDATA%\deep-read-summarize`、macOS `~/Library/Caches/deep-read-summarize`、Linux `$XDG_CACHE_HOME/deep-read-summarize`。 装 Python 时默认走镜像加速,**镜像失败会自动改用直连 GitHub 重试**(镜像挂掉不该让用户彻底装不上);可用环境变量 `DRS_UV_PYTHON_MIRROR` 覆盖(设为空字符串 = 完全不用镜像)。
272
+ **转写工具链**:脚本用 `uv` 创建 **Python 3.12** 环境,从清华镜像安装 **`faster-whisper==1.2.1`**,通过 `HF_ENDPOINT=https://hf-mirror.com` 下载并缓存模型。健康的缓存环境直接运行,不要求 uv 仍在 PATH,也不再启动单独的 Python 导入探测。实际解释器和依赖版本记录在缓存目录的 `runtime.json`;Python 补丁版本、uv 和传递依赖未全部锁定,不能据此声称环境逐字一致。PyAV 可直接解码音频,无需单独 ffmpeg。三平台使用同一条命令 `node scripts/transcribe.js --audio <音频> --out <txt>`,默认 CPU;`options.device: "auto"` 可尝试自备的 CUDA,失败回退 CPU。缓存目录:Windows `%LOCALAPPDATA%\deep-read-summarize`、macOS `~/Library/Caches/deep-read-summarize`、Linux `$XDG_CACHE_HOME/deep-read-summarize`(未设置时用 `~/.cache`)。创建 Python 环境默认先用镜像,失败再直连 GitHub;`DRS_UV_PYTHON_MIRROR` 可覆盖镜像,空字符串表示直连。
273
+
274
+ > ⚠️ **首次转写会先下载模型(small 约 484MB,走 hf-mirror,非 GitHub),可能需要几分钟**。之后复用模型文件;新音频仍需完整推理。同一音频命中经过校验的逐字稿缓存时,才会跳过推理。**只有「无字幕」视频才会触发转写**(有字幕的视频直接用字幕,不触发、不下载)。
275
+
276
+ **逐字稿缓存与线程**:`options.transcriptCache` 默认开启。缓存键包含音频 SHA-256、受管理的本地模型快照、已安装依赖及解码配置;存储真实分段和起止时间,切换时间戳展示不必重转写。更改内容、配置或本地模型/依赖会失效;自定义本地模型目录、无法确认环境、损坏和过期记录均跳过缓存。缓存复用本地已有快照,不在线探测模型上游是否更新。逐字稿保存在上述缓存根目录的 `transcripts-v1`,单条上限 8 MiB、总量上限 128 MiB、有效期 30 天;清理在写入时执行。它不保存音频,也不会跳过后续精读与原文核验。
277
+
278
+ `options.cpuThreads` 为 0–64 的整数,默认 `0` 保持库的 CPU 线程策略。可用同一音频对比 4/6/8 线程,线程越多不一定越快。CLI 可直接使用:
279
+
280
+ ```bash
281
+ node scripts/transcribe.js --audio sample.m4a --out transcript.txt --cpu-threads 8
282
+ node scripts/transcribe.js --audio sample.m4a --out transcript.txt --refresh-cache
283
+ node scripts/transcribe.js --audio sample.m4a --out transcript.txt --no-cache
284
+ node scripts/transcribe.js --clear-cache
285
+ ```
286
+
287
+ `--clear-cache` 只清理逐字稿,不删除模型或 Python 环境。`--refresh-cache` 强制推理并更新缓存,`--no-cache` 禁用读写。成功调用输出一行 `metrics` JSON,报告 `cacheHit`、实际设备、音频时长,以及哈希、环境准备、依赖安装、Python、发布和总毫秒耗时;未命中时还报告导入、模型加载、准备(含解码/VAD)与完整分段迭代耗时。缓存命中不会冒用上次推理耗时。
268
288
 
269
- > ⚠️ **首次转写会先下载模型(small 约 484MB,走 hf-mirror,非 GitHub),可能需要几分钟**,请耐心等待;**本机只下这一次**,之后所有视频转写用缓存、秒开。**只有「无字幕」视频才会触发转写**(有字幕的视频直接用字幕,不触发、不下载)。
289
+ 笔记的 `qualityPassed` 核验笔记与取得的原文,不证明 ASR 听写准确;调整转写路径仍须对照音频检查误字、漏段、数字、否定句与时间戳。
270
290
 
271
291
  **质量/速度**:`small` + `int8` 量化 + `VAD` 静音过滤,中文质量可用且 CPU 友好。**有字幕的视频不转写(快)**;无字幕视频转写,CPU 上约为视频时长的 0.5–2×。
272
292
 
@@ -297,7 +317,7 @@ parsers: book, paper, video, web
297
317
  - **分块结构**:不同类型分块方式不同(章节/主题/段落),影响**输出结构**与覆盖。
298
318
 
299
319
  ### 硬件
300
- - **GPU vs CPU**:GPU 转写可比 CPU 快一个数量级(`device='auto'` 自动选)。CPU 上无字幕视频约 0.5–2× 时长。
320
+ - **GPU vs CPU**:GPU 收益依赖显卡、运行库、模型与音频;设备可见不代表 CUDA 推理可用,应以实际耗时与文本对照为准。
301
321
 
302
322
  ### 其他
303
323
  - **首次 vs 复用**:首次要下模型(约 484MB),之后缓存复用。
@@ -316,10 +336,22 @@ parsers: book, paper, video, web
316
336
 
317
337
  ---
318
338
 
339
+ ## 保持质量的提速
340
+
341
+ - 支持的新宿主优先使用 `deep_read_summarize`,模型只提交参数,省去复制整份工作流脚本。旧宿主自动保留原入口。
342
+ - 精读仍完整覆盖已知原文,最多三个任务并发;空闲槽位立即领取下一块。按行数均衡工作量,避免一个超大块拖住整批。
343
+ - QC 继续独立读取原文;不再在每轮上下文中重复塞入全部 Map 笔记。缺总行数、精读失败或只有视频简介等问题不能靠改写笔记解决,不再浪费全文修订轮次。
344
+ - 视频 `textSource: "desc"` 明确为降级来源,不能取得完整精读的质量通过状态。B站多 P 链接按 `p` 选择对应 CID、分 P 标题和时长。
345
+ - 返回 `timings` 提供各阶段毫秒耗时,用同一资料、模型、篇幅和质量门槛比较;调度测试不能当成真实 LLM 加速比例。
346
+
347
+ 默认保持 small/int8、既有解码设置、篇幅与独立原文核验。逐字稿缓存主要加速重复处理,新视频可用线程配置和分阶段计时调优;常驻 ASR 模型、批量推理和分层 QC 仍需独立基准,不自动启用。
348
+
319
349
  ## 成本
320
350
 
321
351
  Token 用量取决于原文长度、分块数和成稿篇幅。原文核验会再次读取资料,每轮修订还会增加一次修订和一次核验调用。`fastMode` 减少输出章节;调低 `maxChunks` 限制代理数,同时保留全文覆盖。
322
352
 
353
+ 使用 `lib/cache` 的程序化调用方请注意:URL 尾斜杠、空 query 与 fragment 可能选择不同资料,现在会保留;新版指纹自动与旧版缓存隔离,升级后可能需要重新处理一次。该工具不负责证明远程内容未更新,复用前仍须由调用方确认来源与配置。
354
+
323
355
  ---
324
356
 
325
357
  ## 安全与版权
package/SECURITY.md CHANGED
@@ -6,7 +6,7 @@ Please do **not** open a public issue for security vulnerabilities.
6
6
 
7
7
  To report a vulnerability privately:
8
8
 
9
- 1. Open a [private security advisory](https://github.com/<your-org>/deep-read-summarize/security/advisories/new) (if available), or
9
+ 1. Open a [private security advisory](https://github.com/PensiveFei/deep-read-summarize/security/advisories/new) (if available), or
10
10
  2. Email the maintainer (see repository profile for contact).
11
11
 
12
12
  ## What to include
@@ -24,4 +24,4 @@ To report a vulnerability privately:
24
24
 
25
25
  ## Scope
26
26
 
27
- This project only ships workflow code, prompt templates, and schemas. It does not contain copyrighted content or secrets. Vulnerabilities of interest include prompt injection in parser prompts, sandbox escape attempts, and unsafe file handling.
27
+ This project only ships workflow code, prompt templates, and schemas. It does not contain copyrighted content or secrets. Vulnerabilities of interest include prompt injection in parser prompts, sandbox escape attempts, and unsafe file handling.
package/cordis.patch.yml CHANGED
@@ -14,6 +14,7 @@
14
14
  # = warning + ignore, never a startup failure) and folds them into the default
15
15
  # args carried by the skill, so they become this profile's defaults.
16
16
  # config:
17
+ # customParsersDir: './custom-parsers' # optional plugin-level parser directory (not a workflow option)
17
18
  # outputDir: './output' # where notes are written (default: ./output)
18
19
  # tempDir: './.tmp' # temp files for chunked content (default: ./.tmp)
19
20
  # minWords: 2500 # minimum output length (default: 2500)
@@ -25,6 +26,8 @@
25
26
  # whisperModel: 'small' # transcription model (default: small)
26
27
  # language: 'zh' # transcription language (default: zh)
27
28
  # device: 'cpu' # cpu | auto | cuda; GPU needs a CUDA runtime (default: cpu)
29
+ # cpuThreads: 0 # 0 keeps library defaults; 1-64 for measured CPU tuning
30
+ # transcriptCache: true # local content/config-verified transcript reuse (30 days, 128 MiB)
28
31
  # maxRetries: 0 # quality-check retry rounds (default: 0)
29
32
  # frontmatter: 'obsidian' # obsidian | generic | minimal (default: obsidian)
30
33
  # # generic swaps the keys Hugo reserves: type -> docType,
@@ -33,4 +36,4 @@
33
36
  # Idempotency is deliberately NOT a plugin config key: the workflow script runs
34
37
  # without a filesystem, so it cannot write a fingerprint record. Programmatic
35
38
  # callers can use `deep-read-summarize/lib/cache`
36
- # (fingerprint / hasProcessed / markProcessed).
39
+ # (fingerprint / hasProcessed / markProcessed).
@@ -0,0 +1,46 @@
1
+ # ASR performance check — 2026-11-01
2
+
3
+ This check compares the unmodified `6b95206` runner with the transcript-cache/thread/metrics changes. It measures transcription only, excluding video download, deep reading, drafting and independent source review.
4
+
5
+ ## Setup
6
+
7
+ - Windows, Intel Core i5-12500H (12 cores, 16 logical processors), 16 GB RAM; CPU execution.
8
+ - Node 24.19.0, faster-whisper 1.2.1, CTranslate2 4.8.1, PyAV 18.1.0.
9
+ - Existing `small` model snapshot `536b0662742c02347bc0e980a01041f333bce120`, int8, Chinese, existing beam/VAD/decoding defaults; model files already downloaded. HF offline mode prevented network/download variation.
10
+ - One self-authored Chinese passage, rendered locally by Microsoft Huihui Desktop to a 58.439-second WAV. The passage includes numeric contrasts and negation. No private audio was used.
11
+ - Each row starts a separate Node process. The first two use the library's default CPU thread policy; the final row explicitly selects 8. A transcript-cache hit still hashes the input, validates cache identity and publishes a new output.
12
+
13
+ ## Observations
14
+
15
+ | Run | Wall time | Lazy inference iteration | Text relative to baseline |
16
+ | --- | ---: | ---: | --- |
17
+ | Original runner | 24.517 s | not instrumented | baseline |
18
+ | New runner, forced inference | 23.118 s | 20.184 s | byte-identical |
19
+ | New runner, cached, timestamps enabled | 0.098 s | skipped | byte-identical after removing timestamp prefixes |
20
+ | New runner, 8 threads, cache disabled | 21.275 s | 18.306 s | byte-identical |
21
+
22
+ The new forced-inference run spent 0.549 s importing, 1.378 s loading the model, and 0.334 s preparing audio/VAD before consuming segments. The cache-hit call reported zero Python time. Stored segments were rendered with their real times; enabling timestamps did not call Python again.
23
+
24
+ These are single observations, not a latency distribution or a general speed guarantee. Execution order, OS/model-file caches and machine load can affect the difference between fresh runs. The unchanged default is intentional: this sample does not justify increasing CPU threads for every machine.
25
+
26
+ The normalized reference CER was 9.852% in all four rows. Normalization keeps Unicode letters/numbers, folds case and removes punctuation; it does not equate spoken number words with digits or traditional with simplified Chinese. The result establishes output preservation on this sample, not accurate listening for all videos. Real speech, noise, accents, mixed languages, long silence and long-form omissions need their own reference set. GPU and batch inference were not tested or enabled by this change.
27
+
28
+ ## Repeating the check
29
+
30
+ Generate audio from the reference below or use a licensed clip with a reference transcript. Keep the same audio, model snapshot, device, decoding settings and machine load for paired runs. Measure full process wall time and inspect the `metrics` JSON. The `inference` timer consumes the complete lazy segment iterator; timing only the return of `transcribe()` is insufficient.
31
+
32
+ ```bash
33
+ # Fresh result and cache population
34
+ node scripts/transcribe.js --audio sample.wav --out fresh.txt --refresh-cache
35
+ # Same audio, different presentation, no inference
36
+ node scripts/transcribe.js --audio sample.wav --out timestamped.txt --timestamps
37
+ # Thread comparisons must bypass the cache
38
+ node scripts/transcribe.js --audio sample.wav --out threads4.txt --cpu-threads 4 --no-cache
39
+ node scripts/transcribe.js --audio sample.wav --out threads8.txt --cpu-threads 8 --no-cache
40
+ ```
41
+
42
+ Check exact text, omissions, numbers, negation and timestamps as well as aggregate error rates. Keep cold environment/model download, fresh inference and transcript hits separate. Gather multiple representative clips and repeated runs before making a broader performance claim.
43
+
44
+ ## Self-authored reference
45
+
46
+ 今天讨论怎样检查学习笔记的准确性。第一,完整的音频转写不能只保留开头,也不能把视频简介当成全文。第二,数字和否定句需要特别检查。实验用了五十个样本,不是十五个样本;正确率是百分之九十二,而不是百分之九十七。第三,模型下载到磁盘以后,处理新的音频仍然需要推理。重复处理同一份音频时,可以验证内容哈希并复用逐字稿。第四,缓存命中以后,笔记仍然需要精读和独立核验,不能直接宣布质量通过。最后,请保留每一段的真实时间位置,遇到不确定的术语时明确标记,不要编造。
package/docs/RELEASE.md CHANGED
@@ -30,6 +30,19 @@
30
30
  - <未解决的问题或限制>
31
31
  - ...
32
32
 
33
+ ### 贡献者与致谢
34
+
35
+ - @<GitHub 用户名>:<代码、报告、复现、复测或文档等具体贡献>(<对应 Issue/PR 的完整链接>)。
36
+ - <没有外部贡献时如实说明;需要匿名时按贡献者意愿记录,不猜测真实姓名或邮箱。>
37
+
38
+ **归属清单(维护者发布前后各核对一次)**:
39
+
40
+ - [ ] 对照已合并 PR、已解决 Issue 和复测记录,核对用户名与贡献,保留作者/共同作者信息。
41
+ - [ ] 在目标版本的 **CHANGELOG.md 小节**中填好致谢;`npm run release` 从该小节原样生成 Notes,仅修改此模板不会进入发布说明。
42
+ - [ ] GitHub Generate release notes 可用 `.github/release.yml` 分类并列出 PR 作者;人工补齐报告者和复测者,不覆盖已有署名。
43
+ - [ ] 发布后核对公开 Release 的用户名、贡献说明和来源链接;遗漏时补记,保留原有说明、资产和 tag。
44
+ - [ ] 按 [贡献指南](../CONTRIBUTING.md) 在相关讨论中说明结果并及时致谢;敏感报告先确认披露范围。
45
+
33
46
  ---
34
47
 
35
48
  ## npm 发布流程
@@ -139,7 +152,9 @@ npm view deep-read-summarize --prefer-online # 验证线
139
152
  | v0.3.16 | npm 已发布 | 新增 `options.frontmatter` 三档(obsidian / generic / minimal),解决 Hugo 保留键冲突;默认档不变 |
140
153
  | v0.3.17 | npm 已发布 | 修 `resolve()` 原型链回退与质量清单 required;新增发布脚本 `scripts/release.js` |
141
154
  | v0.3.18 | npm 已发布 | 修复自动解析器分流、并发临时文件串稿和分块上限导致的全文遗漏 |
142
- | **v0.3.19** | **npm 已发布(当前)** | 修复 B04–B10,并让质量检查结合程序门禁和可回查的原文证据(O01)。 |
155
+ | v0.3.19 | npm 已发布 | 修复 B04–B10,并让质量检查结合程序门禁和可回查的原文证据(O01)。 |
156
+ | v0.4.0 | npm 已发布 | 保留完整精读与独立原文核验,减少脚本搬运、并发空等和无效修订,并修复来源质量判断与视频分P等问题。 |
157
+ | **v0.5.0** | **npm 已发布(当前)** | 无字幕视频新增逐字稿内容缓存、转写阶段计时和 CPU 线程配置,保留默认精读与原文核验门槛。 |
143
158
  | v1.0.0 | 计划 | API 稳定后发布 |
144
159
 
145
160
  > 早期迭代用 0.x 表达不稳定;API 稳定后再发 1.0.0。