deep-read-summarize 0.3.17 → 0.3.18
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +20 -0
- package/README.en.md +5 -1
- package/README.md +5 -1
- package/docs/RELEASE.md +3 -2
- package/lib/workflow-script.txt +102 -26
- package/package.json +1 -1
- package/parsers/_prompt.js +1 -1
- package/parsers/video.js +1 -1
- package/scripts/transcribe.js +36 -19
- package/skills/deep-read-summarize/SKILL.md +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,26 @@
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
The format is based on [Keep a Changelog](https://keepachangelog.com/).
|
|
5
5
|
|
|
6
|
+
## [0.3.18] — 2026-10-23
|
|
7
|
+
|
|
8
|
+
修复自动解析器分流、并发临时文件串稿和分块上限导致的全文遗漏。
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
|
|
12
|
+
- **`type: "auto"` 在获取前选择正确解析器**:明确的视频链接、arXiv 论文链接、EPUB/MOBI/AZW/AZW3 直接分流;同站的专栏/频道/帮助页、PDF、未知网页和文本先进行轻量类型识别,再获取全文。无法识别时返回 `stage: "detect"`,避免默默改走网页解析器。显式类型继续优先,成稿类型与实际使用的解析器保持一致。
|
|
13
|
+
- **每次运行隔离临时产物**:`options.tempDir` 作为根目录,任务使用独立 `run-*` 子目录,获取、音频、转写和精读共用本次路径,不再相互覆盖。转写使用独立暂存目录,成功退出且产物非空后原子发布;启动时移除旧目标,失败清理暂存文件,不会把旧逐字稿或半成品当成当前结果。成功信号在发布完成后输出,调用方必须核对本次进程退出状态。
|
|
14
|
+
- **分块数量收敛保留全部已知内容**:先校验、排序、去掉重复覆盖并补齐头部/中间/尾部缺口,再合并相邻区间至 `maxChunks`,不再丢掉计划尾部。保持 12 个精读代理硬上限;无法确认 `totalLines` 时保留已知区间并明确降级质量状态。
|
|
15
|
+
|
|
16
|
+
### Added
|
|
17
|
+
|
|
18
|
+
- 行为回归覆盖自动分流、PDF 书籍/论文分类、显式类型优先、并发 A/B 内容隔离、视频中间文件路径、跨平台临时根目录、超额/乱序/重叠/缺口分块,以及旧转写、空产物、失败清理和原子发布。
|
|
19
|
+
- 返回字段 `coverageComplete`:有有效总行数且计划已覆盖全部行号时为 `true`;总行数缺失时为 `false`,同时 `qualityPassed: false` 并附原因。
|
|
20
|
+
|
|
21
|
+
### Compatibility
|
|
22
|
+
|
|
23
|
+
- 不需要改配置;最终 Markdown 文件路径和 frontmatter 配置不变。临时文件改在配置根目录下的独立子目录,依赖固定 `tempDir/input.txt` 或 `tempDir/transcript.txt` 的外部脚本需要改为使用本次任务路径。
|
|
24
|
+
- 自动模式的模糊输入增加一次类型识别调用;显式设置 `type` 可跳过。
|
|
25
|
+
|
|
6
26
|
## [0.3.17] — 2026-10-20
|
|
7
27
|
|
|
8
28
|
修两个「对外暴露、内部没人盯着」的缺陷,并把发布流程自动化。
|
package/README.en.md
CHANGED
|
@@ -68,7 +68,11 @@ Wave 3 Merge draft + quality check (Reduce)
|
|
|
68
68
|
Markdown note
|
|
69
69
|
```
|
|
70
70
|
|
|
71
|
-
Three waves,
|
|
71
|
+
Three waves, normally N+3 sub-agent calls (fetch, N chunk reads, draft, and quality check). In `auto` mode, ambiguous PDFs, unfamiliar web pages, and text inputs require one additional lightweight classification call. An explicit `type` skips classification.
|
|
72
|
+
|
|
73
|
+
`auto` routes recognized video links, arXiv paper links, and ebook extensions directly to their parsers; columns, channels, and help pages on those sites still require classification. PDFs are classified as books or papers from their content. Failed classification returns `stage: "detect"` and asks for an explicit type. `options.tempDir` is a temporary root: each call creates its own `run-*` subdirectory, keeping fetched text, audio, and transcripts separate from other tasks.
|
|
74
|
+
|
|
75
|
+
Chunk plans are sorted, duplicate ranges trimmed, and gaps filled. Plans above `maxChunks` merge adjacent ranges instead of discarding the document tail. The returned `coverageComplete` indicates coverage of all declared lines based on a valid `totalLines`; without a line count, known ranges are retained, `qualityPassed` is set to `false`, and the coverage uncertainty is reported.
|
|
72
76
|
|
|
73
77
|
```
|
|
74
78
|
parsers/ Parsers for each input type, dispatched by type
|
package/README.md
CHANGED
|
@@ -68,7 +68,11 @@
|
|
|
68
68
|
Markdown 笔记
|
|
69
69
|
```
|
|
70
70
|
|
|
71
|
-
|
|
71
|
+
三个波次,默认约 N+3 次子代理调用(获取、N 块精读、成稿、质量校验)。`auto` 遇到 PDF、未知网页或文本等模糊输入时,额外进行一次轻量类型识别;显式指定 `type` 可跳过识别。
|
|
72
|
+
|
|
73
|
+
`auto` 会将明确的视频链接、arXiv 论文链接和电子书扩展名直接分发到对应解析器;同站的专栏、频道与帮助页仍先分类,PDF 会根据内容区分书籍与论文。无法识别时返回 `stage: "detect"`,请显式指定类型。`options.tempDir` 作为临时根目录,每次调用在其中创建独立的 `run-*` 子目录,获取、音频和转写文件不与其他任务共用。
|
|
74
|
+
|
|
75
|
+
分块计划会排序、裁掉重复区间并补齐缺口,超过 `maxChunks` 时合并相邻块以保留全文。返回的 `coverageComplete` 表示是否依据有效的 `totalLines` 覆盖全部行号;缺少总行数时保留已知范围,同时将 `qualityPassed` 置为 `false` 并说明覆盖无法确认。
|
|
72
76
|
|
|
73
77
|
```
|
|
74
78
|
parsers/ 各输入类型的解析器,按类型分发
|
package/docs/RELEASE.md
CHANGED
|
@@ -137,8 +137,9 @@ npm view deep-read-summarize --prefer-online # 验证线
|
|
|
137
137
|
| v0.3.14 | npm 已发布 | 修:路径尾分隔符、本地日期、config 白名单;README 测试数 |
|
|
138
138
|
| v0.3.15 | npm 已发布 | 工程卫生:锁定 LF、脚本正文独立成文、lint 扩检查、包瘦身、提示词改「标准 Markdown」 |
|
|
139
139
|
| v0.3.16 | npm 已发布 | 新增 `options.frontmatter` 三档(obsidian / generic / minimal),解决 Hugo 保留键冲突;默认档不变 |
|
|
140
|
-
|
|
|
140
|
+
| v0.3.17 | npm 已发布 | 修 `resolve()` 原型链回退与质量清单 required;新增发布脚本 `scripts/release.js` |
|
|
141
|
+
| **v0.3.18** | **npm 已发布(当前)** | 修复自动解析器分流、并发临时文件串稿和分块上限导致的全文遗漏 |
|
|
141
142
|
| v1.0.0 | 计划 | API 稳定后发布 |
|
|
142
143
|
|
|
143
144
|
> 早期迭代用 0.x 表达不稳定;API 稳定后再发 1.0.0。
|
|
144
|
-
> Tag 发布后保持稳定,出问题发修复版,不要反复改同一个 Tag。
|
|
145
|
+
> Tag 发布后保持稳定,出问题发修复版,不要反复改同一个 Tag。
|
package/lib/workflow-script.txt
CHANGED
|
@@ -12,13 +12,16 @@ const forcedType = ((args && args.type) || "auto").toLowerCase();
|
|
|
12
12
|
const options = (args && args.options) || {};
|
|
13
13
|
const minWords = options.minWords || 2500;
|
|
14
14
|
const fastMode = !!options.fastMode;
|
|
15
|
-
const maxChunks = Math.max(1, Math.min(options.maxChunks || 4, 12));
|
|
15
|
+
const maxChunks = Math.max(1, Math.min(Math.floor(Number(options.maxChunks) || 4), 12));
|
|
16
16
|
const requireCitations = options.requireCitations !== false;
|
|
17
17
|
const includeTimestamps = !!options.includeTimestamps;
|
|
18
18
|
const maxRetries = options.maxRetries || 0;
|
|
19
19
|
const outputDir = options.outputDir || "./output";
|
|
20
|
-
|
|
21
|
-
const
|
|
20
|
+
// tempDir 是根目录;每次调用使用独立子目录,获取、转写与 Map 必须共享同一个任务路径。
|
|
21
|
+
const tempRoot = options.tempDir || "./.tmp";
|
|
22
|
+
const runId = Date.now().toString(36) + "-" + Math.random().toString(36).slice(2) + Math.random().toString(36).slice(2);
|
|
23
|
+
const tempDir = tempRoot.replace(/[\\/]+$/, "") + "/run-" + runId;
|
|
24
|
+
const tempFile = tempDir + "/input.txt";
|
|
22
25
|
const whisperModel = options.whisperModel || "small";
|
|
23
26
|
const transcribeLanguage = options.language || "zh";
|
|
24
27
|
const transcribeDevice = options.device || "cpu";
|
|
@@ -46,6 +49,52 @@ if (forcedType !== "auto" && parserNames.indexOf(forcedType) === -1)
|
|
|
46
49
|
throw new Error("[FATAL] args.type 非法:" + forcedType + "(可用:" + ["auto"].concat(parserNames).join("|") + ")");
|
|
47
50
|
|
|
48
51
|
phase("解析器选择");
|
|
52
|
+
// 只对明确的来源做快速识别;PDF、未知网页和文本可能属于不同类型,交给轻量分类。
|
|
53
|
+
// 不依赖 URL 等宿主未承诺的全局 API,保持脚本可在 workflow 沙箱中运行。
|
|
54
|
+
function inferType(source) {
|
|
55
|
+
const url = source.match(/https?:\/\/[^\s<>"']+/i);
|
|
56
|
+
const target = url ? url[0] : source;
|
|
57
|
+
const hostMatch = target.match(/^https?:\/\/(?:[^\/?#]*@)?([^\/:?#]+)/i);
|
|
58
|
+
const host = hostMatch ? hostMatch[1].toLowerCase() : "";
|
|
59
|
+
const pathMatch = target.match(/^https?:\/\/[^\/?#]+([^?#]*)/i);
|
|
60
|
+
const urlPath = pathMatch ? pathMatch[1] : "";
|
|
61
|
+
const queryMatch = target.split("#")[0].match(/\?([\s\S]*)$/);
|
|
62
|
+
const query = queryMatch ? queryMatch[1] : "";
|
|
63
|
+
// 同一站点也有专栏、频道和帮助页,只对明确的视频/论文内容路径快速分流。
|
|
64
|
+
if (/(^|\.)(youtu\.be|b23\.tv|v\.douyin\.com)$/.test(host) && /^\/[^/]+/.test(urlPath)) return "video";
|
|
65
|
+
if (/(^|\.)(youtube\.com|youtube-nocookie\.com)$/.test(host) &&
|
|
66
|
+
(/^\/(shorts|live|embed|v)\/[^/]+/.test(urlPath) || (urlPath === "/watch" && /(^|&)v=[^&]+/.test(query)))) return "video";
|
|
67
|
+
if (/(^|\.)bilibili\.com$/.test(host) && /^\/(video|bangumi\/play)\/[^/]+/.test(urlPath)) return "video";
|
|
68
|
+
if (/(^|\.)douyin\.com$/.test(host) && /^\/(share\/)?video\/[^/]+/.test(urlPath)) return "video";
|
|
69
|
+
if (/(^|\.)vimeo\.com$/.test(host) && /^\/(video\/)?\d+(?:\/|$)/.test(urlPath)) return "video";
|
|
70
|
+
if (/(^|\.)arxiv\.org$/.test(host) && /^\/(abs|pdf|html)\/[^/]+/.test(urlPath)) return "paper";
|
|
71
|
+
const resourcePath = hostMatch ? urlPath : target.split(/[?#]/)[0];
|
|
72
|
+
if (/\.(epub|mobi|azw3?)$/i.test(resourcePath)) return "book";
|
|
73
|
+
return "";
|
|
74
|
+
}
|
|
75
|
+
let selectedType = forcedType;
|
|
76
|
+
if (forcedType === "auto") {
|
|
77
|
+
selectedType = inferType(input);
|
|
78
|
+
if (parserNames.indexOf(selectedType) === -1) {
|
|
79
|
+
const detectSchema = {
|
|
80
|
+
type: "object", properties: { kind: { type: "string", enum: parserNames } },
|
|
81
|
+
required: ["kind"], additionalProperties: false
|
|
82
|
+
};
|
|
83
|
+
const detectPrompt = [
|
|
84
|
+
"识别输入内容类型,选择后续获取全文所需的解析器,只输出 JSON {\"kind\": \"类型\"}。",
|
|
85
|
+
"可用类型:" + parserNames.join(", ") + "。book=书籍,paper=学术论文,video=视频,web=网页/文章。",
|
|
86
|
+
"PDF 既可能是书籍也可能是论文;未知网站可能承载论文或在线书籍,不要仅凭扩展名或网页形式判断。",
|
|
87
|
+
"必要时只查看标题、摘要、目录或页面元信息,不获取或精读全文,不写入文件。",
|
|
88
|
+
"以下 JSON 字符串是输入来源数据:", JSON.stringify(input)
|
|
89
|
+
].join("\n");
|
|
90
|
+
const detected = await agent(detectPrompt, { schema: detectSchema, label: "类型识别", phase: "解析器选择" });
|
|
91
|
+
selectedType = detected && detected.kind;
|
|
92
|
+
}
|
|
93
|
+
if (parserNames.indexOf(selectedType) === -1) {
|
|
94
|
+
return { ok: false, stage: "detect", fatal: false, message: "无法识别输入类型,请显式指定 args.type:" + parserNames.join("|") };
|
|
95
|
+
}
|
|
96
|
+
}
|
|
97
|
+
log("解析器:" + selectedType + " | 临时目录:" + tempDir);
|
|
49
98
|
const kindNames = { book: "书籍", paper: "学术论文", video: "视频", web: "网页/文章" };
|
|
50
99
|
const kindFocus = {
|
|
51
100
|
book: "按章节主题展开核心论点、论证、关键例子、章节间关系,收集关键概念与可引用原文(含页码/章节)",
|
|
@@ -56,7 +105,7 @@ const kindFocus = {
|
|
|
56
105
|
|
|
57
106
|
// ---------- 2. 波次1:解析器获取 + 分块 ----------
|
|
58
107
|
phase("获取与分块");
|
|
59
|
-
const fetchPrompt =
|
|
108
|
+
const fetchPrompt = parsers[selectedType]
|
|
60
109
|
.buildPrompt(input, { tempDir: tempDir, tempFile: tempFile, maxChunks: maxChunks, transcribe: options.transcribe !== false, whisperModel: whisperModel, language: transcribeLanguage, device: transcribeDevice });
|
|
61
110
|
// fetchSchema 由插件在构建时从 schemas/index.js 注入(单一真相,避免两份 schema 漂移)。
|
|
62
111
|
|
|
@@ -66,36 +115,58 @@ const fetched = await agent(fetchPrompt, { schema: fetchSchema, label: "获取+
|
|
|
66
115
|
if (!fetched || !fetched.saved) {
|
|
67
116
|
return { ok: false, stage: "fetch", fatal: false, message: (fetched && fetched.message) || "获取失败(可降级:检查输入或手动提供文本)" };
|
|
68
117
|
}
|
|
69
|
-
//
|
|
70
|
-
//
|
|
118
|
+
// 先校验完整计划,按行排序、裁掉重复覆盖并补齐缺口,再合并相邻块。
|
|
119
|
+
// 块数上限约束的是子代理数量,不能通过丢掉尾部原文来实现。
|
|
71
120
|
const rawPlan = (fetched && fetched.chunkPlan) || [];
|
|
72
121
|
if (rawPlan.length === 0) return { ok: false, stage: "fetch", fatal: false, message: "分块计划为空" };
|
|
73
|
-
const
|
|
74
|
-
const
|
|
75
|
-
|
|
122
|
+
const lineCount = Number(fetched.totalLines);
|
|
123
|
+
const declaredLines = Number.isFinite(lineCount) && lineCount > 0 ? Math.floor(lineCount) : 0;
|
|
124
|
+
const validPlan = [];
|
|
125
|
+
for (let i = 0; i < rawPlan.length; i++) {
|
|
76
126
|
const c = rawPlan[i] || {};
|
|
77
|
-
const s = Math.floor(Number(c.startLine)
|
|
78
|
-
const e = Math.floor(Number(c.endLine)
|
|
79
|
-
if (s < 1 || e < s) continue;
|
|
127
|
+
const s = Math.floor(Number(c.startLine));
|
|
128
|
+
const e = Math.floor(Number(c.endLine));
|
|
129
|
+
if (!Number.isFinite(s) || !Number.isFinite(e) || s < 1 || e < s) continue;
|
|
80
130
|
if (declaredLines > 0 && s > declaredLines) continue;
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
topic: String(c.topic || ("块 " + (chunkPlan.length + 1))),
|
|
131
|
+
validPlan.push({
|
|
132
|
+
topic: String(c.topic || ("块 " + (i + 1))),
|
|
84
133
|
startLine: s,
|
|
85
134
|
endLine: declaredLines > 0 ? Math.min(e, declaredLines) : e
|
|
86
135
|
});
|
|
87
136
|
}
|
|
88
|
-
if (
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
if (
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
137
|
+
if (validPlan.length === 0) return { ok: false, stage: "fetch", fatal: false, message: "分块计划无效:行号区间缺失或非法" };
|
|
138
|
+
validPlan.sort(function (a, b) { return a.startLine - b.startLine || a.endLine - b.endLine; });
|
|
139
|
+
const coveredPlan = [];
|
|
140
|
+
let nextLine = 1;
|
|
141
|
+
for (let i = 0; i < validPlan.length; i++) {
|
|
142
|
+
const c = validPlan[i];
|
|
143
|
+
if (c.startLine > nextLine) {
|
|
144
|
+
coveredPlan.push({ topic: "补充未规划内容", startLine: nextLine, endLine: c.startLine - 1 });
|
|
145
|
+
}
|
|
146
|
+
const startLine = Math.max(nextLine, c.startLine);
|
|
147
|
+
if (c.endLine >= startLine) coveredPlan.push({ topic: c.topic, startLine: startLine, endLine: c.endLine });
|
|
148
|
+
nextLine = Math.max(nextLine, c.endLine + 1);
|
|
149
|
+
}
|
|
150
|
+
if (declaredLines > 0 && nextLine <= declaredLines) {
|
|
151
|
+
coveredPlan.push({ topic: "补充尾部内容", startLine: nextLine, endLine: declaredLines });
|
|
98
152
|
}
|
|
153
|
+
const chunkPlan = [];
|
|
154
|
+
const groups = Math.min(maxChunks, coveredPlan.length);
|
|
155
|
+
for (let i = 0; i < groups; i++) {
|
|
156
|
+
const from = Math.floor(i * coveredPlan.length / groups);
|
|
157
|
+
const to = Math.floor((i + 1) * coveredPlan.length / groups);
|
|
158
|
+
const parts = coveredPlan.slice(from, to);
|
|
159
|
+
chunkPlan.push({
|
|
160
|
+
id: i + 1, topic: parts.map(function (c) { return c.topic; }).join(" / "),
|
|
161
|
+
startLine: parts[0].startLine, endLine: parts[parts.length - 1].endLine
|
|
162
|
+
});
|
|
163
|
+
}
|
|
164
|
+
const coverageComplete = declaredLines > 0;
|
|
165
|
+
if (!coverageComplete) log("警告:获取结果缺少有效 totalLines,只能确认已知行号范围的覆盖");
|
|
166
|
+
log("分块计划:" + rawPlan.length + " -> " + chunkPlan.length + " 块(上限 " + maxChunks + ",保留全部已知内容)");
|
|
167
|
+
|
|
168
|
+
// 成稿类型必须与实际执行的解析器一致,不能在获取后仅改变标签。
|
|
169
|
+
const kind = selectedType;
|
|
99
170
|
const metadata = (fetched && fetched.metadata) || {};
|
|
100
171
|
const textSource = (metadata && metadata.textSource) || "unknown";
|
|
101
172
|
const srcLabels = { desc: "视频简介/desc", subtitle: "平台字幕", transcription: "完整语音转写", manual: "用户提供转写文本" };
|
|
@@ -233,11 +304,16 @@ for (let round = 0; round <= maxRetries; round++) {
|
|
|
233
304
|
}
|
|
234
305
|
|
|
235
306
|
// ---------- 结果 ----------
|
|
307
|
+
if (!coverageComplete) {
|
|
308
|
+
passed = false;
|
|
309
|
+
issues = issues.concat(["获取结果缺少有效 totalLines,无法确认原文尾部是否完整覆盖"]);
|
|
310
|
+
}
|
|
236
311
|
return {
|
|
237
312
|
ok: true, kind: kind, title: finalTitle,
|
|
238
313
|
filePath: mdPath, chunksCount: chunkPlan.length, fastMode: fastMode,
|
|
239
314
|
qualityPassed: passed, qualityIssues: issues,
|
|
240
315
|
failedChunks: failedChunks,
|
|
316
|
+
coverageComplete: coverageComplete,
|
|
241
317
|
textSource: textSource,
|
|
242
318
|
note: verified
|
|
243
|
-
};
|
|
319
|
+
};
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "deep-read-summarize",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.18",
|
|
4
4
|
"description": "Deep reading & summarization for DSH: books/papers/videos/web → structured Markdown notes (Obsidian / Logseq / static sites). Plugin parsers, MapReduce deep-read, JSON Schema-constrained sub-tasks.",
|
|
5
5
|
"type": "commonjs",
|
|
6
6
|
"main": "index.js",
|
package/parsers/_prompt.js
CHANGED
|
@@ -34,7 +34,7 @@ function buildFetchPrompt(cfg) {
|
|
|
34
34
|
"目标文件:" + tempFile + "(先创建目录 " + tempDir + ",用 write 工具写入)",
|
|
35
35
|
"步骤:"
|
|
36
36
|
].concat(steps, [
|
|
37
|
-
(steps.length + 1) + ". 分块:按" + cfg.chunkRule + "切分,每块约 100-200 行或 4000-6000 字,块数不超过 " + maxChunks + " 块;每块给 id/topic/startLine/endLine
|
|
37
|
+
(steps.length + 1) + ". 分块:按" + cfg.chunkRule + "切分,每块约 100-200 行或 4000-6000 字,块数不超过 " + maxChunks + " 块;每块给 id/topic/startLine/endLine。必须完整覆盖第 1 行到 totalLines,不遗漏开头、块间或尾部内容;超过块数上限时合并相邻块,不要截断原文。",
|
|
38
38
|
"输出 JSON:",
|
|
39
39
|
cfg.jsonExample,
|
|
40
40
|
"若无法获取文本输出 { \"saved\": false, \"message\": \"原因\" }。只输出 JSON。"
|
package/parsers/video.js
CHANGED
|
@@ -65,7 +65,7 @@ module.exports = {
|
|
|
65
65
|
" ① 先取音频:B站用 playurl(curl 'https://api.bilibili.com/x/player/playurl?avid=<aid>&cid=<cid>&fnval=16&fourk=1' 的 dash.audio[0].baseUrl)或 yt-dlp,下载存到 '" + tempDir + "/audio.m4a';\n" +
|
|
66
66
|
" ② 定位插件转写脚本(文件名固定 `transcribe.js`):Windows 用 `Get-ChildItem -Path $env:USERPROFILE/.dsh -Recurse -Filter transcribe.js`;macOS/Linux 用 `find \"$HOME/.dsh\" -name transcribe.js`(也可直接用 glob 搜 *transcribe.js);\n" +
|
|
67
67
|
" ③ 运行(**三平台命令行完全一致**,脚本是 Node 写的,与 pwsh / bash 无关):`node <脚本路径> --audio '" + tempDir + "/audio.m4a' --out '" + tempDir + "/transcript.txt' --device " + device + " --model " + whisperModel + " --language " + language + "`;\n" +
|
|
68
|
-
" ④
|
|
68
|
+
" ④ 转写可能较慢:可用 run_in_background: true 启动,但必须跟踪**本次启动的进程**直到结束;只有确认**本次进程退出码为 0**,且本次目标文件 '" + tempDir + "/transcript.txt' 存在并非空,才算成功。脚本先写本次独有的暂存文件,校验后才原子发布目标文件;**不要仅凭文件非空/稳定或日志出现 DONE 判断完成,不要复用其他任务或上次运行的逐字稿**。设合理超时,超时则终止本次进程;超时/非 0 退出/空产物均按降级处理;\n" +
|
|
69
69
|
" ⑤ 成功后读取 transcript.txt 作为全文(textSource=transcription)。脚本自举:uv 建 Python 3.12 环境 + 镜像装 faster-whisper + hf-mirror 下模型并缓存;Windows/macOS/Linux 是**同一份实现**。**⚠️ 首次转写会先下载模型(small 约 484MB,hf-mirror,非 GitHub)并较慢——先向用户说明这是正常的一次性下载,勿当成卡死;之后缓存复用、秒开。**\n" +
|
|
70
70
|
" ⑥ 若 `node` 不在 PATH(macOS 从图形界面启动时常见),先用 `command -v node` 确认;确认不可用就直接降级,不要为了跑转写脚本去安装 Node。"
|
|
71
71
|
);
|
package/scripts/transcribe.js
CHANGED
|
@@ -152,15 +152,15 @@ function venvEnvAttempts(baseEnv, mirror) {
|
|
|
152
152
|
/**
|
|
153
153
|
* 判定转写产物是否可用。
|
|
154
154
|
* 0 字节 = VAD 把整段音频滤掉了(纯静音 / 纯 BGM)或解码失败。此时**不能算成功**:
|
|
155
|
-
*
|
|
156
|
-
*
|
|
155
|
+
* 只有本次进程退出码为 0 且发布了非空输出才算完成;空产物保留在本次暂存目录,
|
|
156
|
+
* 清理后以非 0 退出(code 3),不会发布到调用方读取的最终路径。
|
|
157
157
|
*/
|
|
158
158
|
function judgeTranscript(bytes) {
|
|
159
159
|
if (bytes > 0) return { ok: true, code: 0, message: '' };
|
|
160
160
|
return {
|
|
161
161
|
ok: false,
|
|
162
162
|
code: 3,
|
|
163
|
-
message: '转写结果为空(0 字节):音频可能是纯静音 /
|
|
163
|
+
message: '转写结果为空(0 字节):音频可能是纯静音 / 纯音乐,或解码失败。未发布空输出文件,请按降级处理(让用户提供文案,或用 options.transcribe=false)。'
|
|
164
164
|
};
|
|
165
165
|
}
|
|
166
166
|
|
|
@@ -223,11 +223,10 @@ const PY_RUNNER = [
|
|
|
223
223
|
" print('[transcribe] device=%s failed (%s); retrying on cpu' % (device, exc), file=sys.stderr)",
|
|
224
224
|
" segments, info = run('cpu')",
|
|
225
225
|
"",
|
|
226
|
-
"#
|
|
227
|
-
"#
|
|
226
|
+
"# DRS_OUT 是本次调用独有的暂存路径;Node 校验后才原子发布到最终路径。",
|
|
227
|
+
"# 先收集全部分段再落盘,中途推理失败不会留下半截文本。",
|
|
228
228
|
"with open(os.environ['DRS_OUT'], 'w', encoding='utf-8') as f:",
|
|
229
229
|
" f.write(' '.join([x for x in segments if x]))",
|
|
230
|
-
"print('DONE', info.language, round(info.duration, 1))",
|
|
231
230
|
""
|
|
232
231
|
].join("\n");
|
|
233
232
|
|
|
@@ -284,16 +283,29 @@ function main(argv) {
|
|
|
284
283
|
process.stderr.write(USAGE + '\n');
|
|
285
284
|
return fail('--audio 与 --out 都是必填的。');
|
|
286
285
|
}
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
if (uv === null) {
|
|
290
|
-
return fail('没有找到 uv。请先安装:' + uvInstallHint(c) + '(装好后重跑本脚本;或改用 options.transcribe=false / 手动提供转写文本降级)');
|
|
291
|
-
}
|
|
292
|
-
|
|
286
|
+
const outputPath = path.resolve(args.out);
|
|
287
|
+
let stagingDir = null;
|
|
293
288
|
try {
|
|
289
|
+
// 清理旧结果前先保护输入,避免 --audio 与 --out 指向同一文件。
|
|
290
|
+
if (path.resolve(args.audio) === outputPath ||
|
|
291
|
+
(fs.existsSync(args.audio) && fs.existsSync(outputPath) &&
|
|
292
|
+
fs.realpathSync(args.audio) === fs.realpathSync(outputPath))) {
|
|
293
|
+
return fail('--audio 与 --out 不能指向同一文件。');
|
|
294
|
+
}
|
|
295
|
+
// 从本次调用开始就不允许旧逐字稿继续充当“已完成”的产物。
|
|
296
|
+
fs.rmSync(outputPath, { force: true });
|
|
297
|
+
if (!fs.existsSync(args.audio)) return fail('音频文件不存在:' + args.audio);
|
|
298
|
+
if (!isFileAt(args.audio)) return fail('--audio 不是文件(是不是给了目录?):' + args.audio);
|
|
299
|
+
if (uv === null) {
|
|
300
|
+
return fail('没有找到 uv。请先安装:' + uvInstallHint(c) + '(装好后重跑本脚本;或改用 options.transcribe=false / 手动提供转写文本降级)');
|
|
301
|
+
}
|
|
302
|
+
|
|
294
303
|
fs.mkdirSync(root, { recursive: true });
|
|
295
304
|
// 输出目录可能在别处(如 ./.tmp),提前建好,免得 python 侧报一句难懂的错
|
|
296
|
-
fs.mkdirSync(path.dirname(
|
|
305
|
+
fs.mkdirSync(path.dirname(outputPath), { recursive: true });
|
|
306
|
+
// 与最终文件处于同一文件系统,rename 才能原子发布;每次调用独占暂存目录。
|
|
307
|
+
stagingDir = fs.mkdtempSync(path.join(path.dirname(outputPath), '.drs-transcribe-'));
|
|
308
|
+
const stagingOutput = path.join(stagingDir, 'transcript.txt');
|
|
297
309
|
|
|
298
310
|
const childEnv = Object.assign({}, process.env);
|
|
299
311
|
|
|
@@ -330,28 +342,33 @@ function main(argv) {
|
|
|
330
342
|
childEnv.HF_HUB_DISABLE_XET = '1';
|
|
331
343
|
childEnv.HF_HUB_DISABLE_SYMLINKS_WARNING = '1';
|
|
332
344
|
childEnv.DRS_AUDIO = args.audio;
|
|
333
|
-
childEnv.DRS_OUT =
|
|
345
|
+
childEnv.DRS_OUT = stagingOutput;
|
|
334
346
|
childEnv.DRS_MODEL = args.model;
|
|
335
347
|
childEnv.DRS_LANG = args.language;
|
|
336
348
|
childEnv.DRS_DEVICE = args.device;
|
|
337
349
|
|
|
338
|
-
const pyScript = path.join(
|
|
350
|
+
const pyScript = path.join(stagingDir, 'transcribe_run.py');
|
|
339
351
|
fs.writeFileSync(pyScript, PY_RUNNER, 'utf8');
|
|
340
352
|
|
|
341
353
|
// 4) 转写(首次会下载模型到 HF_HOME,之后缓存复用)
|
|
342
354
|
runStep(py, [pyScript], childEnv, '转写');
|
|
343
355
|
|
|
344
|
-
const bytes = fs.statSync(
|
|
345
|
-
process.stdout.write('TRANSCRIBED ' + bytes + ' bytes to ' + args.out + '\n');
|
|
356
|
+
const bytes = fs.statSync(stagingOutput).size;
|
|
346
357
|
const verdict = judgeTranscript(bytes);
|
|
347
358
|
if (!verdict.ok) {
|
|
348
|
-
// 删掉空文件:留着它会让「文件存在」被当成完成
|
|
349
|
-
try { fs.rmSync(args.out, { force: true }); } catch (e) { /* 删不掉也要如实报错 */ }
|
|
350
359
|
return fail(verdict.message, verdict.code);
|
|
351
360
|
}
|
|
361
|
+
fs.renameSync(stagingOutput, outputPath);
|
|
362
|
+
process.stdout.write('TRANSCRIBED ' + bytes + ' bytes to ' + args.out + '\n');
|
|
363
|
+
process.stdout.write('DONE\n');
|
|
352
364
|
return 0;
|
|
353
365
|
} catch (e) {
|
|
354
366
|
return fail(e.message + '(可降级:让用户手动提供转写文本,或用 options.transcribe=false 跳过转写)');
|
|
367
|
+
} finally {
|
|
368
|
+
if (stagingDir) {
|
|
369
|
+
try { fs.rmSync(stagingDir, { recursive: true, force: true }); }
|
|
370
|
+
catch (e) { process.stderr.write('[transcribe] 暂存目录清理失败:' + stagingDir + '(' + e.message + ')\n'); }
|
|
371
|
+
}
|
|
355
372
|
}
|
|
356
373
|
}
|
|
357
374
|
|
|
@@ -36,8 +36,8 @@ whenToUse: 用户提供一本书、论文(arXiv/PDF)、视频链接(YouTub
|
|
|
36
36
|
|
|
37
37
|
1. **配置校验**:缺 input 或 type 非法 → 报 FATAL 终止
|
|
38
38
|
2. **幂等检查(由你执行,脚本不做)**:脚本在沙箱内没有文件系统,写不了指纹记录。落盘前先看目标 `filePath` 是否已有同名笔记——已存在就问用户是覆盖还是跳过,别默默重跑一遍烧 token。(要在程序里做去重,可用插件导出的 `lib/cache.js`:fingerprint / hasProcessed / markProcessed。)
|
|
39
|
-
3. **解析器选择**:按类型从 parsers 注册表选解析器(book/paper/video/web
|
|
40
|
-
4. **波次1 获取+分块**:解析器获取全文 →
|
|
39
|
+
3. **解析器选择**:按类型从 parsers 注册表选解析器(book/paper/video/web)。`auto` 对明确视频链接、arXiv 论文链接、电子书扩展名直接分流;同站的图文/频道/帮助页、PDF/未知网页/文本等模糊输入先轻量识别,不能固定当作网页。识别失败返回 `stage: "detect"`,再请用户显式指定类型
|
|
40
|
+
4. **波次1 获取+分块**:解析器获取全文 → 写入本次独立的临时子目录 → 生成分块计划(JSON Schema 校验)。`tempDir` 是根目录,不要复用其他 `run-*` 任务的 input/audio/transcript;分块超过上限时合并相邻区间,保持从第 1 行到 totalLines 的完整覆盖
|
|
41
41
|
5. **波次2 并行精读**:每个分块一个子代理深度精读(Map)
|
|
42
42
|
6. **波次3 合并成稿**:整合为完整笔记(Reduce),内嵌质量自检
|
|
43
43
|
7. **质量校验**:可选重试(maxRetries),检查覆盖度/引用真实性/术语一致/格式完整/篇幅/语言
|
|
@@ -46,7 +46,7 @@ whenToUse: 用户提供一本书、论文(arXiv/PDF)、视频链接(YouTub
|
|
|
46
46
|
|
|
47
47
|
## 输出
|
|
48
48
|
|
|
49
|
-
workflow 返回 `{ ok, filePath, note, qualityPassed, qualityIssues, failedChunks, textSource }
|
|
49
|
+
workflow 返回 `{ ok, filePath, note, qualityPassed, qualityIssues, failedChunks, coverageComplete, textSource }`。`coverageComplete: false` 表示获取结果缺少有效总行数,无法确认尾部覆盖;此时质量状态也会降级,必须向用户说明。
|
|
50
50
|
|
|
51
51
|
> ⚠️ **`note` 就是完整笔记正文,`filePath` 只是建议路径**:脚本运行在无文件系统的沙箱里(DSH 的 workflow 契约:脚本只负责协调子代理),**它不会替你写文件**。必须由你(主代理)在拿到结果后用 `write` 工具把它写到 `filePath`——默认 `./output/`,可用 `options.outputDir` 指向 Obsidian 仓库或静态站点的内容目录。
|
|
52
52
|
|