deep-read-summarize 0.3.18 → 0.3.19
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +25 -0
- package/README.en.md +11 -5
- package/README.md +11 -5
- package/docs/RELEASE.md +2 -1
- package/index.js +4 -53
- package/lib/cache.js +23 -3
- package/lib/options.js +66 -0
- package/lib/parser-bundle.js +197 -0
- package/lib/quality.js +198 -0
- package/lib/workflow-script.txt +94 -36
- package/package.json +4 -1
- package/parsers/index.js +32 -7
- package/parsers/video.js +5 -1
- package/schemas/index.js +3 -2
- package/scripts/transcribe.js +28 -8
- package/skills/deep-read-summarize/SKILL.md +3 -2
- package/workflow.js +10 -16
package/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,31 @@
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
The format is based on [Keep a Changelog](https://keepachangelog.com/).
|
|
5
5
|
|
|
6
|
+
## [0.3.19] — 2026-10-24
|
|
7
|
+
|
|
8
|
+
修复 B04–B10,并让质量检查结合程序门禁和可回查的原文证据(O01)。
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
|
|
12
|
+
- **成稿与修订保留(B04)**:拒绝空白和非字符串初稿;修订调用失败、返回空值或格式/篇幅退化时保留上一份有效笔记。质量检查失败不再导致正常初稿丢失。
|
|
13
|
+
- **自定义解析器(B05)**:使用 Acorn 解析 CommonJS 模块并打包本地依赖,保留方法简写、模块常量、helper、JSON 和循环依赖。无法在沙箱支持的 Node/外部依赖和动态加载在构建时明确报错;独立的程序化函数仍可直接注册。
|
|
14
|
+
- **视频定位(B06)**:`includeTimestamps` 贯穿获取和转写,按真实分段 start/end 输出毫秒时间范围;平台字幕也保留 cue 时间。关闭时间戳仍逐段换行,不再把长逐字稿压成一行。
|
|
15
|
+
- **配置边界(B07)**:workflow 与插件 config 共用同一份默认值和校验器。检查有限整数、区间、布尔、路径和枚举;直接调用非法参数 FATAL,profile config 逐项告警回退。重试次数限制为 0–3。
|
|
16
|
+
- **转写完成信号(B08)**:复验上一版的修复;仅在本次进程成功、非空产物校验并原子发布后输出 DONE,空结果和所有失败路径均不输出成功信号。
|
|
17
|
+
- **缓存中断恢复(B09)**:每条 JSONL 记录以独立前后换行包围,单次 O_APPEND 追加,不读改尾部或留下恢复锁。损坏尾行不再吞掉下一条成功记录;短写明确失败,元数据不能覆盖保留字段。
|
|
18
|
+
- **发布身份(B10)**:checkout 前验证完整 tag ref 和远端对象,checkout 后核对 commit/包版本;按 ref、SHA、模式、nonce 和 attempt 精确认领本次 run,不再认领第一个新 ID。版本文件与发布表格先在内存预演,避免坏表格先留下已 bump 的版本;发布时同步更新锁文件的根版本。
|
|
19
|
+
|
|
20
|
+
### Changed
|
|
21
|
+
|
|
22
|
+
- **原文核验(O01)**:格式、必要章节和正文长度由程序检查;核验代理读取本次原文,返回全部分块摘录、关键引文原句及行号。缺证据、错行号、引文不匹配或已知精读缺口均不能通过,单独 `pass: true` 无效。新增 `qualityChecks`、`qualityEvidence` 返回字段。
|
|
23
|
+
- 中文/日文字符按字、其他文字按词计数;排除元数据、代码块、HTML 注释和链接地址。frontmatter 按提供的标量/列表模板校验,关键逐字引用集中在第 4 节。
|
|
24
|
+
- 新增一个锁定的生产依赖 Acorn,用于构建期 JavaScript 解析;CI 与发布工作流通过 `npm ci --ignore-scripts` 安装锁定依赖。
|
|
25
|
+
|
|
26
|
+
### Compatibility
|
|
27
|
+
|
|
28
|
+
- 默认配置仍可使用。依赖宽松强转(如字符串布尔值、负数重试)或超出范围的参数现在会报错;自定义解析器须使用纯提示词构造逻辑和静态本地模块依赖。
|
|
29
|
+
- 原文核验比仅检查成稿多读取资料,质量状态会更严格;引用真实性仍取决于核验代理实际读取原文,返回的摘录和行号用于回查。
|
|
30
|
+
|
|
6
31
|
## [0.3.18] — 2026-10-23
|
|
7
32
|
|
|
8
33
|
修复自动解析器分流、并发临时文件串稿和分块上限导致的全文遗漏。
|
package/README.en.md
CHANGED
|
@@ -74,6 +74,10 @@ Three waves, normally N+3 sub-agent calls (fetch, N chunk reads, draft, and qual
|
|
|
74
74
|
|
|
75
75
|
Chunk plans are sorted, duplicate ranges trimmed, and gaps filled. Plans above `maxChunks` merge adjacent ranges instead of discarding the document tail. The returned `coverageComplete` indicates coverage of all declared lines based on a valid `totalLines`; without a line count, known ranges are retained, `qualityPassed` is set to `false`, and the coverage uncertainty is reported.
|
|
76
76
|
|
|
77
|
+
Quality checks combine code and source review. Code validates the supplied scalar/list frontmatter template, required sections, and length (CJK characters plus other words, excluding metadata, code blocks, HTML comments, and link URLs). A separate reviewer reads the current source file and returns per-chunk excerpts and line-bound evidence for verbatim quotations in section 4. A bare `pass: true` cannot override these checks. `qualityChecks` exposes the gates, and `qualityEvidence` exposes the source path and evidence. Source authenticity still depends on the reviewer's actual source reading; uncertainty fails the gate. Empty, rejected, or structurally degraded repairs retain the preceding usable note.
|
|
78
|
+
|
|
79
|
+
Direct workflow options are strict: `minWords` is an integer in 1–100000, `maxChunks` is a positive integer (values above 12 are capped), and `maxRetries` is an integer in 0–3. Invalid values and unknown keys fail before work begins. Plugin profile configuration instead warns per invalid field and uses its default, preserving host startup.
|
|
80
|
+
|
|
77
81
|
```
|
|
78
82
|
parsers/ Parsers for each input type, dispatched by type
|
|
79
83
|
book.js Books: PDF/EPUB/MOBI text extraction, chapter chunking
|
|
@@ -95,6 +99,8 @@ workflow.js the workflow meta and assembly logic (parser inlining, schema i
|
|
|
95
99
|
|
|
96
100
|
To change how an input type is handled, drop a parser with the same interface into `custom-parsers/`; a type with the same name overrides the built-in implementation, and you can also add a brand-new type name (e.g. `podcast`) - every registered parser is inlined into the workflow script, and both the `type` whitelist and the schema's `kind` enum follow the registry. The interface has only three fields: `name`, `types`, `buildPrompt(input, opts)`.
|
|
97
101
|
|
|
102
|
+
File parsers bundle their CommonJS modules and literal relative `.js`, `.cjs`, and `.json` dependencies, preserving module constants, helpers, method shorthand, and cycles. Acorn parses JavaScript during script construction and is installed automatically by npm; it does not run in the workflow sandbox. Node built-ins, external packages, and dynamic `require`/`import` are unsupported in sandbox parsers and fail at build time. Programmatically registered functions must be self-contained; use a file parser for module closures.
|
|
103
|
+
|
|
98
104
|
---
|
|
99
105
|
|
|
100
106
|
## Platform support
|
|
@@ -119,7 +125,7 @@ Further notes:
|
|
|
119
125
|
deep-read-summarize is a DSH plugin installable via npm or locally:
|
|
120
126
|
|
|
121
127
|
```bash
|
|
122
|
-
# npm install (
|
|
128
|
+
# npm install (includes Acorn for parser construction)
|
|
123
129
|
npm install deep-read-summarize
|
|
124
130
|
|
|
125
131
|
# or local install (dsh profile directory)
|
|
@@ -146,7 +152,7 @@ npm install deep-read-summarize
|
|
|
146
152
|
# Option B: develop from source
|
|
147
153
|
git clone https://github.com/<your-org>/deep-read-summarize.git
|
|
148
154
|
cd deep-read-summarize
|
|
149
|
-
npm
|
|
155
|
+
npm ci # install locked dependencies
|
|
150
156
|
```
|
|
151
157
|
|
|
152
158
|
### 2. Verify the environment (offline, no API key needed)
|
|
@@ -155,7 +161,7 @@ npm install # no third-party dependencies, just initialization
|
|
|
155
161
|
npm test # 35 fixture tests, all run offline
|
|
156
162
|
```
|
|
157
163
|
|
|
158
|
-
Seeing `TOTAL:
|
|
164
|
+
Seeing `TOTAL: 35 passed, 0 failed` means the environment is ready.
|
|
159
165
|
|
|
160
166
|
### 3. Feed a demo file with one command
|
|
161
167
|
|
|
@@ -222,7 +228,7 @@ Notes are written to `options.outputDir` (default `./output`, can point at your
|
|
|
222
228
|
|
|
223
229
|
### Frontmatter profiles (`options.frontmatter`)
|
|
224
230
|
|
|
225
|
-
The default is `obsidian`,
|
|
231
|
+
The default is `obsidian`, with the same keys as earlier versions and safely quoted scalar/list values. Profiles exist because `type`, `aliases` and `url` are **reserved keys in Hugo** with different meanings -- dropping an Obsidian note into Hugo changes how the page renders, rather than adding a few harmless fields.
|
|
226
232
|
|
|
227
233
|
| Key | `obsidian` (default) | `generic` | `minimal` |
|
|
228
234
|
|---|---|---|---|
|
|
@@ -313,7 +319,7 @@ Two classes:
|
|
|
313
319
|
|
|
314
320
|
## Cost
|
|
315
321
|
|
|
316
|
-
|
|
322
|
+
Token use depends on document length, chunk count, and output size. Source review reads the document again, and each repair round adds a repair and a review call. `fastMode` reduces output sections; reducing `maxChunks` limits agent count while preserving the complete source.
|
|
317
323
|
|
|
318
324
|
---
|
|
319
325
|
|
package/README.md
CHANGED
|
@@ -74,6 +74,10 @@ Markdown 笔记
|
|
|
74
74
|
|
|
75
75
|
分块计划会排序、裁掉重复区间并补齐缺口,超过 `maxChunks` 时合并相邻块以保留全文。返回的 `coverageComplete` 表示是否依据有效的 `totalLines` 覆盖全部行号;缺少总行数时保留已知范围,同时将 `qualityPassed` 置为 `false` 并说明覆盖无法确认。
|
|
76
76
|
|
|
77
|
+
质量检查分两部分:程序按提供的标量/列表 frontmatter 模板检查字段、必要章节和篇幅(汉字按字、其他文字按词,排除 frontmatter、代码块、HTML 注释及链接地址);独立核验代理读取本次原文文件,逐块检查覆盖,并为第 4 节的逐字引用返回原句和行号。单独的 `pass: true` 无法通过检查。`qualityChecks` 返回检查项,`qualityEvidence` 返回原文路径、分块摘录和引用证据;原文真实性判断仍由读取原文的代理完成,无法核实就报告未通过。修订为空、调用失败或结构退化时保留上一份有效成稿。
|
|
78
|
+
|
|
79
|
+
直接调用 workflow 的 options 必须类型正确:`minWords` 为 1–100000 的整数,`maxChunks` 为正整数(大于 12 仍夹到 12),`maxRetries` 为 0–3 的整数;非法值和未知键在启动时 FATAL。插件 profile 的 config 则逐项告警并回退默认值,避免阻止宿主启动。
|
|
80
|
+
|
|
77
81
|
```
|
|
78
82
|
parsers/ 各输入类型的解析器,按类型分发
|
|
79
83
|
book.js 书籍:PDF/EPUB/MOBI 文本提取、章节分块
|
|
@@ -94,6 +98,8 @@ workflow.js workflow 的 meta 与组装逻辑(解析器内联、schema
|
|
|
94
98
|
|
|
95
99
|
想换某个输入类型的处理方式,在 `custom-parsers/` 放一个同接口的解析器即可,同名类型会覆盖内置实现;用新类型名(如 `podcast`)也可以——注册表里的解析器会全部内联进 workflow 脚本,`type` 白名单与 schema 的 `kind` 枚举都跟着注册表走。接口只有三个字段:`name`、`types`、`buildPrompt(input, opts)`。
|
|
96
100
|
|
|
101
|
+
解析器的 CommonJS 模块会连同字面量相对路径的 `.js` / `.cjs` / `.json` 依赖一起打包,保留模块级常量、helper、方法简写和循环依赖。构建时使用 Acorn 解析 JavaScript;它由 npm 自动安装,不进入 workflow 沙箱。沙箱中的解析器只负责生成提示词,不支持 Node 内置模块、外部包或动态 `require` / `import`;这些依赖会在构建时明确报错。程序化注册的函数须自包含,模块闭包请使用文件解析器。
|
|
102
|
+
|
|
97
103
|
---
|
|
98
104
|
|
|
99
105
|
## 支持平台
|
|
@@ -118,7 +124,7 @@ workflow.js workflow 的 meta 与组装逻辑(解析器内联、schema
|
|
|
118
124
|
deep-read-summarize 是 DSH 插件,可通过 npm 或本地安装:
|
|
119
125
|
|
|
120
126
|
```bash
|
|
121
|
-
# npm
|
|
127
|
+
# npm 安装(自动安装用于解析器构建的 Acorn)
|
|
122
128
|
npm install deep-read-summarize
|
|
123
129
|
|
|
124
130
|
# 或本地安装(dsh profile 目录)
|
|
@@ -145,7 +151,7 @@ npm install deep-read-summarize
|
|
|
145
151
|
# 方式 B:从源码开发
|
|
146
152
|
git clone https://github.com/<your-org>/deep-read-summarize.git
|
|
147
153
|
cd deep-read-summarize
|
|
148
|
-
npm
|
|
154
|
+
npm ci # 安装锁定依赖
|
|
149
155
|
```
|
|
150
156
|
|
|
151
157
|
### 2. 验证环境(离线,不需要任何 API key)
|
|
@@ -154,7 +160,7 @@ npm install # 无第三方依赖,仅初始化
|
|
|
154
160
|
npm test # 35 项 fixture 测试,全部离线跑通
|
|
155
161
|
```
|
|
156
162
|
|
|
157
|
-
看到 `TOTAL:
|
|
163
|
+
看到 `TOTAL: 35 passed, 0 failed` 即环境就绪。
|
|
158
164
|
|
|
159
165
|
### 3. 一行命令喂入 demo 文件
|
|
160
166
|
|
|
@@ -221,7 +227,7 @@ parsers: book, paper, video, web
|
|
|
221
227
|
|
|
222
228
|
### frontmatter 三档(`options.frontmatter`)
|
|
223
229
|
|
|
224
|
-
默认 `obsidian
|
|
230
|
+
默认 `obsidian`,键集与历史版本一致,标量和列表值使用安全引号。之所以要分档:`type`、`aliases`、`url` 在 Hugo 里是**保留键**,语义和 Obsidian 完全不同——直接丢进 Hugo 会改变渲染行为,而不是多几个无害字段。
|
|
225
231
|
|
|
226
232
|
| 键 | `obsidian`(默认) | `generic` | `minimal` |
|
|
227
233
|
|---|---|---|---|
|
|
@@ -312,7 +318,7 @@ parsers: book, paper, video, web
|
|
|
312
318
|
|
|
313
319
|
## 成本
|
|
314
320
|
|
|
315
|
-
|
|
321
|
+
Token 用量取决于原文长度、分块数和成稿篇幅。原文核验会再次读取资料,每轮修订还会增加一次修订和一次核验调用。`fastMode` 减少输出章节;调低 `maxChunks` 限制代理数,同时保留全文覆盖。
|
|
316
322
|
|
|
317
323
|
---
|
|
318
324
|
|
package/docs/RELEASE.md
CHANGED
|
@@ -138,7 +138,8 @@ npm view deep-read-summarize --prefer-online # 验证线
|
|
|
138
138
|
| v0.3.15 | npm 已发布 | 工程卫生:锁定 LF、脚本正文独立成文、lint 扩检查、包瘦身、提示词改「标准 Markdown」 |
|
|
139
139
|
| v0.3.16 | npm 已发布 | 新增 `options.frontmatter` 三档(obsidian / generic / minimal),解决 Hugo 保留键冲突;默认档不变 |
|
|
140
140
|
| v0.3.17 | npm 已发布 | 修 `resolve()` 原型链回退与质量清单 required;新增发布脚本 `scripts/release.js` |
|
|
141
|
-
|
|
|
141
|
+
| v0.3.18 | npm 已发布 | 修复自动解析器分流、并发临时文件串稿和分块上限导致的全文遗漏 |
|
|
142
|
+
| **v0.3.19** | **npm 已发布(当前)** | 修复 B04–B10,并让质量检查结合程序门禁和可回查的原文证据(O01)。 |
|
|
142
143
|
| v1.0.0 | 计划 | API 稳定后发布 |
|
|
143
144
|
|
|
144
145
|
> 早期迭代用 0.x 表达不稳定;API 稳定后再发 1.0.0。
|
package/index.js
CHANGED
|
@@ -20,69 +20,19 @@ const path = require('node:path');
|
|
|
20
20
|
const workflow = require('./workflow');
|
|
21
21
|
const parsers = require('./parsers');
|
|
22
22
|
const schemas = require('./schemas');
|
|
23
|
+
const { DEFAULT_OPTIONS, normalizeWorkflowOptions } = require('./lib/options');
|
|
23
24
|
|
|
24
25
|
module.exports.name = 'deep-read-summarize';
|
|
25
26
|
module.exports.inject = ['skills'];
|
|
26
27
|
|
|
27
|
-
/** workflow 的 options 默认值(与 workflow 脚本内的默认值保持一致)。 */
|
|
28
|
-
const DEFAULT_OPTIONS = {
|
|
29
|
-
minWords: 2500,
|
|
30
|
-
fastMode: false,
|
|
31
|
-
maxChunks: 4,
|
|
32
|
-
transcribe: true,
|
|
33
|
-
requireCitations: true,
|
|
34
|
-
includeTimestamps: false,
|
|
35
|
-
outputDir: './output',
|
|
36
|
-
tempDir: './.tmp',
|
|
37
|
-
whisperModel: 'small',
|
|
38
|
-
language: 'zh',
|
|
39
|
-
device: 'cpu',
|
|
40
|
-
maxRetries: 0,
|
|
41
|
-
frontmatter: 'obsidian'
|
|
42
|
-
};
|
|
43
|
-
|
|
44
|
-
const OPTION_TYPES = {
|
|
45
|
-
minWords: 'number',
|
|
46
|
-
fastMode: 'boolean',
|
|
47
|
-
maxChunks: 'number',
|
|
48
|
-
transcribe: 'boolean',
|
|
49
|
-
requireCitations: 'boolean',
|
|
50
|
-
includeTimestamps: 'boolean',
|
|
51
|
-
outputDir: 'string',
|
|
52
|
-
tempDir: 'string',
|
|
53
|
-
whisperModel: 'string',
|
|
54
|
-
language: 'string',
|
|
55
|
-
device: 'string',
|
|
56
|
-
maxRetries: 'number',
|
|
57
|
-
frontmatter: 'string'
|
|
58
|
-
};
|
|
59
|
-
|
|
60
28
|
/**
|
|
61
29
|
* 归一化插件 config:只认白名单里的键,类型不符就告警并忽略(不抛错,
|
|
62
30
|
* 免得一个拼错的配置项把整个 profile 拖死)。
|
|
63
31
|
* 支持 `config: { outputDir: ... }` 与 `config: { options: { outputDir: ... } }` 两种写法。
|
|
64
32
|
*/
|
|
65
33
|
function normalizeOptions(config) {
|
|
66
|
-
const
|
|
67
|
-
|
|
68
|
-
for (const key of Object.keys(DEFAULT_OPTIONS)) {
|
|
69
|
-
const v = raw[key];
|
|
70
|
-
if (v === undefined) continue;
|
|
71
|
-
if (typeof v !== OPTION_TYPES[key]) {
|
|
72
|
-
console.warn('[deep-read-summarize] ignoring config.' + key + ' — expected ' + OPTION_TYPES[key] + ', got ' + typeof v);
|
|
73
|
-
continue;
|
|
74
|
-
}
|
|
75
|
-
opts[key] = v;
|
|
76
|
-
}
|
|
77
|
-
// 未知键此前是完全无声的:循环只遍历白名单,写错的键名既不生效也不告警。
|
|
78
|
-
const unknown = Object.keys(raw).filter(function (k) {
|
|
79
|
-
return !Object.prototype.hasOwnProperty.call(DEFAULT_OPTIONS, k);
|
|
80
|
-
});
|
|
81
|
-
if (unknown.length) {
|
|
82
|
-
console.warn('[deep-read-summarize] ignoring unknown config key(s): ' + unknown.join(', '));
|
|
83
|
-
console.warn('[deep-read-summarize] allowed keys: ' + Object.keys(DEFAULT_OPTIONS).join(', '));
|
|
84
|
-
}
|
|
85
|
-
return opts;
|
|
34
|
+
const raw = config && Object.prototype.hasOwnProperty.call(config, 'options') ? config.options : config;
|
|
35
|
+
return normalizeWorkflowOptions(raw, { mode: 'config', warn: function (message) { console.warn(message); } });
|
|
86
36
|
}
|
|
87
37
|
|
|
88
38
|
/** 组装技能内容:SKILL.md + workflow meta/script/args 示例(script 为自包含版本)。 */
|
|
@@ -145,3 +95,4 @@ module.exports.schemas = schemas;
|
|
|
145
95
|
module.exports.buildSkillContent = buildSkillContent;
|
|
146
96
|
module.exports.DEFAULT_OPTIONS = DEFAULT_OPTIONS;
|
|
147
97
|
module.exports.normalizeOptions = normalizeOptions;
|
|
98
|
+
module.exports.normalizeWorkflowOptions = normalizeWorkflowOptions;
|
package/lib/cache.js
CHANGED
|
@@ -84,14 +84,34 @@ function hasProcessed(cacheFile, key, opts) {
|
|
|
84
84
|
return true;
|
|
85
85
|
}
|
|
86
86
|
|
|
87
|
-
/**
|
|
87
|
+
/**
|
|
88
|
+
* 标记已处理:每条记录前后带换行,以一次 O_APPEND write 写入。
|
|
89
|
+
* 前导换行隔开无结尾 LF 的合法旧记录和崩溃留下的残行;不读取/改写文件尾,
|
|
90
|
+
* 并发调用无需争抢恢复锁,也不会因进程崩溃留下永久残锁。
|
|
91
|
+
* 依赖本地文件系统的 append 写入语义,不提供网络文件系统事务保证。
|
|
92
|
+
* 写失败或短写仅告警并返回 false;残行由下一条记录的前导换行隔离。
|
|
93
|
+
*/
|
|
88
94
|
function markProcessed(cacheFile, key, outputPath, meta) {
|
|
89
95
|
if (!key) return false;
|
|
90
96
|
try {
|
|
91
97
|
const dir = path.dirname(cacheFile);
|
|
92
98
|
if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
|
|
93
|
-
|
|
94
|
-
|
|
99
|
+
// 元数据只扩展记录,不能伪造指纹、产物路径、处理时间或 JSON 序列化行为。
|
|
100
|
+
const rec = Object.create(null);
|
|
101
|
+
for (const name of Object.keys(meta || {})) {
|
|
102
|
+
if (name !== "key" && name !== "outputPath" && name !== "ts" && name !== "toJSON") rec[name] = meta[name];
|
|
103
|
+
}
|
|
104
|
+
rec.key = key;
|
|
105
|
+
rec.outputPath = outputPath;
|
|
106
|
+
rec.ts = new Date().toISOString();
|
|
107
|
+
const record = Buffer.from("\n" + JSON.stringify(rec) + "\n", "utf8");
|
|
108
|
+
const fd = fs.openSync(cacheFile, "a", 0o600);
|
|
109
|
+
try {
|
|
110
|
+
const written = fs.writeSync(fd, record, 0, record.length, null);
|
|
111
|
+
if (written !== record.length) throw new Error("short append: " + written + "/" + record.length + " bytes");
|
|
112
|
+
} finally {
|
|
113
|
+
fs.closeSync(fd);
|
|
114
|
+
}
|
|
95
115
|
return true;
|
|
96
116
|
} catch (e) {
|
|
97
117
|
console.warn("[cache] markProcessed failed:", e.message);
|
package/lib/options.js
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
|
|
3
|
+
/** Kept self-contained: the exact same validator is embedded in the workflow. */
|
|
4
|
+
function normalizeWorkflowOptions(raw, behavior) {
|
|
5
|
+
const defaults = {
|
|
6
|
+
minWords: 2500, fastMode: false, maxChunks: 4, transcribe: true,
|
|
7
|
+
requireCitations: true, includeTimestamps: false, outputDir: './output',
|
|
8
|
+
tempDir: './.tmp', whisperModel: 'small', language: 'zh', device: 'cpu',
|
|
9
|
+
maxRetries: 0, frontmatter: 'obsidian'
|
|
10
|
+
};
|
|
11
|
+
const settings = behavior || {};
|
|
12
|
+
const tolerant = settings.mode === 'config';
|
|
13
|
+
const warn = typeof settings.warn === 'function' ? settings.warn : function () {};
|
|
14
|
+
const result = Object.assign({}, defaults);
|
|
15
|
+
function invalid(key, reason) {
|
|
16
|
+
if (!tolerant) throw new Error('[FATAL] options' + (key ? '.' + key : '') + ' 非法:' + reason);
|
|
17
|
+
warn('[deep-read-summarize] ignoring config' + (key ? '.' + key : '') + ' — ' + reason);
|
|
18
|
+
}
|
|
19
|
+
if (raw === undefined) return result;
|
|
20
|
+
if (raw === null || typeof raw !== 'object' || Array.isArray(raw)) {
|
|
21
|
+
invalid('', 'expected an options object');
|
|
22
|
+
return result;
|
|
23
|
+
}
|
|
24
|
+
const keys = Object.keys(defaults);
|
|
25
|
+
const unknown = Object.keys(raw).filter(function (key) { return keys.indexOf(key) < 0; });
|
|
26
|
+
if (unknown.length) {
|
|
27
|
+
if (!tolerant) throw new Error('[FATAL] options 未知配置键:' + unknown.join(', ') + '(allowed keys: ' + keys.join(', ') + ')');
|
|
28
|
+
warn('[deep-read-summarize] ignoring unknown config key(s): ' + unknown.join(', '));
|
|
29
|
+
warn('[deep-read-summarize] allowed keys: ' + keys.join(', '));
|
|
30
|
+
}
|
|
31
|
+
const models = [
|
|
32
|
+
'tiny', 'tiny.en', 'base', 'base.en', 'small', 'small.en', 'medium', 'medium.en',
|
|
33
|
+
'large', 'large-v1', 'large-v2', 'large-v3', 'large-v3-turbo', 'turbo',
|
|
34
|
+
'distil-large-v2', 'distil-medium.en', 'distil-small.en', 'distil-large-v3'
|
|
35
|
+
];
|
|
36
|
+
for (const key of keys) {
|
|
37
|
+
if (!Object.prototype.hasOwnProperty.call(raw, key) || raw[key] === undefined) continue;
|
|
38
|
+
let value = raw[key];
|
|
39
|
+
let reason = '';
|
|
40
|
+
if (key === 'minWords' || key === 'maxChunks' || key === 'maxRetries') {
|
|
41
|
+
const minimum = key === 'maxRetries' ? 0 : 1;
|
|
42
|
+
const maximum = key === 'minWords' ? 100000 : (key === 'maxRetries' ? 3 : Infinity);
|
|
43
|
+
if (typeof value !== 'number' || !Number.isFinite(value) || !Number.isInteger(value) || value < minimum || value > maximum) {
|
|
44
|
+
reason = 'expected a finite integer in ' + minimum + '..' + (maximum === Infinity ? '12 (larger values are capped)' : maximum);
|
|
45
|
+
} else if (key === 'maxChunks' && value > 12) {
|
|
46
|
+
if (tolerant) warn('[deep-read-summarize] config.maxChunks capped at 12 (received ' + value + ')');
|
|
47
|
+
value = 12;
|
|
48
|
+
}
|
|
49
|
+
} else if (typeof defaults[key] === 'boolean') {
|
|
50
|
+
if (typeof value !== 'boolean') reason = 'expected boolean';
|
|
51
|
+
} else if (typeof value !== 'string' || !value.trim() || /[\u0000\r\n]/.test(value)) {
|
|
52
|
+
reason = 'expected a non-empty string without NUL or line breaks';
|
|
53
|
+
} else if (key !== 'outputDir' && key !== 'tempDir') {
|
|
54
|
+
value = value.trim().toLowerCase();
|
|
55
|
+
if (key === 'frontmatter' && ['obsidian', 'generic', 'minimal'].indexOf(value) < 0) reason = 'expected obsidian|generic|minimal';
|
|
56
|
+
if (key === 'device' && ['cpu', 'auto', 'cuda'].indexOf(value) < 0) reason = 'expected cpu|auto|cuda';
|
|
57
|
+
if (key === 'whisperModel' && models.indexOf(value) < 0) reason = 'unsupported Whisper model';
|
|
58
|
+
if (key === 'language' && !/^[a-z]{2,3}$/.test(value)) reason = 'expected a two- or three-letter language code';
|
|
59
|
+
}
|
|
60
|
+
if (reason) invalid(key, reason);
|
|
61
|
+
else result[key] = value;
|
|
62
|
+
}
|
|
63
|
+
return result;
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
module.exports = { normalizeWorkflowOptions, DEFAULT_OPTIONS: normalizeWorkflowOptions() };
|
|
@@ -0,0 +1,197 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
|
|
3
|
+
const fs = require('node:fs');
|
|
4
|
+
const path = require('node:path');
|
|
5
|
+
const acorn = require('acorn');
|
|
6
|
+
|
|
7
|
+
// Acorn handles the regexp/division, template and modern syntax boundaries.
|
|
8
|
+
// The two-pass scope walk distinguishes free host APIs from local bindings and
|
|
9
|
+
// property names. Only literal relative CommonJS imports are bundled.
|
|
10
|
+
function analyze(source, filename) {
|
|
11
|
+
const comments = [];
|
|
12
|
+
let tree;
|
|
13
|
+
try { tree = acorn.parse(source, { ecmaVersion: 'latest', sourceType: 'commonjs', onComment: comments }); }
|
|
14
|
+
catch (error) { throw new Error('[FATAL] cannot parse parser module ' + filename + ': ' + error.message); }
|
|
15
|
+
const nodes = [];
|
|
16
|
+
const scopes = new WeakMap();
|
|
17
|
+
const bindings = new WeakSet();
|
|
18
|
+
const identifiers = new Set();
|
|
19
|
+
function scope(parent, kind) { return { parent, kind, names: new Set() }; }
|
|
20
|
+
function children(node, visit) {
|
|
21
|
+
for (const key of Object.keys(node)) {
|
|
22
|
+
const value = node[key];
|
|
23
|
+
if (Array.isArray(value)) {
|
|
24
|
+
for (const child of value) if (child && typeof child.type === 'string') visit(child, node, key);
|
|
25
|
+
} else if (value && typeof value.type === 'string') visit(value, node, key);
|
|
26
|
+
}
|
|
27
|
+
}
|
|
28
|
+
function bind(pattern, target) {
|
|
29
|
+
if (!pattern) return;
|
|
30
|
+
if (pattern.type === 'Identifier') { target.names.add(pattern.name); bindings.add(pattern); }
|
|
31
|
+
else if (pattern.type === 'RestElement') bind(pattern.argument, target);
|
|
32
|
+
else if (pattern.type === 'AssignmentPattern') bind(pattern.left, target);
|
|
33
|
+
else if (pattern.type === 'ArrayPattern') pattern.elements.forEach(item => bind(item, target));
|
|
34
|
+
else if (pattern.type === 'ObjectPattern') pattern.properties.forEach(item => bind(item.type === 'RestElement' ? item.argument : item.value, target));
|
|
35
|
+
}
|
|
36
|
+
function collect(node, active, parent, key) {
|
|
37
|
+
if (/^Function/.test(node.type) || node.type === 'ArrowFunctionExpression') {
|
|
38
|
+
if (node.type === 'FunctionDeclaration') bind(node.id, active);
|
|
39
|
+
const parameters = scope(active, 'parameters');
|
|
40
|
+
bind(node.id, parameters);
|
|
41
|
+
node.params.forEach(param => bind(param, parameters));
|
|
42
|
+
scopes.set(node, parameters);
|
|
43
|
+
nodes.push({ node, parent, key });
|
|
44
|
+
node.params.forEach(param => collect(param, parameters, node, 'params'));
|
|
45
|
+
if (node.id) collect(node.id, parameters, node, 'id');
|
|
46
|
+
collect(node.body, scope(parameters, 'function'), node, 'body');
|
|
47
|
+
return;
|
|
48
|
+
}
|
|
49
|
+
if (node.type === 'ClassDeclaration') bind(node.id, active);
|
|
50
|
+
if (node.type === 'ClassDeclaration' || node.type === 'ClassExpression') {
|
|
51
|
+
active = scope(active, 'block'); bind(node.id, active);
|
|
52
|
+
} else if (['BlockStatement', 'CatchClause', 'ForStatement', 'ForInStatement', 'ForOfStatement', 'SwitchStatement'].indexOf(node.type) >= 0) {
|
|
53
|
+
active = scope(active, 'block');
|
|
54
|
+
} else if (node.type === 'StaticBlock') active = scope(active, 'function');
|
|
55
|
+
if (node.type === 'CatchClause') bind(node.param, active);
|
|
56
|
+
if (node.type === 'VariableDeclaration') {
|
|
57
|
+
let target = active;
|
|
58
|
+
if (node.kind === 'var') while (target.parent && target.kind !== 'function' && target.kind !== 'program') target = target.parent;
|
|
59
|
+
node.declarations.forEach(declaration => bind(declaration.id, target));
|
|
60
|
+
}
|
|
61
|
+
scopes.set(node, active);
|
|
62
|
+
if (node.type === 'Identifier') identifiers.add(node.name);
|
|
63
|
+
nodes.push({ node, parent, key });
|
|
64
|
+
children(node, (child, owner, field) => collect(child, active, owner, field));
|
|
65
|
+
}
|
|
66
|
+
collect(tree, scope(null, 'program'), null, '');
|
|
67
|
+
function bound(node) {
|
|
68
|
+
for (let current = scopes.get(node); current; current = current.parent) if (current.names.has(node.name)) return true;
|
|
69
|
+
return false;
|
|
70
|
+
}
|
|
71
|
+
function reference(item) {
|
|
72
|
+
const { node, parent, key } = item;
|
|
73
|
+
if (node.type !== 'Identifier' || bindings.has(node)) return false;
|
|
74
|
+
if (!parent) return true;
|
|
75
|
+
if ((parent.type === 'MemberExpression' && key === 'property' && !parent.computed) ||
|
|
76
|
+
(['Property', 'MethodDefinition', 'PropertyDefinition'].indexOf(parent.type) >= 0 && key === 'key' && !parent.computed) ||
|
|
77
|
+
(['LabeledStatement', 'BreakStatement', 'ContinueStatement'].indexOf(parent.type) >= 0 && key === 'label')) return false;
|
|
78
|
+
return true;
|
|
79
|
+
}
|
|
80
|
+
return { nodes, comments, identifiers, bound, reference };
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
function standaloneParserSource(parser, name) {
|
|
84
|
+
let fn = Function.prototype.toString.call(parser.buildPrompt);
|
|
85
|
+
// Method shorthand is valid inside an object, but not after a property colon.
|
|
86
|
+
if (!/^(?:async\s+)?function\b/.test(fn) && /^(?:async\s+)?[\w$]+\s*\(/.test(fn)) {
|
|
87
|
+
fn = fn.replace(/^(async\s+)?/, function (_, asyncPrefix) { return (asyncPrefix || '') + 'function '; });
|
|
88
|
+
}
|
|
89
|
+
try { new Function('return (' + fn + ');'); }
|
|
90
|
+
catch (error) { throw new Error('[FATAL] cannot serialize parser ' + name + ': ' + error.message); }
|
|
91
|
+
const data = Object.create(null);
|
|
92
|
+
for (const key of Object.keys(parser)) {
|
|
93
|
+
if (typeof parser[key] !== 'function' && parser[key] !== undefined) data[key] = parser[key];
|
|
94
|
+
}
|
|
95
|
+
const fields = JSON.stringify(data).slice(1, -1);
|
|
96
|
+
return '{ buildPrompt: ' + fn + (fields ? ', ' + fields : '') + ' }';
|
|
97
|
+
}
|
|
98
|
+
|
|
99
|
+
/** Bundle local CommonJS modules while retaining each module's lexical scope. */
|
|
100
|
+
function buildParserBundle(registry) {
|
|
101
|
+
const modules = [];
|
|
102
|
+
const ids = new Map();
|
|
103
|
+
function addModule(filename) {
|
|
104
|
+
const resolved = fs.realpathSync(filename);
|
|
105
|
+
if (ids.has(resolved)) return ids.get(resolved);
|
|
106
|
+
const id = 'm' + modules.length;
|
|
107
|
+
ids.set(resolved, id);
|
|
108
|
+
const entry = { id, code: '', importName: '__drsLocalImport' };
|
|
109
|
+
modules.push(entry); // reserve before recursion, so CommonJS cycles work
|
|
110
|
+
const ext = path.extname(resolved).toLowerCase();
|
|
111
|
+
if (ext === '.json') {
|
|
112
|
+
const json = JSON.parse(fs.readFileSync(resolved, 'utf8').replace(/^\uFEFF/, ''));
|
|
113
|
+
entry.code = 'module.exports = JSON.parse(' + JSON.stringify(JSON.stringify(json)) + ');';
|
|
114
|
+
return id;
|
|
115
|
+
}
|
|
116
|
+
if (ext !== '.js' && ext !== '.cjs') throw new Error('[FATAL] parser module must be local CommonJS (.js/.cjs/.json): ' + resolved);
|
|
117
|
+
const source = fs.readFileSync(resolved, 'utf8').replace(/^\uFEFF/, '').replace(/^#![^\n]*/, '');
|
|
118
|
+
const scanned = analyze(source, resolved);
|
|
119
|
+
while (scanned.identifiers.has(entry.importName)) entry.importName += '_';
|
|
120
|
+
const edits = scanned.comments.map(function (comment) {
|
|
121
|
+
return { start: comment.start, end: comment.end, text: source.slice(comment.start, comment.end).replace(/[^\n]/g, ' ') };
|
|
122
|
+
});
|
|
123
|
+
const unavailable = ['process', 'Buffer', '__dirname', '__filename', 'global'];
|
|
124
|
+
for (const item of scanned.nodes) {
|
|
125
|
+
const current = item.node;
|
|
126
|
+
if (current.type === 'ImportExpression' || current.type === 'WithStatement') {
|
|
127
|
+
throw new Error('[FATAL] ' + (current.type === 'ImportExpression' ? 'dynamic import is unavailable' : 'with scopes are unsupported') + ' in parser bundle: ' + resolved);
|
|
128
|
+
}
|
|
129
|
+
if (current.type === 'MemberExpression' && current.object.type === 'Identifier' && !scanned.bound(current.object)) {
|
|
130
|
+
const owner = current.object.name;
|
|
131
|
+
const member = !current.computed && current.property.type === 'Identifier' ? current.property.name :
|
|
132
|
+
(current.computed && current.property.type === 'Literal' ? current.property.value : undefined);
|
|
133
|
+
if ((owner === 'globalThis' && (member === undefined || unavailable.indexOf(member) >= 0 || member === 'require')) ||
|
|
134
|
+
(owner === 'module' && member === 'require')) {
|
|
135
|
+
throw new Error('[FATAL] unavailable workflow API ' + owner + '.' + (member === undefined ? '<dynamic>' : member) + ' in parser module: ' + resolved);
|
|
136
|
+
}
|
|
137
|
+
}
|
|
138
|
+
if (!scanned.reference(item) || scanned.bound(current)) continue;
|
|
139
|
+
if (unavailable.indexOf(current.name) >= 0) {
|
|
140
|
+
throw new Error('[FATAL] parser module uses unavailable workflow global ' + current.name + ': ' + resolved);
|
|
141
|
+
}
|
|
142
|
+
if (current.name === 'eval' || current.name === 'Function') {
|
|
143
|
+
throw new Error('[FATAL] dynamic code generation is unsupported in parser bundle: ' + resolved);
|
|
144
|
+
}
|
|
145
|
+
if (current.name === 'globalThis' && (!item.parent || item.parent.type !== 'MemberExpression' || item.key !== 'object')) {
|
|
146
|
+
throw new Error('[FATAL] aliases of host globals are unsupported in parser bundle: ' + resolved);
|
|
147
|
+
}
|
|
148
|
+
if (current.name !== 'require') continue;
|
|
149
|
+
const call = item.parent;
|
|
150
|
+
const request = call && call.type === 'CallExpression' && item.key === 'callee' && !call.optional && call.arguments.length === 1 ? call.arguments[0] : null;
|
|
151
|
+
if (!request || request.type !== 'Literal' || typeof request.value !== 'string') {
|
|
152
|
+
throw new Error('[FATAL] parser imports must use require with a literal local path: ' + resolved);
|
|
153
|
+
}
|
|
154
|
+
const specifier = request.value;
|
|
155
|
+
if (!/^\.\.?[\\/]/.test(specifier)) {
|
|
156
|
+
throw new Error('[FATAL] unsupported parser dependency ' + JSON.stringify(specifier) + ' in ' + resolved + '; workflow bundles accept relative local modules only');
|
|
157
|
+
}
|
|
158
|
+
let dependency;
|
|
159
|
+
try { dependency = require.resolve(path.resolve(path.dirname(resolved), specifier)); }
|
|
160
|
+
catch (error) { throw new Error('[FATAL] cannot resolve parser dependency ' + specifier + ' in ' + resolved + ': ' + error.message); }
|
|
161
|
+
edits.push({ start: call.start, end: call.end, text: entry.importName + '(' + JSON.stringify(addModule(dependency)) + ')' });
|
|
162
|
+
}
|
|
163
|
+
edits.sort(function (a, b) { return b.start - a.start; });
|
|
164
|
+
let rewritten = source;
|
|
165
|
+
for (const edit of edits) rewritten = rewritten.slice(0, edit.start) + edit.text + rewritten.slice(edit.end);
|
|
166
|
+
try { new Function('module', 'exports', entry.importName, rewritten); }
|
|
167
|
+
catch (error) { throw new Error('[FATAL] cannot compile parser module ' + resolved + ': ' + error.message); }
|
|
168
|
+
entry.code = rewritten;
|
|
169
|
+
return id;
|
|
170
|
+
}
|
|
171
|
+
const parserEntries = Object.keys(registry.parsers).map(function (name) {
|
|
172
|
+
const parser = registry.parsers[name];
|
|
173
|
+
if (!parser || typeof parser.buildPrompt !== 'function') throw new Error('[FATAL] parser ' + name + ' must expose buildPrompt(input, options)');
|
|
174
|
+
const filename = typeof registry.sourceFor === 'function' ? registry.sourceFor(parser) : undefined;
|
|
175
|
+
const source = filename
|
|
176
|
+
? '{ buildPrompt: function (input, options) { return __bundleLoad(' + JSON.stringify(addModule(filename)) + ').buildPrompt(input, options); } }'
|
|
177
|
+
: standaloneParserSource(parser, name);
|
|
178
|
+
return ' ' + (name === '__proto__' ? '[' + JSON.stringify(name) + ']' : JSON.stringify(name)) + ': ' + source;
|
|
179
|
+
});
|
|
180
|
+
return [
|
|
181
|
+
'const __bundleFactories = {',
|
|
182
|
+
modules.map(function (entry) { return JSON.stringify(entry.id) + ': function (module, exports, ' + entry.importName + ') {\n' + entry.code + '\n}'; }).join(',\n'),
|
|
183
|
+
'};',
|
|
184
|
+
'const __bundleCache = Object.create(null);',
|
|
185
|
+
'function __bundleLoad(id) {',
|
|
186
|
+
' if (__bundleCache[id]) return __bundleCache[id].exports;',
|
|
187
|
+
' const module = { exports: {} };',
|
|
188
|
+
' __bundleCache[id] = module;',
|
|
189
|
+
' try { __bundleFactories[id].call(module.exports, module, module.exports, __bundleLoad); }',
|
|
190
|
+
' catch (error) { delete __bundleCache[id]; throw error; }',
|
|
191
|
+
' return module.exports;',
|
|
192
|
+
'}',
|
|
193
|
+
'const __parsers = {', parserEntries.join(',\n'), '};'
|
|
194
|
+
].join('\n');
|
|
195
|
+
}
|
|
196
|
+
|
|
197
|
+
module.exports = { buildParserBundle };
|