deep-read-summarize 0.3.1 → 0.3.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +123 -109
- package/README.md +4 -4
- package/lib/cache.js +102 -87
- package/package.json +69 -69
- package/parsers/_prompt.js +44 -44
- package/tests/cache.test.js +116 -82
- package/workflow.js +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,110 +1,124 @@
|
|
|
1
|
-
# Changelog
|
|
2
|
-
|
|
3
|
-
All notable changes to this project will be documented in this file.
|
|
4
|
-
The format is based on [Keep a Changelog](https://keepachangelog.com/).
|
|
5
|
-
|
|
6
|
-
## [0.3.
|
|
7
|
-
|
|
8
|
-
### Fixed
|
|
9
|
-
-
|
|
10
|
-
-
|
|
11
|
-
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
-
|
|
18
|
-
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
-
|
|
24
|
-
-
|
|
25
|
-
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
-
|
|
38
|
-
-
|
|
39
|
-
-
|
|
40
|
-
|
|
41
|
-
### Fixed
|
|
42
|
-
-
|
|
43
|
-
|
|
44
|
-
## [0.
|
|
45
|
-
|
|
46
|
-
### Added
|
|
47
|
-
-
|
|
48
|
-
-
|
|
49
|
-
|
|
50
|
-
### Changed
|
|
51
|
-
- package.json
|
|
52
|
-
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
-
|
|
62
|
-
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
-
|
|
72
|
-
-
|
|
73
|
-
-
|
|
74
|
-
-
|
|
75
|
-
-
|
|
76
|
-
-
|
|
77
|
-
-
|
|
78
|
-
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
-
|
|
92
|
-
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/).
|
|
5
|
+
|
|
6
|
+
## [0.3.3] — 2026-08-25
|
|
7
|
+
|
|
8
|
+
### Fixed
|
|
9
|
+
- lib/cache.js:URL 归一化真正去掉尾部空 query(`doc` 与 `doc?` 现在共享指纹;此前 `url.search === "?"` 是死代码,`doc?` 会生成不同指纹导致幂等缓存失效)
|
|
10
|
+
- lib/cache.js:`fingerprint` 用 `??` 替代 `||`,空文件(size=0)不再与「无 meta」混淆
|
|
11
|
+
- workflow.js:`tempFile` 拼接先去掉 `tempDir` 尾部斜杠,避免 `./.tmp//input.txt` 双斜杠路径(与 parsers/_prompt.js 的 0.3.2 修复对齐)
|
|
12
|
+
- tests:+2 用例(URL 尾部 `?` 归一、size=0 指纹区分),25/25 通过;均由 DSH 代码审查发现
|
|
13
|
+
|
|
14
|
+
## [0.3.2] — 2026-08-23
|
|
15
|
+
|
|
16
|
+
### Fixed
|
|
17
|
+
- parsers/_prompt.js:目标文件路径优先使用调用方传入的 `tempFile`,回退路径统一正斜杠,修复 macOS/Linux 上硬编码 Windows 反斜杠导致的路径错误(issue #1,由 DSH 代码审查发现)
|
|
18
|
+
- lib/cache.js:`markProcessed` 写缓存失败不再抛错中断主流程(降级为告警并返回 false);`hasProcessed` 支持可选 TTL(`maxAgeMs`)过期判定(issue #2,由 DSH 代码审查发现)
|
|
19
|
+
|
|
20
|
+
## [0.3.1] — 2026-08-21
|
|
21
|
+
|
|
22
|
+
### Fixed
|
|
23
|
+
- 发布前检查:移除 README 中的本地 Obsidian 仓库绝对路径(改为参数化描述)
|
|
24
|
+
- 移除 docs/RELEASE.md 中的本机绝对路径(改为通用写法)
|
|
25
|
+
- README 同步版本号:tgz 0.2.0 → 0.3.0、测试 17 → 21 项
|
|
26
|
+
|
|
27
|
+
## [0.3.0] — 2026-08-21
|
|
28
|
+
|
|
29
|
+
### Added
|
|
30
|
+
- 幂等缓存 URL 规范化(normalizeInput):host 大小写 / 默认端口 / fragment / 尾部斜杠归一,同一 URL 不同写法共享指纹
|
|
31
|
+
- 安全扫描覆盖 .env.example 与 forward-slash 用户路径;导出 scanDir 供测试调用
|
|
32
|
+
- 测试 17 → 21:parser 模板回归、URL 规范化、安全扫描正/负样本
|
|
33
|
+
|
|
34
|
+
### Changed
|
|
35
|
+
- parsers 四解析器去重:抽取共享提示词模板 parsers/_prompt.js(buildPrompt 输出逐字不变,经新旧对比验证)
|
|
36
|
+
- lint 排除 .tmp*/dist/coverage 等临时目录
|
|
37
|
+
- CI 增加 Node 18/20/22 版本矩阵
|
|
38
|
+
- package.json 增加 publishConfig.access=public(显式公开发布)
|
|
39
|
+
- docs/RELEASE.md 增加 npm 发布流程与版本速览;README 开发节补充发布门禁说明
|
|
40
|
+
|
|
41
|
+
### Fixed
|
|
42
|
+
- security-check 对测试样本误报(样本改为拼接构造,源码不含完整密钥模式)
|
|
43
|
+
|
|
44
|
+
## [0.2.0] — 2026-08-21
|
|
45
|
+
|
|
46
|
+
### Added
|
|
47
|
+
- 首次发布到 npm registry(`npm install deep-read-summarize`),含此前未发布的 0.1.1 变更
|
|
48
|
+
- prepublishOnly 质量门禁:发布前自动跑 17 项测试 + lint + 安全检查
|
|
49
|
+
|
|
50
|
+
### Changed
|
|
51
|
+
- package.json version 对齐 git tag(0.1.0 → 0.2.0)
|
|
52
|
+
- keywords 移除 `deepseek-harness` 商标词(安全合规要求,与 v0.1.0 决策一致)
|
|
53
|
+
- README 补充 npm 安装方式
|
|
54
|
+
|
|
55
|
+
### Fixed
|
|
56
|
+
- git 历史整理:以本地完整开发历史为准 force push 覆盖 API 拼凑历史,补上缺失的 `.github/workflows/ci.yml`
|
|
57
|
+
|
|
58
|
+
## [0.1.1] — 2026-08-19
|
|
59
|
+
|
|
60
|
+
### Added
|
|
61
|
+
- DSH 插件化:cordis.patch.yml(bundle patch)、index.js 插件入口、skills/ 打包
|
|
62
|
+
- node:test 风格测试(tests/index.test.js,`npm run test:node`)
|
|
63
|
+
|
|
64
|
+
### Changed
|
|
65
|
+
- package.json:dsh.bundle 字段、exports、repository、keywords 完善
|
|
66
|
+
- 废弃 cordis.yml → 由 cordis.patch.yml 替代
|
|
67
|
+
|
|
68
|
+
## [0.1.0] — 2026-08-18
|
|
69
|
+
|
|
70
|
+
### Added
|
|
71
|
+
- 首个公开预发布版(pre-release)
|
|
72
|
+
- 四种输入解析器插件:书籍/论文/视频/网页
|
|
73
|
+
- 三波次 MapReduce 精读流水线
|
|
74
|
+
- JSON Schema 结构化输出约束
|
|
75
|
+
- 幂等缓存(lib/cache.js)
|
|
76
|
+
- 失败分级(FATAL vs 可降级)
|
|
77
|
+
- 质量校验循环
|
|
78
|
+
- CI(lint + 测试 + Gitleaks 密钥扫描)
|
|
79
|
+
- README/SECURITY/CONTRIBUTING/CHANGELOG 文档
|
|
80
|
+
|
|
81
|
+
## [3.0.0] — 2026-08-18
|
|
82
|
+
|
|
83
|
+
### Added
|
|
84
|
+
- Plugin parser architecture: `parsers/` registry with book/paper/video/web adapters
|
|
85
|
+
- JSON Schema structured output constraints (`schemas/index.js`)
|
|
86
|
+
- Failure discipline: FATAL (config) vs degradable (content) error classes
|
|
87
|
+
- Quality control loop with retry (`maxRetries`)
|
|
88
|
+
- Configurable citations (`requireCitations`) and video timestamps (`includeTimestamps`)
|
|
89
|
+
- Fixture-driven test suite (`npm test`, 10 tests)
|
|
90
|
+
- Quick validation script (`npm run validate`)
|
|
91
|
+
- CI workflow (lint + tests)
|
|
92
|
+
- cordis.yml plugin tree example
|
|
93
|
+
|
|
94
|
+
### Changed
|
|
95
|
+
- Workflow reduced to 3 serial waves (fetch/chunk → parallel read → merge+QC)
|
|
96
|
+
- Subagent count reduced to N+2~3
|
|
97
|
+
- Defaults changed to general-purpose: `requireCitations: true`, `includeTimestamps: false`
|
|
98
|
+
|
|
99
|
+
### Removed
|
|
100
|
+
- Standalone quality-check subagent (merged into wave 3 + optional lightweight retry)
|
|
101
|
+
|
|
102
|
+
## [2.1.0] — 2026-08-18
|
|
103
|
+
|
|
104
|
+
### Added
|
|
105
|
+
- One-click write-to-Obsidian from the draft agent
|
|
106
|
+
- fastMode (skip sections 5-7)
|
|
107
|
+
|
|
108
|
+
## [2.0.0] — 2026-08-18
|
|
109
|
+
|
|
110
|
+
### Changed
|
|
111
|
+
- Merged identify+fetch+chunk into one subagent (3 waves total)
|
|
112
|
+
- Removed standalone QC subagent
|
|
113
|
+
|
|
114
|
+
## [1.1.0] — 2026-08-18
|
|
115
|
+
|
|
116
|
+
### Fixed
|
|
117
|
+
- Long-content overflow: write-to-temp-file + line-range chunking
|
|
118
|
+
|
|
119
|
+
## [1.0.0] — 2026-08-18
|
|
120
|
+
|
|
121
|
+
### Added
|
|
122
|
+
- Initial deep-reading pipeline (5 serial waves)
|
|
123
|
+
- Template prompts for books/papers/videos/web
|
|
110
124
|
- Obsidian note template with YAML frontmatter
|
package/README.md
CHANGED
|
@@ -77,7 +77,7 @@ deep-read-summarize 是 DSH 插件,可通过 npm、dsh.so 生态或本地安
|
|
|
77
77
|
npm install deep-read-summarize
|
|
78
78
|
|
|
79
79
|
# 或本地安装(dsh profile 目录)
|
|
80
|
-
pnpm add ./deep-read-summarize-0.3.
|
|
80
|
+
pnpm add ./deep-read-summarize-0.3.3.tgz
|
|
81
81
|
# 然后在 dsh 配置的 dsh.profile.bundles 追加:
|
|
82
82
|
# - deep-read-summarize
|
|
83
83
|
# 重启 dsh web 即可(POST /dsh-market/restart)
|
|
@@ -106,10 +106,10 @@ npm install # 无第三方依赖,仅初始化
|
|
|
106
106
|
### 2. 验证环境(离线,不需要任何 API key)
|
|
107
107
|
|
|
108
108
|
```bash
|
|
109
|
-
npm test #
|
|
109
|
+
npm test # 25 项 fixture 测试,全部离线跑通
|
|
110
110
|
```
|
|
111
111
|
|
|
112
|
-
看到 `TOTAL:
|
|
112
|
+
看到 `TOTAL: 25 passed, 0 failed` 即环境就绪。
|
|
113
113
|
|
|
114
114
|
### 3. 一行命令喂入 demo 文件
|
|
115
115
|
|
|
@@ -198,7 +198,7 @@ parsers: book, paper, video, web
|
|
|
198
198
|
```bash
|
|
199
199
|
npm install
|
|
200
200
|
npm run lint # node --check 全部 JS
|
|
201
|
-
npm test # fixture 测试(
|
|
201
|
+
npm test # fixture 测试(25 项,全部离线)
|
|
202
202
|
npm run validate # 发布前验证(含安全检查)
|
|
203
203
|
```
|
|
204
204
|
|
package/lib/cache.js
CHANGED
|
@@ -1,87 +1,102 @@
|
|
|
1
|
-
// ============================================================================
|
|
2
|
-
// lib/cache.js — idempotency: input fingerprint cache
|
|
3
|
-
// ============================================================================
|
|
4
|
-
// 对输入内容计算指纹(URL 或文件路径 + 文件大小/修改时间),
|
|
5
|
-
// 已处理过的输入直接跳过,避免重复消耗 token。
|
|
6
|
-
// 指纹记录以 JSONL 追加式存储,支持断点恢复。
|
|
7
|
-
// ============================================================================
|
|
8
|
-
// 用法:
|
|
9
|
-
// const cache = require("./lib/cache");
|
|
10
|
-
// const key = cache.fingerprint(input, fileMeta);
|
|
11
|
-
// if (cache.hasProcessed(cacheFile, key)) { skip; }
|
|
12
|
-
// cache.markProcessed(cacheFile, key, outputPath);
|
|
13
|
-
// ============================================================================
|
|
14
|
-
|
|
15
|
-
const fs = require("fs");
|
|
16
|
-
const path = require("path");
|
|
17
|
-
const crypto = require("crypto");
|
|
18
|
-
|
|
19
|
-
/**
|
|
20
|
-
* 规范化输入:对 http(s) URL 做归一化(host 小写、去默认端口、去 fragment、
|
|
21
|
-
* 去空 query 的 "?"、去尾部斜杠),避免同一 URL 的不同写法产生不同指纹;
|
|
22
|
-
* 非 URL(本地路径/纯文本)原样返回。
|
|
23
|
-
*/
|
|
24
|
-
function normalizeInput(input) {
|
|
25
|
-
const s = typeof input === "string" ? input.trim() : "";
|
|
26
|
-
if (!/^https?:\/\//i.test(s)) return s;
|
|
27
|
-
try {
|
|
28
|
-
const url = new URL(s);
|
|
29
|
-
url.hash = "";
|
|
30
|
-
url.hostname = url.hostname.toLowerCase();
|
|
31
|
-
if ((url.protocol === "http:" && url.port === "80") || (url.protocol === "https:" && url.port === "443")) {
|
|
32
|
-
url.port = "";
|
|
33
|
-
}
|
|
34
|
-
if (url.
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
}
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
}
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
return true;
|
|
85
|
-
}
|
|
86
|
-
|
|
87
|
-
|
|
1
|
+
// ============================================================================
|
|
2
|
+
// lib/cache.js — idempotency: input fingerprint cache
|
|
3
|
+
// ============================================================================
|
|
4
|
+
// 对输入内容计算指纹(URL 或文件路径 + 文件大小/修改时间),
|
|
5
|
+
// 已处理过的输入直接跳过,避免重复消耗 token。
|
|
6
|
+
// 指纹记录以 JSONL 追加式存储,支持断点恢复。
|
|
7
|
+
// ============================================================================
|
|
8
|
+
// 用法:
|
|
9
|
+
// const cache = require("./lib/cache");
|
|
10
|
+
// const key = cache.fingerprint(input, fileMeta);
|
|
11
|
+
// if (cache.hasProcessed(cacheFile, key)) { skip; }
|
|
12
|
+
// cache.markProcessed(cacheFile, key, outputPath);
|
|
13
|
+
// ============================================================================
|
|
14
|
+
|
|
15
|
+
const fs = require("fs");
|
|
16
|
+
const path = require("path");
|
|
17
|
+
const crypto = require("crypto");
|
|
18
|
+
|
|
19
|
+
/**
|
|
20
|
+
* 规范化输入:对 http(s) URL 做归一化(host 小写、去默认端口、去 fragment、
|
|
21
|
+
* 去空 query 的 "?"、去尾部斜杠),避免同一 URL 的不同写法产生不同指纹;
|
|
22
|
+
* 非 URL(本地路径/纯文本)原样返回。
|
|
23
|
+
*/
|
|
24
|
+
function normalizeInput(input) {
|
|
25
|
+
const s = typeof input === "string" ? input.trim() : "";
|
|
26
|
+
if (!/^https?:\/\//i.test(s)) return s;
|
|
27
|
+
try {
|
|
28
|
+
const url = new URL(s);
|
|
29
|
+
url.hash = "";
|
|
30
|
+
url.hostname = url.hostname.toLowerCase();
|
|
31
|
+
if ((url.protocol === "http:" && url.port === "80") || (url.protocol === "https:" && url.port === "443")) {
|
|
32
|
+
url.port = "";
|
|
33
|
+
}
|
|
34
|
+
if (url.pathname.length > 1 && url.pathname.endsWith("/")) {
|
|
35
|
+
url.pathname = url.pathname.slice(0, -1);
|
|
36
|
+
}
|
|
37
|
+
// 去掉 URL 尾部的空 query "?"(new URL("...doc?").search 返回 "",无法靠 search 判断,
|
|
38
|
+
// 需检查 toString() 输出;同一 URL 的 "doc" 与 "doc?" 两种写法应共享指纹)
|
|
39
|
+
let normalized = url.toString();
|
|
40
|
+
if (normalized.endsWith("?")) normalized = normalized.slice(0, -1);
|
|
41
|
+
return normalized;
|
|
42
|
+
} catch (e) {
|
|
43
|
+
return s; // 非法 URL 原样处理
|
|
44
|
+
}
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
/** 计算输入指纹:URL 规范化后 hash;本地文件加大小 + 修改时间 */
|
|
48
|
+
function fingerprint(input, fileMeta) {
|
|
49
|
+
const base = normalizeInput(input);
|
|
50
|
+
if (!base) return null;
|
|
51
|
+
const metaStr = fileMeta
|
|
52
|
+
? "|" + (fileMeta.size ?? "") + "|" + (fileMeta.mtimeMs ?? "")
|
|
53
|
+
: "";
|
|
54
|
+
return crypto.createHash("sha256").update(base + metaStr).digest("hex").slice(0, 16);
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
/** 从缓存文件读取已处理记录(JSONL) */
|
|
58
|
+
function readCache(cacheFile) {
|
|
59
|
+
if (!fs.existsSync(cacheFile)) return new Map();
|
|
60
|
+
const map = new Map();
|
|
61
|
+
try {
|
|
62
|
+
const lines = fs.readFileSync(cacheFile, "utf8").split("\n");
|
|
63
|
+
for (const line of lines) {
|
|
64
|
+
if (!line.trim()) continue;
|
|
65
|
+
try {
|
|
66
|
+
const rec = JSON.parse(line);
|
|
67
|
+
if (rec && rec.key) map.set(rec.key, rec);
|
|
68
|
+
} catch (e) { /* 跳过损坏行 */ }
|
|
69
|
+
}
|
|
70
|
+
} catch (e) { /* 缓存不可读则视为空 */ }
|
|
71
|
+
return map;
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
/** 是否已处理过该指纹;opts.maxAgeMs 提供时,过期(记录时间早于阈值)视为未处理 */
|
|
75
|
+
function hasProcessed(cacheFile, key, opts) {
|
|
76
|
+
if (!key) return false;
|
|
77
|
+
const rec = readCache(cacheFile).get(key);
|
|
78
|
+
if (!rec) return false;
|
|
79
|
+
const maxAgeMs = opts && opts.maxAgeMs;
|
|
80
|
+
if (maxAgeMs && rec.ts) {
|
|
81
|
+
const age = Date.now() - new Date(rec.ts).getTime();
|
|
82
|
+
if (!isFinite(age) || age > maxAgeMs) return false;
|
|
83
|
+
}
|
|
84
|
+
return true;
|
|
85
|
+
}
|
|
86
|
+
|
|
87
|
+
/** 标记已处理(追加 JSONL 记录);写缓存失败仅告警,不中断主流程 */
|
|
88
|
+
function markProcessed(cacheFile, key, outputPath, meta) {
|
|
89
|
+
if (!key) return false;
|
|
90
|
+
try {
|
|
91
|
+
const dir = path.dirname(cacheFile);
|
|
92
|
+
if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
|
|
93
|
+
const rec = { key, outputPath, ts: new Date().toISOString(), ...(meta || {}) };
|
|
94
|
+
fs.appendFileSync(cacheFile, JSON.stringify(rec) + "\n", "utf8");
|
|
95
|
+
return true;
|
|
96
|
+
} catch (e) {
|
|
97
|
+
console.warn("[cache] markProcessed failed:", e.message);
|
|
98
|
+
return false;
|
|
99
|
+
}
|
|
100
|
+
}
|
|
101
|
+
|
|
102
|
+
module.exports = { fingerprint, normalizeInput, readCache, hasProcessed, markProcessed };
|
package/package.json
CHANGED
|
@@ -1,69 +1,69 @@
|
|
|
1
|
-
{
|
|
2
|
-
"name": "deep-read-summarize",
|
|
3
|
-
"version": "0.3.
|
|
4
|
-
"description": "Deep reading & summarization for DSH: books/papers/videos/web → structured Obsidian notes. Plugin parsers, MapReduce deep-read, JSON Schema output, idempotent cache.",
|
|
5
|
-
"type": "commonjs",
|
|
6
|
-
"main": "index.js",
|
|
7
|
-
"exports": {
|
|
8
|
-
".": "./index.js",
|
|
9
|
-
"./workflow": "./workflow.js",
|
|
10
|
-
"./parsers": "./parsers/index.js",
|
|
11
|
-
"./schemas": "./schemas/index.js",
|
|
12
|
-
"./package.json": "./package.json"
|
|
13
|
-
},
|
|
14
|
-
"files": [
|
|
15
|
-
"index.js",
|
|
16
|
-
"workflow.js",
|
|
17
|
-
"parsers",
|
|
18
|
-
"schemas",
|
|
19
|
-
"lib",
|
|
20
|
-
"skills",
|
|
21
|
-
"scripts",
|
|
22
|
-
"tests",
|
|
23
|
-
"docs",
|
|
24
|
-
"cordis.patch.yml",
|
|
25
|
-
"CHANGELOG.md",
|
|
26
|
-
"README.md",
|
|
27
|
-
"LICENSE",
|
|
28
|
-
"SECURITY.md",
|
|
29
|
-
".env.example"
|
|
30
|
-
],
|
|
31
|
-
"scripts": {
|
|
32
|
-
"test": "node tests/run-tests.js",
|
|
33
|
-
"lint": "node scripts/lint.js",
|
|
34
|
-
"validate": "node tests/quick_validate.js && node scripts/security-check.js",
|
|
35
|
-
"security": "node scripts/security-check.js",
|
|
36
|
-
"test:node": "node --test tests/",
|
|
37
|
-
"prepublishOnly": "node tests/run-tests.js && node scripts/lint.js && node scripts/security-check.js"
|
|
38
|
-
},
|
|
39
|
-
"dsh": {
|
|
40
|
-
"bundle": {
|
|
41
|
-
"patch": "./cordis.patch.yml"
|
|
42
|
-
},
|
|
43
|
-
"lockedRuntime": "workflow tool semantics — see CHANGELOG.md",
|
|
44
|
-
"minVersion": "0.1.0",
|
|
45
|
-
"breakingPolicy": "re-validate with npm test after each DSH upgrade"
|
|
46
|
-
},
|
|
47
|
-
"keywords": [
|
|
48
|
-
"dsh",
|
|
49
|
-
"workflow",
|
|
50
|
-
"summarization",
|
|
51
|
-
"deep-read",
|
|
52
|
-
"obsidian",
|
|
53
|
-
"llm",
|
|
54
|
-
"subagent",
|
|
55
|
-
"mapreduce",
|
|
56
|
-
"parsers"
|
|
57
|
-
],
|
|
58
|
-
"license": "MIT",
|
|
59
|
-
"publishConfig": {
|
|
60
|
-
"access": "public"
|
|
61
|
-
},
|
|
62
|
-
"engines": {
|
|
63
|
-
"node": ">=18"
|
|
64
|
-
},
|
|
65
|
-
"repository": {
|
|
66
|
-
"type": "git",
|
|
67
|
-
"url": "git+https://github.com/PensiveFei/deep-read-summarize.git"
|
|
68
|
-
}
|
|
69
|
-
}
|
|
1
|
+
{
|
|
2
|
+
"name": "deep-read-summarize",
|
|
3
|
+
"version": "0.3.3",
|
|
4
|
+
"description": "Deep reading & summarization for DSH: books/papers/videos/web → structured Obsidian notes. Plugin parsers, MapReduce deep-read, JSON Schema output, idempotent cache.",
|
|
5
|
+
"type": "commonjs",
|
|
6
|
+
"main": "index.js",
|
|
7
|
+
"exports": {
|
|
8
|
+
".": "./index.js",
|
|
9
|
+
"./workflow": "./workflow.js",
|
|
10
|
+
"./parsers": "./parsers/index.js",
|
|
11
|
+
"./schemas": "./schemas/index.js",
|
|
12
|
+
"./package.json": "./package.json"
|
|
13
|
+
},
|
|
14
|
+
"files": [
|
|
15
|
+
"index.js",
|
|
16
|
+
"workflow.js",
|
|
17
|
+
"parsers",
|
|
18
|
+
"schemas",
|
|
19
|
+
"lib",
|
|
20
|
+
"skills",
|
|
21
|
+
"scripts",
|
|
22
|
+
"tests",
|
|
23
|
+
"docs",
|
|
24
|
+
"cordis.patch.yml",
|
|
25
|
+
"CHANGELOG.md",
|
|
26
|
+
"README.md",
|
|
27
|
+
"LICENSE",
|
|
28
|
+
"SECURITY.md",
|
|
29
|
+
".env.example"
|
|
30
|
+
],
|
|
31
|
+
"scripts": {
|
|
32
|
+
"test": "node tests/run-tests.js",
|
|
33
|
+
"lint": "node scripts/lint.js",
|
|
34
|
+
"validate": "node tests/quick_validate.js && node scripts/security-check.js",
|
|
35
|
+
"security": "node scripts/security-check.js",
|
|
36
|
+
"test:node": "node --test tests/",
|
|
37
|
+
"prepublishOnly": "node tests/run-tests.js && node scripts/lint.js && node scripts/security-check.js"
|
|
38
|
+
},
|
|
39
|
+
"dsh": {
|
|
40
|
+
"bundle": {
|
|
41
|
+
"patch": "./cordis.patch.yml"
|
|
42
|
+
},
|
|
43
|
+
"lockedRuntime": "workflow tool semantics — see CHANGELOG.md",
|
|
44
|
+
"minVersion": "0.1.0",
|
|
45
|
+
"breakingPolicy": "re-validate with npm test after each DSH upgrade"
|
|
46
|
+
},
|
|
47
|
+
"keywords": [
|
|
48
|
+
"dsh",
|
|
49
|
+
"workflow",
|
|
50
|
+
"summarization",
|
|
51
|
+
"deep-read",
|
|
52
|
+
"obsidian",
|
|
53
|
+
"llm",
|
|
54
|
+
"subagent",
|
|
55
|
+
"mapreduce",
|
|
56
|
+
"parsers"
|
|
57
|
+
],
|
|
58
|
+
"license": "MIT",
|
|
59
|
+
"publishConfig": {
|
|
60
|
+
"access": "public"
|
|
61
|
+
},
|
|
62
|
+
"engines": {
|
|
63
|
+
"node": ">=18"
|
|
64
|
+
},
|
|
65
|
+
"repository": {
|
|
66
|
+
"type": "git",
|
|
67
|
+
"url": "git+https://github.com/PensiveFei/deep-read-summarize.git"
|
|
68
|
+
}
|
|
69
|
+
}
|
package/parsers/_prompt.js
CHANGED
|
@@ -1,44 +1,44 @@
|
|
|
1
|
-
// ============================================================================
|
|
2
|
-
// parsers/_prompt.js — 共享「获取+分块」提示词模板(内部模块)
|
|
3
|
-
// ============================================================================
|
|
4
|
-
// 四个内置解析器共用同一提示词骨架,各自只提供差异化配置:
|
|
5
|
-
// - label/kindLabel:类型名称(中文 / 中英混合)
|
|
6
|
-
// - steps:获取文本的具体步骤(不含分块步骤,模板自动编号)
|
|
7
|
-
// - chunkRule:分块规则描述
|
|
8
|
-
// - jsonExample:输出 JSON 示例(含 kind 字段)
|
|
9
|
-
// 保持各解析器 buildPrompt(input, opts) 插件接口不变;fork 时仍可整体重写。
|
|
10
|
-
// ============================================================================
|
|
11
|
-
|
|
12
|
-
/**
|
|
13
|
-
* 构建「获取+分块」提示词。
|
|
14
|
-
* @param {object} cfg
|
|
15
|
-
* @param {string} cfg.input 输入(URL 或本地路径)
|
|
16
|
-
* @param {object} [cfg.opts] 选项(tempDir/tempFile/maxChunks)
|
|
17
|
-
* @param {string} cfg.fetchTarget 获取对象描述(如「书籍全文」「视频字幕/转写」)
|
|
18
|
-
* @param {string} cfg.kindLabel 内容类型描述(如「书籍(book)」)
|
|
19
|
-
* @param {string[]} cfg.steps 获取步骤(不含分块步骤;模板统一编号并追加分块步骤)
|
|
20
|
-
* @param {string} cfg.chunkRule 分块规则(如「章节(或章节内小节)」)
|
|
21
|
-
* @param {string} cfg.jsonExample 输出 JSON 示例
|
|
22
|
-
* @returns {string} 完整提示词
|
|
23
|
-
*/
|
|
24
|
-
function buildFetchPrompt(cfg) {
|
|
25
|
-
const opts = cfg.opts || {};
|
|
26
|
-
const tempDir = opts.tempDir || "./.tmp";
|
|
27
|
-
const tempFile = tempDir + "
|
|
28
|
-
const maxChunks = opts.maxChunks || 6;
|
|
29
|
-
const steps = (cfg.steps || []).map(function (s, i) { return (i + 1) + ". " + s; });
|
|
30
|
-
return [
|
|
31
|
-
"你的任务:获取" + cfg.fetchTarget + "并保存为纯文本,然后给出分块计划。",
|
|
32
|
-
"内容类型:" + cfg.kindLabel,
|
|
33
|
-
"输入来源:" + cfg.input,
|
|
34
|
-
"目标文件:" + tempFile + "(先创建目录 " + tempDir + ",用 write 工具写入)",
|
|
35
|
-
"步骤:"
|
|
36
|
-
].concat(steps, [
|
|
37
|
-
(steps.length + 1) + ". 分块:按" + cfg.chunkRule + "切分,每块约 100-200 行或 4000-6000 字,块数不超过 " + maxChunks + " 块;每块给 id/topic/startLine/endLine。",
|
|
38
|
-
"输出 JSON:",
|
|
39
|
-
cfg.jsonExample,
|
|
40
|
-
"若无法获取文本输出 { \"saved\": false, \"message\": \"原因\" }。只输出 JSON。"
|
|
41
|
-
]).join("\n");
|
|
42
|
-
}
|
|
43
|
-
|
|
44
|
-
module.exports = { buildFetchPrompt };
|
|
1
|
+
// ============================================================================
|
|
2
|
+
// parsers/_prompt.js — 共享「获取+分块」提示词模板(内部模块)
|
|
3
|
+
// ============================================================================
|
|
4
|
+
// 四个内置解析器共用同一提示词骨架,各自只提供差异化配置:
|
|
5
|
+
// - label/kindLabel:类型名称(中文 / 中英混合)
|
|
6
|
+
// - steps:获取文本的具体步骤(不含分块步骤,模板自动编号)
|
|
7
|
+
// - chunkRule:分块规则描述
|
|
8
|
+
// - jsonExample:输出 JSON 示例(含 kind 字段)
|
|
9
|
+
// 保持各解析器 buildPrompt(input, opts) 插件接口不变;fork 时仍可整体重写。
|
|
10
|
+
// ============================================================================
|
|
11
|
+
|
|
12
|
+
/**
|
|
13
|
+
* 构建「获取+分块」提示词。
|
|
14
|
+
* @param {object} cfg
|
|
15
|
+
* @param {string} cfg.input 输入(URL 或本地路径)
|
|
16
|
+
* @param {object} [cfg.opts] 选项(tempDir/tempFile/maxChunks)
|
|
17
|
+
* @param {string} cfg.fetchTarget 获取对象描述(如「书籍全文」「视频字幕/转写」)
|
|
18
|
+
* @param {string} cfg.kindLabel 内容类型描述(如「书籍(book)」)
|
|
19
|
+
* @param {string[]} cfg.steps 获取步骤(不含分块步骤;模板统一编号并追加分块步骤)
|
|
20
|
+
* @param {string} cfg.chunkRule 分块规则(如「章节(或章节内小节)」)
|
|
21
|
+
* @param {string} cfg.jsonExample 输出 JSON 示例
|
|
22
|
+
* @returns {string} 完整提示词
|
|
23
|
+
*/
|
|
24
|
+
function buildFetchPrompt(cfg) {
|
|
25
|
+
const opts = cfg.opts || {};
|
|
26
|
+
const tempDir = opts.tempDir || "./.tmp";
|
|
27
|
+
const tempFile = opts.tempFile || (tempDir.replace(/[\\/]+$/, "") + "/input.txt");
|
|
28
|
+
const maxChunks = opts.maxChunks || 6;
|
|
29
|
+
const steps = (cfg.steps || []).map(function (s, i) { return (i + 1) + ". " + s; });
|
|
30
|
+
return [
|
|
31
|
+
"你的任务:获取" + cfg.fetchTarget + "并保存为纯文本,然后给出分块计划。",
|
|
32
|
+
"内容类型:" + cfg.kindLabel,
|
|
33
|
+
"输入来源:" + cfg.input,
|
|
34
|
+
"目标文件:" + tempFile + "(先创建目录 " + tempDir + ",用 write 工具写入)",
|
|
35
|
+
"步骤:"
|
|
36
|
+
].concat(steps, [
|
|
37
|
+
(steps.length + 1) + ". 分块:按" + cfg.chunkRule + "切分,每块约 100-200 行或 4000-6000 字,块数不超过 " + maxChunks + " 块;每块给 id/topic/startLine/endLine。",
|
|
38
|
+
"输出 JSON:",
|
|
39
|
+
cfg.jsonExample,
|
|
40
|
+
"若无法获取文本输出 { \"saved\": false, \"message\": \"原因\" }。只输出 JSON。"
|
|
41
|
+
]).join("\n");
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
module.exports = { buildFetchPrompt };
|
package/tests/cache.test.js
CHANGED
|
@@ -1,83 +1,117 @@
|
|
|
1
|
-
// tests/cache.test.js — idempotency cache tests
|
|
2
|
-
const assert = require('assert');
|
|
3
|
-
const fs = require('fs');
|
|
4
|
-
const os = require('os');
|
|
5
|
-
const path = require('path');
|
|
6
|
-
const cache = require('../lib/cache');
|
|
7
|
-
|
|
8
|
-
let passed = 0, failed = 0;
|
|
9
|
-
function test(name, fn) {
|
|
10
|
-
try { fn(); passed++; console.log(' ✅', name); }
|
|
11
|
-
catch (e) { failed++; console.error(' ❌', name, '—', e.message); }
|
|
12
|
-
}
|
|
13
|
-
|
|
14
|
-
const tmp = path.join(os.tmpdir(), 'drs-cache-test-' + Date.now());
|
|
15
|
-
const cacheFile = path.join(tmp, 'cache.jsonl');
|
|
16
|
-
|
|
17
|
-
test('fingerprint is deterministic for same input', () => {
|
|
18
|
-
const a = cache.fingerprint('https://arxiv.org/abs/2307.09042');
|
|
19
|
-
const b = cache.fingerprint('https://arxiv.org/abs/2307.09042');
|
|
20
|
-
assert.strictEqual(a, b);
|
|
21
|
-
assert.ok(a && a.length >= 16);
|
|
22
|
-
});
|
|
23
|
-
|
|
24
|
-
test('fingerprint differs for different inputs', () => {
|
|
25
|
-
const a = cache.fingerprint('input-a');
|
|
26
|
-
const b = cache.fingerprint('input-b');
|
|
27
|
-
assert.notStrictEqual(a, b);
|
|
28
|
-
});
|
|
29
|
-
|
|
30
|
-
test('fingerprint normalizes URL variants', () => {
|
|
31
|
-
const a = cache.fingerprint('https://EXAMPLE.com/doc/');
|
|
32
|
-
const b = cache.fingerprint('https://example.com/doc#frag');
|
|
33
|
-
assert.strictEqual(a, b, 'URL variants (case/slash/fragment) should share fingerprint');
|
|
34
|
-
const c = cache.fingerprint('https://example.com/doc?x=1');
|
|
35
|
-
assert.notStrictEqual(a, c, 'different query should produce different fingerprint');
|
|
36
|
-
});
|
|
37
|
-
|
|
38
|
-
test('
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
assert.strictEqual(
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
const
|
|
49
|
-
|
|
50
|
-
const
|
|
51
|
-
assert.notStrictEqual(
|
|
52
|
-
});
|
|
53
|
-
|
|
54
|
-
test('
|
|
55
|
-
assert.strictEqual(cache.
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
});
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
const
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
1
|
+
// tests/cache.test.js — idempotency cache tests
|
|
2
|
+
const assert = require('assert');
|
|
3
|
+
const fs = require('fs');
|
|
4
|
+
const os = require('os');
|
|
5
|
+
const path = require('path');
|
|
6
|
+
const cache = require('../lib/cache');
|
|
7
|
+
|
|
8
|
+
let passed = 0, failed = 0;
|
|
9
|
+
function test(name, fn) {
|
|
10
|
+
try { fn(); passed++; console.log(' ✅', name); }
|
|
11
|
+
catch (e) { failed++; console.error(' ❌', name, '—', e.message); }
|
|
12
|
+
}
|
|
13
|
+
|
|
14
|
+
const tmp = path.join(os.tmpdir(), 'drs-cache-test-' + Date.now());
|
|
15
|
+
const cacheFile = path.join(tmp, 'cache.jsonl');
|
|
16
|
+
|
|
17
|
+
test('fingerprint is deterministic for same input', () => {
|
|
18
|
+
const a = cache.fingerprint('https://arxiv.org/abs/2307.09042');
|
|
19
|
+
const b = cache.fingerprint('https://arxiv.org/abs/2307.09042');
|
|
20
|
+
assert.strictEqual(a, b);
|
|
21
|
+
assert.ok(a && a.length >= 16);
|
|
22
|
+
});
|
|
23
|
+
|
|
24
|
+
test('fingerprint differs for different inputs', () => {
|
|
25
|
+
const a = cache.fingerprint('input-a');
|
|
26
|
+
const b = cache.fingerprint('input-b');
|
|
27
|
+
assert.notStrictEqual(a, b);
|
|
28
|
+
});
|
|
29
|
+
|
|
30
|
+
test('fingerprint normalizes URL variants', () => {
|
|
31
|
+
const a = cache.fingerprint('https://EXAMPLE.com/doc/');
|
|
32
|
+
const b = cache.fingerprint('https://example.com/doc#frag');
|
|
33
|
+
assert.strictEqual(a, b, 'URL variants (case/slash/fragment) should share fingerprint');
|
|
34
|
+
const c = cache.fingerprint('https://example.com/doc?x=1');
|
|
35
|
+
assert.notStrictEqual(a, c, 'different query should produce different fingerprint');
|
|
36
|
+
});
|
|
37
|
+
|
|
38
|
+
test('fingerprint normalizes trailing empty query (doc vs doc?)', () => {
|
|
39
|
+
const a = cache.fingerprint('https://example.com/doc');
|
|
40
|
+
const b = cache.fingerprint('https://example.com/doc?');
|
|
41
|
+
assert.strictEqual(a, b, 'trailing "?" (empty query) should not create a distinct fingerprint');
|
|
42
|
+
assert.strictEqual(cache.normalizeInput('https://example.com/doc?'), 'https://example.com/doc');
|
|
43
|
+
});
|
|
44
|
+
|
|
45
|
+
test('fingerprint keeps size 0 distinct from absent meta', () => {
|
|
46
|
+
// 空文件(size=0)与「无 meta」不应共享指纹;0 值必须参与拼接
|
|
47
|
+
const zero = cache.fingerprint('path-to-empty-file', { size: 0, mtimeMs: 123 });
|
|
48
|
+
const absent = cache.fingerprint('path-to-empty-file');
|
|
49
|
+
assert.notStrictEqual(zero, absent, 'size=0 must not collapse into absent meta');
|
|
50
|
+
const five = cache.fingerprint('path-to-empty-file', { size: 5, mtimeMs: 123 });
|
|
51
|
+
assert.notStrictEqual(zero, five, 'size 0 vs 5 must differ');
|
|
52
|
+
});
|
|
53
|
+
|
|
54
|
+
test('normalizeInput keeps non-URL input and root path unchanged', () => {
|
|
55
|
+
assert.strictEqual(cache.normalizeInput('C:\\Users\\x\\doc.md'), 'C:\\Users\\x\\doc.md');
|
|
56
|
+
assert.strictEqual(cache.normalizeInput('https://example.com/'), 'https://example.com/');
|
|
57
|
+
assert.strictEqual(cache.normalizeInput(' '), '');
|
|
58
|
+
});
|
|
59
|
+
|
|
60
|
+
test('fingerprint includes file meta for local files', () => {
|
|
61
|
+
const f = path.join(tmp, 'sample.txt');
|
|
62
|
+
fs.mkdirSync(tmp, { recursive: true });
|
|
63
|
+
fs.writeFileSync(f, 'hello');
|
|
64
|
+
const st = fs.statSync(f);
|
|
65
|
+
const a = cache.fingerprint(f, { size: st.size, mtimeMs: st.mtimeMs });
|
|
66
|
+
const b = cache.fingerprint(f, { size: st.size + 1, mtimeMs: st.mtimeMs });
|
|
67
|
+
assert.notStrictEqual(a, b, 'size change should change fingerprint');
|
|
68
|
+
});
|
|
69
|
+
|
|
70
|
+
test('hasProcessed returns false for unknown key', () => {
|
|
71
|
+
assert.strictEqual(cache.hasProcessed(cacheFile, 'unknown-key'), false);
|
|
72
|
+
});
|
|
73
|
+
|
|
74
|
+
test('markProcessed then hasProcessed returns true', () => {
|
|
75
|
+
const key = cache.fingerprint('https://example.com/doc');
|
|
76
|
+
cache.markProcessed(cacheFile, key, '/tmp/out.md');
|
|
77
|
+
assert.strictEqual(cache.hasProcessed(cacheFile, key), true);
|
|
78
|
+
});
|
|
79
|
+
|
|
80
|
+
test('cache file is JSONL (append-only)', () => {
|
|
81
|
+
const raw = fs.readFileSync(cacheFile, 'utf8').trim().split('\n');
|
|
82
|
+
assert.ok(raw.length >= 1);
|
|
83
|
+
const rec = JSON.parse(raw[0]);
|
|
84
|
+
assert.ok(rec.key && rec.outputPath && rec.ts);
|
|
85
|
+
});
|
|
86
|
+
|
|
87
|
+
test('corrupted line is skipped', () => {
|
|
88
|
+
fs.appendFileSync(cacheFile, '{broken json}\n', 'utf8');
|
|
89
|
+
const map = cache.readCache(cacheFile);
|
|
90
|
+
assert.ok(map.size >= 1, 'should still read valid lines');
|
|
91
|
+
});
|
|
92
|
+
|
|
93
|
+
test('hasProcessed TTL: stale record is treated as unprocessed', () => {
|
|
94
|
+
const key = cache.fingerprint('https://example.com/stale');
|
|
95
|
+
cache.markProcessed(cacheFile, key, '/tmp/stale.md', { ts: '2020-01-01T00:00:00.000Z' });
|
|
96
|
+
assert.strictEqual(cache.hasProcessed(cacheFile, key), true, 'no TTL -> still processed');
|
|
97
|
+
assert.strictEqual(cache.hasProcessed(cacheFile, key, { maxAgeMs: 60 * 1000 }), false, 'stale -> unprocessed');
|
|
98
|
+
assert.strictEqual(cache.hasProcessed(cacheFile, key, { maxAgeMs: 1e12 }), true, 'recent enough -> processed');
|
|
99
|
+
});
|
|
100
|
+
|
|
101
|
+
test('markProcessed tolerates unwritable cache location', () => {
|
|
102
|
+
const blocker = path.join(tmp, 'blocked-file');
|
|
103
|
+
fs.writeFileSync(blocker, 'x');
|
|
104
|
+
const badFile = path.join(blocker, 'cache.jsonl');
|
|
105
|
+
let threw = false, ret = null;
|
|
106
|
+
try { ret = cache.markProcessed(badFile, 'some-key', '/out.md'); } catch (e) { threw = true; }
|
|
107
|
+
assert.strictEqual(threw, false, 'should not throw');
|
|
108
|
+
assert.strictEqual(ret, false, 'should report failure instead');
|
|
109
|
+
});
|
|
110
|
+
|
|
111
|
+
// cleanup
|
|
112
|
+
fs.rmSync(tmp, { recursive: true, force: true });
|
|
113
|
+
|
|
114
|
+
console.log('');
|
|
115
|
+
console.log(passed + ' cache tests passed, ' + failed + ' failed');
|
|
116
|
+
if (require.main === module) process.exit(failed ? 1 : 0);
|
|
83
117
|
module.exports = { passed, failed };
|
package/workflow.js
CHANGED
|
@@ -35,7 +35,7 @@ const meta = {
|
|
|
35
35
|
};
|
|
36
36
|
|
|
37
37
|
// ===== script(复制到 workflow 工具的 script 参数)=====
|
|
38
|
-
const script = "// ========== deep-read-summarize v3.0 ==========\n// 3 波次:解析器获取分块 → 并行精读 → 成稿+质量校验\n// 特性:解析器插件化 / 失败纪律(fatal vs 降级) / 幂等 / 成本控制\n\n// ---------- 0. 配置校验(fatal 级错误:必须终止) ----------\nconst input = (args && args.input || \"\").trim();\nif (!input) throw new Error(\"[FATAL] args.input 缺失:请输入内容链接或文件路径\");\nconst forcedType = ((args && args.type) || \"auto\").toLowerCase();\nif ([\"auto\",\"book\",\"paper\",\"video\",\"web\"].indexOf(forcedType) === -1)\n throw new Error(\"[FATAL] args.type 非法:必须为 auto|book|paper|video|web\");\nconst options = (args && args.options) || {};\nconst minWords = options.minWords || 2500;\nconst fastMode = !!options.fastMode;\nconst maxChunks = Math.max(1, Math.min(options.maxChunks || 6, 12));\nconst requireCitations = options.requireCitations !== false;\nconst includeTimestamps = !!options.includeTimestamps;\nconst maxRetries = options.maxRetries || 1;\nconst cacheEnabled = options.cache !== false;\nconst outputDir = options.outputDir || \"./output\";\nconst tempDir = options.tempDir || \"./.tmp\";\nconst tempFile = tempDir + \"/input.txt\";\n\n// ---------- 0.5 幂等检查:输入指纹缓存 ----------\nlet cacheHit = false;\nif (cacheEnabled && Array.isArray(args._processedKeys)) {\n const fpKey = input.trim().slice(0, 300);\n cacheHit = args._processedKeys.indexOf(fpKey) !== -1;\n if (cacheHit) log(\"幂等命中:该输入已处理过,跳过\");\n}\nif (cacheHit) {\n return { ok: true, cached: true, kind: forcedType, title: (args && args.title) || \"cached\", filePath: \"\", chunksCount: 0, fastMode: fastMode, qualityPassed: true, qualityIssues: [], failedChunks: 0, note: \"\" };\n}\n\n// ---------- 1. 解析器选择(由 args._parsers 注入) ----------\nphase(\"解析器选择\");\nconst parsers = args._parsers || {};\nconst kindNames = { book: \"书籍\", paper: \"学术论文\", video: \"视频\", web: \"网页/文章\" };\nconst kindFocus = {\n book: \"按章节主题展开核心论点、论证、关键例子、章节间关系,收集关键概念与可引用原文(含页码/章节)\",\n paper: \"四维精读:1)研究问题与方法 2)结论与贡献 3)局限与批判 4)与同类工作比较;收集术语与可引用原文(含章节/段落)\",\n video: \"按主题脉络组织,提取核心论点、证据、例子、可操作建议,收集关键概念与主讲人原话\" + (includeTimestamps ? \"(含时间戳定位)\" : \"(不标时间戳)\"),\n web: \"提取核心论点、结构拆解、关键概念、可引用原文\"\n};\n\n// ---------- 2. 波次1:解析器获取 + 分块 ----------\nphase(\"获取与分块\");\nconst fetchPrompt = (parsers[forcedType !== \"auto\" ? forcedType : \"web\"] || parsers.web || { buildPrompt: function() { return [\"未知解析器\"]; } })\n .buildPrompt(input, { tempDir: tempDir, tempFile: tempFile, maxChunks: maxChunks });\nconst fetchSchema = {\n type: \"object\",\n properties: {\n kind: { type: \"string\" }, saved: { type: \"boolean\" }, message: { type: \"string\" }, totalLines: { type: \"number\" },\n metadata: { type: \"object\", properties: { title: { type: \"string\" }, author: { type: \"string\" }, year: { type: \"string\" }, language: { type: \"string\" } }, required: [], additionalProperties: true },\n chunkPlan: {\n type: \"array\",\n items: { type: \"object\", properties: { id: { type: \"number\" }, topic: { type: \"string\" }, startLine: { type: \"number\" }, endLine: { type: \"number\" } }, required: [\"id\",\"topic\",\"startLine\",\"endLine\"], additionalProperties: false }\n }\n },\n required: [\"saved\"], additionalProperties: false\n};\n\nconst fetched = await agent(fetchPrompt, { schema: fetchSchema, label: \"获取+分块\", phase: \"获取与分块\" });\n\n// 失败纪律:内容获取失败 = 可降级错误(返回失败结果,不抛 fatal)\nif (!fetched || !fetched.saved) {\n return { ok: false, stage: \"fetch\", fatal: false, message: (fetched && fetched.message) || \"获取失败(可降级:检查输入或手动提供文本)\" };\n}\nlet kind = (fetched.kind && forcedType === \"auto\") ? fetched.kind : forcedType;\nconst chunkPlan = (fetched && fetched.chunkPlan) || [];\nif (chunkPlan.length === 0) return { ok: false, stage: \"fetch\", fatal: false, message: \"分块计划为空\" };\nconst metadata = (fetched && fetched.metadata) || {};\nlog(\"类型:\" + kind + \" 行数:\" + fetched.totalLines + \" 块数:\" + chunkPlan.length);\n\n// ---------- 3. 波次2:并行精读(Map) ----------\nphase(\"并行精读\");\nconst chunkPrompts = chunkPlan.map(function (c) {\n return [\n \"请深度精读以下内容块,输出结构化精读笔记。\",\n \"内容类型:\" + (kindNames[kind] || kind) + \" | 本块主题:\" + c.topic,\n \"精读侧重:\" + (kindFocus[kind] || kindFocus.web),\n \"用 read 工具读取 \" + tempFile + \" 第 \" + c.startLine + \" 到 \" + c.endLine + \" 行(offset=\" + c.startLine + \", limit=\" + (c.endLine - c.startLine + 1) + \"),然后输出:\",\n \"## 本块主题:\" + c.topic,\n \"### 要点(详细展开,保留数据/例子/论证)\",\n \"### 关键概念(术语+解释)\",\n \"### 关键引用(最多2句原文+位置)\" + (requireCitations ? \",必须标注页码/章节/段落位置\" : \"\"),\n \"### 批判观察(如有)\"\n ].join(\"\\n\");\n});\n\nconst deepReads = await parallel(chunkPrompts.map(function (p) {\n return function () { return agent(p, { label: \"精读块\", phase: \"并行精读\" }); };\n}));\nconst validReads = deepReads.filter(function (r) { return r && r.trim && r.trim().length > 0; });\n\n// 失败纪律:某块精读失败 = 可降级(跳过该块,标记缺口继续)\nconst failedChunks = chunkPlan.length - validReads.length;\nif (validReads.length === 0) return { ok: false, stage: \"read\", fatal: false, message: \"所有分块精读均失败\" };\nif (failedChunks > 0) log(\"警告:\" + failedChunks + \" 块精读失败,标记缺口继续成稿\");\nlog(\"精读完成:\" + validReads.length + \"/\" + chunkPlan.length);\n\n// ---------- 4. 波次3:成稿 + 内嵌质量自检(Reduce) ----------\nphase(\"汇总成稿\");\nconst today = new Date().toISOString().slice(0, 10);\nconst finalTitle = metadata.title || \"未命名内容\";\nconst safeTitle = finalTitle.replace(/[\\\\/:*?\"<>|]/g, \" \").trim();\nconst mdPath = outputDir.replace(/\\\\$/, \"\") + \"/\" + safeTitle + \".md\";\n\nconst extraSections = fastMode ? [] : [\n \"\", \"## 5. 批判性分析(论证强处/弱处/可质疑之处)\", \"\",\n \"## 6. 个人思考与应用(与已有知识联系、如何应用、行动要点)\", \"\",\n \"## 7. 延伸阅读(相关主题/作者其他作品/参考文献)\", \"\"\n];\n\nconst draftPrompt = [\n \"将以下多块精读结果整合为完整 Obsidian 精读笔记,只输出 Markdown(含 YAML frontmatter)。\",\n \"模板:\", \"---\",\n 'title: \"《' + finalTitle + '》深度精读笔记\"',\n \"aliases: [\" + JSON.stringify(finalTitle) + \"]\",\n \"author: \" + (metadata.author || \"\"),\n \"year: \" + (metadata.year || \"\"),\n \"type: \" + kind,\n \"url: \" + (metadata.url || input),\n \"tags: [精读, \" + (kindNames[kind] || kind) + \"]\",\n \"created: \" + today,\n \"status: 已完成\",\n \"---\",\n \"# 《\" + finalTitle + \"》深度精读笔记\", \"\",\n \"> 一句话核心概括。\", \"\",\n \"## 1. 概述(约500字:主题/背景/核心问题/总体结论)\", \"\",\n \"## 2. 结构拆解(按主题组织,含详细要点;关键结论标注原文出处)\", \"\",\n \"## 3. 关键概念与术语表(表格)\", \"\",\n \"## 4. 关键引用(仅关键结论\" + (requireCitations ? \",必须标注页码/章节/段落\" : \"\") + \")\",\n].concat(extraSections, [\n \"---\",\n \"*笔记整理于 \" + today + \"*\", \"\",\n \"硬性要求:总篇幅≥\" + minWords + \"字;语言与原文一致;术语统一;引用必须真实;视频\" + (includeTimestamps ? \"标注时间戳\" : \"不标注时间戳\") + \"。\",\n \"输出前自检:1)覆盖所有块 2)frontmatter完整 3)各小节齐全 4)引用真实 5)篇幅达标 6)语言一致。\",\n \"\",\n \"各块精读结果:\",\n validReads.map(function (r, i) { return \"\\\\n===== 块 \" + (i + 1) + \" =====\\\\n\" + r; }).join(\"\\\\n\")\n]).join(\"\\n\");\n\nlet draft = await agent(draftPrompt, { label: \"汇总成稿\", phase: \"汇总成稿\" });\nif (!draft || !draft.trim) return { ok: false, stage: \"draft\", fatal: false, message: \"成稿失败\" };\n\n// ---------- 5. 质量校验(轻量重试) ----------\nphase(\"质量校验\");\nlet verified = draft;\nlet passed = false;\nlet issues = [];\nfor (let round = 0; round <= maxRetries; round++) {\n const checkPrompt = [\n \"请对总结笔记做质量校验,逐项检查并输出 JSON:\",\n \"1.coverage 2.citationsReal 3.terminologyConsistent 4.formatComplete 5.lengthMet(≥\" + minWords + \"字) 6.languageConsistent\",\n \"输出:{ \\\"pass\\\": true/false, \\\"issues\\\": [\\\"问题\\\"] },只输出 JSON。\",\n \"笔记:\", verified\n ].join(\"\\n\");\n const checkSchema = { type: \"object\", properties: { pass: { type: \"boolean\" }, issues: { type: \"array\", items: { type: \"string\" } } }, required: [\"pass\",\"issues\"], additionalProperties: false };\n const check = await agent(checkPrompt, { schema: checkSchema, label: \"质量校验\", phase: \"质量校验\" });\n passed = !!(check && check.pass);\n issues = (check && check.issues) || [];\n log(\"校验:\" + (passed ? \"通过\" : \"发现问题 \" + issues.length + \" 条\"));\n if (passed) break;\n if (round < maxRetries) {\n const fixPrompt = [\n \"修正以下笔记以解决校验问题,输出完整修正版:\",\n issues.map(function (s) { return \"- \" + s; }).join(\"\\n\"),\n \"当前笔记:\", verified\n ].join(\"\\n\");\n verified = await agent(fixPrompt, { label: \"修正笔记\", phase: \"质量校验\" });\n }\n}\n\n// ---------- 结果 ----------\nreturn {\n ok: true, kind: kind, title: finalTitle,\n filePath: mdPath, chunksCount: chunkPlan.length, fastMode: fastMode,\n qualityPassed: passed, qualityIssues: issues,\n failedChunks: failedChunks,\n note: verified\n};";
|
|
38
|
+
const script = "// ========== deep-read-summarize v3.0 ==========\n// 3 波次:解析器获取分块 → 并行精读 → 成稿+质量校验\n// 特性:解析器插件化 / 失败纪律(fatal vs 降级) / 幂等 / 成本控制\n\n// ---------- 0. 配置校验(fatal 级错误:必须终止) ----------\nconst input = (args && args.input || \"\").trim();\nif (!input) throw new Error(\"[FATAL] args.input 缺失:请输入内容链接或文件路径\");\nconst forcedType = ((args && args.type) || \"auto\").toLowerCase();\nif ([\"auto\",\"book\",\"paper\",\"video\",\"web\"].indexOf(forcedType) === -1)\n throw new Error(\"[FATAL] args.type 非法:必须为 auto|book|paper|video|web\");\nconst options = (args && args.options) || {};\nconst minWords = options.minWords || 2500;\nconst fastMode = !!options.fastMode;\nconst maxChunks = Math.max(1, Math.min(options.maxChunks || 6, 12));\nconst requireCitations = options.requireCitations !== false;\nconst includeTimestamps = !!options.includeTimestamps;\nconst maxRetries = options.maxRetries || 1;\nconst cacheEnabled = options.cache !== false;\nconst outputDir = options.outputDir || \"./output\";\nconst tempDir = options.tempDir || \"./.tmp\";\nconst tempFile = tempDir.replace(/[\\/]+$/, \"\") + \"/input.txt\";\n\n// ---------- 0.5 幂等检查:输入指纹缓存 ----------\nlet cacheHit = false;\nif (cacheEnabled && Array.isArray(args._processedKeys)) {\n const fpKey = input.trim().slice(0, 300);\n cacheHit = args._processedKeys.indexOf(fpKey) !== -1;\n if (cacheHit) log(\"幂等命中:该输入已处理过,跳过\");\n}\nif (cacheHit) {\n return { ok: true, cached: true, kind: forcedType, title: (args && args.title) || \"cached\", filePath: \"\", chunksCount: 0, fastMode: fastMode, qualityPassed: true, qualityIssues: [], failedChunks: 0, note: \"\" };\n}\n\n// ---------- 1. 解析器选择(由 args._parsers 注入) ----------\nphase(\"解析器选择\");\nconst parsers = args._parsers || {};\nconst kindNames = { book: \"书籍\", paper: \"学术论文\", video: \"视频\", web: \"网页/文章\" };\nconst kindFocus = {\n book: \"按章节主题展开核心论点、论证、关键例子、章节间关系,收集关键概念与可引用原文(含页码/章节)\",\n paper: \"四维精读:1)研究问题与方法 2)结论与贡献 3)局限与批判 4)与同类工作比较;收集术语与可引用原文(含章节/段落)\",\n video: \"按主题脉络组织,提取核心论点、证据、例子、可操作建议,收集关键概念与主讲人原话\" + (includeTimestamps ? \"(含时间戳定位)\" : \"(不标时间戳)\"),\n web: \"提取核心论点、结构拆解、关键概念、可引用原文\"\n};\n\n// ---------- 2. 波次1:解析器获取 + 分块 ----------\nphase(\"获取与分块\");\nconst fetchPrompt = (parsers[forcedType !== \"auto\" ? forcedType : \"web\"] || parsers.web || { buildPrompt: function() { return [\"未知解析器\"]; } })\n .buildPrompt(input, { tempDir: tempDir, tempFile: tempFile, maxChunks: maxChunks });\nconst fetchSchema = {\n type: \"object\",\n properties: {\n kind: { type: \"string\" }, saved: { type: \"boolean\" }, message: { type: \"string\" }, totalLines: { type: \"number\" },\n metadata: { type: \"object\", properties: { title: { type: \"string\" }, author: { type: \"string\" }, year: { type: \"string\" }, language: { type: \"string\" } }, required: [], additionalProperties: true },\n chunkPlan: {\n type: \"array\",\n items: { type: \"object\", properties: { id: { type: \"number\" }, topic: { type: \"string\" }, startLine: { type: \"number\" }, endLine: { type: \"number\" } }, required: [\"id\",\"topic\",\"startLine\",\"endLine\"], additionalProperties: false }\n }\n },\n required: [\"saved\"], additionalProperties: false\n};\n\nconst fetched = await agent(fetchPrompt, { schema: fetchSchema, label: \"获取+分块\", phase: \"获取与分块\" });\n\n// 失败纪律:内容获取失败 = 可降级错误(返回失败结果,不抛 fatal)\nif (!fetched || !fetched.saved) {\n return { ok: false, stage: \"fetch\", fatal: false, message: (fetched && fetched.message) || \"获取失败(可降级:检查输入或手动提供文本)\" };\n}\nlet kind = (fetched.kind && forcedType === \"auto\") ? fetched.kind : forcedType;\nconst chunkPlan = (fetched && fetched.chunkPlan) || [];\nif (chunkPlan.length === 0) return { ok: false, stage: \"fetch\", fatal: false, message: \"分块计划为空\" };\nconst metadata = (fetched && fetched.metadata) || {};\nlog(\"类型:\" + kind + \" 行数:\" + fetched.totalLines + \" 块数:\" + chunkPlan.length);\n\n// ---------- 3. 波次2:并行精读(Map) ----------\nphase(\"并行精读\");\nconst chunkPrompts = chunkPlan.map(function (c) {\n return [\n \"请深度精读以下内容块,输出结构化精读笔记。\",\n \"内容类型:\" + (kindNames[kind] || kind) + \" | 本块主题:\" + c.topic,\n \"精读侧重:\" + (kindFocus[kind] || kindFocus.web),\n \"用 read 工具读取 \" + tempFile + \" 第 \" + c.startLine + \" 到 \" + c.endLine + \" 行(offset=\" + c.startLine + \", limit=\" + (c.endLine - c.startLine + 1) + \"),然后输出:\",\n \"## 本块主题:\" + c.topic,\n \"### 要点(详细展开,保留数据/例子/论证)\",\n \"### 关键概念(术语+解释)\",\n \"### 关键引用(最多2句原文+位置)\" + (requireCitations ? \",必须标注页码/章节/段落位置\" : \"\"),\n \"### 批判观察(如有)\"\n ].join(\"\\n\");\n});\n\nconst deepReads = await parallel(chunkPrompts.map(function (p) {\n return function () { return agent(p, { label: \"精读块\", phase: \"并行精读\" }); };\n}));\nconst validReads = deepReads.filter(function (r) { return r && r.trim && r.trim().length > 0; });\n\n// 失败纪律:某块精读失败 = 可降级(跳过该块,标记缺口继续)\nconst failedChunks = chunkPlan.length - validReads.length;\nif (validReads.length === 0) return { ok: false, stage: \"read\", fatal: false, message: \"所有分块精读均失败\" };\nif (failedChunks > 0) log(\"警告:\" + failedChunks + \" 块精读失败,标记缺口继续成稿\");\nlog(\"精读完成:\" + validReads.length + \"/\" + chunkPlan.length);\n\n// ---------- 4. 波次3:成稿 + 内嵌质量自检(Reduce) ----------\nphase(\"汇总成稿\");\nconst today = new Date().toISOString().slice(0, 10);\nconst finalTitle = metadata.title || \"未命名内容\";\nconst safeTitle = finalTitle.replace(/[\\\\/:*?\"<>|]/g, \" \").trim();\nconst mdPath = outputDir.replace(/\\\\$/, \"\") + \"/\" + safeTitle + \".md\";\n\nconst extraSections = fastMode ? [] : [\n \"\", \"## 5. 批判性分析(论证强处/弱处/可质疑之处)\", \"\",\n \"## 6. 个人思考与应用(与已有知识联系、如何应用、行动要点)\", \"\",\n \"## 7. 延伸阅读(相关主题/作者其他作品/参考文献)\", \"\"\n];\n\nconst draftPrompt = [\n \"将以下多块精读结果整合为完整 Obsidian 精读笔记,只输出 Markdown(含 YAML frontmatter)。\",\n \"模板:\", \"---\",\n 'title: \"《' + finalTitle + '》深度精读笔记\"',\n \"aliases: [\" + JSON.stringify(finalTitle) + \"]\",\n \"author: \" + (metadata.author || \"\"),\n \"year: \" + (metadata.year || \"\"),\n \"type: \" + kind,\n \"url: \" + (metadata.url || input),\n \"tags: [精读, \" + (kindNames[kind] || kind) + \"]\",\n \"created: \" + today,\n \"status: 已完成\",\n \"---\",\n \"# 《\" + finalTitle + \"》深度精读笔记\", \"\",\n \"> 一句话核心概括。\", \"\",\n \"## 1. 概述(约500字:主题/背景/核心问题/总体结论)\", \"\",\n \"## 2. 结构拆解(按主题组织,含详细要点;关键结论标注原文出处)\", \"\",\n \"## 3. 关键概念与术语表(表格)\", \"\",\n \"## 4. 关键引用(仅关键结论\" + (requireCitations ? \",必须标注页码/章节/段落\" : \"\") + \")\",\n].concat(extraSections, [\n \"---\",\n \"*笔记整理于 \" + today + \"*\", \"\",\n \"硬性要求:总篇幅≥\" + minWords + \"字;语言与原文一致;术语统一;引用必须真实;视频\" + (includeTimestamps ? \"标注时间戳\" : \"不标注时间戳\") + \"。\",\n \"输出前自检:1)覆盖所有块 2)frontmatter完整 3)各小节齐全 4)引用真实 5)篇幅达标 6)语言一致。\",\n \"\",\n \"各块精读结果:\",\n validReads.map(function (r, i) { return \"\\\\n===== 块 \" + (i + 1) + \" =====\\\\n\" + r; }).join(\"\\\\n\")\n]).join(\"\\n\");\n\nlet draft = await agent(draftPrompt, { label: \"汇总成稿\", phase: \"汇总成稿\" });\nif (!draft || !draft.trim) return { ok: false, stage: \"draft\", fatal: false, message: \"成稿失败\" };\n\n// ---------- 5. 质量校验(轻量重试) ----------\nphase(\"质量校验\");\nlet verified = draft;\nlet passed = false;\nlet issues = [];\nfor (let round = 0; round <= maxRetries; round++) {\n const checkPrompt = [\n \"请对总结笔记做质量校验,逐项检查并输出 JSON:\",\n \"1.coverage 2.citationsReal 3.terminologyConsistent 4.formatComplete 5.lengthMet(≥\" + minWords + \"字) 6.languageConsistent\",\n \"输出:{ \\\"pass\\\": true/false, \\\"issues\\\": [\\\"问题\\\"] },只输出 JSON。\",\n \"笔记:\", verified\n ].join(\"\\n\");\n const checkSchema = { type: \"object\", properties: { pass: { type: \"boolean\" }, issues: { type: \"array\", items: { type: \"string\" } } }, required: [\"pass\",\"issues\"], additionalProperties: false };\n const check = await agent(checkPrompt, { schema: checkSchema, label: \"质量校验\", phase: \"质量校验\" });\n passed = !!(check && check.pass);\n issues = (check && check.issues) || [];\n log(\"校验:\" + (passed ? \"通过\" : \"发现问题 \" + issues.length + \" 条\"));\n if (passed) break;\n if (round < maxRetries) {\n const fixPrompt = [\n \"修正以下笔记以解决校验问题,输出完整修正版:\",\n issues.map(function (s) { return \"- \" + s; }).join(\"\\n\"),\n \"当前笔记:\", verified\n ].join(\"\\n\");\n verified = await agent(fixPrompt, { label: \"修正笔记\", phase: \"质量校验\" });\n }\n}\n\n// ---------- 结果 ----------\nreturn {\n ok: true, kind: kind, title: finalTitle,\n filePath: mdPath, chunksCount: chunkPlan.length, fastMode: fastMode,\n qualityPassed: passed, qualityIssues: issues,\n failedChunks: failedChunks,\n note: verified\n};";
|
|
39
39
|
|
|
40
40
|
// ===== parsers 注册表(运行时注入,实现插件化)=====
|
|
41
41
|
const parsers = require("./parsers");
|