dsh-memory-engine 0.7.0 → 0.7.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +51 -3
- package/lib/auto-import.js +162 -10
- package/lib/automation-config.js +15 -2
- package/lib/client.js +19 -1
- package/lib/client.src.js +19 -1
- package/lib/config-write.js +12 -1
- package/lib/index.js +2 -1
- package/lib/noise.js +47 -0
- package/lib/onboard.js +19 -4
- package/lib/routes.js +18 -4
- package/lib/session-filter.js +47 -0
- package/package.json +2 -1
- package/scripts/cleanup-imports.mjs +116 -0
package/README.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
|
-
# DSH 记忆引擎
|
|
1
|
+
# DSH 记忆引擎
|
|
2
2
|
|
|
3
3
|
DSH 插件的本地记忆引擎,带神经网络可视化。它让 agent 保存、召回和管理跨会话记忆,也能把你已有的记忆文件导进来。
|
|
4
4
|
|
|
5
|
-
当前版本:0.7.
|
|
5
|
+
当前版本:0.7.5。Node ≥22,零新增依赖。自动记忆默认关闭,由用户在图谱头部或设置页手动打开;开启后每轮对话会额外调用一次模型,可能产生费用。自动导入默认只收聊天会话的用户消息,支持来源白名单、工作指令过滤、流程话过滤、规则分类、错误原因、一次性重扫和账本清理。
|
|
6
6
|
|
|
7
7
|
## 从 0.6.1 升级到 0.7.0 的变化
|
|
8
8
|
|
|
@@ -136,7 +136,7 @@ peerDependencies 已放宽为 >=0.1.0-rc.8 <0.3.0 并标为 optional——宿主
|
|
|
136
136
|
|
|
137
137
|
JSON 成功返回 `{ "ok": true, "data": ... }`,失败返回 `{ "ok": false, "error": "说明" }`。既有接口返回结构保留;stats/list/export 仍只统计实际记忆。接口仅接受本机 Host,严格限制路径,拒绝路径穿越;不允许的方法返回 405。只读接口不接收请求体,PUT 和 POST 只接收 JSON 对象,声明大小和实际分块累计大小都限制在 1MB。
|
|
138
138
|
|
|
139
|
-
配置 PUT 使用嵌套对象,例如 `{ "autoMemory": { "enabled": false } }`。只允许 `autoMemory.enabled`、`autoImport.enabled`、`autoMemory.maxPerTurn`(1~5 整数)、`autoMemory.dailyCap`(0~500 整数)、`spinSpeed`(0.05~1.5 数字)、`showGuideOnEmpty` 和 `hideButton
|
|
139
|
+
配置 PUT 使用嵌套对象,例如 `{ "autoMemory": { "enabled": false } }`。只允许 `autoMemory.enabled`、`autoImport.enabled`、`autoMemory.maxPerTurn`(1~5 整数)、`autoMemory.dailyCap`(0~500 整数)、`spinSpeed`(0.05~1.5 数字)、`showGuideOnEmpty` 和 `hideButton`;所有启用/显示开关只接受布尔值。0.7.2 新增 `autoImport.importAssistant`(布尔值)、`autoImport.workMarkers` 和 `autoImport.sessionFilter`,校验规则见下节;sessionFilter 的局部更新保留未提供的旧字段。任何未知字段或类型错误会整次返回 400,不改任何字段。角色卡、来源路径、模型等不能通过此接口修改;GET 的公开字段保持原样。
|
|
140
140
|
|
|
141
141
|
配置写入前生成 `config.json.bak-<时间戳>`,同毫秒加序号,保留最近五份;先写临时文件再 rename。备份或写入失败保留原配置。首次成功启用时,服务端自行保存 `showAutoMemoryCostHint: false`,不向客户端额外开放这个字段的写权限。关闭会清空当前收集的对话、取消排队和进行中的抽取,重新开启时重置冷却和失败退避;自动导入独立运行,若要冻结所有自动入库,需同时关闭它。
|
|
142
142
|
|
|
@@ -176,6 +176,10 @@ memory_map(out="D:\\memory-map.html")
|
|
|
176
176
|
"enabled": true,
|
|
177
177
|
"importAssistant": false,
|
|
178
178
|
"noisePatterns": [],
|
|
179
|
+
"workMarkers": [],
|
|
180
|
+
"sessionFilter": { "mode": "include", "cwdContains": ["state/agents", "qq-bridge", "astrbot", "nonebot", "koishi"], "maxToolCalls": 0 },
|
|
181
|
+
"ledgerRetentionDays": { "skip": 30, "session": 120 },
|
|
182
|
+
"ledgerMaxEntries": 2000,
|
|
179
183
|
"sources": [{ "type": "dsh-sessions" }],
|
|
180
184
|
"intervalMinutes": 60,
|
|
181
185
|
"maxPerRun": 200
|
|
@@ -191,6 +195,49 @@ memory_map(out="D:\\memory-map.html")
|
|
|
191
195
|
|
|
192
196
|
会话导入仅通过 `sessionQuery.listSessions/readSession`,不直接读取或解压会话文件,不激活会话。默认 `importAssistant=false`,只收 `user/message`;改为 true 才额外收 `assistant/message`,两者都过滤流程话。此开关不影响自定义路径中的日记等正文。保留事件发生时间;连续重复、寒暄、纯代码块和敏感词仍被过滤,超过 2000 字的消息拆段保存。每会话每轮最多 200 条,全局默认最多 200 条,批间让出 20 毫秒。`import-ledger.json` 保存 seq、文件 size/mtime、JSONL 行数/字节位置及未完成批次,超限下轮继续。重复导入不强化旧记忆。
|
|
193
197
|
|
|
198
|
+
0.7.2 在读取消息和写入候选两个位置检查 `importAssistant`。运行中关闭后,尚未写入的 assistant 候选也会被拦住;运行中开启从下一轮生效。已有 assistant 存量不会自动删除。
|
|
199
|
+
|
|
200
|
+
`autoImport.sessionFilter` 只影响 dsh-sessions 来源,读取事件后先判断整段会话,再收集消息:
|
|
201
|
+
|
|
202
|
+
0.7.5 默认使用 `include` 来源白名单。常见聊天目录为 `state/agents`、`qq-bridge`、`astrbot`、`nonebot` 和 `koishi`;这些默认提示按完整目录段匹配,避免把 `qq-bridge-probe` 当成聊天目录。自定义 `cwdContains` 仍可填写路径片段。没有会话命中时,设置页会提示并提供「改用宽松模式」按钮;点击后保存 `mode: auto`,立即重新尝试此前跳过的会话。普通 DSH 聊天如不在这些目录,可在 `config.json` 设置 `{ "autoImport": { "sessionFilter": { "mode": "auto" } } }`;需要收全部会话则设 `mode: all`。聊天机器人可配置 `{ "mode": "include", "cwdContains": ["state/agents", "qq-bridge"] }`。
|
|
203
|
+
|
|
204
|
+
0.7.4 在自动导入时直接丢弃桥接聊天记录、运行时快照、`<message source_id=...>`、会话令牌和 Runtime/Context 包装文本。`dsh-sessions` 消息默认至少 12 个 Unicode 字符,可在 `config.json` 的 `autoImport.minChars` 设为 4~200;手动 `memory_import` 和 QQ 文件来源不受此门槛影响。自动导入的长文本若前 50 个字符与已存条目相同,只保留先入库的一条。配置 `personaFile` 后,导入器还会跳过与角色卡正文高度重合的会话文本。在 auto 会话模式下,工作区中的无工具会话若用户消息命中至少两组工作词,也会整段跳过。这些规则只影响新导入,不删除已有记忆。
|
|
205
|
+
|
|
206
|
+
| mode | 行为 |
|
|
207
|
+
|---|---|
|
|
208
|
+
| `auto`(默认) | 至少 2 条 user/message;工作工具调用次数不超过 maxToolCalls(默认 0) |
|
|
209
|
+
| `include` | cwd 包含 cwdContains 中任一路径片段;忽略大小写并统一斜杠。空列表全部排除,不附加 auto 的工具条件 |
|
|
210
|
+
| `all` | 不限制会话类型或消息条数,角色和内容过滤仍然生效 |
|
|
211
|
+
|
|
212
|
+
工作工具包括 `pwsh/bash/read/write/edit/grep/glob/subagent/workflow/notebook`,也识别带命名空间的名称;`qq_*` 和记忆工具不计入。maxToolCalls 只能为 0~10000 的整数;cwdContains 最多 50 个非空片段,每个最长 1000 字符,未知字段会被拒绝。QQ 会话如果也调用过工作工具,auto 同样跳过,可按实际需要选择 include。
|
|
213
|
+
|
|
214
|
+
跳过结果保存在 `import-ledger.json` 的 `session:<id>.skip`,含 sessionId、模式和原因:`work-tools`、`cwd-excluded` 或 `too-few-user-messages`。前两种在同一规则下直接复用账本,日志汇总跳过的工作会话及工具;不足两条消息的会话下轮再检查,以免永远漏掉后续聊天。修改过滤规则或使用 rescanFrom/rescanAll 会重新判断。跳过不会删除已经入库的记忆,也不把该会话的旧水位推进到结尾。
|
|
215
|
+
|
|
216
|
+
工作指令单独按六组特征计数:记忆工具名、插件配置名、任务/验收词、构建发布词、版本号、录屏演示词。通常命中至少两组才拦截,单独说“npm 这个包挺好用的”仍保留。0.7.3 收紧一个弱组合:只命中构建/软件词和版本号两组时保留,避免将“新版已发布,朋友可安装”误判为工作指令;再命中验收、工具调用等第三组时仍拦截。`autoImport.workMarkers` 可追加最多 20 个正则源码字符串(每个最长 300 字符,使用 iu 标志);无效正则拒绝保存,手工写入无效配置则停用自动导入并提示。会话消息在拆段前检查完整正文,路径来源在写入前也检查。旧流程话规则独立保留。
|
|
217
|
+
|
|
218
|
+
### 导入账本清理
|
|
219
|
+
|
|
220
|
+
每轮自动导入结束时检查一次 `import-ledger.json`,只有实际删除记录或过期 skip 缓存时才调用已有 automation-state 原子写;无变化时不写盘。默认无水位 skip 缓存按 skip.at 超过 30 天删除,会话水位按 importedAt 超过 120 天未更新才删除。重新判断 skip 不会延长旧水位的寿命;不认识对应时间的旧记录保留。
|
|
221
|
+
|
|
222
|
+
同一记录同时包含水位和 skip 时,水位按 120 天保留,过期 30 天的 skip 字段可以单独清除,下轮重新判断会话。文件来源的字节/行数进度、未完成分段、重扫任务均保留。导入中刚更新的水位不会因清理而丢失。
|
|
223
|
+
|
|
224
|
+
条目数超过 `ledgerMaxEntries`(默认 2000)时,按更新时间从旧到新淘汰**没有水位**的 skip 缓存,目标为上限的 75%(默认 1500)。容量压力下这种可重新判断的缓存可以提前淘汰;仍不动近期水位。如果全部都是受保护的水位或文件进度,允许暂时超过上限并告警,不能为凑数量造成重新导入。
|
|
225
|
+
|
|
226
|
+
在 config.json 的 autoImport 中设置 `ledgerRetentionDays` 和 `ledgerMaxEntries`;本版没有增加配置 PUT 白名单或界面入口。skip 天数范围为 30~36500,session 为 120~36500,条目上限为 1~100000,均为整数。非法配置会中止本轮导入并记录原因,不会按错误期限清库。日志分别报告过期删除、单独清除 skip 字段、容量淘汰及清理前后数量。近期账本即使有很多 skip,也不会为了显示“条数下降”提前按过期规则删除。
|
|
227
|
+
|
|
228
|
+
### 存量清理(逐步确认)
|
|
229
|
+
|
|
230
|
+
`scripts/cleanup-imports.mjs` 默认只预览。依次运行 assistant、work、dedupe,每一步先看完整 JSON 清单和终端前 10 条样本,获得用户确认后才执行。清单包含库指纹和确认 token,库或规则变化时必须重新预览。执行前关闭 DSH,避免另一个进程同时写库。
|
|
231
|
+
|
|
232
|
+
```sh
|
|
233
|
+
node scripts/cleanup-imports.mjs --home <DSH目录> --stage assistant --out <assistant-preview.json>
|
|
234
|
+
node scripts/cleanup-imports.mjs --home <DSH目录> --preview <assistant-preview.json> --confirm <该清单token>
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
第一步完成后对 work 重复预览和确认,再对 dedupe 重复;不能越过尚未完成的前一步。删除调用现成 store.backup()/write(),合并调用 store.dedupe({ dry_run: false }),默认阈值 0.7。每一步自动备份、写后核对数量;备份失败立即中止。QQ 来源匹配项保护保留并列在 protected 中。去重预览如果会减少 QQ 来源条数,blocked 为 true,执行会拒绝;先审阅冲突组,不自动更改来源或去重算法。无变更时不写库。
|
|
238
|
+
|
|
239
|
+
需要保留 QQ 冲突组、只合并其余组时,在 dedupe 预览命令增加 `--protect-qq`。清单的 protected 列出整个冲突组的所有条目,确认后这些条目原样保留;其余范围仍调用现成 store.dedupe,备份保存完整原库,最终写入仍走 store.write。必须审阅这份新清单后再确认,不能沿用未保护的合并清单。
|
|
240
|
+
|
|
194
241
|
流程话过滤采用固定词表,可在 `autoImport.noisePatterns` 追加字面短语。命中两个不同特征即拦截;只命中一个时,仅拦截不足 150 字的文本,长内容保留。开头 12 字内出现「已回:」「已回复:」「已发送:」「已处理:」「已确认:」「已读:」「已标记:」「回过了:」「发过了:」的工作汇报,无论长度都拦截;中间提到「他说已回复了」不算前缀。自动记忆的抽取结果也通过同一过滤器。规则只作用于后续入库,不追删旧条目。
|
|
195
242
|
|
|
196
243
|
新导入的会话按顺序粗分:日期命名的 Markdown 日记 → 日记;价格或解释 → 知识;过敏、忌口或偏好 → 印象;约定或计划 → 事件;有日期的 Markdown 标题 → 日记;其余 → 对话。印象默认 heat=1,其余为 2。自定义来源已有明确分类时保留原分类,日期日记优先。**导入的分类是启发式猜测,可能分错,可以在管理面板手动编辑。** 过滤和分类都不调用 LLM。
|
|
@@ -373,3 +420,4 @@ node --test test/*.test.mjs
|
|
|
373
420
|
## 许可
|
|
374
421
|
|
|
375
422
|
MIT。可以商用、可以修改、可以再发布,保留版权声明即可(见 [LICENSE](./LICENSE))。
|
|
423
|
+
|
package/lib/auto-import.js
CHANGED
|
@@ -2,16 +2,90 @@ import fs from 'node:fs/promises';
|
|
|
2
2
|
import path from 'node:path';
|
|
3
3
|
import { createHash, randomBytes } from 'node:crypto';
|
|
4
4
|
import { readConfig } from './config.js';
|
|
5
|
+
import { onConfigUpdate } from './config-write.js';
|
|
5
6
|
import { inferEmotion } from './decay.js';
|
|
6
7
|
import { findDuplicate, prepare } from './dedupe.js';
|
|
7
8
|
import { parseTime } from './import.js';
|
|
8
9
|
import { eventText, usefulText } from './session-text.js';
|
|
9
10
|
import { parseSourceFile, readJsonl, sourceFiles, verifySource } from './auto-sources.js';
|
|
10
11
|
import { backupForAutomation, readState, service, warningLogger, writeState } from './automation-state.js';
|
|
11
|
-
import { isNoise } from './noise.js';
|
|
12
|
+
import { isInjected, isNoise, isWorkRecord, workMatches } from './noise.js';
|
|
12
13
|
import { classifyImport } from './classify.js';
|
|
14
|
+
import { assessSession, normalizeSessionFilter, sessionFilterKey } from './session-filter.js';
|
|
13
15
|
|
|
14
16
|
const digest = text => createHash('sha256').update(text).digest('hex');
|
|
17
|
+
const day = 86400000;
|
|
18
|
+
|
|
19
|
+
export function importMinChars(value = 12) {
|
|
20
|
+
if (!Number.isSafeInteger(value) || value < 4 || value > 200) throw new Error('autoImport.minChars 必须是 4~200 的整数');
|
|
21
|
+
return value;
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
const longPrefix = text => Array.from(text).length >= 50 ? Array.from(text).slice(0, 50).join('') : null;
|
|
25
|
+
|
|
26
|
+
export function overlapsPersona(text, persona) {
|
|
27
|
+
if (typeof persona !== 'string' || Array.from(text).length < 100) return false;
|
|
28
|
+
const content = text.replace(/\s+/gu, '');
|
|
29
|
+
const reference = persona.replace(/\s+/gu, '');
|
|
30
|
+
const characters = Array.from(content);
|
|
31
|
+
if (characters.length < 100) return false;
|
|
32
|
+
let matches = 0;
|
|
33
|
+
let checked = 0;
|
|
34
|
+
for (let offset = 0; offset + 50 <= characters.length; offset += 50) {
|
|
35
|
+
checked++;
|
|
36
|
+
if (reference.includes(characters.slice(offset, offset + 50).join(''))) matches++;
|
|
37
|
+
}
|
|
38
|
+
return checked > 0 && matches / checked >= 0.6;
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
export function normalizeLedgerPolicy(settings = {}) {
|
|
42
|
+
const retention = settings.ledgerRetentionDays === undefined ? {} : settings.ledgerRetentionDays;
|
|
43
|
+
if (!retention || typeof retention !== 'object' || Array.isArray(retention)) throw new Error('ledgerRetentionDays 必须是对象');
|
|
44
|
+
for (const key of Object.keys(retention)) if (!['skip', 'session'].includes(key)) throw new Error(`ledgerRetentionDays 不支持 ${key}`);
|
|
45
|
+
const policy = { skip: retention.skip === undefined ? 30 : retention.skip, session: retention.session === undefined ? 120 : retention.session,
|
|
46
|
+
maximum: settings.ledgerMaxEntries === undefined ? 2000 : settings.ledgerMaxEntries };
|
|
47
|
+
for (const [key, minimum, maximum] of [['skip', 30, 36500], ['session', 120, 36500], ['maximum', 1, 100000]]) {
|
|
48
|
+
if (!Number.isSafeInteger(policy[key]) || policy[key] < minimum || policy[key] > maximum) throw new Error(`账本 ${key} 必须为 ${minimum}~${maximum} 的整数`);
|
|
49
|
+
}
|
|
50
|
+
return policy;
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
export function cleanupLedger(ledger, settings = {}, now = Date.now()) {
|
|
54
|
+
const policy = normalizeLedgerPolicy(settings);
|
|
55
|
+
const result = { expired: 0, expiredSkipCaches: 0, evicted: 0, before: Object.keys(ledger).length, after: 0, overLimit: false };
|
|
56
|
+
const candidates = [];
|
|
57
|
+
for (const [key, entry] of Object.entries(ledger)) {
|
|
58
|
+
if (!key.startsWith('session:') || !entry || typeof entry !== 'object' || entry.pending || entry.rescan) continue;
|
|
59
|
+
const watermark = Number.isSafeInteger(entry.lastSeq) && entry.lastSeq >= 0;
|
|
60
|
+
const updated = entry.skip && !watermark ? entry.skip.at : entry.importedAt;
|
|
61
|
+
if (!Number.isFinite(updated) || updated < 0) continue;
|
|
62
|
+
const lifetime = entry.skip && !watermark ? policy.skip : policy.session;
|
|
63
|
+
if (now - updated > lifetime * day) {
|
|
64
|
+
delete ledger[key];
|
|
65
|
+
result.expired++;
|
|
66
|
+
continue;
|
|
67
|
+
}
|
|
68
|
+
if (watermark && entry.skip && Number.isFinite(entry.skip.at) && now - entry.skip.at > policy.skip * day) {
|
|
69
|
+
delete entry.skip;
|
|
70
|
+
result.expiredSkipCaches++;
|
|
71
|
+
}
|
|
72
|
+
if (entry.skip && !watermark) candidates.push({ key, updated });
|
|
73
|
+
}
|
|
74
|
+
let remaining = Object.keys(ledger).length;
|
|
75
|
+
if (remaining > policy.maximum) {
|
|
76
|
+
const target = Math.max(1, Math.floor(policy.maximum * 0.75));
|
|
77
|
+
candidates.sort((left, right) => left.updated - right.updated || left.key.localeCompare(right.key));
|
|
78
|
+
for (const entry of candidates) {
|
|
79
|
+
if (remaining <= target) break;
|
|
80
|
+
delete ledger[entry.key];
|
|
81
|
+
remaining--;
|
|
82
|
+
result.evicted++;
|
|
83
|
+
}
|
|
84
|
+
}
|
|
85
|
+
result.after = remaining;
|
|
86
|
+
result.overLimit = remaining > policy.maximum;
|
|
87
|
+
return result;
|
|
88
|
+
}
|
|
15
89
|
|
|
16
90
|
export function createAutoImport({ ctx, store, now = Date.now, warn = () => {}, getConfig,
|
|
17
91
|
sleep = milliseconds => new Promise(resolve => setTimeout(resolve, milliseconds)), startupDelayMs = 8000, schedule = true } = {}) {
|
|
@@ -27,6 +101,7 @@ export function createAutoImport({ ctx, store, now = Date.now, warn = () => {},
|
|
|
27
101
|
async function perform() {
|
|
28
102
|
const settings = config();
|
|
29
103
|
if (stopped || !settings.autoImport.enabled) return { added: 0, skipped: 0, processed: 0 };
|
|
104
|
+
const ledgerPolicy = normalizeLedgerPolicy(settings.autoImport);
|
|
30
105
|
const ledger = readState(ledgerFile, {});
|
|
31
106
|
if (!ledger || typeof ledger !== 'object' || Array.isArray(ledger)) throw new Error('import-ledger.json 必须是对象');
|
|
32
107
|
const request = settings.autoImport.rescanAll ? 'all' : settings.autoImport.rescanFrom?.trim();
|
|
@@ -45,22 +120,43 @@ export function createAutoImport({ ctx, store, now = Date.now, warn = () => {},
|
|
|
45
120
|
}
|
|
46
121
|
consumedRescan = request;
|
|
47
122
|
}
|
|
48
|
-
const totals = { added: 0, skipped: 0, processed: 0, noise: 0, assistant: 0
|
|
123
|
+
const totals = { added: 0, skipped: 0, processed: 0, noise: 0, assistant: 0, work: 0,
|
|
124
|
+
sessionsSkipped: 0, workSessions: 0, cachedSessions: 0 };
|
|
49
125
|
const maximum = settings.autoImport.maxPerRun;
|
|
126
|
+
const minimum = importMinChars(settings.autoImport.minChars);
|
|
127
|
+
let persona = null;
|
|
128
|
+
if (settings.personaFile) {
|
|
129
|
+
try { persona = await fs.readFile(settings.personaFile, 'utf8'); }
|
|
130
|
+
catch (error) { warning(`无法读取角色卡 ${settings.personaFile}:${error.message}`, { once: true }); }
|
|
131
|
+
}
|
|
50
132
|
const patterns = settings.autoMemory.excludePatterns;
|
|
133
|
+
const allowAssistant = () => settings.autoImport.importAssistant === true && config().autoImport.importAssistant === true;
|
|
134
|
+
const sessionFilter = normalizeSessionFilter(settings.autoImport.sessionFilter);
|
|
135
|
+
const skippedTools = new Set();
|
|
51
136
|
|
|
52
137
|
function commit(entries, key, cursor) {
|
|
53
138
|
if (stopped) return;
|
|
54
139
|
if (entries.length) {
|
|
55
140
|
const rows = store.read();
|
|
56
141
|
const prepared = rows.map(row => prepare(row.text));
|
|
142
|
+
const prefixes = new Set(rows.map(row => longPrefix(row.text)).filter(Boolean));
|
|
57
143
|
let added = 0;
|
|
58
144
|
for (const candidate of entries) {
|
|
145
|
+
if (candidate.origin?.source === 'dsh-sessions' && candidate.origin.role !== 'user' && !allowAssistant()) {
|
|
146
|
+
totals.assistant++; totals.skipped++; continue;
|
|
147
|
+
}
|
|
148
|
+
if (isInjected(candidate.text)) { totals.noise++; totals.skipped++; continue; }
|
|
149
|
+
if (candidate.origin?.source === 'dsh-sessions' && Array.from(candidate.text).length < minimum) { totals.skipped++; continue; }
|
|
150
|
+
if (candidate.origin?.source === 'dsh-sessions' && overlapsPersona(candidate.text, persona)) { totals.skipped++; continue; }
|
|
59
151
|
if (isNoise(candidate.text, settings.autoImport.noisePatterns)) { totals.noise++; totals.skipped++; continue; }
|
|
152
|
+
if (isWorkRecord(candidate.text, settings.autoImport.workMarkers)) { totals.work++; totals.skipped++; continue; }
|
|
153
|
+
const prefix = longPrefix(candidate.text);
|
|
154
|
+
if (prefix && prefixes.has(prefix)) { totals.skipped++; continue; }
|
|
60
155
|
if (findDuplicate(rows, candidate, prepared, 0.85)) { totals.skipped++; continue; }
|
|
61
156
|
const { category: _category, ...fields } = candidate;
|
|
62
157
|
rows.push({ ...fields, id: `m-${now()}-${randomBytes(3).toString('hex')}`, createdAt: now() });
|
|
63
158
|
prepared.push(prepare(candidate.text));
|
|
159
|
+
if (prefix) prefixes.add(prefix);
|
|
64
160
|
added++;
|
|
65
161
|
}
|
|
66
162
|
if (added) {
|
|
@@ -74,17 +170,51 @@ export function createAutoImport({ ctx, store, now = Date.now, warn = () => {},
|
|
|
74
170
|
writeState(ledgerFile, ledger);
|
|
75
171
|
}
|
|
76
172
|
|
|
77
|
-
async function importSession(query,
|
|
173
|
+
async function importSession(query, session) {
|
|
174
|
+
const id = session.header.id;
|
|
78
175
|
const key = `session:${id}`;
|
|
79
176
|
const saved = ledger[key] ?? { lastSeq: -1 };
|
|
80
177
|
if (!Number.isSafeInteger(saved.lastSeq) || saved.lastSeq < -1) throw new Error(`${key} 水位无效`);
|
|
178
|
+
const policy = sessionFilterKey(session.header, sessionFilter);
|
|
179
|
+
function skipped(reason, cached = false) {
|
|
180
|
+
totals.sessionsSkipped++;
|
|
181
|
+
if (cached) totals.cachedSessions++;
|
|
182
|
+
if (reason.reason === 'work-tools') {
|
|
183
|
+
totals.workSessions++;
|
|
184
|
+
for (const name of reason.tools) skippedTools.add(name);
|
|
185
|
+
}
|
|
186
|
+
}
|
|
187
|
+
if (saved.skip?.cacheable && saved.skip.policy === policy && !saved.rescan) { skipped(saved.skip, true); return; }
|
|
81
188
|
const read = await query.readSession(id);
|
|
82
189
|
if (stopped) return;
|
|
83
190
|
if (!Array.isArray(read?.events)) throw new Error('readSession 未返回 events 数组');
|
|
191
|
+
let decision = assessSession({ ...read.session, ...session.header }, read.events, sessionFilter);
|
|
192
|
+
if (decision.accepted && sessionFilter.mode === 'auto') {
|
|
193
|
+
const cwd = session.header.cwd ?? read.session?.cwd;
|
|
194
|
+
if (typeof cwd === 'string' && cwd.trim()) {
|
|
195
|
+
let workingDirectory = /(^|[\\/])default-workspace([\\/]|$)/iu.test(cwd);
|
|
196
|
+
if (!workingDirectory) {
|
|
197
|
+
try { workingDirectory = (await fs.stat(path.join(cwd, 'package.json'))).isFile(); }
|
|
198
|
+
catch {}
|
|
199
|
+
}
|
|
200
|
+
if (workingDirectory && read.events.some(event => event.type === 'user/message'
|
|
201
|
+
&& workMatches(eventText(event), settings.autoImport.workMarkers).length >= 2)) {
|
|
202
|
+
decision = { accepted: false, reason: 'work-message', cacheable: true };
|
|
203
|
+
}
|
|
204
|
+
}
|
|
205
|
+
}
|
|
206
|
+
if (!decision.accepted) {
|
|
207
|
+
skipped(decision);
|
|
208
|
+
ledger[key] = { ...saved, skip: { ...decision, sessionId: id, mode: sessionFilter.mode, policy, at: now() } };
|
|
209
|
+
delete ledger[key].rescan;
|
|
210
|
+
writeState(ledgerFile, ledger);
|
|
211
|
+
return;
|
|
212
|
+
}
|
|
84
213
|
let cursor = { ...saved };
|
|
214
|
+
delete cursor.skip;
|
|
85
215
|
let batch = [];
|
|
86
216
|
let processed = 0;
|
|
87
|
-
let dirty =
|
|
217
|
+
let dirty = !!saved.skip;
|
|
88
218
|
let scanned = 0;
|
|
89
219
|
async function flush() {
|
|
90
220
|
if (dirty) commit(batch, key, cursor);
|
|
@@ -96,19 +226,24 @@ export function createAutoImport({ ctx, store, now = Date.now, warn = () => {},
|
|
|
96
226
|
if (stopped) return;
|
|
97
227
|
if (++scanned % 200 === 0) await sleep(20);
|
|
98
228
|
if (!Number.isSafeInteger(event.seq) || event.seq <= cursor.lastSeq) continue;
|
|
99
|
-
|
|
229
|
+
const role = event.type === 'user/message' ? 'user' : event.type === 'assistant/message' ? 'assistant' : null;
|
|
230
|
+
if (!role || role === 'assistant' && !allowAssistant()) {
|
|
100
231
|
cursor.lastSeq = event.seq;
|
|
101
232
|
delete cursor.pending;
|
|
102
233
|
dirty = true;
|
|
103
|
-
totals.assistant++;
|
|
234
|
+
if (role === 'assistant') totals.assistant++;
|
|
104
235
|
continue;
|
|
105
236
|
}
|
|
106
237
|
const text = eventText(event);
|
|
107
238
|
const hash = text ? digest(text) : null;
|
|
108
239
|
const timestamp = parseTime(event.time);
|
|
109
240
|
const noise = isNoise(text, settings.autoImport.noisePatterns);
|
|
241
|
+
const work = isWorkRecord(text, settings.autoImport.workMarkers);
|
|
242
|
+
const injected = isInjected(text);
|
|
110
243
|
if (noise) totals.noise++;
|
|
111
|
-
if (
|
|
244
|
+
if (work) totals.work++;
|
|
245
|
+
if (!text || injected || Array.from(text).length < minimum || overlapsPersona(text, persona)
|
|
246
|
+
|| !usefulText(text) || noise || work || timestamp === null || hash === cursor.lastTextHash || patterns.some(pattern => text.includes(pattern))) {
|
|
112
247
|
cursor.lastSeq = event.seq;
|
|
113
248
|
delete cursor.pending;
|
|
114
249
|
if (hash) cursor.lastTextHash = hash;
|
|
@@ -122,7 +257,7 @@ export function createAutoImport({ ctx, store, now = Date.now, warn = () => {},
|
|
|
122
257
|
const part = characters.slice(offset, offset + 2000).join('');
|
|
123
258
|
const inferred = inferEmotion(part);
|
|
124
259
|
batch.push({ text: part, ...classifyImport(text), ts: timestamp, emotion: inferred.emotion, boosts: 0,
|
|
125
|
-
origin: { source: 'dsh-sessions', sessionId: id, seq: event.seq, offset, role
|
|
260
|
+
origin: { source: 'dsh-sessions', sessionId: id, seq: event.seq, offset, role } });
|
|
126
261
|
processed++;
|
|
127
262
|
offset += 2000;
|
|
128
263
|
if (offset >= characters.length) {
|
|
@@ -186,7 +321,7 @@ export function createAutoImport({ ctx, store, now = Date.now, warn = () => {},
|
|
|
186
321
|
for (const session of prioritized) {
|
|
187
322
|
if (stopped || totals.processed >= maximum) break;
|
|
188
323
|
if (typeof session.header?.id !== 'string') continue;
|
|
189
|
-
try { await importSession(query, session
|
|
324
|
+
try { await importSession(query, session); }
|
|
190
325
|
catch (error) { warning(`导入会话 ${session.header.id} 失败:${error.message}`); }
|
|
191
326
|
}
|
|
192
327
|
} else {
|
|
@@ -199,6 +334,20 @@ export function createAutoImport({ ctx, store, now = Date.now, warn = () => {},
|
|
|
199
334
|
}
|
|
200
335
|
} catch (error) { warning(`略过来源 ${source.path ?? source.type}:${error.message}`, { once: true }); }
|
|
201
336
|
}
|
|
337
|
+
if (totals.workSessions) {
|
|
338
|
+
try { warn(`跳过 ${totals.workSessions} 个工作会话(原因:用过 ${[...skippedTools].join('/')};账本跳过 ${totals.cachedSessions} 个会话)`); } catch {}
|
|
339
|
+
}
|
|
340
|
+
if (!stopped) {
|
|
341
|
+
const cleaned = cleanupLedger(ledger, settings.autoImport, now());
|
|
342
|
+
if (cleaned.expired || cleaned.expiredSkipCaches || cleaned.evicted) {
|
|
343
|
+
writeState(ledgerFile, ledger);
|
|
344
|
+
try { warn(`清理账本:删除 ${cleaned.expired} 条过期记录(skip ${ledgerPolicy.skip} 天 / session ${ledgerPolicy.session} 天),清除 ${cleaned.expiredSkipCaches} 条过期 skip 缓存,超限淘汰 ${cleaned.evicted} 条无水位缓存(${cleaned.before} → ${cleaned.after})`); } catch {}
|
|
345
|
+
}
|
|
346
|
+
if (cleaned.overLimit) {
|
|
347
|
+
try { warn(`账本仍有 ${cleaned.after} 条,超过上限 ${ledgerPolicy.maximum};保留近期水位、文件进度和未完成任务,未强制删除`); } catch {}
|
|
348
|
+
}
|
|
349
|
+
totals.ledgerCleanup = cleaned;
|
|
350
|
+
}
|
|
202
351
|
return totals;
|
|
203
352
|
}
|
|
204
353
|
|
|
@@ -222,6 +371,9 @@ export function createAutoImport({ ctx, store, now = Date.now, warn = () => {},
|
|
|
222
371
|
}, delay);
|
|
223
372
|
timer.unref?.();
|
|
224
373
|
}
|
|
374
|
+
const removeConfigListener = onConfigUpdate(home, changes => {
|
|
375
|
+
if (changes.autoImport?.sessionFilter?.mode) void (running ? running.finally(() => runNow()) : runNow());
|
|
376
|
+
});
|
|
225
377
|
if (schedule) next(startupDelayMs);
|
|
226
|
-
return { runNow, dispose() { stopped = true; clearTimeout(timer); } };
|
|
378
|
+
return { runNow, dispose() { stopped = true; clearTimeout(timer); removeConfigListener(); } };
|
|
227
379
|
}
|
package/lib/automation-config.js
CHANGED
|
@@ -1,7 +1,17 @@
|
|
|
1
|
+
import { assessSession, normalizeSessionFilter } from './session-filter.js';
|
|
2
|
+
import { normalizeWorkMarkers } from './noise.js';
|
|
3
|
+
|
|
1
4
|
export const memoryDefaults = Object.freeze({ enabled: false, provider: 'deepseek-account', model: 'deepseek-flash',
|
|
2
5
|
maxPerTurn: 2, minChars: 12, dailyCap: 40, cooldownSec: 30,
|
|
3
6
|
excludePatterns: Object.freeze(['密码', '身份证', '银行卡', '手机号', '验证码']) });
|
|
4
7
|
|
|
8
|
+
export function sessionImportHint(autoImport, sessions) {
|
|
9
|
+
if (!Array.isArray(sessions) || !autoImport.enabled || autoImport.sessionFilter.mode !== 'include'
|
|
10
|
+
|| !autoImport.sources.some(source => source.type === 'dsh-sessions')) return null;
|
|
11
|
+
if (sessions.some(session => assessSession(session.header ?? session, [], autoImport.sessionFilter).accepted)) return null;
|
|
12
|
+
return { hint: 'no-session-matched', message: '没有会话命中来源白名单,导入可能全部被跳过。请检查 sessionFilter 配置' };
|
|
13
|
+
}
|
|
14
|
+
|
|
5
15
|
function object(value, name) {
|
|
6
16
|
if (!value || typeof value !== 'object' || Array.isArray(value)) throw new Error(`${name} 必须是对象`);
|
|
7
17
|
}
|
|
@@ -19,7 +29,7 @@ export function automationConfig(config = {}, warn = () => {}) {
|
|
|
19
29
|
const sources = [{ type: 'dsh-sessions' }];
|
|
20
30
|
if (config.bridgeRoot) sources.push({ type: 'bridge', path: config.bridgeRoot });
|
|
21
31
|
if (config.diaryDir) sources.push({ type: 'diary', path: config.diaryDir });
|
|
22
|
-
let autoImport = { enabled: true, importAssistant: false, noisePatterns: [], rescanAll: false,
|
|
32
|
+
let autoImport = { enabled: true, importAssistant: false, noisePatterns: [], workMarkers: [], sessionFilter: normalizeSessionFilter(), rescanAll: false,
|
|
23
33
|
sources, intervalMinutes: 60, maxPerRun: 200 };
|
|
24
34
|
try {
|
|
25
35
|
if (config.autoMemory !== undefined) object(config.autoMemory, 'autoMemory');
|
|
@@ -47,6 +57,8 @@ export function automationConfig(config = {}, warn = () => {}) {
|
|
|
47
57
|
if (typeof autoImport.rescanAll !== 'boolean') throw new Error('rescanAll 必须是布尔值');
|
|
48
58
|
if (autoImport.rescanFrom !== undefined && (typeof autoImport.rescanFrom !== 'string' || !autoImport.rescanFrom.trim())) throw new Error('rescanFrom 必须是非空 sessionId');
|
|
49
59
|
if (!Array.isArray(autoImport.noisePatterns) || autoImport.noisePatterns.some(pattern => typeof pattern !== 'string' || !pattern.trim())) throw new Error('noisePatterns 必须是非空字符串数组');
|
|
60
|
+
autoImport.workMarkers = normalizeWorkMarkers(autoImport.workMarkers);
|
|
61
|
+
autoImport.sessionFilter = normalizeSessionFilter(autoImport.sessionFilter);
|
|
50
62
|
range(autoImport.intervalMinutes, 'intervalMinutes', 1, 10080);
|
|
51
63
|
range(autoImport.maxPerRun, 'maxPerRun', 1, 10000);
|
|
52
64
|
if (!Array.isArray(autoImport.sources)) throw new Error('sources 必须是数组');
|
|
@@ -59,8 +71,9 @@ export function automationConfig(config = {}, warn = () => {}) {
|
|
|
59
71
|
autoImport.sources.sort((left, right) => Number(right.type === 'dsh-sessions') - Number(left.type === 'dsh-sessions'));
|
|
60
72
|
} catch (error) {
|
|
61
73
|
warn(`自动导入配置无效,已停用:${error.message}`);
|
|
62
|
-
autoImport = { enabled: false, importAssistant: false, noisePatterns: [], rescanAll: false,
|
|
74
|
+
autoImport = { enabled: false, importAssistant: false, noisePatterns: [], workMarkers: [], sessionFilter: normalizeSessionFilter(), rescanAll: false,
|
|
63
75
|
sources: [], intervalMinutes: 60, maxPerRun: 200 };
|
|
64
76
|
}
|
|
65
77
|
return { autoMemory, autoImport };
|
|
66
78
|
}
|
|
79
|
+
|
package/lib/client.js
CHANGED
|
@@ -899,6 +899,11 @@ function frame(now) {
|
|
|
899
899
|
return React.createElement("section", { "aria-label": "记忆设置", "data-memory-engine-settings": "true",
|
|
900
900
|
style: { position: "absolute", inset: "76px 0 0", overflowY: "auto", padding: "0 24px 32px", color: "#dbeaf8", font: "14px/1.7 system-ui,sans-serif" } },
|
|
901
901
|
React.createElement("div", { style: { maxWidth: "900px", margin: "0 auto", padding: "28px", border: "1px solid #263f54", borderRadius: "14px", background: "#0b1624" } },
|
|
902
|
+
props.settings.autoImport?.hint === "no-session-matched" ? React.createElement("div", { role: "alert", "data-memory-engine-import-hint": "true",
|
|
903
|
+
style: { marginBottom: "20px", padding: "12px 16px", border: "1px solid #9c7243", borderRadius: "8px", background: "#2a2118", color: "#ffe1ac" } },
|
|
904
|
+
React.createElement("span", null, props.settings.autoImport.message), " ",
|
|
905
|
+
React.createElement("button", { type: "button", "data-memory-engine-use-auto-import": "true", disabled: props.configBusy,
|
|
906
|
+
onClick: props.onUseAutoImport, style: actionStyle }, "改用宽松模式")) : null,
|
|
902
907
|
React.createElement("section", { style: { marginBottom: "24px", paddingBottom: "20px", borderBottom: "1px solid #263f54" } },
|
|
903
908
|
React.createElement("h2", { style: { margin: "0 0 12px", fontSize: "20px", color: "#b7e9ff" } }, "自动记忆"),
|
|
904
909
|
React.createElement(AutoMemoryToggle, { settings: props.settings, busy: props.configBusy, onToggle: props.onToggle, location: "settings" }),
|
|
@@ -1000,6 +1005,19 @@ function frame(now) {
|
|
|
1000
1005
|
if (!enabled && settings.showAutoMemoryCostHint !== false) setCostHint(true);
|
|
1001
1006
|
else saveAutoMemory(!enabled);
|
|
1002
1007
|
}
|
|
1008
|
+
function useAutoImport() {
|
|
1009
|
+
if (configBusy || configRequest.current) return;
|
|
1010
|
+
var controller = new AbortController();
|
|
1011
|
+
var timer = setTimeout(function () { controller.abort(); }, 10000);
|
|
1012
|
+
configRequest.current = controller;
|
|
1013
|
+
setConfigBusy(true); setConfigError("");
|
|
1014
|
+
requestApi("/memory-engine/api/config", { method: "PUT", headers: { "Content-Type": "application/json" },
|
|
1015
|
+
body: JSON.stringify({ autoImport: { sessionFilter: { mode: "auto" } } }), signal: controller.signal }).then(function (data) {
|
|
1016
|
+
setSettings(Object.assign({}, defaultSettings, data));
|
|
1017
|
+
}).catch(function (failure) {
|
|
1018
|
+
setConfigError("保存失败:" + (controller.signal.aborted ? "请求超时或已中断,请刷新确认状态" : failure.message));
|
|
1019
|
+
}).finally(function () { clearTimeout(timer); configRequest.current = null; setConfigBusy(false); });
|
|
1020
|
+
}
|
|
1003
1021
|
function closeGuide() {
|
|
1004
1022
|
dismissed.current = true;
|
|
1005
1023
|
setGuideOpen(false);
|
|
@@ -1076,7 +1094,7 @@ function frame(now) {
|
|
|
1076
1094
|
style: { position: "fixed", inset: 0, maxWidth: "none", maxHeight: "none", zIndex: 9999, background: "#03060c" },
|
|
1077
1095
|
}, tab === "map" ? React.createElement(MemoryCanvas, { key: open, rendererRef: renderer, selectionVersion: selectionVersion, onSelect: setSelected })
|
|
1078
1096
|
: tab === "memories" ? React.createElement(MemoryList, { onChanged: function () { setRetry(function (value) { return value + 1; }); } })
|
|
1079
|
-
: React.createElement(MemorySettings, { settings: settings, configBusy: configBusy, onToggle: toggleAutoMemory,
|
|
1097
|
+
: React.createElement(MemorySettings, { settings: settings, configBusy: configBusy, onToggle: toggleAutoMemory, onUseAutoImport: useAutoImport,
|
|
1080
1098
|
onCleared: function () { openMap(); setGuideOpen(true); } }),
|
|
1081
1099
|
tab !== "map" ? React.createElement("strong", { style: { position: "absolute", top: "23px", left: "24px", font: "600 18px system-ui,sans-serif", color: "#b7e9ff" } }, "记忆管理") : null,
|
|
1082
1100
|
React.createElement("style", null, "@keyframes dsh-memory-detail-in{from{opacity:0;transform:translateX(24px)}to{opacity:1;transform:translateX(0)}}@media(prefers-reduced-motion:reduce){[data-memory-engine-detail]{animation:none!important}}"),
|
package/lib/client.src.js
CHANGED
|
@@ -511,6 +511,11 @@ window.__ModuleLoader__.load({
|
|
|
511
511
|
return React.createElement("section", { "aria-label": "记忆设置", "data-memory-engine-settings": "true",
|
|
512
512
|
style: { position: "absolute", inset: "76px 0 0", overflowY: "auto", padding: "0 24px 32px", color: "#dbeaf8", font: "14px/1.7 system-ui,sans-serif" } },
|
|
513
513
|
React.createElement("div", { style: { maxWidth: "900px", margin: "0 auto", padding: "28px", border: "1px solid #263f54", borderRadius: "14px", background: "#0b1624" } },
|
|
514
|
+
props.settings.autoImport?.hint === "no-session-matched" ? React.createElement("div", { role: "alert", "data-memory-engine-import-hint": "true",
|
|
515
|
+
style: { marginBottom: "20px", padding: "12px 16px", border: "1px solid #9c7243", borderRadius: "8px", background: "#2a2118", color: "#ffe1ac" } },
|
|
516
|
+
React.createElement("span", null, props.settings.autoImport.message), " ",
|
|
517
|
+
React.createElement("button", { type: "button", "data-memory-engine-use-auto-import": "true", disabled: props.configBusy,
|
|
518
|
+
onClick: props.onUseAutoImport, style: actionStyle }, "改用宽松模式")) : null,
|
|
514
519
|
React.createElement("section", { style: { marginBottom: "24px", paddingBottom: "20px", borderBottom: "1px solid #263f54" } },
|
|
515
520
|
React.createElement("h2", { style: { margin: "0 0 12px", fontSize: "20px", color: "#b7e9ff" } }, "自动记忆"),
|
|
516
521
|
React.createElement(AutoMemoryToggle, { settings: props.settings, busy: props.configBusy, onToggle: props.onToggle, location: "settings" }),
|
|
@@ -612,6 +617,19 @@ window.__ModuleLoader__.load({
|
|
|
612
617
|
if (!enabled && settings.showAutoMemoryCostHint !== false) setCostHint(true);
|
|
613
618
|
else saveAutoMemory(!enabled);
|
|
614
619
|
}
|
|
620
|
+
function useAutoImport() {
|
|
621
|
+
if (configBusy || configRequest.current) return;
|
|
622
|
+
var controller = new AbortController();
|
|
623
|
+
var timer = setTimeout(function () { controller.abort(); }, 10000);
|
|
624
|
+
configRequest.current = controller;
|
|
625
|
+
setConfigBusy(true); setConfigError("");
|
|
626
|
+
requestApi("/memory-engine/api/config", { method: "PUT", headers: { "Content-Type": "application/json" },
|
|
627
|
+
body: JSON.stringify({ autoImport: { sessionFilter: { mode: "auto" } } }), signal: controller.signal }).then(function (data) {
|
|
628
|
+
setSettings(Object.assign({}, defaultSettings, data));
|
|
629
|
+
}).catch(function (failure) {
|
|
630
|
+
setConfigError("保存失败:" + (controller.signal.aborted ? "请求超时或已中断,请刷新确认状态" : failure.message));
|
|
631
|
+
}).finally(function () { clearTimeout(timer); configRequest.current = null; setConfigBusy(false); });
|
|
632
|
+
}
|
|
615
633
|
function closeGuide() {
|
|
616
634
|
dismissed.current = true;
|
|
617
635
|
setGuideOpen(false);
|
|
@@ -688,7 +706,7 @@ window.__ModuleLoader__.load({
|
|
|
688
706
|
style: { position: "fixed", inset: 0, maxWidth: "none", maxHeight: "none", zIndex: 9999, background: "#03060c" },
|
|
689
707
|
}, tab === "map" ? React.createElement(MemoryCanvas, { key: open, rendererRef: renderer, selectionVersion: selectionVersion, onSelect: setSelected })
|
|
690
708
|
: tab === "memories" ? React.createElement(MemoryList, { onChanged: function () { setRetry(function (value) { return value + 1; }); } })
|
|
691
|
-
: React.createElement(MemorySettings, { settings: settings, configBusy: configBusy, onToggle: toggleAutoMemory,
|
|
709
|
+
: React.createElement(MemorySettings, { settings: settings, configBusy: configBusy, onToggle: toggleAutoMemory, onUseAutoImport: useAutoImport,
|
|
692
710
|
onCleared: function () { openMap(); setGuideOpen(true); } }),
|
|
693
711
|
tab !== "map" ? React.createElement("strong", { style: { position: "absolute", top: "23px", left: "24px", font: "600 18px system-ui,sans-serif", color: "#b7e9ff" } }, "记忆管理") : null,
|
|
694
712
|
React.createElement("style", null, "@keyframes dsh-memory-detail-in{from{opacity:0;transform:translateX(24px)}to{opacity:1;transform:translateX(0)}}@media(prefers-reduced-motion:reduce){[data-memory-engine-detail]{animation:none!important}}"),
|
package/lib/config-write.js
CHANGED
|
@@ -2,11 +2,16 @@ import fs from 'node:fs';
|
|
|
2
2
|
import path from 'node:path';
|
|
3
3
|
import { readState, writeState } from './automation-state.js';
|
|
4
4
|
import { readUiConfig } from './config.js';
|
|
5
|
+
import { normalizeSessionFilter } from './session-filter.js';
|
|
6
|
+
import { normalizeWorkMarkers } from './noise.js';
|
|
5
7
|
|
|
6
8
|
const listeners = new Map();
|
|
7
9
|
const writable = new Map([
|
|
8
10
|
['autoMemory.enabled', value => typeof value === 'boolean'],
|
|
9
11
|
['autoImport.enabled', value => typeof value === 'boolean'],
|
|
12
|
+
['autoImport.importAssistant', value => typeof value === 'boolean'],
|
|
13
|
+
['autoImport.workMarkers', value => { normalizeWorkMarkers(value); return true; }],
|
|
14
|
+
['autoImport.sessionFilter', value => { normalizeSessionFilter(value); return true; }],
|
|
10
15
|
['autoMemory.maxPerTurn', value => Number.isSafeInteger(value) && value >= 1 && value <= 5],
|
|
11
16
|
['autoMemory.dailyCap', value => Number.isSafeInteger(value) && value >= 0 && value <= 500],
|
|
12
17
|
['spinSpeed', value => Number.isFinite(value) && value >= 0.05 && value <= 1.5],
|
|
@@ -37,7 +42,9 @@ export function updateConfig(home, changes, { now = Date.now, warn = () => {} }
|
|
|
37
42
|
if (!fields.length) invalid('至少提供一个可修改字段');
|
|
38
43
|
for (const [key, value] of fields) {
|
|
39
44
|
if (!writable.has(key) || Object.hasOwn(changes, key) && key.includes('.')) invalid(`不支持修改 ${key}`);
|
|
40
|
-
|
|
45
|
+
try {
|
|
46
|
+
if (!writable.get(key)(value)) invalid(`${key} 类型或范围错误`);
|
|
47
|
+
} catch (error) { invalid(error.message); }
|
|
41
48
|
}
|
|
42
49
|
const directory = path.join(home, 'memory-engine');
|
|
43
50
|
const file = path.join(directory, 'config.json');
|
|
@@ -48,6 +55,10 @@ export function updateConfig(home, changes, { now = Date.now, warn = () => {} }
|
|
|
48
55
|
if (key === 'autoMemory' || key === 'autoImport') next[key] = { ...current[key], ...changes[key] };
|
|
49
56
|
else next[key] = changes[key];
|
|
50
57
|
}
|
|
58
|
+
if (changes.autoImport?.sessionFilter) {
|
|
59
|
+
try { next.autoImport.sessionFilter = normalizeSessionFilter({ ...current.autoImport?.sessionFilter, ...changes.autoImport.sessionFilter }); }
|
|
60
|
+
catch (error) { invalid(error.message); }
|
|
61
|
+
}
|
|
51
62
|
if (changes.autoMemory?.enabled === true) next.showAutoMemoryCostHint = false;
|
|
52
63
|
fs.mkdirSync(directory, { recursive: true });
|
|
53
64
|
const prefix = `config.json.bak-${now()}`;
|
package/lib/index.js
CHANGED
|
@@ -200,7 +200,8 @@ export function apply(ctx) {
|
|
|
200
200
|
const store = new MemoryStore();
|
|
201
201
|
for (const tool of createTools(defineTool, store, { warning: message => warn(ctx, message) })) disposers.push(ctx.tools.register(tool));
|
|
202
202
|
try {
|
|
203
|
-
disposers.push(registerMemoryRoutes(ctx.webServer, { store, warn: message => warn(ctx, message)
|
|
203
|
+
disposers.push(registerMemoryRoutes(ctx.webServer, { store, warn: message => warn(ctx, message),
|
|
204
|
+
listSessions: () => ctx.get('sessionQuery')?.listSessions() }));
|
|
204
205
|
} catch (error) {
|
|
205
206
|
warn(ctx, `图谱路由注册失败,记忆工具仍可用:${error.message}`);
|
|
206
207
|
}
|
package/lib/noise.js
CHANGED
|
@@ -7,8 +7,55 @@ export const DEFAULT_NOISE_PATTERNS = Object.freeze([
|
|
|
7
7
|
|
|
8
8
|
const REPORT_PREFIX = /^\s*(已回|已回复|已发送|已处理|已确认|已读|已标记|回过了|发过了)\s*[::]/u;
|
|
9
9
|
|
|
10
|
+
const INJECTED = Object.freeze([
|
|
11
|
+
/^下面是我和他?刚才的聊天记录/u,
|
|
12
|
+
/每行「谁:说了什么」/u,
|
|
13
|
+
/^Current runtime context\. This snapshot supersedes/im,
|
|
14
|
+
/<message\s+source_id=/u,
|
|
15
|
+
/^【会话令牌】/mu,
|
|
16
|
+
/^\s*##\s*(Runtime|Context)\b/im,
|
|
17
|
+
]);
|
|
18
|
+
|
|
19
|
+
export function isInjected(text) {
|
|
20
|
+
return typeof text === 'string' && INJECTED.some(pattern => pattern.test(text));
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
export const WORK_MARKERS = Object.freeze([
|
|
24
|
+
/memory_(save|recall|forget|dedupe|import|map|stats)/iu,
|
|
25
|
+
/hideButton|showGuideOnEmpty|personaFile|autoImport|autoMemory|spinSpeed/iu,
|
|
26
|
+
/验收|任务书|里程碑|回归测试|测试全绿/u,
|
|
27
|
+
/pnpm|\bnpm\b|commit|git push|构建|编译|部署|重启\s*DSH/iu,
|
|
28
|
+
/版本号|0\.\d+\.\d+|beta\.\d/iu,
|
|
29
|
+
/录屏|截图|演示库|演示服务器/u,
|
|
30
|
+
]);
|
|
31
|
+
|
|
32
|
+
export function normalizeWorkMarkers(value = []) {
|
|
33
|
+
if (!Array.isArray(value) || value.length > 20
|
|
34
|
+
|| value.some(pattern => typeof pattern !== 'string' || !pattern.trim() || pattern.length > 300)) throw new Error('workMarkers 必须是最多 20 个非空正则表达式字符串');
|
|
35
|
+
for (const pattern of value) new RegExp(pattern, 'iu');
|
|
36
|
+
return [...value];
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
export function workMatches(text, extra = []) {
|
|
40
|
+
if (typeof text !== 'string') return [];
|
|
41
|
+
const patterns = [...WORK_MARKERS, ...extra.map(pattern => new RegExp(pattern, 'iu'))];
|
|
42
|
+
const seen = new Set();
|
|
43
|
+
return patterns.filter(pattern => {
|
|
44
|
+
if (seen.has(pattern.source)) return false;
|
|
45
|
+
seen.add(pattern.source);
|
|
46
|
+
return pattern.test(text);
|
|
47
|
+
}).map(pattern => pattern.source);
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
export function isWorkRecord(text, extra = []) {
|
|
51
|
+
const matches = workMatches(text, extra);
|
|
52
|
+
const releaseOnly = matches.length === 2 && matches.includes(WORK_MARKERS[3].source) && matches.includes(WORK_MARKERS[4].source);
|
|
53
|
+
return matches.length >= 2 && !releaseOnly;
|
|
54
|
+
}
|
|
55
|
+
|
|
10
56
|
export function isNoise(text, extra = []) {
|
|
11
57
|
if (typeof text !== 'string') return false;
|
|
58
|
+
if (isInjected(text)) return true;
|
|
12
59
|
if (REPORT_PREFIX.test(Array.from(text).slice(0, 12).join(''))) return true;
|
|
13
60
|
const features = text.replace(/(?:他|她|对方|朋友|同事)(?:说|表示|告诉我)[\s“「『"']*(?:已经回过|回过了)/gu, '');
|
|
14
61
|
const patterns = new Set([...DEFAULT_NOISE_PATTERNS, ...extra].filter(pattern => typeof pattern === 'string' && pattern.length > 0));
|
package/lib/onboard.js
CHANGED
|
@@ -1,4 +1,11 @@
|
|
|
1
|
+
// 引导文案(0.7.0):先说自动记忆,再说手动的路
|
|
2
|
+
// 原则:看到「三条」就知道该干什么,不需要读文档。
|
|
3
|
+
|
|
4
|
+
// 可选的手动提示词:自动记忆关闭时才需要它。
|
|
5
|
+
// 不要再把它说成「必须加进角色卡」——0.7.0 起自动记忆能自己抽。
|
|
1
6
|
export const ONBOARD_PROMPT = [
|
|
7
|
+
'(自动记忆已开启时可以不用这一段)',
|
|
8
|
+
'',
|
|
2
9
|
'聊天时遇到这些,用 memory_save 记下来:',
|
|
3
10
|
' · 对方的称呼、偏好、忌讳、习惯(kind="印象",who 填对方名字)',
|
|
4
11
|
' · 答应过的事、约定、承诺(kind="对话")',
|
|
@@ -11,8 +18,16 @@ export const ONBOARD_PROMPT = [
|
|
|
11
18
|
].join('\n');
|
|
12
19
|
|
|
13
20
|
export const ONBOARD_STEPS = [
|
|
14
|
-
{
|
|
15
|
-
|
|
16
|
-
|
|
21
|
+
{
|
|
22
|
+
title: '先打开自动记忆',
|
|
23
|
+
body: '图谱右上角有个「自动记忆」开关,默认是关的。点一下变成 ● 之后,每轮对话结束会自动抽 0~2 条记下来——这时候你什么都不用做。会调用模型,大约每轮几分钱,随时可以再点掉。',
|
|
24
|
+
},
|
|
25
|
+
{
|
|
26
|
+
title: '想把旧记录也收进来',
|
|
27
|
+
body: '跟 AI 说「把 D:\\bot\\memories.json 导进记忆库」,它会调 memory_import。支持 JSON、Markdown、纯文本和 DSH 会话记录;默认只预览,确认后才写入。',
|
|
28
|
+
},
|
|
29
|
+
{
|
|
30
|
+
title: '回来这里看',
|
|
31
|
+
body: '「记忆」页签能搜索、改错字、删除、导出;「图谱」页签是所有记忆画成的球体,中间那颗金色的是你的人设卡(在 config.json 里配 personaFile)。拖拽旋转,点任意光点看它连着什么。数据存在本机 ~/.dsh/memory-engine/,纯 JSON 明文,不会上传到任何地方。',
|
|
32
|
+
},
|
|
17
33
|
];
|
|
18
|
-
|
package/lib/routes.js
CHANGED
|
@@ -2,8 +2,9 @@ import path from 'node:path';
|
|
|
2
2
|
import { computeMapData } from './map.js';
|
|
3
3
|
import { renderHtml } from './render/render.mjs';
|
|
4
4
|
import { ONBOARD_PROMPT, ONBOARD_STEPS } from './onboard.js';
|
|
5
|
-
import { readUiConfig } from './config.js';
|
|
5
|
+
import { readConfig, readUiConfig } from './config.js';
|
|
6
6
|
import { updateConfig } from './config-write.js';
|
|
7
|
+
import { sessionImportHint } from './automation-config.js';
|
|
7
8
|
import { chars, retention, titleOf } from './decay.js';
|
|
8
9
|
import { scoreMemories } from './render/score.mjs';
|
|
9
10
|
|
|
@@ -75,7 +76,19 @@ function listMemories(store, query) {
|
|
|
75
76
|
items: items.slice(offset, offset + size).map(row => ({ ...row, preview: chars(row.text, 60) + (Array.from(row.text).length > 60 ? '…' : '') })) };
|
|
76
77
|
}
|
|
77
78
|
|
|
78
|
-
export function createMemoryHandler({ store, warn = () => {}, beforeConfigUpdate = () => {} }) {
|
|
79
|
+
export function createMemoryHandler({ store, warn = () => {}, beforeConfigUpdate = () => {}, listSessions }) {
|
|
80
|
+
const home = path.dirname(path.dirname(store.file));
|
|
81
|
+
async function withSessionHint(data) {
|
|
82
|
+
if (typeof listSessions !== 'function') return data;
|
|
83
|
+
try {
|
|
84
|
+
const sessions = await listSessions();
|
|
85
|
+
const hint = sessionImportHint(readConfig(home).autoImport, sessions);
|
|
86
|
+
return hint ? { ...data, autoImport: { ...data.autoImport, ...hint } } : data;
|
|
87
|
+
} catch (error) {
|
|
88
|
+
warn(`读取会话来源提示失败:${error.message}`);
|
|
89
|
+
return data;
|
|
90
|
+
}
|
|
91
|
+
}
|
|
79
92
|
return async (request, response) => {
|
|
80
93
|
let methods = ['GET'];
|
|
81
94
|
const reply = (status, payload, html = false) => {
|
|
@@ -107,7 +120,8 @@ export function createMemoryHandler({ store, warn = () => {}, beforeConfigUpdate
|
|
|
107
120
|
const body = await readBody(request);
|
|
108
121
|
if (pathname === `${base}/api/config`) {
|
|
109
122
|
beforeConfigUpdate(body);
|
|
110
|
-
|
|
123
|
+
const data = updateConfig(home, body, { warn });
|
|
124
|
+
return reply(200, { ok: true, data: await withSessionHint(data) });
|
|
111
125
|
}
|
|
112
126
|
return reply(200, { ok: true, data: item ? store.updateItem(item[1], body) : store.clear(body) });
|
|
113
127
|
}
|
|
@@ -125,7 +139,7 @@ export function createMemoryHandler({ store, warn = () => {}, beforeConfigUpdate
|
|
|
125
139
|
}
|
|
126
140
|
if (pathname === `${base}/api/map`) return reply(200, { ok: true, data: computeMapData(store, { warn }) });
|
|
127
141
|
if (pathname === `${base}/api/stats`) return reply(200, { ok: true, data: store.stats() });
|
|
128
|
-
if (pathname === `${base}/api/config`) return reply(200, { ok: true, data: readUiConfig(
|
|
142
|
+
if (pathname === `${base}/api/config`) return reply(200, { ok: true, data: await withSessionHint(readUiConfig(home, { warn })) });
|
|
129
143
|
if (pathname === `${base}/api/onboard`) return reply(200, { ok: true, data: {
|
|
130
144
|
empty: store.read().length === 0, prompt: ONBOARD_PROMPT, steps: ONBOARD_STEPS, dataDir: path.dirname(store.file),
|
|
131
145
|
} });
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
export const CHAT_CWD_HINTS = Object.freeze(['state/agents', 'qq-bridge', 'astrbot', 'nonebot', 'koishi']);
|
|
2
|
+
export const sessionFilterDefaults = Object.freeze({ mode: 'include', cwdContains: CHAT_CWD_HINTS, maxToolCalls: 0 });
|
|
3
|
+
const workTool = /(?:^|[./:]|__)(pwsh|bash|read|write|edit|grep|glob|subagent|workflow|notebook)$/iu;
|
|
4
|
+
const normalizePath = value => String(value ?? '').replaceAll('\\', '/').toLowerCase();
|
|
5
|
+
const matchCwd = (cwd, part) => {
|
|
6
|
+
const normalizedCwd = normalizePath(cwd);
|
|
7
|
+
const normalizedPart = normalizePath(part).replace(/^\/+|\/+$/gu, '');
|
|
8
|
+
if (!CHAT_CWD_HINTS.includes(normalizedPart)) return normalizedCwd.includes(normalizedPart);
|
|
9
|
+
return `/${normalizedCwd.replace(/^\/+|\/+$/gu, '')}/`.includes(`/${normalizedPart}/`);
|
|
10
|
+
};
|
|
11
|
+
|
|
12
|
+
export function normalizeSessionFilter(value = {}) {
|
|
13
|
+
if (!value || typeof value !== 'object' || Array.isArray(value)) throw new Error('sessionFilter 必须是对象');
|
|
14
|
+
for (const key of Object.keys(value)) if (!Object.hasOwn(sessionFilterDefaults, key)) throw new Error(`sessionFilter 不支持 ${key}`);
|
|
15
|
+
const filter = { ...sessionFilterDefaults, ...value };
|
|
16
|
+
if (!['auto', 'include', 'all'].includes(filter.mode)) throw new Error('sessionFilter.mode 必须是 auto、include 或 all');
|
|
17
|
+
if (!Array.isArray(filter.cwdContains) || filter.cwdContains.length > 50
|
|
18
|
+
|| filter.cwdContains.some(item => typeof item !== 'string' || !item.trim() || item.length > 1000)) throw new Error('cwdContains 必须是最多 50 个非空路径片段');
|
|
19
|
+
if (!Number.isSafeInteger(filter.maxToolCalls) || filter.maxToolCalls < 0 || filter.maxToolCalls > 10000) throw new Error('maxToolCalls 必须是 0~10000 的整数');
|
|
20
|
+
return { ...filter, cwdContains: filter.cwdContains.map(item => item.trim()) };
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
export function sessionFilterKey(header, filter) {
|
|
24
|
+
return JSON.stringify({ version: 2, filter, cwd: normalizePath(header?.cwd) });
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
export function assessSession(header, events, filter = sessionFilterDefaults) {
|
|
28
|
+
if (filter.mode === 'all') return { accepted: true };
|
|
29
|
+
if (filter.mode === 'include') {
|
|
30
|
+
return filter.cwdContains.some(part => matchCwd(header?.cwd, part)) ? { accepted: true }
|
|
31
|
+
: { accepted: false, reason: 'cwd-excluded', message: 'cwd 未命中包含列表', cacheable: true };
|
|
32
|
+
}
|
|
33
|
+
let userMessages = 0;
|
|
34
|
+
let workToolCalls = 0;
|
|
35
|
+
const tools = new Set();
|
|
36
|
+
for (const event of events) {
|
|
37
|
+
if (event.type === 'user/message' && (!event.data?.role || event.data.role === 'user')) userMessages++;
|
|
38
|
+
if (event.type === 'tool/call' && typeof event.data?.name === 'string' && workTool.test(event.data.name)) {
|
|
39
|
+
workToolCalls++;
|
|
40
|
+
tools.add(event.data.name);
|
|
41
|
+
if (workToolCalls > filter.maxToolCalls) return { accepted: false, reason: 'work-tools',
|
|
42
|
+
message: `用过 ${[...tools].join('/')}`, tools: [...tools], workToolCalls, userMessages, cacheable: true };
|
|
43
|
+
}
|
|
44
|
+
}
|
|
45
|
+
return userMessages >= 2 ? { accepted: true } : { accepted: false, reason: 'too-few-user-messages',
|
|
46
|
+
message: `user 消息不足 2 条(当前 ${userMessages} 条)`, userMessages, cacheable: false };
|
|
47
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-memory-engine",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.5",
|
|
4
4
|
"description": "Local DSH memory engine with neuron network visualization and private file import",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"dsh",
|
|
@@ -29,6 +29,7 @@
|
|
|
29
29
|
"lib/",
|
|
30
30
|
"scripts/render-map.mjs",
|
|
31
31
|
"scripts/sync-qq.mjs",
|
|
32
|
+
"scripts/cleanup-imports.mjs",
|
|
32
33
|
"scripts/build-client.mjs",
|
|
33
34
|
"scripts/lib/",
|
|
34
35
|
"cordis.patch.yml",
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
import fs from 'node:fs';
|
|
3
|
+
import path from 'node:path';
|
|
4
|
+
import { createHash } from 'node:crypto';
|
|
5
|
+
import { parseArgs } from 'node:util';
|
|
6
|
+
import { pathToFileURL } from 'node:url';
|
|
7
|
+
import { MemoryStore } from '../lib/store.js';
|
|
8
|
+
import { duplicateGroups } from '../lib/dedupe.js';
|
|
9
|
+
import { isWorkRecord, normalizeWorkMarkers, workMatches } from '../lib/noise.js';
|
|
10
|
+
import { readState } from '../lib/automation-state.js';
|
|
11
|
+
|
|
12
|
+
const digest = value => createHash('sha256').update(value).digest('hex');
|
|
13
|
+
const isQQ = row => ['qq-bridge', 'bridge'].includes(row.origin?.source);
|
|
14
|
+
const qqCount = rows => rows.filter(isQQ).length;
|
|
15
|
+
const fingerprint = file => digest(fs.existsSync(file) ? fs.readFileSync(file) : '[]');
|
|
16
|
+
|
|
17
|
+
export function previewCleanup({ home, stage, threshold = 0.7, protectQQ = false }) {
|
|
18
|
+
if (!home) throw new Error('必须明确指定 home');
|
|
19
|
+
if (!['assistant', 'work', 'dedupe'].includes(stage)) throw new Error('stage 必须是 assistant、work 或 dedupe');
|
|
20
|
+
if (typeof protectQQ !== 'boolean' || protectQQ && stage !== 'dedupe') throw new Error('protectQQ 仅支持 dedupe 步骤的布尔开关');
|
|
21
|
+
const store = new MemoryStore({ home });
|
|
22
|
+
const beforeHash = fingerprint(store.file);
|
|
23
|
+
const rows = store.read();
|
|
24
|
+
const config = readState(path.join(path.dirname(store.file), 'config.json'), {});
|
|
25
|
+
const markers = normalizeWorkMarkers(config.autoImport?.workMarkers);
|
|
26
|
+
const assistants = rows.filter(row => !isQQ(row) && row.origin?.role === 'assistant');
|
|
27
|
+
const work = rows.filter(row => !isQQ(row) && isWorkRecord(row.text, markers));
|
|
28
|
+
if (stage !== 'assistant' && assistants.length) throw new Error(`请先完成 assistant 步骤(剩余 ${assistants.length} 条)`);
|
|
29
|
+
if (stage === 'dedupe' && work.length) throw new Error(`请先完成 work 步骤(剩余 ${work.length} 条)`);
|
|
30
|
+
let selected;
|
|
31
|
+
let protectedRows = [];
|
|
32
|
+
let groups = [];
|
|
33
|
+
let qqAfter = qqCount(rows);
|
|
34
|
+
if (stage === 'dedupe') {
|
|
35
|
+
const allGroups = duplicateGroups(rows, threshold);
|
|
36
|
+
const conflicts = protectQQ ? allGroups.filter(group => qqCount(group.indices.map(index => rows[index])) > Number(isQQ(group.memory))) : [];
|
|
37
|
+
const excluded = new Set(conflicts);
|
|
38
|
+
const matches = allGroups.filter(group => !excluded.has(group));
|
|
39
|
+
protectedRows = conflicts.flatMap(group => group.indices.map(index => rows[index]));
|
|
40
|
+
groups = matches.map(group => ({ keep: group.memory.id, ids: group.indices.map(index => rows[index].id), memory: group.memory }));
|
|
41
|
+
selected = matches.flatMap(group => group.indices.slice(1).map(index => rows[index]));
|
|
42
|
+
qqAfter -= matches.reduce((sum, group) => sum + qqCount(group.indices.map(index => rows[index])) - Number(isQQ(group.memory)), 0);
|
|
43
|
+
} else {
|
|
44
|
+
const matches = rows.filter(row => stage === 'assistant' ? row.origin?.role === 'assistant' : isWorkRecord(row.text, markers));
|
|
45
|
+
selected = matches.filter(row => !isQQ(row));
|
|
46
|
+
protectedRows = matches.filter(isQQ);
|
|
47
|
+
}
|
|
48
|
+
if (fingerprint(store.file) !== beforeHash) throw new Error('预览期间记忆库已变化,请重新预览');
|
|
49
|
+
const plan = { version: 1, stage, protectQQ, file: store.file, beforeHash, before: rows.length, after: rows.length - selected.length,
|
|
50
|
+
removed: selected.length, qqBefore: qqCount(rows), qqAfter, blocked: qqAfter < qqCount(rows),
|
|
51
|
+
threshold: stage === 'dedupe' ? threshold : null, workMarkers: markers,
|
|
52
|
+
items: selected.map(row => ({ id: row.id, text: row.text, origin: row.origin ?? null,
|
|
53
|
+
reason: stage === 'work' ? workMatches(row.text, markers) : [stage] })),
|
|
54
|
+
protected: protectedRows.map(row => ({ id: row.id, text: row.text, origin: row.origin })), groups };
|
|
55
|
+
return { ...plan, token: digest(JSON.stringify(plan)) };
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
export function applyCleanup({ home, preview, confirm }) {
|
|
59
|
+
if (!preview?.token || confirm !== preview.token) throw new Error('必须先审阅清单,再用该清单的 token 明确确认');
|
|
60
|
+
const current = previewCleanup({ home, stage: preview.stage, threshold: preview.threshold ?? 0.7, protectQQ: preview.protectQQ ?? false });
|
|
61
|
+
const { token, ...reviewed } = preview;
|
|
62
|
+
if (digest(JSON.stringify(reviewed)) !== token || current.token !== token) throw new Error('清单已修改、库已变化或规则已改变,请重新预览并确认');
|
|
63
|
+
if (current.blocked) throw new Error('本次合并会减少 QQ 来源数量,已拒绝执行;请先审阅冲突组');
|
|
64
|
+
const store = new MemoryStore({ home });
|
|
65
|
+
if (!current.removed) return { stage: current.stage, removed: 0, total: current.before, qqBefore: current.qqBefore, qqAfter: current.qqBefore };
|
|
66
|
+
let backup;
|
|
67
|
+
if (current.stage === 'dedupe') {
|
|
68
|
+
const protectedIds = new Set(current.protected.map(row => row.id));
|
|
69
|
+
const scoped = Object.create(store);
|
|
70
|
+
if (protectedIds.size) {
|
|
71
|
+
const original = store.read();
|
|
72
|
+
scoped.read = () => store.read().filter(row => !protectedIds.has(row.id));
|
|
73
|
+
scoped.write = merged => {
|
|
74
|
+
if (fingerprint(store.file) !== current.beforeHash) throw new Error('合并期间记忆库已变化,未写入;请重新预览');
|
|
75
|
+
const byId = new Map(merged.map(row => [row.id, row]));
|
|
76
|
+
store.write(original.flatMap(row => protectedIds.has(row.id) ? [row] : byId.has(row.id) ? [byId.get(row.id)] : []));
|
|
77
|
+
};
|
|
78
|
+
}
|
|
79
|
+
const result = scoped.dedupe({ threshold: current.threshold, dry_run: false });
|
|
80
|
+
backup = result.backup;
|
|
81
|
+
} else {
|
|
82
|
+
const rows = store.read();
|
|
83
|
+
const selected = new Set(current.items.map(row => row.id));
|
|
84
|
+
backup = store.backup();
|
|
85
|
+
if (fingerprint(store.file) !== current.beforeHash) throw new Error('备份期间记忆库已变化,未删除;请重新预览');
|
|
86
|
+
store.write(rows.filter(row => !selected.has(row.id)));
|
|
87
|
+
}
|
|
88
|
+
const after = store.read();
|
|
89
|
+
if (after.length !== current.after || qqCount(after) !== current.qqAfter) throw new Error(`写后校验不符,请检查并保留备份:${backup}`);
|
|
90
|
+
return { stage: current.stage, removed: current.removed, total: after.length, qqBefore: current.qqBefore, qqAfter: qqCount(after), backup };
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
function main() {
|
|
94
|
+
const { values } = parseArgs({ options: { home: { type: 'string' }, stage: { type: 'string' }, out: { type: 'string' },
|
|
95
|
+
preview: { type: 'string' }, confirm: { type: 'string' }, threshold: { type: 'string' }, 'protect-qq': { type: 'boolean' }, help: { type: 'boolean' } } });
|
|
96
|
+
if (values.help) {
|
|
97
|
+
console.log('预览:node scripts/cleanup-imports.mjs --home <DSH目录> --stage assistant|work|dedupe --out <清单.json>\n保护 QQ 冲突组:预览 dedupe 时加 --protect-qq\n执行:node scripts/cleanup-imports.mjs --home <DSH目录> --preview <清单.json> --confirm <清单token>\n逐步审阅并确认;执行前关闭 DSH,避免其他进程同时写库。');
|
|
98
|
+
return;
|
|
99
|
+
}
|
|
100
|
+
if (values.preview) {
|
|
101
|
+
if (values.stage || values.out || values.threshold || values['protect-qq']) throw new Error('执行时只接受 home、preview 和 confirm');
|
|
102
|
+
const preview = JSON.parse(fs.readFileSync(values.preview, 'utf8').replace(/^\uFEFF/u, ''));
|
|
103
|
+
console.log(JSON.stringify(applyCleanup({ home: values.home, preview, confirm: values.confirm }), null, 2));
|
|
104
|
+
return;
|
|
105
|
+
}
|
|
106
|
+
if (values.confirm) throw new Error('确认时必须指定 preview 文件');
|
|
107
|
+
const preview = previewCleanup({ home: values.home, stage: values.stage, threshold: values.threshold === undefined ? 0.7 : Number(values.threshold),
|
|
108
|
+
protectQQ: values['protect-qq'] ?? false });
|
|
109
|
+
if (values.out) fs.writeFileSync(values.out, JSON.stringify(preview, null, 2) + '\n', { flag: 'wx', mode: 0o600 });
|
|
110
|
+
console.log(`将${preview.stage === 'dedupe' ? '合并删除' : '删除'} ${preview.removed} 条;QQ 来源 ${preview.qqBefore} → ${preview.qqAfter};保护保留 ${preview.protected.length} 条${preview.blocked ? ';禁止执行:QQ 数量会减少' : ''}`);
|
|
111
|
+
console.log(JSON.stringify({ token: preview.token, first10: preview.items.slice(0, 10) }, null, 2));
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) {
|
|
115
|
+
try { main(); } catch (error) { console.error(error.message); process.exitCode = 1; }
|
|
116
|
+
}
|