@ilaomianyang/dsh-memory-graph 0.10.4 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +21 -0
- package/README.en.md +24 -3
- package/README.md +24 -3
- package/client.js +1 -1
- package/package.json +5 -3
- package/src/auto.js +10 -7
- package/src/capture.js +37 -175
- package/src/distillapi.js +4 -1
- package/src/distillrun.js +7 -3
- package/src/extract.js +42 -18
- package/src/graph.js +10 -5
- package/src/graphdata.js +10 -1
- package/src/history.js +440 -0
- package/src/pageapi.js +2 -2
- package/src/tools.js +52 -1
- package/src/turns.js +265 -0
- package/src/vault.js +20 -10
- package/test/fixtures/session.v4.jsonl.zstd +0 -0
- package/test/history.mjs +249 -0
- package/test/tools.mjs +2 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,26 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.11.0 — 2026-11-03
|
|
4
|
+
|
|
5
|
+
历史会话也能进库了,宿主侧的文案统一成中文。
|
|
6
|
+
|
|
7
|
+
**Added**
|
|
8
|
+
|
|
9
|
+
- `memory_backfill`:把**插件装上之前**的会话历史一次性导入记忆库。采集只看得见宿主运行时发生的对话,所以新库一开始是空的;这个工具回放磁盘上的 `~/.dsh/sessions/<项目>/<会话>/session.v4.jsonl.zstd`,走**与实时采集完全相同**的过滤与脱敏(`src/turns.js` 是唯一实现),因此回填出来的情节和实时采集的是同一种记录。时间戳取自会话文件里那一轮,情节落进**它当天**的 `episodes/YYYY-MM-DD.jsonl`,不是全堆在今天。默认 `dryRun`,只统计不写库。
|
|
10
|
+
- 幂等:一轮对话的身份是 `(会话, 轮次)`,库里已有就跳过,所以还在长的会话下次再跑只补新的部分;删档重装、换机器、装晚了都是同一条命令。
|
|
11
|
+
- 实测(本机):133 个会话 → 2078 条情节,85 条已有跳过,0 坏行、0 读不了,17 秒。
|
|
12
|
+
- `test/history.mjs` 12 项 + `test/fixtures/session.v4.jsonl.zstd`(真的 zstd 帧、真的 v4 事件形状)。检查数 232 → 244。
|
|
13
|
+
|
|
14
|
+
**Fixed**
|
|
15
|
+
|
|
16
|
+
- 会话文件是**拼接的 zstd 容器**(每次追加写一帧),而 `zstdDecompressSync` 和 fzstd 都只解第一帧——按流读只会拿到第一行会话头。`src/history.js` 现在带一个帧扫描器(和 harness 自己用的算法同构:帧头 → 3 字节块头 → 可选校验和),逐帧解码;Node ≥ 22.15 用原生 zstd(全保真),更老的 Node 退回 fzstd,报告里会写明用的是哪个解码器。单帧坏掉只丢那一批,不丢整个会话。
|
|
17
|
+
- `vault.appendEpisode` 的日期改为取记录自己的 `t`,不再一律写今天——否则回填三个月历史会让 `episodes/` 看起来全发生在今天(实测 27 个日期文件)。
|
|
18
|
+
|
|
19
|
+
**Changed**
|
|
20
|
+
|
|
21
|
+
- 宿主侧的用户可见文本统一成中文(英文保留在后,形如「中文 · English」):抽取器/蒸馏的失败说明、路由错误体、失败页兜底、审计行、自动蒸馏的日志、图谱页的按钮与 tooltip、`confidence` 徽章的三种取值。共 45 条 / 10 个文件。`autoDecision` 的 reason 码、枚举码、发给模型的 `SYSTEM_PROMPT` 保持英文。
|
|
22
|
+
- `src/graphdata.js` 的 400 响应补了 `message`,页签不再把机器码 `missing-session` 直接印出来(`reason` 未动)。
|
|
23
|
+
|
|
3
24
|
## 0.10.4 — 2026-11-03
|
|
4
25
|
|
|
5
26
|
**Changed**
|
package/README.en.md
CHANGED
|
@@ -267,11 +267,32 @@ Point 3 means that if the key is already configured for the harness (`DEEPSEEK_A
|
|
|
267
267
|
| `memory_forget` | `retract` (keeps the trace) or `purge` (deletes the file) |
|
|
268
268
|
| `memory_distill` | turn raw episodes into cards with the extractor model |
|
|
269
269
|
| `memory_stats` | vault location, counts, last distillation |
|
|
270
|
+
| `memory_backfill` | import the sessions from **before the plugin was installed** (re-runnable; known turns are skipped) |
|
|
270
271
|
|
|
271
272
|
`memory_graph` is the "neuron" part made concrete: an edge is not invented, it is **two cards mentioning the same entity**. It claims co-mention, never causation — the model decides what that means.
|
|
272
273
|
|
|
273
274
|
**Nothing is injected automatically.** Memory only enters the context when the model asks for it. Most turns do not need long-term memory, and injecting it would be pure cost.
|
|
274
275
|
|
|
276
|
+
## Past sessions count too
|
|
277
|
+
|
|
278
|
+
Capture only sees conversations that happen **while the host is running**, so a freshly installed vault starts empty: the dozens of sessions before it will never appear on their own. `memory_backfill` fills that gap by replaying the session files on disk (`~/.dsh/sessions/<project>/<session>/session.v4.jsonl.zstd`) into episodes of **the same shape** live capture writes.
|
|
279
|
+
|
|
280
|
+
- **Same filter, one implementation**: `<system-reminder>` blocks stripped, runtime snapshots (runtime context, compaction checkpoints) never counted as something a person said, length caps, redaction — all shared with live capture through `src/turns.js`, so the two paths cannot quietly diverge.
|
|
281
|
+
- **The original timestamp**: `t` comes from the turn in the log, and the episode lands in **its own** `episodes/YYYY-MM-DD.jsonl` rather than piling up under today.
|
|
282
|
+
- **Safe to run twice**: a turn is identified by `(session, turn)` and skipped when the vault already has it, so re-running after a session grew imports only the new part. Reinstalls, new machines and late installs are all the same command.
|
|
283
|
+
- **Dry run by default** (`dryRun`): see the numbers before anything is written.
|
|
284
|
+
|
|
285
|
+
In one line:
|
|
286
|
+
|
|
287
|
+
```text
|
|
288
|
+
memory_backfill {dryRun: true} → 133 sessions scanned / 2078 episodes would be written
|
|
289
|
+
memory_backfill {dryRun: false} → 2078 episodes written; 85 already present, skipped
|
|
290
|
+
```
|
|
291
|
+
|
|
292
|
+
What lands is **episodes**, not cards: distil once (`memory_distill`, the Settings button, or the daily auto run at 40 episodes a batch) and they enter the retrieval layer. Two thousand episodes is fifty model batches — that is a spend worth deciding on deliberately.
|
|
293
|
+
|
|
294
|
+
A session file is a **concatenated zstd container** (one frame per appended batch), so reading it means cutting it frame by frame — `src/history.js` carries a scanner equivalent to the one the harness itself uses.
|
|
295
|
+
|
|
275
296
|
## Look at your memory
|
|
276
297
|
|
|
277
298
|
`memory_stats` tells you where the vault is; in its root there is also **`graph.html`** — double-click it, nothing to install, no server:
|
|
@@ -383,7 +404,7 @@ Side benefit: an injected message no longer cuts a turn in half, so an episode's
|
|
|
383
404
|
No network, no real vault:
|
|
384
405
|
|
|
385
406
|
```bash
|
|
386
|
-
npm test # 43 + 13 + 35 + 12 + 22 + 54 + 15 + 8 + 11 + 19 =
|
|
407
|
+
npm test # 43 + 13 + 35 + 12 + 22 + 54 + 15 + 8 + 11 + 19 + 12 = 244 checks, plus the 25-question retrieval eval
|
|
387
408
|
npm run eval # retrieval eval only (offline, deterministic)
|
|
388
409
|
```
|
|
389
410
|
|
|
@@ -392,7 +413,7 @@ npm run eval # retrieval eval only (offline, deterministic)
|
|
|
392
413
|
- `test/client.mjs` — mounts the page on a minimal React stand-in: the controls drawn equal the fields that should show, switching `transport` reveals the http knobs, an empty inline key is never sent as a deletion, an out-of-range number never reaches the wire, a refused save clears the drafts and explains itself, restore sends `unset` rather than `set`, a value that merely repeats the inherited default is not badged as a customisation, the stylesheet is a single `<style>` tagged with the package name, and `client.js` contains no `export`/`import` (the loader evaluates it as a script). It then evaluates `client.js` in a fake page carrying `window.__ModuleLoader__` and walks the real browser path — factory, `apply`, both slot registrations — checking that `react` is the only module required, the registration id is the package name, the orders are 25 and 30, and the labels follow the page language. Finally it drives the **Memory tab**: one request per paint carrying this `sessionId`, every node and edge the host sent drawn, a click opening the card's title and body in the side panel, and both the empty session and the failing route explaining themselves instead of going blank. It also checks the **sidebar entry + main panel**: one rail entry and one page keyed by the same id, the action bar a sibling of the scroll box rather than a sticky child of it, the panel embedding the served route instead of a second renderer, and no vault path in the `src`
|
|
393
414
|
- `test/graphdata.mjs` — the Memory tab's data: episodes filtered by `session`, majority project, the recall query taking the newest turns, a deterministic layout inside the box, every edge pointing at a node that exists, a cross-project card arriving through a shared entity, the caps reporting that they held, no assistant text in the payload, and the route answering 400 / 200 / 500 for missing session, normal, and a broken vault
|
|
394
415
|
- `test/capture.mjs` — drives the capture listener with real `session/event` shapes, including injected context
|
|
395
|
-
- `test/tools.mjs` — runs `execute` + `render` for all
|
|
416
|
+
- `test/tools.mjs` — runs `execute` + `render` for all nine tools
|
|
396
417
|
- `test/page.mjs` — the route behind the sidebar panel: `/api/memory-graph.page` sits on the authenticated channel, an empty vault still answers with a page, it is **the same renderer as the `graph.html` on disk** (only the timestamp differs), `X-Frame-Options: SAMEORIGIN` so only the app can frame it, a broken vault prints the reason on the page (escaped), no vault at all is a 500 rather than a crash, and no vault path ever appears in the bytes
|
|
397
418
|
- `test/distillapi.mjs` — the Distill now door: `/api/memory-graph.distill` on the authenticated `/api` channel, running the same `distill()` (live vault, the model selected in Settings, the live redaction rules), one press is one run, a concurrent second press is refused instead of queued, an extractor refusal is a 200 and not a 500, a throw is a 500 with the lock still released, views are re-rendered only when cards were written, a render failure only warns, the card list is capped at 40, and a route with no runner answers 500 instead of throwing at the browser
|
|
398
419
|
- `test/auto.mjs` — the automatic distiller: each rule fires on its own, whichever matches first runs when both are on, the minimum gap is checked before the rules, a quiet window may cross midnight, a tick only decides (an off switch does not even read the vault), one tick is at most one run, the gap is armed before the call, a restart reads `.auto-state.json` instead of starting over, a run that wrote nothing still arms the gap, an in-flight run is not stacked on, failures are logged with their reason, the project reaches the run, an unreadable vault is a skip and not a crash, and the tick asks for enough episodes to see the threshold
|
|
@@ -422,7 +443,7 @@ The tag triggers `.github/workflows/publish.yml`, which does four things:
|
|
|
422
443
|
|
|
423
444
|
1. checks that the tag and `package.json` agree — a mismatch fails instead of publishing a package whose version lies;
|
|
424
445
|
2. checks whether that version is already on the registry — if so it skips, so re-running a tag is a no-op rather than an error;
|
|
425
|
-
3. `npm publish --provenance --access public` (`prepublishOnly` runs the
|
|
446
|
+
3. `npm publish --provenance --access public` (`prepublishOnly` runs the 244 checks again; a red build cannot reach the registry);
|
|
426
447
|
4. adds "Built and signed on GitHub Actions" to the npm page, proving the tarball came from this commit of this repository.
|
|
427
448
|
|
|
428
449
|
Two ways to authenticate; the workflow accepts both (it uses `NPM_TOKEN` when the secret exists and falls back to OIDC when it does not):
|
package/README.md
CHANGED
|
@@ -259,11 +259,32 @@ plugin_manager action=install_bundle target="<clone 下来的绝对路径>"
|
|
|
259
259
|
| `memory_forget` | `retract` 撤回(留痕)或 `purge` 删除 |
|
|
260
260
|
| `memory_distill` | 用抽取模型把原始事件蒸馏成卡片 |
|
|
261
261
|
| `memory_stats` | 库位置、数量、上次蒸馏时间 |
|
|
262
|
+
| `memory_backfill` | 把**插件装上之前**的会话历史导入记忆库(可重复跑,已有轮次跳过) |
|
|
262
263
|
|
|
263
264
|
`memory_graph` 是"神经元"那部分的具体形态:边不是凭空生成的,而是**两张卡片提到同一个实体**。它不声称因果,只声称"这两个东西在同一段记忆里出现过",模型看到之后自己判断。
|
|
264
265
|
|
|
265
266
|
**没有自动注入。** 记忆只在模型主动调用工具时才进入上下文——这是刻意的:大多数轮次根本不需要长期记忆,注入了就是纯成本。
|
|
266
267
|
|
|
268
|
+
## 历史会话也能进
|
|
269
|
+
|
|
270
|
+
采集只看得见**宿主开着的时候**发生的对话,所以新装的库一开始是空的——装之前的几十个会话不会自己冒出来。`memory_backfill` 补这一段:它回放磁盘上的会话文件(`~/.dsh/sessions/<项目目录>/<会话>/session.v4.jsonl.zstd`),转成和实时采集**同样格式**的情节。
|
|
271
|
+
|
|
272
|
+
- **走的还是那套过滤**:`<system-reminder>` 剥掉、运行时快照(runtime context、压缩检查点)不算人话、限长、脱敏,全部复用 `src/turns.js`——实时和回填只有一份实现,不会两边慢慢走偏。
|
|
273
|
+
- **时间是原来的时间**:`t` 取自会话文件里那一轮的时间戳,情节落进它**当天**的 `episodes/YYYY-MM-DD.jsonl`,不是全堆在今天。
|
|
274
|
+
- **可以重复跑**:一轮对话的唯一身份是 `(会话, 轮次)`,库里已经有了就跳过;所以一个还在长的会话,下次再跑只补新的部分。删档重装、换机器、插件装晚了,都是这一条命令的事。
|
|
275
|
+
- **默认只统计不写库**(`dryRun`),先看数字再决定。
|
|
276
|
+
|
|
277
|
+
一句话的形状:
|
|
278
|
+
|
|
279
|
+
```text
|
|
280
|
+
memory_backfill {dryRun: true} → 133 个会话 / 将写入 2078 条情节
|
|
281
|
+
memory_backfill {dryRun: false} → 已写入 2078 条情节;已存在跳过 85 条
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
回填出来的是**情节**,还没变成卡片:蒸馏一次(`memory_distill`、设置页按钮,或让自动蒸馏一天跑一批,默认每批 40 条)才会进检索层。2000 条情节要花的就是 50 批模型调用,这件事该由你决定什么时候花。
|
|
285
|
+
|
|
286
|
+
会话文件是**拼接的 zstd 容器**(每次追加写一帧),所以读它要先把帧切开——`src/history.js` 里有一个和 harness 自己用的算法同构的帧扫描器。
|
|
287
|
+
|
|
267
288
|
## 看一眼你的记忆
|
|
268
289
|
|
|
269
290
|
`memory_stats` 会告诉你库在哪;库里根目录还有一个 **`graph.html`**,双击就能开,不用装东西、不用起服务:
|
|
@@ -376,7 +397,7 @@ Harness 会把运行时快照、技能目录、时间戳、审批通知、压缩
|
|
|
376
397
|
零依赖,不联网,不碰真实记忆库:
|
|
377
398
|
|
|
378
399
|
```bash
|
|
379
|
-
npm test # 43 + 13 + 35 + 12 + 22 + 54 + 15 + 8 + 11 + 19 =
|
|
400
|
+
npm test # 43 + 13 + 35 + 12 + 22 + 54 + 15 + 8 + 11 + 19 + 12 = 244 项检查,另加 25 题检索评测
|
|
380
401
|
npm run eval # 只跑检索评测(离线,确定性的)
|
|
381
402
|
```
|
|
382
403
|
|
|
@@ -385,7 +406,7 @@ npm run eval # 只跑检索评测(离线,确定性的)
|
|
|
385
406
|
- `test/client.mjs` — 把设置页挂在一个够用的 React 替身上跑:页面画出的控件 = 该显示的字段、切换 `transport` 后 http 那几项才出现、空的内联 key 不会被当成"删除"发出去、越界的数字到不了线上、宿主拒绝后清空草稿并报错、「恢复默认」发的是 `unset` 而不是 `set`、只在真的和继承值不同时才打「已自定义」(同一次保存会钉住整行,值等于默认的不算改过)、样式表只有一个带包名标记的 `<style>`、`client.js` 里不许出现 `export`/`import`(loader 按脚本求值);再把 `client.js` 真的放进一个带 `window.__ModuleLoader__` 的假页面里求值,走一遍 factory → `apply` 的浏览器路径(只 `require('react')`、注册 id 是包名、四处 order 分别 25 / 30 / 30、标签跟随页面语言、动作栏是滚动区的兄弟而不是它里面的 sticky 子元素);再验证**侧栏入口 + 主面板**(`sidebar.panellist` 与 `main` 用同一个 id 才连得上、面板里嵌的是那条路由而不是第二套渲染、src 里不带库路径);最后驱动**会话页签**:只按 `sessionId` 请求 `/api/memory-graph.graph`、宿主给的节点和边一个不少地画出来、点卡片在右侧展开标题与正文、点空白收起、空会话和失败的路由都给人话
|
|
386
407
|
- `test/graphdata.mjs` — 会话页签的数据:按 `session` 过滤情节(别的会话一条都不带)、项目多数决、召回查询取最近几轮并截断、布局确定且不越界、每条边都指向存在的节点、跨项目卡片通过共享实体挂进来、上限与截断照实报、assistant 原文不进 payload;路由对"缺 session / 正常 / 库坏掉"三种情况分别回 400 / 200 / 500
|
|
387
408
|
- `test/capture.mjs` — 用真实的 `session/event` 形状驱动采集,验证一轮一情节、密钥被拦、短轮不入库、注入消息不入库
|
|
388
|
-
- `test/tools.mjs` — 逐个跑
|
|
409
|
+
- `test/tools.mjs` — 逐个跑 9 个工具的 `execute` + `render`;另外验证模型要的那次蒸馏走的是共享 runner(而不是绕开它直接调 `distill`),否则它可能和一次自动蒸馏撞在一起
|
|
389
410
|
- `test/page.mjs` — 侧栏面板背后那条路由:`/api/memory-graph.page` 在受认证通道上、空库也给一页而不是报错、**和落到磁盘的 `graph.html` 是同一份渲染**(只差时间戳)、`X-Frame-Options: SAMEORIGIN` 所以只有应用能嵌、库读坏了把原因印在页面上(转义后)、没有库是 500 不是崩溃、页面里不带任何库路径
|
|
390
411
|
- `test/distillapi.mjs` — 「蒸馏一次」那条路:`/api/memory-graph.distill` 在受认证的 `/api` 通道上、跑的是同一个 `distill()`(活库 + 设置里的模型 + 打码规则)、一次按下一次运行、并发第二次被拒(不排队)、抽取器的拒绝是 200 而不是 500、抛错是 500 且锁照样解开、写了卡片才重建视图、渲染失败只 warn、卡片列表截到 40、没有 runner 时回 500 而不是把异常扔给浏览器
|
|
391
412
|
- `test/auto.mjs` — 自动蒸馏:两条规则各自触发、都开时谁先满足谁先跑、最短间隔先于两条规则判断、静默时段跨午夜、tick 只做决定(关着时连库都不读)、一次 tick 最多一次运行、间隔在调用前就记下、重启后读回 `.auto-state.json` 所以不会重跑、跑空了也记间隔、正在跑时不叠加、失败照原因记日志、项目名透传、库读不了是跳过不是崩、阈值大时请求的条数够看到它
|
|
@@ -415,7 +436,7 @@ npm version minor -m "chore(release): v%s" && git push --follow-tags # minor /
|
|
|
415
436
|
|
|
416
437
|
1. 核对 tag 和 `package.json` 的版本号——不一致就失败,不会发出一个版本号对不上的包;
|
|
417
438
|
2. 查这个版本是不是已经在注册表上——在就跳过,重推同一个 tag 是空操作而不是报错;
|
|
418
|
-
3. `npm publish --provenance --access public`(`prepublishOnly` 会再跑一遍那
|
|
439
|
+
3. `npm publish --provenance --access public`(`prepublishOnly` 会再跑一遍那 244 项,红的进不去);
|
|
419
440
|
4. npm 页面上多一条「Built and signed on GitHub Actions」,证明这个 tarball 是从这个仓库的这个提交构建的。
|
|
420
441
|
|
|
421
442
|
凭据有两条路,工作流两条都认(有 `NPM_TOKEN` 就用它,没有就走 OIDC):
|
package/client.js
CHANGED
|
@@ -195,7 +195,7 @@ const SECTIONS = [
|
|
|
195
195
|
path: ['extractor', 'provider'],
|
|
196
196
|
type: 'text',
|
|
197
197
|
placeholder: 'deepseek / wb / agy-local',
|
|
198
|
-
label: t('provider', 'provider'),
|
|
198
|
+
label: t('模型提供方 provider', 'provider'),
|
|
199
199
|
hint: t('只有关掉"跟随当前模型"才需要。', 'Only needed when "follow the current model" is off.'),
|
|
200
200
|
visible: (value) =>
|
|
201
201
|
getPath(value, ['extractor', 'transport']) === 'harness' && getPath(value, ['extractor', 'useCurrentModel']) === false,
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ilaomianyang/dsh-memory-graph",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.11.0",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"author": "ISKING <admin@isking.cc>",
|
|
6
6
|
"repository": {
|
|
@@ -34,6 +34,7 @@
|
|
|
34
34
|
"client.js",
|
|
35
35
|
"src/*.js",
|
|
36
36
|
"test/*.mjs",
|
|
37
|
+
"test/fixtures/*",
|
|
37
38
|
"eval/*.mjs",
|
|
38
39
|
"eval/questions.json",
|
|
39
40
|
"eval/REPORT.md",
|
|
@@ -47,7 +48,7 @@
|
|
|
47
48
|
"LICENSE"
|
|
48
49
|
],
|
|
49
50
|
"scripts": {
|
|
50
|
-
"test": "node test/smoke.mjs && node test/schema.mjs && node test/client.mjs && node test/capture.mjs && node test/tools.mjs && node test/extract.mjs && node test/graphdata.mjs && node test/page.mjs && node test/distillapi.mjs && node test/auto.mjs && node eval/run.mjs",
|
|
51
|
+
"test": "node test/smoke.mjs && node test/schema.mjs && node test/client.mjs && node test/capture.mjs && node test/tools.mjs && node test/extract.mjs && node test/graphdata.mjs && node test/page.mjs && node test/distillapi.mjs && node test/auto.mjs && node test/history.mjs && node eval/run.mjs",
|
|
51
52
|
"eval": "node eval/run.mjs",
|
|
52
53
|
"live": "node test/live.mjs",
|
|
53
54
|
"live:supersede": "node test/live-supersede.mjs",
|
|
@@ -74,6 +75,7 @@
|
|
|
74
75
|
},
|
|
75
76
|
"license": "MIT",
|
|
76
77
|
"dependencies": {
|
|
77
|
-
"@deepseek-ai/schemastery": "~3.18.5"
|
|
78
|
+
"@deepseek-ai/schemastery": "~3.18.5",
|
|
79
|
+
"fzstd": "^0.1.1"
|
|
78
80
|
}
|
|
79
81
|
}
|
package/src/auto.js
CHANGED
|
@@ -96,7 +96,7 @@ export function createAutoDistiller({ cfg, vault, run, log = () => {}, now = ()
|
|
|
96
96
|
lastAttemptAt = Number.isFinite(attempt) && attempt > 0 ? attempt : null
|
|
97
97
|
startedAt = Number.isFinite(start) && start > 0 ? start : null
|
|
98
98
|
} catch (error) {
|
|
99
|
-
log('warn',
|
|
99
|
+
log('warn', `自动蒸馏:读不了 .auto-state.json · auto: could not read .auto-state.json: ${error?.message ?? error}`)
|
|
100
100
|
}
|
|
101
101
|
}
|
|
102
102
|
|
|
@@ -111,7 +111,7 @@ export function createAutoDistiller({ cfg, vault, run, log = () => {}, now = ()
|
|
|
111
111
|
const batch = await vault.readUndistilled(Math.max(1, auto.minEpisodes ?? 0, cfg.extractor?.maxEpisodesPerRun ?? 40))
|
|
112
112
|
pending = Array.isArray(batch) ? batch.length : 0
|
|
113
113
|
} catch (error) {
|
|
114
|
-
log('warn',
|
|
114
|
+
log('warn', `自动蒸馏:数不了还没蒸馏的情节 · auto: could not count undistilled episodes: ${error?.message ?? error}`)
|
|
115
115
|
return { due: false, rule: null, reason: 'the vault could not be read', ran: false }
|
|
116
116
|
}
|
|
117
117
|
|
|
@@ -126,7 +126,7 @@ export function createAutoDistiller({ cfg, vault, run, log = () => {}, now = ()
|
|
|
126
126
|
try {
|
|
127
127
|
await vault.writeAutoState?.({ startedAt: at })
|
|
128
128
|
} catch (error) {
|
|
129
|
-
log('warn',
|
|
129
|
+
log('warn', `自动蒸馏:没能启动计划时钟 · auto: could not start the schedule clock: ${error?.message ?? error}`)
|
|
130
130
|
}
|
|
131
131
|
}
|
|
132
132
|
|
|
@@ -139,21 +139,24 @@ export function createAutoDistiller({ cfg, vault, run, log = () => {}, now = ()
|
|
|
139
139
|
try {
|
|
140
140
|
await vault.writeAutoState?.({ lastAttemptAt: at })
|
|
141
141
|
} catch (error) {
|
|
142
|
-
log('warn',
|
|
142
|
+
log('warn', `自动蒸馏:没能记下这次尝试 · auto: could not record the attempt: ${error?.message ?? error}`)
|
|
143
143
|
}
|
|
144
144
|
|
|
145
145
|
const report = await run({ project: auto.project ? String(auto.project) : undefined })
|
|
146
146
|
lastReport = report
|
|
147
147
|
if (report?.busy === true) {
|
|
148
|
-
log('info', 'auto: skipped, a distillation is already running')
|
|
148
|
+
log('info', '自动蒸馏:跳过,已经有一次蒸馏在跑 · auto: skipped, a distillation is already running')
|
|
149
149
|
} else if (report?.ok === true) {
|
|
150
150
|
log(
|
|
151
151
|
'info',
|
|
152
|
-
|
|
152
|
+
`自动蒸馏 auto (${decision.rule}): read ${report.episodes} episode(s) → wrote ${report.written} card(s)` +
|
|
153
153
|
(report.model ? ` · ${report.model}` : ''),
|
|
154
154
|
)
|
|
155
155
|
} else {
|
|
156
|
-
log(
|
|
156
|
+
log(
|
|
157
|
+
'warn',
|
|
158
|
+
`自动蒸馏 auto (${decision.rule}): ${report?.message ?? report?.reason ?? '没有运行 · did not run'}`,
|
|
159
|
+
)
|
|
157
160
|
}
|
|
158
161
|
return { ...decision, pending, ran: true, report }
|
|
159
162
|
}
|
package/src/capture.js
CHANGED
|
@@ -4,117 +4,24 @@
|
|
|
4
4
|
* Deliberately model-free. Every turn is appended as it closes, so the "never
|
|
5
5
|
* loses anything" guarantee holds even while the extractor is unconfigured,
|
|
6
6
|
* offline, or out of quota. Distillation happens later and separately.
|
|
7
|
+
*
|
|
8
|
+
* What counts as a real message, how it is truncated and how it is redacted
|
|
9
|
+
* lives in `turns.js`, shared with the history importer — see the note there for
|
|
10
|
+
* why there is exactly one copy of that logic.
|
|
7
11
|
*/
|
|
8
|
-
import {
|
|
12
|
+
import { createTurnAssembler, projectOf } from './turns.js'
|
|
9
13
|
|
|
10
14
|
const FLUSH_AFTER_MS = 90_000
|
|
11
|
-
const MAX_ASSISTANT_PARTS = 6
|
|
12
|
-
|
|
13
|
-
function collectText(node, depth = 0, seen = new Set(), out = []) {
|
|
14
|
-
if (node === null || node === undefined || depth > 4) return out
|
|
15
|
-
if (typeof node === 'string') return out
|
|
16
|
-
if (Array.isArray(node)) {
|
|
17
|
-
for (const item of node) collectText(item, depth + 1, seen, out)
|
|
18
|
-
return out
|
|
19
|
-
}
|
|
20
|
-
if (typeof node !== 'object') return out
|
|
21
|
-
if (seen.has(node)) return out
|
|
22
|
-
seen.add(node)
|
|
23
|
-
|
|
24
|
-
if (node.type === 'text' && typeof node.text === 'string') {
|
|
25
|
-
out.push(node.text)
|
|
26
|
-
return out
|
|
27
|
-
}
|
|
28
|
-
for (const key of ['content', 'message', 'text', 'parts', 'blocks']) {
|
|
29
|
-
if (key in node) collectText(node[key], depth + 1, seen, out)
|
|
30
|
-
}
|
|
31
|
-
return out
|
|
32
|
-
}
|
|
33
|
-
|
|
34
|
-
function eventText(event) {
|
|
35
|
-
const parts = collectText(event?.data)
|
|
36
|
-
return parts.join('\n').replace(/\n{3,}/g, '\n\n').trim()
|
|
37
|
-
}
|
|
38
|
-
|
|
39
|
-
function projectOf(session, fallback) {
|
|
40
|
-
const cwd = session?.header?.cwd ?? session?.meta?.cwd ?? session?.cwd
|
|
41
|
-
if (typeof cwd === 'string' && cwd.trim() !== '') {
|
|
42
|
-
// Both separators, whatever platform this is running on: the path comes from
|
|
43
|
-
// the session header, which may well have been written on the other OS.
|
|
44
|
-
const parts = cwd.replace(/[\\/]+$/, '').split(/[\\/]/)
|
|
45
|
-
const base = parts[parts.length - 1] ?? ''
|
|
46
|
-
if (base !== '') return base
|
|
47
|
-
}
|
|
48
|
-
return fallback
|
|
49
|
-
}
|
|
50
|
-
|
|
51
|
-
function truncate(text, max) {
|
|
52
|
-
const value = String(text ?? '').trim()
|
|
53
|
-
if (value.length <= max) return value
|
|
54
|
-
return `${value.slice(0, max)}…`
|
|
55
|
-
}
|
|
56
15
|
|
|
57
16
|
export function registerCapture(ctx, { cfg, vault, redactor }) {
|
|
58
|
-
const
|
|
17
|
+
const assembler = createTurnAssembler({ cfg, redactor })
|
|
59
18
|
const disposers = []
|
|
60
19
|
const timers = new Map()
|
|
61
20
|
|
|
62
|
-
// Both the switch and the skip list are editable while the plugin runs (they
|
|
63
|
-
// are `.volatile()` in the schema), so neither is baked in here: the switch is
|
|
64
|
-
// read per event, and the compiled rules are rebuilt only when the list's text
|
|
65
|
-
// actually changes — a `new RegExp` on every token would be pure waste.
|
|
66
|
-
let skipKey
|
|
67
|
-
let skipRules = []
|
|
68
|
-
|
|
69
|
-
function compiledSkipRules() {
|
|
70
|
-
const list = cfg.capture.skipPatterns ?? []
|
|
71
|
-
const key = list.join('\u0000')
|
|
72
|
-
if (key !== skipKey) {
|
|
73
|
-
skipKey = key
|
|
74
|
-
skipRules = list.flatMap((pattern) => {
|
|
75
|
-
try {
|
|
76
|
-
return [new RegExp(pattern, 'i')]
|
|
77
|
-
} catch {
|
|
78
|
-
return []
|
|
79
|
-
}
|
|
80
|
-
})
|
|
81
|
-
}
|
|
82
|
-
return skipRules
|
|
83
|
-
}
|
|
84
|
-
|
|
85
|
-
/** True when the "user" side is really a harness injection, not a person typing. */
|
|
86
|
-
const injected = (text) => text !== '' && compiledSkipRules().some((re) => re.test(text))
|
|
87
|
-
|
|
88
|
-
/**
|
|
89
|
-
* Delimited harness blocks come off before the patterns are consulted.
|
|
90
|
-
*
|
|
91
|
-
* They are the one kind of injection whose extent is knowable, so removing them
|
|
92
|
-
* keeps the person's own words when a real message happens to carry one — the
|
|
93
|
-
* alternative (dropping the whole message) loses the part that was worth
|
|
94
|
-
* keeping. Undelimited blocks (a runtime snapshot) are handled by the patterns.
|
|
95
|
-
*/
|
|
96
|
-
function stripInjectedBlocks(text) {
|
|
97
|
-
return String(text ?? '')
|
|
98
|
-
.replace(/<system-reminder>[\s\S]*?<\/system-reminder>/gi, '')
|
|
99
|
-
.replace(/[ \t]+\n/g, '\n')
|
|
100
|
-
.replace(/\n{3,}/g, '\n\n')
|
|
101
|
-
.trim()
|
|
102
|
-
}
|
|
103
|
-
|
|
104
21
|
const idOf = (session) => String(session?.id ?? session?.header?.id ?? 'unknown')
|
|
105
22
|
|
|
106
|
-
function bufferOf(session) {
|
|
107
|
-
const id = idOf(session)
|
|
108
|
-
let buffer = buffers.get(id)
|
|
109
|
-
if (!buffer) {
|
|
110
|
-
buffer = { project: projectOf(session, cfg.defaultProject), user: '', assistant: [], tools: 0, turn: null }
|
|
111
|
-
buffers.set(id, buffer)
|
|
112
|
-
}
|
|
113
|
-
return { id, buffer }
|
|
114
|
-
}
|
|
115
|
-
|
|
116
23
|
function drop(id) {
|
|
117
|
-
|
|
24
|
+
assembler.forget(id)
|
|
118
25
|
const timer = timers.get(id)
|
|
119
26
|
if (timer) {
|
|
120
27
|
clearTimeout(timer)
|
|
@@ -123,33 +30,9 @@ export function registerCapture(ctx, { cfg, vault, redactor }) {
|
|
|
123
30
|
}
|
|
124
31
|
|
|
125
32
|
async function flush(id) {
|
|
126
|
-
const
|
|
127
|
-
if (!buffer) return
|
|
128
|
-
const toolCount = buffer.tools
|
|
129
|
-
const user = truncate(buffer.user, cfg.capture.userChars)
|
|
130
|
-
const assistant = truncate(buffer.assistant.join('\n'), cfg.capture.assistantChars)
|
|
131
|
-
|
|
132
|
-
// Every episode is anchored to something the user actually said. A turn with
|
|
133
|
-
// no user side is context noise, and one too short to mean anything — such as
|
|
134
|
-
// "ok" — would only fill the vault with lines the extractor cannot use.
|
|
135
|
-
if (user === '' || user.length + assistant.length < cfg.capture.minChars) {
|
|
136
|
-
drop(id)
|
|
137
|
-
return
|
|
138
|
-
}
|
|
139
|
-
|
|
33
|
+
const episode = assembler.flush(id)
|
|
140
34
|
drop(id)
|
|
141
|
-
|
|
142
|
-
const redactedUser = redactor.redact(user).text
|
|
143
|
-
const redactedAssistant = redactor.redact(assistant).text
|
|
144
|
-
await vault.appendEpisode({
|
|
145
|
-
t: nowIso(),
|
|
146
|
-
project: buffer.project,
|
|
147
|
-
session: id,
|
|
148
|
-
turn: buffer.turn,
|
|
149
|
-
user: redactedUser,
|
|
150
|
-
assistant: redactedAssistant,
|
|
151
|
-
...(cfg.capture.includeToolOutput && toolCount > 0 ? { tools: toolCount } : {}),
|
|
152
|
-
})
|
|
35
|
+
if (episode) await vault.appendEpisode(episode)
|
|
153
36
|
}
|
|
154
37
|
|
|
155
38
|
function schedule(id) {
|
|
@@ -164,65 +47,42 @@ export function registerCapture(ctx, { cfg, vault, redactor }) {
|
|
|
164
47
|
|
|
165
48
|
disposers.push(
|
|
166
49
|
ctx.on('session/event', (session, event) => {
|
|
167
|
-
// The switch is read
|
|
168
|
-
// 设置 must stop writes immediately without unregistering the listener.
|
|
50
|
+
// The switch is read inside the assembler, per event: turning capture off
|
|
51
|
+
// in 设置 must stop writes immediately without unregistering the listener.
|
|
169
52
|
if (!cfg.capture.enabled) return
|
|
170
53
|
|
|
171
|
-
const
|
|
54
|
+
const id = idOf(session)
|
|
55
|
+
const episodes = assembler.push({
|
|
56
|
+
sessionId: id,
|
|
57
|
+
project: projectOf(session, cfg.defaultProject),
|
|
58
|
+
event,
|
|
59
|
+
})
|
|
172
60
|
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
const id = idOf(session)
|
|
177
|
-
const buffer = buffers.get(id)
|
|
178
|
-
if (buffer) {
|
|
179
|
-
if (typeof event?.data?.turn === 'number') buffer.turn = event.data.turn
|
|
180
|
-
flush(id).catch(() => {})
|
|
61
|
+
if (episodes.length > 0) {
|
|
62
|
+
for (const episode of episodes) {
|
|
63
|
+
vault.appendEpisode(episode).catch(() => {})
|
|
181
64
|
}
|
|
182
|
-
return
|
|
183
65
|
}
|
|
184
66
|
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
// approval notices and compaction checkpoints through the same channel as
|
|
190
|
-
// the person typing — and not always with a telling `source.kind`: a real
|
|
191
|
-
// runtime snapshot has arrived here tagged `user` (it was captured as if
|
|
192
|
-
// the person had typed it). So provenance can only *add* a skip, never
|
|
193
|
-
// veto the patterns; both are consulted, and delimited blocks come off
|
|
194
|
-
// first so a message that merely carried a reminder keeps its own words.
|
|
195
|
-
const kind = event?.data?.source?.kind
|
|
196
|
-
const text = stripInjectedBlocks(eventText(event))
|
|
197
|
-
const synthetic = (typeof kind === 'string' && kind !== 'user') || text === '' || injected(text)
|
|
198
|
-
if (synthetic) return
|
|
67
|
+
const type = event?.type
|
|
68
|
+
const relevant =
|
|
69
|
+
type === 'turn/end' || type === 'user/message' || type === 'assistant/message' || type === 'tool/result'
|
|
70
|
+
if (!relevant) return
|
|
199
71
|
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
72
|
+
// A turn is the natural write unit: flush it as soon as it closes, so an
|
|
73
|
+
// episode exists even if the next turn never comes. The timer is the safety
|
|
74
|
+
// net for a turn that never closes at all.
|
|
75
|
+
if (type === 'turn/end') {
|
|
76
|
+
const timer = timers.get(id)
|
|
77
|
+
if (timer) {
|
|
78
|
+
clearTimeout(timer)
|
|
79
|
+
timers.delete(id)
|
|
205
80
|
}
|
|
206
|
-
|
|
207
|
-
const { buffer } = bufferOf(session)
|
|
208
|
-
buffer.user = text
|
|
209
|
-
if (typeof event?.data?.turn === 'number') buffer.turn = event.data.turn
|
|
210
|
-
schedule(id)
|
|
81
|
+
assembler.forget(id)
|
|
211
82
|
return
|
|
212
83
|
}
|
|
213
84
|
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
if (type === 'assistant/message') {
|
|
217
|
-
const text = eventText(event)
|
|
218
|
-
if (text !== '' && buffer.assistant.length < MAX_ASSISTANT_PARTS) buffer.assistant.push(text)
|
|
219
|
-
if (typeof event?.data?.turn === 'number') buffer.turn = event.data.turn
|
|
220
|
-
schedule(id)
|
|
221
|
-
return
|
|
222
|
-
}
|
|
223
|
-
|
|
224
|
-
buffer.tools += 1
|
|
225
|
-
schedule(id)
|
|
85
|
+
if (assembler.isOpen(id)) schedule(id)
|
|
226
86
|
}),
|
|
227
87
|
)
|
|
228
88
|
|
|
@@ -231,7 +91,9 @@ export function registerCapture(ctx, { cfg, vault, redactor }) {
|
|
|
231
91
|
() => {
|
|
232
92
|
for (const timer of timers.values()) clearTimeout(timer)
|
|
233
93
|
timers.clear()
|
|
234
|
-
for (const
|
|
94
|
+
for (const episode of assembler.flushAll()) {
|
|
95
|
+
vault.appendEpisode(episode).catch(() => {})
|
|
96
|
+
}
|
|
235
97
|
},
|
|
236
98
|
]
|
|
237
99
|
}
|
package/src/distillapi.js
CHANGED
|
@@ -23,7 +23,10 @@ export function createDistillHandler({ getRunner }) {
|
|
|
23
23
|
return async function handleDistill() {
|
|
24
24
|
const runner = getRunner()
|
|
25
25
|
if (typeof runner?.run !== 'function') {
|
|
26
|
-
return json(
|
|
26
|
+
return json(
|
|
27
|
+
{ ok: false, reason: 'unavailable', message: '蒸馏器不可用 · the distiller is not available' },
|
|
28
|
+
500,
|
|
29
|
+
)
|
|
27
30
|
}
|
|
28
31
|
return respond(await runner.run())
|
|
29
32
|
}
|
package/src/distillrun.js
CHANGED
|
@@ -42,7 +42,11 @@ export function createDistillRunner(options) {
|
|
|
42
42
|
*/
|
|
43
43
|
function run({ project, dryRun = false, signal } = {}) {
|
|
44
44
|
if (inflight) {
|
|
45
|
-
return Promise.resolve({
|
|
45
|
+
return Promise.resolve({
|
|
46
|
+
ok: false,
|
|
47
|
+
busy: true,
|
|
48
|
+
reason: '已有一次蒸馏在跑,等它结束 · A distillation is already running.',
|
|
49
|
+
})
|
|
46
50
|
}
|
|
47
51
|
|
|
48
52
|
const task = (async () => {
|
|
@@ -66,7 +70,7 @@ export function createDistillRunner(options) {
|
|
|
66
70
|
try {
|
|
67
71
|
await refreshViews()
|
|
68
72
|
} catch (error) {
|
|
69
|
-
log('warn',
|
|
73
|
+
log('warn', `蒸馏完成了,但有一个记忆库视图没能渲染出来 · distilled, but a vault view did not render: ${error?.message ?? error}`)
|
|
70
74
|
}
|
|
71
75
|
}
|
|
72
76
|
return {
|
|
@@ -85,7 +89,7 @@ export function createDistillRunner(options) {
|
|
|
85
89
|
reason: report.reason ?? null,
|
|
86
90
|
}
|
|
87
91
|
} catch (error) {
|
|
88
|
-
log('warn',
|
|
92
|
+
log('warn', `蒸馏失败 · distillation failed: ${error?.message ?? error}`)
|
|
89
93
|
return {
|
|
90
94
|
ok: false,
|
|
91
95
|
busy: false,
|