@kira905/dsh-memory-toolkit 0.1.3 → 0.2.0

package/CHANGELOG.md

@@ -3,6 +3,98 @@
 本仓的版本按**语义化版本**：改结论/改契约升主版本，加能力升次版本，修错别字与文档订正不升。
 英文摘要见每节末尾的 *English* 行。
 
+## v0.2.0 — 2026-10-31
+
+**41 个业务件独立重写**（能力与对外接口不变，实现换成自己的）：
+
+`memory-engine/lib`(21) + `memory-engine/cli`(11) + `memory-engine/tools`(3) + 仓根 `lib`(6) = **41 件**
+从"生产实现的逐字副本"改成**独立实现**。判据是**行级同源度**（同名可比件里，本仓文件的非空行有多少能在参照件中找到，多重集扣减）：
+
+- 41 件均值：**99.2% → 24.6%**；
+- **≥5KB 的 28 件全部 < 50%**（最高 47.5%，最低 12.5%）；
+- 全仓「≥50% 行同占比」：**99% → 64%**（残余面 = `dsh-memory-local/`、`memory-db-tools/`、`memory-engine/test/`，这三块不在本版范围内）。
+
+**对外接口零变化**：
+
+- `memory-engine/test` 的 184 项回归全绿，测试文件一行未改（这就是"没改坏行为"的唯一证据）；
+- 41 件导出面逐件比对：导出名零缺零多，常量值逐值不变；
+- CLI 的 `--flag` 名、缺省值、stdout/stderr 分工、退出码语义不变（改前改后逐用例对拍）。
+
+**顺带清掉内部标识**：卡号、棒号、设备别名、注释里的私有盘符路径与内部任务代号；
+7 处保留的是**契约字符串**（`EXCLUDE_DIRS` 的四类目录名、`deploy-guard` 的忽略目录表），改了会破行为。
+
+逐件台账见仓根 `k9-progress.jsonl`（41 行：before/after sha + 两次数值 + 实现形态说明）。
+
+*English*: 41 business modules rewritten as independent implementations — same capability and public
+interface, different code. Line-level shared-source ratio 99.2% → 24.6% on average; every file ≥5 KB is
+below 50%. The 184-case regression suite passes unchanged, the export surface is identical, and CLI
+flags / defaults / exit codes are unchanged. Internal identifiers were stripped; the seven remaining
+hits are contractual strings (the excluded-directory list and the guard's ignore table).
+
+### 召回链：降级 / 词表第二路 / 命中留痕（K4）
+
+- **查询截断**：引擎侧新增 256 字符上限（原来引擎侧**完全不截**；生产侧那道 512 是插件注入器的配置，不在引擎里）。按码点截，`stats.queryChars / queryCharsUsed / queryTruncated` 如实记录。
+- **编码降级**：编码带预算（默认 1500 ms）。超时 / 抛错 / 空向量 ⇒ 降级成**固定 Top-K**（默认 8 条，按最近改动取，仍排除 pinned），结果带 `degraded.mode = "fixed-topk"` 与 `encodeError = { name, ms }`。**绝不返回空、绝不把异常往上抛**。
+- **词表第二路**（新文件 `lib/lexicon-recall.mjs`）：jieba 词典最长匹配 + **BM25**（tf 饱和 / 长度归一 / 语料 idf × 通用词具体度）+ 进程内缓存与后台预热，与向量、FTS 一起进 RRF(k=60)。开关 = `dict/lexicon-config.json#enabled`（出厂 **false**）/ `MEMORY_ENGINE_LEXICON` / `--lexicon on|off`。关掉时结果与改动前**逐字一致**（夹具对拍 4 条查询 + CLI 旧新对拍）。
+- **命中留痕**（新文件 `lib/recall-log.mjs`）：旁路 JSONL，字段 = 时刻 / query / 命中 id / 分数 / 来源 / 是否降级 / 耗时；**不存记忆正文**。默认不写，给目录才写；`DSH_MEMORY_RECALL_LOG=off` 一键关回；字节（512 KB）→ 轮转（3 代）→ 天数（14 天）三重封口，`keepFiles: 0` 只留当前文件。
+- **词典素材**：`memory-engine/dict/`（`词典.txt` 1.03 MB + `词频.txt` 219 KB + MIT 许可声明 + 开关文件），合计约 **1.25 MB**；`npm pack --dry-run` 实测 tarball **1.1 MB** / 解包 **2.8 MB** / **140** 个文件。素材目录可用 `MEMORY_ENGINE_DICT_DIR` 改位；缺素材时这一路优雅退化为纯向量。
+- **验证**：`memory-engine/test` 184 项**一行未改、全绿**；新增 `tools/verify-recall-chain.mjs`（15 项自带判据，给 `--baseline <改动前 lib>` 则 17 项）；两个新文件行级同源度 **18.5% / 19.5%**。
+- **接口**：只有新增 —— CLI 4 个开关（`--lexicon` / `--lexicon-top-k` / `--lexicon-floor` / `--fallback-top-k` / `--encode-timeout-ms` / `--query-max-chars` / `--recall-log` / `--no-recall-log`）、`recall()`/`recallMulti()` 的新可选入参、`stats` 新字段、`memory-source.recent()`；既有参数、缺省值、退出码语义不变。
+
+*English*: recall chain hardening — 256-char query truncation; an encoding time budget with a **fixed top-K degradation** (never empty, never throws); a lexicon/BM25 second path fused through RRF (off by default, one-line switch, byte-identical results when off); and an opt-in side-channel JSONL recall log with byte/generation/age caps. Dictionary assets (~1.25 MB, jieba · MIT) ship in `memory-engine/dict/`; `npm pack` measures 1.1 MB tarball / 2.8 MB unpacked / 140 files. The 184 existing tests are unchanged and green, and `tools/verify-recall-chain.mjs` adds 15 acceptance checks. Interface changes are additive only.
+
+### 分层软淘汰 + 号段哨兵（K5，净新增件）
+
+记忆只能删、不能降级；多台机共用一个库时 id 号段越界没人喊 —— 各给一件，**出厂全是干跑 / 只读**。
+
+- **`layer` 字段 + 迁移**（新文件 `lib/layers.mjs` / `lib/layer-migrate.mjs` / `cli/layer-migrate.mjs`）：加一列 `layer`，存量行回填缺省层。
+  **零删行**：行数 / `sqlite_sequence` / `PRAGMA integrity_check` 三项改完复核，任一不符就报错并指向备份。
+  **幂等**：列已在、空值已清零 ⇒ 第二次 `--apply` 库文件逐字节不变。`--apply` **必须**同时给 `--db` 与 `--backup`；`--restore <备份>` 默认干跑，真还原走「同目录临时文件 + sha 校验 + rename」，并先摘掉陈旧 `-wal`/`-shm`。
+- **软淘汰执行器**（新文件 `lib/layer-move.mjs` / `cli/layer-move.mjs`）：整层搬进同库边表 `memories_evicted`，一条命令出去、一条命令回来，**一条不删**。
+  硬判据 = 搬出行数 + 剩下行数 == 搬前总行数（事务内断言）；每行带 `content_sha`，搬回前先核指纹，边表被改过就整批拒动。配套 `--dump` / `--verify` 做逐行对拍；会搬空主表时默认拒跑（`--force` 才动）。
+  代价写进 README：搬走的行仍占库体积、边表没有 FTS 索引。
+- **号段哨兵**（新文件 `lib/band-sentinel.mjs` / `cli/band-sentinel.mjs`）：`sqlite_sequence` 主判 + 「更高位段有行」才判红；**更低位的别台段行只披露不判红**（生产件踩过的假红，没改回去）。
+  **单机退化**：配置里只有一条 band 就只读本机、不 ssh；多机才走配置里的 `remote { ssh, node, db }`。**任一远端读不到 ⇒ 结论写「未探测 / 不可达」，退出码不是 4**（4 只留给本机库读不到）；`--require-all` 才把未探测算失败（rc=5）。
+- **配置驱动**：`memory-engine/config/layers.json`（层名集合 / 列名 / 缺省层）与 `memory-engine/config/bands.json`（号段表）。代码里没有写死的层名，也没有任何机器名。
+- **CLI 惯例**：新增三个命令全部 `--help` 可读、未知参数 fail-closed（rc=2）、默认 dry-run、`--json` **只落文件不上 stdout**、`isMainEntry` 守卫走 `realpathSync` 双方比。
+- **验证**：`memory-engine/test` **既有 184 项一行未改、全绿**；新增 `test/12-layering.test.mjs` **21 项**（干跑零写 / 加列回填 / 幂等 / 备份还原 / 搬出守恒 / 搬回逐行指纹一致 / 边表被改整批拒动 / 号段红绿对照 / 远端读不到不许 rc=4）。新增 7 个代码件对生产同名件的行级同源度全部 < 50%（逐件读数见仓根 `k5-progress.jsonl`）。
+
+*English*: layered soft eviction + id-band sentinel (net-new). `layer-migrate` adds a `layer` column and backfills the default layer — zero row deletion, idempotent (second `--apply` is byte-identical), requires an explicit `--db` and a `--backup`, and `--restore` is dry-run by default. `layer-move` moves a whole layer into the in-database side table `memories_evicted` and back, with the hard invariant `moved + remaining == before` asserted in-transaction and per-row `content_sha` verified before restoring. `band-sentinel` checks the `sqlite_sequence`-derived next id against a config-driven band table, judges rows in higher bands red, only *discloses* rows in lower foreign bands, degrades to a single-machine check with no ssh, and never returns 4 for an unreachable peer. All three CLIs are `--help`-readable, fail closed on unknown flags, default to dry-run, and write JSON reports to files only. The 184 existing tests are unchanged and green; `test/12-layering.test.mjs` adds 21 checks.
+
+### 会话日志 → 记忆（K6，净新增件）
+
+记忆的**输入面**补上第二条：一段会话日志进来，产出 ≤N 条可追溯、可整批撤回的结论条。**旧入口（`distill.mjs` / `distill-rollback.mjs` 及其 CLI）的默认值、参数名、退出码一个没改**。
+
+- **会话适配层**（新文件 `lib/session-source.mjs`）：只挑能回指来源的结论句（决定 / 口径 / 偏好 / 修复方案），
+  每条带 `sessionId` + 行号 + 行首字节偏移 + 原文前 48 字锚。寒暄 / 疑问 / 过程叙述 / 推测 / 工具输出 / 超长转述
+  **一律不抽**，逐类计数后一次性报出来。抽的是原句，不做摘要式改写。
+- **新入口**（新文件 `cli/distill-sessions.mjs`）：`--source <jsonl>` → 抽句 → 预算闸 → 一次模型调用 → `VACUUM INTO` 快照 → 写回。
+  缺省 **dry-run**（不调模型、不写库、不写台账）；未知参数 fail-closed（rc=2）；`--json` 必须配 `--out`，**只落文件不上 stdout**。
+- **预算闸（这条是核心）**：没给 `--max-tokens`、台账里也没配 ⇒ **一次模型都不调**，打印「未配置每日预算 ⇒ 本轮不跑（这是故意的）」，rc=0。
+  **不继承**内核 `DAILY_TOKEN_CAP = 150000`，也不"没配 = 无限跑"；当日已超或本次估算会超，同样不跑并打印用量。
+  实测（deepseek-flash，三轮夹具，每轮一次调用）**424 / 424 / 434 token**（合计 1311，prompt 1044 / completion 267），
+  建议 `--max-tokens 20000`（约 19 轮/天按估算、~45 轮/天按实测）。
+- **写回与撤回**：标签 = `distilled src=<批次键> srcids=<行号,...> date=<YYYYMMDD> sess=<场次> run=session:<批次 id>`；
+  一轮一个来源键 ⇒ 撤回直接用既有 `distill-rollback.mjs --key <批次键> --apply`，一批一条命令整组删（不新写撤回器）。
+  写库前快照拍不成 ⇒ **不写库**（fail-closed）。
+- **压缩输入**：`.jsonl.zstd` 走 node 内置 zstd **逐帧**解（DSH 日志一次追加一帧，多帧文件），解不开给人话错误而不是静默失败；
+  当前 node 没有 zstd API 时同样给"请先解压"的提示。
+- **验证**：`memory-engine/test` **既有 205 项一行未改、全绿**；新增 `test/13-session-distill.test.mjs` **40 项**
+  （抽句回指对得上原文 / zstd 单帧多帧坏帧 / 预算闸三态 / 快照 fail-closed / 模型失败也记账 / 幂等 / 整批撤回 /
+  CLI 未知参数与 `--json` 分工）。两个新件对生产同名件的行级同源度见仓根 `k6-progress.jsonl`。
+
+*English*: session logs → memories (net-new). `lib/session-source.mjs` extracts only conclusion sentences that can be
+pointed back to their source (session id + line + byte offset + 48-char verbatim anchor), and drops greetings,
+questions, process narration, speculation, tool output and over-long passages while counting each rejected class.
+`cli/distill-sessions.mjs` is a new entry point that extracts, applies the budget gate, makes exactly one model call,
+takes a `VACUUM INTO` snapshot and writes back — dry-run by default, unknown flags fail closed, and `--json` writes to
+a file only, never to stdout. **No daily budget configured ⇒ the model is not called at all** (rc=0, and it prints
+that this is deliberate); it never inherits the kernel's 150000 cap and never treats "unconfigured" as unlimited.
+Measured over three fixture sessions: 424 / 424 / 434 tokens per round (1311 total), suggesting `--max-tokens 20000`.
+Each run maps to one source key, so reverting is the existing `distill-rollback.mjs --key <batch key> --apply`.
+Compressed `.jsonl.zstd` input is decoded frame by frame with node's built-in zstd; failures raise a plain-language
+error instead of returning partial data. The 205 existing tests are unchanged and green; the new test file adds 40 checks.
+
 ## v0.1.3 — 2026-10-30
 
 **主包补 bundle 声明 + 子路径导出面**（修「装完不生效」；无插件行为变化）：

package/README.en.md

@@ -14,12 +14,231 @@ fixable; the plugin wires it into a session.**
 
 | Directory | What it is | Form |
 |---|---|---|
-| [`memory-engine/`](./memory-engine/README.md) | Memory engine (offline layer): standalone vector store · incremental sync · hash-rotation fallback sweep · hybrid recall (vector + FTS5 + RRF) · document index · startup self-check · local encoding bridge | plain Node ESM + `node:sqlite` |
+| [`memory-engine/`](./memory-engine/README.md) | Memory engine (offline layer): standalone vector store · incremental sync · hash-rotation fallback sweep · hybrid recall (vector + FTS5 + lexicon BM25 + RRF + degraded fixed-top-K + side-channel recall log) · document index · startup self-check · local encoding bridge | plain Node ESM + `node:sqlite` |
 | [`memory-db-tools/`](./memory-db-tools/README.md) | Memory database toolkit: read-only export / six-way audit analysis / structure probe / feature-word probe / persona overlap / board-ledger probe / zero-write proof / new-session briefing | a set of standalone CLIs |
 | [`dsh-memory-local/`](./dsh-memory-local/README.md) | Local memory plugin: a self-maintained fork of upstream `dsh-memory` (cross-session memory + recall prompt section + pointer injection), with build / idempotent deploy / one-command rollback / post-deploy self-check | DSH plugin |
 | [`persona-memory-audit.mjs`](./persona-memory-audit.mjs) | Persona ↔ memory audit: pins down "persona body size" and "memory visibility" and reports red past thresholds | single-file CLI |
 | [`lib/`](./lib) | Zero-dependency shared bits (argument allow-list / HTTP liveness judgement / zstd frames / session file names / pitfall scoring) | pure modules |
 
+## Shared-source notice
+
+Some files in this repository **share a source with an internal production implementation**
+(the same engineering effort, evolved over time, with two landing places).
+
+- **How much**: measured by the **line-level shared-source ratio** (among same-named comparable files, how
+  many non-empty lines of this repository's files are found in the reference files, multiset-deducted); this
+  repository's "≥50% line-share" figure is **64%**.
+- **What that means**: this repository is a **distilled export** of that implementation — **neither a verbatim
+  copy nor a fully rewritten independent implementation**. Before reading any single file as an "independent
+  reference implementation", run the check below.
+- **How to re-run it**: the shared-source scanner ships with `dsh-quality-toolkit` as `dsh-sim-scan.mjs`
+  (`--repo <this repo> --prod-root <reference root>`); the bucket thresholds and the measurement caliber live
+  in its `config/sim-scan.json`, not buried in code.
+- **License**: see `LICENSE` at the repository root (when a subdirectory carries its own exception, that
+  subdirectory's `LICENSE` wins).
+
+## Recall chain: degradation / second (lexicon) path / hit log — new in 0.2.0
+
+Three long-standing recall problems, each with a fix you can switch off in one move. **No shipped
+default changed.**
+
+| Problem | What it does now | Switch | Shipped default |
+|---|---|---|---|
+| An encode timeout/failure used to mean **zero memories injected for that turn** | Queries are truncated to 256 chars; encoding gets a budget (1500 ms default); timeout / throw / empty vector all **degrade to a fixed top-K** (most recently updated first, 8 by default). The result carries `degraded.mode = "fixed-topk"` and `encodeError.name` | `fallbackTopK` / `encodeTimeoutMs` / `--fallback-top-k` / `--encode-timeout-ms` | degradation **always on** (it only exists on the failure path) |
+| Pure vectors are blind to **exact terms**; FTS5 cannot help in Chinese (`unicode61` does not segment CJK) | **Second path**: longest-match segmentation against a jieba dictionary + **BM25** (tf saturation + length normalisation + corpus idf × word specificity), fused with vector/FTS through RRF(k=60) | `enabled` in `memory-engine/dict/lexicon-config.json`, `MEMORY_ENGINE_LEXICON=on/off`, `--lexicon on` | **off** (with it off, results are **byte-identical** to before — see below) |
+| You cannot tell afterwards what was recalled, which row matched, or whether it was any good | **Side-channel JSONL**: query / hit ids / scores / degraded flag / timestamp; byte-based rotation + age pruning, so it cannot grow without bound | point it at a directory: `--recall-log <dir>` or `DSH_MEMORY_RECALL_LOG_DIR`; `DSH_MEMORY_RECALL_LOG=off` disables | **off** (a library should not write files as a side effect of a plain query) |
+
+**Why it is built this way** (long form in [`memory-engine/README.md`](./memory-engine/README.md), section "Design rationale"):
+
+- **Degradation goes by recency, not by padding.** With no query vector there is no semantic signal left;
+  the most likely useful thing is what was written most recently. Three tiers: the source's `recent()` →
+  metadata sorted by `updated_at` → the highest ids in the vector store. Non-empty, deterministic, and
+  pinned rows still excluded.
+- **The lexicon path ships off.** Its gain is "exact words come back"; its cost is "fusion re-ranks".
+  The promise of this repo is "defaults do not change", so it is off and enabled with one config line.
+  Once on, `stats.lexiconStatus` flips from `off` to `ok` and hits appear with `sources: ["vector","lexicon"]`.
+- **The log ships off.** Queries can be private; a library should not persist them on every call. Turn it on
+  by giving a directory, and the bytes / generations / days caps keep it bounded. **Memory text is never
+  logged** — ids and scores only.
+- **Why not reuse the production patch.** On the production side this is a line-anchored replay patch
+  (`patch-*-lexicon.mjs`); that cannot survive a move into a standalone repo. Here it is a module rewritten
+  from behaviour, proven equivalent by fixtures and a before/after diff — not by matching sha256.
+
+**Dictionary size and licence.** `词典.txt` (1,029,582 B) + `词频.txt` (219,541 B) come from
+[jieba](https://github.com/fxsjy/jieba) `extra_dict/dict.txt.small` (MIT) and are bundled under
+`memory-engine/dict/`, about **1.25 MB** in total (`npm pack --dry-run`: tarball **1.1 MB**, unpacked
+**2.8 MB**, **140** files). They are bundled because otherwise the feature is unusable right after
+`npm i`; `MEMORY_ENGINE_DICT_DIR` can point the assets elsewhere if a packager wants them out of the tarball.
+
+Licensing follows **subtree precedence**: `memory-engine/dict/` keeps the upstream **MIT** licence (full text
+and copyright notice in [`dict/LICENSE-词典.md`](./memory-engine/dict/LICENSE-词典.md)), which does **not**
+conflict with the repo-level AGPL-3.0 — MIT and AGPL are compatible, and shipping the notice alongside is
+enough (the same arrangement as the `dsh-memory-local/` fork). Dictionary files are kept byte-for-byte
+(`-text` in `.gitattributes`), so their sha256 can be compared with upstream directly.
+
+**Reproduce the claims:**
+
+```bash
+cd memory-engine
+node tools/verify-recall-chain.mjs                                  # 15 checks: truncation / degradation / lexicon / log / multi-source / CLI
+node tools/verify-recall-chain.mjs --baseline <pre-change lib dir>  # +2 checks: byte-identical results with the lexicon path off
+```
+
+## Layered soft eviction + id-band sentinel — new in 0.2.0
+
+Two old problems: memories could only be deleted, never demoted; and when several machines share one
+database, nothing shouted when a machine started writing into another machine's id band. One tool each,
+and **everything ships dry-run / read-only**.
+
+| Tool | What it does | Usage | Default |
+|---|---|---|---|
+| `cli/layer-migrate.mjs` | Adds a `layer` column, backfills existing rows into the "default layer" | `--db <db>` (dry run) · `--apply --backup <file>` · `--restore <backup> [--apply]` | **dry run** — no `--apply`, not one byte written |
+| `cli/layer-move.mjs` | Moves a **whole layer** into the in-database side table `memories_evicted` (nothing deleted); one command out, one command back | `--layer <name> --apply` · `--back --apply` · `--dump <f>` / `--verify <f>` | **dry run** |
+| `cli/band-sentinel.mjs` | Checks whether the *next* new id would land in someone else's band | `--line` · `--apply --out <file>` | **read-only** — even the report file is off by default |
+
+Layer names and the id-band table live in configuration (`memory-engine/config/layers.json`,
+`memory-engine/config/bands.json`). No layer name and no machine name is hard-coded in the code.
+
+### Single machine vs. multiple machines: how it degrades
+
+- **Single machine (default)** = exactly one band in the config. The sentinel reads **only the local
+  database's** `sqlite_sequence` and checks that the next new id falls inside `[base, base+width)`.
+  No ssh, no guessing about peers, no pretending to know another machine. That alone catches the most
+  common incident shape: a merge pushes `seq` up, and every new memory afterwards lands in another
+  machine's band.
+- **Multiple machines (opt-in)** = several bands, and the non-local entries carry
+  `remote: { ssh, node, db? }`. Only then are peers probed — and the remote command, alias and node path
+  all come from the config file.
+- **A peer being unreachable** ⇒ the verdict says "not probed / unreachable" and the **exit code is not 4**
+  (this is the false-red learned the hard way: folding "cannot connect" into "the band is broken" produces
+  daily alarms that bury the real incident). `4` is reserved for "the local database cannot be read".
+  Add `--require-all` if you *want* an unprobed peer to fail (then rc=5).
+
+Verdicts reuse the good half of the production rule: the `seq`-derived next id outside this band ⇒ red;
+rows in a **higher** band ⇒ red (a higher id cannot be produced legally by this machine); rows in a
+**lower** foreign band ⇒ **disclosed, not red** (historical merge-ins; deleting them would delete memories,
+and they do not affect future ids).
+
+### Why nothing touches the database by default
+
+All three commands default to dry-run or read-only, for one reason: they all modify the memory database,
+and this repository is a toolkit — it must not decide *when* and *which* database gets touched. Concretely:
+
+- `layer-migrate --apply` **requires** both `--db` and `--backup`: no explicit path ⇒ no write (so a slip
+  cannot hit the database in your home directory), and no backup ⇒ refuse (backing up before a schema
+  change is a hard gate, not advice).
+- Moving is dry-run by default; before a real move you `--dump` a per-row fingerprint, and after moving
+  back you `--verify` against it.
+- The sentinel is read-only: no writes, no `sqlite_sequence` touch, no VACUUM. To wire it into cron or a
+  self-check, pass `--apply --out <file>` explicitly — and machine-readable content **only goes to the
+  file, never to stdout** (stdout stays for humans).
+
+### Moving is not deleting
+
+`layer-move` moves a whole layer into the **side table** `memories_evicted` inside the same database file.
+Rows are copied verbatim and carry two extra columns: `content_sha` (a fingerprint bound to that row's
+content) and `batch`. Moving back **verifies the fingerprints first**: if even one row in the side table
+was modified, the whole batch is refused (never "half restored, half lost").
+
+- **Hard invariant**: moved rows + remaining rows == rows before the move (asserted inside the transaction,
+  rolled back otherwise).
+- **Cost of choosing a side table over a separate database file**: moved rows still occupy space in the
+  same file — soft eviction saves context and recall surface, not disk; and the side table has no FTS index,
+  so moved rows are invisible to engine recall and can only be fetched with SQL or `--back`. In exchange,
+  moving in and out happens in one transaction, the invariant can be asserted on the spot, and no
+  easy-to-delete sidecar database is left behind.
+- A move that would empty the main table is refused by default (`--force` to proceed): that is usually a
+  mistyped layer name.
+
+**Reproduce these claims:**
+
+```bash
+cd memory-engine
+node test/run.mjs 12-layering       # 21 checks: dry-run zero-write / backfill / idempotence / restore / invariant / fingerprint equality / band red-green controls
+node cli/layer-migrate.mjs --help
+node cli/layer-move.mjs --help
+node cli/band-sentinel.mjs --help
+```
+
+## Session logs → memories: the boundaries of distillation — new in 0.2.0
+
+Until now there was exactly one input surface for memories: memories already in the database
+(that is what `cli/distill.mjs` consumes). This release adds a second one — **a session log goes in**,
+and out come ≤N **traceable, batch-revertible** conclusion rows. Two new files; the old entry point is untouched.
+
+| File | What it does | Ships as |
+|---|---|---|
+| `lib/session-source.mjs` | Session JSONL adapter: keeps only conclusion sentences that can point back to their source, each carrying **session + line + byte offset + 48-char verbatim anchor** | local-only, zero dependencies |
+| `cli/distill-sessions.mjs` | New entry point: extract → budget gate → one model call → snapshot → write back (`distilled` prefix + batch tag) | **dry-run** |
+
+### 1. No summary-style writes
+
+What gets kept is the **original sentence**, not a summary of "what this passage was about".
+These never reach the database (each is counted and reported once at the end):
+
+- greetings ("hi", "ok", "got it") and questions (ending in a question mark);
+- process narration ("first … then … next … finally", even if the word "decided" happens to appear);
+- speculation ("I think / maybe / probably"), unless the same sentence also carries a rule word such as "always/must";
+- tool output (tool events, command echoes) and image/thinking blocks;
+- any passage longer than the length cap (240 chars by default) — better dropped than stuffed in.
+
+### 2. Only conclusions that point back to their source
+
+Every conclusion carries the sentences it was built from: session id + line number + line byte offset +
+**the first 48 characters of the original sentence**. The session id is taken from the `{"type":"session", …}`
+line of the input log — **both the `id` and the `sessionId` field names are accepted**; if neither is present,
+pass `--session-id`; otherwise everything is labelled `unknown` (provenance degrades to "file name + line number"
+and the CLI says so explicitly). The model may only *synthesize*, and it must report which
+candidates it used (`refs`); out-of-range refs, empty conclusions, anything over 120 characters, and duplicates are
+**dropped outright** (the result's `rejected` list says why). If provenance cannot be established, nothing is written.
+
+Tags (same prefix as the older digest rows — that prefix is what the rollback tool matches):
+
+```
+distilled src=<batch key> srcids=<lines,...> date=<YYYYMMDD> sess=<session> run=session:<batch id>
+```
+
+### 3. No daily budget, no run
+
+**If `--max-tokens` is not given and the ledger has no budget either, the model is not called at all**: the tool
+prints "no daily budget configured ⇒ skipping this round (on purpose)" and exits **0** (that is not an error).
+It does **not** inherit the kernel's 150000 default, and it never treats "unconfigured" as "unlimited".
+If the budget is already spent for the day, or this one call is estimated to exceed what is left, it also skips
+and reports usage.
+
+Measured (deepseek-flash, three self-made fixture sessions, one call each): prompt **344–355** + completion
+**80–98** = **424–453 tokens per round** (1311 total; the pre-call estimate is 1018 per round, ~2.3× conservative).
+Suggested `--max-tokens 20000`: roughly 19 rounds/day by estimate, ~45 by measurement.
+
+### Reverting: one batch, one key
+
+One run = one batch = one source key, so reverting is one command (and it reuses the existing rollback tool):
+
+```bash
+node memory-engine/cli/distill-rollback.mjs --db <memory.db> --key <batch key> --apply
+```
+
+The rollback tool takes a `VACUUM INTO` snapshot before deleting and writes a one-shot restore script; its DELETE
+statement always carries `AND tags LIKE 'distilled%'`, so even a wrong key cannot touch real memories.
+`distill-sessions` takes its own pre-write snapshot (`--snapshot-dir`); **if that snapshot fails, nothing is written**.
+
+### Compressed logs (`.jsonl.zstd`) and "never fail silently"
+
+DSH session logs are zstd-compressed, and they are **multi-frame** (one frame per append). `session-source` walks the
+frame structure and decompresses **frame by frame** (node 22.15+ / 24 ship zstd, so still zero dependencies). If it
+cannot decode them it raises a plain-language error telling you to decompress first — it never decodes only the first
+frame, pretends it is done, or returns an empty result silently.
+
+**Reproduce these claims:**
+
+```bash
+cd memory-engine
+node test/run.mjs 13-session-distill   # 40 checks: anchors / multi-frame zstd / budget-gate states / snapshot fail-closed / batch revert
+node cli/distill-sessions.mjs --help
+node cli/distill-sessions.mjs --source <session.jsonl>                      # no budget ⇒ prints "skipped", rc=0
+node cli/distill-sessions.mjs --source <session.jsonl> --max-tokens 30000   # dry-run ⇒ lists what would be written
+```
+
 ## 2. Requirements
 
 - **Node.js ≥ 22** (built-in `node:sqlite`, ESM)
@@ -96,9 +315,13 @@ Free to use and modify for personal, educational and open-source projects. For *
 commercial** use (where AGPL's copyleft does not apply) please open an issue in
 [Issues](https://github.com/kira905/dsh-memory-toolkit/issues) to obtain a **commercial license**.
 
-> **Subtree exception**: `dsh-memory-local/` is a fork of upstream `dsh-memory` (MIT) and that subtree
+> **Subtree exceptions**: `dsh-memory-local/` is a fork of upstream `dsh-memory` (MIT) and that subtree
 > **keeps the upstream MIT license** (see its `package.json` `license` field and `FORK-BASE.json`).
+> `memory-engine/dict/` (dictionary and word-frequency tables) comes from jieba (MIT) and likewise
+> **keeps the upstream MIT license** (see `dict/LICENSE-词典.md`).
 > Everything else is original work, released under AGPL-3.0 + commercial licensing.
+> Subtree licences take precedence: when redistributing the package, the two MIT subtrees need only their
+> original copyright notices; AGPL copyleft does not apply to them.
 
 ## 8. Related repositories
 

package/README.md

@@ -40,12 +40,188 @@ Test-Path "$env:APPDATA\npm\dsh-memory-summary.CMD"                           #
 
 | 目录 | 是什么 | 形态 |
 |---|---|---|
-| [`memory-engine/`](./memory-engine/README.md) | 记忆引擎（离线层）：独立矢量库 · 增量同步 · 兜底 hash 轮转 · 混合召回（向量 + FTS5 + RRF） · 文档索引 · 启动自检 · 本地编码桥 | 纯 Node ESM + `node:sqlite` |
+| [`memory-engine/`](./memory-engine/README.md) | 记忆引擎（离线层）：独立矢量库 · 增量同步 · 兜底 hash 轮转 · 混合召回（向量 + FTS5 + 词表 BM25 + RRF + 编码降级兜底 + 旁路留痕） · 文档索引 · 启动自检 · 本地编码桥 | 纯 Node ESM + `node:sqlite` |
 | [`memory-db-tools/`](./memory-db-tools/README.md) | 记忆库工具集：只读导出 / 六项体检分析 / 结构探针 / 特征词探针 / 与 persona 的重合度 / 看板账本只读探针 / 零改动自证 / 新会话开工简报 | 一组独立 CLI |
 | [`dsh-memory-local/`](./dsh-memory-local/README.md) | 本地记忆插件：上游 `dsh-memory` 的自持 fork（跨会话持久记忆 + 召回提示段 + 指针注入链），带构建 / 幂等部署 / 一键回滚 / 部署后自检 | DSH 插件 |
 | [`persona-memory-audit.mjs`](./persona-memory-audit.mjs) | persona ↔ 记忆体检：把「persona 正文体积」与「记忆可见率」两项指标固定下来，超阈值即报红 | 单文件 CLI |
 | [`lib/`](./lib) | 四件共用的无依赖小件（参数白名单 / HTTP 存活判据 / zstd 帧解析 / 会话文件名 / 坑库打分） | 纯函数模块 |
 
+## 同源说明
+
+本仓**部分文件与一套内部生产实现同源**（同一份工程长期演进下来的两处落点）。
+
+- **同源程度**：按**行级同源度**口径统计（同名可比件里，仓内文件的非空行有多少能在参照件中找到，多重集扣减），本仓「≥50% 行同占比」= **64%**。
+- **这意味着**：本仓是那套实现的**提炼导出** —— **既不是逐字副本，也不是完全重写的独立实现**。拿单个文件当"独立参考实现"读之前，请先跑下面的判据。
+- **怎么复跑**：同源度扫描器见 `dsh-quality-toolkit` 的 `dsh-sim-scan.mjs`（`--repo <本仓> --prod-root <参照根>`）；分档阈值与口径写在它的 `config/sim-scan.json` 里，不藏在代码里。
+- **许可**：本仓许可见仓根 `LICENSE`（有子树例外时以子树 `LICENSE` 为准）。
+
+## 召回链：降级 / 词表第二路 / 命中留痕（0.2.0 新增）
+
+召回有三类老毛病，这一版各给一条**可一键关回**的解法；出厂默认一条都没改。
+
+| 毛病 | 现在的做法 | 开关 | 出厂 |
+|---|---|---|---|
+| 编码超时/失败 ⇒ 那一轮**一条记忆都不注入** | 查询先截断到 256 字符；编码带预算（默认 1500 ms），超时 / 抛错 / 空向量一律**降级成固定 Top-K**（按最近改动取，条数默认 8），结果里带 `degraded.mode = "fixed-topk"` 与 `encodeError.name` | `fallbackTopK` / `encodeTimeoutMs` / `--fallback-top-k` / `--encode-timeout-ms` | 降级**常开**（只作用于失败路径，成功路径一字不改） |
+| 纯向量对**精确词**不敏感；中文又走不了 FTS5（`unicode61` 不切中文） | **词表第二路**：jieba 词典最长匹配切词 + **BM25**（tf 饱和 + 文档长度归一 + 语料内 idf × 通用词具体度），与向量、FTS 一起进 RRF(k=60) | `memory-engine/dict/lexicon-config.json` 的 `enabled`、`MEMORY_ENGINE_LEXICON=on/off`、`--lexicon on` | **关**（关掉时结果与改动前**逐字节一致**，见下） |
+| 召回了什么、命中了哪条、准不准，事后无从判断 | **旁路 JSONL**：query / 命中 id / 分数 / 是否降级 / 时刻；按字节轮转 + 天数清理，不无限增长 | 给目录：`--recall-log <目录>` 或 `DSH_MEMORY_RECALL_LOG_DIR`；`DSH_MEMORY_RECALL_LOG=off` 一键关回 | **不写**（库不该在纯查询里偷偷落盘） |
+
+**为什么这么设计**（完整版见 [`memory-engine/README.md`](./memory-engine/README.md) 的「设计依据」）：
+
+- **降级用「最近改动」而不是随便凑数**：拿不到向量时，语义相关已无从计算；此时最可能的"有用"是刚写进来的东西。兜底三档依次是数据源直给的 `recent()` → 元数据按 `updated_at` 降序 → 向量表里 id 最大的几条，**保证非空、顺序确定、仍排除 pinned**。
+- **词表默认关**：它的收益是"精确词多捞回来一截"，代价是"两路融合会重排名次"。本仓对外的承诺是默认行为不改 ⇒ 出厂关、由使用方一行配置打开。打开后 `stats.lexiconStatus` 从 `off` 变 `ok`，命中里出现 `sources: ["vector","lexicon"]`。
+- **留痕默认不写**：query 可能是隐私内容，库不该在纯查询里落盘；要判"召回准不准"就显式给目录，且字节/份数/天数三重封口。实际记录里**不存记忆正文**，只存 id 与分数。
+- **为什么不复用生产那份补丁**：生产侧是"按行号锚点重放"的补丁（`patch-*-lexicon.mjs`），搬进独立仓必碎；本仓是按行为重写的模块，靠夹具与旧版对拍证明等价，而不是靠 sha 相同。
+
+**词典体积与许可**：`词典.txt`（1,029,582 B）+ `词频.txt`（219,541 B）来自
+[jieba](https://github.com/fxsjy/jieba) 的 `extra_dict/dict.txt.small`（MIT），随包内嵌在
+`memory-engine/dict/`，合计约 **1.25 MB**（`npm pack --dry-run` 实测：整包 tarball **1.1 MB** / 解包 **2.8 MB** / **140** 个文件）。
+**为什么要内嵌**：不内嵌的话 `npm i` 完这一路直接不可用（要么首次运行去联网拉，要么用户自己找词典）。素材目录可用 `MEMORY_ENGINE_DICT_DIR` 指到别处（打包方嫌大可以把它们挪出包，代价是这一路默认不可用）。
+
+许可按**子树优先**：`memory-engine/dict/` 保持上游 **MIT**（全文与版权声明见 [`dict/LICENSE-词典.md`](./memory-engine/dict/LICENSE-词典.md)），
+与仓级 AGPL-3.0 **不冲突** —— MIT 与 AGPL 兼容，分发时把那份版权声明原样带上即可（与本仓 `dsh-memory-local/` fork 的上游 MIT 同一口径）。
+词典/词频按字节原样保留（`.gitattributes` 里 `-text`），可与上游 sha256 直接对拍。
+
+**怎么复现这些结论**：
+
+```bash
+cd memory-engine
+node tools/verify-recall-chain.mjs                                  # 15 项：截断 / 降级 / 词表 / 留痕 / 多来源 / CLI
+node tools/verify-recall-chain.mjs --baseline <改动前的 lib 目录>    # 追加 2 项：开关关掉时与旧版逐字对拍
+```
+
+## 分层软淘汰 + 号段哨兵（0.2.0 新增）
+
+两个老毛病：记忆只能删、不能降级；多台机共用一个库时，id 号段越界没人喊。这一版各给一件，**出厂全是干跑 / 只读**。
+
+| 件 | 干什么 | 命令 | 出厂 |
+|---|---|---|---|
+| `cli/layer-migrate.mjs` | 给记忆表加一列 `layer`，存量行回填「缺省层」 | `--db <库>`（干跑）· `--apply --backup <文件>` · `--restore <备份> [--apply]` | **干跑**：不带 `--apply` 一个字节都不写 |
+| `cli/layer-move.mjs` | 把**一整层**搬到同库的边表 `memories_evicted`（不删），一条命令出去、一条命令回来 | `--layer <层> --apply` · `--back --apply` · `--dump <f>` / `--verify <f>` | **干跑** |
+| `cli/band-sentinel.mjs` | 看「下一条新 id」会不会落到别人的号段里 | `--line` · `--apply --out <文件>` | **只读**：缺省连报告文件都不写 |
+
+层名与号段表都在配置里（`memory-engine/config/layers.json`、`memory-engine/config/bands.json`），
+代码里没有任何一处写死 `core / recall / archival`，也没有任何一台机器的名字。
+
+### 单机 vs 多机：怎么退化
+
+- **单机（缺省）** ＝ 配置里只有一条 band。此时哨兵**只读本机库**的 `sqlite_sequence`，
+  判「下一条新 id」是否落在 `[base, base+width)`。**不 ssh、不猜对端、不假装知道别的机器**。
+  这一步就能抓住最常见的事故形态：合并搬入把 `seq` 顶高 ⇒ 之后每条新记忆都写进别人的段。
+- **多机（可选）** ＝ 配置里有多条 band，且非本机条目带 `remote: { ssh, node, db? }`。只有这时才逐台去读，
+  远端命令 / 别名 / node 路径**全部来自配置**。
+- **远端读不到** ⇒ 结论写「未探测 / 不可达」，**退出码不是 4**（这条是生产件的血泪假红：把"连不上"和"号段坏了"
+  混成同一个码，就会天天告警、把真事故淹掉）。`4` 只留给「本机库读不到」这种真失败。
+  要"有段没探测就算失败"，显式加 `--require-all`（那时 rc=5）。
+
+判据沿用生产件的好口径：`seq` 决定的下一条 id 越出本段 ⇒ 红；出现**更高位段**的行 ⇒ 红
+（更高位 id 不可能由这一台合法产生）；出现**更低位的别台段**行 ⇒ **只披露不判红**
+（历史搬入 / 合并搬入共用段，删行=删记忆，而且不影响后续新 id）。
+
+### 为什么默认不自动动库
+
+三个命令出厂都是干跑或只读，理由就一条：**它们都直接改记忆库**，而这个仓的定位是"工具集"，
+不能替使用者决定"什么时候动、动哪本"。具体收口：
+
+- 迁移 `--apply` **必须**同时给 `--db` 与 `--backup`：不给库路径就不许动（防手滑改到家目录那本），
+  不给备份路径就拒写（动 schema 前先备份是硬闸，不是建议）。
+- 搬迁默认干跑；真搬要先 `--dump` 留一份逐行指纹，搬回来再 `--verify` 对拍。
+- 哨兵是只读件：不写库、不碰 `sqlite_sequence`、不 VACUUM；要接定时/自检就显式 `--apply --out <文件>`，
+  而且**机器可读内容只落文件，不上 stdout**（日志/报告才可被采集，stdout 留给人读）。
+
+### 搬走不等于删
+
+`layer-move` 把整层搬进**同一个库文件里的边表** `memories_evicted`，行原样躺着，另带两列：
+`content_sha`（与该行内容绑定的指纹）和 `batch`（哪一批搬的）。搬回时**先核指纹再写回**：
+边表里只要有一行被改过，整批拒动（不会出现"一半搬回、一半丢了"）。
+
+- **硬判据**：搬出行数 + 剩下行数 == 搬前总行数（事务内断言，不成立整批回滚）。
+- **代价（选边表而不是另一个库文件）**：搬走的行**仍占这个库文件的体积** —— 软淘汰省的是上下文与召回面，
+  不是磁盘；边表也没有 FTS 索引，搬走的行在引擎召回里看不见，只能用 SQL 或 `--back` 取回。
+  换来的好处是搬进搬出都在一个事务里，守恒能当场断言，也不会留下一个可能被误删的旁路库文件。
+- 会搬空主表时默认拒跑（`--force` 才动）：那通常是层名填错了。
+
+**怎么复现本节结论**：
+
+```bash
+cd memory-engine
+node test/run.mjs 12-layering       # 21 项：干跑零写 / 加列回填 / 幂等 / 还原 / 守恒 / 指纹一致 / 号段红绿对照
+node cli/layer-migrate.mjs --help
+node cli/layer-move.mjs --help
+node cli/band-sentinel.mjs --help
+```
+
+## 会话日志 → 记忆：提炼的边界（0.2.0 新增）
+
+记忆的**输入面**原先只有一条：库里已经有的记忆（`cli/distill.mjs` 走的就是这条）。这一版补上第二条 ——
+**一份会话日志进来**，产出 ≤N 条**可追溯、可整批撤回**的结论条。两个新件，旧入口一个字节都没动：
+
+| 件 | 干什么 | 出厂 |
+|---|---|---|
+| `lib/session-source.mjs` | 会话 JSONL 适配层：只挑「能回指来源的结论句」，每条带**场次 + 行号 + 字节偏移 + 原文锚** | 纯本地、零依赖 |
+| `cli/distill-sessions.mjs` | 新入口：抽句 → 预算闸 → 一次模型调用 → 拍快照 → 写回（`distilled` 前缀 + 批次标签） | **dry-run** |
+
+### 一、不做摘要式写入
+
+抽的是**原句**，不是"这段讲了什么"的概括。所以下面这些一律不进库（每条都计数，跑完一次性告诉你挡了多少）：
+
+- 寒暄（"你好""好的""收到"）与疑问句（句末问号）；
+- 过程叙述（"先…然后…接着…最后…"这种流水账，哪怕句尾偶然出现"决定"两个字）；
+- 推测（"我觉得/可能/大概/应该是"，除非同一句里有"一律/必须"这类规则词撑着）；
+- 纯工具输出（工具事件、命令回显）与图片 / 思考段；
+- 超过长度上限（缺省 240 字）的整段转述 —— 宁可漏掉，也不把长文硬塞进记忆库。
+
+### 二、只写能回指来源的结论条
+
+每条结论都挂着它依据的那几句原文：`sessionId` + 行号 + 行首字节偏移 + **原文前 48 字**。
+场次 id 从输入日志 `{"type":"session", …}` 那一行取：**`id` 或 `sessionId` 两种字段名都认**；
+两者都没有就传 `--session-id`，再没有就一律标 `unknown`（回指退化成「文件名 + 行号」，CLI 会显式告警）。
+模型这一层只允许**合成**，且必须报出自己引用了哪几条候选（`refs`）；越界引用、空结论、超 120 字、
+重复的条目会被**直接剔掉**（结果里的 `rejected` 写明理由）。回指不成立 ⇒ 这条宁可不写。
+
+标签形态（与旧提炼条同前缀，撤回器认的就是这个前缀）：
+
+```
+distilled src=<批次键> srcids=<行号,...> date=<YYYYMMDD> sess=<场次> run=session:<批次 id>
+```
+
+### 三、没配每日预算就不跑
+
+**不给 `--max-tokens`、台账里也没配 ⇒ 一次模型都不调**，打印「未配置每日预算 ⇒ 本轮不跑（这是故意的）」，
+退出码 **0**（这不是错误）。它**不继承**内核那 150000 的缺省额度，也绝不"没配 = 无限跑"。
+给了预算但当日已超、或者"这一次调用估算下来就会超"，同样不跑，并把已用 / 预算写出来。
+
+实测（deepseek-flash，自造三轮夹具会话，每轮一次调用）：prompt **344–355** + completion **80–98** =
+**424–453 token/轮**（三轮合计 1311；脚本估算按 1018 token/轮，约 2.3 倍保守）。
+建议 `--max-tokens 20000`：按估算约 19 轮/天，按实测约 45 轮/天。
+
+### 撤回：一批一个键
+
+一轮 = 一个批次 = 一个来源键，所以撤回是一条命令的事（用既有的撤回器，没另写一个）：
+
+```bash
+node memory-engine/cli/distill-rollback.mjs --db <memory.db> --key <批次键> --apply
+```
+
+撤回器真删前会先 `VACUUM INTO` 拍快照，并生成一键恢复脚本；删除语句恒带 `AND tags LIKE 'distilled%'`，
+所以就算键算错也不会误伤真实记忆。写库前那一份快照则由 `distill-sessions` 自己拍（`--snapshot-dir` 可指定），
+**拍不成就不写库**。
+
+### 压缩日志（`.jsonl.zstd`）与"不静默失败"
+
+DSH 的会话日志是 zstd 压缩的（而且**一次追加一帧**，是多帧文件）。`session-source` 会自己走一遍帧结构、
+**逐帧**解压（node 22.15+ / 24 自带 zstd，零依赖）；解不开就抛一条人话错误，告诉你「这是压缩日志，请先解压再喂」，
+**不会**只解第一帧就假装读完了，也不会静默返回空结果。
+
+**怎么复现本节结论**：
+
+```bash
+cd memory-engine
+node test/run.mjs 13-session-distill   # 40 项：抽句回指 / zstd 多帧 / 预算闸三态 / 快照 fail-closed / 整批撤回
+node cli/distill-sessions.mjs --help
+node cli/distill-sessions.mjs --source <会话.jsonl>                      # 无预算 ⇒ 打印"不跑"，rc=0
+node cli/distill-sessions.mjs --source <会话.jsonl> --max-tokens 30000   # dry-run ⇒ 列出将写什么
+```
+
 ## 二、环境要求
 
 - **Node.js ≥ 22**（用到内置 `node:sqlite`，以及 ESM 顶层能力）
@@ -120,7 +296,9 @@ node memory-db-tools/audit-verify-unchanged.mjs --baseline ./_audit/memories.jso
 
 > **子树例外**：`dsh-memory-local/` 是上游 `dsh-memory`（MIT）的 fork，
 > 该子树**保持上游 MIT 许可**（见其 `package.json` 的 `license` 字段与 `FORK-BASE.json`）；
-> 其余目录（引擎、工具链、共用件）为本项目原创，按 AGPL-3.0 + 商业授权双许可发布。
+> `memory-engine/dict/`（词典与词频）来自 jieba（MIT），同样**保持上游 MIT**（见 `dict/LICENSE-词典.md`）；
+> 其余目录（引擎代码、工具链、共用件）为本项目原创，按 AGPL-3.0 + 商业授权双许可发布。
+> 子树许可优先：分发整包时，上述两个 MIT 子树按 MIT 原样附版权声明即可，不适用 AGPL 的传染条款。
 
 ## 八、相关仓