@kira905/dsh-memory-toolkit 0.1.2 → 0.2.0

package/CHANGELOG.md

@@ -3,6 +3,135 @@
 本仓的版本按**语义化版本**：改结论/改契约升主版本，加能力升次版本，修错别字与文档订正不升。
 英文摘要见每节末尾的 *English* 行。
 
+## v0.2.0 — 2026-10-31
+
+**41 个业务件独立重写**（能力与对外接口不变，实现换成自己的）：
+
+`memory-engine/lib`(21) + `memory-engine/cli`(11) + `memory-engine/tools`(3) + 仓根 `lib`(6) = **41 件**
+从"生产实现的逐字副本"改成**独立实现**。判据是**行级同源度**（同名可比件里，本仓文件的非空行有多少能在参照件中找到，多重集扣减）：
+
+- 41 件均值：**99.2% → 24.6%**；
+- **≥5KB 的 28 件全部 < 50%**（最高 47.5%，最低 12.5%）；
+- 全仓「≥50% 行同占比」：**99% → 64%**（残余面 = `dsh-memory-local/`、`memory-db-tools/`、`memory-engine/test/`，这三块不在本版范围内）。
+
+**对外接口零变化**：
+
+- `memory-engine/test` 的 184 项回归全绿，测试文件一行未改（这就是"没改坏行为"的唯一证据）；
+- 41 件导出面逐件比对：导出名零缺零多，常量值逐值不变；
+- CLI 的 `--flag` 名、缺省值、stdout/stderr 分工、退出码语义不变（改前改后逐用例对拍）。
+
+**顺带清掉内部标识**：卡号、棒号、设备别名、注释里的私有盘符路径与内部任务代号；
+7 处保留的是**契约字符串**（`EXCLUDE_DIRS` 的四类目录名、`deploy-guard` 的忽略目录表），改了会破行为。
+
+逐件台账见仓根 `k9-progress.jsonl`（41 行：before/after sha + 两次数值 + 实现形态说明）。
+
+*English*: 41 business modules rewritten as independent implementations — same capability and public
+interface, different code. Line-level shared-source ratio 99.2% → 24.6% on average; every file ≥5 KB is
+below 50%. The 184-case regression suite passes unchanged, the export surface is identical, and CLI
+flags / defaults / exit codes are unchanged. Internal identifiers were stripped; the seven remaining
+hits are contractual strings (the excluded-directory list and the guard's ignore table).
+
+### 召回链：降级 / 词表第二路 / 命中留痕（K4）
+
+- **查询截断**：引擎侧新增 256 字符上限（原来引擎侧**完全不截**；生产侧那道 512 是插件注入器的配置，不在引擎里）。按码点截，`stats.queryChars / queryCharsUsed / queryTruncated` 如实记录。
+- **编码降级**：编码带预算（默认 1500 ms）。超时 / 抛错 / 空向量 ⇒ 降级成**固定 Top-K**（默认 8 条，按最近改动取，仍排除 pinned），结果带 `degraded.mode = "fixed-topk"` 与 `encodeError = { name, ms }`。**绝不返回空、绝不把异常往上抛**。
+- **词表第二路**（新文件 `lib/lexicon-recall.mjs`）：jieba 词典最长匹配 + **BM25**（tf 饱和 / 长度归一 / 语料 idf × 通用词具体度）+ 进程内缓存与后台预热，与向量、FTS 一起进 RRF(k=60)。开关 = `dict/lexicon-config.json#enabled`（出厂 **false**）/ `MEMORY_ENGINE_LEXICON` / `--lexicon on|off`。关掉时结果与改动前**逐字一致**（夹具对拍 4 条查询 + CLI 旧新对拍）。
+- **命中留痕**（新文件 `lib/recall-log.mjs`）：旁路 JSONL，字段 = 时刻 / query / 命中 id / 分数 / 来源 / 是否降级 / 耗时；**不存记忆正文**。默认不写，给目录才写；`DSH_MEMORY_RECALL_LOG=off` 一键关回；字节（512 KB）→ 轮转（3 代）→ 天数（14 天）三重封口，`keepFiles: 0` 只留当前文件。
+- **词典素材**：`memory-engine/dict/`（`词典.txt` 1.03 MB + `词频.txt` 219 KB + MIT 许可声明 + 开关文件），合计约 **1.25 MB**；`npm pack --dry-run` 实测 tarball **1.1 MB** / 解包 **2.8 MB** / **140** 个文件。素材目录可用 `MEMORY_ENGINE_DICT_DIR` 改位；缺素材时这一路优雅退化为纯向量。
+- **验证**：`memory-engine/test` 184 项**一行未改、全绿**；新增 `tools/verify-recall-chain.mjs`（15 项自带判据，给 `--baseline <改动前 lib>` 则 17 项）；两个新文件行级同源度 **18.5% / 19.5%**。
+- **接口**：只有新增 —— CLI 4 个开关（`--lexicon` / `--lexicon-top-k` / `--lexicon-floor` / `--fallback-top-k` / `--encode-timeout-ms` / `--query-max-chars` / `--recall-log` / `--no-recall-log`）、`recall()`/`recallMulti()` 的新可选入参、`stats` 新字段、`memory-source.recent()`；既有参数、缺省值、退出码语义不变。
+
+*English*: recall chain hardening — 256-char query truncation; an encoding time budget with a **fixed top-K degradation** (never empty, never throws); a lexicon/BM25 second path fused through RRF (off by default, one-line switch, byte-identical results when off); and an opt-in side-channel JSONL recall log with byte/generation/age caps. Dictionary assets (~1.25 MB, jieba · MIT) ship in `memory-engine/dict/`; `npm pack` measures 1.1 MB tarball / 2.8 MB unpacked / 140 files. The 184 existing tests are unchanged and green, and `tools/verify-recall-chain.mjs` adds 15 acceptance checks. Interface changes are additive only.
+
+### 分层软淘汰 + 号段哨兵（K5，净新增件）
+
+记忆只能删、不能降级；多台机共用一个库时 id 号段越界没人喊 —— 各给一件，**出厂全是干跑 / 只读**。
+
+- **`layer` 字段 + 迁移**（新文件 `lib/layers.mjs` / `lib/layer-migrate.mjs` / `cli/layer-migrate.mjs`）：加一列 `layer`，存量行回填缺省层。
+  **零删行**：行数 / `sqlite_sequence` / `PRAGMA integrity_check` 三项改完复核，任一不符就报错并指向备份。
+  **幂等**：列已在、空值已清零 ⇒ 第二次 `--apply` 库文件逐字节不变。`--apply` **必须**同时给 `--db` 与 `--backup`；`--restore <备份>` 默认干跑，真还原走「同目录临时文件 + sha 校验 + rename」，并先摘掉陈旧 `-wal`/`-shm`。
+- **软淘汰执行器**（新文件 `lib/layer-move.mjs` / `cli/layer-move.mjs`）：整层搬进同库边表 `memories_evicted`，一条命令出去、一条命令回来，**一条不删**。
+  硬判据 = 搬出行数 + 剩下行数 == 搬前总行数（事务内断言）；每行带 `content_sha`，搬回前先核指纹，边表被改过就整批拒动。配套 `--dump` / `--verify` 做逐行对拍；会搬空主表时默认拒跑（`--force` 才动）。
+  代价写进 README：搬走的行仍占库体积、边表没有 FTS 索引。
+- **号段哨兵**（新文件 `lib/band-sentinel.mjs` / `cli/band-sentinel.mjs`）：`sqlite_sequence` 主判 + 「更高位段有行」才判红；**更低位的别台段行只披露不判红**（生产件踩过的假红，没改回去）。
+  **单机退化**：配置里只有一条 band 就只读本机、不 ssh；多机才走配置里的 `remote { ssh, node, db }`。**任一远端读不到 ⇒ 结论写「未探测 / 不可达」，退出码不是 4**（4 只留给本机库读不到）；`--require-all` 才把未探测算失败（rc=5）。
+- **配置驱动**：`memory-engine/config/layers.json`（层名集合 / 列名 / 缺省层）与 `memory-engine/config/bands.json`（号段表）。代码里没有写死的层名，也没有任何机器名。
+- **CLI 惯例**：新增三个命令全部 `--help` 可读、未知参数 fail-closed（rc=2）、默认 dry-run、`--json` **只落文件不上 stdout**、`isMainEntry` 守卫走 `realpathSync` 双方比。
+- **验证**：`memory-engine/test` **既有 184 项一行未改、全绿**；新增 `test/12-layering.test.mjs` **21 项**（干跑零写 / 加列回填 / 幂等 / 备份还原 / 搬出守恒 / 搬回逐行指纹一致 / 边表被改整批拒动 / 号段红绿对照 / 远端读不到不许 rc=4）。新增 7 个代码件对生产同名件的行级同源度全部 < 50%（逐件读数见仓根 `k5-progress.jsonl`）。
+
+*English*: layered soft eviction + id-band sentinel (net-new). `layer-migrate` adds a `layer` column and backfills the default layer — zero row deletion, idempotent (second `--apply` is byte-identical), requires an explicit `--db` and a `--backup`, and `--restore` is dry-run by default. `layer-move` moves a whole layer into the in-database side table `memories_evicted` and back, with the hard invariant `moved + remaining == before` asserted in-transaction and per-row `content_sha` verified before restoring. `band-sentinel` checks the `sqlite_sequence`-derived next id against a config-driven band table, judges rows in higher bands red, only *discloses* rows in lower foreign bands, degrades to a single-machine check with no ssh, and never returns 4 for an unreachable peer. All three CLIs are `--help`-readable, fail closed on unknown flags, default to dry-run, and write JSON reports to files only. The 184 existing tests are unchanged and green; `test/12-layering.test.mjs` adds 21 checks.
+
+### 会话日志 → 记忆（K6，净新增件）
+
+记忆的**输入面**补上第二条：一段会话日志进来，产出 ≤N 条可追溯、可整批撤回的结论条。**旧入口（`distill.mjs` / `distill-rollback.mjs` 及其 CLI）的默认值、参数名、退出码一个没改**。
+
+- **会话适配层**（新文件 `lib/session-source.mjs`）：只挑能回指来源的结论句（决定 / 口径 / 偏好 / 修复方案），
+  每条带 `sessionId` + 行号 + 行首字节偏移 + 原文前 48 字锚。寒暄 / 疑问 / 过程叙述 / 推测 / 工具输出 / 超长转述
+  **一律不抽**，逐类计数后一次性报出来。抽的是原句，不做摘要式改写。
+- **新入口**（新文件 `cli/distill-sessions.mjs`）：`--source <jsonl>` → 抽句 → 预算闸 → 一次模型调用 → `VACUUM INTO` 快照 → 写回。
+  缺省 **dry-run**（不调模型、不写库、不写台账）；未知参数 fail-closed（rc=2）；`--json` 必须配 `--out`，**只落文件不上 stdout**。
+- **预算闸（这条是核心）**：没给 `--max-tokens`、台账里也没配 ⇒ **一次模型都不调**，打印「未配置每日预算 ⇒ 本轮不跑（这是故意的）」，rc=0。
+  **不继承**内核 `DAILY_TOKEN_CAP = 150000`，也不"没配 = 无限跑"；当日已超或本次估算会超，同样不跑并打印用量。
+  实测（deepseek-flash，三轮夹具，每轮一次调用）**424 / 424 / 434 token**（合计 1311，prompt 1044 / completion 267），
+  建议 `--max-tokens 20000`（约 19 轮/天按估算、~45 轮/天按实测）。
+- **写回与撤回**：标签 = `distilled src=<批次键> srcids=<行号,...> date=<YYYYMMDD> sess=<场次> run=session:<批次 id>`；
+  一轮一个来源键 ⇒ 撤回直接用既有 `distill-rollback.mjs --key <批次键> --apply`，一批一条命令整组删（不新写撤回器）。
+  写库前快照拍不成 ⇒ **不写库**（fail-closed）。
+- **压缩输入**：`.jsonl.zstd` 走 node 内置 zstd **逐帧**解（DSH 日志一次追加一帧，多帧文件），解不开给人话错误而不是静默失败；
+  当前 node 没有 zstd API 时同样给"请先解压"的提示。
+- **验证**：`memory-engine/test` **既有 205 项一行未改、全绿**；新增 `test/13-session-distill.test.mjs` **40 项**
+  （抽句回指对得上原文 / zstd 单帧多帧坏帧 / 预算闸三态 / 快照 fail-closed / 模型失败也记账 / 幂等 / 整批撤回 /
+  CLI 未知参数与 `--json` 分工）。两个新件对生产同名件的行级同源度见仓根 `k6-progress.jsonl`。
+
+*English*: session logs → memories (net-new). `lib/session-source.mjs` extracts only conclusion sentences that can be
+pointed back to their source (session id + line + byte offset + 48-char verbatim anchor), and drops greetings,
+questions, process narration, speculation, tool output and over-long passages while counting each rejected class.
+`cli/distill-sessions.mjs` is a new entry point that extracts, applies the budget gate, makes exactly one model call,
+takes a `VACUUM INTO` snapshot and writes back — dry-run by default, unknown flags fail closed, and `--json` writes to
+a file only, never to stdout. **No daily budget configured ⇒ the model is not called at all** (rc=0, and it prints
+that this is deliberate); it never inherits the kernel's 150000 cap and never treats "unconfigured" as unlimited.
+Measured over three fixture sessions: 424 / 424 / 434 tokens per round (1311 total), suggesting `--max-tokens 20000`.
+Each run maps to one source key, so reverting is the existing `distill-rollback.mjs --key <batch key> --apply`.
+Compressed `.jsonl.zstd` input is decoded frame by frame with node's built-in zstd; failures raise a plain-language
+error instead of returning partial data. The 205 existing tests are unchanged and green; the new test file adds 40 checks.
+
+## v0.1.3 — 2026-10-30
+
+**主包补 bundle 声明 + 子路径导出面**（修「装完不生效」；无插件行为变化）：
+
+根因是**声明层错位**：`dsh-memory-local/` 的 bundle 声明写在私有子包自己的 `package.json` 里，主包没有 `dsh.bundle`
+⇒ `dsh plugin add` 只打一行 `declares no dsh.bundle` 警告、profile 的 `dsh.profile.bundles` 不变，**装了等于没装**
+（rc 仍是 0，所以用户会当成装好了）。本版把声明提到主包根，并给出宿主 loader（ESM `import()`）能解析的子路径。
+
+- 新增仓根 `cordis.patch.yml`：`id: kira905-memory`（**不是** `memory` —— 那个 id 归官方 `dsh-memory`，撞 id 会静默顶掉或启动抛错）、
+  `name: '@kira905/dsh-memory-toolkit/dsh-memory-local'`；`config` 与子包那份逐键相同（含 `path`）。
+- 主包 `package.json` 新增 `dsh.bundle.patch = ./cordis.patch.yml`。
+- 主包 `package.json` 新增 `exports`：`./dsh-memory-local` → `./dsh-memory-local/lib/index.js`（**文件**，不是目录）、
+  `./package.json` → `./package.json`；**没有** `"."` 条目（主包 `lib/` 没有 `index.js`，包根导入本来就不通）。
+- `files` 补上 `cordis.patch.yml`（否则 tarball 里没有它，装了照样不生效）。
+
+**迁移四条（"收窄包面"变更 —— 只升 patch 号，但请读一遍）**：
+
+1. **从包根导入的写法会失败**（`import '@kira905/dsh-memory-toolkit'`）—— 旧版本来也不通（没有 `main` / `exports`），
+   本版给它一个**明确的** `ERR_PACKAGE_PATH_NOT_EXPORTED`，而不是 "Cannot find package …/index.js" 这种像"文件丢了"的报错。
+   没有可迁移的既有用法。
+2. **子路径导入改走 `exports`**：`@kira905/dsh-memory-toolkit/dsh-memory-local` 从 0.1.2 的
+   `ERR_UNSUPPORTED_DIR_IMPORT`（ESM 不认目录）变成**可用**；反过来，深路径
+   `@kira905/dsh-memory-toolkit/dsh-memory-local/lib/index.js` 从 0.1.2 的"可达"变成 **`ERR_PACKAGE_PATH_NOT_EXPORTED`**。
+   踩到的把 specifier 换成映射后的 `…/dsh-memory-local`（**不要**反过来删 `exports` 图省事 —— 那会把 ESM 这条路重新掐死）。
+3. **与官方 `dsh-memory` 二选一**：本包定位是**替代实现**（`dsh-memory-local` 是 `dsh-memory@0.1.0` 的 fork），两者抢同一个
+   "记忆"位置。装之前读 `<profile>/package.json` 的 `dsh.profile.bundles`，含 `dsh-memory` 就**别装**（反之亦然）；
+   要换实现先把另一个摘掉，别让两份同时加载。
+4. **旧版仍在 npm、可回退**：`0.1.2` 及以前都在，`npm i @kira905/dsh-memory-toolkit@0.1.2` 可回退；回退后深路径恢复可达，
+   同时"不挂载"的老行为也回来（不坏，但等于白装）。
+
+*English: 0.1.3 declares the bundle at the package root (new `cordis.patch.yml`, `id: kira905-memory`) and adds an
+`exports` map that points the `./dsh-memory-local` subpath at a real file, so `dsh plugin add` mounts the fork instead
+of printing "declares no dsh.bundle". No plugin behaviour changed. This narrows the package surface: the deep
+`…/dsh-memory-local/lib/index.js` path is no longer importable (use the mapped subpath), the package root still has no
+entry point (now with an explicit `ERR_PACKAGE_PATH_NOT_EXPORTED`), and this package replaces — never coexists with —
+the official `dsh-memory`. 0.1.2 stays on npm for rollback.*
+
 ## v0.1.2 — 2026-10-30
 
 **内部标识残留清剿**（无 API / 行为契约变化）：
@@ -17,7 +146,7 @@
 **部署面补齐 + 用语刷净**（无 API / 行为契约变化）：
 
 - `dsh-memory-local/` 部署侧支持 `.modules.yaml` profiles（主包 0.2.0-rc.x 的文件布局），新增「全新安装」分支，备份目录改为按需创建。
-- 内部用语刷净：构建机的设备标记改为中性写法（`【设备：ROG】`），文档标签示例由设备名改为通用词（`工作记录`）；对应单测断言同步更新，`memory-engine` 184 项全绿。
+- 内部用语刷净：构建机的设备标记改为中性写法（原来的构建机设备标记），文档标签示例由设备名改为通用词（`工作记录`）；对应单测断言同步更新，`memory-engine` 184 项全绿。
 - 本版起 `npm publish` 走 `next` 标签、稳定后再挂 `latest`。
 
 *English: deploy side now supports `.modules.yaml` profiles (host 0.2.0-rc.x) with a fresh-install branch and lazy backup-dir creation; internal build-machine wording scrubbed (no API/behaviour change).*

package/README.en.md

@@ -14,12 +14,231 @@ fixable; the plugin wires it into a session.**
 
 | Directory | What it is | Form |
 |---|---|---|
-| [`memory-engine/`](./memory-engine/README.md) | Memory engine (offline layer): standalone vector store · incremental sync · hash-rotation fallback sweep · hybrid recall (vector + FTS5 + RRF) · document index · startup self-check · local encoding bridge | plain Node ESM + `node:sqlite` |
+| [`memory-engine/`](./memory-engine/README.md) | Memory engine (offline layer): standalone vector store · incremental sync · hash-rotation fallback sweep · hybrid recall (vector + FTS5 + lexicon BM25 + RRF + degraded fixed-top-K + side-channel recall log) · document index · startup self-check · local encoding bridge | plain Node ESM + `node:sqlite` |
 | [`memory-db-tools/`](./memory-db-tools/README.md) | Memory database toolkit: read-only export / six-way audit analysis / structure probe / feature-word probe / persona overlap / board-ledger probe / zero-write proof / new-session briefing | a set of standalone CLIs |
 | [`dsh-memory-local/`](./dsh-memory-local/README.md) | Local memory plugin: a self-maintained fork of upstream `dsh-memory` (cross-session memory + recall prompt section + pointer injection), with build / idempotent deploy / one-command rollback / post-deploy self-check | DSH plugin |
 | [`persona-memory-audit.mjs`](./persona-memory-audit.mjs) | Persona ↔ memory audit: pins down "persona body size" and "memory visibility" and reports red past thresholds | single-file CLI |
 | [`lib/`](./lib) | Zero-dependency shared bits (argument allow-list / HTTP liveness judgement / zstd frames / session file names / pitfall scoring) | pure modules |
 
+## Shared-source notice
+
+Some files in this repository **share a source with an internal production implementation**
+(the same engineering effort, evolved over time, with two landing places).
+
+- **How much**: measured by the **line-level shared-source ratio** (among same-named comparable files, how
+  many non-empty lines of this repository's files are found in the reference files, multiset-deducted); this
+  repository's "≥50% line-share" figure is **64%**.
+- **What that means**: this repository is a **distilled export** of that implementation — **neither a verbatim
+  copy nor a fully rewritten independent implementation**. Before reading any single file as an "independent
+  reference implementation", run the check below.
+- **How to re-run it**: the shared-source scanner ships with `dsh-quality-toolkit` as `dsh-sim-scan.mjs`
+  (`--repo <this repo> --prod-root <reference root>`); the bucket thresholds and the measurement caliber live
+  in its `config/sim-scan.json`, not buried in code.
+- **License**: see `LICENSE` at the repository root (when a subdirectory carries its own exception, that
+  subdirectory's `LICENSE` wins).
+
+## Recall chain: degradation / second (lexicon) path / hit log — new in 0.2.0
+
+Three long-standing recall problems, each with a fix you can switch off in one move. **No shipped
+default changed.**
+
+| Problem | What it does now | Switch | Shipped default |
+|---|---|---|---|
+| An encode timeout/failure used to mean **zero memories injected for that turn** | Queries are truncated to 256 chars; encoding gets a budget (1500 ms default); timeout / throw / empty vector all **degrade to a fixed top-K** (most recently updated first, 8 by default). The result carries `degraded.mode = "fixed-topk"` and `encodeError.name` | `fallbackTopK` / `encodeTimeoutMs` / `--fallback-top-k` / `--encode-timeout-ms` | degradation **always on** (it only exists on the failure path) |
+| Pure vectors are blind to **exact terms**; FTS5 cannot help in Chinese (`unicode61` does not segment CJK) | **Second path**: longest-match segmentation against a jieba dictionary + **BM25** (tf saturation + length normalisation + corpus idf × word specificity), fused with vector/FTS through RRF(k=60) | `enabled` in `memory-engine/dict/lexicon-config.json`, `MEMORY_ENGINE_LEXICON=on/off`, `--lexicon on` | **off** (with it off, results are **byte-identical** to before — see below) |
+| You cannot tell afterwards what was recalled, which row matched, or whether it was any good | **Side-channel JSONL**: query / hit ids / scores / degraded flag / timestamp; byte-based rotation + age pruning, so it cannot grow without bound | point it at a directory: `--recall-log <dir>` or `DSH_MEMORY_RECALL_LOG_DIR`; `DSH_MEMORY_RECALL_LOG=off` disables | **off** (a library should not write files as a side effect of a plain query) |
+
+**Why it is built this way** (long form in [`memory-engine/README.md`](./memory-engine/README.md), section "Design rationale"):
+
+- **Degradation goes by recency, not by padding.** With no query vector there is no semantic signal left;
+  the most likely useful thing is what was written most recently. Three tiers: the source's `recent()` →
+  metadata sorted by `updated_at` → the highest ids in the vector store. Non-empty, deterministic, and
+  pinned rows still excluded.
+- **The lexicon path ships off.** Its gain is "exact words come back"; its cost is "fusion re-ranks".
+  The promise of this repo is "defaults do not change", so it is off and enabled with one config line.
+  Once on, `stats.lexiconStatus` flips from `off` to `ok` and hits appear with `sources: ["vector","lexicon"]`.
+- **The log ships off.** Queries can be private; a library should not persist them on every call. Turn it on
+  by giving a directory, and the bytes / generations / days caps keep it bounded. **Memory text is never
+  logged** — ids and scores only.
+- **Why not reuse the production patch.** On the production side this is a line-anchored replay patch
+  (`patch-*-lexicon.mjs`); that cannot survive a move into a standalone repo. Here it is a module rewritten
+  from behaviour, proven equivalent by fixtures and a before/after diff — not by matching sha256.
+
+**Dictionary size and licence.** `词典.txt` (1,029,582 B) + `词频.txt` (219,541 B) come from
+[jieba](https://github.com/fxsjy/jieba) `extra_dict/dict.txt.small` (MIT) and are bundled under
+`memory-engine/dict/`, about **1.25 MB** in total (`npm pack --dry-run`: tarball **1.1 MB**, unpacked
+**2.8 MB**, **140** files). They are bundled because otherwise the feature is unusable right after
+`npm i`; `MEMORY_ENGINE_DICT_DIR` can point the assets elsewhere if a packager wants them out of the tarball.
+
+Licensing follows **subtree precedence**: `memory-engine/dict/` keeps the upstream **MIT** licence (full text
+and copyright notice in [`dict/LICENSE-词典.md`](./memory-engine/dict/LICENSE-词典.md)), which does **not**
+conflict with the repo-level AGPL-3.0 — MIT and AGPL are compatible, and shipping the notice alongside is
+enough (the same arrangement as the `dsh-memory-local/` fork). Dictionary files are kept byte-for-byte
+(`-text` in `.gitattributes`), so their sha256 can be compared with upstream directly.
+
+**Reproduce the claims:**
+
+```bash
+cd memory-engine
+node tools/verify-recall-chain.mjs                                  # 15 checks: truncation / degradation / lexicon / log / multi-source / CLI
+node tools/verify-recall-chain.mjs --baseline <pre-change lib dir>  # +2 checks: byte-identical results with the lexicon path off
+```
+
+## Layered soft eviction + id-band sentinel — new in 0.2.0
+
+Two old problems: memories could only be deleted, never demoted; and when several machines share one
+database, nothing shouted when a machine started writing into another machine's id band. One tool each,
+and **everything ships dry-run / read-only**.
+
+| Tool | What it does | Usage | Default |
+|---|---|---|---|
+| `cli/layer-migrate.mjs` | Adds a `layer` column, backfills existing rows into the "default layer" | `--db <db>` (dry run) · `--apply --backup <file>` · `--restore <backup> [--apply]` | **dry run** — no `--apply`, not one byte written |
+| `cli/layer-move.mjs` | Moves a **whole layer** into the in-database side table `memories_evicted` (nothing deleted); one command out, one command back | `--layer <name> --apply` · `--back --apply` · `--dump <f>` / `--verify <f>` | **dry run** |
+| `cli/band-sentinel.mjs` | Checks whether the *next* new id would land in someone else's band | `--line` · `--apply --out <file>` | **read-only** — even the report file is off by default |
+
+Layer names and the id-band table live in configuration (`memory-engine/config/layers.json`,
+`memory-engine/config/bands.json`). No layer name and no machine name is hard-coded in the code.
+
+### Single machine vs. multiple machines: how it degrades
+
+- **Single machine (default)** = exactly one band in the config. The sentinel reads **only the local
+  database's** `sqlite_sequence` and checks that the next new id falls inside `[base, base+width)`.
+  No ssh, no guessing about peers, no pretending to know another machine. That alone catches the most
+  common incident shape: a merge pushes `seq` up, and every new memory afterwards lands in another
+  machine's band.
+- **Multiple machines (opt-in)** = several bands, and the non-local entries carry
+  `remote: { ssh, node, db? }`. Only then are peers probed — and the remote command, alias and node path
+  all come from the config file.
+- **A peer being unreachable** ⇒ the verdict says "not probed / unreachable" and the **exit code is not 4**
+  (this is the false-red learned the hard way: folding "cannot connect" into "the band is broken" produces
+  daily alarms that bury the real incident). `4` is reserved for "the local database cannot be read".
+  Add `--require-all` if you *want* an unprobed peer to fail (then rc=5).
+
+Verdicts reuse the good half of the production rule: the `seq`-derived next id outside this band ⇒ red;
+rows in a **higher** band ⇒ red (a higher id cannot be produced legally by this machine); rows in a
+**lower** foreign band ⇒ **disclosed, not red** (historical merge-ins; deleting them would delete memories,
+and they do not affect future ids).
+
+### Why nothing touches the database by default
+
+All three commands default to dry-run or read-only, for one reason: they all modify the memory database,
+and this repository is a toolkit — it must not decide *when* and *which* database gets touched. Concretely:
+
+- `layer-migrate --apply` **requires** both `--db` and `--backup`: no explicit path ⇒ no write (so a slip
+  cannot hit the database in your home directory), and no backup ⇒ refuse (backing up before a schema
+  change is a hard gate, not advice).
+- Moving is dry-run by default; before a real move you `--dump` a per-row fingerprint, and after moving
+  back you `--verify` against it.
+- The sentinel is read-only: no writes, no `sqlite_sequence` touch, no VACUUM. To wire it into cron or a
+  self-check, pass `--apply --out <file>` explicitly — and machine-readable content **only goes to the
+  file, never to stdout** (stdout stays for humans).
+
+### Moving is not deleting
+
+`layer-move` moves a whole layer into the **side table** `memories_evicted` inside the same database file.
+Rows are copied verbatim and carry two extra columns: `content_sha` (a fingerprint bound to that row's
+content) and `batch`. Moving back **verifies the fingerprints first**: if even one row in the side table
+was modified, the whole batch is refused (never "half restored, half lost").
+
+- **Hard invariant**: moved rows + remaining rows == rows before the move (asserted inside the transaction,
+  rolled back otherwise).
+- **Cost of choosing a side table over a separate database file**: moved rows still occupy space in the
+  same file — soft eviction saves context and recall surface, not disk; and the side table has no FTS index,
+  so moved rows are invisible to engine recall and can only be fetched with SQL or `--back`. In exchange,
+  moving in and out happens in one transaction, the invariant can be asserted on the spot, and no
+  easy-to-delete sidecar database is left behind.
+- A move that would empty the main table is refused by default (`--force` to proceed): that is usually a
+  mistyped layer name.
+
+**Reproduce these claims:**
+
+```bash
+cd memory-engine
+node test/run.mjs 12-layering       # 21 checks: dry-run zero-write / backfill / idempotence / restore / invariant / fingerprint equality / band red-green controls
+node cli/layer-migrate.mjs --help
+node cli/layer-move.mjs --help
+node cli/band-sentinel.mjs --help
+```
+
+## Session logs → memories: the boundaries of distillation — new in 0.2.0
+
+Until now there was exactly one input surface for memories: memories already in the database
+(that is what `cli/distill.mjs` consumes). This release adds a second one — **a session log goes in**,
+and out come ≤N **traceable, batch-revertible** conclusion rows. Two new files; the old entry point is untouched.
+
+| File | What it does | Ships as |
+|---|---|---|
+| `lib/session-source.mjs` | Session JSONL adapter: keeps only conclusion sentences that can point back to their source, each carrying **session + line + byte offset + 48-char verbatim anchor** | local-only, zero dependencies |
+| `cli/distill-sessions.mjs` | New entry point: extract → budget gate → one model call → snapshot → write back (`distilled` prefix + batch tag) | **dry-run** |
+
+### 1. No summary-style writes
+
+What gets kept is the **original sentence**, not a summary of "what this passage was about".
+These never reach the database (each is counted and reported once at the end):
+
+- greetings ("hi", "ok", "got it") and questions (ending in a question mark);
+- process narration ("first … then … next … finally", even if the word "decided" happens to appear);
+- speculation ("I think / maybe / probably"), unless the same sentence also carries a rule word such as "always/must";
+- tool output (tool events, command echoes) and image/thinking blocks;
+- any passage longer than the length cap (240 chars by default) — better dropped than stuffed in.
+
+### 2. Only conclusions that point back to their source
+
+Every conclusion carries the sentences it was built from: session id + line number + line byte offset +
+**the first 48 characters of the original sentence**. The session id is taken from the `{"type":"session", …}`
+line of the input log — **both the `id` and the `sessionId` field names are accepted**; if neither is present,
+pass `--session-id`; otherwise everything is labelled `unknown` (provenance degrades to "file name + line number"
+and the CLI says so explicitly). The model may only *synthesize*, and it must report which
+candidates it used (`refs`); out-of-range refs, empty conclusions, anything over 120 characters, and duplicates are
+**dropped outright** (the result's `rejected` list says why). If provenance cannot be established, nothing is written.
+
+Tags (same prefix as the older digest rows — that prefix is what the rollback tool matches):
+
+```
+distilled src=<batch key> srcids=<lines,...> date=<YYYYMMDD> sess=<session> run=session:<batch id>
+```
+
+### 3. No daily budget, no run
+
+**If `--max-tokens` is not given and the ledger has no budget either, the model is not called at all**: the tool
+prints "no daily budget configured ⇒ skipping this round (on purpose)" and exits **0** (that is not an error).
+It does **not** inherit the kernel's 150000 default, and it never treats "unconfigured" as "unlimited".
+If the budget is already spent for the day, or this one call is estimated to exceed what is left, it also skips
+and reports usage.
+
+Measured (deepseek-flash, three self-made fixture sessions, one call each): prompt **344–355** + completion
+**80–98** = **424–453 tokens per round** (1311 total; the pre-call estimate is 1018 per round, ~2.3× conservative).
+Suggested `--max-tokens 20000`: roughly 19 rounds/day by estimate, ~45 by measurement.
+
+### Reverting: one batch, one key
+
+One run = one batch = one source key, so reverting is one command (and it reuses the existing rollback tool):
+
+```bash
+node memory-engine/cli/distill-rollback.mjs --db <memory.db> --key <batch key> --apply
+```
+
+The rollback tool takes a `VACUUM INTO` snapshot before deleting and writes a one-shot restore script; its DELETE
+statement always carries `AND tags LIKE 'distilled%'`, so even a wrong key cannot touch real memories.
+`distill-sessions` takes its own pre-write snapshot (`--snapshot-dir`); **if that snapshot fails, nothing is written**.
+
+### Compressed logs (`.jsonl.zstd`) and "never fail silently"
+
+DSH session logs are zstd-compressed, and they are **multi-frame** (one frame per append). `session-source` walks the
+frame structure and decompresses **frame by frame** (node 22.15+ / 24 ship zstd, so still zero dependencies). If it
+cannot decode them it raises a plain-language error telling you to decompress first — it never decodes only the first
+frame, pretends it is done, or returns an empty result silently.
+
+**Reproduce these claims:**
+
+```bash
+cd memory-engine
+node test/run.mjs 13-session-distill   # 40 checks: anchors / multi-frame zstd / budget-gate states / snapshot fail-closed / batch revert
+node cli/distill-sessions.mjs --help
+node cli/distill-sessions.mjs --source <session.jsonl>                      # no budget ⇒ prints "skipped", rc=0
+node cli/distill-sessions.mjs --source <session.jsonl> --max-tokens 30000   # dry-run ⇒ lists what would be written
+```
+
 ## 2. Requirements
 
 - **Node.js ≥ 22** (built-in `node:sqlite`, ESM)
@@ -96,9 +315,13 @@ Free to use and modify for personal, educational and open-source projects. For *
 commercial** use (where AGPL's copyleft does not apply) please open an issue in
 [Issues](https://github.com/kira905/dsh-memory-toolkit/issues) to obtain a **commercial license**.
 
-> **Subtree exception**: `dsh-memory-local/` is a fork of upstream `dsh-memory` (MIT) and that subtree
+> **Subtree exceptions**: `dsh-memory-local/` is a fork of upstream `dsh-memory` (MIT) and that subtree
 > **keeps the upstream MIT license** (see its `package.json` `license` field and `FORK-BASE.json`).
+> `memory-engine/dict/` (dictionary and word-frequency tables) comes from jieba (MIT) and likewise
+> **keeps the upstream MIT license** (see `dict/LICENSE-词典.md`).
 > Everything else is original work, released under AGPL-3.0 + commercial licensing.
+> Subtree licences take precedence: when redistributing the package, the two MIT subtrees need only their
+> original copyright notices; AGPL copyleft does not apply to them.
 
 ## 8. Related repositories