dsh-layered-memory 0.5.4 → 0.6.1

package/README.en.md

@@ -0,0 +1,211 @@
+[简体中文](README.md) | **English**
+
+<p align="center">
+  <img src="./assets/readme/hero.svg" width="100%"
+       alt="dsh-layered-memory: conversations are automatically distilled into layered memories and injected before every model step (L0 raw conversation → L1 atomic memories → L2 scene blocks → L3 core persona)">
+</p>
+
+# dsh-layered-memory
+
+A **layered distillation memory plugin** for DeepSeek Harness (persistent composition
+plugin): conversations are processed in the background through L0 capture → L1 atomic
+memories → L2 scene consolidation → L3 persona distillation, and relevant memories are
+automatically injected into context before every model step — neither the user nor the
+model needs to do anything. Ported from the pipeline design of
+[MemoryCore](https://github.com/TencentDB-Agent-Memory) (TencentDB Agent Memory):
+prompts are kept as-is; only the L2/L3 "LLM manipulates files" flow is adapted to
+"LLM outputs, engineering side executes".
+
+## Core Capabilities
+
+### Layered Memory (L0–L3)
+
+| Layer | Content | Storage |
+| --- | --- | --- |
+| L0 | Per-turn user/assistant messages (cleaned, code blocks and injected tags stripped) | `conversations/YYYY-MM-DD.jsonl` + SQLite |
+| L1 | Scene segmentation + memory extraction (chat: persona/episodic/instruction; work: work_fact/work_task/work_method/work_artifact) + conflict-detection dedup & merge, each record family-tagged | `records/YYYY-MM-DD.jsonl` + SQLite (FTS5 + optional vectors, family column) |
+| L2 | New memories consolidated into Markdown scene documents (META blocks, heat management, merge caps), consolidated per family | `scenes/chat/*.md`, `scenes/work/*.md` |
+| L3 | Persona distilled from changed scenes (chat: user persona ≤2000 chars; work: Team Operating Doctrine ≤1200 chars), one per family | `persona-chat.md`, `persona-work.md` |
+
+Pipeline principles: all distillation calls reuse DSH's own `ctx.llm`; any stage
+failure is logged only and never blocks the agent loop; recall injection is wrapped
+in tags that are automatically stripped on the capture side, preventing feedback loops.
+
+### Per-Session Memory Modes
+
+Each session can independently choose a memory mode — **write and recall share the
+same mode**:
+
+| Mode | Distillation (write) | Recall (injection) |
+| --- | --- | --- |
+| `Auto` (default) | Single extraction with a merged-vocabulary prompt, all personal 3 + work 4 types enabled, family tag assigned by type prefix | Both families recalled; persona/scene navigation grouped by category and injected with structured `<domain>` blocks |
+| `chat` | Narrow prompt with personal 3 types only, chat family only | chat memories only + chat persona/scene navigation |
+| `work` | Narrow prompt with work 4 types only, work family only | work memories only + work persona/scene navigation |
+| `Off` | No L0 writes, no distillation | No recall; the three memory tools return a "cloaked" notice |
+
+- **Control**: the pill next to the mode selector in the input bar (`Memory · Auto`);
+  clicking opens a macOS-style sliding picker above — release to snap to the nearest
+  mode;
+- **Default mode** = config `family` (`auto|chat|work`, default `auto`); each session's
+  choice is persisted by sessionId to `session-modes.json`, surviving restarts/session
+  restore; switching mid-session takes effect next turn, already-extracted memories
+  stay in their original family;
+- Stacks with the global switches (global is the master gate); L2/L3 are fully
+  family-isolated — content never leaks across families.
+
+### Memory Browser (Settings → Memory)
+
+Multi-tab page with a mixed view of both families: **Overview** (per-layer counts +
+memory mode switch panel, auto-refresh every 5s), **Memories** (L1 card list with
+keyword/type/scene filters), **Scenes** (L2 full text), **Persona** (L3 full text),
+**Log** (last 200 lines of `memory.log`). Switches go through the official settings
+service (namespace `dsh-memory`, effective immediately, persisted across restarts);
+effective rule = **static config (deployment ceiling) AND runtime switch**; data
+channel is loopback RPC (`dsh-memory/*`).
+
+## Getting Started
+
+Requires Node ≥ 22.16. Two invocation styles — the `npx` prefix can replace `dsh` in
+any command below:
+
+```bash
+# Option 1: run the official CLI directly via npx (no pre-installed dsh; version can be pinned, e.g. dsh-layered-memory@0.5.4)
+npx -y @deepseek-ai/dsh plugin --profile web add dsh-layered-memory
+
+# Option 2: with the dsh CLI installed (dsh is a pnpm forwarder; npm i -g pnpm first if missing)
+dsh plugin --profile web add dsh-layered-memory
+
+# Alternative sources: GitHub repo / local path (dev & debugging, link: points at the repo; npm run build + restart dsh to apply)
+dsh plugin --profile web add https://github.com/JunNanLYS/dsh-layered-memory
+dsh plugin --profile web add /path/to/dsh-layered-memory
+```
+
+This package declares a `dsh.bundle` composition layer (`cordis.patch.yml`); after
+installation the **plugin entry is mounted automatically** — no need to hand-edit
+`$DSH_HOME/profiles/web/cordis.patch.yml`. Then restart DeepSeek Harness and verify:
+the appearance of `conversations/ records/ scenes/` and `memory.db` under
+`~/.dsh/memory/` means the plugin applied successfully; the "Memory" page in settings
+and the mode pill in the input bar mean the client half is ready.
+
+> ⚠️ **Security note**: installing a plugin = running third-party code with your
+> privileges. This plugin reads session content, writes files in its data directory,
+> and calls the LLM/embedding services you configured; if that concerns you, review
+> the source first (`src/`).
+
+**Uninstall**: `dsh plugin --profile web remove dsh-layered-memory` + restart. Data
+stays in `~/.dsh/memory/`; delete the whole directory manually if you don't need it.
+
+### Development from Source
+
+```bash
+git clone https://github.com/JunNanLYS/dsh-layered-memory
+cd dsh-layered-memory
+npm install && npm run build
+dsh plugin --profile web add .        # link: install; after code changes, npm run build + restart dsh
+npm run smoke                         # smoke test (rebuild first: see command below)
+npx tsc src/smoke.ts --outDir dist-smoke --module nodenext --moduleResolution nodenext --target es2022 --strict --skipLibCheck --esModuleInterop
+```
+
+## Configuration
+
+Override configs go into the profile's own `cordis.patch.yml` as a **top-level bare
+patch entry** (direct `id:`, not wrapped in `insert:` — an insert with the same id as
+the bundle layer appends and causes `duplicate loader entry id` startup failure):
+
+```yaml
+- id: dsh-memory
+  name: dsh-layered-memory
+  config:                    # keys replace whole lines (no deep merge); write out all keys you want to keep
+    family: auto             # default mode for new sessions: auto | chat | work
+    llm:                     # distillation model route (falls back to the current default model if empty)
+      provider: ''
+      model: ''
+```
+
+| Field | Default | Description |
+| --- | --- | --- |
+| `family` | `auto` | Default memory mode for new sessions: `auto` (both families) \| `chat` (personal) \| `work` (work); switchable per session via the input-bar control |
+| `dataDir` | `$DSH_HOME/memory` | Data directory |
+| `capture.enabled` | `true` | L0 capture |
+| `capture.stripCodeBlocks` | `true` | Strip code blocks from assistant messages |
+| `capture.maxMessageChars` | `4000` | Max characters per message |
+| `extract.enabled` | `true` | L1 extraction |
+| `extract.minMessages` | `1` | Run L1 extraction after N new messages accumulate |
+| `extract.backgroundMessages` | `10` | Background messages attached to extraction |
+| `extract.candidatePool` | `5` | Dedup candidate pool size |
+| `l2.enabled` | `true` | L2 scene consolidation |
+| `l2.minNewMemories` | `5` | New-memory threshold since last L2 consolidation |
+| `l2.maxScenes` | `12` | Scene block count cap |
+| `l2.sceneContextLimit` | `3` | Max similar-scene full texts attached to the L2 prompt |
+| `l3.enabled` | `true` | L3 persona distillation |
+| `l3.interval` | `20` | L3 distillation interval (new-memory count) |
+| `recall.enabled` | `true` | Auto recall |
+| `recall.maxResults` | `5` | L1 records injected per step |
+| `recall.strategy` | `hybrid` | Retrieval strategy: `keyword` / `embedding` / `hybrid` |
+| `recall.scoreThreshold` | `0.3` | Recall score threshold (below is not injected; applies to keyword/embedding only, not pre-fusion hybrid; tool path unfiltered) |
+| `embedding.enabled` | `false` | Vector retrieval switch; off = pure FTS |
+| `embedding.baseUrl` | empty | OpenAI-compatible /embeddings endpoint (e.g. `https://api.siliconflow.cn/v1`) |
+| `embedding.apiKey` | empty | API key |
+| `embedding.model` | empty | embedding model name |
+| `embedding.dimensions` | `0` | Vector dimensions (required when enabled; must match model output) |
+| `llm.provider/model` | empty | Distillation model override (defaults to current selection) |
+| `llm.maxTokens` | `20000` | Output token cap per distillation call (unified across stages; a reasoning model's reasoning shares this budget — too low gets fully consumed by thinking, leaving 0 chars of text) |
+| `llm.temperature` | `0.3` | Distillation temperature |
+| `llm.maxInputChars` | `700000` | Input character budget per distillation call (over-budget L1 inputs are chunked automatically) |
+| `tools` | `true` | Whether to register model-callable memory tools |
+
+## Storage Layout
+
+Aligned with MemoryCore's official dual-write architecture: append-only JSONL files
+(`conversations/`, `records/` sharded by day) are the source of truth for
+backup/restore and are **never rewritten**; `memory.db` (node:sqlite + WAL + FTS5
+BM25 + sqlite-vec cosine vectors) is the primary retrieval engine — updates/deletes
+from dedup merges touch the retrieval DB only.
+
+- **Three retrieval strategies** (`recall.strategy`): `keyword` (FTS5 BM25) /
+  `embedding` (vec0 cosine KNN) / `hybrid` (both lanes in parallel + RRF k=60
+  fusion, default); `conversation_search` (L0) uses the same fusion;
+- **Vectors optional**: off by default (pure FTS). DSH's `ctx.llm` has no embeddings
+  endpoint; enabling requires any OpenAI-compatible `/embeddings` service (configure
+  `embedding.*`); config changes automatically drop the vector table and re-embed
+  everything in the background;
+- **Degradation chain**: sqlite-vec load failure → pure FTS; embedding call failure
+  → degrade to FTS for that call with a one-time warning; retrieval DB init failure
+  → memory features disabled entirely but dsh itself starts normally;
+- **Replacement seam**: `L1Store.search()` is the single retrieval entry point.
+
+```
+memory/
+├── memory.db              # SQLite retrieval DB (L0/L1 metadata + FTS5 + optional vectors; L1 has a family column)
+├── conversations/2026-01-01.jsonl  # L0 raw conversation source of truth (one file per day, append-only; not family-split)
+├── records/2026-01-01.jsonl        # L1 atomic memory source of truth (one file per day, append-only; family field included)
+├── scenes/chat/*.md       # L2 scene blocks (chat family)
+├── scenes/work/*.md       # L2 scene blocks (work family)
+├── persona-chat.md        # L3 persona (chat family: user persona)
+├── persona-work.md        # L3 persona (work family: Team Operating Doctrine)
+├── state.json             # pipeline checkpoint (v2 family-split: families.chat / families.work)
+├── session-modes.json     # session mode map (sessionId → mode, auto-pruned after >90 days)
+└── memory.log             # diagnostic log (info+, rotates to memory.log.1 beyond 2MB)
+```
+
+## Logging & Troubleshooting
+
+The dsh host prints plugin logs to the console; the plugin mirrors info and above to
+`memory.log` in its data directory. The typical log path of one conversation turn:
+`L0 捕获 turn=N …条` → `L0 落盘 N 条` → `蒸馏管线开始（…待重试 M 条）` →
+`LLM 调用 provider/model：输入 x → 输出 y 字符（z s）` → `L1 抽取完成（…新增 Y 条）` →
+`蒸馏管线结束`; the next turn shows `召回命中 N 条 L1`. Empty LLM output carries full
+diagnostics (finish reason / token counts / reasoning excerpt); JSON parse failures
+include the first 400 characters of the raw model output; all failure warns carry the
+first stack frame.
+
+## Differences from MemoryCore
+
+- The full pipeline is embedded (no external Gateway); distillation reuses DSH's own LLM;
+- L2/L3 changed from "LLM manipulates file tools" to "LLM outputs operation JSON / full documents, engineering side executes";
+- Recall injection happens at `agent/pre-step` + agent-scoped `systemPrompt.context` (DSH-native events/services);
+- Storage/retrieval is a single-machine slimmed version of the official sqlite backend (drops multi-tenant isolation columns, TCVDB cloud backend, audit tables; tokenization uses a bundled CJK bigram instead of jieba, keeping zero native dependencies — sqlite-vec is the only native extension, auto-degrading on load failure).
+
+## License
+
+[MIT](LICENSE)

package/README.md

@@ -1,3 +1,5 @@
+**简体中文** | [English](README.en.md)
+
 <p align="center">
   <img src="./assets/readme/hero.svg" width="100%"
        alt="dsh-layered-memory：对话自动分层蒸馏成记忆，模型每步前自动召回注入（L0 原始对话 → L1 原子记忆 → L2 场景块 → L3 核心画像）">
@@ -25,6 +27,16 @@ L0 捕获 → L1 原子记忆 → L2 场景整合 → L3 画像蒸馏，模型
 管线原则：所有蒸馏调用复用 DSH 自身的 `ctx.llm`；任何阶段失败只记日志、绝不阻塞
 Agent 循环；召回注入用标签包裹、捕获侧自动剥离，防止反馈循环。
 
+**未蒸馏缓冲持久化**：抽取失败的待重试消息与攒触发阈值中途的消息都暂存在按档分桶的
+缓冲里（`pending.json`，每次蒸馏尝试后原子落盘）——重启不丢，启动 20 秒后自动补跑一次，
+失败则维持"等下一轮同档对话"的语义。
+
+**重建**（设置页 → 记忆 → 概览 → 重建记忆）：以 L0 原始对话为事实源重新推导全部派生层。
+旧 `records/`、`scenes/`、`persona-*.md` 整体归档（改名 `*.bak.<时间戳>`，不删除），检索库
+L1 清空、checkpoint 重置，随后按会话分块、统一 auto 档重蒸馏 L1→L2→L3（收尾强制跑一轮
+L2 残余 + L3 冷启动）。重建分块走低优先级队列——期间正常对话的蒸馏优先进行；带确认弹窗
+（会话数/消息数/预计调用数）、进度条与取消（已重建部分保留）。
+
 ### 会话级记忆档位
 
 每个会话可独立选择记忆档位，**写入与召回同档**：
@@ -44,24 +56,26 @@ Agent 循环；召回注入用标签包裹、捕获侧自动剥离，防止反
 
 ### 记忆浏览器（设置页 → 记忆）
 
-多 Tab 页面，两族混合视图：**概览**（各层计数 + 记忆模式开关面板，5 秒自动刷新）、
+多 Tab 页面，两族混合视图：**概览**（各层计数 + 记忆模式开关面板 + 蒸馏思考档位选择器，5 秒自动刷新）、
 **记忆**（L1 卡片列表，关键词/类型/情境筛选）、**场景**（L2 全文）、**画像**（L3 全文）、
-**日志**（`memory.log` 尾部 200 行）。开关走官方 settings 服务（命名空间 `dsh-memory`，
-实时生效、重启保留）；生效规则 = **静态 config（部署上限）AND 运行时开关**，
+**日志**（`memory.log` 尾部 200 行）。开关与思考档位走官方 settings 服务（命名空间 `dsh-memory`，
+实时生效、重启保留）；开关生效规则 = **静态 config（部署上限）AND 运行时开关**；
+思考档位为运行时覆盖（选择器选"跟随配置"则用部署配置 `llm.reasoningEffort` 作默认），
 数据通道为 loopback RPC（`dsh-memory/*`）。
 
 ## 快速开始
 
-需要 Node ≥ 22.16 与已安装的 dsh CLI（dsh 是 pnpm 转发器，未装 pnpm 时先 `npm i -g pnpm`）。
+需要 Node ≥ 22.16。两种调用方式任选（`npx` 前缀可替换下面任何 `dsh` 命令）：
 
 ```bash
-# 从 npm 安装（推荐；可 pin 版本，如 @0.5.3）
+# 方式一：npx 直接跑官方 CLI（无需预装 dsh；可 pin 版本，如 dsh-layered-memory@0.5.4）
+npx -y @deepseek-ai/dsh plugin --profile web add dsh-layered-memory
+
+# 方式二：已装 dsh CLI（dsh 是 pnpm 转发器，未装 pnpm 时先 npm i -g pnpm）
 dsh plugin --profile web add dsh-layered-memory
 
-# 或从 GitHub 仓库安装
+# 包源备选：GitHub 仓库 / 本地路径（开发调试，link: 指向仓库，npm run build + 重启 dsh 即生效）
 dsh plugin --profile web add https://github.com/JunNanLYS/dsh-layered-memory
-
-# 或从本地路径安装（开发/调试，link: 指向仓库，npm run build + 重启 dsh 即生效）
 dsh plugin --profile web add /path/to/dsh-layered-memory
 ```
 
@@ -130,7 +144,8 @@ npx tsc src/smoke.ts --outDir dist-smoke --module nodenext --moduleResolution no
 | `embedding.model` | 空 | embedding 模型名 |
 | `embedding.dimensions` | `0` | 向量维度（启用时必填，须与模型输出一致） |
 | `llm.provider/model` | 空 | 蒸馏模型覆盖（默认用当前默认选择） |
-| `llm.maxTokens` | `20000` | 单次蒸馏输出 token 上限（全阶段统一；推理模型的 reasoning 与正文共享该预算，过低会被思考吃光导致正文 0 字符） |
+| `llm.maxTokens` | `256000` | 单次蒸馏输出 token 上限（全阶段统一；推理模型的 reasoning 与正文共享该预算，过低会被思考吃光导致正文 0 字符） |
+| `llm.reasoningEffort` | `off` | 蒸馏思考档位（部署默认）：`off` / `high` / `max`，空串不传（跟随模型默认）。蒸馏是结构化抽取任务，默认关思考——推理模型（如 v4-flash）默认 high 档的思考可把任意输出预算全部吃光导致正文 0 字符；非推理模型不认识 effort 时需设为空串。运行时可在设置页 → 记忆 → 概览临时切换（选"跟随配置"即回退本值） |
 | `llm.temperature` | `0.3` | 蒸馏温度 |
 | `llm.maxInputChars` | `700000` | 单次蒸馏输入字符预算（超限的 L1 输入自动分块抽取） |
 | `tools` | `true` | 是否注册模型可调用的记忆工具 |
@@ -160,7 +175,9 @@ memory/
 ├── persona-chat.md        # L3 画像（chat 族：用户画像）
 ├── persona-work.md        # L3 画像（work 族：Team Operating Doctrine）
 ├── state.json             # 管线 checkpoint（v2 分族：families.chat / families.work）
+├── pending.json           # 未蒸馏缓冲（按档分桶；重启恢复 + 启动自动补跑）
 ├── session-modes.json     # 会话档位映射（sessionId → 档位，>90 天自动清理）
+├── records.bak.<ts>/      # 重建时归档的旧产物（scenes/persona 同款 *.bak.<ts>）
 └── memory.log             # 诊断日志（info+，超 2MB 轮转为 memory.log.1）
 ```