dsh-layered-memory 0.6.1 → 0.7.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. package/README.en.md +106 -81
  2. package/README.md +93 -80
  3. package/assets/img/Hero.png +0 -0
  4. package/assets/img/Layers.png +0 -0
  5. package/assets/img/Modes.png +0 -0
  6. package/assets/img/ui-dark.jpg +0 -0
  7. package/assets/img/ui-light.jpg +0 -0
  8. package/assets/readme/flow.svg +189 -0
  9. package/assets/readme/storage.svg +115 -0
  10. package/dist/client.js +1100 -402
  11. package/dist/config.d.ts +12 -0
  12. package/dist/config.js +20 -17
  13. package/dist/hooks/capture.d.ts +17 -1
  14. package/dist/hooks/capture.js +45 -11
  15. package/dist/hooks/recall.d.ts +7 -0
  16. package/dist/hooks/recall.js +20 -4
  17. package/dist/index.d.ts +8 -0
  18. package/dist/index.js +107 -44
  19. package/dist/pipeline/rebuild.d.ts +2 -0
  20. package/dist/pipeline/rebuild.js +7 -0
  21. package/dist/pipeline/runner.d.ts +7 -1
  22. package/dist/pipeline/runner.js +22 -7
  23. package/dist/runtime-package-lock.json +982 -0
  24. package/dist/settings.js +79 -20
  25. package/dist/stats.d.ts +7 -1
  26. package/dist/stats.js +131 -14
  27. package/dist/store/download-queue.d.ts +71 -0
  28. package/dist/store/download-queue.js +313 -0
  29. package/dist/store/embedding-source.d.ts +160 -0
  30. package/dist/store/embedding-source.js +421 -0
  31. package/dist/store/embedding.d.ts +3 -1
  32. package/dist/store/embedding.js +5 -0
  33. package/dist/store/l0.d.ts +13 -5
  34. package/dist/store/l0.js +40 -10
  35. package/dist/store/l1.d.ts +15 -5
  36. package/dist/store/l1.js +47 -19
  37. package/dist/store/local-embedding.d.ts +64 -0
  38. package/dist/store/local-embedding.js +120 -0
  39. package/dist/store/model-catalog.d.ts +45 -0
  40. package/dist/store/model-catalog.js +78 -0
  41. package/dist/store/runtime-installer.d.ts +65 -0
  42. package/dist/store/runtime-installer.js +220 -0
  43. package/dist/store/sqlite.d.ts +68 -7
  44. package/dist/store/sqlite.js +422 -73
  45. package/dist/tools/index.js +6 -4
  46. package/dist/util/filelog.d.ts +2 -0
  47. package/dist/util/filelog.js +20 -3
  48. package/package.json +1 -1
  49. package/assets/readme/hero.svg +0 -58
package/README.en.md CHANGED
@@ -1,8 +1,8 @@
1
1
  [简体中文](README.md) | **English**
2
2
 
3
3
  <p align="center">
4
- <img src="./assets/readme/hero.svg" width="100%"
5
- alt="dsh-layered-memory: conversations are automatically distilled into layered memories and injected before every model step (L0 raw conversation L1 atomic memories L2 scene blocks L3 core persona)">
4
+ <img src="./assets/img/Hero.png" width="100%"
5
+ alt="DeepSeek Harness hero banner: conversations distilled into layered memories and recalled before every model step chat bubbles dissolve into three progressively brighter light layers flowing into a frosted-glass capsule with a glowing orb and gradient track (tick labels: 日常·工作·智能·关闭), with light threads looping back to suggest recall">
6
6
  </p>
7
7
 
8
8
  # dsh-layered-memory
@@ -11,57 +11,46 @@ A **layered distillation memory plugin** for DeepSeek Harness (persistent compos
11
11
  plugin): conversations are processed in the background through L0 capture → L1 atomic
12
12
  memories → L2 scene consolidation → L3 persona distillation, and relevant memories are
13
13
  automatically injected into context before every model step — neither the user nor the
14
- model needs to do anything. Ported from the pipeline design of
15
- [MemoryCore](https://github.com/TencentDB-Agent-Memory) (TencentDB Agent Memory):
16
- prompts are kept as-is; only the L2/L3 "LLM manipulates files" flow is adapted to
17
- "LLM outputs, engineering side executes".
14
+ model needs to do anything.
18
15
 
19
- ## Core Capabilities
16
+ > The core memory capabilities of this plugin (the L0–L3 layered distillation pipeline,
17
+ > prompts, and dual-write storage design) are modeled after **MemoryCore** from
18
+ > [TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory):
19
+ > prompts are kept as-is; only the L2/L3 "LLM manipulates files" flow is adapted to
20
+ > "LLM outputs, engineering side executes".
20
21
 
21
- ### Layered Memory (L0–L3)
22
+ ## Runtime Data Flow
22
23
 
23
- | Layer | Content | Storage |
24
- | --- | --- | --- |
25
- | L0 | Per-turn user/assistant messages (cleaned, code blocks and injected tags stripped) | `conversations/YYYY-MM-DD.jsonl` + SQLite |
26
- | L1 | Scene segmentation + memory extraction (chat: persona/episodic/instruction; work: work_fact/work_task/work_method/work_artifact) + conflict-detection dedup & merge, each record family-tagged | `records/YYYY-MM-DD.jsonl` + SQLite (FTS5 + optional vectors, family column) |
27
- | L2 | New memories consolidated into Markdown scene documents (META blocks, heat management, merge caps), consolidated per family | `scenes/chat/*.md`, `scenes/work/*.md` |
28
- | L3 | Persona distilled from changed scenes (chat: user persona ≤2000 chars; work: Team Operating Doctrine ≤1200 chars), one per family | `persona-chat.md`, `persona-work.md` |
24
+ <p align="center">
25
+ <img src="./assets/readme/flow.svg" width="100%"
26
+ alt="Runtime data flow: session events from User and Assistant (left) flow into the plugin (L0 capture, L1–L3 distillation, retrieval, memory tools), which injects relevant memories into the DeepSeek Harness core (right) at agent/pre-step; distillation reuses the core's ctx.llm and data is dual-written to ~/.dsh/memory/">
27
+ </p>
29
28
 
30
- Pipeline principles: all distillation calls reuse DSH's own `ctx.llm`; any stage
31
- failure is logged only and never blocks the agent loop; recall injection is wrapped
32
- in tags that are automatically stripped on the capture side, preventing feedback loops.
29
+ The plugin attaches to DSH-native event seams (`session/event` for capture,
30
+ `agent/pre-step` for injection), reuses the host's `ctx.llm` for distillation, and stays
31
+ fully transparent to both user and model. It also registers three model-callable memory
32
+ tools: `memory_search` / `conversation_search` / `memory_read_scene`.
33
33
 
34
- ### Per-Session Memory Modes
34
+ ## Layered Memory (L0–L3)
35
+
36
+ <p align="center">
37
+ <img src="./assets/img/Layers.png" width="100%"
38
+ alt="Four memory layers refining from top-left to bottom-right: L0 raw conversation (chat bubbles) → L1 atomic memories (glowing fact particles) → L2 scene blocks (glass document slabs) → L3 core persona (radiant crystal core); stages connected by LLM extract/consolidate/distill light beams, shrinking width shows progressive refinement">
39
+ </p>
35
40
 
36
- Each session can independently choose a memory mode — **write and recall share the
37
- same mode**:
41
+ ## Per-Session Memory Modes
38
42
 
39
- | Mode | Distillation (write) | Recall (injection) |
40
- | --- | --- | --- |
41
- | `Auto` (default) | Single extraction with a merged-vocabulary prompt, all personal 3 + work 4 types enabled, family tag assigned by type prefix | Both families recalled; persona/scene navigation grouped by category and injected with structured `<domain>` blocks |
42
- | `chat` | Narrow prompt with personal 3 types only, chat family only | chat memories only + chat persona/scene navigation |
43
- | `work` | Narrow prompt with work 4 types only, work family only | work memories only + work persona/scene navigation |
44
- | `Off` | No L0 writes, no distillation | No recall; the three memory tools return a "cloaked" notice |
43
+ <p align="center">
44
+ <img src="./assets/img/Modes.png" width="100%"
45
+ alt="Per-session memory modes: a glass capsule track with four stops (日常 · 工作 · 智能 · 关闭), the glowing orb resting on 智能 (default); a vignette above each personal chat bubbles for 日常, code/document panes for 工作, two streams merging brightest for 智能, a dim dashed ghost bubble for 关闭">
46
+ </p>
45
47
 
46
48
  - **Control**: the pill next to the mode selector in the input bar (`Memory · Auto`);
47
49
  clicking opens a macOS-style sliding picker above — release to snap to the nearest
48
- mode;
49
- - **Default mode** = config `family` (`auto|chat|work`, default `auto`); each session's
50
- choice is persisted by sessionId to `session-modes.json`, surviving restarts/session
51
- restore; switching mid-session takes effect next turn, already-extracted memories
52
- stay in their original family;
53
- - Stacks with the global switches (global is the master gate); L2/L3 are fully
54
- family-isolated — content never leaks across families.
55
-
56
- ### Memory Browser (Settings → Memory)
57
-
58
- Multi-tab page with a mixed view of both families: **Overview** (per-layer counts +
59
- memory mode switch panel, auto-refresh every 5s), **Memories** (L1 card list with
60
- keyword/type/scene filters), **Scenes** (L2 full text), **Persona** (L3 full text),
61
- **Log** (last 200 lines of `memory.log`). Switches go through the official settings
62
- service (namespace `dsh-memory`, effective immediately, persisted across restarts);
63
- effective rule = **static config (deployment ceiling) AND runtime switch**; data
64
- channel is loopback RPC (`dsh-memory/*`).
50
+ mode; adapts to light/dark themes;
51
+ - Each session's choice is persisted by sessionId to `session-modes.json`, surviving
52
+ restarts/session restore; stacks with the global switches (global is the master gate);
53
+ L2/L3 are fully family-isolated content never leaks across families.
65
54
 
66
55
  ## Getting Started
67
56
 
@@ -69,7 +58,7 @@ Requires Node ≥ 22.16. Two invocation styles — the `npx` prefix can replace
69
58
  any command below:
70
59
 
71
60
  ```bash
72
- # Option 1: run the official CLI directly via npx (no pre-installed dsh; version can be pinned, e.g. dsh-layered-memory@0.5.4)
61
+ # Option 1: run the official CLI directly via npx (no pre-installed dsh; version can be pinned, e.g. dsh-layered-memory@0.6.1)
73
62
  npx -y @deepseek-ai/dsh plugin --profile web add dsh-layered-memory
74
63
 
75
64
  # Option 2: with the dsh CLI installed (dsh is a pnpm forwarder; npm i -g pnpm first if missing)
@@ -106,6 +95,15 @@ npm run smoke # smoke test (rebuild first: see command b
106
95
  npx tsc src/smoke.ts --outDir dist-smoke --module nodenext --moduleResolution nodenext --target es2022 --strict --skipLibCheck --esModuleInterop
107
96
  ```
108
97
 
98
+ ## UI Preview
99
+
100
+ <p align="center">
101
+ <img src="./assets/img/ui-dark.jpg" width="49.5%"
102
+ alt="Settings memory browser overview in dark theme: status card (plugin version, capture/distill/recall switch states, FTS and vector capabilities, L1 memory count, distillation model) and stat tiles, glassy controls with a cold-blue accent">
103
+ <img src="./assets/img/ui-light.jpg" width="49.5%"
104
+ alt="The same settings memory browser overview in light theme: identical layout and information on light card backgrounds with the same accent family, theme switch without reload">
105
+ </p>
106
+
109
107
  ## Configuration
110
108
 
111
109
  Override configs go into the profile's own `cordis.patch.yml` as a **top-level bare
@@ -141,6 +139,8 @@ the bundle layer appends and causes `duplicate loader entry id` startup failure)
141
139
  | `l3.interval` | `20` | L3 distillation interval (new-memory count) |
142
140
  | `recall.enabled` | `true` | Auto recall |
143
141
  | `recall.maxResults` | `5` | L1 records injected per step |
142
+ | `recall.includePersona` | `true` | Inject persona context on recall (`<user-persona>`) |
143
+ | `recall.includeSceneNav` | `true` | Inject scene navigation on recall (`<scene-navigation>`) |
144
144
  | `recall.strategy` | `hybrid` | Retrieval strategy: `keyword` / `embedding` / `hybrid` |
145
145
  | `recall.scoreThreshold` | `0.3` | Recall score threshold (below is not injected; applies to keyword/embedding only, not pre-fusion hybrid; tool path unfiltered) |
146
146
  | `embedding.enabled` | `false` | Vector retrieval switch; off = pure FTS |
@@ -148,56 +148,74 @@ the bundle layer appends and causes `duplicate loader entry id` startup failure)
148
148
  | `embedding.apiKey` | empty | API key |
149
149
  | `embedding.model` | empty | embedding model name |
150
150
  | `embedding.dimensions` | `0` | Vector dimensions (required when enabled; must match model output) |
151
+ | `embedding.maxInputChars` | `5000` | Max characters per text (overlong inputs truncated) |
152
+ | `embedding.timeoutMs` | `10000` | Per-call embedding timeout (ms) |
153
+ | `embedding.allowLocalModels` | `true` | Allow the local embedding tier (deployment ceiling; when off, no model downloads and no local tier in settings) |
154
+ | `embedding.mirror` | `https://hf-mirror.com` | Download mirror root for local models (can be changed back to `https://huggingface.co`) |
151
155
  | `llm.provider/model` | empty | Distillation model override (defaults to current selection) |
152
- | `llm.maxTokens` | `20000` | Output token cap per distillation call (unified across stages; a reasoning model's reasoning shares this budget — too low gets fully consumed by thinking, leaving 0 chars of text) |
156
+ | `llm.maxTokens` | `256000` | Output token cap per distillation call (unified across stages; a reasoning model's reasoning shares this budget — too low gets fully consumed by thinking, leaving 0 chars of text) |
157
+ | `llm.reasoningEffort` | `off` | Distillation reasoning-effort tier (deployment default): `off` / `high` / `max`; empty string = don't send (follow model default). Distillation is structured extraction, so thinking is off by default — a reasoning model (e.g. v4-flash) at its default `high` tier can consume the entire output budget on thinking, leaving 0 chars of text; set to empty string for models that don't recognize the effort parameter. Switchable at runtime in Settings → Memory → Overview ("follow config" falls back to this value) |
153
158
  | `llm.temperature` | `0.3` | Distillation temperature |
154
159
  | `llm.maxInputChars` | `700000` | Input character budget per distillation call (over-budget L1 inputs are chunked automatically) |
160
+ | `llm.timeoutMs` | `120000` | Per-call distillation timeout (ms) |
155
161
  | `tools` | `true` | Whether to register model-callable memory tools |
156
162
 
157
163
  ## Storage Layout
158
164
 
159
- Aligned with MemoryCore's official dual-write architecture: append-only JSONL files
160
- (`conversations/`, `records/` sharded by day) are the source of truth for
161
- backup/restore and are **never rewritten**; `memory.db` (node:sqlite + WAL + FTS5
162
- BM25 + sqlite-vec cosine vectors) is the primary retrieval engine — updates/deletes
163
- from dedup merges touch the retrieval DB only.
164
-
165
- - **Three retrieval strategies** (`recall.strategy`): `keyword` (FTS5 BM25) /
166
- `embedding` (vec0 cosine KNN) / `hybrid` (both lanes in parallel + RRF k=60
167
- fusion, default); `conversation_search` (L0) uses the same fusion;
168
- - **Vectors optional**: off by default (pure FTS). DSH's `ctx.llm` has no embeddings
169
- endpoint; enabling requires any OpenAI-compatible `/embeddings` service (configure
170
- `embedding.*`); config changes automatically drop the vector table and re-embed
171
- everything in the background;
172
- - **Degradation chain**: sqlite-vec load failure → pure FTS; embedding call failure
173
- → degrade to FTS for that call with a one-time warning; retrieval DB init failure
174
- → memory features disabled entirely but dsh itself starts normally;
175
- - **Replacement seam**: `L1Store.search()` is the single retrieval entry point.
165
+ <p align="center">
166
+ <img src="./assets/readme/storage.svg" width="100%"
167
+ alt="Storage layout: dual-write architecture (append-only JSONL source of truth + memory.db retrieval engine); file forms include conversations/records/scenes/persona/state/pending/session-modes/embedding-source/model catalog/inference runtime/log and rebuild archives; three retrieval strategies keyword/embedding/hybrid (RRF k=60); a degradation chain never blocks the host">
168
+ </p>
176
169
 
177
- ```
178
- memory/
179
- ├── memory.db # SQLite retrieval DB (L0/L1 metadata + FTS5 + optional vectors; L1 has a family column)
180
- ├── conversations/2026-01-01.jsonl # L0 raw conversation source of truth (one file per day, append-only; not family-split)
181
- ├── records/2026-01-01.jsonl # L1 atomic memory source of truth (one file per day, append-only; family field included)
182
- ├── scenes/chat/*.md # L2 scene blocks (chat family)
183
- ├── scenes/work/*.md # L2 scene blocks (work family)
184
- ├── persona-chat.md # L3 persona (chat family: user persona)
185
- ├── persona-work.md # L3 persona (work family: Team Operating Doctrine)
186
- ├── state.json # pipeline checkpoint (v2 family-split: families.chat / families.work)
187
- ├── session-modes.json # session mode map (sessionId → mode, auto-pruned after >90 days)
188
- └── memory.log # diagnostic log (info+, rotates to memory.log.1 beyond 2MB)
189
- ```
170
+ Vectors are off by default (pure FTS). DSH's `ctx.llm` has no embeddings endpoint;
171
+ semantic retrieval is provided by a **three-state embedding source** (off / remote /
172
+ local), switchable at runtime in the settings page see the next section.
173
+
174
+ ## Semantic Retrieval (Embedding Source)
175
+
176
+ Pick the embedding source in Settings → Memory → Overview → Semantic Retrieval;
177
+ it takes effect immediately, no config edit or restart:
178
+
179
+ | Source | Description |
180
+ | --- | --- |
181
+ | **Off** (default) | No vector embedding at all; pure BM25 keyword retrieval |
182
+ | **Remote** | Bring any OpenAI-compatible `/embeddings` service (selectable only when the `embedding.*` quartet is configured) |
183
+ | **Local** | Pick from a built-in model catalog, ONNX-quantized **CPU inference** — no API key, data never leaves the machine |
184
+
185
+ The local catalog is a built-in allowlist (each model pinned to a revision with
186
+ per-file sha256; arbitrary repos cannot be downloaded):
187
+
188
+ | Model | Dims | Context | Size | Notes |
189
+ | --- | --- | --- | --- | --- |
190
+ | BGE small Chinese | 512 | 512 | ~25MB | Fastest on CPU; good first taste of semantic retrieval |
191
+ | EmbeddingGemma 300M | 768 | 2048 | ~330MB | 100+ languages incl. Chinese; balanced quality/cost (same as upstream MemoryCore) |
192
+ | BGE-M3 | 1024 | 8192 | ~590MB | Best Chinese quality; a single embedding can take seconds |
193
+
194
+ - **Download**: one click on the model card (default mirror `hf-mirror.com`, resumable
195
+ downloads + sha256 integrity checks); stored under `models/<id>/` in the data
196
+ directory, deletable from the settings page at any time;
197
+ - **On-demand runtime**: the inference runtime (transformers.js, ~100–200MB) is
198
+ installed only on first switch to the local tier, into `runtime/` in the data
199
+ directory — never in the plugin's dependency tree or install directory;
200
+ - **Live switching**: one click to swap sources — everything is re-embedded in the
201
+ background (visible progress, cancellable; retrieval silently degrades to keywords
202
+ in the meantime, conversations unaffected; a dimension change rebuilds the vector
203
+ table at the new size); a failed switch keeps the old source, which a restart
204
+ still uses;
205
+ - **Effective = deployment ceiling AND runtime choice**: `embedding.allowLocalModels=false`
206
+ disables the local tier entirely; without the `embedding.*` quartet the remote tier
207
+ is unavailable (enterprise deployments can lock this down). The choice persists in
208
+ `embedding-source.json`.
190
209
 
191
210
  ## Logging & Troubleshooting
192
211
 
193
212
  The dsh host prints plugin logs to the console; the plugin mirrors info and above to
194
213
  `memory.log` in its data directory. The typical log path of one conversation turn:
195
- `L0 捕获 turn=N …条` → `L0 落盘 N 条` `蒸馏管线开始(…待重试 M 条)` →
196
- `LLM 调用 provider/model:输入 x 输出 y 字符(z s)` → `L1 抽取完成(…新增 Y 条)`
197
- `蒸馏管线结束`; the next turn shows `召回命中 N L1`. Empty LLM output carries full
198
- diagnostics (finish reason / token counts / reasoning excerpt); JSON parse failures
199
- include the first 400 characters of the raw model output; all failure warns carry the
200
- first stack frame.
214
+ `L0 capture` `L0 flush` → `distillation pipeline start``LLM call (input/output
215
+ chars, duration)` → `L1 extraction done` → `pipeline end`; the next turn shows
216
+ `recall hit N L1 records`. Empty LLM output carries full diagnostics (finish reason /
217
+ token counts / reasoning excerpt); JSON parse failures include the first 400 characters
218
+ of the raw model output; all failure warns carry the first stack frame.
201
219
 
202
220
  ## Differences from MemoryCore
203
221
 
@@ -206,6 +224,13 @@ first stack frame.
206
224
  - Recall injection happens at `agent/pre-step` + agent-scoped `systemPrompt.context` (DSH-native events/services);
207
225
  - Storage/retrieval is a single-machine slimmed version of the official sqlite backend (drops multi-tenant isolation columns, TCVDB cloud backend, audit tables; tokenization uses a bundled CJK bigram instead of jieba, keeping zero native dependencies — sqlite-vec is the only native extension, auto-degrading on load failure).
208
226
 
227
+ ## Credits
228
+
229
+ The core memory capabilities (layered distillation pipeline, prompt design, and the
230
+ dual-write storage architecture) are modeled after **MemoryCore** from
231
+ [TencentCloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory).
232
+ Thanks to the original project for open-sourcing its design and implementation.
233
+
209
234
  ## License
210
235
 
211
236
  [MIT](LICENSE)
package/README.md CHANGED
@@ -1,74 +1,57 @@
1
1
  **简体中文** | [English](README.en.md)
2
2
 
3
3
  <p align="center">
4
- <img src="./assets/readme/hero.svg" width="100%"
5
- alt="dsh-layered-memory:对话自动分层蒸馏成记忆,模型每步前自动召回注入(L0 原始对话 L1 原子记忆 → L2 场景块 → L3 核心画像)">
4
+ <img src="./assets/img/Hero.png" width="100%"
5
+ alt="DeepSeek Harness hero 横幅:对话自动分层蒸馏成记忆,模型每步前自动召回注入——右侧对话气泡逐层溶解为三层渐亮光带,流入带发光圆球与渐变轨道的玻璃胶囊(下有 日常·工作·智能·关闭 四档刻度),光丝回流示意召回注入">
6
6
  </p>
7
7
 
8
8
  # dsh-layered-memory
9
9
 
10
10
  DeepSeek Harness 的**分层蒸馏记忆插件**(持久组合插件):对话在后台自动完成
11
11
  L0 捕获 → L1 原子记忆 → L2 场景整合 → L3 画像蒸馏,模型每一步前自动把相关记忆
12
- 注入上下文——用户与模型都不需要做任何操作。移植自
13
- [MemoryCore](https://github.com/TencentDB-Agent-Memory)(TencentDB Agent Memory)的
14
- 管线设计:Prompt 原样保留,仅把"LLM 操作文件"的 L2/L3 流程适配为"LLM 输出、工程侧执行"。
12
+ 注入上下文——用户与模型都不需要做任何操作。
15
13
 
16
- ## 核心能力
14
+ > 本插件的记忆核心能力(L0–L3 分层蒸馏管线、Prompt 与双写存储设计)参考自
15
+ > [TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)
16
+ > 中的 **MemoryCore**:Prompt 原样保留,仅把"LLM 操作文件"的 L2/L3 流程适配为
17
+ > "LLM 输出、工程侧执行"。
17
18
 
18
- ### 分层记忆(L0–L3)
19
+ ## 运行时数据流
19
20
 
20
- | 层 | 内容 | 存储 |
21
- | --- | --- | --- |
22
- | L0 | 每轮 user/assistant 消息(清洗去噪、剥离代码块与注入标签) | `conversations/YYYY-MM-DD.jsonl` + SQLite |
23
- | L1 | 情境切分 + 记忆提取(chat: persona/episodic/instruction;work: work_fact/work_task/work_method/work_artifact)+ 冲突检测去重合并,每条带族标签 | `records/YYYY-MM-DD.jsonl` + SQLite(FTS5 + 可选向量,family 列) |
24
- | L2 | 把新记忆整合为 Markdown 场景文档(META 块、热度管理、合并上限),按族各自整合 | `scenes/chat/*.md`、`scenes/work/*.md` |
25
- | L3 | 从变化场景蒸馏画像(chat: 用户画像 ≤2000 字;work: Team Operating Doctrine ≤1200 字),每族一份 | `persona-chat.md`、`persona-work.md` |
21
+ <p align="center">
22
+ <img src="./assets/readme/flow.svg" width="100%"
23
+ alt="dsh-layered-memory 运行时数据流:左侧 User Assistant 的会话事件流入插件(L0 捕获、L1–L3 蒸馏、检索召回、记忆工具),插件经 agent/pre-step 把相关记忆注入右侧 DSH 核心;蒸馏复用核心的 ctx.llm,数据双写 ~/.dsh/memory/">
24
+ </p>
26
25
 
27
- 管线原则:所有蒸馏调用复用 DSH 自身的 `ctx.llm`;任何阶段失败只记日志、绝不阻塞
28
- Agent 循环;召回注入用标签包裹、捕获侧自动剥离,防止反馈循环。
26
+ 插件挂在 dsh 原生事件缝上(`session/event` 捕获、`agent/pre-step` 注入),
27
+ 蒸馏调用复用宿主 `ctx.llm`,全程对用户与模型透明。另注册三个模型可主动调用的
28
+ 记忆工具:`memory_search` / `conversation_search` / `memory_read_scene`。
29
29
 
30
- **未蒸馏缓冲持久化**:抽取失败的待重试消息与攒触发阈值中途的消息都暂存在按档分桶的
31
- 缓冲里(`pending.json`,每次蒸馏尝试后原子落盘)——重启不丢,启动 20 秒后自动补跑一次,
32
- 失败则维持"等下一轮同档对话"的语义。
30
+ ## 分层记忆(L0–L3)
33
31
 
34
- **重建**(设置页 → 记忆 → 概览 → 重建记忆):以 L0 原始对话为事实源重新推导全部派生层。
35
- `records/`、`scenes/`、`persona-*.md` 整体归档(改名 `*.bak.<时间戳>`,不删除),检索库
36
- L1 清空、checkpoint 重置,随后按会话分块、统一 auto 档重蒸馏 L1→L2→L3(收尾强制跑一轮
37
- L2 残余 + L3 冷启动)。重建分块走低优先级队列——期间正常对话的蒸馏优先进行;带确认弹窗
38
- (会话数/消息数/预计调用数)、进度条与取消(已重建部分保留)。
32
+ <p align="center">
33
+ <img src="./assets/img/Layers.png" width="100%"
34
+ alt="分层记忆四层(自左上向右下逐层精炼):L0 原始对话(对话气泡)→ L1 原子记忆(发光事实粒子)→ L2 场景块(玻璃文档板)→ L3 核心画像(发光晶核);层间由 LLM 提取/整合/蒸馏光束相连,宽度递减表示数据逐层精炼">
35
+ </p>
39
36
 
40
- ### 会话级记忆档位
37
+ ## 会话级记忆档位
41
38
 
42
- 每个会话可独立选择记忆档位,**写入与召回同档**:
39
+ <p align="center">
40
+ <img src="./assets/img/Modes.png" width="100%"
41
+ alt="会话级记忆档位:一条玻璃胶囊滑轨四个停点(日常·工作·智能·关闭),发光圆球停在智能(默认)档;各档上方微场景——日常为个人聊天气泡、工作为代码文档窗格、智能为双流合流最亮、关闭为暗淡虚线幽灵泡">
42
+ </p>
43
43
 
44
- | 档位 | 蒸馏(写入) | 召回(注入) |
45
- | --- | --- | --- |
46
- | `自动`(默认) | 合并词表 prompt 单次抽取,个人三类 + 工作四类全开,按 type 前缀落族标签 | 两族全召回;画像/场景导航按类别归组、`<domain>` 分域结构化注入 |
47
- | `chat` | 窄 prompt 只提个人三类,只入 chat 族 | 只查 chat 记忆 + chat 画像/场景导航 |
48
- | `work` | 窄 prompt 只提工作四类,只入 work 族 | 只查 work 记忆 + work 画像/场景导航 |
49
- | `关闭` | 不写 L0、不蒸馏 | 不召回;三个记忆工具返回"已隐身"提示 |
50
-
51
- - **控件**:输入栏内、模式选择器右侧的 pill(`记忆·自动`),点击在上方浮出 macOS 风格
52
- 滑动选择器——拖拽松手吸附最近档位;
53
- - **默认档** = 配置 `family`(`auto|chat|work`,默认 `auto`);每会话的选择按 sessionId
54
- 持久化到 `session-modes.json`,重启/恢复会话不丢;中途切档下一轮生效,已提取记忆留在原族;
55
- - 与全局开关叠加(全局是总闸);L2/L3 完全分族,间族内容不渗透。
56
-
57
- ### 记忆浏览器(设置页 → 记忆)
58
-
59
- 多 Tab 页面,两族混合视图:**概览**(各层计数 + 记忆模式开关面板 + 蒸馏思考档位选择器,5 秒自动刷新)、
60
- **记忆**(L1 卡片列表,关键词/类型/情境筛选)、**场景**(L2 全文)、**画像**(L3 全文)、
61
- **日志**(`memory.log` 尾部 200 行)。开关与思考档位走官方 settings 服务(命名空间 `dsh-memory`,
62
- 实时生效、重启保留);开关生效规则 = **静态 config(部署上限)AND 运行时开关**;
63
- 思考档位为运行时覆盖(选择器选"跟随配置"则用部署配置 `llm.reasoningEffort` 作默认),
64
- 数据通道为 loopback RPC(`dsh-memory/*`)。
44
+ - **控件**:输入栏内、模式选择器右侧的 pill(`记忆·自动`),点击在上方浮出 macOS
45
+ 风格滑动选择器——拖拽松手吸附最近档位;深浅主题自适应;
46
+ - 每会话的选择按 sessionId 持久化到 `session-modes.json`,重启/恢复会话不丢;
47
+ 与全局开关叠加(全局是总闸);L2/L3 完全分族,跨族内容不渗透。
65
48
 
66
49
  ## 快速开始
67
50
 
68
51
  需要 Node ≥ 22.16。两种调用方式任选(`npx` 前缀可替换下面任何 `dsh` 命令):
69
52
 
70
53
  ```bash
71
- # 方式一:npx 直接跑官方 CLI(无需预装 dsh;可 pin 版本,如 dsh-layered-memory@0.5.4
54
+ # 方式一:npx 直接跑官方 CLI(无需预装 dsh;可 pin 版本,如 dsh-layered-memory@0.6.1
72
55
  npx -y @deepseek-ai/dsh plugin --profile web add dsh-layered-memory
73
56
 
74
57
  # 方式二:已装 dsh CLI(dsh 是 pnpm 转发器,未装 pnpm 时先 npm i -g pnpm)
@@ -101,6 +84,15 @@ npm run smoke # 冒烟测试(先重编:见下方命
101
84
  npx tsc src/smoke.ts --outDir dist-smoke --module nodenext --moduleResolution nodenext --target es2022 --strict --skipLibCheck --esModuleInterop
102
85
  ```
103
86
 
87
+ ## 界面预览
88
+
89
+ <p align="center">
90
+ <img src="./assets/img/ui-dark.jpg" width="49.5%"
91
+ alt="深色主题下的设置页记忆浏览器概览:状态卡(插件版本、捕获/蒸馏/召回开关状态、FTS 与向量能力、L1 记忆计数、蒸馏模型)与统计瓦片,玻璃质感控件与冷蓝强调色">
92
+ <img src="./assets/img/ui-light.jpg" width="49.5%"
93
+ alt="浅色主题下的同一设置页记忆浏览器概览:同款布局与信息,浅色卡片底与同套强调色,主题切换无需重载">
94
+ </p>
95
+
104
96
  ## 配置
105
97
 
106
98
  覆盖配置写在 profile 自己的 `cordis.patch.yml`,用**顶层裸 patch 条目**(直接 `id:`,
@@ -136,6 +128,8 @@ npx tsc src/smoke.ts --outDir dist-smoke --module nodenext --moduleResolution no
136
128
  | `l3.interval` | `20` | L3 蒸馏间隔(新记忆条数) |
137
129
  | `recall.enabled` | `true` | 自动召回 |
138
130
  | `recall.maxResults` | `5` | 每步召回注入的 L1 条数 |
131
+ | `recall.includePersona` | `true` | 召回时注入画像上下文(`<user-persona>`) |
132
+ | `recall.includeSceneNav` | `true` | 召回时注入场景导航(`<scene-navigation>`) |
139
133
  | `recall.strategy` | `hybrid` | 检索策略:`keyword` / `embedding` / `hybrid` |
140
134
  | `recall.scoreThreshold` | `0.3` | 召回分数阈值(低于不注入;仅 keyword/embedding 策略生效,hybrid 融合前不过滤;工具路径不过滤) |
141
135
  | `embedding.enabled` | `false` | 向量检索开关;关闭即纯 FTS 运行 |
@@ -143,52 +137,64 @@ npx tsc src/smoke.ts --outDir dist-smoke --module nodenext --moduleResolution no
143
137
  | `embedding.apiKey` | 空 | API Key |
144
138
  | `embedding.model` | 空 | embedding 模型名 |
145
139
  | `embedding.dimensions` | `0` | 向量维度(启用时必填,须与模型输出一致) |
140
+ | `embedding.maxInputChars` | `5000` | 单条文本最大字符数(超长截断) |
141
+ | `embedding.timeoutMs` | `10000` | 单次 embedding 调用超时(ms) |
142
+ | `embedding.allowLocalModels` | `true` | 允许本地嵌入档(部署上限:关闭后设置页不能下载模型、不能切本地档) |
143
+ | `embedding.mirror` | `https://hf-mirror.com` | 本地模型下载镜像根地址(可改回官方 `https://huggingface.co`) |
146
144
  | `llm.provider/model` | 空 | 蒸馏模型覆盖(默认用当前默认选择) |
147
145
  | `llm.maxTokens` | `256000` | 单次蒸馏输出 token 上限(全阶段统一;推理模型的 reasoning 与正文共享该预算,过低会被思考吃光导致正文 0 字符) |
148
146
  | `llm.reasoningEffort` | `off` | 蒸馏思考档位(部署默认):`off` / `high` / `max`,空串不传(跟随模型默认)。蒸馏是结构化抽取任务,默认关思考——推理模型(如 v4-flash)默认 high 档的思考可把任意输出预算全部吃光导致正文 0 字符;非推理模型不认识 effort 时需设为空串。运行时可在设置页 → 记忆 → 概览临时切换(选"跟随配置"即回退本值) |
149
147
  | `llm.temperature` | `0.3` | 蒸馏温度 |
150
148
  | `llm.maxInputChars` | `700000` | 单次蒸馏输入字符预算(超限的 L1 输入自动分块抽取) |
149
+ | `llm.timeoutMs` | `120000` | 单次蒸馏调用超时(ms) |
151
150
  | `tools` | `true` | 是否注册模型可调用的记忆工具 |
152
151
 
153
152
  ## 存储布局
154
153
 
155
- 对齐 MemoryCore 官方双写架构:JSONL 追加文件(`conversations/`、`records/` 按天分片)是
156
- 备份/恢复的事实源,**只增不改**;`memory.db`(node:sqlite + WAL + FTS5 BM25 + sqlite-vec
157
- 余弦向量)是主检索引擎,去重合并的更新/删除只动检索库。
154
+ <p align="center">
155
+ <img src="./assets/readme/storage.svg" width="100%"
156
+ alt="存储布局:双写架构(JSONL 事实源只增不改 + memory.db 主检索库);文件形态含 conversations/records/scenes/persona/state/pending/session-modes/embedding-source/模型目录/推理运行时/日志与重建归档;检索三策略 keyword/embedding/hybrid(RRF k=60);降级链保证永不阻塞宿主">
157
+ </p>
158
158
 
159
- - **检索三策略**(`recall.strategy`):`keyword`(FTS5 BM25)/ `embedding`(vec0 余弦 KNN)/
160
- `hybrid`(双路并行 + RRF k=60 融合,默认);`conversation_search`(L0)同款融合;
161
- - **向量能力可选**:默认关闭(纯 FTS)。DSH 的 `ctx.llm` 无 embeddings 端点,启用需自备
162
- 任意 OpenAI 兼容 `/embeddings` 服务(配置 `embedding.*`);配置变化自动 drop 向量表并
163
- 后台全量重嵌入;
164
- - **降级链**:sqlite-vec 加载失败 → 纯 FTS;embedding 调用失败 → 该次降级 FTS 并告警一次;
165
- 检索库初始化失败 → 记忆功能整体停用但 dsh 本体照常启动;
166
- - **替换缝**:`L1Store.search()` 是唯一检索入口。
159
+ 向量能力默认关闭(纯 FTS)。DSH `ctx.llm` embeddings 端点,语义检索由
160
+ **三态嵌入源**提供(关闭 / 远程 / 本地),设置页可运行时切换——见下节。
167
161
 
168
- ```
169
- memory/
170
- ├── memory.db # SQLite 检索库(L0/L1 元数据 + FTS5 + 可选向量;L1 带 family 列)
171
- ├── conversations/2026-01-01.jsonl # L0 原始对话事实源(每天一个文件,追加;不分族)
172
- ├── records/2026-01-01.jsonl # L1 原子记忆事实源(每天一个文件,追加;含 family 字段)
173
- ├── scenes/chat/*.md # L2 场景块(chat 族)
174
- ├── scenes/work/*.md # L2 场景块(work 族)
175
- ├── persona-chat.md # L3 画像(chat 族:用户画像)
176
- ├── persona-work.md # L3 画像(work 族:Team Operating Doctrine)
177
- ├── state.json # 管线 checkpoint(v2 分族:families.chat / families.work)
178
- ├── pending.json # 未蒸馏缓冲(按档分桶;重启恢复 + 启动自动补跑)
179
- ├── session-modes.json # 会话档位映射(sessionId → 档位,>90 天自动清理)
180
- ├── records.bak.<ts>/ # 重建时归档的旧产物(scenes/persona 同款 *.bak.<ts>)
181
- └── memory.log # 诊断日志(info+,超 2MB 轮转为 memory.log.1)
182
- ```
162
+ ## 语义检索(嵌入源)
163
+
164
+ 设置页(记忆 概览 语义检索)选择嵌入源,即时生效、无需改配置重启:
165
+
166
+ | 嵌入源 | 说明 |
167
+ | --- | --- |
168
+ | **关闭**(默认) | 不做任何向量嵌入,纯 BM25 关键词检索 |
169
+ | **远程** | 自备任意 OpenAI 兼容 `/embeddings` 服务(`embedding.*` 四件套配齐才可选) |
170
+ | **本地** | 内置模型目录选一款,ONNX 量化 **CPU 推理**——无需 API Key,数据不出本机 |
171
+
172
+ 本地模型目录是插件内置白名单(每款锁定 revision + 每文件 sha256,不可下载任意仓库):
173
+
174
+ | 模型 | 维度 | 上下文 | 体积 | 特点 |
175
+ | --- | --- | --- | --- | --- |
176
+ | BGE small 中文 | 512 | 512 | ~25MB | CPU 嵌入最快,适合先体验语义检索 |
177
+ | EmbeddingGemma 300M | 768 | 2048 | ~330MB | 100+ 语言含中文,质量与开销均衡(上游 MemoryCore 同款) |
178
+ | BGE-M3 | 1024 | 8192 | ~590MB | 中文质量最强,单条嵌入可达秒级 |
179
+
180
+ - **下载**:模型卡一键下载(默认镜像 `hf-mirror.com`,断点续传 + sha256 完整性
181
+ 校验),落盘数据目录 `models/<id>/`,不用了随时在设置页删除;
182
+ - **按需运行时**:首次切换本地档才安装推理运行时(transformers.js,约 100~200MB,
183
+ 装进数据目录 `runtime/`——不进插件依赖树,不碰插件安装目录);
184
+ - **活切换**:一键换源——自动后台全量重嵌(进度可见、可取消,期间检索自动降级
185
+ 关键词,不影响对话;维度变化时向量表按新维度重建);切换失败保持旧源,重启仍
186
+ 按原源运行;
187
+ - **生效规则 = 部署上限 AND 运行时选择**:`embedding.allowLocalModels=false` 可整体
188
+ 禁用本地档、未配 `embedding.*` 四件套则远程档不可选(企业部署可收口),状态持久
189
+ 化在 `embedding-source.json`。
183
190
 
184
191
  ## 日志与排查
185
192
 
186
- dsh 宿主把插件日志打到控制台,插件将 info 及以上镜像到数据目录 `memory.log`。一轮对话的
187
- 典型日志路径:`L0 捕获 turn=N …条` → `L0 落盘 N 条``蒸馏管线开始(…待重试 M 条)` →
188
- `LLM 调用 provider/model:输入 x输出 y 字符(z s)` → `L1 抽取完成(…新增 Y 条)` →
189
- `蒸馏管线结束`;下一轮有 `召回命中 N L1`。LLM 空输出会带完整诊断(finish 原因 /
190
- token 计数 / reasoning 摘录);JSON 解析失败附带模型原始输出前 400 字符;所有失败 warn
191
- 携带错误堆栈首帧。
193
+ dsh 宿主把插件日志打到控制台,插件将 info 及以上镜像到数据目录 `memory.log`。一轮
194
+ 对话的典型日志路径:`L0 捕获` → `L0 落盘` `蒸馏管线开始``LLM 调用(输入/输出
195
+ 字符数、耗时)` `L1 抽取完成``蒸馏管线结束`;下一轮有 `召回命中 N L1`。
196
+ LLM 空输出带完整诊断(finish 原因 / token 计数 / reasoning 摘录),JSON 解析失败
197
+ 附模型原始输出前 400 字符,失败 warn 均带堆栈首帧。
192
198
 
193
199
  ## 与 MemoryCore 的差异
194
200
 
@@ -196,7 +202,14 @@ token 计数 / reasoning 摘录);JSON 解析失败附带模型原始输出
196
202
  - L2/L3 由"LLM 操作文件工具"改为"LLM 输出操作 JSON / 完整文档,工程侧执行";
197
203
  - 召回注入点在 `agent/pre-step` + agent 作用域 `systemPrompt.context`(DSH 原生事件/服务);
198
204
  - 存储/检索即官方 sqlite 后端的单机裁剪版(裁掉多租户隔离列、TCVDB 云后端、审计表;
199
- 分词用自带 CJK 二元组替代 jieba,保持零原生依赖——仅 sqlite-vec 一个原生扩展,加载失败自动降级)。
205
+ 分词用自带 CJK 二元组替代 jieba,保持零原生依赖——仅 sqlite-vec 一个原生扩展,
206
+ 加载失败自动降级)。
207
+
208
+ ## 致谢
209
+
210
+ 记忆核心能力(分层蒸馏管线、Prompt 设计、双写存储架构)参考自
211
+ [TencentCloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)
212
+ 项目中的 **MemoryCore**,感谢原项目开放的设计与实现。
200
213
 
201
214
  ## License
202
215
 
Binary file
Binary file
Binary file
Binary file
Binary file