dsh-memoir 0.4.3 → 0.5.0

package/README.en.md

@@ -0,0 +1,265 @@
+# dsh-memoir
+
+[![npm version](https://img.shields.io/npm/v/dsh-memoir.svg)](https://www.npmjs.com/package/dsh-memoir)
+
+[中文](./README.md) · English
+
+**dsh-memoir is a local project-memory layer for DeepSeek Harness: it persists an agent's work conclusions, lessons learned, and next actions, then carries them across sessions through bounded Hot Memory injection, on-demand ranked recall, and Web GUI management.**
+
+> Cache-aware local project memory for DeepSeek Harness.
+
+- **Local-only** — all data stays on your machine (`~/.dsh/dsh-memoir.json` + per-project `PROJECT_MEMORY.md`)
+- **Zero external memory service** — no vector database, no embedding API, no cloud memory service
+- **Bounded hot-memory injection** — token-budgeted Hot Memory is injected into the system prompt (default 900/1200)
+- **Ranked local recall** — inverted index + BM25 local ranked retrieval; `memoir_read` fetches long-tail history on demand
+- **Web GUI** — a sidebar "Memory" panel with project/global browsing, relevance-ranked search, Hot Memory Inspector, and Retrieval Diagnostics
+
+## Quick Start
+
+```bash
+# install into the web profile from npm (recommended)
+dsh plugin --profile web add dsh-memoir
+
+# or install latest source from GitHub
+dsh plugin --profile web add github:Qinling-Melon-Farmers/dsh-memoir
+
+# or local development (after cloning)
+dsh plugin --profile web add link:/absolute/path/dsh-memoir
+```
+
+Restart DSH to take effect (`dsh web`), then use it normally:
+
+```text
+use the Agent as usual
+      ↓
+end of each worked turn: an automatic distill reminder
+      ↓
+memoir_record persists work / lessons / next steps
+      ↓
+future sessions auto-inherit Hot Memory (bounded, ranked, frozen per session)
+      ↓
+need long-tail history? memoir_read (local relevance-ranked recall)
+```
+
+## Architecture
+
+```text
+                   ~/.dsh/dsh-memoir.json
+                            │
+                            │ SSOT (single source of truth)
+                            ▼
+                      MemoirStore
+              ┌─────────────┴─────────────┐
+              │                           │
+              ▼                           ▼
+        PROJECT_MEMORY.md          Retrieval Index
+         human-readable             ranked recall
+         (git-committable)                │
+              │                           ▼
+              │                       memoir_read
+              │                       GUI /search
+              │
+              ▼
+       Hot Memory Selector
+         (token budget)
+              │
+              ▼
+       Session Snapshot
+         (frozen per session)
+              │
+              ▼
+         System Prompt
+```
+
+## Memory Model: Full Memory vs Hot Memory
+
+**Full Memory (complete history)** — the structured JSON SSOT plus the regenerated `PROJECT_MEMORY.md` projection. Used for: complete history, GUI browsing, git commits, manual inspection, and as the source data for ranked recall.
+
+**Hot Memory (bounded injection)** — high-value memories selected by the selector within a token budget, injected into the system prompt. Properties: **bounded / ranked / compact / session-frozen**.
+
+> v0.4+ no longer injects the full PROJECT_MEMORY.md into the model: Hot Memory goes to the prompt, long-tail history goes through ranked recall.
+
+**Session Snapshot freezing semantics**: one session's injected text is built once and frozen (stable prompt prefix, maximizing prompt-prefix cache hits); the current session does not re-consume memory it just wrote, and a new session rebuilds and sees the latest memory. Since v0.4.2, when there is no unique session identity (session.id / agent.id), freezing is skipped — a cache miss beats wrongly reusing another session's snapshot.
+
+## v0.5.0 lifecycle and rc8 compatibility
+
+- The development and peer-dependency baseline is `@deepseek-ai/dsh-* 0.1.0-rc.8`.
+- Store format v3 migrates v2 entries without changing their `id`, content, or timestamp. The first mutation materializes `importance`, `pinned`, `status`, `supersedes`, and `tags`; startup reads do not rewrite old files.
+- Retrieval defaults to `active`. Archived and superseded history is retained and can be inspected from the Web panel. Explicit `supersedes` marks its targets as superseded; history is never deleted automatically.
+- `PROJECT_MEMORY.md` is a human-readable projection. Only bounded Hot Memory enters the system prompt; the full file is not injected.
+- GET routes no longer register browser-supplied paths as active workspaces. Only the trusted system-prompt cwd grants panel write authorization. Lock metadata now includes pid, creation time, and nonce, with conservative reclaim only after 60 seconds and a dead owner.
+- `memoir_read(scope: 'all')` uses a deduplicated global ranking so project and global results are not repeated.
+
+## Tools
+
+| Tool | Purpose |
+| --- | --- |
+| `memoir_record` | write work / lessons / actions / note entries |
+| `memoir_read` | local relevance retrieval across project (default) / global / all, with limit and compact/full output shapes |
+
+`memoir_read`'s query description matches its real behavior: **local relevance retrieval over titles and content — supports Chinese phrases, English keywords, code identifiers, and paths, ordered by relevance**.
+
+## Retrieval
+
+- No embeddings, no vector database, no external memory service
+- Tokenization: Chinese 2/3-grams + English words + code/path identifiers
+- BM25 (documents keep true term frequency; queries are deduplicated)
+- 2.5× title boost, exact-phrase boost, section weight, recency decay
+- Separate length normalization for titles and bodies (v0.4.2)
+- Epoch-aware LRU query cache with 1-hour time buckets: limit/detail stay out of the cache key, so every output shape shares one ranked result (v0.4.2)
+- Query-cache metrics (hits/misses/evictions/hit rate) and Last Query (latency/candidates/returned) observability (v0.4.2)
+- Global recall limit is a true global Top-K; output truncation preserves the top-ranked head (v0.4.2)
+
+Curated-query Top-5 hit rate: 100% (quality gate ≥ 90%, see `test/recall-quality.test.ts`).
+
+## GUI
+
+The v0.4 Project / Global / Search / Add / Delete / Diagnostics architecture is kept; since v0.4.2:
+
+- **Search unified on RetrievalEngine**: a non-empty query calls `GET /api/dsh-memoir/search` — the same BM25 ranking as the agent's `memoir_read` — results ordered by relevance with scores shown
+- **Hot Memory Inspector**: expand to see the Hot Memory that will actually be injected for the current workspace (Actions / Lessons / Recent state) — i.e. "what exactly the next session inherits"
+- **Retrieval Diagnostics**: Retrieval Index (docs/terms/epoch), Query Cache (hits/misses/evictions/hit rate/size/capacity), Last Query (latency/returned), Session Snapshot (hash/createdAt/storeRevision)
+
+## Screenshots
+
+**1. Plugin active & overall UI**: the sidebar gains a "Memory" entry (alongside SSH / Task Board, mutually exclusive panels); clicking opens the memory panel in the center column.
+
+![Plugin active & overall UI](picture/插件生效和UI效果1.png)
+
+**2. Project memory**: the current project session's persistent memory grouped into Work Log / Lessons Learned / Action Guide / Notes; each entry shows time, section chip, title, content, and session origin, with search, refresh, and per-entry delete.
+
+![Project memory](picture/项目记忆2.png)
+
+**3. Manually adding memory**: a form to pick a section, a one-line title, and content — written to the same data the agent's `memoir_record` writes; PROJECT_MEMORY.md regenerates automatically after submit.
+
+![Manually adding memory](picture/手动添加记忆3.png)
+
+**4. Global memory management**: memory buckets for all projects (name, path, updated time, count) with cross-project search and per-entry maintenance.
+
+![Global memory management](picture/全局记忆管理4.png)
+
+**5. Ranked search + Hot Memory Inspector + Memory Diagnostics (v0.4.2)**: a typed query triggers RetrievalEngine-ranked recall with a relevance score on each result; at the bottom you can expand the Hot Memory Inspector (what the next session will inherit for the current workspace) and the extended Memory Diagnostics (Retrieval index / Query cache / Last query / Session snapshot).
+
+![Ranked search with Hot Memory Inspector and Memory Diagnostics](picture/hot%20memory预览与记忆诊断5.png)
+
+## Storage & Privacy
+
+```text
+~/.dsh/dsh-memoir.json        ← structured JSON (single source of truth / SSOT)
+<workspace>/PROJECT_MEMORY.md ← human-readable projection regenerated from the JSON (git-friendly)
+
+No cloud memory DB · No embedding API · No vector DB
+```
+
+JSON is the source of truth and Markdown is the generated projection: the panel, the tools, and the agent write the same data. Since v0.4.2 the panel write API is also workspace-authorized — an absolute path submitted by the browser is not authorization by itself; only the current active cwd or an existing store project can be written to.
+
+## Configuration
+
+Add a `config` block on the plugin row in `cordis.patch.yml` (all optional; defaults shown):
+
+```yaml
+- insert:
+    - id: memoir
+      name: dsh-memoir
+      config:
+        enabled: true            # master switch (tools, routes, prompt section)
+        announceToAgent: true    # system-prompt announcement section
+        autoDistill: true        # auto distill reminder after each worked turn
+        hotMemoryTokens: 900     # Hot Memory target tokens
+        hotMemoryMaxTokens: 1200 # Hot Memory hard ceiling (never exceeded)
+        readDefaultLimit: 8      # memoir_read default result count
+        readMaxLimit: 30         # memoir_read maximum result count
+        sessionSnapshotMax: 128  # per-session snapshot LRU cap
+        queryCacheSize: 128      # ranked-query LRU cache size
+```
+
+## Design Trade-offs
+
+- **Bounded vs full injection**: v0.3 injected the full history into the prompt and it kept growing; v0.4+ injects only budgeted Hot Memory, with long-tail history recalled on demand. Token benchmarks below.
+- **Frozen vs fresh**: within a session the injected text is frozen to gain prompt-prefix cache hits; without a unique session identity it is not frozen (v0.4.2), so new sessions always see new memory.
+- **Hot Memory quota**: Recent state (newest work, 1–3 entries) is guaranteed a floor, actions/lessons fill by ranking, and work only appears in Recent state — never injected twice (v0.4.2).
+- **Multi-process safety**: store record/remove runs inside a cross-process critical section on `~/.dsh/dsh-memoir.lock` (exclusive O_EXCL creation with timeout); the section force-reloads from disk before mutating, so two interleaved DSH processes lose no updates (v0.4.2).
+- **Windows paths**: canonical keys are fully lowercased (`C:\A` / `c:\a\` / `C:/A` share one bucket) while display paths keep the original casing (v0.4.2).
+- **GUI and Agent share one engine**: panel search and `memoir_read` use the same RetrievalEngine instead of separate filter logic (v0.4.2).
+
+## Use Cases
+
+| Scenario | How to use it |
+| --- | --- |
+| Recurring environment pitfalls (encoding / escaping / paths / permissions) | record a `lessons` entry with copy-pasteable fix commands |
+| Project rules and conventions (no emoji, run tests before release, branch policy) | record as `actions`, auto-injected for whoever takes over |
+| Root cause of a hard-to-find bug | record as `lessons` / `work` to avoid re-investigation |
+| Fixed deployment/release checklist | record as `actions`; new sessions follow it |
+| Reuse experience across projects | global tab or `memoir_read(scope: 'global', query: ...)` |
+
+Typical example: after solving "console Chinese mojibake" the first time, record the diagnosis and fix commands as a `lessons` entry (e.g. `chcp 65001 first … always write UTF-8 without BOM`); every new session in this project then inherits the lesson automatically instead of re-debugging, and cross-project global search hits it too. The memory plugin distills "root cause + fix command" into project knowledge — it does not fix the terminal's own encoding defects.
+
+## Comparison
+
+| Project | Primary focus |
+| --- | --- |
+| dsh-memory | citation / source-traceable reference memory |
+| dsh-mnemon | a heavier long-term memory system |
+| distill | distilling sessions into skills |
+| **dsh-memoir** | **lightweight project workflow memory: local, bounded injection, ranked recall** |
+
+Each plugin has its own focus — pick per need; no "which is stronger" narrative.
+
+## Development / Benchmark / Tests
+
+```bash
+pnpm install          # install devDeps (typescript, esbuild, @deepseek-ai/* type packages)
+pnpm run build        # tsc builds the host + esbuild builds the client bundle
+pnpm run typecheck    # full type check (src + test)
+pnpm test             # 131 tests: store (incl. multi-process lock) / snapshot / selector / retrieval / tools / routes / auto-distill / integration / client pure logic / bundle protocol & purity
+npm run bench         # benchmark (100/1k/10k/100k entries); results written to bench/report.md
+```
+
+Quality gates: **Top-5 recall ≥ 90% · Hot Memory ≤ configured hardMax · same-session prompt-prefix stability · global recall ≤ limit · zero lost updates across processes**.
+
+v0.4.2 benchmark summary (node v22.23.2, budget 900/1200 tokens; full report in `bench/report.md`. Methodology fixed: uncached queries measure `search()` directly; cached queries warm the same query first, then time it):
+
+| Entries | Cold load | Warm read | Hot Memory build | Index build | Uncached query | Cached query | Cache hit rate | Full markdown tokens | Injected tokens | Reduction |
+|---|---|---|---|---|---|---|---|---|---|---|
+| 100 | 1.3 ms | 2.22 µs | 0.54 ms | 2.9 ms | 0.224 ms | 2.87 µs | 50.0% | 3870 | 902 | 76.7% |
+| 1,000 | 1.6 ms | 0.40 µs | 0.70 ms | 15.0 ms | 1.419 ms | 1.45 µs | 50.0% | 38182 | 916 | 97.6% |
+| 10,000 | 25.3 ms | 0.42 µs | 2.60 ms | 142.9 ms | 11.889 ms | 1.14 µs | 50.0% | 385807 | 902 | 99.8% |
+| 100,000 | 158.2 ms | 0.42 µs | 31.97 ms | 2238.7 ms | 153.551 ms | 1.15 µs | 50.0% | 3907057 | 917 | 100.0% |
+
+## Implementation
+
+- **Full-stack TypeScript**: `src/host/*.ts` (store / tools / retrieval / selector / snapshot / routes / autodistill / index — tsc emits `lib/*.js`) + `src/client/*.ts(x)` (esbuild emits the `lib/client.js` closure-factory bundle).
+- **Two-sided plugin**: the host half registers the agent tools, `/api/dsh-memoir` routes, the `agent/turn-stopping` auto-distill listener, and the per-project system-prompt injection section; the client half renders the panel. Runtime deps are official NPM SDK packages only.
+- Mounted via the `dsh.bundle.patch` manifest (`insert` row in `cordis.patch.yml`); no DSH source changes.
+- Auto-distill safety boundaries: top-level sessions only (subagents / nested delegations skipped), turns with tool activity that haven't recorded yet, aborted turns skipped, at most one steer per turn.
+
+## Contributing
+
+PRs and issues are managed with templates and automation:
+
+- [CONTRIBUTING.md](CONTRIBUTING.md) — PR scope, commit conventions and checklist;
+- [ISSUE_TRIAGE.md](ISSUE_TRIAGE.md) — issue labels, classification and closing criteria;
+- `.github/ISSUE_TEMPLATE` — bug / request templates; `.github/pull_request_template.md` — PR template.
+
+Bug reports must include screenshot / log evidence, a smoke test, code references and a patch. New features and documentation-only PRs must first be discussed in an issue.
+
+## Release
+
+Version releases run automatically in `.github/workflows/publish.yml` when a `v*` tag is pushed: install deps, verify the tag matches the `package.json` version, run typecheck/test, then publish to npm. Configure either of these auth options in the repo:
+
+- npm Trusted Publishing: GitHub repo `Qinling-Melon-Farmers/dsh-memoir`, workflow `publish.yml`
+- GitHub Actions secret `NPM_TOKEN`: a granular token with publish rights and 2FA bypass allowed
+
+Publishing a patch release:
+
+```bash
+npm version patch
+git push
+git push origin vX.Y.Z  # use the actual version printed by npm version
+```
+
+`npm version patch` updates `package.json`, creates the version commit and the tag; no manual `git tag` or local `npm publish` needed.
+
+## License
+
+Apache-2.0

package/README.md

@@ -1,8 +1,10 @@
-# dsh-memoir
-
-[![npm version](https://img.shields.io/npm/v/dsh-memoir.svg)](https://www.npmjs.com/package/dsh-memoir)
-
-**dsh-memoir 是 DeepSeek Harness 的本地项目记忆层：把 Agent 的工作结论、经验教训和后续行动持久化，并通过有界 Hot Memory 自动继承、按需排序召回和 Web GUI 管理，实现跨会话项目记忆。**
+# dsh-memoir
+
+[![npm version](https://img.shields.io/npm/v/dsh-memoir.svg)](https://www.npmjs.com/package/dsh-memoir)
+
+[English](./README.en.md) · 中文
+
+**dsh-memoir 是 DeepSeek Harness 的本地项目记忆层：把 Agent 的工作结论、经验教训和后续行动持久化，并通过有界 Hot Memory 自动继承、按需排序召回和 Web GUI 管理，实现跨会话项目记忆。**
 
 > Cache-aware local project memory for DeepSeek Harness.
 
@@ -14,16 +16,16 @@
 
 ## Quick Start
 
-```bash
-# 从 npm 安装到 web profile（推荐）
-dsh plugin --profile web add dsh-memoir
-
-# 或从 GitHub 安装最新源码
-dsh plugin --profile web add github:Qinling-Melon-Farmers/dsh-memoir
-
-# 或本地开发（克隆后）
-dsh plugin --profile web add link:/绝对路径/dsh-memoir
-```
+```bash
+# 从 npm 安装到 web profile（推荐）
+dsh plugin --profile web add dsh-memoir
+
+# 或从 GitHub 安装最新源码
+dsh plugin --profile web add github:Qinling-Melon-Farmers/dsh-memoir
+
+# 或本地开发（克隆后）
+dsh plugin --profile web add link:/绝对路径/dsh-memoir
+```
 
 安装后重启 DSH 生效（`dsh web`）。正常使用即可：
 
@@ -79,6 +81,15 @@ memoir_record 沉淀工作 / 教训 / 下一步
 
 **Session Snapshot 冻结语义**：同一 session 的注入文本只构建一次并冻结（prompt 前缀稳定，最大化 prompt-prefix cache 命中）；当前 session 不重新消费自己刚写的记忆，新 session 重建并看到最新记忆。v0.4.2 起，没有唯一会话身份（session.id / agent.id）时**不做冻结**——宁可 cache miss，不可跨 session 错复用旧快照。
 
+## v0.5.0 生命周期与 rc8 兼容性
+
+- 当前开发基线为 `@deepseek-ai/dsh-* 0.1.0-rc.8`；peer dependency 与开发依赖已统一到 rc8。
+- 存储格式从 v2 迁移到 v3：旧条目保持原有 `id`、内容和时间，首次变更时补齐 `importance`、`pinned`、`status`、`supersedes` 与 `tags`；启动读取不会重写旧文件。
+- 默认只召回 `active` 条目；归档和被替代条目保留在历史中，可在 Web 面板切换状态查看。显式 `supersedes` 会把目标条目标记为 `superseded`，不会自动删除历史。
+- `PROJECT_MEMORY.md` 是人类可读投影；system prompt 只注入有界 Hot Memory，完整文件不会整体注入。
+- GET 路由不再把浏览器传入的路径登记为活动工作区；只有可信 system-prompt cwd 才能获得面板写权限。锁文件现在带 pid、创建时间和 nonce，只在超过 60 秒且 pid 已死亡时保守回收。
+- `memoir_read(scope: 'all')` 使用去重后的全局排序结果，避免当前项目与全局结果重复。
+
 ## Tools
 
 | 工具 | 作用 |
@@ -127,7 +138,9 @@ curated 查询 Top-5 命中率 100%（质量门禁 ≥90%，见 `test/recall-qua
 
 ![全局记忆管理](picture/全局记忆管理4.png)
 
-> v0.4.2 新增的带 query 的 ranked results 与 Hot Memory Inspector / Retrieval Diagnostics 截图将随版本发布补充。
+**5. 排序搜索 + Hot Memory 预览 + 记忆诊断（v0.4.2）**：搜索框输入 query 后走 RetrievalEngine 排序召回，每条结果带相关性分数；底部可展开「Hot Memory 预览」（查看当前工作区下一会话将自动继承的内容）与扩展后的 Memory Diagnostics（Retrieval 索引 / Query cache / 最近查询 / 会话快照）。
+
+![排序搜索与 Hot Memory 预览 / 记忆诊断](picture/hot%20memory预览与记忆诊断5.png)
 
 ## Storage & Privacy
 
@@ -213,30 +226,41 @@ v0.4.2 benchmark 摘要（node v22.23.2，budget 900/1200 tokens；完整报告
 | 10,000 | 25.3 ms | 0.42 µs | 2.60 ms | 142.9 ms | 11.889 ms | 1.14 µs | 50.0% | 385807 | 902 | 99.8% |
 | 100,000 | 158.2 ms | 0.42 µs | 31.97 ms | 2238.7 ms | 153.551 ms | 1.15 µs | 50.0% | 3907057 | 917 | 100.0% |
 
-## 实现说明
+## 实现说明
 
 - **TypeScript 全栈**：`src/host/*.ts`（store / tools / retrieval / selector / snapshot / routes / autodistill / index，tsc 构建出 `lib/*.js`）+ `src/client/*.ts(x)`（esbuild 打出 `lib/client.js` 闭包工厂 bundle）。
 - **双面插件**：host 半注册 agent 工具、`/api/dsh-memoir` 路由、`agent/turn-stopping` 自动收尾监听与按项目求值的 system prompt 注入段；client 半提供面板。运行时仅依赖官方 NPM SDK。
 - 通过 `dsh.bundle.patch` manifest（`cordis.patch.yml` 的 `insert` 行）挂载，不改 DSH 源码。
-- 自动收尾安全边界：仅顶级会话（跳过 subagent / 嵌套委托）、仅「有工具调用且未记录过」的回合、已中止回合不打扰、每回合至多一次。
-
-## Release
-
-版本发布由 `.github/workflows/publish.yml` 在 `v*` tag 推送后自动执行：安装依赖、校验 tag 与 `package.json` 版本一致、运行 typecheck/test，再发布到 npm。仓库需配置以下任一认证方式：
-
-- npm Trusted Publishing：GitHub 仓库 `Qinling-Melon-Farmers/dsh-memoir`，workflow `publish.yml`
-- GitHub Actions secret `NPM_TOKEN`：使用具有发布权限且允许绕过发布 2FA 的 granular token
-
-发布 patch 版本：
-
-```bash
-npm version patch
-git push
-git push origin vX.Y.Z  # 使用 npm version 输出的实际版本号
-```
-
-`npm version patch` 会修改 `package.json`、创建版本提交并创建对应 tag；无需再次执行 `git tag` 或在本机执行 `npm publish`。
-
-## 许可
+- 自动收尾安全边界：仅顶级会话（跳过 subagent / 嵌套委托）、仅「有工具调用且未记录过」的回合、已中止回合不打扰、每回合至多一次。
+
+## 贡献
+
+PR 与 Issue 采用模板化 + 自动化管理：
+
+- [CONTRIBUTING.md](CONTRIBUTING.md) — PR 范围、提交规范与检查清单；
+- [ISSUE_TRIAGE.md](ISSUE_TRIAGE.md) — Issue 标签体系、分类与关闭标准；
+- `.github/ISSUE_TEMPLATE` — Bug / 功能请求模板；`.github/pull_request_template.md` — PR 模板。
+
+Bug 报告需附截图 / 日志证据、冒烟测试、引用代码与补丁；全新功能与仅文档类
+PR 请先提 Issue 讨论。
+
+## Release
+
+版本发布由 `.github/workflows/publish.yml` 在 `v*` tag 推送后自动执行：安装依赖、校验 tag 与 `package.json` 版本一致、运行 typecheck/test，再发布到 npm。仓库需配置以下任一认证方式：
+
+- npm Trusted Publishing：GitHub 仓库 `Qinling-Melon-Farmers/dsh-memoir`，workflow `publish.yml`
+- GitHub Actions secret `NPM_TOKEN`：使用具有发布权限且允许绕过发布 2FA 的 granular token
+
+发布 patch 版本：
+
+```bash
+npm version patch
+git push
+git push origin vX.Y.Z  # 使用 npm version 输出的实际版本号
+```
+
+`npm version patch` 会修改 `package.json`、创建版本提交并创建对应 tag；无需再次执行 `git tag` 或在本机执行 `npm publish`。
+
+## 许可
 
 Apache-2.0