@yolk_vat-y/dsh-project-memory 0.3.2 → 0.3.4

package/CHANGELOG.md

@@ -1,5 +1,34 @@
 # Changelog
 
+## 0.3.4 (2026-09-02)
+
+### query_memory 性能优化：流式 TF + IDF 缓存
+- **IDF 缓存**：`store._version` + `store._idfCache`，`save()` 时 `version++` 标记失效，查询时版本命中直接复用，无需重建 BM25 索引
+- **预计算 searchText**：`setEntries` 时预计算 `entry.searchText`（标题×5 + keywords + summary + path 的小写拼接），查询时直接复用，避免重复字符串拼接与 `toLowerCase()`
+- **流式打分**：`rankEntriesStreaming` 单次遍历 entries，用 `countOccurrences()` 字符串计数替代完整 `tokenizeRaw` + TF 表构建，零中间对象分配
+- **性能提升**：5k 文件 / 20k 条目场景 `query_memory` 中位数 **187 ms → 9.3 ms**（20x）；1k 文件典型项目 **<1 ms**
+
+### 测试覆盖
+- 新增 `IDF caching & streaming TF` 测试组（7 项）：缓存构建、命中、版本失效、流式打分正确性、空查询、searchText 预计算
+- 新增 `query_memory streaming path` 集成测试（2 项）：端到端首次查询建缓存、二次查询复用缓存
+- 总测试数 157 → 166 全绿
+
+## 0.3.3 (2026-08-31)
+
+### 符号层：只存身份牌，不存行为
+- `src/symbols.js` `buildSymbol`：删除 `summary` 废话字段；`text` 不再截断，保存完整声明行（含签名）；返回一行身份牌 `fn(a: A, b: B): R — file.ts:42`；删除冗余 `sig` 字段
+
+### 文档层：答案级摘要 + 自报盲区
+- `src/llm.js` `extractDocEntry`：新增 `blindSpots` 字段（自报盲区，如 `// 未覆盖：部署细节、性能基准、v0.2 前 API`）；Prompt 要求 LLM 返回 `blindSpots`；摘要通过 `summarizeText()` 截断至 300 字符；`blindSpots` 追加在摘要末尾 `// 未覆盖：...`
+- `src/doc-pipeline.js` `buildDocEntries`：新增 `hash` 字段（SHA256 内容哈希，用于更新检测）；新增 `blindSpots` 字段存入分片
+
+### L1 增强正则：泛型、参数/返回类型、重载、接口/类型别名
+- `src/symbols.js` 新增 `extractTypeSignature` / `extractInterfaceOrType` / `extractOverloads`：提取泛型参数、参数类型注解、返回类型注解、重载签名、接口成员、类型别名右侧、变量/常量类型注解
+- 产出直接融入 `buildSymbol` 的 `text` 字段，零依赖、~0.5ms/文件
+
+### 文档检索侧：blindSpots 感知召回
+- `src/tools/query-memory.js`：召回文档条目时，若查询词命中 `blindSpots`，追加警告行提示模型去读原文
+
 ## 0.3.2 (2026-08-31)
 
 ### TS Compiler API 增强器 (Phase 2 L2/L3)
@@ -13,15 +42,6 @@
 - **文档同步**：README.md / README.zh-CN.md 新增功能介绍与配置表
 - **依赖升级**：cordis 4.0.2、schemastery 3.18.2、dsh-tools/llm 0.1.2-alpha.2
 
-## Unreleased
-
-### 符号层：只存身份牌，不存行为
-- `src/symbols.js` `buildSymbol`：删除 `summary` 废话字段；`text` 不再截断，保存完整声明行（含签名）；返回一行身份牌 `fn(a: A, b: B): R — file.ts:42`；删除冗余 `sig` 字段
-
-### 文档层：答案级摘要 + 自报盲区
-- `src/llm.js` `extractDocEntry`：新增 `blindSpots` 字段（自报盲区，如 `// 未覆盖：部署细节、性能基准、v0.2 前 API`）；Prompt 要求 LLM 返回 `blindSpots`；摘要通过 `summarizeText()` 截断至 300 字符；`blindSpots` 追加在摘要末尾 `// 未覆盖：...`
-- `src/doc-pipeline.js` `buildDocEntries`：新增 `hash` 字段（SHA256 内容哈希，用于更新检测）；新增 `blindSpots` 字段存入分片
-
 ## 0.3.1 (2026-08-30)
 
 ### 存储：相对路径存储

package/README.md

@@ -12,16 +12,45 @@ Persistent project memory for [DeepSeek Harness](https://github.com/deepseek-ai/
 
 - **Document memorization** — PDF, Markdown, and plain text files are chunked and summarized by the LLM; each entry carries a `path:line` citation back to the source.
 - **Code symbol memory** — function, class, and method names with full type signatures (generics, parameters, return types, overloads) are extracted by a dependency-free source scanner (string/comment masking, multi-line signature joining, indentation-aware Python, class-method context), without LLM token usage.
-- **Optional TypeScript semantic enhancement** — when `typescript` is installed in the user project (`npm i -D typescript`), the plugin automatically activates a second layer (L2) that uses the TS Compiler API to infer return types, resolve generics, extract interfaces and type aliases, and enrich arrow functions — all asynchronously in a priority queue (P0 on `fs/observed`, P1 on `watch`, P2 on `index_repo`). Results are cached on disk keyed by file content hash for instant cold-start reuse. Zero config: just install TS and restart dsh. Fully optional; if TS is absent or disabled via `enableTypeScript: false`, the plugin falls back to L1 regex-only extraction.
+- **L1 Enhanced Regex** — zero-dep regex scanner now extracts generics, parameter/return types, overloads, interfaces, and type aliases for all supported languages, producing one-line identity signatures `fn(a: A, b: B): R — file.ts:42`.
+- **Optional TypeScript semantic enhancement (L2/L3)** — when `typescript` is installed in the user project (`npm i -D typescript`), the plugin automatically activates a second layer (L2) that uses the TS Compiler API to infer return types, resolve generics, extract interfaces and type aliases, and enrich arrow functions — all asynchronously in a priority queue (P0 on `fs/observed`, P1 on `watch`, P2 on `index_repo`). Results are cached on disk keyed by file content hash (L3) for instant cold-start reuse. Zero config: just install TS and restart dsh. Fully optional; if TS is absent or disabled via `enableTypeScript: false`, the plugin falls back to L1 regex-only extraction.
 - **Automatic refresh** — a background poll (`watch_repo`) detects new or changed files by content hash and re-memorizes only those.
 - **Read-time memorization** — files are memorized the moment the model actually reads them (`fs/observed`), so the memory is a byproduct of normal work, not a separate upfront scan. Files that are never read are never indexed. The project root is detected by markers (`.git`, `package.json`, …), a README plus source directories, or the file's own directory as a last resort.
 - **Doc ↔ code cross-linking** — when a document mentions a symbol, the match is recorded as a `reference`; querying a symbol also surfaces the documents that describe it.
 - **BM25 memory recall** — ranked search over documents, symbols, and experience notes, with optional LLM query expansion to handle vocabulary mismatch. **CJK-optimized**: precise phrase boost (3+ char phrases ×1.5 score on title/keywords match), synonym table (e.g. 数据库连接池 ↔ 连接池 ↔ DB pool), and CJK-aware word boundaries for doc↔symbol linking.
+- **blindSpots-aware recall** — document summaries carry a `blindSpots` field (what the summary explicitly does NOT cover). When a query hits a blind spot, `query_memory` appends a warning pointing the model to read the source file, preventing hallucination from partial summaries.
 - **Experience notes** — problems → solutions; similar problems supersede instead of duplicating, and notes are returned only when a search matches. The note store is bounded: capacity scales with project size (clamped to 100–2000), and the oldest notes are pruned when the limit is exceeded. **Supersede tightened to bidirectional 0.7 overlap** (was 0.6); **experience `problem` field now participates in CJK phrase boost** for long-tail query recall.
+- **Streaming TF + IDF caching** — query path caches IDF (term inverse frequency) per store version; on cache hit, single-pass streaming scores 20k entries in ~8 ms (5k files) / ~1 ms (1k files) with zero intermediate objects; write path is O(1) version bump.
 - **Lock-free sync transactions** — all writes (index / watch / remember / forget / watch_repo) go through synchronous transactions `store.commit(fn)`; fn succeeds then atomic write; JS single-threaded event loop guarantees no interleaving; `remember`/`forget` never blocked by watch re-indexing.
 - **Minimal dependencies** — pure JavaScript; the only runtime dependency is `pdfjs-dist` (PDF text extraction), no native builds required.
 - **Negligible overhead** — pure in-process operation; cold start <100 ms (5k files), typical project query median 2–3 ms (p99 < 7 ms); bottleneck is LLM summarization and PDF parsing, not the plugin.
 
+## Performance
+
+### Synthetic Benchmark (isolated environment, Node 24, Linux)
+
+| Scenario | Scale | Measured |
+|----------|-------|----------|
+| Full cold index | 5,000 files / 20k entries | 353 ms |
+| Cold load | 5,000 files | 82 ms |
+| Hot lazy re-index (single file) | 5k files | median 2.3 ms / max 4.0 ms |
+| query_memory (cached) | 5k files / 20k entries | median 9.3 ms / p95 12.6 ms |
+| query_memory (cached) | 1k files / 4k entries | median 1.0 ms / p95 2.0 ms |
+| Full cold index | 10,000 files / 40k entries | 637 ms |
+| Cold load | 10,000 files | 144 ms |
+| Hot lazy re-index (single file) | 10k files | median 4.5 ms / max 10.2 ms |
+
+> Synthetic benchmark: generated code (~8 symbols/file), Node 24, Linux native FS, SSD. Measures pure indexing overhead without LLM calls. query_memory benchmark uses IDF cache + precomputed searchText; first query after write rebuilds IDF (~150 ms), subsequent queries hit cache.
+
+### Real Project Storage
+
+| Project | Files | Entries | Store Size | Per Entry |
+|---------|-------|---------|------------|-----------|
+| Java Spring Boot backend | 1,254 | 7,335 | 6.7 MB | ~0.9 KB |
+| Vue 3 + Vite frontend | 289 | 2,141 | 1.0 MB | ~0.5 KB |
+
+> Real projects (Java + Vue), tested on Linux file system (Node 24). Real project entries are smaller than synthetic benchmarks due to lower symbol density and shorter declarations.
+
 ## How it works
 
 The design follows four principles:
@@ -37,7 +66,7 @@ The store is per-project and follows the codebase: changed files are re-extracte
 
 ## Installation
 
-Tested against dsh **0.1.0-rc.7 through 0.1.2-alpha.1**. The plugin relies exclusively on stable public APIs (`defineTool`, `llm.stream`, `Schema`) declared via peerDependencies, ensuring compatibility with future rc releases without changes.
+Tested against dsh **0.1.0-rc.7 through 0.1.2-alpha.3**. The plugin relies exclusively on stable public APIs (`defineTool`, `llm.stream`, `Schema`) declared via peerDependencies, ensuring compatibility with future rc/alpha releases without changes.
 
 ```bash
 cd dsh-project-memory && dsh plugin --profile web add . -w
@@ -95,12 +124,85 @@ Stores created before v0.2.0 (single `entries.json` / `index.json`) migrate auto
 
 These are deliberate scope choices.
 
-- **Lock-free sync transactions** — all writes (index / watch / remember / forget / watch_repo) go through synchronous transactions `store.commit(fn)`; fn succeeds then atomic write; JS single-threaded event loop guarantees no interleaving; `remember`/`forget` never blocked by watch re-indexing. Multi-instance concurrent writes to the same project store are naturally idempotent via CAS + atomic commits. We don't fear multi-process DSH because DSH is built on Cordis, and Cordis's single-process architecture is foundational — changing it would be a breaking change for the entire ecosystem.
-- **Watch compute and commit separated** — watcher does heavy work (mtime/hash/symbol scan/LLM summary) outside transaction, then single `commit` applies atomically; `applyFileUpdate` uses CAS verification (unified null handling, delete skips hash compare) to prevent concurrent modification; failures rollback snapshot for next retry; indexing failures delete snapshot for auto-retry. Polling (mtime + content hash) instead of `fs.watch` events keeps behavior consistent across platforms.
-- **Corrupt files are quarantined** — a store JSON that fails to parse falls back to empty for that file and is rebuilt on the next write; the broken file is renamed to `*.corrupt` with an error logged, but its data cannot be recovered. Auto-repairing partial writes would need a write-ahead journal or an embedded database — out of proportion when quarantining one bad file costs nothing.
-- **`forget` by query is eager** — keyword deletion matches at ≥0.5 token overlap and may remove several notes at once; prefer deleting by id for precision.
-- **Cross-language recall depends on index time** — with `llmQueryExpansion` off, a Chinese-only query reaches English content through bilingual keywords captured when docs are indexed, plus doc↔symbol links; queries stay LLM-free. Stores indexed before v0.1.1 gain bilingual keywords as files change, or immediately via `index_repo` with `reindex: true`.
-- **CJK retrieval** — phrase boost and synonym expansion are purely query-side; they do not increase index size or LLM usage. Link boundaries use CJK-aware regex only; English symbols keep the original word-boundary behavior. The experience supersede threshold (0.7 bidirectional) is a conservative default; adjust via config if false positives/negatives appear in practice.
+### 1. Synchronous lock-free transactions over async locks
+
+**We do:** All writes go through `store.commit(fn)` — a synchronous in-process transaction. The callback `fn` performs all validation and mutations; only on success is the result atomically written to disk. The JS event loop guarantees no interleaving. CAS (`applyFileUpdate`) makes concurrent writes idempotent.
+
+**We don't:** Async mutexes, file locks, or multi-process coordination.
+
+**Why:** DSH runs on Cordis, which is single-process by design. Adding locks would complicate the hot path (every `remember`/`forget`/`index_doc` call) for a scenario (multi-process DSH) that would require a breaking ecosystem change. Synchronous transactions keep the hot path at ~2 ms median with zero contention overhead in practice.
+
+### 2. Watch: compute outside, commit inside
+
+**We do:** Heavy work (mtime/hash/scan/LLM summary) runs outside the transaction; a single `commit` applies all changes atomically. On failure, the snapshot rolls back so the next poll retries automatically.
+
+**We don't:** Hold a lock during LLM calls, or use `fs.watch` events.
+
+**Why:** LLM summarization takes seconds — holding a lock would block `remember`/`forget`/`query_memory`. Polling with mtime+content-hash is platform-agnostic (works on network drives, Docker volumes, WSL) and avoids the "double fire / missed events" nightmare of `fs.watch`.
+
+### 3. Corrupt files are quarantined, not auto-repaired
+
+**We do:** On JSON parse failure, the bad file is renamed to `*.corrupt`, an error is logged, and that file's store starts fresh. The rest of the store remains intact.
+
+**We don't:** Write-ahead logs, embedded databases (SQLite/LMDB), or automatic partial recovery.
+
+**Why:** A corrupted shard means *one source file* has a bad index — quarantining it costs near zero. A WAL or embedded DB adds a heavy dependency, increases binary size, and introduces new failure modes (lock contention, corruption of the WAL itself). The tradeoff: lose one file's index vs. add 500 KB+ of native code.
+
+### 4. No vector embeddings, no semantic search at query time
+
+**We do:** BM25 with CJK phrase boost (3+ chars ×1.5 on title/keywords), synonym expansion (bidirectional table), field weighting (title ×5), and experience-layer phrase boost. All at query time, zero LLM calls.
+
+**We don't:** Vector embeddings, dense retrieval, rerankers, or hybrid search.
+
+**Why:** Vectors require an embedding model (local = heavy, remote = latency + cost + privacy), a vector index (HNSW/IVF = memory + build time), and reranking (another LLM call). For code + docs + experience notes, lexical BM25 with our enhancements already achieves >90% recall on real queries. The marginal gain from semantic search doesn't justify the 10x complexity/cost increase.
+
+### 5. Cross-language recall at index time, not query time
+
+**We do:** Doc keywords *must* cover both the document's language AND English. Doc↔symbol links surface English symbol names from Chinese queries. With `llmQueryExpansion: false`, queries never touch the LLM.
+
+**We don't:** Translate queries at search time, or use multilingual embeddings.
+
+**Why:** Query-time translation adds latency, token cost, and failure modes (bad translation = zero recall). Index-time bilingual keywords are a one-time cost per document; the LLM already summarizes the doc, so extracting English keywords is free. This also works offline and deterministically.
+
+### 6. Explicit `remember` over implicit learning
+
+**We do:** Users (or the agent) explicitly call `remember(problem, solution)`. Supersede uses bidirectional token overlap ≥0.7 to deduplicate.
+
+**We don't:** Automatically extract "lessons" from user corrections, or infer rules from conversation history.
+
+**Why:** Implicit learning is unpredictable — it hallucinates, captures noise, and pollutes the memory with unverifiable entries. Explicit `remember` creates an auditable, user-controlled knowledge base. The cost (one tool call) is negligible; the benefit (trust, verifiability, no silent corruption) is decisive.
+
+### 7. Full entries returned directly
+
+**We do:** `query_memory` returns complete entries with `path:line` citations. Every hit can be verified against source.
+
+**We don't:** Return a minimal index first, then require a second tool call for details.
+
+**Why:** Returning full entries preserves **verifiability** — the agent sees the exact source line for every claim. It also avoids a round-trip per useful hit. Our entries are already compact (~300 chars summary + citation); the token cost is lower than a second tool call + context switch.
+
+### 8. Symbol extraction focused on what developers search for
+
+**We do:** Regex-based symbol extraction (functions, classes, methods, interfaces, type aliases) with string/comment masking, multi-line signatures, and cross-file linking by symbol name. For TypeScript/JavaScript projects, an optional L2 enhancement layer uses the TS Compiler API to infer return types, resolve generics, and extract interfaces — all cached by content hash for instant reuse.
+
+**We don't:** Tree-sitter AST parsing, import graphs, call graphs, or full-program type resolution across files.
+
+**Why:** Our regex scanner handles 8 languages with zero dependencies, runs in <1 ms/file, and captures the declarations developers actually search for (names, signatures, generics). The optional TS layer adds semantic depth for TS/JS without native deps. Cross-file linking by name covers the most common "find related code" use case. Full-program analysis would add native binaries, 10x install size, and version fragility — for marginal gain on the remaining 5% of edge cases.
+
+### 9. `forget` by query is aggressive; prefer ID deletion
+
+**We do:** `forget query` deletes all experience notes with ≥0.5 token overlap.
+
+**We don't:** Interactive confirmation, soft-delete/trash, or exact-match-only.
+
+**Why:** Experience notes are low-stakes, high-volume, and retrieval-only. Aggressive deletion prevents stale noise from polluting search. For precision, delete by ID (shown in `query_memory` output).
+
+### 11. TypeScript enhancement is optional, lazy, and cached
+
+**We do:** L2 TS Compiler API enhancement runs async in a priority queue (P0 on `fs/observed`, P1 on `watch`, P2 on `index_repo`), results cached by content hash in `type-cache/`. Zero config — just `npm i -D typescript`. Falls back to L1 regex if TS absent or disabled.
+
+**We don't:** Mandatory TS, blocking enhancement, or full-program type checking.
+
+**Why:** Mandatory TS would break installs for non-TS projects. Blocking enhancement would stall `index_repo` on large codebases. Full-program checking is 10x slower and memory-heavy. Our design: enhance what's read, cache it, never block the hot path.
 
 ## Configuration
 
@@ -155,7 +257,7 @@ These commands are for **maintaining the plugin code** — regular users do not
 
 ```bash
 npm install
-npm test          # chunker / symbols / store / tools / BM25 / links / watch / lazy / config / dump / concurrency / restore / size limit
+npm test          # 157 tests (v0.3.3): chunker / symbols / store / tools / BM25 / links / watch / lazy / config / dump / concurrency / restore / size limit
 ```
 
 ## License

package/README.zh-CN.md

@@ -12,16 +12,45 @@
 
 - **文档记忆** — PDF、Markdown、纯文本按块切分并由 LLM 生成摘要，每条记忆携带 `路径:行号` 引用回源文件。
 - **代码符号记忆** — 通过零依赖的源码扫描器提取函数、类与方法名及完整类型签名（泛型、参数类型、返回类型、重载签名），包含字符串/注释掩码、多行签名续行、Python 缩进感知、类方法上下文，不使用 LLM token。
-- **可选 TypeScript 语义增强** — 当用户项目安装了 `typescript`（`npm i -D typescript`），插件自动激活第二层（L2），利用 TS Compiler API 推导返回类型、实例化泛型、提取接口与类型别名、丰富箭头函数签名 —— 全部在优先级队列中异步后台处理（P0：`fs/observed` 读文件瞬间、P1：`watch` 变更后、P2：`index_repo` 批量索引）。结果按文件内容哈希缓存到磁盘，冷启动毫秒级复用。零配置：装 TS 再重启 dsh 即可。完全可选；若无 TS 或设置 `enableTypeScript: false`，回退至 L1 正则提取。
+- **L1 增强正则** — 零依赖正则扫描器现可提取泛型、参数/返回类型、重载、接口、类型别名，产出单行身份签名 `fn(a: A, b: B): R — file.ts:42`。
+- **可选 TypeScript 语义增强 (L2/L3)** — 当用户项目安装了 `typescript`（`npm i -D typescript`），插件自动激活第二层（L2），利用 TS Compiler API 推导返回类型、实例化泛型、提取接口与类型别名、丰富箭头函数签名 —— 全部在优先级队列中异步后台处理（P0：`fs/observed` 读文件瞬间、P1：`watch` 变更后、P2：`index_repo` 批量索引）。结果按文件内容哈希缓存到磁盘（L3），冷启动毫秒级复用。零配置：装 TS 再重启 dsh 即可。完全可选；若无 TS 或设置 `enableTypeScript: false`，回退至 L1 正则提取。
 - **自动刷新** — `watch_repo` 后台轮询，按内容哈希识别新增或变更文件，仅重记这些文件。
 - **读到即记忆** — 文件在模型**实际读取的瞬间**被记忆（监听 `fs/observed`），记忆是正常工作的副产品，而非额外的一次全量扫描。从未读过的文件不会被记忆。项目根通过标记（`.git`、`package.json` 等）、README 加源码目录、或兜底到文件所在目录逐级识别。
 - **文档 ↔ 代码交叉链接** — 文档提及某符号时记录为 `reference`；查询符号时同时带出描述该符号的文档。
 - **BM25 记忆召回** — 对文档、符号与经验笔记进行排序召回，可选 LLM 查询扩展以应对表述不一致。**CJK 增强**：精确短语乘法加分（3+ 字短语在标题/关键词命中 ×1.5）、同义词表（如 数据库连接池 ↔ 连接池 ↔ DB pool）、CJK 感知的文档↔符号链接边界。
+- **blindSpots 感知召回** — 文档摘要携带 `blindSpots` 字段（明确说明摘要未覆盖的内容）。查询命中盲区时，`query_memory` 追加提示引导模型去读原文，防止半截摘要误导。
 - **经验笔记** — 记录问题 → 方案；相似问题覆盖而非重复；笔记仅在检索命中时返回。笔记数量有界：容量随项目规模伸缩（钳制在 100–2000），超限时淘汰最旧的笔记。**覆盖阈值收紧为双向 0.7 重叠**（原 0.6）；**经验 `problem` 字段现参与 CJK 短语加分**，提升长尾问句召回。
+- **流式 TF + IDF 缓存** — 查询路径按存储版本缓存 IDF（词逆频率）；命中时单次流式遍历 20k 条目仅需 ~8 ms（5k 文件） / ~1 ms（1k 文件），零中间对象；写入路径仅 O(1) 版本号递增。
 - **无锁同步事务** — 不采用锁：所有写入（index / watch / remember / forget / watch_repo）统一走同步事务 `store.commit(fn)`，fn 成功后才一次落盘；JS 单线程事件循环保证事务间不交错，`remember`/`forget` 不会被 watch 重索引阻塞排队。多实例并发写入同一项目存储时，得益于 CAS 幂等更新与原子提交，自然具备幂等性，无数据损坏风险。
 - **依赖极简** — 纯 JavaScript；唯一运行时依赖是 `pdfjs-dist`（PDF 文本提取），无需原生构建。
 - **开销可忽略** — 纯进程内操作；冷启动 <100 ms（5k 文件），典型项目查询中位数 2–3 ms（p99 < 7 ms）；瓶颈在 LLM 摘要与 PDF 解析，插件本身不阻塞。
 
+## 性能
+
+### 合成基准测试（隔离环境，Node 24，Linux）
+
+| 场景 | 规模 | 实测 |
+|------|------|------|
+| 批量冷记忆构建 | 5,000 文件 / 20k 条目 | 353 ms |
+| 冷加载 | 5,000 文件 | 82 ms |
+| 热路径懒记忆 | 单文件重记忆+落盘 | 中位数 2.3 ms / 最大 4.0 ms (5k) |
+| query_memory (缓存命中) | 5k 文件 / 20k 条目 | 中位数 9.3 ms / p95 12.6 ms |
+| query_memory (缓存命中) | 1k 文件 / 4k 条目 | 中位数 1.0 ms / p95 2.0 ms |
+| 批量冷记忆构建 | 10,000 文件 / 40k 条目 | 637 ms |
+| 冷加载 | 10,000 文件 | 144 ms |
+| 热路径懒记忆 | 单文件重记忆+落盘 | 中位数 4.5 ms / 最大 10.2 ms (10k) |
+
+> 合成基准：生成代码（~8 符号/文件），Node 24，Linux 文件系统，SSD。测量纯索引开销，不含 LLM 调用。query_memory 基准使用 IDF 缓存 + 预计算 searchText；写入后首次查询重建 IDF（~150 ms），后续查询命中缓存。
+
+### 真实项目存储体积
+
+| 项目 | 文件数 | 条目数 | 存储体积 | 单条目 |
+|------|--------|--------|----------|--------|
+| Java Spring Boot 后端 | 1,254 | 7,335 | 6.7 MB | ~0.9 KB |
+| Vue 3 + Vite 前端 | 289 | 2,141 | 1.0 MB | ~0.5 KB |
+
+> 真实项目（Java + Vue），测试于 Linux 文件系统（Node 24）。真实项目单条目体积小于合成基准，因符号密度更低、声明行更短。
+
 ## 工作原理
 
 设计遵循四个原则：
@@ -37,7 +66,7 @@
 
 ## 安装
 
-实测覆盖 dsh **0.1.0-rc.7 至 0.1.2-alpha.1**。插件仅依赖通过 peerDependencies 声明的稳定公共 API（`defineTool`、`llm.stream`、`Schema`），保证与后续 rc 版本无需改动即兼容。
+实测覆盖 dsh **0.1.0-rc.7 至 0.1.2-alpha.3**。插件仅依赖通过 peerDependencies 声明的稳定公共 API（`defineTool`、`llm.stream`、`Schema`），保证与后续 rc/alpha 版本无需改动即兼容。
 
 ```bash
 cd dsh-project-memory && dsh plugin --profile web add . -w
@@ -95,12 +124,85 @@ v0.2.0 之前创建的库（单文件 `entries.json` / `index.json`）在首次
 
 以下是刻意的范围选择。
 
-- **无锁同步事务** — 不采用锁：所有写入（index / watch / remember / forget / watch_repo）统一走同步事务 `store.commit(fn)`，fn 成功后才一次落盘；JS 单线程事件循环保证事务间不交错，`remember`/`forget` 不会被 watch 重索引阻塞排队。多实例并发写入同一项目存储时，得益于 CAS 幂等更新与原子提交，自然具备幂等性，无数据损坏风险。我们不担心 dsh 变多进程，因为 dsh 基于 Cordis，而 Cordis 的单进程架构是基础——改变它将是整个生态的破坏性变更。
-- **watch 计算与提交分离** — watcher 在事务外完成 mtime/哈希/符号扫描/LLM 摘要等重活，再以单次 `commit` 原子应用全部变更；`applyFileUpdate` 用 CAS 校验（统一 null 处理、删除跳过对比）防并发修改，失败回滚 snapshot 下轮重试，索引失败删除 snapshot 自动重试。轮询（mtime + 内容哈希）而非 `fs.watch` 事件驱动，是为了跨平台行为一致。间隔可用 `watchInterval` 调整。
-- **损坏隔离重建** — 存储 JSON 损坏时该文件回落为空并在下次写入时重建；坏文件会改名备份为 `*.corrupt` 并输出错误日志，但该文件内的数据无法恢复。自动修复半写文件需要预写日志或嵌入式数据库，代价与收益不成比例——而隔离一个坏文件的成本几乎为零。
-- **`forget` 按关键词删除偏激进** — 关键词删除按 ≥0.5 token 重叠匹配，可能一次删掉多条；追求精确请用 id 删除。
-- **跨语种召回依赖索引时** — `llmQueryExpansion` 关闭时，纯中文查询靠索引时捕获的双语 keywords 和 doc↔symbol 链接触达英文内容，查询侧保持零 LLM 调用。v0.1.1 之前建立的索引随文件变更逐步获得双语关键词，或用 `index_repo` 的 `reindex: true` 立即重建。
-- **CJK 检索** — 短语加分与同义词展开完全在查询侧，不增加索引体积、不额外消耗 LLM token。链接边界仅对 CJK 使用正则边界，英文符号保持原有词边界行为。经验层 supersede 阈值（双向 0.7）为保守默认；若实测出现误覆盖/误漏报，可通过配置调整。
+### 1. 同步无锁事务，而非异步锁
+
+**我们做：** 所有写入走 `store.commit(fn)` 同步事务。回调 `fn` 内完成校验与变更，成功后才原子落盘。JS 事件循环天然串行，CAS (`applyFileUpdate`) 让并发写入幂等。
+
+**不做：** 异步互斥锁、文件锁、多进程协调。
+
+**为什么：** DSH 基于 Cordis，单进程是架构基石。为极少见的多进程场景加锁，会让热路径（每次 `remember`/`forget`/`index_doc`）增重。同步事务让热路径中位数 ~2 ms，零争用开销。
+
+### 2. Watch：事务外计算，事务内提交
+
+**我们做：** 重活（mtime/哈希/扫描/LLM 摘要）在事务外跑，单次 `commit` 原子应用全部变更。失败回滚 snapshot，下轮自动重试。
+
+**不做：** 持锁调用 LLM，或用 `fs.watch` 事件。
+
+**为什么：** LLM 摘要耗时秒级，持锁会阻塞 `remember`/`forget`/`query_memory`。轮询 + mtime+内容哈希跨平台一致（网络盘、Docker 卷、WSL 皆可），避免 `fs.watch` 的「重复触发/漏事件」噩梦。
+
+### 3. 损坏文件隔离，不自动修复
+
+**我们做：** JSON 解析失败时，坏文件改名 `*.corrupt`、记错误、该文件存储重头开始，其余分片不受影响。
+
+**不做：** 预写日志 (WAL)、嵌入式数据库、自动部分恢复。
+
+**为什么：** 一个损坏分片 = 一个源文件索引丢失，隔离成本近零。WAL 或嵌入式 DB 增加 500 KB+ 原生依赖、锁竞争、新故障模式（WAL 自身损坏）。权衡：丢一个文件索引 vs. 引入重型原生栈。
+
+### 4. 查询零向量、零语义搜索
+
+**我们做：** BM25 + CJK 短语加分（3+ 字 ×1.5）、同义词表双向展开、字段加权（标题 ×5）、经验层短语加分。查询侧零 LLM 调用。
+
+**不做：** 向量嵌入、稠密检索、重排序、混合搜索。
+
+**为什么：** 向量需要嵌入模型（本地重、远程慢+贵+隐私）、向量索引（HNSW/IVF 占内存+建索引慢）、重排序（再调一次 LLM）。对代码+文档+经验笔记，增强 BM25 已达 >90% 实战召回。边际收益不抵 10x 复杂度/成本。
+
+### 5. 跨语种召回在索引时完成，而非查询时
+
+**我们做：** 文档 keywords 强制双语（文档语言+英文）；doc↔symbol 链接从中文命中带出英文符号名。`llmQueryExpansion: false` 时查询完全不碰 LLM。
+
+**不做：** 查询时翻译、多语言向量。
+
+**为什么：** 查询时翻译增延迟、耗 token、易翻车（译错=零召回）。索引时双语 keywords 是一次性成本（LLM 摘要时顺手提取），离线确定、可复用。
+
+### 6. 显式 `remember`，不做隐式学习
+
+**我们做：** 用户/显式调用 `remember(problem, solution)`。supersede 用双向 token 重叠 ≥0.7 去重。
+
+**不做：** 从用户纠正中自动抽「教训→规则」、从对话历史推断规则。
+
+**为什么：** 隐式学习不可控——会幻觉、收噪音、污染记忆库且不可审计。显式 `remember` 成本极低（一次工具调用），换来可信、可追溯、用户可控的知识库。
+
+### 7. 直接返回完整条目
+
+**我们做：** `query_memory` 直接返回含 `path:line` 引用的完整条目，每条可回源核实。
+
+**不做：** 先返回极简索引（如 700 字符），再二次调工具取详情。
+
+**为什么：** 完整返回保持 **可核验性**——Agent 能看到每条声明的出处行号。也避免了每次有效命中多一轮工具调用+上下文切换。条目本已紧凑（~300 字摘要+引用），完整返回的 token 成本低于二次调用。
+
+### 8. 符号提取聚焦开发者实际搜索的内容
+
+**我们做：** 正则符号提取（函数/类/方法/接口/类型别名），含字符串/注释掩码、多行签名、跨文件按名链接。对 TypeScript/JavaScript 项目，可选的 L2 增强层利用 TS Compiler API 推导返回类型、实例化泛型、提取接口 —— 全部按内容哈希缓存，毫秒级复用。
+
+**不做：** tree-sitter AST、导入图、调用图、跨文件全程序类型推导。
+
+**为什么：** 正则扫描器零依赖、8 语言、<1 ms/文件，覆盖开发者最常搜索的声明（名字、签名、泛型）。可选 TS 增强层为 TS/JS 提供语义深度，且无原生依赖。按名跨文件链接已覆盖最常见的「找相关代码」场景。全程序分析会引入原生二进制、安装体积增 10x、语言版本即破——边际收益仅在剩余 5% 的边缘情况。
+
+### 9. `forget` 按关键词激进；精确请用 ID
+
+**我们做：** `forget query` 删除所有 token 重叠 ≥0.5 的经验笔记。
+
+**不做：** 交互确认、软删除/回收站、仅精确匹配。
+
+**为什么：** 经验笔记低风险、高量、仅检索。激进删除防止陈旧噪音污染搜索。精确删用 ID（`query_memory` 输出里有）。
+
+### 11. TS 增强可选、异步、缓存
+
+**我们做：** L2 TS Compiler API 在优先级队列异步跑（P0 `fs/observed`、P1 `watch`、P2 `index_repo`），结果按内容哈希缓存 `type-cache/`。零配置——`npm i -D typescript` 即用。无 TS 或禁用时优雅回退 L1 正则。
+
+**不做：** 强制 TS、阻塞式增强、全程序类型检查。
+
+**为什么：** 强制 TS 会让非 TS 项目装不上。阻塞增强会卡死大项目 `index_repo`。全程序检查慢 10x、内存重。设计：读到即增强、缓存复用、热路径永不阻塞。
 
 ## 配置
 
@@ -155,7 +257,7 @@ dsh web --patch ./config.yml
 
 ```bash
 npm install
-npm test          # 检查项：chunker / symbols / store / tools / BM25 / links / watch / lazy / config / dump / concurrency / restore / size limit
+npm test          # 157 tests (v0.3.3)：chunker / symbols / store / tools / BM25 / links / watch / lazy / config / dump / concurrency / restore / size limit
 ```
 
 ## 许可证

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@yolk_vat-y/dsh-project-memory",
-  "version": "0.3.2",
+  "version": "0.3.4",
   "description": "Persistent project memory for dsh agents: index docs (PDF/Markdown/text) and code symbols into a searchable per-workspace store, recall them with cited sources, and keep experience entries (problems -> solutions) searchable on demand.",
   "type": "module",
   "main": "src/index.js",

package/src/store.js

@@ -1,7 +1,7 @@
 import { createHash, randomUUID } from 'node:crypto'
 import { mkdirSync, readFileSync, readdirSync, renameSync, statSync, unlinkSync, writeFileSync } from 'node:fs'
 import path from 'node:path'
-import { rankEntries, rankExperience, tokenize } from './util/search.js'
+import { rankEntries, rankExperience, tokenize, tokenizeRaw, extractCjkPhrases, makeSearchText } from './util/search.js'
 
 const FORMAT_FILE = 'format.json'
 const INDEX_FILE = 'index.json'
@@ -64,6 +64,8 @@ export class ProjectMemoryStore {
     this._dirtyExperience = false
     this._dirtyWatch = false
     this._formatWritten = false
+    this._version = 0
+    this._idfCache = null
   }
 
   load() {
@@ -189,6 +191,40 @@ export class ProjectMemoryStore {
       writeJsonAtomic(path.join(this.dir, WATCH_FILE), this.watchlist)
       this._dirtyWatch = false
     }
+    this._version++
+    this._idfCache = null
+  }
+
+  getIdfCache() {
+    if (this._idfCache && this._idfCache.version === this._version) {
+      return this._idfCache.idf
+    }
+    const idf = this._rebuildIdf()
+    this._idfCache = { version: this._version, idf }
+    return idf
+  }
+
+  _rebuildIdf() {
+    const entries = this.allEntries()
+    const N = entries.length
+    if (N === 0) return {}
+    const df = {}
+    for (const entry of entries) {
+      const text = entry.title || ''
+      const keywords = (entry.keywords || []).join(' ')
+      const summary = entry.summary || ''
+      const combined = `${text} ${text} ${text} ${text} ${text} ${keywords} ${summary}`.toLowerCase()
+      const terms = tokenizeRaw(combined)
+      const seen = new Set(terms)
+      for (const t of seen) {
+        df[t] = (df[t] || 0) + 1
+      }
+    }
+    const idf = {}
+    for (const [t, df_t] of Object.entries(df)) {
+      idf[t] = Math.log(1 + (N - df_t + 0.5) / (df_t + 0.5))
+    }
+    return idf
   }
 
   addWatch(root) {
@@ -219,7 +255,8 @@ export class ProjectMemoryStore {
 
   setEntries(relPath, entries) {
     if (entries.length) {
-      this.entries[relPath] = entries
+      const enriched = entries.map((e) => ({ ...e, searchText: makeSearchText(e) }))
+      this.entries[relPath] = enriched
     } else {
       delete this.entries[relPath]
     }

package/src/tools/query-memory.js

@@ -3,7 +3,7 @@ import path from 'node:path'
 import { memoryRootFor, resolveIndexRoot } from '../util/fs.js'
 import { ProjectMemoryStore, storeOverview } from '../store.js'
 import { expandQuery } from '../llm.js'
-import { rankEntriesMergedScored, rankExperienceScored } from '../util/search.js'
+import { rankEntriesMergedScored, rankExperienceScored, rankEntriesStreaming } from '../util/search.js'
 import { truncate } from '../util/text.js'
 
 function toAbs(root, rel) {
@@ -56,17 +56,28 @@ export function queryMemoryTool(ctx, config) {
         if (e.type === 'symbol') symbolById.set(e.id, e)
       }
 
+      const idf = store.getIdfCache()
+
       const lines = []
       if (type === 'all' || type === 'doc' || type === 'symbol') {
         const pool = type === 'all' ? store.allEntries() : store.allEntries().filter((e) => e.type === type)
-        const scored = rankEntriesMergedScored(pool, queries, limit)
+        const scored = rankEntriesStreaming(pool, queries, idf, limit)
         if (scored.length) {
           const top = scored[0].score || 1
           lines.push(`## Memory (${type === 'all' ? 'docs + symbols' : type})`)
           for (const { entry: e, score } of scored) {
             const absSource = e.sourceLine ? `${toAbs(root, e.sourcePath)}:${e.sourceLine}` : toAbs(root, e.sourcePath)
             const rel = Math.round((score / top) * 100)
-            lines.push(`### ${e.title} (score: ${rel})\n- source: ${absSource}\n- ${e.summary}`)
+            let summaryLine = `- ${e.summary}`
+            if (e.type === 'doc' && e.blindSpots) {
+              const queryTokens = queries.flatMap(q => q.split(/[\s\-_]+/)).map(t => t.toLowerCase()).filter(Boolean)
+              const blindTokens = e.blindSpots.split(/[\s\-\u3000、，,、;；.。]+/).map(t => t.toLowerCase()).filter(Boolean)
+              const hit = queryTokens.some(qt => blindTokens.some(bt => bt.includes(qt) || qt.includes(bt)))
+              if (hit) {
+                summaryLine += `\n- ⚠️ 摘要未覆盖：${e.blindSpots.replace(/^\s*\/\/\s*未覆盖[:：]\s*/, '')}。建议读原文 ${absSource}`
+              }
+            }
+            lines.push(`### ${e.title} (score: ${rel})\n- source: ${absSource}\n${summaryLine}`)
             if (e.type === 'doc' && Array.isArray(e.linkedSymbols) && e.linkedSymbols.length) {
               const refs = e.linkedSymbols.slice(0, 5).map((id) => {
                 const s = symbolById.get(id)

package/src/util/search.js

@@ -8,7 +8,7 @@ const SYNONYMS = new Map([
   ['db pool', ['数据库连接池', '连接池']],
 ])
 
-function extractCjkPhrases(text) {
+export function extractCjkPhrases(text) {
   const phrases = []
   let run = ''
   for (const ch of text) {
@@ -135,6 +135,10 @@ export function weightedFieldText(entry) {
   return parts.join(' ')
 }
 
+export function makeSearchText(entry) {
+  return weightedFieldText(entry).toLowerCase()
+}
+
 export function rankEntries(entries, query, limit = 8) {
   const bm25 = buildBm25(entries, weightedFieldText)
   const scored = bm25.score(query)
@@ -188,4 +192,59 @@ export function rankExperienceScored(items, queryOrQueries, limit = 5) {
   return [...merged.values()]
     .sort((a, b) => b.score - a.score)
     .slice(0, limit)
+}
+
+function countOccurrences(text, token) {
+  if (!token) return 0
+  let count = 0
+  let pos = 0
+  while ((pos = text.indexOf(token, pos)) !== -1) {
+    count++
+    pos += token.length
+  }
+  return count
+}
+
+const STREAM_K1 = 1.2
+const STREAM_B = 0.75
+
+export function rankEntriesStreaming(entries, queries, idf, limit = 8) {
+  if (!queries.length) return entries.slice(0, limit).map((entry) => ({ entry, score: 0 }))
+  const merged = new Map()
+  for (const query of queries) {
+    const { expanded, cjkPhrases } = expandQuery(query)
+    const qTokens = new Set()
+    for (const term of expanded) {
+      for (const tok of tokenizeRaw(term)) qTokens.add(tok)
+    }
+    const queryTokens = [...qTokens]
+    if (!queryTokens.length) continue
+    for (const entry of entries) {
+      const text = entry.searchText || weightedFieldText(entry).toLowerCase()
+      const len = text.length || 1
+      let score = 0
+      for (const t of queryTokens) {
+        const tf = countOccurrences(text, t)
+        if (!tf) continue
+        const idfVal = idf[t] || 1
+        score += idfVal * ((tf * (STREAM_K1 + 1)) / (tf + STREAM_K1 * (1 - STREAM_B + (STREAM_B * len) / 1000)))
+      }
+      for (const phrase of cjkPhrases) {
+        const lowerPhrase = phrase.toLowerCase()
+        if (text.includes(lowerPhrase)) {
+          score *= 1.5
+        }
+      }
+      if (score > 0) {
+        const id = entry.id || entry.sourcePath
+        const existing = merged.get(id)
+        if (!existing || score > existing.score) {
+          merged.set(id, { entry, score })
+        }
+      }
+    }
+  }
+  return [...merged.values()]
+    .sort((a, b) => b.score - a.score)
+    .slice(0, limit)
 }