@yolk_vat-y/dsh-project-memory 0.3.3 → 0.3.4

package/CHANGELOG.md

@@ -1,5 +1,18 @@
 # Changelog
 
+## 0.3.4 (2026-09-02)
+
+### query_memory 性能优化：流式 TF + IDF 缓存
+- **IDF 缓存**：`store._version` + `store._idfCache`，`save()` 时 `version++` 标记失效，查询时版本命中直接复用，无需重建 BM25 索引
+- **预计算 searchText**：`setEntries` 时预计算 `entry.searchText`（标题×5 + keywords + summary + path 的小写拼接），查询时直接复用，避免重复字符串拼接与 `toLowerCase()`
+- **流式打分**：`rankEntriesStreaming` 单次遍历 entries，用 `countOccurrences()` 字符串计数替代完整 `tokenizeRaw` + TF 表构建，零中间对象分配
+- **性能提升**：5k 文件 / 20k 条目场景 `query_memory` 中位数 **187 ms → 9.3 ms**（20x）；1k 文件典型项目 **<1 ms**
+
+### 测试覆盖
+- 新增 `IDF caching & streaming TF` 测试组（7 项）：缓存构建、命中、版本失效、流式打分正确性、空查询、searchText 预计算
+- 新增 `query_memory streaming path` 集成测试（2 项）：端到端首次查询建缓存、二次查询复用缓存
+- 总测试数 157 → 166 全绿
+
 ## 0.3.3 (2026-08-31)
 
 ### 符号层：只存身份牌，不存行为

package/README.md

@@ -12,16 +12,45 @@ Persistent project memory for [DeepSeek Harness](https://github.com/deepseek-ai/
 
 - **Document memorization** — PDF, Markdown, and plain text files are chunked and summarized by the LLM; each entry carries a `path:line` citation back to the source.
 - **Code symbol memory** — function, class, and method names with full type signatures (generics, parameters, return types, overloads) are extracted by a dependency-free source scanner (string/comment masking, multi-line signature joining, indentation-aware Python, class-method context), without LLM token usage.
-- **Optional TypeScript semantic enhancement** — when `typescript` is installed in the user project (`npm i -D typescript`), the plugin automatically activates a second layer (L2) that uses the TS Compiler API to infer return types, resolve generics, extract interfaces and type aliases, and enrich arrow functions — all asynchronously in a priority queue (P0 on `fs/observed`, P1 on `watch`, P2 on `index_repo`). Results are cached on disk keyed by file content hash for instant cold-start reuse. Zero config: just install TS and restart dsh. Fully optional; if TS is absent or disabled via `enableTypeScript: false`, the plugin falls back to L1 regex-only extraction.
+- **L1 Enhanced Regex** — zero-dep regex scanner now extracts generics, parameter/return types, overloads, interfaces, and type aliases for all supported languages, producing one-line identity signatures `fn(a: A, b: B): R — file.ts:42`.
+- **Optional TypeScript semantic enhancement (L2/L3)** — when `typescript` is installed in the user project (`npm i -D typescript`), the plugin automatically activates a second layer (L2) that uses the TS Compiler API to infer return types, resolve generics, extract interfaces and type aliases, and enrich arrow functions — all asynchronously in a priority queue (P0 on `fs/observed`, P1 on `watch`, P2 on `index_repo`). Results are cached on disk keyed by file content hash (L3) for instant cold-start reuse. Zero config: just install TS and restart dsh. Fully optional; if TS is absent or disabled via `enableTypeScript: false`, the plugin falls back to L1 regex-only extraction.
 - **Automatic refresh** — a background poll (`watch_repo`) detects new or changed files by content hash and re-memorizes only those.
 - **Read-time memorization** — files are memorized the moment the model actually reads them (`fs/observed`), so the memory is a byproduct of normal work, not a separate upfront scan. Files that are never read are never indexed. The project root is detected by markers (`.git`, `package.json`, …), a README plus source directories, or the file's own directory as a last resort.
 - **Doc ↔ code cross-linking** — when a document mentions a symbol, the match is recorded as a `reference`; querying a symbol also surfaces the documents that describe it.
 - **BM25 memory recall** — ranked search over documents, symbols, and experience notes, with optional LLM query expansion to handle vocabulary mismatch. **CJK-optimized**: precise phrase boost (3+ char phrases ×1.5 score on title/keywords match), synonym table (e.g. 数据库连接池 ↔ 连接池 ↔ DB pool), and CJK-aware word boundaries for doc↔symbol linking.
+- **blindSpots-aware recall** — document summaries carry a `blindSpots` field (what the summary explicitly does NOT cover). When a query hits a blind spot, `query_memory` appends a warning pointing the model to read the source file, preventing hallucination from partial summaries.
 - **Experience notes** — problems → solutions; similar problems supersede instead of duplicating, and notes are returned only when a search matches. The note store is bounded: capacity scales with project size (clamped to 100–2000), and the oldest notes are pruned when the limit is exceeded. **Supersede tightened to bidirectional 0.7 overlap** (was 0.6); **experience `problem` field now participates in CJK phrase boost** for long-tail query recall.
+- **Streaming TF + IDF caching** — query path caches IDF (term inverse frequency) per store version; on cache hit, single-pass streaming scores 20k entries in ~8 ms (5k files) / ~1 ms (1k files) with zero intermediate objects; write path is O(1) version bump.
 - **Lock-free sync transactions** — all writes (index / watch / remember / forget / watch_repo) go through synchronous transactions `store.commit(fn)`; fn succeeds then atomic write; JS single-threaded event loop guarantees no interleaving; `remember`/`forget` never blocked by watch re-indexing.
 - **Minimal dependencies** — pure JavaScript; the only runtime dependency is `pdfjs-dist` (PDF text extraction), no native builds required.
 - **Negligible overhead** — pure in-process operation; cold start <100 ms (5k files), typical project query median 2–3 ms (p99 < 7 ms); bottleneck is LLM summarization and PDF parsing, not the plugin.
 
+## Performance
+
+### Synthetic Benchmark (isolated environment, Node 24, Linux)
+
+| Scenario | Scale | Measured |
+|----------|-------|----------|
+| Full cold index | 5,000 files / 20k entries | 353 ms |
+| Cold load | 5,000 files | 82 ms |
+| Hot lazy re-index (single file) | 5k files | median 2.3 ms / max 4.0 ms |
+| query_memory (cached) | 5k files / 20k entries | median 9.3 ms / p95 12.6 ms |
+| query_memory (cached) | 1k files / 4k entries | median 1.0 ms / p95 2.0 ms |
+| Full cold index | 10,000 files / 40k entries | 637 ms |
+| Cold load | 10,000 files | 144 ms |
+| Hot lazy re-index (single file) | 10k files | median 4.5 ms / max 10.2 ms |
+
+> Synthetic benchmark: generated code (~8 symbols/file), Node 24, Linux native FS, SSD. Measures pure indexing overhead without LLM calls. query_memory benchmark uses IDF cache + precomputed searchText; first query after write rebuilds IDF (~150 ms), subsequent queries hit cache.
+
+### Real Project Storage
+
+| Project | Files | Entries | Store Size | Per Entry |
+|---------|-------|---------|------------|-----------|
+| Java Spring Boot backend | 1,254 | 7,335 | 6.7 MB | ~0.9 KB |
+| Vue 3 + Vite frontend | 289 | 2,141 | 1.0 MB | ~0.5 KB |
+
+> Real projects (Java + Vue), tested on Linux file system (Node 24). Real project entries are smaller than synthetic benchmarks due to lower symbol density and shorter declarations.
+
 ## How it works
 
 The design follows four principles:
@@ -37,7 +66,7 @@ The store is per-project and follows the codebase: changed files are re-extracte
 
 ## Installation
 
-Tested against dsh **0.1.0-rc.7 through 0.1.2-alpha.2**. The plugin relies exclusively on stable public APIs (`defineTool`, `llm.stream`, `Schema`) declared via peerDependencies, ensuring compatibility with future rc releases without changes.
+Tested against dsh **0.1.0-rc.7 through 0.1.2-alpha.3**. The plugin relies exclusively on stable public APIs (`defineTool`, `llm.stream`, `Schema`) declared via peerDependencies, ensuring compatibility with future rc/alpha releases without changes.
 
 ```bash
 cd dsh-project-memory && dsh plugin --profile web add . -w
@@ -228,7 +257,7 @@ These commands are for **maintaining the plugin code** — regular users do not
 
 ```bash
 npm install
-npm test          # 157 tests (v0.3.2): chunker / symbols / store / tools / BM25 / links / watch / lazy / config / dump / concurrency / restore / size limit
+npm test          # 157 tests (v0.3.3): chunker / symbols / store / tools / BM25 / links / watch / lazy / config / dump / concurrency / restore / size limit
 ```
 
 ## License

package/README.zh-CN.md

@@ -12,16 +12,45 @@
 
 - **文档记忆** — PDF、Markdown、纯文本按块切分并由 LLM 生成摘要，每条记忆携带 `路径:行号` 引用回源文件。
 - **代码符号记忆** — 通过零依赖的源码扫描器提取函数、类与方法名及完整类型签名（泛型、参数类型、返回类型、重载签名），包含字符串/注释掩码、多行签名续行、Python 缩进感知、类方法上下文，不使用 LLM token。
-- **可选 TypeScript 语义增强** — 当用户项目安装了 `typescript`（`npm i -D typescript`），插件自动激活第二层（L2），利用 TS Compiler API 推导返回类型、实例化泛型、提取接口与类型别名、丰富箭头函数签名 —— 全部在优先级队列中异步后台处理（P0：`fs/observed` 读文件瞬间、P1：`watch` 变更后、P2：`index_repo` 批量索引）。结果按文件内容哈希缓存到磁盘，冷启动毫秒级复用。零配置：装 TS 再重启 dsh 即可。完全可选；若无 TS 或设置 `enableTypeScript: false`，回退至 L1 正则提取。
+- **L1 增强正则** — 零依赖正则扫描器现可提取泛型、参数/返回类型、重载、接口、类型别名，产出单行身份签名 `fn(a: A, b: B): R — file.ts:42`。
+- **可选 TypeScript 语义增强 (L2/L3)** — 当用户项目安装了 `typescript`（`npm i -D typescript`），插件自动激活第二层（L2），利用 TS Compiler API 推导返回类型、实例化泛型、提取接口与类型别名、丰富箭头函数签名 —— 全部在优先级队列中异步后台处理（P0：`fs/observed` 读文件瞬间、P1：`watch` 变更后、P2：`index_repo` 批量索引）。结果按文件内容哈希缓存到磁盘（L3），冷启动毫秒级复用。零配置：装 TS 再重启 dsh 即可。完全可选；若无 TS 或设置 `enableTypeScript: false`，回退至 L1 正则提取。
 - **自动刷新** — `watch_repo` 后台轮询，按内容哈希识别新增或变更文件，仅重记这些文件。
 - **读到即记忆** — 文件在模型**实际读取的瞬间**被记忆（监听 `fs/observed`），记忆是正常工作的副产品，而非额外的一次全量扫描。从未读过的文件不会被记忆。项目根通过标记（`.git`、`package.json` 等）、README 加源码目录、或兜底到文件所在目录逐级识别。
 - **文档 ↔ 代码交叉链接** — 文档提及某符号时记录为 `reference`；查询符号时同时带出描述该符号的文档。
 - **BM25 记忆召回** — 对文档、符号与经验笔记进行排序召回，可选 LLM 查询扩展以应对表述不一致。**CJK 增强**：精确短语乘法加分（3+ 字短语在标题/关键词命中 ×1.5）、同义词表（如 数据库连接池 ↔ 连接池 ↔ DB pool）、CJK 感知的文档↔符号链接边界。
+- **blindSpots 感知召回** — 文档摘要携带 `blindSpots` 字段（明确说明摘要未覆盖的内容）。查询命中盲区时，`query_memory` 追加提示引导模型去读原文，防止半截摘要误导。
 - **经验笔记** — 记录问题 → 方案；相似问题覆盖而非重复；笔记仅在检索命中时返回。笔记数量有界：容量随项目规模伸缩（钳制在 100–2000），超限时淘汰最旧的笔记。**覆盖阈值收紧为双向 0.7 重叠**（原 0.6）；**经验 `problem` 字段现参与 CJK 短语加分**，提升长尾问句召回。
+- **流式 TF + IDF 缓存** — 查询路径按存储版本缓存 IDF（词逆频率）；命中时单次流式遍历 20k 条目仅需 ~8 ms（5k 文件） / ~1 ms（1k 文件），零中间对象；写入路径仅 O(1) 版本号递增。
 - **无锁同步事务** — 不采用锁：所有写入（index / watch / remember / forget / watch_repo）统一走同步事务 `store.commit(fn)`，fn 成功后才一次落盘；JS 单线程事件循环保证事务间不交错，`remember`/`forget` 不会被 watch 重索引阻塞排队。多实例并发写入同一项目存储时，得益于 CAS 幂等更新与原子提交，自然具备幂等性，无数据损坏风险。
 - **依赖极简** — 纯 JavaScript；唯一运行时依赖是 `pdfjs-dist`（PDF 文本提取），无需原生构建。
 - **开销可忽略** — 纯进程内操作；冷启动 <100 ms（5k 文件），典型项目查询中位数 2–3 ms（p99 < 7 ms）；瓶颈在 LLM 摘要与 PDF 解析，插件本身不阻塞。
 
+## 性能
+
+### 合成基准测试（隔离环境，Node 24，Linux）
+
+| 场景 | 规模 | 实测 |
+|------|------|------|
+| 批量冷记忆构建 | 5,000 文件 / 20k 条目 | 353 ms |
+| 冷加载 | 5,000 文件 | 82 ms |
+| 热路径懒记忆 | 单文件重记忆+落盘 | 中位数 2.3 ms / 最大 4.0 ms (5k) |
+| query_memory (缓存命中) | 5k 文件 / 20k 条目 | 中位数 9.3 ms / p95 12.6 ms |
+| query_memory (缓存命中) | 1k 文件 / 4k 条目 | 中位数 1.0 ms / p95 2.0 ms |
+| 批量冷记忆构建 | 10,000 文件 / 40k 条目 | 637 ms |
+| 冷加载 | 10,000 文件 | 144 ms |
+| 热路径懒记忆 | 单文件重记忆+落盘 | 中位数 4.5 ms / 最大 10.2 ms (10k) |
+
+> 合成基准：生成代码（~8 符号/文件），Node 24，Linux 文件系统，SSD。测量纯索引开销，不含 LLM 调用。query_memory 基准使用 IDF 缓存 + 预计算 searchText；写入后首次查询重建 IDF（~150 ms），后续查询命中缓存。
+
+### 真实项目存储体积
+
+| 项目 | 文件数 | 条目数 | 存储体积 | 单条目 |
+|------|--------|--------|----------|--------|
+| Java Spring Boot 后端 | 1,254 | 7,335 | 6.7 MB | ~0.9 KB |
+| Vue 3 + Vite 前端 | 289 | 2,141 | 1.0 MB | ~0.5 KB |
+
+> 真实项目（Java + Vue），测试于 Linux 文件系统（Node 24）。真实项目单条目体积小于合成基准，因符号密度更低、声明行更短。
+
 ## 工作原理
 
 设计遵循四个原则：
@@ -37,7 +66,7 @@
 
 ## 安装
 
-实测覆盖 dsh **0.1.0-rc.7 至 0.1.2-alpha.2**。插件仅依赖通过 peerDependencies 声明的稳定公共 API（`defineTool`、`llm.stream`、`Schema`），保证与后续 rc 版本无需改动即兼容。
+实测覆盖 dsh **0.1.0-rc.7 至 0.1.2-alpha.3**。插件仅依赖通过 peerDependencies 声明的稳定公共 API（`defineTool`、`llm.stream`、`Schema`），保证与后续 rc/alpha 版本无需改动即兼容。
 
 ```bash
 cd dsh-project-memory && dsh plugin --profile web add . -w
@@ -228,7 +257,7 @@ dsh web --patch ./config.yml
 
 ```bash
 npm install
-npm test          # 157 tests (v0.3.2)：chunker / symbols / store / tools / BM25 / links / watch / lazy / config / dump / concurrency / restore / size limit
+npm test          # 157 tests (v0.3.3)：chunker / symbols / store / tools / BM25 / links / watch / lazy / config / dump / concurrency / restore / size limit
 ```
 
 ## 许可证

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@yolk_vat-y/dsh-project-memory",
-  "version": "0.3.3",
+  "version": "0.3.4",
   "description": "Persistent project memory for dsh agents: index docs (PDF/Markdown/text) and code symbols into a searchable per-workspace store, recall them with cited sources, and keep experience entries (problems -> solutions) searchable on demand.",
   "type": "module",
   "main": "src/index.js",

package/src/store.js

@@ -1,7 +1,7 @@
 import { createHash, randomUUID } from 'node:crypto'
 import { mkdirSync, readFileSync, readdirSync, renameSync, statSync, unlinkSync, writeFileSync } from 'node:fs'
 import path from 'node:path'
-import { rankEntries, rankExperience, tokenize } from './util/search.js'
+import { rankEntries, rankExperience, tokenize, tokenizeRaw, extractCjkPhrases, makeSearchText } from './util/search.js'
 
 const FORMAT_FILE = 'format.json'
 const INDEX_FILE = 'index.json'
@@ -64,6 +64,8 @@ export class ProjectMemoryStore {
     this._dirtyExperience = false
     this._dirtyWatch = false
     this._formatWritten = false
+    this._version = 0
+    this._idfCache = null
   }
 
   load() {
@@ -189,6 +191,40 @@ export class ProjectMemoryStore {
       writeJsonAtomic(path.join(this.dir, WATCH_FILE), this.watchlist)
       this._dirtyWatch = false
     }
+    this._version++
+    this._idfCache = null
+  }
+
+  getIdfCache() {
+    if (this._idfCache && this._idfCache.version === this._version) {
+      return this._idfCache.idf
+    }
+    const idf = this._rebuildIdf()
+    this._idfCache = { version: this._version, idf }
+    return idf
+  }
+
+  _rebuildIdf() {
+    const entries = this.allEntries()
+    const N = entries.length
+    if (N === 0) return {}
+    const df = {}
+    for (const entry of entries) {
+      const text = entry.title || ''
+      const keywords = (entry.keywords || []).join(' ')
+      const summary = entry.summary || ''
+      const combined = `${text} ${text} ${text} ${text} ${text} ${keywords} ${summary}`.toLowerCase()
+      const terms = tokenizeRaw(combined)
+      const seen = new Set(terms)
+      for (const t of seen) {
+        df[t] = (df[t] || 0) + 1
+      }
+    }
+    const idf = {}
+    for (const [t, df_t] of Object.entries(df)) {
+      idf[t] = Math.log(1 + (N - df_t + 0.5) / (df_t + 0.5))
+    }
+    return idf
   }
 
   addWatch(root) {
@@ -219,7 +255,8 @@ export class ProjectMemoryStore {
 
   setEntries(relPath, entries) {
     if (entries.length) {
-      this.entries[relPath] = entries
+      const enriched = entries.map((e) => ({ ...e, searchText: makeSearchText(e) }))
+      this.entries[relPath] = enriched
     } else {
       delete this.entries[relPath]
     }

package/src/tools/query-memory.js

@@ -3,7 +3,7 @@ import path from 'node:path'
 import { memoryRootFor, resolveIndexRoot } from '../util/fs.js'
 import { ProjectMemoryStore, storeOverview } from '../store.js'
 import { expandQuery } from '../llm.js'
-import { rankEntriesMergedScored, rankExperienceScored } from '../util/search.js'
+import { rankEntriesMergedScored, rankExperienceScored, rankEntriesStreaming } from '../util/search.js'
 import { truncate } from '../util/text.js'
 
 function toAbs(root, rel) {
@@ -56,10 +56,12 @@ export function queryMemoryTool(ctx, config) {
         if (e.type === 'symbol') symbolById.set(e.id, e)
       }
 
+      const idf = store.getIdfCache()
+
       const lines = []
       if (type === 'all' || type === 'doc' || type === 'symbol') {
         const pool = type === 'all' ? store.allEntries() : store.allEntries().filter((e) => e.type === type)
-        const scored = rankEntriesMergedScored(pool, queries, limit)
+        const scored = rankEntriesStreaming(pool, queries, idf, limit)
         if (scored.length) {
           const top = scored[0].score || 1
           lines.push(`## Memory (${type === 'all' ? 'docs + symbols' : type})`)

package/src/util/search.js

@@ -8,7 +8,7 @@ const SYNONYMS = new Map([
   ['db pool', ['数据库连接池', '连接池']],
 ])
 
-function extractCjkPhrases(text) {
+export function extractCjkPhrases(text) {
   const phrases = []
   let run = ''
   for (const ch of text) {
@@ -135,6 +135,10 @@ export function weightedFieldText(entry) {
   return parts.join(' ')
 }
 
+export function makeSearchText(entry) {
+  return weightedFieldText(entry).toLowerCase()
+}
+
 export function rankEntries(entries, query, limit = 8) {
   const bm25 = buildBm25(entries, weightedFieldText)
   const scored = bm25.score(query)
@@ -188,4 +192,59 @@ export function rankExperienceScored(items, queryOrQueries, limit = 5) {
   return [...merged.values()]
     .sort((a, b) => b.score - a.score)
     .slice(0, limit)
+}
+
+function countOccurrences(text, token) {
+  if (!token) return 0
+  let count = 0
+  let pos = 0
+  while ((pos = text.indexOf(token, pos)) !== -1) {
+    count++
+    pos += token.length
+  }
+  return count
+}
+
+const STREAM_K1 = 1.2
+const STREAM_B = 0.75
+
+export function rankEntriesStreaming(entries, queries, idf, limit = 8) {
+  if (!queries.length) return entries.slice(0, limit).map((entry) => ({ entry, score: 0 }))
+  const merged = new Map()
+  for (const query of queries) {
+    const { expanded, cjkPhrases } = expandQuery(query)
+    const qTokens = new Set()
+    for (const term of expanded) {
+      for (const tok of tokenizeRaw(term)) qTokens.add(tok)
+    }
+    const queryTokens = [...qTokens]
+    if (!queryTokens.length) continue
+    for (const entry of entries) {
+      const text = entry.searchText || weightedFieldText(entry).toLowerCase()
+      const len = text.length || 1
+      let score = 0
+      for (const t of queryTokens) {
+        const tf = countOccurrences(text, t)
+        if (!tf) continue
+        const idfVal = idf[t] || 1
+        score += idfVal * ((tf * (STREAM_K1 + 1)) / (tf + STREAM_K1 * (1 - STREAM_B + (STREAM_B * len) / 1000)))
+      }
+      for (const phrase of cjkPhrases) {
+        const lowerPhrase = phrase.toLowerCase()
+        if (text.includes(lowerPhrase)) {
+          score *= 1.5
+        }
+      }
+      if (score > 0) {
+        const id = entry.id || entry.sourcePath
+        const existing = merged.get(id)
+        if (!existing || score > existing.score) {
+          merged.set(id, { entry, score })
+        }
+      }
+    }
+  }
+  return [...merged.values()]
+    .sort((a, b) => b.score - a.score)
+    .slice(0, limit)
 }