@yolk_vat-y/dsh-project-memory 0.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md ADDED
@@ -0,0 +1,57 @@
1
+ # Changelog
2
+
3
+ ## Unreleased
4
+
5
+ ### 新增
6
+ - 发布 npm 包 `@yolk_vat-y/dsh-project-memory`,支持 `dsh plugin --profile web add @yolk_vat-y/dsh-project-memory -w` 直接安装;`publishConfig.access` 设为 `public`(scoped 包默认 restricted,不设置会导致他人安装 403)
7
+
8
+ ## 0.1.3 (2026-08-23)
9
+
10
+ ### 修复
11
+ - 项目根定位重写:`.git` / `.hg` / `.svn` 作为强边界向上无限爬(不再受 8 层限制)——深层路径(Java 式 9 层以上)不再静默把存储落在中间目录,monorepo 里就近的 package.json 不再把库从仓库根拆散;无版本控制时回退到弱标记 + 目录启发式,启发式命中按就近优先,且系统临时目录及其之上不参与定位
12
+ - DEFAULT_IGNORE 增加 `vendor` / `third_party` / `thirdparty` / `obj`,Go/C++/C# 项目的依赖与构建目录不再被索引进记忆
13
+ - Windows / macOS 大小写不敏感文件系统上,模型以不同大小写路径读取同一文件会生成双键索引,并被 watch 轮询反复"清理→重建"(每轮重复消耗 LLM token);存储键统一按平台规范化
14
+ - `index_doc` 未显式传 `root` 时默认定位到项目根存储(与读到即索引一致);此前落在文档所在目录,造成存储碎片和同一文档的双份索引费用
15
+ - chunker 对 `chunkChars ≤ 0` 的配置值不再死循环(此前会同步阻塞整个 dsh 事件循环);非法 `maxChunks` 同样回落默认值
16
+ - watch 轮询间隔下限钳制到 1 秒(此前 `watchInterval: 0` 会以毫秒级频率全仓库扫描)
17
+ - `watch_repo` 工具的存储写入补上互斥锁——它是唯一绕过锁的写入方,与 lazy/watch 并发时可能丢失更新
18
+ - doc↔symbol 链接只按符号名匹配:此前泛化词(function/class 等)混进链接索引,文档摘要里出现"function"一词就会链接到项目全部函数符号;链接计数同步修正为按唯一对统计
19
+ - `maxFileSizeMb: 0` 对文本文档表示不限制(此前会变成 0 字节上限,拒绝所有非空文件)
20
+ - index_repo 对已变更文件不再计算两次 SHA-256
21
+
22
+ ### 清理
23
+ - 移除无生产调用方的 `store.searchExperience` / `scoreExperience` 与 `isCjkText`
24
+ - 原子写失败时自动删除本次临时文件;进程崩溃的残留 `.tmp` 由下次保存兜底清理(超 60 秒)
25
+
26
+ ### 文档
27
+ - 实测 dsh 版本覆盖更新至 0.1.1-rc.1
28
+
29
+ ## 0.1.2 (2026-08-21)
30
+
31
+ ### 修复
32
+ - BM25 词频信号失效:tokenize 去重导致 tf 恒为 1、标题 ×5 加权被抵消;检索层改用不去重分词,排序恢复词频与字段权重信号(supersede/forget 的重叠判定不受影响)
33
+ - watch 轮询索引失败后回滚 mtime 快照,下一轮自动重试;此前失败文件会被一直跳过直到再次修改
34
+ - Rust `pub fn` / `pub(crate) async fn` 公开函数纳入符号表
35
+ - lazy 索引对读取瞬间被删除的文件静默跳过,不再刷错误日志
36
+ - PDF 补上字节大小上限(此前只限页数),index_repo / watch / lazy 三条路径统一生效
37
+ - 修正 peerDependencies:cordis 实际为 ^4.0.1、schemastery 实际为 ^3.18.1(此前按 dsh 的 0.x rc 线声明,匹配不到任何已发布版本);dsh-tools/dsh-llm 范围补充 0.1.1-rc 线
38
+
39
+ ### 性能
40
+ - lazy 队列 code 文件优先处理,不被大文档的 LLM 摘要阻塞
41
+ - 文档分块摘要改为 4 并发池(保持块顺序),墙钟时间约降至 1/4,token 成本不变
42
+ - 存储写盘去掉缩进,体积约减半
43
+ - 扩展名过滤前置到项目根探测之前,无关文件不再白扫约 80 次 stat
44
+
45
+ ### 行为变更
46
+ - CJK 查询不再绕过 `llmQueryExpansion` 开关:关闭时查询严格零 LLM 调用
47
+ - 跨语种召回改由索引时承担:文档摘要的 keywords 现要求同时覆盖文档语言与英文;旧索引随文件变更逐步获得双语关键词,或用 `index_repo reindex: true` 立即重建
48
+
49
+ ### 文档
50
+ - README 新增"已知限制"一节(进程内锁、watch 持锁、损坏静默重建、绝对路径引用、forget 聚合删除、跨语种召回依赖索引时关键词)
51
+
52
+ ## 0.1.0 (2026-08-20)
53
+
54
+ - 初始版本:文档(PDF/Markdown/txt)与代码符号的持久化项目记忆
55
+ - 读取时索引(lazy indexing)、增量刷新、watch 后台保鲜
56
+ - BM25 检索(含 CJK 查询扩展)与经验笔记(remember/forget)
57
+ - dump 反射转储自动过滤、并发写串行化
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 dsh-project-memory contributors
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,151 @@
1
+ # dsh-project-memory
2
+
3
+ [English](README.md) | [简体中文](README.zh-CN.md)
4
+
5
+ [![ci](https://github.com/00080000/dsh-project-memory/actions/workflows/ci.yml/badge.svg)](https://github.com/00080000/dsh-project-memory/actions/workflows/ci.yml) [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) [![Listed on dsh-plugin.org](https://dsh-plugin.org/badges/listed.svg)](https://dsh-plugin.org/plugins/00080000/dsh-project-memory)
6
+
7
+ Persistent project memory for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness)(dsh) agents. Indexes documents (PDF / Markdown / txt) and code symbols into a per-workspace store, refreshes them automatically, and recalls them with source citations — documents are cross-linked to the code symbols they reference.
8
+
9
+ > The plugin keeps a compact project index on disk, with every entry pointing to a concrete file and line — the agent can reorient quickly instead of re-reading the whole project.
10
+
11
+ ## Features
12
+
13
+ - **Document indexing** — PDF, Markdown, and plain text files are chunked and summarized by the LLM; each entry carries a `path:line` citation back to the source.
14
+ - **Code symbol table** — function and class names are extracted with a lightweight regex scanner, without LLM token usage.
15
+ - **Automatic refresh** — a background poll (`watch_repo`) detects new or changed files by content hash and re-indexes only those.
16
+ - **Read-time indexing** — files are indexed the moment the model actually reads them (`fs/observed`), so the index is a byproduct of normal work, not a separate upfront scan. Files that are never read are never indexed. The project root is detected by markers (`.git`, `package.json`, …), a README plus source directories, or the file's own directory as a last resort.
17
+ - **Doc ↔ code cross-linking** — when a document mentions a symbol, the match is recorded as a `reference`; querying a symbol also surfaces the documents that describe it.
18
+ - **BM25 retrieval** — ranked search over documents, symbols, and experience notes, with optional LLM query expansion to handle vocabulary mismatch.
19
+ - **Experience notes** — problems → solutions; similar problems supersede instead of duplicating, and notes are returned only when a search matches. The note store is bounded: capacity scales with project size (clamped to 100–2000), and the oldest notes are pruned when the limit is exceeded.
20
+ - **Minimal dependencies** — pure JavaScript; the only runtime dependency is `pdfjs-dist` (PDF text extraction), no native builds required.
21
+
22
+ ## How it works
23
+
24
+ The design follows four principles:
25
+
26
+ - **Volatility** — context is ephemeral; it is lost when a session is compacted.
27
+ - **Persistence** — the index is stored on disk and survives compaction and new sessions.
28
+ - **Compactness** — only summaries are stored; the index runs around 0.5% the size of the source it covers (8.8 MB of source → 49 KB of index in the example project), so retrieval replaces re-reading the full file.
29
+ - **Verifiability** — hits carry a `path:line` citation where applicable, so the agent can confirm details against the source.
30
+
31
+ Building the index does not require an upfront scan: files are indexed as the model reads them, so the index grows to cover exactly what has been worked with. Re-reading a file that has not changed is a no-op (content hash), so the index stays fresh with minimal ongoing overhead.
32
+
33
+ The store is per-project and follows the codebase: changed files are re-extracted by content hash, deleted files are removed. Experience notes are retrieval-only, so accumulation does not affect context.
34
+
35
+ ## Installation
36
+
37
+ Tested against dsh **0.1.0-rc.7 through 0.1.1-rc.2**. Newer rc lines are expected to work — the plugin only touches stable surfaces (`defineTool`, `llm.stream`, `Schema`) — but they are not individually verified. The host provides `@deepseek-ai/cordis` ^4.0.1 and `@deepseek-ai/schemastery` ^3.18.1, declared via peerDependencies.
38
+
39
+ ```bash
40
+ cd dsh-project-memory
41
+ dsh plugin --profile web add . -w
42
+ ```
43
+
44
+ The `-w` (workspace-root) flag is required: the profile directory is a pnpm workspace root, and pnpm rejects `add` there without it. From any other directory, the path form works the same: `dsh plugin --profile web add /path/to/dsh-project-memory -w`.
45
+
46
+ The plugin is also published on npm as a scoped package:
47
+
48
+ ```bash
49
+ dsh plugin --profile web add @yolk_vat-y/dsh-project-memory -w
50
+ ```
51
+
52
+ A prebuilt tarball is published with each release, installable without a build step:
53
+
54
+ ```bash
55
+ dsh plugin --profile web add /path/to/dsh-project-memory-0.1.3.tgz
56
+ ```
57
+
58
+ Each indexed project has its own store at `<root>/.dsh-project-memory/`. Add it to `.gitignore` if it should not be committed.
59
+
60
+ ## Usage
61
+
62
+ The tools below are **invoked by the agent**, not typed by the user. In the chat, just ask naturally — e.g. "index this project" or "what does the auth module do?" — or simply keep working, and the agent calls the matching tool automatically. By default (`lazyIndexing`) files are indexed the moment the model reads them, so memory fills in while you work. `watch_repo` keeps explicitly-watched roots fresh in the background; `index_repo` forces a full backfill of a project (unchanged files are skipped).
63
+
64
+ | Tool | Purpose |
65
+ |---|---|
66
+ | `index_doc file_path` | Index one document (PDF/MD/txt): chunk → LLM summary → store with `path:line`. Unchanged files are skipped. |
67
+ | `index_repo root` | Index a whole project: docs get LLM summaries, code files get a zero-token symbol table. Incremental, cleans up deleted files, cross-links docs to symbols. |
68
+ | `watch_repo root` | Enable automatic refresh: a background poll detects new/changed files (mtime + content hash) and re-indexes only those. Watched roots persist across plugin restarts. |
69
+ | `query_memory query` | BM25 search over docs + symbols + experience, optionally query-expanded by the LLM. Returns ranked hits with relative scores, sources, and doc→symbol references. |
70
+ | `remember problem solution` | Save an experience note. Similar problems supersede instead of duplicating. |
71
+ | `forget id_or_query` | Delete stale experience notes. |
72
+
73
+ ## Design
74
+
75
+ ```
76
+ .dsh-project-memory/
77
+ index.json file-level content-hash map (incremental)
78
+ entries.json doc summaries + symbol table entries, keyed by file
79
+ experience.json problem → solution notes (retrieval-only)
80
+ watch.json watched roots
81
+ ```
82
+
83
+ - **Incremental** — content hash per file; only changed files are re-extracted.
84
+ - **Cross-linking** — after indexing, doc summaries are matched against symbol names; matches are attached to the doc entry as `references` and surfaced by `query_memory`.
85
+ - **Query expansion** — when `llmQueryExpansion` is on, `query_memory` asks `ctx.llm` to rewrite the query into several variants (synonyms, EN/CN, identifier guesses) and merges BM25 scores across variants; when off, queries never touch the LLM. Cross-language recall (a Chinese question hitting English content) comes from index time instead: doc keywords are required to cover the document's own language AND English, and doc↔symbol links surface English symbol names from Chinese hits.
86
+ - **Consistency** — the fact layer follows the codebase (hash re-extract / remove-on-delete); the experience layer is retrieval-only with supersede and `forget`. Store writes are serialized per memory directory; the lock is in-process, so avoid running multiple dsh instances against the same project store concurrently.
87
+
88
+ ## Known limitations
89
+
90
+ - **In-process locking** — store writes are serialized per memory directory within one dsh process; two dsh instances sharing a project store is last-writer-wins.
91
+ - **Watch poll holds the lock** — while the watcher re-indexes changed docs (LLM summarization), `remember`/`forget` queue behind it. Overlapping polls serialize on the same lock: safe, but they can pile up on very large diffs.
92
+ - **Silent corruption recovery** — a corrupt store JSON silently falls back to empty and is rebuilt on the next write; there is no warning.
93
+ - **Absolute source paths** — entries cite absolute paths; moving a project invalidates citations until the next re-index.
94
+ - **`forget` by query is eager** — keyword deletion matches at ≥0.5 token overlap and may remove several notes at once; prefer deleting by id for precision.
95
+ - **Cross-language recall depends on index time** — with `llmQueryExpansion` off, a Chinese-only query reaches English content through bilingual keywords captured when docs are indexed, plus doc↔symbol links. Stores indexed before v0.1.1 gain bilingual keywords as files change, or immediately via `index_repo` with `reindex: true`.
96
+
97
+ ## Configuration
98
+
99
+ | Key | Default | Meaning |
100
+ |---|---|---|
101
+ | `memoryDir` | `.dsh-project-memory` | store directory inside each indexed root |
102
+ | `chunkChars` | 3000 | max chars per document chunk |
103
+ | `maxChunksPerFile` | 40 | max chunks per document |
104
+ | `maxFileSizeMb` | 50 | skip documents (incl. PDF) and code files larger than this (MB) |
105
+ | `maxOutputChars` | 8000 | cap for `query_memory` result text (chars) |
106
+ | `maxPdfPages` | 1000 | PDF page cap when pages are not otherwise limited |
107
+ | `llmQueryExpansion` | false | expand queries via `ctx.llm` before BM25 (off by default to save tokens) |
108
+ | `expansionCount` | 6 | max expansion variants |
109
+ | `lazyIndexing` | true | index files the moment the model reads them (`fs/observed`) |
110
+ | `autoIndexOnFirstUse` | false | full scan of the current working directory on plugin load (opt-in) |
111
+ | `watch` | true | enable the background refresh |
112
+ | `watchInterval` | 15 | poll interval (seconds) |
113
+
114
+ ### Toggling features
115
+
116
+ The two most relevant switches are `lazyIndexing` (index a file the moment the model reads it; default on) and `autoIndexOnFirstUse` (full scan of the current working directory on plugin load; default off). Lazily indexed project roots are automatically registered with the watcher, so changed files stay fresh without an explicit `watch_repo`.
117
+
118
+ Settings live in the plugin's config object. To change them, add an override entry to your profile's `cordis.patch.yml` — for the web profile that is `~/.dsh/profiles/web/cordis.patch.yml`:
119
+
120
+ ```yaml
121
+ - id: project-memory
122
+ config:
123
+ lazyIndexing: true # on: index files as the model reads them (default)
124
+ autoIndexOnFirstUse: false # off: no upfront full scan (default)
125
+ llmQueryExpansion: false # off: do not spend tokens on LLM query expansion (default)
126
+ watch: true # on: background refresh for watched roots (default)
127
+ watchInterval: 15 # poll interval in seconds
128
+ ```
129
+
130
+ Only list the keys you want to change; the rest fall back to the plugin defaults. Verify the result with `dsh --profile web --dump-config`.
131
+
132
+ For a one-off run without editing the profile, pass the override as a CLI patch overlay:
133
+
134
+ ```bash
135
+ dsh web --patch ./config.yml
136
+ ```
137
+
138
+ where `config.yml` contains the same override block.
139
+
140
+ ## Development (for contributors)
141
+
142
+ These commands are for **maintaining the plugin code** — regular users do not need them. Installing the plugin only requires the command in [Installation](#installation).
143
+
144
+ ```bash
145
+ npm install
146
+ npm test # 101 checks: chunker / symbols / store / tools / BM25 / links / watch / lazy / config / dump / concurrency / restore / size limit
147
+ ```
148
+
149
+ ## License
150
+
151
+ MIT
@@ -0,0 +1,151 @@
1
+ # dsh-project-memory
2
+
3
+ [English](README.md) | [简体中文](README.zh-CN.md)
4
+
5
+ [![ci](https://github.com/00080000/dsh-project-memory/actions/workflows/ci.yml/badge.svg)](https://github.com/00080000/dsh-project-memory/actions/workflows/ci.yml) [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) [![Listed on dsh-plugin.org](https://dsh-plugin.org/badges/listed.svg)](https://dsh-plugin.org/plugins/00080000/dsh-project-memory)
6
+
7
+ 为 [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness)(dsh)agent 提供持久化的项目记忆。将文档(PDF / Markdown / txt)与代码符号索引进每个工作区独立的存储库,自动维护更新,召回时附源文件引用——文档自动交叉链接到其提及的代码符号。
8
+
9
+ > 插件在磁盘上维护一份精简的项目索引,每条记录指向具体的文件与行号;agent 需要快速了解项目时先查索引,无需重读整个项目。
10
+
11
+ ## 特性
12
+
13
+ - **文档索引** — PDF、Markdown、纯文本按块切分并由 LLM 生成摘要,每条索引携带 `路径:行号` 引用回源文件。
14
+ - **代码符号表** — 通过轻量正则扫描提取函数与类名,不使用 LLM token。
15
+ - **自动刷新** — `watch_repo` 后台轮询,按内容哈希识别新增或变更文件,仅重抽这些文件。
16
+ - **读到即索引** — 文件在模型**实际读取的瞬间**被索引(监听 `fs/observed`),索引是正常工作的副产品,而非额外的一次全量扫描。从未读过的文件不会被索引。项目根通过标记(`.git`、`package.json` 等)、README 加源码目录、或兜底到文件所在目录逐级识别。
17
+ - **文档 ↔ 代码交叉链接** — 文档提及某符号时记录为 `reference`;查询符号时同时带出描述该符号的文档。
18
+ - **BM25 检索** — 对文档、符号与经验笔记进行排序召回,可选 LLM 查询扩展以应对表述不一致。
19
+ - **经验笔记** — 记录问题 → 方案;相似问题覆盖而非重复;笔记仅在检索命中时返回。笔记数量有界:容量随项目规模伸缩(钳制在 100–2000),超限时淘汰最旧的笔记。
20
+ - **依赖极简** — 纯 JavaScript;唯一运行时依赖是 `pdfjs-dist`(PDF 文本提取),无需原生构建。
21
+
22
+ ## 工作原理
23
+
24
+ 设计遵循四个原则:
25
+
26
+ - **易失性** — 上下文是临时的,会话压缩即丢失。
27
+ - **持久性** — 索引存于磁盘,跨压缩与会话保留。
28
+ - **紧凑性** — 仅存摘要;索引规模约为其覆盖源码的 0.5%(示例项目中 8.8 MB 源码 → 49 KB 索引),检索替代了通读整个文件。
29
+ - **可核验性** — 命中在适用时携带 `路径:行号` 引用,agent 可对照源文件核实。
30
+
31
+ 构建索引无需预先全量扫描:文件在模型读取时被索引,索引恰好覆盖实际处理过的内容。未变更的文件重读是空操作(内容哈希),因此索引的持续维护开销很低。
32
+
33
+ 存储按项目独立存放,并跟随代码库变化:文件变更按内容哈希重新抽取,文件删除则同步移除。经验层仅检索,累积不影响上下文。
34
+
35
+ ## 安装
36
+
37
+ 实测覆盖 dsh **0.1.0-rc.7 → 0.1.1-rc.2**。更高的 rc 版本预期可用——插件只使用稳定接口(`defineTool`、`llm.stream`、`Schema`)——但未经逐一验证。宿主需提供 `@deepseek-ai/cordis` ^4.0.1 与 `@deepseek-ai/schemastery` ^3.18.1,已通过 peerDependencies 声明。
38
+
39
+ ```bash
40
+ cd dsh-project-memory
41
+ dsh plugin --profile web add . -w
42
+ ```
43
+
44
+ `-w`(workspace-root)标志是必需的:profile 目录是 pnpm 工作区根目录,不带该标志 pnpm 会拒绝 add。其他目录下同样可用路径形式:`dsh plugin --profile web add /path/to/dsh-project-memory -w`。
45
+
46
+ 插件同时发布在 npm 上(scoped 包):
47
+
48
+ ```bash
49
+ dsh plugin --profile web add @yolk_vat-y/dsh-project-memory -w
50
+ ```
51
+
52
+ 每个版本会附带预构建 tarball,无需构建步骤即可安装:
53
+
54
+ ```bash
55
+ dsh plugin --profile web add /path/to/dsh-project-memory-0.1.3.tgz
56
+ ```
57
+
58
+ 每个被索引的项目在 `<root>/.dsh-project-memory/` 下有独立存储。如无需入库,可加入 `.gitignore`。
59
+
60
+ ## 用法
61
+
62
+ 以下工具由 **agent 自动调用**,无需用户手动输入。在对话中直接说自然语言即可——例如「给这个项目建个索引」或「auth 模块是干嘛的」,或者正常开发即可——agent 会自动调用对应工具。默认开启「读到即索引」(`lazyIndexing`):模型读哪个文件,就顺便索引哪个文件,记忆在你干活的过程中自然积累。`watch_repo` 让显式监听的根目录在后台保持新鲜;`index_repo` 强制对项目做一次全量回填(未变更文件自动跳过)。
63
+
64
+ | 工具 | 用途 |
65
+ |---|---|
66
+ | `index_doc file_path` | 索引单个文档(PDF/MD/txt):分块 → LLM 摘要 → 带 `路径:行号` 入库。未变更文件自动跳过。 |
67
+ | `index_repo root` | 索引整个项目:文档由 LLM 生成摘要,代码文件生成零 token 符号表。增量更新、清理已删除文件、文档与符号交叉链接。 |
68
+ | `watch_repo root` | 启用自动刷新:后台轮询检测新增/变更文件(mtime + 内容哈希),仅重抽这些文件。监听的项目在插件重启后自动恢复。 |
69
+ | `query_memory query` | 对文档、符号、经验执行 BM25 检索,可选 LLM 查询扩展。返回带相对分数(0-100)、引用与文档→符号链接的排序结果。 |
70
+ | `remember problem solution` | 保存经验笔记。相似问题覆盖而非重复。 |
71
+ | `forget id_or_query` | 删除过期经验笔记。 |
72
+
73
+ ## 设计
74
+
75
+ ```
76
+ .dsh-project-memory/
77
+ index.json 文件级内容哈希表(增量)
78
+ entries.json 文档摘要 + 符号表条目,按文件
79
+ experience.json 问题 → 方案笔记(仅检索)
80
+ watch.json 被监听根目录
81
+ ```
82
+
83
+ - **增量** — 按文件内容哈希,仅重新抽取变更文件。
84
+ - **交叉链接** — 索引后将文档摘要与符号名匹配,命中符号以 `references` 挂载到文档条目,由 `query_memory` 带出。
85
+ - **查询扩展** — `llmQueryExpansion` 开启时,`query_memory` 让 `ctx.llm` 将查询改写为多个变体(同义词、中英、符号名猜测),再跨变体合并 BM25 分数;关闭时查询完全不碰 LLM。跨语种召回(中文问题命中英文内容)改由索引时承担:文档 keywords 要求同时覆盖文档语言与英文,doc↔symbol 链接也会从中文命中带出英文符号名。
86
+ - **一致性** — 事实层跟随代码库(哈希重抽 / 删除即移除);经验层仅检索,配合覆盖与 `forget` 机制。每个记忆目录的写入按进程内互斥锁串行化;请避免多个 dsh 实例同时写同一项目存储。
87
+
88
+ ## 已知限制
89
+
90
+ - **进程内锁** — 存储写入按记忆目录在一个 dsh 进程内串行化;两个 dsh 实例共享同一项目存储时后写覆盖先写。
91
+ - **watch 轮询持锁** — watcher 重索引变更文档(LLM 摘要)期间,`remember`/`forget` 会排队等待。重叠轮询靠同一把锁串行:安全,但大改动时可能堆积。
92
+ - **损坏静默重建** — 存储 JSON 损坏时静默回落为空并在下次写入时重建,无告警。
93
+ - **绝对路径引用** — 条目引用绝对路径;项目搬家后引用失效,重建索引即恢复。
94
+ - **`forget` 按关键词删除偏激进** — 关键词删除按 ≥0.5 token 重叠匹配,可能一次删掉多条;追求精确请用 id 删除。
95
+ - **跨语种召回依赖索引时** — `llmQueryExpansion` 关闭时,纯中文查询靠索引时捕获的双语 keywords 和 doc↔symbol 链接触达英文内容。v0.1.1 之前建立的索引随文件变更逐步获得双语关键词,或用 `index_repo` 的 `reindex: true` 立即重建。
96
+
97
+ ## 配置
98
+
99
+ | 键 | 默认值 | 含义 |
100
+ |---|---|---|
101
+ | `memoryDir` | `.dsh-project-memory` | 每个被索引根目录内的存储目录 |
102
+ | `chunkChars` | 3000 | 每个文档块最大字符数 |
103
+ | `maxChunksPerFile` | 40 | 每文档最大块数 |
104
+ | `maxFileSizeMb` | 50 | 大于该值(MB)的文档(含 PDF)/代码文件跳过 |
105
+ | `maxOutputChars` | 8000 | `query_memory` 返回文本上限(字符) |
106
+ | `maxPdfPages` | 1000 | 未另行限制时 PDF 的页数上限 |
107
+ | `llmQueryExpansion` | false | BM25 检索前通过 `ctx.llm` 扩展查询(默认关闭,节省 token) |
108
+ | `expansionCount` | 6 | 扩展变体上限 |
109
+ | `lazyIndexing` | true | 模型读取文件的瞬间即索引(`fs/observed`) |
110
+ | `autoIndexOnFirstUse` | false | 插件加载时对当前工作目录做全量扫描(可选) |
111
+ | `watch` | true | 启用后台刷新 |
112
+ | `watchInterval` | 15 | 轮询间隔(秒) |
113
+
114
+ ### 功能开关
115
+
116
+ 两个最常用的开关是 `lazyIndexing`(模型读取文件的瞬间即索引;默认开启)和 `autoIndexOnFirstUse`(插件加载时对当前工作目录做全量扫描;默认关闭)。懒加载建立的索引根会自动注册到 watcher,文件变更无需手动 `watch_repo` 也能保持新鲜。
117
+
118
+ 配置存放在插件的 config 对象中。修改方式:在 profile 的 `cordis.patch.yml` 里加一条覆盖项——web profile 对应 `~/.dsh/profiles/web/cordis.patch.yml`:
119
+
120
+ ```yaml
121
+ - id: project-memory
122
+ config:
123
+ lazyIndexing: true # 开启:模型读到哪个文件就索引哪个(默认)
124
+ autoIndexOnFirstUse: false # 关闭:不做加载时的全量扫描(默认)
125
+ llmQueryExpansion: false # 关闭:不用 LLM 扩展查询,节省 token(默认)
126
+ watch: true # 开启:被监听根目录后台保持新鲜(默认)
127
+ watchInterval: 15 # 轮询间隔(秒)
128
+ ```
129
+
130
+ 只需列出要改的键,其余键回落到插件默认值。用 `dsh --profile web --dump-config` 验证生效。
131
+
132
+ 不想改 profile 文件、只想临时试一次,可用 CLI 补丁覆盖:
133
+
134
+ ```bash
135
+ dsh web --patch ./config.yml
136
+ ```
137
+
138
+ 其中 `config.yml` 内容就是上面的覆盖块。
139
+
140
+ ## 开发(面向贡献者)
141
+
142
+ 以下命令用于**维护插件源码**,普通用户无需执行。安装插件只需使用[安装](#安装)一节中的命令。
143
+
144
+ ```bash
145
+ npm install
146
+ npm test # 101 项检查:chunker / symbols / store / tools / BM25 / links / watch / lazy / config / dump / concurrency / restore / size limit
147
+ ```
148
+
149
+ ## 许可证
150
+
151
+ MIT
@@ -0,0 +1,13 @@
1
+ - insert:
2
+ - id: project-memory
3
+ name: dsh-project-memory
4
+ config:
5
+ memoryDir: .dsh-project-memory
6
+ chunkChars: 3000
7
+ maxChunksPerFile: 40
8
+ llmQueryExpansion: false
9
+ expansionCount: 6
10
+ lazyIndexing: true
11
+ autoIndexOnFirstUse: false
12
+ watch: true
13
+ watchInterval: 15
package/package.json ADDED
@@ -0,0 +1,64 @@
1
+ {
2
+ "name": "@yolk_vat-y/dsh-project-memory",
3
+ "version": "0.1.3",
4
+ "description": "Persistent project memory for dsh agents: index docs (PDF/Markdown/text) and code symbols into a searchable per-workspace store, recall them with cited sources, and keep experience entries (problems -> solutions) searchable on demand.",
5
+ "type": "module",
6
+ "main": "src/index.js",
7
+ "files": [
8
+ "src",
9
+ "cordis.patch.yml",
10
+ "README.md",
11
+ "README.zh-CN.md",
12
+ "CHANGELOG.md"
13
+ ],
14
+ "repository": {
15
+ "type": "git",
16
+ "url": "https://github.com/00080000/dsh-project-memory.git"
17
+ },
18
+ "scripts": {
19
+ "test": "node test/run-test.mjs"
20
+ },
21
+ "keywords": [
22
+ "dsh",
23
+ "dsh-plugin",
24
+ "deepseek-harness",
25
+ "memory",
26
+ "project-memory",
27
+ "rag",
28
+ "codebase",
29
+ "cordis"
30
+ ],
31
+ "license": "MIT",
32
+ "publishConfig": {
33
+ "access": "public"
34
+ },
35
+ "engines": {
36
+ "node": ">=18"
37
+ },
38
+ "dependencies": {
39
+ "pdfjs-dist": "^4.10.38"
40
+ },
41
+ "devDependencies": {
42
+ "@deepseek-ai/cordis": "4.0.1",
43
+ "@deepseek-ai/dsh-tools": "0.1.1-rc.2",
44
+ "@deepseek-ai/schemastery": "3.18.1",
45
+ "@deepseek-ai/dsh-llm": "0.1.1-rc.2"
46
+ },
47
+ "peerDependencies": {
48
+ "@deepseek-ai/cordis": "^4.0.1",
49
+ "@deepseek-ai/dsh-tools": ">=0.0.1-rc.1 <0.1.0 || >=0.1.0-rc.1 <0.2.0-0 || >=0.1.1-rc.1 <0.2.0-0",
50
+ "@deepseek-ai/schemastery": "^3.18.1",
51
+ "@deepseek-ai/dsh-llm": ">=0.0.1-rc.1 <0.1.0 || >=0.1.0-rc.1 <0.2.0-0 || >=0.1.1-rc.1 <0.2.0-0"
52
+ },
53
+ "peerDependenciesMeta": {
54
+ "@deepseek-ai/cordis": { "optional": true },
55
+ "@deepseek-ai/dsh-tools": { "optional": true },
56
+ "@deepseek-ai/schemastery": { "optional": true },
57
+ "@deepseek-ai/dsh-llm": { "optional": true }
58
+ },
59
+ "dsh": {
60
+ "bundle": {
61
+ "patch": "./cordis.patch.yml"
62
+ }
63
+ }
64
+ }
package/src/chunker.js ADDED
@@ -0,0 +1,43 @@
1
+ export function chunkText(text, chunkChars = 3000, maxChunks = 40) {
2
+ if (!Number.isFinite(chunkChars) || chunkChars < 1) chunkChars = 3000
3
+ if (!Number.isFinite(maxChunks) || maxChunks < 1) maxChunks = 40
4
+ const lines = text.split(/\r?\n/)
5
+ const sections = []
6
+ let current = { title: '', lines: [], line: 1 }
7
+ for (let i = 0; i < lines.length; i++) {
8
+ const line = lines[i]
9
+ if (/^\s*#{1,6}\s/.test(line)) {
10
+ if (current.lines.length) {
11
+ sections.push(current)
12
+ current = { title: '', lines: [], line: i + 1 }
13
+ }
14
+ current.title = line.replace(/^\s*#{1,6}\s*/, '').trim()
15
+ current.line = i + 1
16
+ }
17
+ current.lines.push(line)
18
+ }
19
+ if (current.lines.length) sections.push(current)
20
+
21
+ const chunks = []
22
+ for (const section of sections) {
23
+ if (!section.lines.join('').trim()) continue
24
+ let block = section.lines.join('\n').trim()
25
+ let line = section.line
26
+ while (block.length > chunkChars) {
27
+ let splitAt = block.lastIndexOf('\n\n', chunkChars)
28
+ if (splitAt < chunkChars * 0.5) splitAt = block.lastIndexOf(' ', chunkChars)
29
+ if (splitAt < chunkChars * 0.5) splitAt = chunkChars
30
+ const part = block.slice(0, splitAt).trim()
31
+ if (part) chunks.push({ title: section.title, text: part, line })
32
+ line += part.split('\n').length
33
+ block = block.slice(splitAt).trim()
34
+ if (chunks.length >= maxChunks) break
35
+ }
36
+ if (block) {
37
+ chunks.push({ title: section.title, text: block, line })
38
+ }
39
+ if (chunks.length >= maxChunks) break
40
+ }
41
+
42
+ return chunks.filter((c) => c.text).slice(0, maxChunks)
43
+ }
@@ -0,0 +1,54 @@
1
+ import path from 'node:path'
2
+ import { stat } from 'node:fs/promises'
3
+ import { looksLikeDump, readTextFile } from './util/fs.js'
4
+ import { parsePdf } from './parsers/pdfjs-parser.js'
5
+ import { chunkText } from './chunker.js'
6
+ import { extractDocEntry } from './llm.js'
7
+
8
+ export async function extractTextFromFile(filePath, { maxFileSizeMb = 50, maxPdfPages = 1000 } = {}) {
9
+ const ext = path.extname(filePath).toLowerCase()
10
+ if (ext === '.pdf') {
11
+ if (maxFileSizeMb) {
12
+ const stats = await stat(filePath)
13
+ if (stats.size > maxFileSizeMb * 1024 * 1024) {
14
+ throw new Error(`File too large to index (${(stats.size / 1024 / 1024).toFixed(1)} MB), limit is ${maxFileSizeMb} MB`)
15
+ }
16
+ }
17
+ const result = await parsePdf(filePath, { maxPages: maxPdfPages })
18
+ return result.markdown
19
+ }
20
+ return readTextFile(filePath, maxFileSizeMb ? maxFileSizeMb * 1024 * 1024 : Infinity)
21
+ }
22
+
23
+ const DOC_CONCURRENCY = 4
24
+
25
+ export async function buildDocEntries(llm, filePath, { chunkChars = 3000, maxChunks = 40, maxFileSizeMb = 50, maxPdfPages = 1000 } = {}) {
26
+ const text = await extractTextFromFile(filePath, { maxFileSizeMb, maxPdfPages })
27
+ if (looksLikeDump(text)) return null
28
+ const chunks = chunkText(text, chunkChars, maxChunks)
29
+ const metas = new Array(chunks.length)
30
+ let cursor = 0
31
+ await Promise.all(
32
+ Array.from({ length: Math.min(DOC_CONCURRENCY, chunks.length) }, () =>
33
+ (async () => {
34
+ while (cursor < chunks.length) {
35
+ const i = cursor++
36
+ metas[i] = await extractDocEntry(llm, chunks[i], filePath)
37
+ }
38
+ })(),
39
+ ),
40
+ )
41
+ return metas.map((meta, i) => ({
42
+ id: `${relativeId(filePath)}#${i}`,
43
+ sourcePath: filePath,
44
+ sourceLine: chunks[i].line,
45
+ type: 'doc',
46
+ title: meta.title,
47
+ summary: meta.summary,
48
+ keywords: meta.keywords,
49
+ }))
50
+ }
51
+
52
+ function relativeId(filePath) {
53
+ return String(filePath).replace(/[\\/:\s]/g, '_')
54
+ }
package/src/index.js ADDED
@@ -0,0 +1,62 @@
1
+ import Schema from '@deepseek-ai/schemastery'
2
+ import { indexDocTool } from './tools/index-doc.js'
3
+ import { indexRepoTool, indexRepository } from './tools/index-repo.js'
4
+ import { queryMemoryTool } from './tools/query-memory.js'
5
+ import { rememberTool } from './tools/remember.js'
6
+ import { forgetTool } from './tools/forget.js'
7
+ import { watchRepoTool } from './tools/watch-repo.js'
8
+ import { WatchManager } from './watch.js'
9
+ import { setupLazyIndexing } from './lazy.js'
10
+
11
+ export const name = 'dsh-project-memory'
12
+ export const inject = ['llm', 'tools']
13
+
14
+ export const Config = Schema.object({
15
+ memoryDir: Schema.string().default('.dsh-project-memory'),
16
+ chunkChars: Schema.number().default(3000),
17
+ maxChunksPerFile: Schema.number().default(40),
18
+ maxFileSizeMb: Schema.number().default(50),
19
+ maxOutputChars: Schema.number().default(8000),
20
+ maxPdfPages: Schema.number().default(1000),
21
+ llmQueryExpansion: Schema.boolean().default(false),
22
+ expansionCount: Schema.number().default(6),
23
+ lazyIndexing: Schema.boolean().default(true),
24
+ autoIndexOnFirstUse: Schema.boolean().default(false),
25
+ watch: Schema.boolean().default(true),
26
+ watchInterval: Schema.number().default(15),
27
+ })
28
+
29
+ export function apply(ctx, config) {
30
+ const watchManager = new WatchManager(ctx, config)
31
+ if (config.watch) {
32
+ watchManager.restorePersisted()
33
+ ctx.effect(() => {
34
+ watchManager.start(config.watchInterval * 1000)
35
+ return () => watchManager.stop()
36
+ })
37
+ }
38
+
39
+ if (config.lazyIndexing) {
40
+ setupLazyIndexing(ctx, config, watchManager)
41
+ }
42
+
43
+ ctx.tools.register(indexDocTool(ctx, config))
44
+ ctx.tools.register(indexRepoTool(ctx, config))
45
+ ctx.tools.register(queryMemoryTool(ctx, config))
46
+ ctx.tools.register(rememberTool(config))
47
+ ctx.tools.register(forgetTool(config))
48
+ ctx.tools.register(watchRepoTool(watchManager, config))
49
+
50
+ if (config.autoIndexOnFirstUse) {
51
+ ctx.effect(async () => {
52
+ const root = process.cwd()
53
+ try {
54
+ watchManager.addRoot(root)
55
+ const report = await indexRepository(ctx, config, root)
56
+ console.log(`[dsh-project-memory] ${report}`)
57
+ } catch (err) {
58
+ console.error(`[dsh-project-memory] auto-index failed for ${root}: ${err.message}`)
59
+ }
60
+ })
61
+ }
62
+ }