dsh-memory-dba 0.1.4 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +137 -0
- package/MEMORY.md +83 -0
- package/README.md +76 -35
- package/README.zh-CN.md +38 -14
- package/bin/cli.mjs +44 -1
- package/package.json +47 -46
- package/src/assemble.mjs +67 -12
- package/src/catalog.mjs +114 -0
- package/src/events.mjs +138 -0
- package/src/index.mjs +192 -28
- package/src/search.mjs +88 -12
- package/src/store.mjs +18 -3
- package/src/stream.mjs +66 -25
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,142 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.3.0 — 2026-11-04
|
|
4
|
+
|
|
5
|
+
- Persist complete sanitized text events before stream publication; retain drafts, source roles and candidate observations. Recover completed unpublished turns without duplicate event references.
|
|
6
|
+
- Keep bounded head-and-tail stream excerpts with explicit event references.
|
|
7
|
+
- Add shared topic cards and evidence-backed fact revisions, retraction and conflict labels, with explicit CLI authoring and history inspection.
|
|
8
|
+
- Validate Python keyword index freshness against content manifests; rebuild atomically on append, edit, rename, deletion or schema changes.
|
|
9
|
+
- Keep workspace bodies behind explicit file selection and versioned facts out of independent Python vector caches. CLI semantic recall requires opt-in.
|
|
10
|
+
- Preserve the previous session snapshot, isolation and retrieval-budget fixes.
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
## 0.2.2 — 2026-11-03
|
|
14
|
+
|
|
15
|
+
Session-scoped prompt state, a cold-path hint, and a smaller default surface.
|
|
16
|
+
|
|
17
|
+
- **Core is snapshotted per session.** The `memory-dba:core` provider now reads
|
|
18
|
+
once on the first assembly of a session (`context.agent.id`) and replays
|
|
19
|
+
`{text, bytes, clipped}` for the rest of that session, keeping the cacheable
|
|
20
|
+
head stable. New session = new id = latest file. `coreRefresh: 'always'` opts
|
|
21
|
+
back into a read on every assembly. Without an agent id the provider rereads
|
|
22
|
+
every time and keeps only a last-read slot. Session snapshots remain until
|
|
23
|
+
`session/disposed`; memory usage therefore scales with the number and size of
|
|
24
|
+
live sessions, and disposal releases each snapshot.
|
|
25
|
+
- **Hits are isolated per session.** `latestHits` was one process-wide slot, so
|
|
26
|
+
two live sessions overwrote each other. Hits now live in a
|
|
27
|
+
`Map<sessionId, text>` written by `user/message` and read by
|
|
28
|
+
`context.agent.id`; contexts without an id keep the old single-slot fallback.
|
|
29
|
+
Disposal clears the session entry.
|
|
30
|
+
- **Empty recall gets a cold-path hint.** `buildHits()` returns one line naming
|
|
31
|
+
the Python index (`py -3.14 <home>/../scripts/nb_search.py "查询"`) instead of
|
|
32
|
+
`''` when there are no hits; a non-empty hit list never carries it. The new
|
|
33
|
+
`coldHint` config replaces the sentence; `'off'` or an explicit empty value
|
|
34
|
+
disables it.
|
|
35
|
+
- **Semantic recall is off by default.** `vector.enabled` now defaults to
|
|
36
|
+
`false`. Level 1 is already narrow (core injected in full, workspace names
|
|
37
|
+
only, stream short lines), while the synchronous worker costs up to 1500 ms
|
|
38
|
+
and a GPU for a measured 0/143 non-core coverage (core 67/74, stream 0/124,
|
|
39
|
+
workspace 0/19). The Python CLI remains the semantic path over full text.
|
|
40
|
+
`vector: { enabled: true }` explicitly restores the old behavior; when it is
|
|
41
|
+
false neither the warmup nor `semanticRecall` runs.
|
|
42
|
+
- **Tests.** Default config never calls the vector runtime; a stub proves one
|
|
43
|
+
core read per session, retention past 32 live sessions, disposal refresh,
|
|
44
|
+
`always` rereading, and the no-id fallback; two sessions keep separate hits
|
|
45
|
+
and disposal clears them;
|
|
46
|
+
empty recall shows the hint while a disabled/overridden hint behaves; the
|
|
47
|
+
semantic-only injection test now opts in explicitly.
|
|
48
|
+
|
|
49
|
+
## 0.2.1 — 2026-11-03
|
|
50
|
+
|
|
51
|
+
Retrieval-corpus change: `core.md` is injection-only, matching the Python side.
|
|
52
|
+
|
|
53
|
+
- **`core` leaves the recall corpus.** `loadCorpus()` now indexes `stream.md` and
|
|
54
|
+
the workspace names only. `buildCore()` is unchanged and still injects the whole
|
|
55
|
+
core layer as a system-prompt section every turn. Core was previously in both
|
|
56
|
+
places, so the same text could consume the prompt budget twice; the Python
|
|
57
|
+
`nb_search.py` already excludes `core` by default for exactly this reason.
|
|
58
|
+
- **Semantic channel.** Its keys come from the same corpus, so core is no longer
|
|
59
|
+
scored there either; a key set without core entries is valid.
|
|
60
|
+
- **Tests.** The corpus contains no `source === 'core'` document, `buildCore()`
|
|
61
|
+
still returns the core text, a core-only term produces an empty recall block,
|
|
62
|
+
and the semantic key set from a core-less corpus is accepted.
|
|
63
|
+
|
|
64
|
+
## 0.2.0 — 2026-11-03
|
|
65
|
+
|
|
66
|
+
Runtime change: workspace layers are retrieved in two levels, and level 1 knows
|
|
67
|
+
only file names.
|
|
68
|
+
|
|
69
|
+
- **Level-1 corpus.** One document per `ws/*.md` file. Its text is exactly
|
|
70
|
+
`ws/<area>.md` plus the first line when that line is a `# ...` heading; the
|
|
71
|
+
body is never indexed. `date` is `null` for these documents because a section
|
|
72
|
+
date cannot exist without the section body.
|
|
73
|
+
- **Rendering.** Workspace hits render as `- [ws/<area>.md]` with no body (the
|
|
74
|
+
`(semantic)` marker is preserved). The hits block adds one hint per block:
|
|
75
|
+
read the body with `py -3.14 <home>/../scripts/nb_search.py "查询" --file <区名>`.
|
|
76
|
+
The script path is derived from the configured `home` once at mount; the new
|
|
77
|
+
`detailHint` config replaces the whole hint line (non-Windows installs, another
|
|
78
|
+
Python launcher). The hint is placed before the hit bullets so a tight budget
|
|
79
|
+
cannot clip it away.
|
|
80
|
+
- **Python output.** `nb_search.py` level 1 uses one numbering sequence, so a
|
|
81
|
+
name-only workspace row is `1. [ws/<area>.md]` and the following stream hits
|
|
82
|
+
continue `2.`, `3.`; level 2 always prints the level and the files it searched,
|
|
83
|
+
including when nothing matched — a zero-hit second pass can never look like a
|
|
84
|
+
plain level-1 miss.
|
|
85
|
+
- **Semantic channel.** The embedding cache is built from the same corpus, so
|
|
86
|
+
workspace entries are name-only there too and a semantic hit renders as the
|
|
87
|
+
file name. The vector path can no longer bypass level 1.
|
|
88
|
+
- **Why.** Recall runs on every user message, while a workspace section is long.
|
|
89
|
+
With body indexing, any unrelated query could pull a whole section into the
|
|
90
|
+
prompt; in the DBA tree this happened with `ws/角色卡.md`. Level 1 answers
|
|
91
|
+
"which note is relevant", level 2 answers "what it says", and only an explicit
|
|
92
|
+
`--file` unlocks the second answer.
|
|
93
|
+
- **Python side.** `nb_index.py` writes name-only workspace documents and rejects
|
|
94
|
+
old indexes by version; `nb_search.py --file` re-reads the file, splits on
|
|
95
|
+
`##`, reuses BM25 and errors non-zero when a file is missing; `port/dba.py`
|
|
96
|
+
forwards `files=` without changing the default level-1 behavior.
|
|
97
|
+
- **`exclude` stays.** It is an extra guardrail, not the gate: the gate is that
|
|
98
|
+
level 1 contains no body at all. Relying on an exclusion list while indexing
|
|
99
|
+
bodies means every unlisted file can still be recalled in full.
|
|
100
|
+
- **Tests.** Area-name retrieval, body text absent from rendered hits, the
|
|
101
|
+
`--file` hint, body-only queries not surfacing a workspace file, name-only
|
|
102
|
+
semantic excerpts, and first-line-title extraction.
|
|
103
|
+
|
|
104
|
+
## 0.1.6 — 2026-11-03
|
|
105
|
+
|
|
106
|
+
Runtime change: per-turn retrieval hits moved out of the system prompt.
|
|
107
|
+
|
|
108
|
+
- The `hits` block is now registered with `systemPrompt.context()` (default order
|
|
109
|
+
`300`, above the harness's own contexts at 100–210) instead of
|
|
110
|
+
`systemPrompt.section()`. The harness materializes runtime context as a user-role
|
|
111
|
+
snapshot appended after the conversation.
|
|
112
|
+
- Why it matters: a section is the first message of a request, so a hit list that
|
|
113
|
+
changed inside it moved the prompt head and invalidated prefix caching for every
|
|
114
|
+
message behind it — the whole conversation, once per turn. Measured on a local
|
|
115
|
+
corpus of 13,734 requests: step 2 of a turn ran at a **51.4%** hit rate (47.3M
|
|
116
|
+
uncached tokens, 23.7% of all misses) while step 3 and later ran at 92.9–96%.
|
|
117
|
+
A single 550k-token session showed a constant 4,992-token hit, exactly the common
|
|
118
|
+
prompt head, on every affected request.
|
|
119
|
+
- `core` stays a system-prompt section (order `-50`) — it changes every few days,
|
|
120
|
+
not every turn.
|
|
121
|
+
- Config: `hitsOrder` default `200` → `300`, now a runtime-context order.
|
|
122
|
+
- Tests: a regression test pins `hits` to runtime context so the block cannot drift
|
|
123
|
+
back into the system prompt.
|
|
124
|
+
|
|
125
|
+
## 0.1.5 — 2026-10-26
|
|
126
|
+
|
|
127
|
+
Documentation-only revision. No runtime source file, API, or behavior changed.
|
|
128
|
+
|
|
129
|
+
- Separated the two state interfaces that the earlier scope text merged into one:
|
|
130
|
+
**(A) memory-token prefix** (frozen RWKV state → *k* continuous tokens prepended
|
|
131
|
+
to a frozen LLM's instruction embeddings, trained on downstream task error) and
|
|
132
|
+
**(B) state-to-text slot decoding** (state → four readable slots → text).
|
|
133
|
+
- Recorded route (A)'s actual standing: 0.5477 vs 0.6774 direct-MLP baseline on 50
|
|
134
|
+
sealed test episodes, episode-specific per the correct/wrong/zero/random ablation,
|
|
135
|
+
but token-count ranking unsettled, 64-token seeds incomplete, and behind the MLP
|
|
136
|
+
baseline on a 192k-sample budget.
|
|
137
|
+
- Bounded every `0/1779` figure to route (B), and named the runtime's actual third
|
|
138
|
+
path: text summaries plus retrieval.
|
|
139
|
+
|
|
3
140
|
## 0.1.4 — 2026-10-26
|
|
4
141
|
|
|
5
142
|
Documentation-only revision. No runtime source file, API, or behavior changed.
|
package/MEMORY.md
ADDED
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# 记忆记录与检索(0.3.0)
|
|
2
|
+
|
|
3
|
+
本版保留 core 的会话快照和工作区的两级读取,增加完整文本事件、主题导航卡及可纠正的事实目录。不新增模型调用,不默认开启向量检索。
|
|
4
|
+
|
|
5
|
+
## 文件与职责
|
|
6
|
+
|
|
7
|
+
| 文件 | 内容 | 自动召回范围 |
|
|
8
|
+
| --- | --- | --- |
|
|
9
|
+
| `core.md` | 每轮需要的长期规则 | 按会话快照,受原有预算限制 |
|
|
10
|
+
| `stream.md` | 回合的短摘录,含 `[event=ID]` | Node 的近期窗口;Python 的全历史关键词索引 |
|
|
11
|
+
| `events-v1/ID.json` | 完整脱敏用户/助手文本、来源、轮次状态和候选观察项 | 不把整个档案自动注入;根据 ID 显式读取 |
|
|
12
|
+
| `catalog.json` | 主题卡和带来源、纠正关系的事实 | 导航词用于选册;有效事实按相关性召回 |
|
|
13
|
+
| `ws/名称.md` | 原始工作区正文 | 一级只返回文件名;`--file` 才读取正文 |
|
|
14
|
+
| `.index/inverted.json` | Python 派生关键词索引与源文件摘要清单 | 查询前检查新鲜度,必要时原子重建 |
|
|
15
|
+
|
|
16
|
+
完整文本指宿主发送的用户和助手**文本消息**,不包含图片、音频、模型隐藏推理或完整工具执行轨迹。启发式脱敏沿用原有规则。旧流水没有完整来源,不能从被截掉的文字恢复档案;本版只为新事件留存。
|
|
17
|
+
|
|
18
|
+
## 事件写入与读取
|
|
19
|
+
|
|
20
|
+
用户消息到达时先保存 pending 档案;收到助手完整消息时更新;回合结束先保存 complete,再发布带事件 ID 的流水。重启仅重试已完成但尚未发布的队列项。中断事件保留档案,不把它伪装成完成结果,也不自动重演对话。
|
|
21
|
+
|
|
22
|
+
长消息的流水摘录保留首尾,仍受 300/600 字符预算限制。摘录用于导航,完整语义应回读事件。`observations` 仅识别行首的“决定:”“约束:”“结果:”“计划:”“纠正:”,保存为候选陈述,不自动成为有效事实。
|
|
23
|
+
|
|
24
|
+
在仓库根目录执行(`--home` 指向记忆目录本身):
|
|
25
|
+
|
|
26
|
+
```powershell
|
|
27
|
+
node packages/dsh-memory-dba/bin/cli.mjs event --id EVENT_ID --home D:/AI/DBA/memory --json
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
档案是不可再生的来源文件,应随整个记忆目录备份。复制 `catalog.json` 时也要复制其引用的 `events-v1` 档案。维护单一写入宿主;本版不提供跨进程并发会话消息合并。
|
|
31
|
+
|
|
32
|
+
## 主题卡:允许检索导航信息,正文仍显式解锁
|
|
33
|
+
|
|
34
|
+
`catalog.json` 的顶层结构为 `{"version":1,"cards":[],"facts":[]}`。每张卡只允许下列字段:
|
|
35
|
+
|
|
36
|
+
```json
|
|
37
|
+
{
|
|
38
|
+
"file": "ws/桌面自动化.md",
|
|
39
|
+
"topics": ["浏览器控制"],
|
|
40
|
+
"aliases": ["远程调试", "已登录网页"],
|
|
41
|
+
"entities": ["CDP", "Chrome"],
|
|
42
|
+
"questions": ["如何读取现有浏览器页面?"]
|
|
43
|
+
}
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
把这段 JSON 存为文件,通过 `card-put --input 卡片.json --home ...` 写入。字段用于一级 BM25 选册,不会将工作区正文复制到索引;命中仍只渲染文件引用。排除文件仍不进入候选集。主题卡需要随文件改名和主题变化维护,不从全文自动抽取隐含事实。
|
|
47
|
+
|
|
48
|
+
## 事实:来源陈述、有效状态和纠正关系
|
|
49
|
+
|
|
50
|
+
事实由显式的 `fact-add --input 事实.json --home ...` 写入。示例中的事件 ID 必须替换为真实档案,并且 quote 必须确实出现在对应角色的文本中:
|
|
51
|
+
|
|
52
|
+
```json
|
|
53
|
+
{
|
|
54
|
+
"id": "request-timeout-v2",
|
|
55
|
+
"key": "request-timeout",
|
|
56
|
+
"claim": "请求超时设为 60 秒",
|
|
57
|
+
"kind": "constraint",
|
|
58
|
+
"status": "active",
|
|
59
|
+
"source": {
|
|
60
|
+
"event_id": "REAL_EVENT_ID",
|
|
61
|
+
"role": "user",
|
|
62
|
+
"quote": "请求超时设为 60 秒"
|
|
63
|
+
},
|
|
64
|
+
"supersedes": ["request-timeout-v1"],
|
|
65
|
+
"observed_at": "2026-11-04T00:00:00.000Z"
|
|
66
|
+
}
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
- `kind`:decision、constraint、result 或 plan。计划不会自动变成结果。
|
|
70
|
+
- `source.role`:user 或 assistant。展示为用户陈述/助手报告;quote 校验只证明来源存在,不证明 claim 的解释正确或执行已获工具验证。
|
|
71
|
+
- `supersedes`:只能引用同一个 key 下已经存在的前驱记录。新值生效后,旧值退出默认事实召回。
|
|
72
|
+
- `status: retracted`:撤回记录。撤回后不会自动恢复曾被它替代的旧值;恢复旧值需要新的显式记录。
|
|
73
|
+
- 同 key 下多个未建立替代关系的有效值标记为 conflict,不自动按时间挑一个“真值”。
|
|
74
|
+
|
|
75
|
+
普通查询只返回未被替代或撤回的条目;历史通过 `facts 查询 --history --json --home ...` 显式查看。原始流水仍保留,检索文本会提醒它可能包含计划、失败或已被纠正的说法。当前事实不能自动覆盖 core 快照中的旧规则;修改长期规则时也应维护 core,并遵守原有的会话生效边界。
|
|
76
|
+
|
|
77
|
+
## 索引更新与边界
|
|
78
|
+
|
|
79
|
+
Python 检索比较源文件内容摘要,而不只依赖修改时间,覆盖文件追加、修改、删除、改名及目录变化。缺失、损坏或过期索引会尝试重建;构建失败明确返回故障,不以“无命中”替代。指定 `--file` 的现场正文检索仍不依赖一级索引。
|
|
80
|
+
|
|
81
|
+
Node 会使变动后的主题卡、事实与证据失效并重新加载。Node 的近期 stream 窗口与 Python 全历史范围仍有区别;本版没有声称二者排名完全相同。事实版本暂只走关键词通道,独立 Python 向量缓存不会保存或返回它们;Node 的可选向量通道只能在当前有效候选中补充匹配。
|
|
82
|
+
|
|
83
|
+
向量依旧为显式选项。CLI `recall` 默认也只走关键词,需加 `--semantic` 才尝试语义召回。没有为每本工作区另建向量数据库。
|
package/README.md
CHANGED
|
@@ -1,17 +1,19 @@
|
|
|
1
1
|
# dsh-memory-dba
|
|
2
2
|
|
|
3
|
-
[中文功能范围说明](README.zh-CN.md)
|
|
3
|
+
[中文功能范围说明](README.zh-CN.md) · [记忆操作与数据格式](MEMORY.md)
|
|
4
4
|
|
|
5
5
|
File-based long-term memory for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness), stored as plain files you can read, diff, and commit.
|
|
6
6
|
|
|
7
7
|
Session logs answer "what happened in this conversation". This plugin answers "what did we decide three weeks ago" — across sessions, across machines, and across models.
|
|
8
8
|
|
|
9
|
-
This package is the **file-memory and retrieval plugin** of
|
|
9
|
+
This package is the **file-memory and retrieval plugin** of DBA. Version 0.3.0 adds its own complete sanitized text events, recoverable stream publication, topic cards and versioned facts. The original Python RWKV backend and processing pipeline remain separate; no state model is enabled by this change.
|
|
10
10
|
|
|
11
11
|
```
|
|
12
12
|
core.md injected in full every turn, byte-capped
|
|
13
13
|
stream.md one line per finished turn, retrieved only
|
|
14
|
-
ws/<area>.md detail notes,
|
|
14
|
+
ws/<area>.md detail notes; retrieval returns the file name only, body on --file
|
|
15
|
+
catalog.json navigation cards and evidence-backed fact revisions
|
|
16
|
+
events-v1/ complete sanitized text events, read explicitly by ID
|
|
15
17
|
```
|
|
16
18
|
|
|
17
19
|
Everything lives in one directory. Copy it, put it in git, point two harness installs at the same folder, or read it with `cat`. There is no database or required embedding service.
|
|
@@ -23,20 +25,21 @@ Everything lives in one directory. Copy it, put it in git, point two harness ins
|
|
|
23
25
|
| Capability | What this package actually does |
|
|
24
26
|
|---|---|
|
|
25
27
|
| Three-layer file memory | `core.md`, `stream.md`, `ws/<area>.md` under one directory, plain UTF-8, no database. `init` creates the layout; existing files are never overwritten. |
|
|
26
|
-
|
|
|
27
|
-
| BM25 retrieval | BM25 (`k1=1.5`, `b=0.75`)
|
|
28
|
-
| Optional Ollama / bge-m3 semantic recall | Cosine hits above `minScore` from a `<home>/.vec/embeddings.json` cache are appended after keyword hits without reordering the keyword results. Synchronous 1,500 ms budget (`Atomics.wait` on a worker), 60 s cooldown after a failure, silent degradation to BM25. The cache is built by the CLI, not automatically. |
|
|
29
|
-
| DSH session-event integration |
|
|
30
|
-
| Current stream writing |
|
|
28
|
+
| Split prompt injection | `core` is a system-prompt section (default order `-50`), snapshotted once per session. Retrieval `hits` are runtime context (default order `300`), which the harness materializes as a trailing user snapshot. A hit list inside the system prompt would change the head of the request and invalidate every message behind it. |
|
|
29
|
+
| BM25 retrieval | BM25 (`k1=1.5`, `b=0.75`) with CJK unigrams/bigrams. Level 1 searches `stream` text plus **workspace file names, first-line titles and explicit topic-card fields**; `core.md` is injected in full and stays out of the recall corpus, so the same text cannot spend the budget twice (same rule as the Python side). Workspace bodies are not in the corpus at all (see Retrieval). Recency reranking only reorders candidates that relevance already found, and only when they overflow the byte budget. |
|
|
30
|
+
| Optional Ollama / bge-m3 semantic recall | Cosine hits above `minScore` from a `<home>/.vec/embeddings.json` cache are appended after keyword hits without reordering the keyword results. Workspace entries in that cache are name-only documents, so a semantic hit cannot bypass level 1 and surface a body. Synchronous 1,500 ms budget (`Atomics.wait` on a worker), 60 s cooldown after a failure, silent degradation to BM25. The cache is built by the CLI, not automatically. |
|
|
31
|
+
| DSH session-event integration | Persists text drafts, archives completion before publishing the stream, and recovers completed unpublished events on mount. Interrupted records remain inspectable without replaying them as completed turns. |
|
|
32
|
+
| Current stream writing | Bounded head-and-tail excerpts (300/600 characters), with `[event=ID]` pointing to complete sanitized text in `events-v1`. Explicit observation labels are candidates, not automatically verified facts. |
|
|
33
|
+
| Cards and facts | `catalog.json` holds explicit workspace navigation and evidence-backed fact revisions. Superseded/retracted records leave default retrieval; unresolved values are marked as conflicts. See [MEMORY.md](MEMORY.md). |
|
|
31
34
|
|
|
32
35
|
### 2. Exists in the original Python DBA, not migrated into this package
|
|
33
36
|
|
|
34
37
|
| Capability | Original Python DBA | In this package |
|
|
35
38
|
|---|---|---|
|
|
36
39
|
| RWKV-7 continuous session state and summary backend | `scripts/nb_rwkv.py`, `scripts/nb_rwkv_bridge.py`; `scripts/nb_turn.py` uses the local RWKV backend by default for session recursion, summaries, and branch isolation. | No RWKV backend, no state interface, no summary generation. |
|
|
37
|
-
|
|
|
38
|
-
| Persistent outbox
|
|
39
|
-
| Message drafts and interrupted
|
|
40
|
+
| Complete events and event-vector recall | Original complete events and vector recall under `.turns`. | Own complete text archive under `events-v1`; no event vectors or old-archive migration. |
|
|
41
|
+
| Persistent outbox and retries | Original multi-stage pipeline. | Own durable text drafts and recoverable stream-publication queue; no RWKV processing stages. |
|
|
42
|
+
| Message drafts and interrupted turns | Original resume pipeline. | Text drafts persist; interrupted dialogue is not automatically resumed. |
|
|
40
43
|
| Controlled `core` / `ws` writes | Dedicated `remember_core` / `remember_ws` write paths with their own checks. | CLI `write` appends to `stream.md` only; `core.md` and `ws/*.md` are edited with ordinary file tools. |
|
|
41
44
|
| Index-maintenance loop | Chained maintenance for the keyword index, notebook vectors, and event vectors. | BM25 invalidates its corpus cache on file metadata changes; the semantic cache is rebuilt only by a manual `index` run. |
|
|
42
45
|
|
|
@@ -44,9 +47,16 @@ These are real capabilities of the original Python system, not hypothetical futu
|
|
|
44
47
|
|
|
45
48
|
### 3. Research not yet accepted for use in the overall DBA project
|
|
46
49
|
|
|
50
|
+
Two state interfaces were tried, and neither validates the other:
|
|
51
|
+
|
|
52
|
+
- **(A) Memory-token prefix.** A frozen RWKV-7 state passes through an adapter into *k* continuous tokens at the LLM hidden width, which are concatenated in front of the instruction embeddings of a frozen Qwen2.5. Only the adapter and the output head train, on downstream task error — no text is produced in between. This is the interface the architecture-judgment experiment tested: best 0.5477 against a direct-MLP baseline of 0.6774 on 50 sealed test episodes, and the ablation (correct vs wrong-episode vs zero vs random tokens) shows the tokens carry episode-specific information. It is not yet accepted either: the rank among token counts is unsettled, the 64-token seeds are incomplete, and on a small 192k-sample budget memory tokens still trail the MLP baseline — they overtake only after enough training data.
|
|
53
|
+
- **(B) State-to-text slot decoding.** A decoder turns the state into four readable slots and the text is fed onward. All six rounds failed acceptance (best 9/1773 all-slot exact). **Every `0/1779` figure on this page belongs to (B) only and says nothing about (A).**
|
|
54
|
+
|
|
55
|
+
What the DBA runtime actually uses today is a third path: text summaries plus retrieval. It is neither (A) nor (B).
|
|
56
|
+
|
|
47
57
|
| Research item | Actual state |
|
|
48
58
|
|---|---|
|
|
49
|
-
| Decoding RWKV state into
|
|
59
|
+
| Decoding RWKV state into readable text slots (route B) | State-decoder training experiments were performed on frozen RWKV features, but they did not reach the acceptance standard. The v4 decoder run completed 5,000 steps; on dev (1,779 cases) all-slot exact matches were **0/1779** and content-slot accuracy was 1.6%. The state itself was not useless: true-state token loss 1.185 beat no-state 1.575 and wrong-state 1.260, and per-label linear probes on the same pooled features reached AUC 0.98 — the information is present, the joint multi-label decode is what fails. It was judged **Not qualified** and never entered the final test. There is no usable state-to-text decoder today. |
|
|
50
60
|
| Cross-model consumption of state | Cloud models consume text summaries and retrieved text, not RWKV state tensors or dedicated memory codes. The interface is not built. |
|
|
51
61
|
| A genuinely resident service | RWKV runs through a local runner/bridge. Continuity across turns is not the same as an accepted always-on service; no resident deployment has been accepted. |
|
|
52
62
|
|
|
@@ -60,17 +70,26 @@ The old five-layer architecture's periodic consolidation / retire / dedupe path
|
|
|
60
70
|
|
|
61
71
|
Read these before relying on the plugin as shared or historical memory.
|
|
62
72
|
|
|
63
|
-
- **
|
|
64
|
-
- **
|
|
65
|
-
- **
|
|
73
|
+
- **Workspace bodies are not retrievable from this package.** Level 1 returns file names only by design. The second-level read is the Python DBA CLI (`scripts/nb_search.py --file <区名>`), which the hits block names using `<home>/../scripts/nb_search.py` (derived from `home`; override with `detailHint`); an install that does not have the DBA tree can still recall names, but there is no in-plugin command that expands a body. This is a deliberate boundary, not a missing index.
|
|
74
|
+
- **Retrieval text and core snapshots are per-session.** `user/message` stores the hits under `session.id`, and `memory-dba:hits` resolves them with `context.agent.id`; `memory-dba:core` snapshots per `context.agent.id` as well. Two live sessions therefore no longer overwrite each other. Contexts without an agent id (diagnostics, tests, non-agent assembly) cannot be isolated: hits fall back to the latest single slot, and core rereads on every assembly while keeping only a last-read slot. Disposing a session clears its hits and core snapshot; live core snapshots remain until disposal, so memory scales with the number and size of active sessions.
|
|
75
|
+
- **Stream retrieval only covers a recent window.** The index takes at most the most recent **1,000 non-empty stream entries**, within at most **128 KiB** of entry text (an entry larger than 128 KiB is skipped; the scan stops when the byte cap would be exceeded). Older entries remain in `stream.md` but are not in the retrieval candidate set. The 1,000-entry cutoff was exercised in the audit; the 128 KiB cap is enforced in `src/search.mjs` and was **not** independently exercised end-to-end. Move durable facts into `ws/*.md`, which is not capped and is read on demand through the `--file` second pass.
|
|
76
|
+
- **Excerpts are navigation, not semantic summaries.** Full sanitized text is retained in `events-v1`; labelled observations remain candidates. Add facts explicitly with evidence. Old truncated text cannot be reconstructed.
|
|
66
77
|
- **Semantic recall needs manual indexing.** The vector cache is not updated by the plugin while it runs. Run `dsh-memory-dba index` after editing memory; documents that are new or changed are still searchable by BM25 but not by semantics until then.
|
|
67
78
|
- **An over-budget core is clipped, not compressed.** When `core.md` exceeds `coreLimitKB`, the injected section is cut and marked `[...clipped to fit the memory budget]`. The file itself is not rewritten, summarized, or reorganized.
|
|
68
|
-
- **
|
|
79
|
+
- **Recovery requires archival storage.** Completed archived events with failed stream publication are retried on mount. Archive write failure is logged and readable recall remains available. Keep one active writing host per memory directory.
|
|
69
80
|
- **The default memory directory is not the old Python DBA directory.** On Windows it is `%LOCALAPPDATA%\dsh-memory-dba`; elsewhere `~/.dsh-memory-dba`, unless `home` or `DSH_MEMORY_HOME` says otherwise. To reuse an existing Python DBA memory library, set `home` explicitly (for example `D:/AI/DBA/memory`).
|
|
70
81
|
|
|
71
82
|
## Release notes
|
|
72
83
|
|
|
73
|
-
`0.
|
|
84
|
+
`0.3.0` adds complete text archives, explicit navigation cards, evidence-backed fact corrections and Python content-manifest freshness checks. See [MEMORY.md](MEMORY.md). Vector recall stays opt-in.
|
|
85
|
+
|
|
86
|
+
`0.2.2` makes retrieval state match the harness's session model. `core.md` is snapshotted per session (`coreRefresh: 'session'`, or `'always'` to opt out), retrieval hits are keyed by session id instead of one process-wide slot, and an empty recall contributes a cold-path hint naming the Python index (`coldHint`, `'off'` to disable). Semantic recall is now **off by default** (`vector.enabled: false`) because the name-only level-1 corpus gave it 0/143 useful non-core coverage while costing up to 1500 ms and a GPU; set it to `true` to opt back in. See `CHANGELOG.md`.
|
|
87
|
+
|
|
88
|
+
`0.2.1` keeps `core.md` out of the recall corpus. Core is still injected in full as a system-prompt section, but retrieval no longer indexes it, so the same text cannot consume the prompt budget twice; this matches the Python DBA side, which excludes `core` by default. See `CHANGELOG.md`.
|
|
89
|
+
|
|
90
|
+
`0.2.0` changes retrieval behavior: workspace layers are indexed as names plus first-line titles and rendered without bodies; the body read moves to an explicit `--file` second pass in the Python DBA CLI. See `CHANGELOG.md`.
|
|
91
|
+
|
|
92
|
+
`0.1.2` through `0.1.5` are documentation-only revisions. No runtime source file or behavior changed in either release; the changes are the scope tables, the limits above, and the corrected release history, so that the npm page matches what is actually delivered. See `CHANGELOG.md` for the full history.
|
|
74
93
|
|
|
75
94
|
## Install
|
|
76
95
|
|
|
@@ -95,9 +114,9 @@ Then append one entry to the `insert:` list in that profile's `cordis.patch.yml`
|
|
|
95
114
|
home: '~/.dsh-memory-dba'
|
|
96
115
|
```
|
|
97
116
|
|
|
98
|
-
Omit `config` entirely to use the defaults. Restart the harness process after adding the entry; a profile reload is not enough because the prompt
|
|
117
|
+
Omit `config` entirely to use the defaults. Restart the harness process after adding the entry; a profile reload is not enough because the prompt section and runtime context are registered at startup.
|
|
99
118
|
|
|
100
|
-
The plugin hooks
|
|
119
|
+
The plugin hooks one prompt section, one runtime context, and the session event stream. Nothing else is required.
|
|
101
120
|
|
|
102
121
|
## Configuration
|
|
103
122
|
|
|
@@ -105,33 +124,51 @@ The plugin hooks two prompt sections and the session event stream. Nothing else
|
|
|
105
124
|
|---|---|---|
|
|
106
125
|
| `home` | platform default | Memory directory. `DSH_MEMORY_HOME` overrides the default when `home` is unset. |
|
|
107
126
|
| `coreLimitKB` | `15` | Byte cap for the core section. Text past the cap is clipped and marked. |
|
|
108
|
-
| `hitBudgetKB` | `6` | Byte cap for the retrieval
|
|
127
|
+
| `hitBudgetKB` | `6` | Byte cap for the retrieval runtime context. |
|
|
109
128
|
| `topK` | `5` | Maximum retrieved notes per turn. |
|
|
110
129
|
| `timeDecay` | `true` | When the candidate set overflows the budget, rerank with a recency weight. |
|
|
111
130
|
| `halfLifeDays` | `30` | Recency weight halves every N days, floor 0.5. |
|
|
112
131
|
| `writeStream` | `true` | Append one line per finished turn to `stream.md`. |
|
|
113
132
|
| `tag` | `''` | Optional label written into stream lines. |
|
|
114
|
-
| `exclude` | `[]` | Workspace file names
|
|
115
|
-
| `
|
|
116
|
-
| `
|
|
117
|
-
| `
|
|
133
|
+
| `exclude` | `[]` | Workspace file names kept out of retrieval entirely, e.g. role-play notes. Extra guardrail only: the real gate is that level 1 indexes workspace names, never bodies. Matches by file name, so a rename requires updating this list; the Python DBA's `nb_doctor` flags an entry that no longer exists. |
|
|
134
|
+
| `detailHint` | `''` | Full text of the level-1 workspace hint. Empty derives `py -3.14 <home>/../scripts/nb_search.py "查询" --file <区名>` from `home`; a non-empty value is used verbatim (non-Windows installs, another Python launcher). |
|
|
135
|
+
| `coldHint` | `''` | Text contributed when level-1 recall has no hits. Empty derives a sentence naming the full Python index; `'off'` (or an explicit empty value/`false`) disables it. A non-empty string replaces the whole sentence. |
|
|
136
|
+
| `coreRefresh` | `'session'` | `'session'` snapshots `core.md` on the first assembly of each session so the cacheable head stays stable; `'always'` re-reads on every assembly when edits must apply immediately. |
|
|
137
|
+
| `coreOrder` | `-50` | System-prompt section order for the core layer. |
|
|
138
|
+
| `hitsOrder` | `300` | Runtime-context order for retrieval hits; the harness's own contexts use 100–210. |
|
|
139
|
+
| `vector.enabled` | `false` | Optional semantic recall. Off by default: level 1 is already name-only, and the synchronous worker measured 0/143 non-core coverage against a 1500 ms/GPU cost. Set `true` and build the cache to opt in. |
|
|
118
140
|
| `vector.url` | `http://127.0.0.1:11434` | Ollama embedding server. |
|
|
119
141
|
| `vector.model` | `bge-m3` | Model used for indexing and queries. |
|
|
120
142
|
| `vector.timeoutMs` | `1500` | Total synchronous worker-join budget in milliseconds; also caps the embedding request. |
|
|
121
143
|
| `vector.minScore` | `0.45` | Cosine floor for an appended semantic hit. |
|
|
122
144
|
| `vector.keepAlive` | `30m` | Passed to Ollama so the model stays resident between turns. |
|
|
123
145
|
|
|
124
|
-
Partial `vector` objects inherit unspecified defaults.
|
|
146
|
+
Partial `vector` objects inherit unspecified defaults. Semantic recall is **off by default**; set `vector: { enabled: true }` and run the `index` command to opt in.
|
|
147
|
+
|
|
148
|
+
## Why the two placements differ
|
|
125
149
|
|
|
126
|
-
|
|
150
|
+
The core layer changes every few days; retrieval hits change on every user turn. Both are model-visible, but only one of them may sit in the system prompt.
|
|
127
151
|
|
|
128
|
-
|
|
152
|
+
A system-prompt section is the first message of every request, so any change to it moves the head of the prompt: prefix caching then misses from that point on, and every message after the system prompt — the entire conversation — is re-sent uncached. Runtime context does not have that cost. The harness materializes it as a user-role snapshot appended after the conversation, so replacing it invalidates only the tokens that follow it, which for a per-turn block is the turn itself.
|
|
129
153
|
|
|
130
|
-
Core text is
|
|
154
|
+
Core text is read on the first assembly of each session and replayed from that session's snapshot, so `core.md` edits take effect in a new session without paying a disk read on every assembly. Snapshots remain until `session/disposed` and the cache therefore grows with the number and size of live session snapshots; disposal releases each one. Set `coreRefresh: 'always'` when edits must be visible immediately inside a live session. Without a session id (diagnostics, tests, non-agent assembly), the plugin rereads every time and keeps only a single last-read slot.
|
|
131
155
|
|
|
132
156
|
## Retrieval
|
|
133
157
|
|
|
134
|
-
BM25 over
|
|
158
|
+
BM25 over `stream.md` and the **names** of `ws/*.md`, tokenized for mixed Chinese/English text: Latin runs stay whole words (so identifiers such as `1234567890` remain searchable), CJK runs emit unigrams and bigrams (a trigram index cannot match a two-character Chinese query).
|
|
159
|
+
|
|
160
|
+
`core.md` is **not** in the recall corpus: it is injected in full every turn by `buildCore`, so recalling it as well would let the same text consume the prompt budget twice. The Python side (`nb_search.py`) follows the same rule by excluding `core` by default; this package matches it.
|
|
161
|
+
|
|
162
|
+
Workspace retrieval is deliberately **two-level**:
|
|
163
|
+
|
|
164
|
+
1. The level-1 corpus contains one document per workspace file, and that document is only `ws/<area>.md` plus the file's first line when it is a `# ...` heading. The body is never indexed. A hit renders as `- [ws/记忆系统.md]` with no text, and the hits block adds one hint: read the body with `py -3.14 <home>/../scripts/nb_search.py "查询" --file <区名>`. The path is derived from the configured `home` at mount time; set `detailHint` to replace the whole line.
|
|
165
|
+
2. Reading the body is an explicit second step, done outside the plugin with the Python CLI's repeatable `--file` flag. That command re-reads the file, splits it on `##`, scores the sections with the same BM25, and returns the text. There is no second content index to drift out of sync with the file. Its human output always states that it is level 2 and lists the files searched, even for zero hits, so "searched this file and found nothing" cannot look like "the flag did not take effect".
|
|
166
|
+
|
|
167
|
+
Why not just index the body: recall runs on **every** user message, and a workspace section is long. Any unrelated query could then pull a whole section into the prompt; in the DBA tree this actually happened with `ws/角色卡.md` (role-play notes recalled by an unrelated query). Level 1 answers "which note", level 2 answers "what it says", and only an explicit `--file` unlocks the second answer.
|
|
168
|
+
|
|
169
|
+
`exclude` remains supported (`exclude: ['角色卡.md']`), but it is an **extra guardrail**, not the gate. It matches by file name, so renaming a workspace file requires updating the list; the Python DBA's `nb_doctor` flags an entry that no longer exists on disk. The gate is that level 1 has no body to leak: even if an entry were missing from `exclude`, the corpus still contains only names and titles. Conversely, relying on `exclude` while indexing bodies means every file not on the list can still be recalled in full.
|
|
170
|
+
|
|
171
|
+
The semantic cache follows the same rule because it is built from the same corpus: workspace entries are name-only texts, so a semantic hit is rendered as `- [ws/<area>.md] (semantic)` and cannot surface a body. `core` is not in the corpus at all, so the semantic channel cannot score the injected text a second time; `stream` is unchanged.
|
|
135
172
|
|
|
136
173
|
Ranking is two-step:
|
|
137
174
|
|
|
@@ -140,15 +177,19 @@ Ranking is two-step:
|
|
|
140
177
|
|
|
141
178
|
Recency never introduces a candidate that relevance did not find.
|
|
142
179
|
|
|
143
|
-
Core
|
|
180
|
+
Core is not indexed here at all (see above); stream entries split per non-empty line. Workspace files are not split at level 1 at all — one name-only document per file — and reported line numbers for them are `1`. Workspace additions, edits, removals, and renames invalidate the cached corpus using file names, sizes, and modification times.
|
|
144
181
|
|
|
145
|
-
For predictable stream retrieval latency, only the most recent **1,000 non-empty stream entries**, within **128 KiB of entry text**, are indexed. Individual entries larger than 128 KiB are skipped. Earlier entries remain on disk but are outside the candidate set; move lasting details into `ws/*.md` to keep them searchable.
|
|
182
|
+
For predictable stream retrieval latency, only the most recent **1,000 non-empty stream entries**, within **128 KiB of entry text**, are indexed. Individual entries larger than 128 KiB are skipped. Earlier entries remain on disk but are outside the candidate set; move lasting details into `ws/*.md` to keep them searchable. Workspace documents are not capped. A 5 MiB mixed Chinese/English stream is covered by a cold-load-plus-search benchmark test targeting 300 ms; actual timing depends on hardware. Reading and UTF-8 validation still scan the whole stream, so this is not a constant-time guarantee for arbitrarily large files.
|
|
146
183
|
|
|
147
184
|
If the first query finds no shared term, recall retries once with CJK unigrams plus Latin words. Existing shared-term hits are never relaxed. The result reports `relaxed: true` only when the retry supplies hits; there are at most two ranking attempts.
|
|
148
185
|
|
|
186
|
+
When level-1 recall returns no hits, the runtime context contributes one cold-path line instead of an empty block: it says so and names the Python CLI that searches the full history (`py -3.14 <home>/../scripts/nb_search.py "查询"`). A non-empty hit list never carries that line. `coldHint` overrides the sentence; `'off'` or an explicit empty value disables it.
|
|
187
|
+
|
|
149
188
|
## Semantic recall (optional)
|
|
150
189
|
|
|
151
|
-
|
|
190
|
+
Semantic recall is **off by default**. Level 1 is already narrow — core is injected in full, workspace documents are names only, and stream entries are short lines — while the synchronous worker waits up to 1500 ms and needs a GPU. On the current corpus it covered **0/143** non-core documents (core 67/74, stream 0/124, workspace 0/19), and core has since left the recall corpus entirely. The Python DBA CLI is the better path for semantic search over full text. To opt in on a machine with a warm `bge-m3`, set `vector: { enabled: true }`, then build the derived cache:
|
|
191
|
+
|
|
192
|
+
BM25 always works on a bare Node install. Install an Ollama embedding model separately, then build the derived cache:
|
|
152
193
|
|
|
153
194
|
```bash
|
|
154
195
|
ollama pull bge-m3
|
|
@@ -159,11 +200,11 @@ Indexing uses four concurrent requests. Duplicate document texts share one entry
|
|
|
159
200
|
|
|
160
201
|
The portable `<home>/.vec/embeddings.json` cache stores version, model, dimension, and SHA-1 document-text keys, never document paths. Each vector is quantized to signed int8 with its own `max(abs(vector)) / 127` scale and base64 bytes. `bge-m3` normally uses 1024 dimensions; the cache records the actual model dimension.
|
|
161
202
|
|
|
162
|
-
On each user message, a lazily reused, unreferenced worker embeds the query using `/api/embed` (falling back once to `/api/embeddings` on HTTP 404), then compares it with cached vectors by cosine similarity. Only document keys are sent, not corpus text. A fresh shared buffer returns at most eight matching corpus indices and scores. The main thread joins synchronously using `Atomics.wait`, with a **1500 ms total budget**, including worker startup and message posting. Prompt
|
|
203
|
+
On each user message, a lazily reused, unreferenced worker embeds the query using `/api/embed` (falling back once to `/api/embeddings` on HTTP 404), then compares it with cached vectors by cosine similarity. Only document keys are sent, not corpus text. A fresh shared buffer returns at most eight matching corpus indices and scores. The main thread joins synchronously using `Atomics.wait`, with a **1500 ms total budget**, including worker startup and message posting. Prompt callbacks remain synchronous; no background result mutates a later turn's hits.
|
|
163
204
|
|
|
164
205
|
Keyword ranking is never rewritten by semantics. Hits above `minScore` that keywords missed are **appended** after the keyword hits, marked `(semantic)`, and cut to a 600-byte excerpt; the prompt budget keeps at most 70% for keyword hits so a long workspace section cannot crowd them out entirely. Fusion of the two rankings was tried and rejected: with no score floor, a semantic table of contents outranked an exact keyword match.
|
|
165
206
|
|
|
166
|
-
A cold `bge-m3` takes about two seconds to load, longer than the join budget, so the plugin fires one background warmup request
|
|
207
|
+
A cold `bge-m3` takes about two seconds to load, longer than the join budget, so when semantic recall is enabled the plugin fires one background warmup request on mount and asks Ollama to keep the model resident for `keepAlive`. After a failed or timed-out lookup the endpoint is skipped for 60 seconds, which keeps a down or cold embedder from costing the budget on every turn. When `vector.enabled` is false (the default), neither the warmup nor the worker path runs.
|
|
167
208
|
|
|
168
209
|
Missing caches, unavailable models (including Ollama HTTP 404), network errors, model/dimension mismatches, corrupt caches, and timeouts silently degrade to BM25 during recall. No model is installed automatically. `doctor` reports vector entry counts and corpus coverage; a missing cache is optional even without an embedder, while a corrupt cache fails the vector check. `stats` includes cache bytes when present.
|
|
169
210
|
|
|
@@ -178,7 +219,7 @@ dsh-memory-dba stats # bytes per layer
|
|
|
178
219
|
dsh-memory-dba doctor # layout, budget, corpus, retrieval checks
|
|
179
220
|
```
|
|
180
221
|
|
|
181
|
-
Useful while tuning `coreLimitKB` and inspecting why a note did or did not surface. The CLI is an operator tool; the plugin does not need it at runtime.
|
|
222
|
+
Useful while tuning `coreLimitKB` and inspecting why a note did or did not surface. The CLI is an operator tool; the plugin does not need it at runtime. `recall` is level 1 only — it prints `ws/<file>` for workspace hits; the body is read with the Python DBA CLI's `--file` second pass.
|
|
182
223
|
|
|
183
224
|
## Writing memory
|
|
184
225
|
|
package/README.zh-CN.md
CHANGED
|
@@ -1,30 +1,44 @@
|
|
|
1
1
|
# dsh-memory-dba:功能范围
|
|
2
2
|
|
|
3
|
-
[English README](README.md) · 版本 0.
|
|
3
|
+
[English README](README.md) · [记忆操作与数据格式](MEMORY.md) · 版本 0.3.0
|
|
4
4
|
|
|
5
5
|
本包是 DBA(Dual-Brain Architecture,双脑架构)的**文件记忆与检索插件**,不是完整双脑运行时。安装它不会同时安装 RWKV 小脑、连续状态后端或原 Python DBA 的可靠事件处理链。
|
|
6
6
|
|
|
7
|
-
0.
|
|
7
|
+
0.3.0:新增完整脱敏文本事件、可恢复流水发布、主题导航卡和带证据的事实纠正关系;Python 查询前按内容摘要检查索引新鲜度。向量仍默认关闭。详见 [MEMORY.md](MEMORY.md)。
|
|
8
|
+
|
|
9
|
+
0.2.2:core 改为按会话快照(`coreRefresh` 默认 `session`,`always` 可恢复逐次读盘);召回块按会话 id 隔离;空命中时给一行「全量历史去 Python 索引查」的冷提示(`coldHint` 可覆盖/关闭);语义召回默认关闭(`vector.enabled: false`,理由见下)。0.2.1 让 `core.md` 退出召回语料:它仍每轮全量注入,但不再被检索索引,同一份内容不会既注入又召回(与 Python 侧一致)。0.2.0 是检索行为改动:工作区册在**一级索引里只有文件名 + 首行标题**,命中只渲染 `- [ws/<区>.md]`;要看正文必须用 Python DBA 的 `nb_search.py --file <区名>` 显式二次检索。0.1.2–0.1.5 是说明文档修订版;0.1.6 把召回块从系统提示词中段移到会话尾部,命中率不再被它拖垮。下列尚未实现的能力不受影响。
|
|
8
10
|
|
|
9
11
|
## 本 npm 包已实现
|
|
10
12
|
|
|
11
13
|
| 能力 | 实际行为 |
|
|
12
14
|
| --- | --- |
|
|
13
15
|
| 三层文件记忆 | core.md、stream.md、ws/*.md;可直接阅读和迁移 |
|
|
14
|
-
|
|
|
15
|
-
| BM25 关键词检索 |
|
|
16
|
-
|
|
|
16
|
+
| 分段提示词注入 | 核心记忆是系统提示词的一段(默认 order -50,可缓存的前缀);本轮召回是运行时上下文(默认 order 300),被宿主渲染成会话尾部的用户快照。召回若放进系统提示词,每轮都会改掉请求的头,把它后面的整段对话全部作废。默认预算分别为 15 KiB 和 6 KiB。core 与召回都按 `context.agent.id` 分会话:core 首次装配时快照、召回按会话取;拿不到 id 时退回逐次读盘/单槽 |
|
|
17
|
+
| BM25 关键词检索 | 支持中英文混合、中文单字和双字;预算溢出时可按时间重排。一级 0 命中时给一行冷提示,指向 Python 全量索引(`coldHint` 可覆盖/关闭) |
|
|
18
|
+
| 工作区册两级检索 | 一级索引「文件名 + 首行标题 + 显式主题卡导航字段」,正文绝不进索引;命中渲染成 `- [ws/<区>.md]`,并提示用 Python DBA 的 `<home>/../scripts/nb_search.py "查询" --file <区名>` 二次检索正文(整句可用 `detailHint` 覆盖)。语义通道同样只拿到文件名,不能绕过一级 |
|
|
19
|
+
| 可选语义召回 | **默认关闭**(`vector.enabled: false`):一级已只剩短行/文件名,同步 worker 最长等 1500ms 且依赖 GPU,实测非 core 覆盖仅 0/143;需要时显式开启并手动建向量缓存。语义检索建议优先用 Python CLI 的全量语料 |
|
|
17
20
|
| DSH 会话接入 | 监听用户/助手消息,在轮次结束时追加记忆流 |
|
|
18
|
-
| 当前记忆流写法 |
|
|
21
|
+
| 当前记忆流写法 | 首尾摘录保留用户 300/助手 600 字符,含 `[event=ID]`;完整脱敏文本保存在 `events-v1`,可显式回读。明确标记的观察项仅作候选,不自动成为有效事实 |
|
|
22
|
+
| 主题卡与事实目录 | `catalog.json` 导航字段帮助选册;事实带事件来源、纠正关系和撤回状态,同键冲突明确标记 |
|
|
23
|
+
|
|
24
|
+
### 为什么工作区册默认不索引正文
|
|
25
|
+
|
|
26
|
+
这是**召回边界**,不是省空间:检索每轮都跑,而一本册的正文动辄几千字。全文入索引时,任何一次闲聊查询都可能把某段正文整段拉进 prompt —— DBA 仓库实测过 `ws/角色卡.md` 被无关查询召回。一级只回答「该看哪本册」,二级用 `--file` 显式指定文件才回答「正文说什么」。二级现场读文件、按 `##` 切段、复用同一套 BM25,不再建第二份内容索引:两份内容索引早晚不同步,而「同一段记忆在两处内容不同」比没有索引更难查。
|
|
27
|
+
|
|
28
|
+
`core.md` 不进召回语料,只作全量注入(与 Python 侧一致):同一份内容既每轮全量注入、又作为命中重复召回,等于白占注入预算;语义通道同样不索引它。
|
|
29
|
+
|
|
30
|
+
提示里的脚本路径从 `home` 推导(`<home>/../scripts/nb_search.py`),不是写死的绝对路径;记忆树搬走或换机器后提示仍指向实际位置。非 Windows / 其他 Python 启动器用配置项 `detailHint` 覆盖整句提示。
|
|
31
|
+
|
|
32
|
+
`exclude` 仍然保留(按文件名匹配,如某本角色设定册),但它只是**额外护栏**:按文件名匹配意味着改名后必须同步,否则排除会静默失效;DBA 的 `nb_doctor` 会抓「条目指向盘上不存在的文件」。真正的闸门是一级索引里根本没有正文。只靠排除表而全文入索引,任何没被列入的册子仍可被整段召回。
|
|
19
33
|
|
|
20
34
|
## 原 Python DBA 已有,但本包尚未迁入
|
|
21
35
|
|
|
22
36
|
| 能力 | 本包现状 |
|
|
23
37
|
| --- | --- |
|
|
24
38
|
| RWKV-7 会话连续状态、恢复与摘要后端 | 未接入 |
|
|
25
|
-
|
|
|
26
|
-
|
|
|
27
|
-
|
|
|
39
|
+
| 原 Python 事件向量召回 | 未迁入;本包已有独立的完整文本档案 `events-v1`,不自动迁移旧 `.turns` 档案 |
|
|
40
|
+
| 原 Python 多阶段 outbox/RWKV 重试 | 未迁入;本包已有独立的完成事件待发布队列,挂载时恢复流水发布 |
|
|
41
|
+
| 原 Python 消息恢复协议 | 本包已持久保存文本草稿;中断轮次不自动续跑或伪装成完成 |
|
|
28
42
|
| core/ws 受控写入接口 | 未迁入;CLI write 只写 stream,core/ws 依赖普通文件操作 |
|
|
29
43
|
| 原系统的关键词、记事本向量与事件向量维护链 | 未完整迁入;本包语义缓存须手动执行 index |
|
|
30
44
|
|
|
@@ -32,7 +46,14 @@
|
|
|
32
46
|
|
|
33
47
|
## 整体 DBA 仍在研究、未达验收标准
|
|
34
48
|
|
|
35
|
-
|
|
49
|
+
试过两种状态接口,两者互不构成证据:
|
|
50
|
+
|
|
51
|
+
- **(A)记忆 Token 前缀**:冻结的 RWKV-7 状态经适配器变成 k 个连续 token(宽度等于 LLM 隐层),直接拼在冻结 Qwen2.5 的指令嵌入前面;只训练适配器和输出头,用下游任务误差反传,**中间不产生任何文本**。这是架构判定实验验证的路线:50 个密封测试 episode 上最佳 0.5477,对直接 MLP 基线 0.6774;消融(正确 / 错误 episode / 置零 / 随机 token)证明 token 携带 episode 特异信息。它也还没验收:档位名次未定、64 Token 种子不全,而且小预算(192k 样本)下记忆 Token 仍落后 MLP,训够样本才反超。
|
|
52
|
+
- **(B)状态解码为文本槽**:把状态解成四个可读槽位再往下喂。六轮全部未达验收(最好 9/1773 全槽全对)。**本页所有 0/1779 只属于(B),不能用来评价(A)。**
|
|
53
|
+
|
|
54
|
+
DBA 运行时目前实际走的是第三条:文本摘要 + 检索,既不是(A)也不是(B)。
|
|
55
|
+
|
|
56
|
+
- **RWKV 状态可靠解码为文本槽(路线 B)**:做过基于冻结 RWKV 特征的解码器训练实验。v4 完成 5000 步,验证集 1779 例里全槽精确命中 0/1779、内容槽准确率 1.6%,未通过验收,也未进入最终测试。但状态本身不是没用:真实状态 token loss 1.185 优于无状态 1.575、错配状态 1.260,同一池化特征上的逐标签线性探针 AUC 达 0.98 —— 信息在,失败的是联合多标签解码。这不等于 RWKV 主干微调已成功或已有可用状态接口。
|
|
36
57
|
- **跨模型状态消费**:云端目前消费文本摘要与检索原文,不是直接接收 RWKV 状态张量或专用记忆码。
|
|
37
58
|
- **经验证的模型常驻服务**:跨轮状态连续与进程/权重始终常驻是两回事,原系统也没有完成常驻部署验收。
|
|
38
59
|
|
|
@@ -40,12 +61,13 @@
|
|
|
40
61
|
|
|
41
62
|
## 当前限制
|
|
42
63
|
|
|
43
|
-
1.
|
|
64
|
+
1. **按会话隔离,但拿不到 agent.id 时仍有共享槽**:召回按 `session.id` 存、按 `context.agent.id` 取;core 首次装配按 id 快照,并保留到 `session/disposed`。快照内存随仍存活的会话数及 core 大小增长;会话释放时清理。拿不到 id 的装配(子代理、测试、非 agent 装配)只能退回:召回用最后一条单槽,core 每次重新读盘。两个带 id 的会话不会再互相覆盖。
|
|
44
65
|
2. **旧流水可能搜不到**:仅最近 1000 条、合计最多 128 KiB 的非空流水记录进入检索;更旧的内容仍在文件里,却不在候选集中。1000 条窗口有隔离测试依据;128 KiB 上限来自源码,未单独进行端到端边界测试。长期细节应放入 ws。
|
|
45
|
-
3.
|
|
46
|
-
4.
|
|
47
|
-
5.
|
|
66
|
+
3. **摘录不等于语义摘要**:流水首尾摘录用于导航,完整脱敏文本在事件档案;旧流水被截掉的内容无法恢复。核心记忆仍只按预算裁切。
|
|
67
|
+
4. **语义缓存不会自动更新,且默认不启用**:先设 `vector: { enabled: true }`,编辑记忆后还需执行 index。未建立缓存、模型不可用或超时,会降级为 BM25;运行时不会主动提示所有降级原因,可用 doctor 检查缓存。
|
|
68
|
+
5. **归档成功是恢复前提**:已归档的完成事件可重试发布流水;磁盘无法写入档案时仍无法保证持久化,会记录错误。脱敏沿用启发式规则。
|
|
48
69
|
6. **默认目录与旧 DBA 不同**:Windows 默认 `%LOCALAPPDATA%/dsh-memory-dba`,其他平台默认 `~/.dsh-memory-dba`。要复用原库,须明确设置 home;仅安装插件不会自动找到旧 Python 仓库。
|
|
70
|
+
7. **工作区册正文不在本包检索范围内**:一级只给文件名,读正文要用 Python DBA 的 `<home>/../scripts/nb_search.py --file`(路径由 `home` 推导,`detailHint` 可覆盖)。没装 DBA 树时插件仍能召回名字,但不能展开正文——这是设计边界,不是索引缺失。
|
|
49
71
|
|
|
50
72
|
## 安装与配置
|
|
51
73
|
|
|
@@ -60,3 +82,5 @@
|
|
|
60
82
|
```
|
|
61
83
|
|
|
62
84
|
完整安装步骤、配置字段、向量索引与 CLI 用法见 [English README](README.md)。新增插件后需重启宿主。本轮发布不会替换用户已运行的插件或移动既有记忆。
|
|
85
|
+
|
|
86
|
+
新增配置:`coreRefresh: 'session' | 'always'`(默认 session,按会话快照 core;always 恢复逐次读盘)、`coldHint`(默认空=自动生成冷提示;`'off'` 或显式空值关闭;其他字符串覆盖整句)、`vector.enabled`(默认 false,显式 true 才启用语义召回)。
|