@kenz1117/dsh-engram 0.9.5 → 0.9.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.en.md CHANGED
@@ -77,7 +77,7 @@ Sensory and emotional dimensions (smell, temperature, emotional weight) are deli
77
77
 
78
78
  - **Cross-session memory**: a user memory profile is injected at session start (entry cap plus token budget, both configurable; a curated profile maintained via `engram_profile_edit` takes precedence over derived facts), so the agent naturally knows who you are and what you are building; tools recall facts across sessions.
79
79
  - **Dual-scope stores**: `user.db` shared globally; `project-<hash>.db` isolated per git origin identity (falls back to a working-directory encoding without git; legacy stores migrate automatically) — personal preferences follow the person, project conventions follow the repo.
80
- - **Hybrid retrieval**: FTS5 (unicode61 + Chinese 2-gram pre-tokenization) fused with local vectors (`Xenova/bge-small-zh-v1.5`, 512-dim, q8) via RRF, plus one-hop expansion over relation edges and a multiplicative recency/proof ranking boost; the embedding model runs offline, and a failed download degrades to keyword-only retrieval with an explicit marker.
80
+ - **Hybrid retrieval**: FTS5 (unicode61 + Chinese 2-gram pre-tokenization) fused with local vectors (by default `Xenova/paraphrase-multilingual-MiniLM-L12-v2`, 384-dim, q8; model and pooling are configurable — see below) via RRF, plus one-hop expansion over relation edges and a multiplicative recency/proof ranking boost; the embedding model runs offline, and a failed download degrades to keyword-only retrieval with an explicit marker.
81
81
  - **Memory-palace information architecture** (v0.7.2+): all four principles live on the core path, not in display-layer paint — **location as index** (writes sort by kind into rooms and pin a `room#slot` coordinate; room capacity 9, overflow opens a new room, slots are never recycled); **fixed route as order** (`tour_routes` is append-only; `engram_tour mode=fixed` walks the whole palace in slot order); **skeleton reused long-term** (a topic always lands in the same room and the same index, so recall is sequential extraction rather than fresh search); **every marker unique** (placard rule scored 0-1: globally unique +0.4 / date anchor +0.3 / no 6-char prefix clash within the room +0.3; low scores enter the refurb list; a palace with fewer than 8 active memories is left unscanned to avoid small-store noise).
82
82
  - **Corridor-routed retrieval**: the profile carries a room directory, and `engram_search` accepts a `room` parameter — decide the room first, then search inside it, instead of always running whole-store RRF. The top 5 hits carry neighbouring slot ids from the same room as encoding-specificity cues.
83
83
  - **Retrieval-practice loop** (spaced repetition): `engram_review_queue` returns palace coordinates and placard cues but **never the body text**, forcing the model to recall first; `engram_review` reveals the entry and `engram_report grade` (0-5) self-rates it, advancing an SM-2 schedule (1 → 6 → round(prev × ease) days, reset on failure, ease floor 1.3). Entries under review scheduling **no longer take part in automatic decay** — their fate is decided by recall. Session-start injection reports how many memories are due.
@@ -141,6 +141,18 @@ Optional configuration (cordis.yml):
141
141
  assessReminder: true # evidence-gate nudge: inject an engram_assess reminder before the next step when unassessed search batches exist
142
142
  modelCacheDir: '~/.dsh/engram/models' # embedding model cache directory
143
143
  hfEndpoint: 'https://huggingface.co' # model download endpoint; set a mirror behind restricted networks
144
+ # Embedder: omitting this whole block means local + the default model; `kind` picks the local model or a remote endpoint
145
+ embedder:
146
+ kind: 'local' # local (transformers.js, offline inference) | remote (OpenAI-compatible /embeddings)
147
+ localModel: 'Xenova/paraphrase-multilingual-MiniLM-L12-v2' # 384-dim; the previous default Xenova/bge-small-zh-v1.5 is 512-dim
148
+ localPooling: 'mean' # mean (most sentence-embedding models) | cls (bge-m3 and the XLM-R family)
149
+ # kind: 'remote'
150
+ # model: 'text-embedding-3-small' # remote: model name, passed through to the server
151
+ # baseUrl: 'https://api.example.com/v1' # remote: root address including the version
152
+ # apiKeyEnv: 'EMBEDDING_API_KEY' # remote: read the key from this environment variable (wins over a literal apiKey)
153
+ # batchSize: 32 # remote: items per batch
154
+ # timeoutMs: 30000 # remote: per-request timeout in ms
155
+ # Changing the model means re-embedding: a dimension that does not match the store is refused on write and on comparison (EMBEDDER_DIMENSION_MISMATCH) rather than truncated into a plausible-looking similarity
144
156
  ingest: 'off' # automatic capture: off | light (user messages only, ≤2/turn) | eager (assistant messages too, ≤5/turn)
145
157
  reflect: 'off' # automatic belief consolidation: off | suggest (only enqueue on session end; drain via engram_reflect) | auto (consolidate in the background after the session); engram_reflect stays manually callable in every mode
146
158
  # provider and model must be given as a pair: aux-LLM route override for capture/distill (parsed from the session log by default)
@@ -195,7 +207,7 @@ Session agent Host half (Node)
195
207
  - **Palace structure (the directory is the room, the path is the route)**: the **grand hall** is the standing core profile injected every turn (few and stable, always present); the **corridor** is the room directory in the profile plus the `engram_search room` parameter (pick the room, then search); **rooms** sort by kind (Fact Hall / Preference Pavilion / Decision Chamber / Episode Gallery / Skill Workshop) with capacity 9, overflowing into `<name>-2`; **placards** are each memory's `placard` marker, scored on the unique · distinctive · dated rule. Existing stores are slotted on first open (idempotent; opening a new room logs a warning so a human can name it).
196
208
  - **Automatic capture** (when `ingest` is on): each new turn's first step extracts candidate facts from the previous turn out of the session log; session end (`session/disposed`) captures the final turn with a 5-second timeout, and on failure/timeout a pending key lands in the operation log for the next session's first step to replay; captured (session, turn) pairs are idempotent. The read source is the session log; aux-call request auditing goes to the plugin's own operation log — unknown events are never appended to the session log. Candidates are written with low confidence and deduplicated by embedding.
197
209
  - **Provenance chain**: every memory records its source session, turn, and event seq, fully traceable via `engram_review`; the operation log table records every write/edit/forget/distill/decay.
198
- - **Offline embeddings**: the model downloads once (q8, ~50MB; mirror endpoint configurable), then runs fully offline; on failure the plugin keeps working and retrieval degrades to keyword-only with an explicit marker.
210
+ - **Offline embeddings**: the model downloads once (q8, ~50MB; mirror endpoint configurable), then runs fully offline; on failure the plugin keeps working and retrieval degrades to keyword-only with an explicit marker. **The dimension is a hard constraint**: when the stored vectors and the current model disagree, writes and similarity comparisons are both refused with `EMBEDDER_DIMENSION_MISMATCH` (naming both dimensions) instead of being truncated into a plausible-looking similarity.
199
211
  - **UI localization**: the client half registers zh/en dictionaries through the host locale service and follows the host language setting live; data-level enums (status/kind) are mapped only in the display layer and stay English in storage.
200
212
 
201
213
  ## Web Management Panel
@@ -260,7 +272,7 @@ Profile text changes as the memory store changes — changes only land at turn b
260
272
  ## Known Limitations and Deferred Work
261
273
 
262
274
  - **Contradiction checks are LLM-free by default** — writes report candidates by vector similarity (≥0.88) and create edges; semantic-contradiction confirmation is left to model/user adjudication and distillation. Optional Jev adjudication (the panel's "Jev" tab or the yml `jev` block) routes the fuzzy band and contradiction edges to an external judge, falling back to this default whenever Jev is unavailable.
263
- - **Memories written during embedder degradation have no vectors** — memories written before the model is ready do not participate in the semantic track; after semantics come online run `pnpm backfill` once to backfill existing vectors (after `pnpm build` has warmed the model cache; `HF_ENDPOINT` configurable).
275
+ - **Memories written during embedder degradation have no vectors** — memories written before the model is ready do not participate in the semantic track; after semantics come online run `pnpm backfill` once to backfill existing vectors (after `pnpm build` has warmed the model cache; `HF_ENDPOINT` configurable). **The same holds when you change the model**: once `embedder.localModel` changes (or you switch between local and remote), the stored vectors no longer match the new model's dimension and writes are refused — either switch the model back, or re-embed everything with the new one: `pnpm backfill -- --force --model <id>` (optionally `--pooling mean|cls`; point `--cache-dir` at an existing model cache to skip a download). The script scans every store under `<dbDir>` (`user.db`, `shared.db`, `project-<hash>.db`); without `--force`, a model whose dimension differs from the stored vectors makes it fail before writing anything, so a mixed-dimension store can never be produced. Remote embedders (`kind: remote`) are out of scope — the script runs local transformers.js inference and cannot reproduce a remote endpoint's vectors.
264
276
 
265
277
  ## Roadmap
266
278
 
package/README.md CHANGED
@@ -77,7 +77,7 @@ dsh plugin --profile web add @kenz1117/dsh-engram
77
77
 
78
78
  - **跨会话记忆**:会话开始注入用户画像摘要(条数 + token 预算双重上限,可配;`engram_profile_edit` 维护的 curated 画像优先于派生事实),Agent 天然"记得"你是谁、在做什么;工具检索跨会话召回历史事实。
79
79
  - **双层分库**:`user.db` 全局共享;`project-<hash>.db` 按 git origin 标识隔离(无 git 时回退 cwd 全量哈希,旧命名库自动迁移)——个人偏好跟人走,项目约定跟仓库走。**项目宫殿随工作区切换**:管理面板「项目」scope 默认跟随 GUI 当前选中的工作区(Header 显示工作区标题与路径),也可在下拉里固定到某个工作区;`engram_save/search` 等工具的 project 读写同样按当前会话 cwd 归属,与面板同一口径。
80
- - **混合检索**:FTS5(unicode61 + 中文 2-gram 预切词)与本地向量(`Xenova/bge-small-zh-v1.5`,512 维,q8)RRF 融合 + 关系边一跳扩展 + 新鲜度/命中次数乘性排序 boost;嵌入模型离线运行,下载失败自动降级纯关键词并显式标记。
80
+ - **混合检索**:FTS5(unicode61 + 中文 2-gram 预切词)与本地向量(默认 `Xenova/paraphrase-multilingual-MiniLM-L12-v2`,384 维,q8;模型与池化可配,见下)RRF 融合 + 关系边一跳扩展 + 新鲜度/命中次数乘性排序 boost;嵌入模型离线运行,下载失败自动降级纯关键词并显式标记。
81
81
  - **记忆宫殿信息架构**(v0.7.2+):四原则全部落进核心路径,而非展示层皮肤——**位置当索引**(写入按 kind 分房并钉「房间#桩位」坐标,房间容量 9,满员开新房,桩位只增不回收);**固定路线定顺序**(`tour_routes` append-only,`engram_tour mode=fixed` 按桩位顺序走全宫);**骨架长期复用**(同一 topic 永远落在同一房间同一序号,顺序提取而非重新检索);**标记独一无二**(门牌规则 0-1 评分:全库唯一 +0.4 / 带日期锚点 +0.3 / 同房前 6 字不重复 +0.3,低分进翻新清单;库内 active 记忆少于 8 条时不扫描,避免小库噪声)。
82
82
  - **走廊路由检索**:画像里附房间目录,`engram_search` 支持 `room` 参数——先决定进哪个房间,再在房内检索,而非一上来全库 RRF。检索命中 top5 附同房间相邻桩位 id 作为编码特异性线索。
83
83
  - **检索练习闭环**(间隔重复):`engram_review_queue` 只给宫殿坐标与门牌线索、**不给正文**,迫使模型先主动回忆;`engram_review` 揭示核对,`engram_report grade`(0-5)自评推进 SM-2 调度(1 → 6 → round(prev × ease) 天,失败重置,ease 下限 1.3)。进入复习调度的条目**不再参与自动衰减**——命运由回忆结果决定。会话开始注入会提示今日待回忆条数。
@@ -141,6 +141,18 @@ dsh plugin --profile web add @kenz1117/dsh-engram
141
141
  assessReminder: true # 证据门收尾提醒:存在未判定检索批次时在下一步开始前注入 engram_assess 提醒
142
142
  modelCacheDir: '~/.dsh/engram/models' # 嵌入模型缓存目录
143
143
  hfEndpoint: 'https://huggingface.co' # 模型下载端点,网络受限可配镜像
144
+ # 嵌入器:整段不写就是 local + 默认模型;kind 决定用本地模型还是远程端点
145
+ embedder:
146
+ kind: 'local' # local(transformers.js 离线推理)| remote(OpenAI 兼容 /embeddings)
147
+ localModel: 'Xenova/paraphrase-multilingual-MiniLM-L12-v2' # 384 维;旧默认 Xenova/bge-small-zh-v1.5 为 512 维
148
+ localPooling: 'mean' # mean(多数句向量模型)| cls(bge-m3 等 XLM-R 同族)
149
+ # kind: 'remote'
150
+ # model: 'text-embedding-3-small' # remote:模型名,原样传给服务端
151
+ # baseUrl: 'https://api.example.com/v1' # remote:含版本号的根地址
152
+ # apiKeyEnv: 'EMBEDDING_API_KEY' # remote:密钥从这个环境变量读(优先于写死在配置里的 apiKey)
153
+ # batchSize: 32 # remote:每批条数
154
+ # timeoutMs: 30000 # remote:单次请求超时(毫秒)
155
+ # 换模型要重建向量:维度与库中不同时写入和比较都直接拒绝(EMBEDDER_DIMENSION_MISMATCH),不截断后假装相似
144
156
  ingest: 'off' # 自动摄取:off | light(仅用户消息,每轮≤2条)| eager(含助手消息,每轮≤5条)
145
157
  reflect: 'off' # 自动信念巩固:off | suggest(会话结束只登记待巩固,engram_reflect 执行)| auto(会话结束后台巩固);任何档位均可手动调 engram_reflect
146
158
  # provider 与 model 必须成对提供:摄取/蒸馏的辅助 LLM 路由覆盖(缺省从会话日志解析)
@@ -196,7 +208,7 @@ dsh plugin --profile web add @kenz1117/dsh-engram
196
208
  - **宫殿结构(目录即房间,路径即路线)**:**主厅** = 每轮注入的常驻核心画像(少而稳,每次都在场);**走廊** = 画像里附的房间目录 + `engram_search room` 参数(先定房间,再检索);**房间** = 按 kind 分房(事实厅 / 偏好阁 / 决策堂 / 往事廊 / 技法坊),容量 9,满员开「房名-2」;**门牌** = 每条记忆的 `placard` 铭牌,受唯一·差异化·带日期纪律评分。存量库首次打开时自动补排桩(幂等,开新房会告警提醒人工命名)。
197
209
  - **自动摄取**(`ingest` 开启时):新一轮第一步从会话日志提取上一轮的候选事实;会话结束(session/disposed)补摄取最后一轮,5 秒超时,失败/超时把 pending 键写入操作日志,下次会话首步自动重放补做;已摄取的 (会话, 轮次) 幂等去重(键固定在私人库,实时与历史回填共用一份,跨路径不重复)。读取源是会话日志;辅助调用的请求审计走插件自身操作日志,不向会话日志 append 未知事件。候选以低 confidence 写入并按**目标分库内**的嵌入近邻去重。**作用域逐条判**:提炼输出带 `scope`,`project` 落当前会话 cwd 的项目库、`user` 落私人库;无 cwd 的会话只能进私人库(此类会话不做逐条判定,避免标记与实际分库不符)。历史回填沿用同一判定,默认按各会话自己的 cwd 落项目库。
198
210
  - **来源链**:每条记忆记录来源会话、轮次与事件 seq,`engram_review` 可完整回查;操作日志表记录全部写入/修改/遗忘/蒸馏/衰减。
199
- - **嵌入离线**:模型首次使用需联网下载(q8 约 50MB,端点可配镜像),此后完全离线;失败时插件照常工作,检索降级纯关键词并显式标记。
211
+ - **嵌入离线**:模型首次使用需联网下载(q8 约 50MB,端点可配镜像),此后完全离线;失败时插件照常工作,检索降级纯关键词并显式标记。**维度是硬约束**:库中向量与当前模型的维度不一致时,写入与相似度比较都直接拒绝并报 `EMBEDDER_DIMENSION_MISMATCH`(说清两边各是多少维),而不是逐维截断后给出一个像样的相似度。
200
212
  - **界面本地化**:client 半经宿主 locale 服务注册 zh/en 词典,跟随宿主语言设置实时切换;状态/种类等数据枚举仅在显示层映射,存储值保持英文。
201
213
 
202
214
  ## Web 管理面板
@@ -261,7 +273,7 @@ pnpm test
261
273
  ## Known Limitations and Deferred Work
262
274
 
263
275
  - **矛盾判定默认无 LLM** —— 写入时默认仅按向量相似度(≥0.88)报告候选并建边,语义矛盾的确认留给模型/用户裁决与蒸馏;可配 Jev 裁决(面板「裁决」tab 或 yml `jev`)让模糊带与矛盾边走外部判决,Jev 不可用时回落本默认。
264
- - **嵌入器降级期间的记忆无向量** —— 模型未就绪时写入的记忆不参与语义道;语义上线后跑一次 `pnpm backfill` 补算存量向量(`pnpm build` 的模型缓存就绪后执行,可经 `HF_ENDPOINT` 配镜像)。
276
+ - **嵌入器降级期间的记忆无向量** —— 模型未就绪时写入的记忆不参与语义道;语义上线后跑一次 `pnpm backfill` 补算存量向量(`pnpm build` 的模型缓存就绪后执行,可经 `HF_ENDPOINT` 配镜像)。**换模型同理**:换了 `embedder.localModel`(或从 local 换到 remote)之后,库中旧向量与新模型维度不同,写入会被拒绝——要么把模型改回去,要么用新模型重建全部向量:`pnpm backfill -- --force --model <id>`(`--pooling mean|cls` 可选,`--cache-dir` 指向已有模型缓存可省一次下载)。脚本扫描 `<dbDir>` 下的全部分库(`user.db`、`shared.db`、`project-<hash>.db`);不带 `--force` 时若模型维度与库中已有向量不符,它在写入前就报错退出,绝不会写出混合维度的库。远程嵌入器(`kind: remote`)不在脚本覆盖范围——它跑的是本地 transformers.js 推理,复现不了远程端点的向量。
265
277
 
266
278
  ## 路线图
267
279