@kenz1117/dsh-engram 0.7.3 → 0.7.5

package/README.en.md

@@ -73,15 +73,17 @@ Sensory and emotional dimensions (smell, temperature, emotional weight) are deli
 - **Capture redaction**: inbound content passes a regex scrub for common secrets and credentials (sk- API keys, Bearer, AWS AKIA, GitHub tokens, PEM private keys, password/token assignments) and matched fragments become `[REDACTED:<type>]`.
 - **Recall placeholder (anti-echo-chamber)**: memory-recall tool output inside captured slices is replaced with `[engram memory result omitted from capture: <tool>]`, and the extraction prompt states that restating existing memory is not new information, breaking the memory self-reinforcement loop.
 - **Multi-query retrieval**: `engram_search` can use the aux LLM to rewrite the query into ≤3 complementary queries, retrieving each and fusing them with cross-query RRF plus a per-query floor; a failed rewrite degrades to the single query (`queryRewrite: false` disables).
+- **Evidence gate (search → assess)**: a hit means "relevant", not "enough to answer". Every retrieval registers an in-process batch (latest 20 per session, released when the session ends) and prints `ref=…` per row plus the batch id; `engram_assess` may only cite refs of that batch, and `sufficient` is enforced by code — all three must hold (the model claims adequacy, at least one valid evidence ref, `nextStrategy=answer`) or the verdict is insufficient with the strategy rewritten back to retrieval. Verdicts and rejected refs go to the audit log, visible under the "retrieval" category in the curator log.
 - **Portable data**: `engram_export` exports Markdown / JSON files in one step, with a redacted-view variant (secondary scrubbing + 40-char preview truncation, share-safe). `engram_mirror` writes an Obsidian / Logseq-friendly mirror directory (one Markdown per memory with YAML frontmatter + `[[id]]` backlinks), turning the palace into a human-readable private knowledge base.
 - **AGI Architecture Exploration (dsh-market · AGI Architecture)**: listed in dsh-market's "AGI Architecture Exploration" category as a cognitive-science reframe of agent long-term memory — memory palace (imagery labels + room placards), corridor topology (force-directed graph), closure questions (the ingest prompt nudges the model to ask clarifying questions), consolidation merging (heuristic dedup + cosine similarity) — sitting alongside MemGPT / Letta in the "agent memory architecture" conversation.
 
-## Tools (16, narrow parameters)
+## Tools (17, narrow parameters)
 
 | Tool | Purpose |
 |---|---|
 | `engram_save` | Save (automatic contradiction-candidate detection when embeddings are available); `items` array saves ≤10 in one call with shared scrubbing and in-batch dedup, one failure not blocking the rest (`count`/`items`/`failed` summary); `placard` attaches a marker (scored unique · distinctive · dated; low scores get a rewrite hint) |
-| `engram_search` | Hybrid semantic + keyword retrieval (hits reinforce confidence); `room` routes through the corridor — search inside one room only; the top 5 hits carry same-room neighbouring-slot cues |
+| `engram_search` | Hybrid semantic + keyword retrieval (hits reinforce confidence); `room` routes through the corridor — search inside one room only; the top 5 hits carry same-room neighbouring-slot cues. Output rows carry `id=` and `ref=`; the tail carries the batch id |
+| `engram_assess` | Evidence gate: before answering, judge whether the retrieved content suffices. Submit `batchId` + ≤8 `evidenceRefs` (only `ref=` values from that batch's output) + `missing` + `nextStrategy`; the code requires all three for `sufficient` (claimed adequate, at least one valid in-batch evidence ref, `nextStrategy=answer`), otherwise it rules insufficient and rewrites the strategy back to retrieval; refs from other batches are rejected and listed, and the verdict is written to the audit log |
 | `engram_timeline` | Timeline browsing: creation-time descending by default; `order: 'tour'` follows the fixed tour route's slot order instead (output carries palace coordinates, entries off-route sorted last) so the agent can re-walk the route |
 | `engram_update` | Correct an entry (supersedes chain); can re-attach a `placard` marker too |
 | `engram_forget` | Forget (soft-delete, restorable) |
@@ -108,7 +110,7 @@ Optional configuration (cordis.yml):
     dbDir: '~/.dsh/engram'          # store and model-cache root directory
     injectProfile: true             # inject the user memory profile at session start
     profileTopN: 8                  # injection entry cap (1-64)
-    injectTokenBudget: 1024         # injection token budget (128-8192, estimated ceil(len/4); over-budget entries degrade to index lines)
+    injectTokenBudget: 1024         # injection token budget (128-8192; CJK text counted at 1.5 tokens/char, other text at 4 chars/token; over-budget entries degrade to index lines)
     modelCacheDir: '~/.dsh/engram/models'  # embedding model cache directory
     hfEndpoint: 'https://huggingface.co'   # model download endpoint; set a mirror behind restricted networks
     ingest: 'off'                   # automatic capture: off | light (user messages only, ≤2/turn) | eager (assistant messages too, ≤5/turn)
@@ -161,6 +163,10 @@ Since v0.7.2 the Today view opens with a **"Due today" card**: each due memory i
 
 Since v0.7.3 there is a dedicated **"History backfill" tab**: you choose every import rule (time window / turns per session / total turn budget / include subagent·seeded·no-cwd sessions), press "Re-estimate" to see candidate sessions and pending turns at zero cost (no LLM calls), then start. While running it shows progress (sessions / turns / written / skipped / failed) and can be paused at any time — finished turns are skipped by idempotency key, so starting again resumes. The same release rebuilds the panel as five views (Today / Palace / Corridor tour / Curator log / History backfill): the always-on nine-cell curator bar folds into a summary card at the top of Today (three headline metrics + last-7-days counts + health ring), and the front page keeps only what to do (due today, refurb) and what to reference (room directory, tour proposal); the corridor bird's-eye and the retrieval bench move to Corridor tour; the curator log becomes a full page filterable by writes / capture / retrieval / organize; and each of the five rooms gets its own hue (fact blue / preference purple / decision teal / episode orange / skill magenta) across tags, the room directory, and corridor nodes. The exhibition list becomes compact rows: hairlines instead of cards, two-line body, action buttons revealed on hover (or keyboard focus) and wrapped below the body on narrow screens — roughly twice as many rows per screen. The refurb list stops scanning a palace with fewer than 8 active memories, avoiding small-store noise.
 
+Since v0.7.4 the panel gets a second pass of polish: Today is rearranged to "tour proposal on the left, room directory + due today + refurb list on the right", with roomier rows in the tour proposal and the room directory; the Room exhibition toolbar becomes one row of search + status + sort with the room filter as its own wrapping chip row; the curator log merges its counts and category filter into a single toolbar and marks every row with a category dot (writes / capture / retrieval / organize); history backfill's rules, estimate, and run blocks are separated by hairlines instead of a tinted estimate box; and section spacing now comes solely from the container gap, removing the asymmetry where a heading hugged the card above but sat far from the one below.
+
+Since v0.7.5: two underlying fixes plus one new capability. ① Profile-injection token estimation is now CJK-aware (CJK at 1.5 tokens/char, other text at 4 chars/token); the old "length / 4" undercounted Chinese by more than 4×, so Chinese users' injections routinely exceeded `injectTokenBudget` by 17%–50% — the trailing `+N more` counter line now counts against the budget too, so the injection never exceeds its promise. ② A new **evidence gate** `engram_assess` (17 tools): a hit means "relevant", not "enough to answer". `engram_search` registers an in-process evidence batch per call (latest 20 per session, released when the session ends) and prints `ref=` per row plus the batch id; `engram_assess` may only cite refs of that batch and `sufficient` is enforced by code — the model's claim, at least one valid evidence ref, and `nextStrategy=answer` must all hold, otherwise the verdict is insufficient with the strategy rewritten back to retrieval; refs from other batches are rejected and listed, and the verdict goes to the audit log (visible under the "retrieval" category of the curator log). ③ The curator log gains the missing op labels and detail formatting (consolidation / review answer / slot assigned / slots backfilled / room opened) instead of raw op names and JSON.
+
 ## Development
 
 ```sh

package/README.md

@@ -73,15 +73,17 @@ dsh plugin --profile web add @kenz1117/dsh-engram
 - **摄取脱敏**：入库前正则清洗常见密钥凭据（sk- 系 API key、Bearer、AWS AKIA、GitHub token、PEM 私钥、password/token 赋值），命中片段替换为 `[REDACTED:<类型>]`。
 - **召回占位（防回声室）**：摄取切片中记忆召回工具的输出替换为 `[engram memory result omitted from capture: <tool>]`，并向提取模型附注"既有记忆的复述不是新信息"，阻断记忆自我强化循环。
 - **多查询检索**：`engram_search` 可用辅助 LLM 把查询改写为 ≤3 个互补查询分别检索，跨查询 RRF 融合 + 每查询保底命中；改写失败自动降级单查询（`queryRewrite: false` 关闭）。
+- **证据门（search → assess）**：检索命中只说明「相关」，不说明「足以回答」。每次检索登记一个进程内批次（每会话保留最近 20 个，会话结束即释放），输出行尾给出 `ref=…` 与批次 id；`engram_assess` 只能引用同一批次的 ref，且 `sufficient` 由代码强制——三者齐备（模型声称充足、至少一条有效证据、`nextStrategy=answer`）才算充足，否则判为不足并把策略改回继续检索。判定与拒绝明细写入审计日志，面板「管家日志」的「检索」类别可见。
 - **数据可携带**：`engram_export` 一键导出 Markdown / JSON 文件，支持脱敏视图（内容二次清洗 + 预览截断，分享安全）。`engram_mirror` 导出可漫游的镜像目录（Obsidian / Logseq 友好：每条记忆一个 Markdown，正文 + YAML frontmatter + 双向链接 `[[id]]`），让「宫殿」也成为可人读的私人知识库。
 - **认知架构探索（dsh-market · AGI 架构探索）**：本仓库是 dsh-market「AGI 架构探索」类目下，对 agent 长期记忆的认知科学方法论重构——记忆宫殿（意象标签 + 房间铭牌）、走廊拓扑（力导向图）、闭环提问（摄入时让模型主动追问用户细节）、巩固合并（启发式去重 + 余弦相似度），与 MemGPT/Letta 同层「agent 记忆架构」叙事。
 
-## 工具（16 个，窄参数）
+## 工具（17 个，窄参数）
 
 | 工具 | 作用 |
 |---|---|
 | `engram_save` | 保存（嵌入可用时自动做矛盾候选检测）；支持 `items` 数组单次批量保存 ≤10 条，统一清洗/批量内去重，单条失败不影响其余（`count`/`items`/`failed` 汇总返回）；`placard` 挂门牌（按唯一·差异化·带日期评分，低分附改写建议） |
-| `engram_search` | 语义 + 关键词混合检索（命中强化置信度）；`room` 参数做走廊路由——只在指定房间内检索；命中 top5 附同房相邻桩位线索 |
+| `engram_search` | 语义 + 关键词混合检索（命中强化置信度）；`room` 参数做走廊路由——只在指定房间内检索；命中 top5 附同房相邻桩位线索。输出行尾给 `id=` 与 `ref=`，末尾给批次 id |
+| `engram_assess` | 证据门：作答前判定「检索到的内容是否足以回答」。提交 `batchId` + ≤8 条 `evidenceRefs`（只能取该批次输出里的 `ref=`）+ `missing` + `nextStrategy`；代码强制 `sufficient` 需同时满足「声称充足」「至少一条属于本批次的有效证据」「nextStrategy=answer」，否则判为不足并把策略改回继续检索；非本批次的 ref 会被拒绝并列出，判定写入审计日志 |
 | `engram_timeline` | 时间线浏览：默认按创建时间倒序；`order: 'tour'` 改按固定巡游路线桩位顺序（输出附宫殿坐标，未上路线者排末尾），让 agent 也能沿固定路线复述 |
 | `engram_update` | 修正（supersedes 取代链）；可同时改挂 `placard` 门牌 |
 | `engram_forget` | 遗忘（软删可恢复） |
@@ -108,7 +110,7 @@ dsh plugin --profile web add @kenz1117/dsh-engram
     dbDir: '~/.dsh/engram'          # 分库与模型缓存根目录
     injectProfile: true             # 会话开始注入用户画像摘要
     profileTopN: 8                  # 注入条数上限（1-64）
-    injectTokenBudget: 1024         # 注入 token 预算（128-8192，估算 ceil(len/4)，超预算条目降级为索引行）
+    injectTokenBudget: 1024         # 注入 token 预算（128-8192，中文按 1.5 token/字、其余按 4 字符/token 估算，超预算条目降级为索引行）
     modelCacheDir: '~/.dsh/engram/models'  # 嵌入模型缓存目录
     hfEndpoint: 'https://huggingface.co'   # 模型下载端点，网络受限可配镜像
     ingest: 'off'                   # 自动摄取：off | light（仅用户消息，每轮≤2条）| eager（含助手消息，每轮≤5条）
@@ -161,6 +163,10 @@ v0.7.2 起「今日」视图首屏新增**「今日待回忆」卡**：按线索
 
 v0.7.3 起新增独立的**「历史回填」tab**：导入规则全部由你选择（时间窗 / 单会话轮数 / 总轮数上限 / 是否包含子代理·种子·无 cwd 会话），点「重新估算」先看候选会话数与待处理轮数（零成本、不调 LLM），确认后「开始回填」；运行中显示进度（会话 / 轮次 / 写入条数 / 跳过 / 失败）并可随时暂停——已完成的轮次按幂等键跳过，再点开始即续做。同一版把面板重做成五个视图（今日管家 / 宫殿陈展 / 走廊巡游 / 管家日志 / 历史回填）：原常驻的九格「管家日报」条收进「今日」视图的速览卡（三个大指标 + 近 7 天计数 + 健康分环），首页只留在办与参考两块（待回忆、待翻新 / 房间目录、入殿导航）；走廊鸟瞰与检索实验台移入「走廊巡游」；管家日志独立成整页，可按落成类 / 发掘 / 检索 / 整理筛选；五间房各一色（事实蓝 / 偏好紫 / 决策青 / 往事橙 / 技法品红）贯穿标签、房间目录与走廊节点。陈展列表改为紧凑行式：分隔线取代卡片描边、正文两行截断、操作按钮 hover（或键盘聚焦）才显现、窄屏折到正文下方，一屏可读条目约翻一倍。翻新清单在库内 active 少于 8 条时不再扫描，避免小库噪声。
 
+v0.7.4 起继续打磨面板细节：今日视图重排为「左入殿导航 · 右房间目录、今日待回忆、翻新清单」，入殿导航与房间目录加大行间距；宫殿陈展工具栏改「搜索 + 状态 + 排序」一行、房间筛选独立成可换行 chips；管家日志把计数与类别筛选合成一条工具条，并给每行加类别色点（落成 / 发掘 / 检索 / 整理）；历史回填的规则、估算、执行三段改用分隔线切块；区域间距统一由容器间距给出，消除「标题贴住上方卡片、下方却过松」的不对称。
+
+v0.7.5 起是两处底层修正加一层新能力。① 画像注入的 token 估算改为 CJK 感知（中文按 1.5 token/字、其余按 4 字符/token）：此前按长度除以 4 会把中文低估四倍以上，中文用户的实际注入长期超出 `injectTokenBudget` 约 17%–50%；末尾 `+N more` 计数行也纳入预算，注入总量不再超承诺。② 新增**证据门** `engram_assess`（工具 17 个）：检索命中只说明「相关」，不说明「足以回答」；`engram_search` 每次登记一个进程内证据批次（每会话保留最近 20 个，会话结束即释放），输出每行带 `ref=`、末尾带批次 id；`engram_assess` 只能引用同一批次的 ref，且 `sufficient` 由代码强制——声称充足、至少一条有效证据、`nextStrategy=answer` 三者齐备才算充足，否则判为不足并把策略改回继续检索；不属于该批次的 ref 会被拒绝并列出，判定写入审计日志（管家日志「检索」类别可见）。③ 管家日志补齐 op 词典与明细格式化（闭馆整理 / 复习答题 / 排桩 / 批量排桩 / 开新房），不再显示英文原名与原始 JSON。
+
 ## 开发
 
 ```sh
@@ -180,7 +186,7 @@ pnpm bundle
 
 #### Token effect
 
-画像注入为条件性固定成本（受条数上限与 token 预算双重约束）；工具 schema 为常驻成本（16 个窄参数工具）。
+画像注入为条件性固定成本（受条数上限与 token 预算双重约束）；工具 schema 为常驻成本（17 个窄参数工具）。
 
 #### KV Cache effect