dsh-experience 1.0.4 → 1.0.8

package/CHANGELOG.md

@@ -1,5 +1,77 @@
 # Changelog
 
+## [1.0.8] — 2026-11-09 — CI 测试与持久化回归
+
+### 新增
+
+- `.github/workflows/test.yml`：每次 push 到 `main` 与每个 pull request 都跑
+  `node tools/selftest.mjs` 与 `node tools/e2e.mjs`，矩阵覆盖 Ubuntu/Windows × Node 20/22。
+  此前仓库只有「打 tag 时发布 npm」的流水线，代码从不经过测试。
+- `lib/store.js`：把 store 读写从 `index.js` 抽出来，成为可测试的纯模块
+  （`emptyStore / normalizeStore / readStore / writeStore`，写入走临时文件 + rename）。
+- `T69`：写入 → 重载 → 断言 `archive` / `evictions` / `nextId` 全部还在。
+- `T70`：未知字段（未来版本新增的表）在旧版本 reload 时不被丢弃。
+- `T71`：损坏 JSON 返回 `{error, store}` 而不是抛异常；缺失文件标记 `missing`。
+
+### 修复
+
+- `loadStore()` 不再自己重建字段，改走 `normalizeStore()`——这正是 1.0.7 里
+  `archive` / `evictions` 被静默抹掉的根因。
+- `e2e` T46 原先断言 `store.archive === undefined`，等于把上面那个 bug 写进了测试；
+  改为断言「归档条数为 0」。
+
+### 测试
+
+`selftest 81 pass / 0 fail`、`e2e 19 pass / 0 fail`（合计 100）。
+
+## [1.0.7] — 2026-11-09 — 检索：把「坑」纳入档案，修流水账挤掉垂直经验
+
+现象：真实任务（plugin-test-5 写 .mjs、require 不存在的模块、报错后排查修复）
+检索 PASS 且 top=0.26，但注入的是 exp-23/24（写文件流水账），exp-26（node.exe / MODULE_NOT_FOUND 避坑指南）
+连前五都没进。
+
+根因两条：
+
+1. **打分只看 title/when/keywords，不看 pitfalls/reverse_examples**。经验真正的价值在坑里
+   （`MODULE_NOT_FOUND`、`CommandNotFoundException`），这些词根本没进索引。
+2. **特征桶权重太平均**。`#error` 和 `#node` 与 `#file` 同权，一句「报错」压不过一堆「文件/新增」。
+
+修法：
+
+- `profileText()` 把 `pitfalls` 与 `reverse_examples` 一起入索引。
+- 特征桶分级权重：`#error` 3、`#node` 2.6、`#module` 2.2、`#shell`/`#debug` 2、`#run`/`#write` 1.8、其余 1.4/1.2。
+- 关键特征（`#error`/`#node`/`#module`）同时命中时再给 0.08/个 的确定性加分。
+- 新增工具 `experience_archive`：把记录移入 `store.archive`（可 `/memory restore` 恢复，写 `evictions` 审计），
+  用于清理已批准的流水账；不删除数据。
+
+实测（真实库，清理 exp-22..25 后剩下 4 条）：四条真实失败查询上 exp-26 全部排第 1
+（0.364 / 0.373 / 0.502 / 0.502），无关查询仍为 0。测试：selftest 78 pass、e2e 19 pass。
+
+## [1.0.6] — 2026-11-09 — 检索：停用词、特征桶与 idf 加权
+
+现象：一条噪声较大的任务描述（"跑一个会踩坑的 .mjs 脚本…内置 node.exe 模块报错"）
+在旧打分下只有 0.07，exp-26 根本没被召回。
+
+修法：
+
+- 停用词扩到口语填充词（随便/故意/场景/试试…），噪声不再稀释覆盖率。
+- 新增特征桶（`#node` `#shell` `#error` `#module` `#test` `#file` `#path` `#run` `#write` `#debug` …），
+  在原始文本上判定，词表不重合也能命中；`#run`/`#write` 区分"跑脚本"与"加函数"这两类意图。
+- 打分改为 idf 加权：跨经验常见的词（报错/模块/脚本）权重低，稀有词（node.exe/MODULE_NOT_FOUND）权重高；
+  中文 bigram 权重 ×0.45（它们最容易误撞）。
+- 索引文件名 basename，"truncate.test.mjs" 也能匹配 "test/truncate.test.mjs"。
+- 阈值可配：`/memory config set recallThreshold 0.1`（默认仍 0.15；实测无关查询得分为 0，
+  噪声查询下第二名 0.143，降到 0.05 会把第二名一起放进来）。
+
+实测（真实库 7 条）：噪声查询 exp-26=0.159 排第一；"新增纯函数+用例" exp-1=0.852；
+"重命名图片" 全部 0。测试 77 pass（selftest）/ 19 pass（e2e）。
+
+## [1.0.5] — 2026-11-09 — 第一个 npm 版本
+
+- 发布到 npm：`dsh-experience@1.0.4` 是首个公开版本，`1.0.5` 修正安装说明。
+- `README.md`：安装段从 `file:`/`link:` 改为 `install_bundle target=dsh-experience@<版本>`，并补升级方式（Web 插件页 Update / 重新 install）与“npm 安装是复制、改代码不生效”的说明。
+- 新增 `tools/pack-check.mjs`（发布前按 `files` 白名单核对入包内容）与 `.github/workflows/publish.yml`（推 `v*` tag 校验版本一致、跑 selftest+e2e 后 `npm publish`）。
+
 ## [1.0.4] — 2026-11-09 — npm 发布通道
 
 - 新增 `.github/workflows/publish.yml`：推送 `v*` tag 时先校验 tag 与 `package.json` 版本一致、跑 selftest 与 e2e，再 `npm publish --access public`。

package/README.md

@@ -66,38 +66,25 @@ flowchart LR
 
 ## 安装
 
-### 方式一：作为 bundle 安装（推荐）
+### 方式一：从 npm 安装（推荐）
 
 ```text
-plugin_manager install_bundle target=file:<PATH-TO-dsh-experience>
+plugin_manager install_bundle target=dsh-experience@1.0.5
 ```
 
-开发模式（改代码即时生效，Junction 链接）：
+安装后 `plugin_manager list_plugins` 应出现 `dsh-experience` 且 `fiberPhase: active`。
+
+**升级**：Web 插件页对该 bundle 点 **Update**，或再执行一次 `install_bundle target=dsh-experience@<新版本>`。升级走 `listVersions` → `updateBundle`，失败会自动回滚到旧版本。
+
+### 方式二：本地开发（Junction 链接）
 
 ```text
 plugin_manager install_bundle target=link:<PATH-TO-dsh-experience>
 ```
 
-安装后 `plugin_manager list_plugins` 应出现 `dsh-experience` 且 `fiberPhase: active`。
+链接安装改代码即时生效；npm 安装是复制，改代码不生效，要发新版本。
 
-### 方式二：手工写进 profile
-
-编辑 `~/.dsh/profiles/<profile>/package.json`：
-
-```json
-{
-  "dependencies": {
-    "dsh-experience": "link:<PATH-TO-dsh-experience>"
-  },
-  "dsh": {
-    "profile": {
-      "bundles": ["dsh-experience"]
-    }
-  }
-}
-```
-
-重启 DSH 后，在任意会话里执行 `/memory list`，看到 `库 0 条` 即安装成功。
+装完在任意会话执行 `/memory list`，看到 `库 0 条` 即安装成功。
 
 ## 命令速查
 
@@ -135,11 +122,19 @@ plugin_manager install_bundle target=link:<PATH-TO-dsh-experience>
 - 注入前做长度截断（默认 2 条 / 1600 字符）；
 - 仓库不包含任何作者运行时数据，骨架见 [store.example.json](store.example.json)。
 
+## 检索加权
+
+当任务包含报错（`#error`）或明确排错意图时，含有避坑经验的历史条目将获得 3 倍以上的检索加权，确保模型优先复用排错捷径。
+
+实现上：`pitfalls` 与 `reverse_examples` 一并入索引，特征桶分级加权（`#error` 3、`#node` 2.6、`#module` 2.2…），
+`#error`/`#node`/`#module` 同时命中再加确定性加分。实测同一批失败查询上，正确条目的分数从 0.07 提升到 0.36–0.50，
+并在清理流水账后稳定排第 1。
+
 ## 已知限制
 
 - 召回仅在每轮第 1 步生效；
 - `auto-verified` 默认不参与召回 —— 错误经验的代价高于多一次点击；
-- 相似度是词覆盖打分，不做向量检索（零依赖的代价）；
+- 相似度是词覆盖打分 + 特征桶，不做向量检索（零依赖的代价）；**同义改写可能漏召回**——记录里写 `MODULE_NOT_FOUND`、提问只说「模块找不到」时不一定匹配，这需要向量检索才能解决，属能力边界；
 - 淘汰只归档不删除，需要 `restore` 才能回到召回池；
 - 经验只描述「怎么做」，不包含业务代码本身。
 

package/index.js

@@ -19,6 +19,8 @@ import crypto from 'node:crypto'
 import * as C from './lib/capture.js'
 import * as E from './lib/evidence.js'
 import * as D from './lib/decay.js'
+import * as R from './lib/recall.js'
+import * as S from './lib/store.js'
 import { renderLookup } from './lib/lookup.js'
 
 export const name = 'dsh-experience'
@@ -33,25 +35,10 @@ const PROBE_FILE = path.join(STORE_DIR, 'probe.json')
 const THRESHOLD = 0.15
 const MAX_EXPERIENCES = 2
 const MAX_CHARS = 1600
-const BUILD = 'r11-supersede'
+const BUILD = 'r15-store-tests'
 
 let CONFIG = { ...C.DEFAULT_CONFIG }
 
-const STOP = new Set([
-  '的', '了', '和', '与', '在', '是', '把', '并', '就', '还', '也', '有', '用', '要', '中',
-  'the', 'a', 'an', 'and', 'or', 'to', 'of', 'in', 'on', 'for', 'with', 'is', 'are', 'be',
-  'it', 'this', 'that', 'then', 'use', 'run', 'can',
-])
-
-const emptyStore = () => ({
-  version: 1,
-  nextId: 1,
-  nextInjectionId: 1,
-  experiences: [],
-  injections: [],
-  turns: [],
-})
-
 function loadConfig() {
   try {
     const raw = JSON.parse(fs.readFileSync(CONFIG_FILE, 'utf8'))
@@ -83,77 +70,39 @@ function saveConfig() {
 }
 
 function loadStore() {
-  try {
-    const raw = fs.readFileSync(STORE_FILE, 'utf8')
-    const parsed = JSON.parse(raw)
-    if (!parsed || typeof parsed !== 'object') throw new Error('not an object')
-    const store = {
-      version: Math.max(2, Number(parsed.version || 1)),
-      nextId: parsed.nextId || 1,
-      nextInjectionId: parsed.nextInjectionId || 1,
-      experiences: Array.isArray(parsed.experiences) ? parsed.experiences : [],
-      injections: Array.isArray(parsed.injections) ? parsed.injections : [],
-      turns: Array.isArray(parsed.turns) ? parsed.turns : [],
-    }
-    E.migrateStore(store)
-    // Records written before the redaction fix may still carry absolute paths.
-    let scrubbed = 0
-    for (const record of store.experiences) if (C.scrubRecord(record)) scrubbed += 1
-    if (scrubbed) console.warn(`[experience] scrubbed ${scrubbed} legacy record(s)`)
-    return store
-  } catch (error) {
-    if (error && error.code === 'ENOENT') return emptyStore()
+  const result = S.readStore(STORE_FILE)
+  if (result.error) {
     try {
       fs.renameSync(STORE_FILE, `${STORE_FILE}.broken-${Date.now()}`)
     } catch {}
-    console.warn('[experience] store unreadable, started empty:', error && error.message)
-    return emptyStore()
+    console.warn('[experience] store unreadable, started empty:', result.error && result.error.message)
   }
+  const store = result.store
+  E.migrateStore(store)
+  // Records written before the redaction fix may still carry absolute paths.
+  let scrubbed = 0
+  for (const record of store.experiences) if (C.scrubRecord(record)) scrubbed += 1
+  if (scrubbed) console.warn(`[experience] scrubbed ${scrubbed} legacy record(s)`)
+  return store
 }
 
 function saveStore(store) {
-  fs.mkdirSync(STORE_DIR, { recursive: true })
-  const tmp = `${STORE_FILE}.tmp`
-  fs.writeFileSync(tmp, JSON.stringify(store, null, 2), 'utf8')
-  fs.renameSync(tmp, STORE_FILE)
+  S.writeStore(STORE_FILE, store)
 }
 
 function tokens(text) {
-  const out = new Set()
-  const source = String(text || '').toLowerCase()
-  for (const match of source.matchAll(/[a-z0-9_][a-z0-9_.\-/]*/g)) {
-    const token = match[0]
-    if (token.length >= 2 && !STOP.has(token)) out.add(token)
-  }
-  const cjk = source.replace(/[^\u4e00-\u9fff]+/g, ' ')
-  for (const chunk of cjk.split(' ')) {
-    for (let i = 0; i + 1 < chunk.length; i++) {
-      const bigram = chunk.slice(i, i + 2)
-      if (!STOP.has(bigram)) out.add(bigram)
-    }
-  }
-  return out
+  return R.expand(text)
 }
 
-function profileTokens(experience) {
-  return tokens([
-    experience.title,
-    experience.taskType,
-    experience.when,
-    (experience.keywords || []).join(' '),
-  ].join(' '))
+function scoreExperience(experience, queryTokens, index) {
+  return R.score(R.profileText(experience), queryTokens, index)
 }
 
-function scoreExperience(experience, queryTokens) {
-  const expTokens = profileTokens(experience)
-  if (!queryTokens.size || !expTokens.size) return 0
-  let queryHits = 0
-  for (const token of queryTokens) if (expTokens.has(token)) queryHits++
-  let expHits = 0
-  for (const token of expTokens) if (queryTokens.has(token)) expHits++
-  const queryCoverage = queryHits / queryTokens.size
-  const expCoverage = expHits / expTokens.size
-  return Math.round((0.7 * queryCoverage + 0.3 * expCoverage) * 1000) / 1000
+// Retrieval floor. Configurable via "/memory config set recallThreshold 0.1"
+// because the right value depends on how noisy the task descriptions are.
+function threshold() {
+  const value = Number(CONFIG.recallThreshold)
+  return Number.isFinite(value) && value > 0 ? value : THRESHOLD
 }
 
 function textOfMessage(message) {
@@ -321,6 +270,7 @@ export function apply(ctx) {
 
     const queryText = (payload.messages || []).map(textOfMessage).join('\n')
     const queryTokens = tokens(queryText)
+    const index = R.corpus(store.experiences)
     const scored = store.experiences
       .map((experience) => ({ experience, score: scoreExperience(experience, queryTokens) }))
       .sort((a, b) => b.score - a.score)
@@ -329,10 +279,10 @@ export function apply(ctx) {
       title: experience.title,
       status: experience.status,
       score,
-      passed: recallable(experience) && score >= THRESHOLD,
+      passed: recallable(experience) && score >= threshold(),
     }))
     const chosen = scored
-      .filter((entry) => recallable(entry.experience) && entry.score >= THRESHOLD)
+      .filter((entry) => recallable(entry.experience) && entry.score >= threshold())
       .slice(0, MAX_EXPERIENCES)
 
     const trace = {
@@ -344,7 +294,7 @@ export function apply(ctx) {
       query: queryText.slice(0, 400),
       queryTokens: [...queryTokens].slice(0, 40),
       candidates,
-      threshold: THRESHOLD,
+      threshold: threshold(),
       injected: chosen.map((entry) => entry.experience.id),
       injectionId: null,
       scores: Object.fromEntries(chosen.map((entry) => [entry.experience.id, entry.score])),
@@ -553,10 +503,11 @@ export function apply(ctx) {
   function noteGatedFailure(trace) {
     try {
       const queryTokens = tokens(trace.query || '')
+      const index = R.corpus(store.experiences)
       let best = null
       for (const existing of store.experiences) {
         if (!['verified', 'auto-verified', 'candidate'].includes(existing.status)) continue
-        const score = scoreExperience(existing, queryTokens)
+        const score = scoreExperience(existing, queryTokens, index)
         if (score >= CONFIG.mergeThreshold && (!best || score > best.score)) best = { existing, score }
       }
       if (best) noteEvidence(best.existing, trace, 'failure', false, C.summarizeCalls(trace.calls || []))
@@ -622,6 +573,7 @@ export function apply(ctx) {
       }
 
       const queryTokens = tokens([draft.title, draft.taskType, draft.when, (draft.keywords || []).join(' ')].join(' '))
+      const index = R.corpus(store.experiences)
       const sameScope = (record) => !CONFIG.scopeMustMatch || !C.scopeConflicts(record, draft)
       const mergeable = ['verified', 'auto-verified', 'candidate']
       const runSteps = C.summarizeCalls(trace.calls || [])
@@ -639,7 +591,7 @@ export function apply(ctx) {
         if (!mergeable.includes(existing.status)) continue
         if (!sameScope(existing)) continue
         if (existing.taskType && draft.taskType && existing.taskType !== draft.taskType) continue
-        const score = scoreExperience(existing, queryTokens)
+        const score = scoreExperience(existing, queryTokens, index)
         if (score >= CONFIG.mergeThreshold && (!best || score > best.score)) best = { existing, score }
       }
       if (best && CONFIG.scopeMustMatch && C.scopeConflicts(best.existing, draft)) best = null
@@ -830,6 +782,39 @@ export function apply(ctx) {
     },
   })
 
+  ctx.tools.register({
+    name: 'experience_archive',
+    description:
+      'Move an experience out of the active library into store.archive (reversible with /memory restore <id>). ' +
+      'Use for records a human explicitly asked to remove; it never deletes data.',
+    parameters: {
+      id: { type: 'string', required: true, description: 'Experience id, e.g. exp-22' },
+      reason: { type: 'string', description: 'Why this record is being archived' },
+    },
+    output: {
+      schema: { type: 'string' },
+      render: (_args, value) => [{ type: 'text', text: String(value) }],
+    },
+    async execute(args) {
+      const index = store.experiences.findIndex((item) => item.id === args.id)
+      if (index < 0) return `未找到 ${args.id}`
+      const [target] = store.experiences.splice(index, 1)
+      store.archive = store.archive || []
+      store.archive.push({ ...target, archivedAt: new Date().toISOString() })
+      store.evictions = store.evictions || []
+      store.evictions.push({
+        id: target.id,
+        at: new Date().toISOString(),
+        by: 'human-request',
+        reason: String(args.reason || '人工要求移出活动库'),
+        rank: null,
+        ageDays: null,
+      })
+      persist()
+      return `已归档 ${target.id}（${target.title}）；活动库剩 ${store.experiences.length} 条，可用 /memory restore ${target.id} 恢复。`
+    },
+  })
+
   ctx.tools.register({
     name: 'experience_review',
     description:

package/lib/capture.js

@@ -39,6 +39,7 @@ export const DEFAULT_CONFIG = {
   stepsJaccard: 0.7,
   allowAutoVerified: false,
   minEffectiveSteps: 2,
+  recallThreshold: 0.15,
 }
 
 export function redact(text) {

package/lib/recall.js

@@ -0,0 +1,195 @@
+// Retrieval scoring: lexical tokens + a small hand-written feature layer.
+//
+// Why this exists: a task description and a stored recipe often share *meaning*
+// while sharing almost no words ("跑一个会踩坑的 .mjs 脚本，node 报错" vs
+// "pwsh / node.exe / load_workspace_dependencies"). Pure lexical coverage then
+// scores near zero and the recipe is never recalled.
+//
+// Two mechanisms, both synchronous and dependency-free:
+//   1. stop words — colloquial fillers ("随便", "故意", "测试场景") must not dilute coverage
+//   2. features   — canonical buckets ("#node", "#error", "#shell") that merge synonyms
+//
+// Features are derived from the raw text *before* stop-word removal, so a task
+// that explicitly says "测试" still produces "#test" even though "测试" is dropped
+// as a token.
+
+export const STOP_WORDS = new Set([
+  // function words
+  '的', '了', '和', '与', '在', '是', '把', '并', '就', '还', '也', '有', '用', '要', '中',
+  '这个', '那个', '一个', '一下', '我们', '你们', '可以', '需要', '请', '帮', '让我', '给我',
+  '然后', '现在', '已经', '还是', '就是', '不要', '不用', '没有', '什么', '怎么', '这样',
+  '那样', '但是', '因为', '所以', '如果', '或者', '以及', '并且', '而且', '之后', '之前',
+  // colloquial fillers that carry no retrieval signal
+  '随便', '随便写', '故意', '特意', '场景', '测试场景', '试试', '尝试', '看看', '跑一个',
+  '随便跑', '简单', '复杂', '大概', '可能', '应该', '一会', '等下',
+  // english
+  'the', 'a', 'an', 'and', 'or', 'to', 'of', 'in', 'on', 'for', 'with', 'is', 'are', 'be',
+  'it', 'this', 'that', 'then', 'use', 'run', 'can', 'will', 'should', 'just', 'some',
+])
+
+// Canonical bucket -> patterns that should activate it. Keep this list small and
+// conservative: a bucket that fires on everything destroys ranking signal.
+export const FEATURES = {
+  node: [/node(?:\.exe)?/, /\bmjs\b/, /\.mjs\b/, /\besm\b/, /内置\s*node/, /nodejs/],
+  shell: [/pwsh/, /powershell/, /\bshell\b/, /命令行/, /终端/, /cmd\b/],
+  error: [/报错/, /错误/, /\berror\b/, /exception/, /抛错/, /失败/, /\bfail/, /stack/],
+  module: [/模块/, /\bmodule\b/, /import\s/, /require\(/, /导出/, /export\s/],
+  test: [/测试/, /\btest\b/, /用例/, /\bspec\b/, /断言/, /assert/],
+  file: [/文件/, /脚本/, /\bfile\b/, /\.js\b/, /\.mjs\b/, /\.json\b/],
+  path: [/路径/, /目录/, /\bpath\b/, /folder/, /文件夹/],
+  python: [/python/, /\.py\b/, /pip\b/],
+  git: [/\bgit\b/, /提交/, /分支/, /仓库/, /commit/],
+  // Intent buckets: "run this script" and "add a function" are different tasks
+  // even when they mention the same files. Without these, a noisy description
+  // matches whichever record happens to share more generic nouns.
+  run: [/跑/, /运行/, /执行/, /\brun\b/, /execute/, /启动/],
+  write: [/新增/, /添加/, /创建/, /写入/, /\bwrite\b/, /\bcreate\b/, /实现/, /修改/, /重构/, /补/],
+  debug: [/排查/, /踩坑/, /修复/, /定位/, /\bdebug\b/, /为什么/, /失败原因/],
+}
+
+const FEATURE_PREFIX = '#'
+
+export function features(text) {
+  const source = String(text || '').toLowerCase()
+  const out = new Set()
+  for (const [name, patterns] of Object.entries(FEATURES)) {
+    if (patterns.some((re) => re.test(source))) out.add(FEATURE_PREFIX + name)
+  }
+  return out
+}
+
+export function tokenize(text) {
+  const out = new Set()
+  const source = String(text || '').toLowerCase()
+  for (const match of source.matchAll(/[a-z0-9_][a-z0-9_.\-/]*/g)) {
+    const token = match[0]
+    if (token.length >= 2 && !STOP_WORDS.has(token)) {
+      out.add(token)
+      // Index the basename too, so "truncate.test.mjs" matches "test/truncate.test.mjs".
+      const slash = token.lastIndexOf('/')
+      if (slash > 0 && token.length - slash > 2) out.add(token.slice(slash + 1))
+    }
+  }
+  const cjk = source.replace(/[^\u4e00-\u9fff]+/g, ' ')
+  for (const chunk of cjk.split(' ')) {
+    for (let i = 0; i + 1 < chunk.length; i++) {
+      const bigram = chunk.slice(i, i + 2)
+      if (!STOP_WORDS.has(bigram)) out.add(bigram)
+    }
+  }
+  return out
+}
+
+/** Tokens + feature buckets for one text (a query or a stored profile). */
+export function expand(text) {
+  const out = tokenize(text)
+  for (const feature of features(text)) out.add(feature)
+  return out
+}
+
+/** The searchable profile of a stored experience. */
+export function profileText(experience) {
+  if (!experience) return ''
+  // Pitfalls and counter-examples are usually the only place a record names the
+  // concrete failure ("MODULE_NOT_FOUND", "CommandNotFoundException"). Scoring
+  // without them ranks "add a file" chores above the actual troubleshooting note.
+  return [
+    experience.title,
+    experience.taskType,
+    experience.when,
+    (experience.keywords || []).join(' '),
+    (experience.pitfalls || []).join(' '),
+    (experience.reverse_examples || []).join(' '),
+  ].filter(Boolean).join(' ')
+}
+
+const round = (value) => Math.round(value * 1000) / 1000
+
+/**
+ * Document frequencies across the whole library. Rare tokens ("node.exe",
+ * "MODULE_NOT_FOUND") discriminate; common ones ("报错", "模块") must not.
+ * Without this, a noisy task description lifts every record at once.
+ */
+export function corpus(experiences) {
+  const df = new Map()
+  let n = 0
+  for (const experience of experiences || []) {
+    n += 1
+    for (const token of expand(profileText(experience))) df.set(token, (df.get(token) || 0) + 1)
+  }
+  return { n, df }
+}
+
+function idf(token, c) {
+  if (!c || !c.n) return 1
+  const d = c.df.get(token) || 0
+  return Math.log(1 + (c.n - d + 0.5) / (d + 0.5))
+}
+
+/** Feature buckets are hand-made and rare by construction: worth a little more than a plain token. */
+const FEATURE_WEIGHTS = {
+  '#error': 3,
+  '#node': 2.6,
+  '#module': 2.2,
+  '#shell': 2,
+  '#debug': 2,
+  '#run': 1.8,
+  '#write': 1.8,
+  '#test': 1.4,
+  '#file': 1.2,
+  '#path': 1.2,
+  '#python': 1.4,
+  '#git': 1.4,
+}
+
+const featureWeight = (token) =>
+  token.startsWith(FEATURE_PREFIX) ? FEATURE_WEIGHTS[token] || 1.4 : 1
+
+// A shared failure signal is worth more than any amount of shared nouns: the
+// user is asking for help with the same class of breakage.
+const CRITICAL_FEATURES = ['#error', '#node', '#module']
+
+/** CJK bigrams are noisy: 脚本 / 模块 / 报错 collide across unrelated tasks. */
+const scriptWeight = (token) => (/^[\u4e00-\u9fff]+$/.test(token) ? 0.45 : 1)
+
+/**
+ * Score a stored profile against an expanded query set.
+ * 0.75 * idf-weighted share of the query the profile covers
+ * 0.25 * unweighted share of the profile the query covers
+ * Both terms are bounded, so the result is always within [0, 1].
+ */
+export function score(profile, queryTokens, c) {
+  const expTokens = expand(profile)
+  if (!queryTokens || !queryTokens.size || !expTokens.size) return 0
+  let weightSum = 0
+  let weightHit = 0
+  for (const token of queryTokens) {
+    const weight = idf(token, c) * featureWeight(token) * scriptWeight(token)
+    weightSum += weight
+    if (expTokens.has(token)) weightHit += weight
+  }
+  const queryCoverage = weightSum ? weightHit / weightSum : 0
+  let expHits = 0
+  for (const token of expTokens) if (queryTokens.has(token)) expHits++
+  const expCoverage = expHits / expTokens.size
+  let critical = 0
+  for (const feature of CRITICAL_FEATURES) {
+    if (queryTokens.has(feature) && expTokens.has(feature)) critical += 1
+  }
+  const criticalBonus = critical ? 0.08 * critical : 0
+  return round(Math.min(1, 0.75 * queryCoverage + 0.25 * expCoverage + criticalBonus))
+}
+
+/** Human-readable breakdown for `/memory why` and `experience_debug`. */
+export function explain(profile, queryText) {
+  const queryTokens = expand(queryText)
+  const expTokens = expand(profile)
+  const shared = [...queryTokens].filter((t) => expTokens.has(t))
+  return {
+    score: score(profile, queryTokens),
+    queryTokens: [...queryTokens].slice(0, 40),
+    expTokens: [...expTokens].slice(0, 40),
+    shared: shared.slice(0, 20),
+    features: shared.filter((t) => t.startsWith(FEATURE_PREFIX)),
+  }
+}

package/lib/store.js

@@ -0,0 +1,66 @@
+// Durable store I/O, split out of index.js so the reload path is testable.
+//
+// The bug this exists to prevent: `loadStore()` used to rebuild only
+// {version,nextId,nextInjectionId,experiences,injections,turns}, so every
+// reload dropped `archive` and `evictions` — and the next persist() rewrote
+// them out of the file, silently losing archived records.
+
+import fs from 'node:fs'
+
+export function emptyStore() {
+  return {
+    version: 1,
+    nextId: 1,
+    nextInjectionId: 1,
+    experiences: [],
+    injections: [],
+    turns: [],
+    archive: [],
+    evictions: [],
+  }
+}
+
+const array = (value) => (Array.isArray(value) ? value : [])
+
+/**
+ * Rebuild a store from parsed JSON without dropping fields.
+ * Unknown extra fields are preserved verbatim on purpose: a plugin upgrade that
+ * adds a table must not lose it when an older build reloads the file.
+ */
+export function normalizeStore(parsed) {
+  if (!parsed || typeof parsed !== 'object') throw new Error('not an object')
+  const known = {
+    version: Math.max(2, Number(parsed.version || 1)),
+    nextId: Number(parsed.nextId) || 1,
+    nextInjectionId: Number(parsed.nextInjectionId) || 1,
+    experiences: array(parsed.experiences),
+    injections: array(parsed.injections),
+    turns: array(parsed.turns),
+    archive: array(parsed.archive),
+    evictions: array(parsed.evictions),
+  }
+  return { ...parsed, ...known }
+}
+
+/**
+ * Read a store file.
+ * @returns {{ store: object, error?: Error, missing?: boolean }}
+ */
+export function readStore(file) {
+  try {
+    const raw = fs.readFileSync(file, 'utf8')
+    return { store: normalizeStore(JSON.parse(raw)) }
+  } catch (error) {
+    if (error && error.code === 'ENOENT') return { store: emptyStore(), missing: true }
+    return { store: emptyStore(), error }
+  }
+}
+
+/** Atomic-ish write: temp file + rename, UTF-8, no BOM. */
+export function writeStore(file, store) {
+  const dir = file.slice(0, Math.max(file.lastIndexOf('/'), file.lastIndexOf('\\')))
+  if (dir) fs.mkdirSync(dir, { recursive: true })
+  const tmp = `${file}.tmp`
+  fs.writeFileSync(tmp, JSON.stringify(store, null, 2), 'utf8')
+  fs.renameSync(tmp, file)
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-experience",
-  "version": "1.0.4",
+  "version": "1.0.8",
   "private": false,
   "publishConfig": {
     "access": "public"

package/tools/e2e.mjs

@@ -186,7 +186,7 @@ await test('T46 decay report is read-only and restore refuses unknown ids', asyn
   const report = await cmd('decay --age 400');
   assert.equal(report.kind, 'success', report.text);
   assert.equal(readStore().experiences.length, before, 'report must not delete anything');
-  assert.equal(readStore().archive, undefined, 'nothing moved to archive');
+  assert.equal((readStore().archive || []).length, 0, 'nothing moved to archive');
   const restore = await cmd('restore exp-1');
   assert.equal(restore.kind, 'error');
   assert.match(restore.text, /archive 里没有/);

package/tools/selftest.mjs

@@ -9,6 +9,11 @@ import * as C from '../lib/capture.js'
 import * as D from '../lib/decay.js'
 import * as E from '../lib/evidence.js'
 import * as L from '../lib/lookup.js'
+import * as R from '../lib/recall.js'
+import * as S from '../lib/store.js'
+import fs from 'node:fs'
+import os from 'node:os'
+import path from 'node:path'
 
 let pass = 0
 const failures = []
@@ -41,6 +46,7 @@ test('config defaults are the agreed numbers', () => {
     stepsJaccard: 0.7,
     allowAutoVerified: false,
     minEffectiveSteps: 2,
+    recallThreshold: 0.15,
   })
   assert.equal(C.DEFAULT_CONFIG.autoPromote, false, 'policy must default to off')
   assert.equal(C.DEFAULT_CONFIG.allowAutoVerified, false, 'auto-verified must never be recalled by default')
@@ -912,6 +918,113 @@ test('T62 renderReview lists rule-rejected records without offering them', () =>
   assert.match(text, /已否决.*1 条/)
 })
 
+// ---------------------------------------------------------------- recall scoring
+
+test('T63 stop words do not dilute the query', () => {
+  const noisy = R.expand('随便跑一个测试场景，故意让它报错')
+  for (const filler of ['随便', '故意', '场景', '测试场景']) {
+    assert.equal(noisy.has(filler), false, filler + ' must be dropped')
+  }
+  assert.equal(noisy.has('#error'), true, 'feature survives even though the words are dropped')
+})
+
+test('T64 vocabulary mismatch still scores through feature buckets', () => {
+  const profile = 'pwsh node.exe load_workspace_dependencies 模块 报错'
+  const query = '跑一个会踩坑的 .mjs 脚本，内置 node 模块报错'
+  const result = R.explain(profile, query)
+  assert.ok(result.score >= 0.15, 'score ' + result.score + ' must clear the default floor')
+  assert.ok(result.features.includes('#node'), 'node feature must match, got ' + result.features.join(','))
+  assert.ok(result.features.includes('#error'), 'error feature must match, got ' + result.features.join(','))
+})
+
+test('T65 unrelated query still scores near zero', () => {
+  const result = R.explain('pwsh node.exe 模块 报错', '把三张图片重命名为 image_001.png')
+  assert.ok(result.score < 0.15, 'score ' + result.score + ' must stay below the floor')
+})
+
+test('T66 score is bounded and self-match is high', () => {
+  const profile = '新增 truncate 函数 并配 node:test 用例'
+  const partial = R.score(profile, R.expand('新增 truncate 函数并配用例'))
+  const self = R.score(profile, R.expand(profile))
+  assert.ok(partial > 0.3 && partial <= 1, 'partial match scored ' + partial)
+  assert.ok(self <= 1, 'self match must not exceed 1, got ' + self)
+})
+
+test('T67 tokenizer keeps file names', () => {
+  const tokens = R.tokenize('node test/truncate.test.mjs && echo done')
+  assert.equal(tokens.has('truncate.test.mjs'), true)
+  assert.equal(tokens.has('node'), true)
+})
+
+test('T68 a query with #error favours the record that documents the failure', () => {
+  const withPitfall = {
+    id: 'exp-a',
+    title: '在 DSH 会话里运行 Node 脚本',
+    taskType: 'node/run',
+    when: '在 desktop 里 跑 .mjs 脚本验证时',
+    keywords: ['node', 'node.exe', 'PATH', 'MODULE_NOT_FOUND'],
+    pitfalls: ['require 一个不存在的模块只会得到 MODULE_NOT_FOUND 报错'],
+    reverse_examples: [],
+  }
+  const chore = {
+    id: 'exp-b',
+    title: '新增 index',
+    taskType: 'node-project/feature',
+    when: '在 desktop 里 新增函数/模块并配用例',
+    keywords: ['node', 'src/index.js', 'test/<name>.mjs'],
+    pitfalls: [],
+    reverse_examples: [],
+  }
+  const index = R.corpus([withPitfall, chore])
+  const query = R.expand('随便写个 .mjs 脚本，require 一个不存在的模块，跑一下报错，然后自己排查修复')
+  const hit = R.score(R.profileText(withPitfall), query, index)
+  const miss = R.score(R.profileText(chore), query, index)
+  assert.equal(query.has('#error'), true, 'query must carry #error')
+  assert.ok(hit > miss, `pitfall record ${hit} must beat chore ${miss}`)
+})
+
+test('T69 reload preserves archive and evictions', () => {
+  const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'exp-store-'))
+  const file = path.join(dir, 'store.json')
+  const store = S.emptyStore()
+  store.experiences = [{ id: 'exp-1', status: 'verified', title: 'kept' }]
+  store.archive = [{ id: 'exp-9', title: 'archived' }]
+  store.evictions = [{ id: 'exp-9', by: 'human-request', at: '2026-11-09T00:00:00.000Z' }]
+  store.nextId = 10
+  S.writeStore(file, store)
+
+  const reloaded = S.readStore(file)
+  assert.equal(reloaded.error, undefined, 'clean file must load without error')
+  assert.equal(reloaded.store.experiences.length, 1)
+  assert.deepEqual(reloaded.store.archive.map((e) => e.id), ['exp-9'], 'archive must survive')
+  assert.equal(reloaded.store.evictions.length, 1, 'evictions must survive')
+  assert.equal(reloaded.store.nextId, 10, 'counters must survive')
+  fs.rmSync(dir, { recursive: true, force: true })
+})
+
+test('T70 unknown fields survive a reload', () => {
+  const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'exp-store-'))
+  const file = path.join(dir, 'store.json')
+  fs.writeFileSync(file, JSON.stringify({ experiences: [], futureTable: [{ id: 'x' }] }), 'utf8')
+  const reloaded = S.readStore(file)
+  assert.deepEqual(reloaded.store.futureTable, [{ id: 'x' }], 'an older build must not drop new fields')
+  assert.deepEqual(reloaded.store.archive, [], 'missing archive defaults to empty')
+  fs.rmSync(dir, { recursive: true, force: true })
+})
+
+test('T71 a corrupt store degrades instead of throwing', () => {
+  const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'exp-store-'))
+  const file = path.join(dir, 'store.json')
+  fs.writeFileSync(file, '{ not json', 'utf8')
+  const result = S.readStore(file)
+  assert.ok(result.error instanceof Error, 'error must be reported, not thrown')
+  assert.equal(result.store.experiences.length, 0, 'falls back to an empty store')
+  const missing = S.readStore(path.join(dir, 'nope.json'))
+  assert.equal(missing.missing, true)
+  assert.deepEqual(missing.store.archive, [])
+  fs.rmSync(dir, { recursive: true, force: true })
+})
+
 // ---------------------------------------------------------------- summary
 
 const total = pass + failures.length