dsh-tool-folder 0.2.5 → 0.3.2

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 hfhh32
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -107,7 +107,10 @@ were never unregistered, so nothing else changes.
 | include | [] | whitelist: exact name or `prefix*`. Non-empty → only matching tools are injected (meta tools exempt; deny still wins). Execution is NOT gated by include. |
 | toonifyResults | false | compact long JSON results after execution (drop empty fields + truncate strings; only when the text block is >2000 chars and parses as JSON) |
 | maxFoldMs | 50 | hard cap; beyond this keep the full list |
-| semanticEnabled | true | **v0.1.8**: tools_search semantic leg — BM25 + local bge-m3 (Ollama) RRF hybrid. Chinese intents hit English-only tools (the documented BM25 gap). Index lazily built + disk-cached (`~/.dsh/state/semantic-cache.json`); Ollama offline/timeout degrades to BM25-only, never slower or worse. |
+| semanticEnabled | true | 语义腿总开关。检索融合顺序：**本地静态嵌入（`lib/static-embed/`）→ 远程 embedding → 本地 Ollama → 纯 BM25**；任一层缺失或失败自动降级，绝不比纯 BM25 更慢或更差 |
+| staticWeight | 0.15 | **v0.3.2**: 本地静态嵌入腿在 RRF 中的权重。实测最优 0.15 —— 静态腿只负责补词法查不到的坑，压过词法会引入噪声（留出集 w=0.15 → recall@3 0.621，w=2 → 0.517） |
+| staticEmbedDir | '' | **v0.3.2**: 静态嵌入目录（含 `vocab.json` / `vectors.bin` / `scales.bin` / `meta.json`）。留空 = 用插件自带的 `lib/static-embed`；目录不存在则该腿自动跳过 |
+| semanticWeight | 3 | **远程** embedding 腿在 RRF 中的权重：1=等权，3=偏语义（默认），0=纯 BM25。与 `staticWeight` 分开配置，因为远程 bge-m3 质量更高 |
 | ollamaBase | http://127.0.0.1:11434 | Ollama endpoint for the semantic leg |
 | embedModel | bge-m3 | embedding model for the semantic leg |
 | feedbackFile | '' | feedback JSON path (default `$DSH_HOME/logs/tool-folder/`) |
@@ -115,7 +118,11 @@ were never unregistered, so nothing else changes.
 
 ## Files
 
-- `lib/bm25.js` — zero-dep BM25 (k1=1.2, b=0.75; CN bigrams + EN tokens)
+- `lib/bm25.js` — zero-dep BM25 (k1=1.2, b=0.75; camelCase split + CN bigrams + EN tokens)
+- `lib/expand.js` — **v0.3.1**: CN/EN dev-intent thesaurus + reverse alias expansion
+- `lib/rerank.js` — **v0.3.1**: body-level rerank (weight 0.45) + optional RRF fusion
+- `lib/static-embed.js` — **v0.3.2**: offline static embedding (lookup + mean-pool + cosine, zero deps)
+- `lib/static-embed/` — **v0.3.2**: 7760 × 256 int8 table, 2.01 MB, built by `tools/build-static-embed.py`
 - `lib/semantic.js` — v0.2.0 semantic leg: local bge-m3 embeddings (Ollama) + RRF fusion, disk-cached
 - `lib/chainguard.js` — ChainGuard firewall: literal patterns + exfil-chain detection + deobfuscation
 - `lib/obfuscation.js` — v0.2.0 anti-obfuscation: concat rebuild, `-enc`/certutil/bitsadmin shapes, homoglyphs
@@ -149,6 +156,106 @@ were never unregistered, so nothing else changes.
 
 ## Changelog
 
+### v0.3.2 — 离线静态嵌入语义腿（不装 Ollama / 不续 Ark 也能有语义，2026-10-31）
+
+**动机**：v0.3.0 的语义腿两条路全断——Ark `doubao-embedding` 返回 400
+`InvalidSubscription`（订阅过期），Ollama 未安装（`11434` HTTP 000）。
+在 29 条**留出查询**上实测，`v0.3.1` 的纯词法+词表只能到 **recall@3 = 0.448**，
+剩下 55% 是词表覆盖不到的改写。
+
+**做法**（`MinishLab/model2vec` 思路：把 sentence transformer 压成一张查找表）：
+
+- 源模型 `minishlab/potion-multilingual-128M`（分词器 `BAAI/bge-m3`），
+  50 万 token × 256 维 = 512 MB fp32
+- **剪枝**：只保留 ① 领域文本用到的 token ② 全部单字 CJK（7688）→ 7760 行
+- **int8 量化**：每行独立 scale，行先 L2 归一化，平均余弦保真度 **0.99995**
+- 产物 **2.01 MB**（`lib/static-embed/`），运行期只做 查表 + 均值池化 + 余弦
+  —— 无推理图、无 onnx、无 WASM、无网络、无外部依赖，单次 embed < 1 ms
+
+构建脚本 `tools/build-static-embed.py`（纯本地计算，读本地 safetensors）。
+
+**融合权重单列**：新增 `staticWeight`（默认 **0.15**），与远程腿的
+`semanticWeight`（默认 3）分开。整组压测（`eval/sweep-static.mjs`，留出集）：
+
+| bm25 : static | 1:0 | **1:0.15** | 1:0.3 | 1:0.5 | 1:1 | 1:2 |
+|---|---|---|---|---|---|---|
+| recall@3 | 0.448 | **0.621** | 0.586 | 0.586 | 0.552 | 0.517 |
+
+**静态腿的价值是「补词法查不到的坑」，不是压过词法**——权重一高，它的噪声就会
+把本来对的词法结果挤下去。本地腿同时固定 `bmWeight = 1`，不叠加 `adaptiveBm` 的中文降权。
+
+**实测（29 条留出查询 / 40 条标注查询）**
+
+| 指标 | v0.3.1 | v0.3.2 |
+|---|---|---|
+| 留出集 recall@3 | 0.448 | **0.621** |
+| 留出集 recall@6 | 0.448 | **0.655** |
+| 留出集 MRR | 0.379 | **0.506** |
+| 标注集 recall@3 | 1.000 | 1.000 |
+| 标注集 MRR | 0.888 | **0.904** |
+
+**修掉一个会让结果变差的隐患**：查询向量为空时（整句 token 全在词表外）原会照常进融合，
+所有文档余弦都是 0，排序退化为插入顺序，给 topK 塞进一批无关工具。现在直接跳过本地腿。
+
+**测过但否掉的方案**（负结果同样重要）：等权/高权重 RRF、按词法证据量自适应打折、
+SIF 加权池化 + 去主成分——均无可靠收益，详见 `CHANGELOG-v0.3.2.md`。
+
+### v0.3.1 — 分词器 + 意图词表 + 精排权重（2026-10-31）
+
+新增 `eval/` 评测基础设施（36 个混合工具 / 40 条查询 + 29 条留出集 /
+`run.mjs` `rerank.mjs` `sweep.mjs`），此后每次调参都有数据依据。
+
+1. **`bm25.js` 修分词**：camelCase 拆分原先在 `toLowerCase()` 之后跑，
+   `searchNotes` 变成整串 `searchnotes`，查 "notes" 零命中；补 CJK 高频功能词
+   bigram 黑名单（`这个/一下/怎么` 等，IDF≈0 却撑大 `avgdl`）。
+2. **`expand.js`（新增）**：约 60 组中英开发意图词表 + 反向别名扩展。
+   这是唯一能救「首轮零命中」的手段——标注集 recall@3 0.725 → 1.000。
+3. **`rerank.js`**：修掉权重天花板（w=0.35 时元数据第 0 位底分 0.65、正文满分仅 0.35，
+   正文以 0.91 输给 0.00），扫描最优值 **0.45**；新增 `fusion: 'rrf'` 选项。
+4. **`index.js`**：把 `eq` 拆成 `eqStr`（喂语义腿，字符串）与 `eq`（喂 BM25，token 数组），
+   修掉一处「语义腿一激活就崩」的隐患。
+
+### v0.3.0 — 语义腿真正可用 + 融合修正（2026-10-17）
+
+**动机（实测发现的真问题，不是推测）**：本机没装 Ollama（`127.0.0.1:11434` 返回
+HTTP 000），`embedText` 恒为 null → `tools_search` 实际一直是**纯 BM25**。在 43 个
+真实工具（`tools-real.json`）、18 条真实中文意图上测：**Hit@1 只有 6/18（33%）**。
+
+1. **远程 embedding 腿（`lib/embed-remote.js` 新增）**：OpenAI 兼容端点，默认火山
+   `doubao-embedding-vision`（2048 维），key 读 `DSH_EMBED_KEY` / `OPENVIKING_ARK_KEY`。
+   带熔断（连续 3 次失败 → 5 分钟冷却）+ 批量（4 条/批、2 并发、429 退避 0.8/2/5s）。
+   纯语义 Hit@1 **11/18（61%）**。
+2. **加权 RRF + 自适应 BM25 权重**：语义腿默认权重 3，BM25 默认 0.25；
+   若查询里的 ASCII 词命中 BM25 首条的名称/描述（真实词匹配、工具名、server 名），
+   BM25 权重回升到 1。实测 1:1 融合只有 8/18，加权后 **11/18**。
+3. **重复 id 修复（关键）**：语料里同名工具出现两次时，旧 RRF 会把同一 id 的分累加
+   两次——`list_sessions` / `get_config` 各两份，直接把正确项从第一顶到第三以外
+   （纯语义 11/18 被拉到 2/18）。现在 bm25/语义两腿都按 id 去重，组装侧也先按名去重。
+4. **索引可用性修复**：旧实现要求**全部** doc 成功才落缓存（一次失败就永远重建），
+   且 60s 硬截止串行调用；改为批量并发 + **部分成功也落盘** + 覆盖率 ≥80% 即可用。
+5. **缓存路径修正**：`~/.dsh/state/` → 优先 `$DSH_HOME/state/`（DSH_HOME 指向
+   `.dsh-community` 时旧实现会把缓存写到另一个目录）。缓存身份键带 provider，
+   远程/本地切换不会复用错向量。
+6. **索引不再阻塞**：在 assemble 后台构建，`tools_search` 绝不等待冷启动（旧实现
+   首次搜索可能卡 30s+）；查询向量有内存缓存 + 15s 节流后台预热。
+7. **正文精排（`lib/rerank.js` 新增，默认关）**：借鉴 SkillRouter 的两阶段检索，
+   在候选集内用「完整描述 + 参数名 + 参数描述 + 枚举」重排。**实测在本语料上
+   Hit@1 持平、Hit@3 反而 -1**（15→14），所以默认关闭，`bodyRerank: true` 可开。
+8. **语义腿参与自动注入**（`semanticDynamic`，默认开）：只用已缓存的查询向量，
+   零额外延迟；命中就把结果并进动态段。
+9. `debugDump`：把当前工具语料 dump 成 JSON，供离线评测复现。
+10. 测试 114 → **123 条**，全绿；新增 `scripts/eval-tools.mjs` 离线评测脚本。
+
+**实测汇总（43 工具 / 18 中文意图）**
+
+| 方案 | Hit@1 | Hit@3 |
+|---|---|---|
+| BM25（v0.2.5 实际状态） | 6/18 | 8/18 |
+| RRF 1:1 | 8/18 | 12/18 |
+| 语义单腿（参考上限） | 11/18 | 15/18 |
+| **v0.3.0 默认（语义3 : BM25 0.25 自适应）** | **11/18** | **15/18** |
+| 再加正文精排 0.35 | 11/18 | 14/18 |
+
 ### v0.2.5 — red-team firewall fixes + retrieval hardening (2026-09-02)
 1. **Exec-tool recognition (red-team F1)** — bare-name matching let MCP-prefixed
    exec tools (`mcp__windows__Cmd`) bypass the high-risk gate; `isExecTool` now

package/lib/bm25.js

@@ -1,127 +1,207 @@
-/**
- * bm25.js — zero-dependency BM25 lexical search for tool retrieval.
- *
- * VENDOR SOURCE (2026-09-02): dsh-skill-folder/lib/bm25.js vendors this file
- * verbatim (comments may diverge per repo). Modify here, then mirror the code
- * change into the skill repo. Sync gate: `node E:\DSH-Data\scripts\check-vendor-sync.mjs`.
- * semantic.js is intentionally DIVERGED (tool has incremental cache/sha1) and
- * is NOT covered by the gate.
- *
- * Design (ported from the xiaowan agent's ToolBM25Index idea; the algorithm is
- * the reference, this code is a clean-room JS implementation):
- *   - BM25 with standard k1=1.2, b=0.75 (RAG-MCP / BoR validated the
- *     retrieval-first approach; BM25 is the cheap lexical leg).
- *   - Tokenizer handles mixed CN/EN: ASCII words + CJK bigrams. No jieba, no
- *     network, no dependencies — must run inside the DSH plugin runtime.
- *   - Synchronous and fast: indexing ~50 tools is <1ms, a query <1ms.
- *
- * Cordis sandbox note: this module only does string/number math. It never
- * touches fs/net/process, so it is safe in any host or sandbox context.
- */
-
-const K1 = 1.2;
-const B = 0.75;
-
-const EN_STOP = new Set([
-  "the", "a", "an", "and", "or", "of", "to", "in", "on", "for", "with",
-  "at", "by", "from", "as", "is", "are", "was", "were", "be", "been",
-  "it", "this", "that", "these", "those", "i", "you", "he", "she", "we",
-  "they", "do", "does", "did", "have", "has", "had", "not", "no", "yes",
-  "will", "would", "can", "could", "should", "may", "might", "must",
-  "your", "my", "our", "their", "its", "all", "any", "some", "each",
-]);
-
-/** Tokenize mixed CN/EN text into lowercase tokens. */
-function tokenize(text) {
-  const out = [];
-  const s = String(text || "").toLowerCase();
-  for (const m of s.matchAll(/[a-z0-9][a-z0-9_+-]{1,}/g)) {
-    const t = m[0];
-    if (!EN_STOP.has(t)) out.push(t);
-    // Sub-words of hyphen/underscore compounds. Without this, a tool name like
-    // "mcp__open-design__start_run" is ONE term, so querying "design" scores
-    // zero. The whole token is kept too, so exact-name matches and existing
-    // IDF are unchanged; sub-words only ADD recall.
-    if (t.length > 2 && (t.includes("-") || t.includes("_"))) {
-      for (const part of t.split(/[-_]+/)) {
-        if (part.length > 1 && part !== t && !EN_STOP.has(part)) out.push(part);
-      }
-    }
-  }
-  // CJK bigrams: "开源框架" -> "开源","源框","框架". Bigrams beat single
-  // chars for short tool descriptions and are dependency-free.
-  const cjk = s.replace(/[^\u4e00-\u9fff]/g, "");
-  if (cjk.length >= 2) {
-    for (let i = 0; i + 2 <= cjk.length; i++) out.push(cjk.slice(i, i + 2));
-  }
-  return out;
-}
-
-/** Build a BM25 index over an array of docs. Each doc: {id, text, ...}. */
-function buildIndex(docs) {
-  const n = docs.length;
-  const docTokens = [];
-  const df = new Map(); // term -> #docs containing it
-  let totalLen = 0;
-
-  for (const doc of docs) {
-    const tokens = tokenize(doc.text);
-    docTokens.push(tokens);
-    totalLen += tokens.length;
-    const seen = new Set();
-    for (const t of tokens) {
-      if (!seen.has(t)) {
-        seen.add(t);
-        df.set(t, (df.get(t) || 0) + 1);
-      }
-    }
-  }
-  const avgdl = n > 0 ? totalLen / n : 1;
-
-  return { n, docTokens, df, avgdl };
-}
-
-/** Score one doc's token list against the query's IDF map. */
-function score(tokens, queryIdf, n, avgdl) {
-  const tf = new Map();
-  for (const t of tokens) tf.set(t, (tf.get(t) || 0) + 1);
-  let acc = 0;
-  const dl = tokens.length;
-  for (const [term, idf] of queryIdf) {
-    const f = tf.get(term);
-    if (!f) continue;
-    acc += idf * ((f * (K1 + 1)) / (f + K1 * (1 - B + (B * dl) / avgdl)));
-  }
-  return acc;
-}
-
-/**
- * Search an index. Returns up to topK doc ids sorted by BM25 score desc.
- * @param index   result of buildIndex
- * @param query   raw query string (mixed CN/EN ok)
- * @param topK    max results (default 10)
- * @param opts    {filter: (docId) => boolean}
- */
-function search(index, query, topK = 10, opts = {}) {
-  const { n, docTokens, df, avgdl } = index;
-  if (!n || !query) return [];
-  const qTerms = tokenize(query);
-  if (qTerms.length === 0) return [];
-
-  const queryIdf = new Map();
-  for (const t of qTerms) {
-    const d = df.get(t) || 0;
-    // BM25+ style: add 1 inside the log to keep IDF positive for unseen terms.
-    queryIdf.set(t, Math.log((n - d + 0.5) / (d + 0.5) + 1));
-  }
-
-  const scored = [];
-  for (let i = 0; i < n; i++) {
-    const s = score(docTokens[i], queryIdf, n, avgdl);
-    if (s > 0 && (!opts.filter || opts.filter(i))) scored.push([i, s]);
-  }
-  scored.sort((a, b) => b[1] - a[1]);
-  return scored.slice(0, topK).map(([i]) => i);
-}
-
-export { tokenize, buildIndex, search, score };
+/**
+ * bm25.js — zero-dependency BM25 lexical search for tool retrieval.
+ *
+ * VENDOR SOURCE (2026-09-02): dsh-skill-folder/lib/bm25.js vendors this file
+ * verbatim (comments may diverge per repo). Modify here, then mirror the code
+ * change into the skill repo. Both copies are kept in sync by the maintainer.
+ * semantic.js is intentionally DIVERGED (tool has incremental cache/sha1) and
+ * is NOT covered by the gate.
+ *
+ * Design (ported from the xiaowan agent's ToolBM25Index idea; the algorithm is
+ * the reference, this code is a clean-room JS implementation):
+ *   - BM25 with standard k1=1.2, b=0.75 (RAG-MCP / BoR validated the
+ *     retrieval-first approach; BM25 is the cheap lexical leg).
+ *   - Tokenizer handles mixed CN/EN: ASCII words + CJK bigrams. No jieba, no
+ *     network, no dependencies — must run inside the DSH plugin runtime.
+ *   - Synchronous and fast: indexing ~50 tools is <1ms, a query <1ms.
+ *
+ * Cordis sandbox note: this module only does string/number math. It never
+ * touches fs/net/process, so it is safe in any host or sandbox context.
+ *
+ * ---- 2026-10-31 升级（v0.3.1）----
+ *   1. camelCase / PascalCase 拆分：`searchNotes` 之前的正则在 lowercase 后
+ *      拿到的是整串 `searchnotes`，查询 "notes" 零命中；现在先按驼峰拆再降小写。
+ *   2. CJK 功能词 bigram 过滤：「这个 / 一下 / 怎么 / 帮我」这类 bigram 几乎每句
+ *      都有，IDF 接近 0 却会撑大 avgdl，把真正有区分度的词稀释掉。
+ *   3. BM25+ delta 项（可选，默认 0）：标准 BM25 对「命中但频次低」的文档给分
+ *      偏低；delta 给每个命中词一个下限分，短查询召回更稳。
+ *      参考 ratel ADR-0023 的实测：改 b 只重排已召回的（recall 不变、MRR 上升），
+ *      而 delta 是真正补召回的手段，两者不冲突。
+ *   4. k1 / b 可配置（默认仍 1.2 / 0.75，与 ratel 收敛后的取值一致）。
+ */
+
+const K1 = 1.2;
+const B = 0.75;
+
+const EN_STOP = new Set([
+  "the", "a", "an", "and", "or", "of", "to", "in", "on", "for", "with",
+  "at", "by", "from", "as", "is", "are", "was", "were", "be", "been",
+  "it", "this", "that", "these", "those", "i", "you", "he", "she", "we",
+  "they", "do", "does", "did", "have", "has", "had", "not", "no", "yes",
+  "will", "would", "can", "could", "should", "may", "might", "must",
+  "your", "my", "our", "their", "its", "all", "any", "some", "each",
+]);
+
+/**
+ * 中文高频功能词 bigram。这些词在几乎每个中文工具描述/查询里都出现，
+ * 信息量接近 0，留在索引里只会抬高 avgdl、稀释有区分度的词。
+ * 只过滤 bigram 层，不影响单字信息（单字本来就不进索引）。
+ */
+const CJK_STOP_BIGRAM = new Set([
+  "这个", "那个", "这些", "那些", "一个", "一下", "一种", "一些",
+  "什么", "怎么", "怎样", "如何", "可以", "是否", "然后", "所以",
+  "但是", "而且", "或者", "还是", "如果", "因为", "虽然", "已经",
+  "我们", "你们", "他们", "自己", "现在", "刚才", "帮我", "给我",
+  "看看", "试一下", "用的", "的是", "了的", "了吗", "的话",
+  "这里", "那里", "里面", "上面", "下面", "之后", "之前", "时候",
+]);
+
+/** camelCase / PascalCase / 数字边界拆分：searchNotes -> search, notes */
+function splitCamel(word) {
+  if (!/[A-Z]/.test(word)) return [];
+  // 在「小写/数字 → 大写」和「连续大写 → 大写+小写」处断
+  const parts = word
+    .replace(/([a-z0-9])([A-Z])/g, "$1 $2")
+    .replace(/([A-Z]+)([A-Z][a-z])/g, "$1 $2")
+    .split(/\s+/)
+    .filter(Boolean);
+  return parts.length > 1 ? parts : [];
+}
+
+/** Tokenize mixed CN/EN text into lowercase tokens. */
+function tokenize(text) {
+  const out = [];
+  const raw = String(text || "");
+  const lower = raw.toLowerCase();
+
+  // --- ASCII（在原文上跑，保留大小写信息以便拆驼峰）---
+  for (const m of raw.matchAll(/[A-Za-z0-9][A-Za-z0-9_+-]{1,}/g)) {
+    const word = m[0];
+    const t = word.toLowerCase();
+    if (!EN_STOP.has(t)) out.push(t);
+
+    // 驼峰拆分（v0.3.1 新增）：searchNotes -> searchnotes + search + notes
+    for (const part of splitCamel(word)) {
+      const p = part.toLowerCase();
+      if (p.length > 1 && p !== t && !EN_STOP.has(p)) out.push(p);
+    }
+
+    // 连字符/下划线拆分。没有这一步，工具名
+    // "mcp__open-design__start_run" 是 ONE term，查 "design" 零分。
+    // 整词保留，子词只增加召回，不动精确匹配与既有 IDF。
+    if (t.length > 2 && (t.includes("-") || t.includes("_"))) {
+      for (const part of t.split(/[-_]+/) ) {
+        if (part.length > 1 && part !== t && !EN_STOP.has(part)) out.push(part);
+      }
+    }
+  }
+
+  // --- CJK bigram ---
+  const cjk = lower.replace(/[^\u4e00-\u9fff]/g, "");
+  if (cjk.length >= 2) {
+    for (let i = 0; i + 2 <= cjk.length; i++) {
+      const bg = cjk.slice(i, i + 2);
+      if (!CJK_STOP_BIGRAM.has(bg)) out.push(bg);
+    }
+  }
+  return out;
+}
+
+/** Build a BM25 index over an array of docs. Each doc: {id, text, ...}. */
+function buildIndex(docs) {
+  const n = docs.length;
+  const docTokens = [];
+  const df = new Map(); // term -> #docs containing it
+  let totalLen = 0;
+
+  for (const doc of docs) {
+    const tokens = tokenize(doc.text);
+    docTokens.push(tokens);
+    totalLen += tokens.length;
+    const seen = new Set();
+    for (const t of tokens) {
+      if (!seen.has(t)) {
+        seen.add(t);
+        df.set(t, (df.get(t) || 0) + 1);
+      }
+    }
+  }
+  const avgdl = n > 0 ? totalLen / n : 1;
+
+  return { n, docTokens, df, avgdl };
+}
+
+/**
+ * Score one doc's token list against the query's IDF map.
+ * opts.delta 启用 BM25+ 的 TF 下限项（默认 0 = 标准 BM25）。
+ */
+function score(tokens, queryIdf, n, avgdl, opts = {}) {
+  const k1 = Number.isFinite(opts.k1) ? opts.k1 : K1;
+  const b = Number.isFinite(opts.b) ? opts.b : B;
+  const delta = Number.isFinite(opts.delta) ? opts.delta : 0;
+
+  const tf = new Map();
+  for (const t of tokens) tf.set(t, (tf.get(t) || 0) + 1);
+  let acc = 0;
+  const dl = tokens.length;
+  for (const [term, idf] of queryIdf) {
+    const f = tf.get(term);
+    if (!f) continue;
+    const sat = (f * (k1 + 1)) / (f + k1 * (1 - b + (b * dl) / avgdl));
+    acc += idf * (sat + delta);
+  }
+  return acc;
+}
+
+/**
+ * Search an index. Returns up to topK doc ids sorted by BM25 score desc.
+ * @param index   result of buildIndex
+ * @param query   raw query string (mixed CN/EN ok) 或已扩展的 token 数组
+ * @param topK    max results (default 10)
+ * @param opts    {filter: (i)=>boolean, k1, b, delta}
+ */
+function search(index, query, topK = 10, opts = {}) {
+  const { n, docTokens, df, avgdl } = index;
+  if (!n || !query) return [];
+
+  // 支持「已扩展的多词查询」：数组直接当 token 用，字符串走分词
+  const qTerms = Array.isArray(query) ? query : tokenize(query);
+  if (qTerms.length === 0) return [];
+
+  const queryIdf = new Map();
+  for (const t of qTerms) {
+    const d = df.get(t) || 0;
+    // BM25+ style: add 1 inside the log to keep IDF positive for unseen terms.
+    queryIdf.set(t, Math.log((n - d + 0.5) / (d + 0.5) + 1));
+  }
+
+  const scored = [];
+  for (let i = 0; i < n; i++) {
+    const s = score(docTokens[i], queryIdf, n, avgdl, opts);
+    if (s > 0 && (!opts.filter || opts.filter(i))) scored.push([i, s]);
+  }
+  scored.sort((a, b2) => b2[1] - a[1]);
+  return scored.slice(0, topK).map(([i]) => i);
+}
+
+/** 带分数的版本，供 RRM / 融合类算法使用。 */
+function searchScored(index, query, topK = 10, opts = {}) {
+  const { n, docTokens, df, avgdl } = index;
+  if (!n || !query) return [];
+  const qTerms = Array.isArray(query) ? query : tokenize(query);
+  if (qTerms.length === 0) return [];
+  const queryIdf = new Map();
+  for (const t of qTerms) {
+    const d = df.get(t) || 0;
+    queryIdf.set(t, Math.log((n - d + 0.5) / (d + 0.5) + 1));
+  }
+  const out = [];
+  for (let i = 0; i < n; i++) {
+    const s = score(docTokens[i], queryIdf, n, avgdl, opts);
+    if (s > 0 && (!opts.filter || opts.filter(i))) out.push([i, s]);
+  }
+  out.sort((a, b2) => b2[1] - a[1]);
+  return out.slice(0, topK);
+}
+
+export { tokenize, buildIndex, search, searchScored, score, splitCamel, CJK_STOP_BIGRAM };