@anionex/dsh-tool-search 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,12 @@
+# Changelog
+
+## 0.1.0 - 2026-09-05
+
+- Add deterministic local BM25 discovery for deferred DSH tools.
+- Add per-Agent monotonic selection with resume and fork recovery.
+- Commit selections only after a successful final result and persist the same canonical recovery payload.
+- Keep the real registry and approval, guard, event, and execution semantics intact.
+- Keep maintained DSH control-plane tools, including `report`, visible by default and reject unsupported code/PTC presentation modes.
+- Add live Web settings for Top-K, always-visible names, and never-search names.
+- Add responsive container-based settings layout and compatibility with DSH through `0.1.2-rc.1`.
+- Add Recall@1/Recall@5, lifecycle, pipeline, UI, package, token, and real Agent E2E acceptance.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Anionex
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,160 @@
+# dsh-tool-search
+
+Codex-style deferred tool loading for [DeepSeek Harness (DSH)](https://github.com/deepseek-ai/DeepSeek-Harness).
+
+`dsh-tool-search` keeps the real DSH tool registry intact while reducing the model-visible tool surface. A new Agent initially sees the maintained core tool set plus `tool_search`. Calling `tool_search` runs deterministic local BM25 over deferred metadata, adds the Top-K matches to that Agent's selected set, and exposes their complete native schemas on the next model call. The model then calls each tool by its real name through the normal DSH execution pipeline.
+
+## Install
+
+```bash
+dsh plugin --profile web add @anionex/dsh-tool-search
+```
+
+Use `--profile headless` or another DSH profile name when appropriate. Restart a running profile and create a new session after installation.
+
+On affected Node.js 24+ standalone DSH launchers, a bare external Profile package can fail to resolve. If startup reports `Cannot find package '@anionex/dsh-tool-search'`, use `node --expose-internals "$(command -v dsh)" --profile <name> ...`; this is an upstream DSH loader limitation rather than a plugin resolution fallback.
+
+Requirements:
+
+- DSH `0.1.1-rc.1` through the current `0.1.2-rc.1` prerelease line
+- Node.js `^22.19.0 || >=24.0.0`
+- Native tool presentation mode
+
+## Behavior
+
+The initial native schema surface contains:
+
+- `tool_search`
+- `apply_patch`
+- DSH core tools maintained in `DEFAULT_CORE_TOOLS`
+- exact names added to `alwaysVisible`
+
+Version `0.1.0` treats these names as core:
+
+```text
+apply_patch, ask_user_question, bash, create_goal, dsh_im_return_file,
+exit_plan_mode, get_goal, interrupt_agent, job_kill, job_list, job_output,
+list_agents, ralph, report, send_message, skill, subagent, subagent_fork,
+todo_write, update_goal, workflow
+```
+
+Every other registered tool is searchable but initially omitted. `neverSearch` hides a name from both the initial surface and the search corpus; it wins a conflict with `alwaysVisible`. `tool_search` itself cannot be blacklisted, preventing a configuration dead end.
+
+The selected set is:
+
+- isolated per live Agent;
+- monotonic during that Agent lifecycle;
+- reconstructed from successful `tool_search` results on resume;
+- reconstructed from the copied history prefix on fork;
+- re-evaluated against live settings without deleting its history.
+
+Tool and prompt registry changes invalidate every cached catalog. The next assembly or search rebuilds from current scoped schemas.
+
+## Settings
+
+The DSH Web settings page exposes:
+
+- default result count, `1-20` and `5` by default;
+- an exact-name always-visible allowlist;
+- an exact-name never-search denylist;
+- a live, filterable registry table with per-tool policy controls.
+
+Equivalent settings:
+
+```yaml
+- id: dsh-tool-search
+  config:
+    defaultLimit: 5
+    alwaysVisible:
+      - my_frequent_tool
+    neverSearch:
+      - internal_debug_tool
+```
+
+## Search Contract
+
+Each function search document contains, in order:
+
+1. `tool.name`
+2. `tool.name.replaceAll("_", " ")`
+3. `tool.description`
+4. a recursive schema traversal that emits each node description, then each sorted property name immediately before that property's nested descriptions; `items` and `anyOf` follow
+
+The local index uses BM25 with `k1=1.2` and `b=0.75`, English stop-word removal, and Porter2 stemming. Exact raw-name matches are always ranked first. Remaining results use descending BM25 score, then code-point lexical tool name as a stable tie-break. Search results contain only names, short descriptions, and scores; they never duplicate parameter schemas.
+
+## Codex Alignment
+
+The implementation was checked against `openai/codex` commit [`3fde89f`](https://github.com/openai/codex/commit/3fde89f628281b9b049376fb0cec4d1577bfeaec), the latest observed `main` commit during the source review. Codex's generic function document builder is the source for the field order and recursive schema traversal. Codex also uses `bm25` `2.3.2` with its English tokenizer and the same BM25 defaults.
+
+Intentional DSH adaptations:
+
+| Area | Codex | This plugin |
+| --- | --- | --- |
+| Default Top-K | 8 in current upstream | 5 |
+| Loading protocol | Responses API `tool_search_output` carries loadable definitions in history | ordinary DSH `tool_search` returns summaries; native schemas appear in the next request |
+| Loaded state | definitions remain usable while retained in model history | per-Agent monotonic selected set recovered from DSH session events |
+| Equal-score order | upstream candidate order can be unstable | lexical name tie-break |
+| Exact name | ordinary BM25 ranking | explicit Recall@1 guarantee |
+| Transliteration | Rust `deunicode` | Unicode NFKD diacritic folding |
+| Result payload | definitions, no scores | summaries and scores, no schemas |
+
+These differences are deliberate. DSH has no Responses API `tool_search_output` history item, while the requested behavior requires stable selected-set recovery and schema injection on the next model turn.
+
+## Execution Semantics
+
+This plugin filters only `PromptAssembly.tools` and matching `tool:<name>` guidance sections. It does not unregister tools and does not call `ctx.tools.restrict()`.
+
+After discovery, a real tool call still follows DSH's standard chain:
+
+```text
+tool/call
+  -> tools/pre-execute
+  -> user approval
+  -> guards
+  -> tools/execute
+  -> ToolDefinition.execute
+  -> tools/post-execute
+  -> finalizer
+  -> tools/result
+  -> tool/result
+```
+
+Deferred visibility is a context and token optimization, not an authorization boundary. A model or caller that already knows a hidden real name can still attempt it, and approval, guards, sandboxing, and tool-owned validation remain responsible for enforcement.
+
+Selection is committed only from a successful final `tools/result`, after post-execute policy. The finalizer persists the same compact, schema-free JSON result that drove the live commit, so resume and fork replay cannot diverge from the in-memory selected set.
+
+## Token Benchmark
+
+`pnpm benchmark` tokenizes deterministic nested OpenAI function-tool wire schemas using `gpt-tokenizer`'s default `o200k_base` encoder:
+
+| Deferred tools | Full initial schemas | `tool_search` initial | Initial reduction | After Top-5 |
+| ---: | ---: | ---: | ---: | ---: |
+| 10 | 1,822 | 103 | 94.3% | 1,013 |
+| 30 | 5,462 | 103 | 98.1% | 1,013 |
+| 50 | 9,102 | 103 | 98.9% | 1,013 |
+| 100 | 18,202 | 103 | 99.4% | 1,013 |
+
+The shipped generator and exact fixture construction are in [`scripts/benchmark.mjs`](scripts/benchmark.mjs). [`docs/benchmark-results.json`](docs/benchmark-results.json) records the package version, generator and corpus SHA-256 hashes, fixture sizes, tokenizer, and machine-readable results. Application totals also include core schemas and system prompt tokens, so production percentage savings depend on the installed tool mix.
+
+## Verification
+
+```bash
+pnpm install --frozen-lockfile
+pnpm validate
+pnpm run e2e:profile
+```
+
+The test suite covers document construction, exact-name Recall@1, capability Recall@5, stable ties, per-Agent isolation, monotonic growth, registry refresh, result shape, final-success commit, post-execute rejection and rewrite, one-turn loading, resume/fork recovery, live settings, Web controls, and the real DSH approval/guard/event/execution pipeline. The profile E2E installs the packed plugin into a clean `DSH_HOME` and drives a scripted real Agent through search, next-turn schema exposure, and a real-name tool call. Release validation runs this E2E on DSH `0.1.1-rc.1` and `0.1.2-rc.1`.
+
+## Known Limits
+
+- Native mode is required. DSH code/PTC and `both` modes independently render a `tools:sdk` prompt section; the stable plugin API does not expose a filtered SDK renderer. The plugin rejects those assemblies with a clear error instead of leaving search unavailable while leaking the unfiltered SDK.
+- On some Node.js 24+ builds, the DSH `0.1.2-rc.1` standalone launcher cannot access Node's internal ESM loader and therefore cannot resolve any bare external Profile package. If startup reports `Cannot find package '@anionex/dsh-tool-search'`, launch with `node --expose-internals "$(command -v dsh)" --profile <name> ...`. CI requires standard bare-package loading on Node 22; the Profile E2E records whether a local run needed this launcher workaround.
+- Tool definitions do not expose package provenance. Newly introduced DSH core names default to deferred until this package updates `DEFAULT_CORE_TOOLS` or the user adds an allowlist entry.
+- Exact `tool:<name>` guidance is filtered with a hidden schema. Arbitrary third-party prompt sections have no tool provenance and cannot be safely rewritten.
+- English stemming and NFKD folding are not semantic or multilingual retrieval. There is no reranker or embedding fallback.
+- Cooperative prompt waterfalls are not a hostile-plugin security boundary.
+
+## License
+
+MIT

package/README.zh.md

@@ -0,0 +1,160 @@
+# dsh-tool-search
+
+面向 [DeepSeek Harness（DSH）](https://github.com/deepseek-ai/DeepSeek-Harness) 的 Codex 风格延迟工具加载插件。
+
+`dsh-tool-search` 保留真实的 DSH 工具注册表，只缩小模型可见的工具集合。新 Agent 起初只看到维护中的核心工具和 `tool_search`。调用 `tool_search` 后，插件在本地延迟工具元数据上运行确定性 BM25，将 Top-K 结果加入该 Agent 的已选集合，并在下一次模型调用中暴露它们的完整原生 schema。模型随后用真实工具名调用，仍经过 DSH 原有执行链。
+
+## 安装
+
+```bash
+dsh plugin --profile web add @anionex/dsh-tool-search
+```
+
+需要时可把 `web` 换成 `headless` 或其他 DSH profile。安装后应重启运行中的 profile，并新建会话。
+
+部分 Node.js 24+ 版本的 DSH 独立启动器无法解析 Profile 中的裸包名。若启动时报 `Cannot find package '@anionex/dsh-tool-search'`，请使用 `node --expose-internals "$(command -v dsh)" --profile <name> ...`；这是 DSH 上游 loader 的限制，不是插件通过其他路径加载。
+
+要求：
+
+- DSH `0.1.1-rc.1` 至当前 `0.1.2-rc.1` 预发布版本线
+- Node.js `^22.19.0 || >=24.0.0`
+- 原生工具展示模式
+
+## 行为
+
+初始原生 schema 集合包含：
+
+- `tool_search`
+- `apply_patch`
+- `DEFAULT_CORE_TOOLS` 中维护的 DSH 核心工具
+- `alwaysVisible` 中配置的精确名称
+
+`0.1.0` 把以下名称视为核心工具：
+
+```text
+apply_patch, ask_user_question, bash, create_goal, dsh_im_return_file,
+exit_plan_mode, get_goal, interrupt_agent, job_kill, job_list, job_output,
+list_agents, ralph, report, send_message, skill, subagent, subagent_fork,
+todo_write, update_goal, workflow
+```
+
+其他已注册工具都可搜索，但默认不显示。`neverSearch` 同时从初始集合和搜索语料中隐藏名称，并在与 `alwaysVisible` 冲突时优先。`tool_search` 自身不能被拉黑，避免配置后无工具可发现。
+
+已选集合具备以下语义：
+
+- 每个活跃 Agent 相互隔离；
+- 一个 Agent 生命周期内只增长；
+- resume 时从成功的 `tool_search` 结果恢复；
+- fork 时从复制的历史前缀恢复；
+- 设置变化只改变当前有效可见性，不删除已选历史。
+
+工具或提示词注册表变化会让全部缓存目录失效。下一次 assembly 或搜索按当前作用域 schema 重建索引。
+
+## 设置
+
+DSH Web 设置页提供：
+
+- 默认结果数，范围 `1-20`，默认 `5`；
+- 精确工具名常驻白名单；
+- 精确工具名搜索黑名单；
+- 可筛选的实时工具目录及逐工具策略选择。
+
+等价配置：
+
+```yaml
+- id: dsh-tool-search
+  config:
+    defaultLimit: 5
+    alwaysVisible:
+      - my_frequent_tool
+    neverSearch:
+      - internal_debug_tool
+```
+
+## 搜索合同
+
+每个函数工具的搜索文档按顺序包含：
+
+1. `tool.name`
+2. `tool.name.replaceAll("_", " ")`
+3. `tool.description`
+4. 递归遍历 schema：先输出当前节点描述，再按排序后的属性名逐项输出“属性名 + 该属性的嵌套描述”；随后处理 `items` 和 `anyOf`
+
+本地索引使用 `k1=1.2`、`b=0.75` 的 BM25，包含英语停用词移除和 Porter2 词干化。精确原始名称命中固定排第一；其他结果先按 BM25 分数降序，再按工具名 Unicode 码点字典序稳定处理同分。搜索结果只有名称、短描述和分数，不重复参数 schema。
+
+## Codex 对齐
+
+实现对照了源码调研期间 `openai/codex` 最新 `main` 提交 [`3fde89f`](https://github.com/openai/codex/commit/3fde89f628281b9b049376fb0cec4d1577bfeaec)。Codex 的通用函数文档构造器是字段顺序和递归 schema 遍历的依据；Codex 同样使用 `bm25` `2.3.2` 的英语 tokenizer 与相同 BM25 默认参数。
+
+DSH 适配差异：
+
+| 范围 | Codex | 本插件 |
+| --- | --- | --- |
+| 默认 Top-K | 当前上游为 8 | 5 |
+| 加载协议 | Responses API 的 `tool_search_output` 在历史中携带可加载定义 | 普通 DSH `tool_search` 返回摘要，下一请求加入原生 schema |
+| 已加载状态 | 定义保留在模型历史时可用 | 每 Agent 单调已选集合，并从 DSH session 事件恢复 |
+| 同分排序 | 上游候选顺序可能不稳定 | 工具名字典序 tie-break |
+| 精确名称 | 普通 BM25 排名 | 显式保证 Recall@1 |
+| 字符转写 | Rust `deunicode` | Unicode NFKD 变音符号折叠 |
+| 结果载荷 | 定义，无分数 | 摘要和分数，无 schema |
+
+这些差异是有意设计。DSH 没有 Responses API 的 `tool_search_output` 历史类型，而需求要求下一模型轮注入 schema、稳定恢复已选集合。
+
+## 执行语义
+
+插件只过滤 `PromptAssembly.tools` 和匹配的 `tool:<name>` 指引段，不注销工具，也不调用 `ctx.tools.restrict()`。
+
+发现后的真实工具调用仍经过 DSH 标准链：
+
+```text
+tool/call
+  -> tools/pre-execute
+  -> 用户审批
+  -> guards
+  -> tools/execute
+  -> ToolDefinition.execute
+  -> tools/post-execute
+  -> finalizer
+  -> tools/result
+  -> tool/result
+```
+
+延迟可见性是上下文与 token 优化，不是授权边界。模型或调用方若已知隐藏工具的真实名称，仍可尝试调用；审批、guard、sandbox 和工具自身校验继续承担安全职责。
+
+已选集合只在 post-execute 策略之后、最终 `tools/result` 成功时提交。finalizer 会持久化驱动实时提交的同一份紧凑、无 schema JSON 结果，因此 resume/fork 重放不会与内存中的已选集合分叉。
+
+## Token 基准
+
+`pnpm benchmark` 使用 `gpt-tokenizer` 默认的 `o200k_base` 编码器，对确定性的嵌套 OpenAI function-tool wire schema 计数：
+
+| 延迟工具数 | 完整初始 schema | 仅 `tool_search` | 初始减少 | 加载 Top-5 后 |
+| ---: | ---: | ---: | ---: | ---: |
+| 10 | 1,822 | 103 | 94.3% | 1,013 |
+| 30 | 5,462 | 103 | 98.1% | 1,013 |
+| 50 | 9,102 | 103 | 98.9% | 1,013 |
+| 100 | 18,202 | 103 | 99.4% | 1,013 |
+
+随包发布的生成器和完整 fixture 构造位于 [`scripts/benchmark.mjs`](scripts/benchmark.mjs)。[`docs/benchmark-results.json`](docs/benchmark-results.json) 记录包版本、生成器与语料 SHA-256、fixture 规模、tokenizer 和机器可读结果。应用总量还包含核心 schema 和系统提示词 token，生产环境的百分比取决于实际安装工具集合。
+
+## 验证
+
+```bash
+pnpm install --frozen-lockfile
+pnpm validate
+pnpm run e2e:profile
+```
+
+测试覆盖文档构造、精确名称 Recall@1、能力 Recall@5、同分稳定性、Agent 隔离、集合单调增长、注册表刷新、结果形状、最终成功后提交、post-execute 拒绝与改写、只多一轮加载、resume/fork 恢复、实时设置、Web 控件，以及真实 DSH approval/guard/event/execute 链。Profile E2E 会把打包产物安装到全新 `DSH_HOME`，再驱动一个脚本化真实 Agent 完成搜索、下一轮 schema 可见和真实名称调用。发布验收会在 DSH `0.1.1-rc.1` 和 `0.1.2-rc.1` 上分别运行该 E2E。
+
+## 已知限制
+
+- 必须使用原生模式。DSH code/PTC 和 `both` 模式会独立渲染 `tools:sdk` 提示词段；稳定插件 API 没有过滤 SDK 的渲染钩子。插件会用明确错误拒绝这些 assembly，避免搜索不可用时仍泄露未过滤 SDK。
+- 部分 Node.js 24+ 版本下，DSH `0.1.2-rc.1` 独立启动器无法访问 Node 内部 ESM loader，因而不能解析任何 Profile 中以裸包名安装的外部插件。若启动时报 `Cannot find package '@anionex/dsh-tool-search'`，请改用 `node --expose-internals "$(command -v dsh)" --profile <name> ...`。CI 会在 Node 22 上强制验证标准裸包名加载；Profile E2E 也会记录本地运行是否用到了这一启动器兼容方案。
+- 工具定义不提供包来源。新加入的 DSH 核心工具默认延迟，直到本包更新 `DEFAULT_CORE_TOOLS`，或用户加入白名单。
+- 插件会随隐藏 schema 一起过滤精确 `tool:<name>` 指引。任意第三方提示词段没有工具来源，无法可靠改写。
+- 英语词干化和 NFKD 折叠不是语义或多语言检索；没有 reranker 或 embedding fallback。
+- 协作式 prompt waterfall 不是抵御恶意插件的安全边界。
+
+## 许可证
+
+MIT

package/cordis.patch.yml

@@ -0,0 +1,3 @@
+- insert:
+    - id: dsh-tool-search
+      name: '@anionex/dsh-tool-search'

package/docs/benchmark-results.json

@@ -0,0 +1,51 @@
+{
+  "schemaVersion": 2,
+  "packageVersion": "0.1.0",
+  "tokenizer": "gpt-tokenizer 4.0.0 (o200k_base default encode)",
+  "fixture": {
+    "description": "Deterministic nested function schemas; full surface versus tool_search-only initial surface.",
+    "generator": "scripts/benchmark.mjs",
+    "generatorSha256": "146898e3a92876fb35d9531c65950e940b664e7d77fa72e9a9c31169337adec1",
+    "corpusSha256": "2a7e824f3891f37b4633f09f3a814abd5d071c28a421b75a41580df9528745e2",
+    "sizes": [
+      10,
+      30,
+      50,
+      100
+    ]
+  },
+  "rows": [
+    {
+      "tools": 10,
+      "fullTokens": 1822,
+      "deferredInitialTokens": 103,
+      "initialTokensSaved": 1719,
+      "initialReductionPercent": 94.3,
+      "afterTop5Tokens": 1013
+    },
+    {
+      "tools": 30,
+      "fullTokens": 5462,
+      "deferredInitialTokens": 103,
+      "initialTokensSaved": 5359,
+      "initialReductionPercent": 98.1,
+      "afterTop5Tokens": 1013
+    },
+    {
+      "tools": 50,
+      "fullTokens": 9102,
+      "deferredInitialTokens": 103,
+      "initialTokensSaved": 8999,
+      "initialReductionPercent": 98.9,
+      "afterTop5Tokens": 1013
+    },
+    {
+      "tools": 100,
+      "fullTokens": 18202,
+      "deferredInitialTokens": 103,
+      "initialTokensSaved": 18099,
+      "initialReductionPercent": 99.4,
+      "afterTop5Tokens": 1013
+    }
+  ]
+}

package/lib/bm25.js

@@ -0,0 +1,83 @@
+/** Deterministic local BM25, aligned with Codex's bm25 2.3.2 defaults. */
+import stem from 'wink-porter2-stemmer';
+const K1 = 1.2;
+const B = 0.75;
+// stop-words 0.9.0 uses the NLTK English list before stemming.
+const ENGLISH_STOP_WORDS = new Set([
+    'a', 'about', 'above', 'after', 'again', 'against', 'ain', 'all', 'am', 'an', 'and', 'any', 'are',
+    'aren', "aren't", 'as', 'at', 'be', 'because', 'been', 'before', 'being', 'below', 'between', 'both',
+    'but', 'by', 'can', 'couldn', "couldn't", 'd', 'did', 'didn', "didn't", 'do', 'does', 'doesn',
+    "doesn't", 'doing', 'don', "don't", 'down', 'during', 'each', 'few', 'for', 'from', 'further', 'had',
+    'hadn', "hadn't", 'has', 'hasn', "hasn't", 'have', 'haven', "haven't", 'having', 'he', 'her', 'here',
+    'hers', 'herself', 'him', 'himself', 'his', 'how', 'i', 'if', 'in', 'into', 'is', 'isn', "isn't",
+    'it', "it's", 'its', 'itself', 'just', 'll', 'm', 'ma', 'me', 'mightn', "mightn't", 'more', 'most',
+    'mustn', "mustn't", 'my', 'myself', 'needn', "needn't", 'no', 'nor', 'not', 'now', 'o', 'of', 'off',
+    'on', 'once', 'only', 'or', 'other', 'our', 'ours', 'ourselves', 'out', 'over', 'own', 're', 's',
+    'same', 'shan', "shan't", 'she', "she's", 'should', "should've", 'shouldn', "shouldn't", 'so', 'some',
+    'such', 't', 'than', 'that', "that'll", 'the', 'their', 'theirs', 'them', 'themselves', 'then',
+    'there', 'these', 'they', 'this', 'those', 'through', 'to', 'too', 'under', 'until', 'up', 've',
+    'very', 'was', 'wasn', "wasn't", 'we', 'were', 'weren', "weren't", 'what', 'when', 'where', 'which',
+    'while', 'who', 'whom', 'why', 'will', 'with', 'won', "won't", 'wouldn', "wouldn't", 'y', 'you',
+    "you'd", "you'll", "you're", "you've", 'your', 'yours', 'yourself', 'yourselves',
+]);
+function asciiFold(value) {
+    return value.normalize('NFKD').replace(/\p{M}+/gu, '');
+}
+export function tokenize(value) {
+    const matches = asciiFold(value).toLowerCase().match(/[\p{L}\p{N}]+(?:['.][\p{L}\p{N}]+)*/gu) ?? [];
+    return matches.filter(token => !ENGLISH_STOP_WORDS.has(token)).map(token => stem(token));
+}
+export class Bm25Index {
+    documents;
+    frequencies = new Map();
+    averageLength;
+    constructor(documents) {
+        this.documents = documents.map((document, order) => {
+            const tokens = tokenize(document.text);
+            const counts = new Map();
+            for (const token of tokens)
+                counts.set(token, (counts.get(token) ?? 0) + 1);
+            for (const token of counts.keys())
+                this.frequencies.set(token, (this.frequencies.get(token) ?? 0) + 1);
+            return { ...document, tokens, counts, order };
+        });
+        this.averageLength = this.documents.length === 0
+            ? 256
+            : this.documents.reduce((total, document) => total + document.tokens.length, 0) / this.documents.length;
+    }
+    search(query, limit) {
+        if (limit <= 0 || this.documents.length === 0)
+            return [];
+        const queryTokens = tokenize(query);
+        const normalizedQuery = query.trim();
+        const matches = [];
+        for (const document of this.documents) {
+            let score = 0;
+            for (const token of queryTokens) {
+                const termFrequency = document.counts.get(token) ?? 0;
+                if (termFrequency === 0)
+                    continue;
+                const documentFrequency = this.frequencies.get(token) ?? 0;
+                const inverseDocumentFrequency = Math.log1p((this.documents.length - documentFrequency + 0.5) / (documentFrequency + 0.5));
+                const lengthRatio = document.tokens.length / this.averageLength;
+                const weight = (termFrequency * (K1 + 1))
+                    / (termFrequency + K1 * (1 - B + B * lengthRatio));
+                score += inverseDocumentFrequency * weight;
+            }
+            const exact = document.name === normalizedQuery;
+            if (score > 0 || exact)
+                matches.push({ id: document.id, name: document.name, score, exact, order: document.order });
+        }
+        matches.sort((left, right) => {
+            if (left.exact !== right.exact)
+                return left.exact ? -1 : 1;
+            if (left.score !== right.score)
+                return right.score - left.score;
+            if (left.name !== right.name)
+                return left.name < right.name ? -1 : 1;
+            return left.order - right.order;
+        });
+        return matches.slice(0, limit).map(({ id, name, score }) => ({ id, name, score }));
+    }
+}
+//# sourceMappingURL=bm25.js.map

package/lib/bm25.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"bm25.js","sourceRoot":"","sources":["../src/bm25.ts"],"names":[],"mappings":"AAAA,0EAA0E;AAE1E,OAAO,IAAI,MAAM,sBAAsB,CAAA;AAEvC,MAAM,EAAE,GAAG,GAAG,CAAA;AACd,MAAM,CAAC,GAAG,IAAI,CAAA;AAEd,+DAA+D;AAC/D,MAAM,kBAAkB,GAAG,IAAI,GAAG,CAAC;IACjC,GAAG,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,SAAS,EAAE,KAAK,EAAE,KAAK,EAAE,IAAI,EAAE,IAAI,EAAE,KAAK,EAAE,KAAK,EAAE,KAAK;IACjG,MAAM,EAAE,QAAQ,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,SAAS,EAAE,MAAM,EAAE,QAAQ,EAAE,OAAO,EAAE,OAAO,EAAE,SAAS,EAAE,MAAM;IACpG,KAAK,EAAE,IAAI,EAAE,KAAK,EAAE,QAAQ,EAAE,UAAU,EAAE,GAAG,EAAE,KAAK,EAAE,MAAM,EAAE,QAAQ,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO;IAC7F,SAAS,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,KAAK,EAAE,KAAK,EAAE,MAAM,EAAE,SAAS,EAAE,KAAK;IACpG,MAAM,EAAE,QAAQ,EAAE,KAAK,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,OAAO,EAAE,SAAS,EAAE,QAAQ,EAAE,IAAI,EAAE,KAAK,EAAE,MAAM;IACpG,MAAM,EAAE,SAAS,EAAE,KAAK,EAAE,SAAS,EAAE,KAAK,EAAE,KAAK,EAAE,GAAG,EAAE,IAAI,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,KAAK,EAAE,OAAO;IAChG,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,QAAQ,EAAE,MAAM,EAAE,IAAI,EAAE,GAAG,EAAE,IAAI,EAAE,IAAI,EAAE,QAAQ,EAAE,UAAU,EAAE,MAAM,EAAE,MAAM;IAClG,OAAO,EAAE,SAAS,EAAE,IAAI,EAAE,QAAQ,EAAE,OAAO,EAAE,SAAS,EAAE,IAAI,EAAE,KAAK,EAAE,KAAK,EAAE,KAAK,EAAE,GAAG,EAAE,IAAI,EAAE,KAAK;IACnG,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,IAAI,EAAE,GAAG;IAChG,MAAM,EAAE,MAAM,EAAE,QAAQ,EAAE,KAAK,EAAE,OAAO,EAAE,QAAQ,EAAE,WAAW,EAAE,SAAS,EAAE,WAAW,EAAE,IAAI,EAAE,MAAM;IACrG,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,MAAM,EAAE,SAAS,EAAE,KAAK,EAAE,OAAO,EAAE,QAAQ,EAAE,MAAM,EAAE,YAAY,EAAE,MAAM;IAC9F,OAAO,EAAE,OAAO,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,SAAS,EAAE,IAAI,EAAE,KAAK,EAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,IAAI;IAC/F,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,QAAQ,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,SAAS,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,OAAO;IACnG,OAAO,EAAE,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,EAAE,QAAQ,EAAE,UAAU,EAAE,GAAG,EAAE,KAAK;IAC/F,OAAO,EAAE,QAAQ,EAAE,QAAQ,EAAE,QAAQ,EAAE,MAAM,EAAE,OAAO,EAAE,UAAU,EAAE,YAAY;CACjF,CAAC,CAAA;AAEF,SAAS,SAAS,CAAC,KAAa;IAC9B,OAAO,KAAK,CAAC,SAAS,CAAC,MAAM,CAAC,CAAC,OAAO,CAAC,UAAU,EAAE,EAAE,CAAC,CAAA;AACxD,CAAC;AAED,MAAM,UAAU,QAAQ,CAAC,KAAa;IACpC,MAAM,OAAO,GAAG,SAAS,CAAC,KAAK,CAAC,CAAC,WAAW,EAAE,CAAC,KAAK,CAAC,uCAAuC,CAAC,IAAI,EAAE,CAAA;IACnG,OAAO,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC,kBAAkB,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,KAAK,CAAC,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAA;AAC1F,CAAC;AAoBD,MAAM,OAAO,SAAS;IACH,SAAS,CAAsB;IAC/B,WAAW,GAAG,IAAI,GAAG,EAAkB,CAAA;IACvC,aAAa,CAAQ;IAEtC,YAAY,SAAqC;QAC/C,IAAI,CAAC,SAAS,GAAG,SAAS,CAAC,GAAG,CAAC,CAAC,QAAQ,EAAE,KAAK,EAAE,EAAE;YACjD,MAAM,MAAM,GAAG,QAAQ,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAA;YACtC,MAAM,MAAM,GAAG,IAAI,GAAG,EAAkB,CAAA;YACxC,KAAK,MAAM,KAAK,IAAI,MAAM;gBAAE,MAAM,CAAC,GAAG,CAAC,KAAK,EAAE,CAAC,MAAM,CAAC,GAAG,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAA;YAC3E,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,IAAI,EAAE;gBAAE,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,KAAK,EAAE,CAAC,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAA;YACtG,OAAO,EAAE,GAAG,QAAQ,EAAE,MAAM,EAAE,MAAM,EAAE,KAAK,EAAE,CAAA;QAC/C,CAAC,CAAC,CAAA;QACF,IAAI,CAAC,aAAa,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,KAAK,CAAC;YAC9C,CAAC,CAAC,GAAG;YACL,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,CAAC,KAAK,EAAE,QAAQ,EAAE,EAAE,CAAC,KAAK,GAAG,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,CAAA;IAC3G,CAAC;IAED,MAAM,CAAC,KAAa,EAAE,KAAa;QACjC,IAAI,KAAK,IAAI,CAAC,IAAI,IAAI,CAAC,SAAS,CAAC,MAAM,KAAK,CAAC;YAAE,OAAO,EAAE,CAAA;QACxD,MAAM,WAAW,GAAG,QAAQ,CAAC,KAAK,CAAC,CAAA;QACnC,MAAM,eAAe,GAAG,KAAK,CAAC,IAAI,EAAE,CAAA;QACpC,MAAM,OAAO,GAA4D,EAAE,CAAA;QAE3E,KAAK,MAAM,QAAQ,IAAI,IAAI,CAAC,SAAS,EAAE,CAAC;YACtC,IAAI,KAAK,GAAG,CAAC,CAAA;YACb,KAAK,MAAM,KAAK,IAAI,WAAW,EAAE,CAAC;gBAChC,MAAM,aAAa,GAAG,QAAQ,CAAC,MAAM,CAAC,GAAG,CAAC,KAAK,CAAC,IAAI,CAAC,CAAA;gBACrD,IAAI,aAAa,KAAK,CAAC;oBAAE,SAAQ;gBACjC,MAAM,iBAAiB,GAAG,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,KAAK,CAAC,IAAI,CAAC,CAAA;gBAC1D,MAAM,wBAAwB,GAAG,IAAI,CAAC,KAAK,CACzC,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,GAAG,iBAAiB,GAAG,GAAG,CAAC,GAAG,CAAC,iBAAiB,GAAG,GAAG,CAAC,CAC9E,CAAA;gBACD,MAAM,WAAW,GAAG,QAAQ,CAAC,MAAM,CAAC,MAAM,GAAG,IAAI,CAAC,aAAa,CAAA;gBAC/D,MAAM,MAAM,GAAG,CAAC,aAAa,GAAG,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC;sBACrC,CAAC,aAAa,GAAG,EAAE,GAAG,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,GAAG,WAAW,CAAC,CAAC,CAAA;gBACpD,KAAK,IAAI,wBAAwB,GAAG,MAAM,CAAA;YAC5C,CAAC;YACD,MAAM,KAAK,GAAG,QAAQ,CAAC,IAAI,KAAK,eAAe,CAAA;YAC/C,IAAI,KAAK,GAAG,CAAC,IAAI,KAAK;gBAAE,OAAO,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,QAAQ,CAAC,EAAE,EAAE,IAAI,EAAE,QAAQ,CAAC,IAAI,EAAE,KAAK,EAAE,KAAK,EAAE,KAAK,EAAE,QAAQ,CAAC,KAAK,EAAE,CAAC,CAAA;QACrH,CAAC;QAED,OAAO,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE;YAC3B,IAAI,IAAI,CAAC,KAAK,KAAK,KAAK,CAAC,KAAK;gBAAE,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAA;YAC1D,IAAI,IAAI,CAAC,KAAK,KAAK,KAAK,CAAC,KAAK;gBAAE,OAAO,KAAK,CAAC,KAAK,GAAG,IAAI,CAAC,KAAK,CAAA;YAC/D,IAAI,IAAI,CAAC,IAAI,KAAK,KAAK,CAAC,IAAI;gBAAE,OAAO,IAAI,CAAC,IAAI,GAAG,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAA;YACpE,OAAO,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC,KAAK,CAAA;QACjC,CAAC,CAAC,CAAA;QACF,OAAO,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,IAAI,EAAE,KAAK,EAAE,EAAE,EAAE,CAAC,CAAC,EAAE,EAAE,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC,CAAC,CAAA;IACpF,CAAC;CACF"}

package/lib/catalog.js

@@ -0,0 +1,53 @@
+/** Codex-compatible tool search document construction. */
+function pushPart(parts, value) {
+    if (typeof value !== 'string')
+        return;
+    const part = value.trim();
+    if (part.length > 0)
+        parts.push(part);
+}
+/**
+ * Append descriptions and property names in the same recursive shape used by
+ * Codex: description, sorted object properties, array items, then anyOf.
+ */
+export function appendSchemaSearchText(schema, parts) {
+    if (typeof schema !== 'object' || schema === null || Array.isArray(schema))
+        return;
+    const node = schema;
+    pushPart(parts, node.description);
+    if (typeof node.properties === 'object' && node.properties !== null && !Array.isArray(node.properties)) {
+        const properties = node.properties;
+        for (const name of Object.keys(properties).sort()) {
+            pushPart(parts, name);
+            appendSchemaSearchText(properties[name], parts);
+        }
+    }
+    appendSchemaSearchText(node.items, parts);
+    if (Array.isArray(node.anyOf)) {
+        for (const variant of node.anyOf)
+            appendSchemaSearchText(variant, parts);
+    }
+}
+export function toolSearchText(tool) {
+    const parts = [];
+    pushPart(parts, tool.name);
+    pushPart(parts, tool.name.replaceAll('_', ' '));
+    pushPart(parts, tool.description);
+    appendSchemaSearchText(tool.parameters, parts);
+    return parts.join(' ');
+}
+export function buildCatalog(tools) {
+    const byName = new Map();
+    for (const tool of tools) {
+        if (byName.has(tool.name))
+            continue;
+        byName.set(tool.name, {
+            name: tool.name,
+            description: tool.description?.trim() ?? '',
+            parameters: structuredClone(tool.parameters),
+            searchText: toolSearchText(tool),
+        });
+    }
+    return [...byName.values()].sort((left, right) => left.name < right.name ? -1 : left.name > right.name ? 1 : 0);
+}
+//# sourceMappingURL=catalog.js.map

package/lib/catalog.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"catalog.js","sourceRoot":"","sources":["../src/catalog.ts"],"names":[],"mappings":"AAAA,0DAA0D;AAsB1D,SAAS,QAAQ,CAAC,KAAe,EAAE,KAAc;IAC/C,IAAI,OAAO,KAAK,KAAK,QAAQ;QAAE,OAAM;IACrC,MAAM,IAAI,GAAG,KAAK,CAAC,IAAI,EAAE,CAAA;IACzB,IAAI,IAAI,CAAC,MAAM,GAAG,CAAC;QAAE,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAA;AACvC,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,sBAAsB,CAAC,MAAe,EAAE,KAAe;IACrE,IAAI,OAAO,MAAM,KAAK,QAAQ,IAAI,MAAM,KAAK,IAAI,IAAI,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC;QAAE,OAAM;IAClF,MAAM,IAAI,GAAG,MAAwB,CAAA;IACrC,QAAQ,CAAC,KAAK,EAAE,IAAI,CAAC,WAAW,CAAC,CAAA;IAEjC,IAAI,OAAO,IAAI,CAAC,UAAU,KAAK,QAAQ,IAAI,IAAI,CAAC,UAAU,KAAK,IAAI,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,UAAU,CAAC,EAAE,CAAC;QACvG,MAAM,UAAU,GAAG,IAAI,CAAC,UAAqC,CAAA;QAC7D,KAAK,MAAM,IAAI,IAAI,MAAM,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC;YAClD,QAAQ,CAAC,KAAK,EAAE,IAAI,CAAC,CAAA;YACrB,sBAAsB,CAAC,UAAU,CAAC,IAAI,CAAC,EAAE,KAAK,CAAC,CAAA;QACjD,CAAC;IACH,CAAC;IACD,sBAAsB,CAAC,IAAI,CAAC,KAAK,EAAE,KAAK,CAAC,CAAA;IACzC,IAAI,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;QAC9B,KAAK,MAAM,OAAO,IAAI,IAAI,CAAC,KAAK;YAAE,sBAAsB,CAAC,OAAO,EAAE,KAAK,CAAC,CAAA;IAC1E,CAAC;AACH,CAAC;AAED,MAAM,UAAU,cAAc,CAAC,IAAoB;IACjD,MAAM,KAAK,GAAa,EAAE,CAAA;IAC1B,QAAQ,CAAC,KAAK,EAAE,IAAI,CAAC,IAAI,CAAC,CAAA;IAC1B,QAAQ,CAAC,KAAK,EAAE,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC,CAAA;IAC/C,QAAQ,CAAC,KAAK,EAAE,IAAI,CAAC,WAAW,CAAC,CAAA;IACjC,sBAAsB,CAAC,IAAI,CAAC,UAAU,EAAE,KAAK,CAAC,CAAA;IAC9C,OAAO,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,CAAA;AACxB,CAAC;AAED,MAAM,UAAU,YAAY,CAAC,KAAgC;IAC3D,MAAM,MAAM,GAAG,IAAI,GAAG,EAA4B,CAAA;IAClD,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,IAAI,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC;YAAE,SAAQ;QACnC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,EAAE;YACpB,IAAI,EAAE,IAAI,CAAC,IAAI;YACf,WAAW,EAAE,IAAI,CAAC,WAAW,EAAE,IAAI,EAAE,IAAI,EAAE;YAC3C,UAAU,EAAE,eAAe,CAAC,IAAI,CAAC,UAAU,CAAC;YAC5C,UAAU,EAAE,cAAc,CAAC,IAAI,CAAC;SACjC,CAAC,CAAA;IACJ,CAAC;IACD,OAAO,CAAC,GAAG,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,IAAI,CAAC,IAAI,GAAG,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,GAAG,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAA;AACjH,CAAC"}