@fffangchenghao/dsh-web-search 0.21.0 → 0.22.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,50 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.22.0
4
+
5
+ ### Changed
6
+ - **Tail-tolerant tier release.** A tier no longer waits for a contributor
7
+ from the leading region. It releases when the result budget is filled and any
8
+ two independent engines have returned rows. Interactive deadlines are 5 s for
9
+ Latin-script queries and 3 s for CJK queries; `wait: 'all'` remains the
10
+ completeness escape hatch.
11
+ - **Official release facts start in parallel.** `officialLookup()` now covers
12
+ PostgreSQL, Rust and explicitly npm-scoped package queries. The Rust resolver
13
+ maps the GitHub release timestamp to the canonical
14
+ `blog.rust-lang.org/YYYY/MM/DD/Rust-X.Y.Z/` page, tries the API directly
15
+ before the tunnel, and is capped at 2.5 s. `Rust 1.90 发布说明` now returns the
16
+ official announcement at rank 1.
17
+ - **Optional Firecrawl key.** `firecrawlApiKey` / `FIRECRAWL_API_KEY` gives
18
+ users a repeatable Firecrawl fallback. Keyless Firecrawl remains the default
19
+ and its quota behavior is still recorded honestly; a client cannot repair a
20
+ server-side keyless quota.
21
+ - **Replayable follow-up measurement.** `bench/compare30.mjs` can run one arm
22
+ with `--only plugin`, enriches the plugin arm with the official lookup, and
23
+ records an optional Firecrawl key. The 30-query follow-up and the five-query
24
+ Chinese latency table are checked in under `bench/compare30/` and
25
+ `docs/latency-and-sources.md`.
26
+
27
+ ### Documentation
28
+ - **Exa architecture and papers.** `docs/latency-and-sources.md` explains why a
29
+ hosted semantic index can answer around 1 s, what tail-tolerant release
30
+ borrows from Dean & Barroso, and why this plugin does not need a vector
31
+ database to fix these specific failures. It links Exa's engineering posts,
32
+ The Tail at Scale, HNSW, Matryoshka representation learning and ColBERT.
33
+
34
+ **Test suite: 680 tests across 55 files** (679 pass, 1 skipped).
35
+
36
+ ## 0.21.1
37
+
38
+ ### Documentation
39
+ - **Public evidence links for the 30-query comparison.** The raw three sweeps
40
+ and the query corpus are linked from `docs/comparison.md` through a public
41
+ gist, so the evidence is reachable without publishing the working tree.
42
+ - **README quickstart and cross-agent paths.** The package now includes
43
+ `docs/quickstart.md`, `docs/cross-agent.md` and the `web_tools` demo asset.
44
+
45
+ No runtime behavior changed since 0.21.0. **Test suite: 668 tests across 52 files**
46
+ (667 pass, 1 skipped).
47
+
3
48
  ## 0.21.0
4
49
 
5
50
  ### Added
package/README.md CHANGED
@@ -3,14 +3,14 @@
3
3
  **Gives DeepSeek Harness a pair of hands for the web: the model can search, then open a page and read it, without you pasting links.**
4
4
 
5
5
  - [中文说明](README.zh-CN.md)
6
- - [Quickstart](docs/quickstart.md) · [Cross-agent MCP](docs/cross-agent.md) · [30-query comparison](docs/comparison.md)
6
+ - [Quickstart](docs/quickstart.md) · [Cross-agent MCP](docs/cross-agent.md) · [30-query comparison](docs/comparison.md) · [Latency & sources](docs/latency-and-sources.md)
7
7
  - [Changelog](CHANGELOG.md) · [Why a plugin's settings go missing: the four-layer chain](docs/plugin-settings-form.md)
8
8
 
9
9
  <p align="center"><img src="https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/banner.png" alt="dsh-web-search — key-free web search and page fetch for DeepSeek Harness" width="100%"></p>
10
10
 
11
11
  <p align="center">
12
12
  <a href="https://www.npmjs.com/package/@fffangchenghao/dsh-web-search"><img src="https://img.shields.io/npm/v/@fffangchenghao/dsh-web-search.svg" alt="npm version"></a>
13
- <img src="https://img.shields.io/badge/tests-668%20passing-brightgreen" alt="tests">
13
+ <img src="https://img.shields.io/badge/tests-680%20tests-brightgreen" alt="tests">
14
14
  <img src="https://img.shields.io/badge/engines-8%2C%20zero%20API%20key-brightgreen" alt="engines">
15
15
  <img src="https://img.shields.io/badge/node-%3E%3D22.18-blue" alt="node">
16
16
  <img src="https://img.shields.io/badge/dsh-%3E%3D0.2.1--rc.2%20%3C0.3.0-blueviolet" alt="dsh">
@@ -20,11 +20,13 @@
20
20
  ## Why this one
21
21
 
22
22
  - **No API key, eight engines.** AnySearch, CN Bing, DuckDuckGo HTML, www Bing, DuckDuckGo Lite and 360 Search are scraped directly; Exa joins the primary tier through its keyless MCP endpoint, and Firecrawl's search API is the measured fallback. The plugin works out of the box.
23
+ - **Official release facts, not generic ranking.** Explicit PostgreSQL, Rust and scoped npm queries start a narrow official lookup in parallel; `Rust 1.90 发布说明` returns the canonical Rust Blog announcement at rank 1. The lookup is capped at 2.5 s and failure leaves the ordinary rows intact.
24
+ - **A tail budget and an honest quota story.** Interactive searches release after two independent contributors and cap at 5 s for Latin-script queries / 3 s for CJK queries. Firecrawl stays a fallback; `firecrawlApiKey` is optional for users who need repeatable quota. Keyless Firecrawl cannot be made dependable from the client side.
23
25
  - **The ranking is measured, not asserted.** Cross-engine reciprocal-rank fusion with authority and freshness boosts, a 20-query offline benchmark, and an 81-configuration parameter sweep — one of the few search plugins here that publishes its metric, per-query output and confidence intervals. See [How the ranking works](docs/rrf-ranking.md).
24
26
  - **One package, two runtimes.** DSH gets the native plugin and one `web_tools` entry; Claude Code, Codex and other MCP clients get the same keyless search through a dependency-free stdio server with `search`, `fetch`, `research` and `domain_search`. See [Cross-agent MCP](docs/cross-agent.md).
25
- - **Measured in public, including the losses.** A 30-query bilingual comparison records availability, latency, unique domains, official-domain presence and every failure against keyless Firecrawl and anonymous Exa. Exa is faster here; this plugin returns more distinct domains. Both test arms were keyless. The method and raw results are in [docs/comparison.md](docs/comparison.md).
27
+ - **Measured in public, including the losses.** A 30-query bilingual comparison records availability, latency, unique domains, official-domain presence and every failure against keyless Firecrawl and anonymous Exa. The follow-up documents the Exa architecture difference, the 3 s CJK budget and the Rust official-source lookup. Exa is faster here; this plugin returns more distinct domains. Both test arms were keyless. The method and raw results are in [docs/comparison.md](docs/comparison.md) and a [public gist](https://gist.github.com/fangchenghao/3db9838fa98928435c1c2df38202c373).
26
28
  - **Fetching admits failure.** Binary documents and empty 200 responses are rejected by name (`SEARCH_UNSUPPORTED`, `SEARCH_EMPTY`) instead of being returned as noise; the direct → tunnel → render → extract ladder shares one 25 s budget and stops when there is no time left. The budget is per request — there is a regression test for exactly that.
27
- - **668 tests across 52 files**, a committed `lib/`, declared `engines.dsh` / `engines.node`, and a [SECURITY.md](SECURITY.md) that lists every process, host and file the plugin touches.
29
+ - **680 tests across 55 files**, a committed `lib/`, declared `engines.dsh` / `engines.node`, and a [SECURITY.md](SECURITY.md) that lists every process, host and file the plugin touches.
28
30
  - **Bilingual documentation.** This README and [中文说明](README.zh-CN.md) are maintained side by side.
29
31
 
30
32
  ![Ranking baseline and parameter sweep](https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/bench.png)
@@ -41,7 +43,7 @@ The web-search corner of the DSH plugin ecosystem is crowded. A fresh snapshot
41
43
  | Key-free path | 8 engines, all key-free | ~30-engine free/paid chain | 5 defaults (seam, Exa, DDG, Bing, Jina) | Firecrawl + Keenable keyless defaults | 1 anonymous AnySearch API |
42
44
  | Dedicated vertical sources | 34 (8 on by default) | 41 | platform registry, not a vertical catalog | web + X, no vertical catalog | vertical API behind AnySearch |
43
45
  | Model-facing tools | 1 (`web_tools`, 8 actions) + 4 MCP tools | ~10 | 2 (`web_index` / `web_call`, 20 actions) | native search/fetch + skill | 5 |
44
- | Tests | 52 files / 668 cases, locally verified | no test script in `package.json` | 774 unit + 94 bench, self-reported | Vitest configured; count not published | Vitest configured; count not published |
46
+ | Tests | 55 files / 680 cases, locally verified | no test script in `package.json` | 774 unit + 94 bench, self-reported | Vitest configured; count not published | Vitest configured; count not published |
45
47
  | GitHub stars | 0 | 620 | 77 | 836 | 663 |
46
48
  | npm downloads, last month | 1,087 | 52,436 | 6,420 | 43,151 | 9,251 |
47
49
  | npm unpacked size | 1.2 MB | 0.70 MB | 3.05 MB | 0.51 MB | 0.27 MB |
@@ -52,7 +54,7 @@ read is:
52
54
  - **Feature depth:** upper-middle. `dsh-free-search` has more engines and
53
55
  verticals; `dsh-web-search-pro` has a deeper evidence/ranking pipeline and far
54
56
  more platform backends; `modsearch` is the stronger cross-host bridge.
55
- - **Engineering and verification:** near the top. 668 local tests, a replayable
57
+ - **Engineering and verification:** near the top. 680 local tests, a replayable
56
58
  ranking benchmark, declared security boundaries and one runtime dependency
57
59
  put it alongside the best-maintained entries, while remaining much smaller
58
60
  than the Playwright-heavy option.
@@ -60,7 +62,7 @@ read is:
60
62
  Chrome, PostgreSQL and AI Act paths and the 34-source vertical layer are not
61
63
  common in the other plugins.
62
64
  - **Adoption and maturity:** bottom third of this sample. The public npm release
63
- is 0.21.0, matching this tree, but the GitHub repository is four days old
65
+ is 0.22.0, matching this tree, but the GitHub repository is four days old
64
66
  with no stars and little independent adoption.
65
67
  - **Tool-surface economy:** now competitive. 0.20.x exposes one `web_tools`
66
68
  entry with eight actions, close to `dsh-web-search-pro`'s two tools with 20
@@ -275,8 +277,8 @@ Each line is the plain version first, the technical note second. The technical n
275
277
  *Technical: `lead = cn` when the query contains CJK characters, `lead = global` for plain ASCII; the other region only appends when the leading one comes up short. `src/relevance.ts` splits a CJK query into character bigrams (an ASCII word counts double), requires two weight units of overlap with title, snippet or URL, and returns English queries untouched. A whole-phrase engine that misses a long query is asked once more with its core words (`src/variants.ts`) — but only when its first answer contains no row that survives the relevance gate, so a working short query pays neither the request nor the latency: `东数西算 八个国家枢纽节点 定位` also asks `东数西算`, `如何配置 postgres 逻辑复制` also asks `postgres 逻辑复制`. Measured on the first query, `cn.bing` answered with ten dictionary/company rows before this; it now contributes the NDRC and gov.cn articles, and the fused list has eight relevant rows.*
276
278
 
277
279
  6. **You choose speed or completeness.**
278
- `auto` waits up to 6 s for English and 3.5 s for Chinese; `fast` takes whoever answers first; `all` waits for everyone, up to 10 s.
279
- *Technical: the three `wait` values with soft deadlines of 6000/3500 ms (`auto`), 3500 ms (`fast`), 10000 ms (`all`); the per-engine hard deadline is `timeoutMs: 15000`.*
280
+ `auto` waits up to 5 s for English and 3 s for Chinese; `fast` takes whoever answers first; `all` waits for everyone, up to 10 s.
281
+ *Technical: the three `wait` values with soft deadlines of 5000/3000 ms (`auto`), 3500 ms (`fast`), 10000 ms (`all`); the per-engine hard deadline is `timeoutMs: 15000`.*
280
282
 
281
283
  7. **Repeating a question does not repeat the work.**
282
284
  The same query within ten minutes is answered from the previous run, almost instantly.
@@ -555,6 +557,7 @@ Optionally, add a config row:
555
557
  proxy: "" # empty means auto-detect
556
558
  anysearchApiKey: "" # empty means anonymous access
557
559
  anysearchDomain: "" # or finance / academic / security / legal / code
560
+ firecrawlApiKey: "" # optional; keyless fallback is quota-limited
558
561
  timeRange: off # off / day / week / month / year
559
562
  renderJs: auto # auto / always / off
560
563
  renderTimeoutMs: 15000
@@ -603,6 +606,7 @@ Open Settings → "Web search (Bing)". Thirty controls, arranged in six groups,
603
606
  | Proxy (relay) | `proxy` | empty (detect) | e.g. `http://127.0.0.1:7890`; empty reads the environment, then the system proxy |
604
607
  | AnySearch key | `anysearchApiKey` | empty (anonymous) | Optional. Sent only to `api.anysearch.com`; raises limits and unlocks page extraction. Several keys may be comma-separated and are rotated, with a throttled or rejected one resting automatically |
605
608
  | Vertical domain | `anysearchDomain` | empty (general) | Typical values: `finance` / `academic` / `security` / `legal` / `code`; a dotted value such as `finance.us_stock` selects a sub-domain |
609
+ | Firecrawl key | `firecrawlApiKey` | empty (keyless) | Optional. Only affects the fallback tier; the keyless endpoint has a tight shared quota and may return `429` after repeated sweeps |
606
610
  | Blocked sites | `blockedDomains` | empty (nothing) | Domains that must never appear, comma-separated; a subdomain matches its parent. Applied after fusion and on a cache hit, so a blocked row cannot come back from the cache |
607
611
  | Safe search | `safeSearch` | `off` | `off` keeps each engine's default; `moderate` or `strict` is forwarded to Bing (`adlt`) and DuckDuckGo (`kp`). Other engines ignore it |
608
612
  | Fallback on | `fallbackOn` | all six classes | Failure classes that may switch to a fallback engine: `quota`, `auth`, `bot-wall`, `transient`, `invalid-response`, `unknown`. Empty means never fall back; a skipped fallback is explained in the result's `note` |
package/README.zh-CN.md CHANGED
@@ -9,7 +9,7 @@
9
9
 
10
10
  <p align="center">
11
11
  <a href="https://www.npmjs.com/package/@fffangchenghao/dsh-web-search"><img src="https://img.shields.io/npm/v/@fffangchenghao/dsh-web-search.svg" alt="npm 版本"></a>
12
- <img src="https://img.shields.io/badge/tests-668%20passing-brightgreen" alt="测试">
12
+ <img src="https://img.shields.io/badge/tests-680%20tests-brightgreen" alt="测试">
13
13
  <img src="https://img.shields.io/badge/engines-8%2C%20zero%20API%20key-brightgreen" alt="引擎">
14
14
  <img src="https://img.shields.io/badge/node-%3E%3D22.18-blue" alt="node">
15
15
  <img src="https://img.shields.io/badge/dsh-%3E%3D0.2.1--rc.2%20%3C0.3.0-blueviolet" alt="dsh">
@@ -19,11 +19,13 @@
19
19
  ## 为什么选这个
20
20
 
21
21
  - **不要 API key,八个引擎。** AnySearch、必应中国、DuckDuckGo HTML、必应国际、DuckDuckGo Lite、360 搜索直接抓取;Exa 走它的免密钥 MCP 通道加入主引擎,Firecrawl 的搜索 API 作为实测后的兜底。装好就能用。
22
+ - **官方发布事实优先。** PostgreSQL、Rust 和显式 npm 包查询会并行做一次窄查询;`Rust 1.90 发布说明` 的第 1 条是 Rust Blog 官方公告。查询上限 2.5 秒,失败就保留普通结果。
23
+ - **尾部有预算,配额不粉饰。** 交互搜索在两个独立引擎返回后提前释放,拉丁文查询 5 秒、中文查询 3 秒封顶。Firecrawl 仍是兜底;需要稳定配额可以配 `firecrawlApiKey`,但免 key 的服务端配额不是客户端能修好的。
22
24
  - **排序是量出来的,不是写出来的。** 跨引擎 RRF 融合,叠加权威度与时效加权;20 条离线基准 + 81 组参数扫描,是这类插件里少数同时公开指标、逐条输出和置信区间的。方法见 [排序是怎么做的](docs/rrf-ranking.md)。
23
25
  - **一个包,两种运行时。** DSH 里是原生插件和唯一的 `web_tools` 入口;Claude Code、Codex 等 MCP 客户端通过零依赖的 stdio server 使用同一套免密钥搜索,提供 `search`、`fetch`、`research`、`domain_search`。见 [跨 Agent MCP](docs/cross-agent.md)。
24
- - **公开测量,也公开输的地方。** 30 条中英查询对比记录可用性、延迟、独立域名、官方域名命中与每一条失败;对比对象是免 key Firecrawl 和匿名 Exa。这里 Exa 更快,本插件返回的独立域名更多;两条对比路径都未使用 key。方法和原始结果在 [docs/comparison.md](docs/comparison.md)。
26
+ - **公开测量,也公开输的地方。** 30 条中英查询对比记录可用性、延迟、独立域名、官方域名命中与每一条失败;对比对象是免 key Firecrawl 和匿名 Exa。这里 Exa 更快,本插件返回的独立域名更多;两条对比路径都未使用 key。方法和原始结果在 [docs/comparison.md](docs/comparison.md);Exa 架构差异、中文 3 秒预算和 Rust 官方源见 [延迟与官方源](docs/latency-and-sources.md)。
25
27
  - **抓取会承认失败。** 二进制文档和「200 但正文为空」会被明确拒绝(`SEARCH_UNSUPPORTED`、`SEARCH_EMPTY`),而不是当噪音返回;direct → tunnel → render → extract 四级阶梯共享 25 秒预算,预算不足就不再起新的一级。预算是「每次请求」而不是「插件启动时」,这点有回归测试守着。
26
- - **52 个测试文件、668 个用例**,`lib/` 产物入库,声明了 `engines.dsh` / `engines.node`,[SECURITY.md](SECURITY.md) 逐项列出插件会碰的进程、主机和文件。
28
+ - **55 个测试文件、680 个用例**,`lib/` 产物入库,声明了 `engines.dsh` / `engines.node`,[SECURITY.md](SECURITY.md) 逐项列出插件会碰的进程、主机和文件。
27
29
  - **双语文档。** 本文件和 [English README](README.md) 同步维护。
28
30
 
29
31
  ![排序基线参数扫描](https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/bench.png)
@@ -40,7 +42,7 @@ DSH 插件生态里的「联网搜索」已经很挤。下面是一张 2026-10-2
40
42
  | 免 key 路径 | 8 个引擎,全部免 key | ~30 个免费/付费引擎链 | 默认 5 个(seam、Exa、DDG、Bing、Jina) | Firecrawl + Keenable 免 key 默认 | 1 个 AnySearch 匿名 API |
41
43
  | 专门数据源 | 34 个(默认开 8 个) | 41 个 | 平台注册表,不是垂直源目录 | 网页 + X,无垂直源目录 | AnySearch 的垂直 API |
42
44
  | 面向模型的工具 | 1 个(`web_tools`,8 个 action)+ 4 个 MCP 工具 | ~10 个 | 2 个(`web_index` / `web_call`,20 个动作) | 原生搜索/抓取 + skill | 5 个 |
43
- | 测试 | 52 文件 / 668 用例,本地实测 | `package.json` 无测试脚本 | 774 单元 + 94 bench,自报 | 配了 Vitest,未公布数量 | 配了 Vitest,未公布数量 |
45
+ | 测试 | 55 文件 / 680 用例,本地实测 | `package.json` 无测试脚本 | 774 单元 + 94 bench,自报 | 配了 Vitest,未公布数量 | 配了 Vitest,未公布数量 |
44
46
  | GitHub stars | 0 | 620 | 77 | 836 | 663 |
45
47
  | npm 近一月下载 | 1,087 | 52,436 | 6,420 | 43,151 | 9,251 |
46
48
  | npm 解包大小 | 1.2 MB | 0.70 MB | 3.05 MB | 0.51 MB | 0.27 MB |
@@ -50,11 +52,11 @@ DSH 插件生态里的「联网搜索」已经很挤。下面是一张 2026-10-2
50
52
  - **功能深度:中上。** `dsh-free-search` 引擎和垂直源更多;
51
53
  `dsh-web-search-pro` 的证据/排序管线和平台后端更深;`modsearch` 的跨宿主
52
54
  能力更强。
53
- - **工程质量与可验证性:接近第一梯队。** 668 个本地测试、可重放的排序基准、
55
+ - **工程质量与可验证性:接近第一梯队。** 680 个本地测试、可重放的排序基准、
54
56
  明确的安全边界、只有 1 个运行时依赖,同时体积远小于 Playwright 重型方案。
55
57
  - **免 key + 官方源纠错:最强卖点。** Chrome、PostgreSQL、AI Act 的官方路径
56
58
  和 34 个垂直源,不是其他插件普遍具备的。
57
- - **采用度与成熟度:这个样本里的后段。** 公开 npm 已是 0.21.0,与这棵树一致;但 GitHub 仓库创建 4 天、0 stars,
59
+ - **采用度与成熟度:这个样本里的后段。** 公开 npm 已是 0.22.0,与这棵树一致;但 GitHub 仓库创建 4 天、0 stars,
58
60
  还缺少独立采用。
59
61
  - **工具面的经济性:现在有竞争力。** 0.20.x 只暴露一个 `web_tools` + 8 个
60
62
  action,接近 `dsh-web-search-pro` 的 2 工具 + 20 动作;旧的八个工具名保留为
@@ -267,8 +269,8 @@ Harness 里有个叫 `web` 的**服务**,它自己不会搜索,只负责「
267
269
  *技术补充:查询含 CJK 字符时 `lead = cn`,纯 ASCII 时 `lead = global`;非主导区只在主导区凑不够条数时才追加。`src/relevance.ts` 把中文查询切成二字 bigram(ASCII 单词权重翻倍),要求与标题、摘要或 URL 至少两个权重单位的重合;英文查询原样放行。对整串匹配型引擎还会用核心词补问一次(`src/variants.ts`),但只在原答案没有任何一条能通过相关性门槛时才补问——短查询正常工作时不付这次请求和延迟:「东数西算 八个国家枢纽节点 定位」会补问「东数西算」,「如何配置 postgres 逻辑复制」会补问「postgres 逻辑复制」。实测第一条查询:`cn.bing` 原来返回十条词典/公司页,现在能贡献发改委和 gov.cn 文章,融合结果八条全部相关。*
268
270
 
269
271
  6. **快慢可以选。**
270
- 默认 `auto`:英文最多等 6 秒、中文最多等 3.5 秒;`fast` 谁先回就用谁;`all` 等所有引擎,最多 10 秒。
271
- *技术补充:`wait` 三值,软截止分别是 `auto` 6000/3500 ms、`fast` 3500 ms、`all` 10000 ms;单引擎硬超时默认 `timeoutMs: 15000`。*
272
+ 默认 `auto`:英文最多等 5 秒、中文最多等 3 秒;`fast` 谁先回就用谁;`all` 等所有引擎,最多 10 秒。
273
+ *技术补充:`wait` 三值,软截止分别是 `auto` 5000/3000 ms、`fast` 3500 ms、`all` 10000 ms;单引擎硬超时默认 `timeoutMs: 15000`。*
272
274
 
273
275
  7. **同一个问题不重复打扰搜索引擎。**
274
276
  十分钟内问同一个问题,直接用上次的结果,几乎瞬间返回。
@@ -526,6 +528,7 @@ pnpm install
526
528
  proxy: "" # 留空自动探测
527
529
  anysearchApiKey: "" # 留空表示匿名访问
528
530
  anysearchDomain: "" # 或 finance / academic / security / legal / code
531
+ firecrawlApiKey: "" # 可选;免 key 兜底配额很紧
529
532
  timeRange: off # off / day / week / month / year
530
533
  renderJs: auto # auto / always / off
531
534
  renderTimeoutMs: 15000
@@ -574,6 +577,7 @@ pnpm install
574
577
  | 代理(中转站) | `proxy` | 空(自动探测) | 如 `http://127.0.0.1:7890`;留空先读环境变量再读系统代理 |
575
578
  | AnySearch 通行证 | `anysearchApiKey` | 空(匿名) | 可选项。只发给 `api.anysearch.com`,用于提高额度、解锁正文抽取。可以用逗号分隔填多个 key,插件轮流使用;某个被限流或拒绝会自动休息 |
576
579
  | 专业领域 | `anysearchDomain` | 空(通用) | 常见取值 `finance` / `academic` / `security` / `legal` / `code`,可带子域如 `finance.us_stock` |
580
+ | Firecrawl 通行证 | `firecrawlApiKey` | 空(匿名) | 可选项,只影响兜底梯队;匿名端点共享配额很紧,连续扫描可能返回 `429` |
577
581
  | 屏蔽网站 | `blockedDomains` | 空(不屏蔽) | 永远不要出现在结果里的网站,逗号分隔;子域名会一起屏蔽。在融合之后和缓存命中时都会过滤,所以被屏蔽的行不会从缓存里回来 |
578
582
  | 安全搜索 | `safeSearch` | `off` | `off` 保持引擎默认;`moderate` / `strict` 会以 Bing `adlt`、DuckDuckGo `kp` 转发,其它引擎忽略 |
579
583
  | 哪些失败可回退 | `fallbackOn` | 六类全开 | 允许切备用引擎的失败类别:`quota`、`auth`、`bot-wall`、`transient`、`invalid-response`、`unknown`。留空表示任何失败都不回退;被跳过的回退会写进结果的 `note` |
package/client.js CHANGED
@@ -74,7 +74,7 @@
74
74
  var ENTRY = 'web-search-bing';
75
75
  var PACKAGE_NAME = 'dsh-web-search-bing';
76
76
  /** Keep in step with package.json; compared with the npm registry. */
77
- var VERSION = '0.21.0';
77
+ var VERSION = '0.22.0';
78
78
  /** Snapshot used while no configuration service is reachable. */
79
79
  var EMPTY_STATE = { status: 'unavailable', value: undefined, base: undefined, user: undefined, revision: undefined, writable: false };
80
80
 
@@ -22,7 +22,7 @@ The failures are part of the result.
22
22
  - **Exa arm:** the anonymous `web_search_exa` MCP endpoint with
23
23
  `Accept: application/json, text/event-stream`; direct first, then tunnel.
24
24
  - **No API keys** were used. All raw responses are in
25
- [`bench/compare30/results-2026-10-26-sweep1.json`](https://github.com/fangchenghao/dsh-web-search/blob/main/bench/compare30/results-2026-10-26-sweep1.json).
25
+ [`results-2026-10-26-sweep1.json`](https://gist.githubusercontent.com/fangchenghao/3db9838fa98928435c1c2df38202c373/raw/results-2026-10-26-sweep1.json).
26
26
 
27
27
  Metrics: `P@8`, `R@8` and `MRR@8` are **not** reported here. They need
28
28
  human-labelled relevance; inventing labels would make the table look stronger
@@ -70,9 +70,9 @@ By category (first sweep; answered / official / mean ms):
70
70
  then 16/30 and 0/30 in two immediate repeats, with HTTP 429 on the tunnel
71
71
  after a direct 403. That is a distribution/reliability limitation, not a
72
72
  ranking difference. The repeat files are
73
- [`sweep2`](https://github.com/fangchenghao/dsh-web-search/blob/main/bench/compare30/results-2026-10-26-sweep2.json)
73
+ [`sweep2`](https://gist.githubusercontent.com/fangchenghao/3db9838fa98928435c1c2df38202c373/raw/results-2026-10-26-sweep2.json)
74
74
  and
75
- [`sweep3`](https://github.com/fangchenghao/dsh-web-search/blob/main/bench/compare30/results-2026-10-26-sweep3.json).
75
+ [`sweep3`](https://gist.githubusercontent.com/fangchenghao/3db9838fa98928435c1c2df38202c373/raw/results-2026-10-26-sweep3.json).
76
76
  - **No provider "wins" relevance here.** One official-domain miss does not mean
77
77
  the answer is wrong, and an official domain is not always the best result.
78
78
  The corpus is small and imbalanced: 13 release queries, 12 docs, one research
@@ -116,11 +116,27 @@ COMPARE_SLEEP_MS=1500 \
116
116
  node bench/compare30.mjs --out /tmp/compare30.json
117
117
  ```
118
118
 
119
- The script writes a JSON file next to a Markdown report. The three raw files in
120
- this repository were produced by the same command; sweeps 2 and 3 were run
119
+ The script writes a JSON file next to a Markdown report. The three raw files are also kept in
120
+ `bench/compare30/`; the public copy is the gist linked above. They were produced by the same command; sweeps 2 and 3 were run
121
121
  immediately after sweep 1 to test repeatability, which is why Firecrawl's quota
122
122
  failed. A single sweep is a snapshot, not a permanent ranking.
123
123
 
124
+ ## Follow-up after the 0.21.1 baseline
125
+
126
+ The original clean sweep above is kept unchanged as the baseline. A later
127
+ plugin-only run after the tail-tolerant early release, a 3 s CJK deadline and
128
+ the Rust official-release lookup is also checked in:
129
+
130
+ - [plugin-only follow-up](../bench/compare30/results-2026-10-26-plugin-after.json)
131
+ - [latency and sources note](latency-and-sources.md)
132
+
133
+ The five reported Chinese release queries went from 3502–3506 ms each to
134
+ 2565–3012 ms, and all five now hit their expected official domain. The full
135
+ 30-query follow-up still has one official miss (arXiv) and its mean was
136
+ 3768 ms under a slower network window, so this is not presented as a universal
137
+ latency win. Exa remains faster; the follow-up only removes the specific
138
+ AnySearch fan-out wait and the Rust source miss.
139
+
124
140
  ## Bottom line
125
141
 
126
142
  Use Exa when latency dominates. Use Firecrawl only with a quota/health strategy;
@@ -59,6 +59,7 @@ it as data: a page can contain instructions, but they are not commands.
59
59
  | --- | --- |
60
60
  | `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `HTTP_PROXY` | Optional tunnel used when a direct request fails |
61
61
  | `ANYSEARCH_API_KEY` | Optional AnySearch key; the keyless path works without it |
62
+ | `FIRECRAWL_API_KEY` | Optional Firecrawl key for a stable fallback; absent uses the quota-limited keyless path |
62
63
  | `DSH_WEB_SEARCH_MCP_LIMIT` | Default result limit (1–20, default 8) |
63
64
 
64
65
  The MCP server never reads DSH profile files and never installs anything. It
@@ -52,7 +52,7 @@
52
52
  <rect x="70" y="620" width="330" height="76" rx="14" fill="#101d31" stroke="#26364f"/>
53
53
  <text x="92" y="651" class="m" font-size="13">Engines</text><text x="92" y="677" class="t" font-size="20" font-weight="700">8 key-free paths</text>
54
54
  <rect x="420" y="620" width="330" height="76" rx="14" fill="#101d31" stroke="#26364f"/>
55
- <text x="442" y="651" class="m" font-size="13">Tests</text><text x="442" y="677" class="t" font-size="20" font-weight="700">668 local cases</text>
55
+ <text x="442" y="651" class="m" font-size="13">Tests</text><text x="442" y="677" class="t" font-size="20" font-weight="700">680 local cases</text>
56
56
  <rect x="770" y="620" width="360" height="76" rx="14" fill="#101d31" stroke="#26364f"/>
57
57
  <text x="792" y="651" class="m" font-size="13">Cross-agent</text><text x="792" y="677" class="t" font-size="20" font-weight="700">MCP stdio server</text>
58
58
  </g>
@@ -0,0 +1,181 @@
1
+ # Latency, Exa, and official sources
2
+
3
+ This note answers three questions from the public 30-query comparison:
4
+
5
+ 1. Why is Exa around 1 s while this plugin was around 3 s?
6
+ 2. Can the AnySearch tail and the Chinese Rust miss be fixed?
7
+ 3. Does the fix require a vector database or a different retrieval model?
8
+
9
+ The short answer: Exa is a different architecture, not a faster RRF loop. This
10
+ plugin can borrow tail-tolerance and source-specific facts, but copying Exa's
11
+ semantic index would be a different product. The changes below improve the
12
+ interactive path without pretending that fan-out scraping is free.
13
+
14
+ ## 1. Why Exa is fast
15
+
16
+ Exa describes its stack as a hosted semantic index, not a wrapper around a
17
+ search-engine results page:
18
+
19
+ - A query is embedded and matched against a web-scale vector database.
20
+ - Exa reports that it truncates 4096-dimensional embeddings to 256 dimensions
21
+ with Matryoshka training (about 20x less memory), replaces 16-bit floats with
22
+ 1-bit document vectors (another 16x), and only searches the query's cluster
23
+ plus nearby clusters instead of every vector. It reports roughly 1000x
24
+ throughput from clustering, then reranks the shortlist with uncompressed
25
+ vectors. Metadata filters use inverted indexes. This is deliberately unlike
26
+ a global HNSW graph, which is harder to shard and filter:
27
+ [engineering post](https://exa.ai/blog/building-web-scale-vector-db).
28
+ - Results come back from one API call. There is no per-query fan-out to several
29
+ scraped HTML endpoints, and no client-side fusion of eight formats.
30
+ - Exa's low-latency mode is explicitly marketed as sub-200 ms in a
31
+ us-west-1 datacenter, with about 50 ms network latency in their test. That is
32
+ a vendor claim measured on their infrastructure, not an independent result.
33
+ The same post notes that, with Exa 3.0, Lite is now the recommended
34
+ low-latency search type: [Exa Instant](https://exa.ai/blog/exa-instant).
35
+
36
+ Our comparison used the anonymous Exa MCP endpoint, not Exa Instant. It measured
37
+ about 1.0 s end to end through a local tunnel. Even so, the structural difference
38
+ is real: **one indexed retrieval call versus a fan-out over independently hosted
39
+ and scraped engines.**
40
+
41
+ The fusion itself is cheap. The milliseconds are spent waiting for network
42
+ legs and parsing heterogeneous result pages, and the slowest leg sets the tail.
43
+
44
+ ## 2. What was changed
45
+
46
+ ### Tail-tolerant early release
47
+
48
+ `engines.ts` used to require contributors from the *leading region* before it
49
+ would release a tier. A Chinese query therefore waited for both AnySearch and
50
+ CN Bing even when a global primary had already supplied useful rows. It now
51
+ counts contributors across regions and releases when:
52
+
53
+ - the result budget is filled, and
54
+ - at least two independent engines contributed.
55
+
56
+ `wait: 'all'` still waits for every engine; `wait: 'fast'` still returns the
57
+ first answer. This follows the practical lesson from
58
+ [The Tail at Scale](https://dl.acm.org/doi/10.1145/2408776.2408794): for an
59
+ interactive request, do not make the response wait for every replica or
60
+ partition when a useful answer is already available. It is early termination,
61
+ not full hedged requests, and the distinction is deliberate: a second copy of a
62
+ scraped request would add load and may trigger more bot checks.
63
+
64
+ The interactive deadlines are now 5 s for Latin-script queries and 3 s for CJK
65
+ queries. That is a hard product decision: the plugin may return fewer rows on a
66
+ pathological query rather than let one slow engine define the tail.
67
+
68
+ ### Official release facts
69
+
70
+ A generic search query is not a reliable way to find a version-specific release
71
+ announcement. `official-sources.ts` now starts a narrow lookup in parallel with
72
+ the ordinary fan-out:
73
+
74
+ - **PostgreSQL**: the official versioning page (existing path).
75
+ - **Rust**: the GitHub release endpoint gives the UTC publication date; the
76
+ canonical official URL is then
77
+ `https://blog.rust-lang.org/YYYY/MM/DD/Rust-X.Y.Z/`.
78
+ - **Scoped npm packages**: an explicitly `npm`-scoped query uses the existing
79
+ npm vertical and puts the package page first.
80
+
81
+ The lookup is capped at 2.5 s and failure is contained: if the official source
82
+ is unavailable, the ordinary rows remain. The Rust GitHub endpoint is tried
83
+ directly first because measurement showed the direct request was much faster
84
+ than the tunnel for this API, with the tunnel kept as fallback.
85
+
86
+ For `Rust 1.90 发布说明`, the official row is now
87
+ `https://blog.rust-lang.org/2025/09/18/Rust-1.90.0/` at rank 1 rather than the
88
+ Rust home page or a third-party changelog.
89
+
90
+ ### Firecrawl is still a fallback, not a primary
91
+
92
+ Nothing in a client can manufacture keyless Firecrawl quota. The honest fix is:
93
+
94
+ - keep Firecrawl in the fallback tier;
95
+ - let the existing health tracker cool it down after `429`/blocks;
96
+ - support an optional `firecrawlApiKey` / `FIRECRAWL_API_KEY` for users who
97
+ need repeatable fallback behavior;
98
+ - record keyless quota failures in the public comparison instead of hiding
99
+ them.
100
+
101
+ The keyless arm is suitable for a one-off check, not for a 30-query sweep
102
+ repeated three times.
103
+
104
+ ## 3. Measurements
105
+
106
+ ### Five Chinese release queries
107
+
108
+ The original clean sweep ran these at the 3.5 s auto deadline. All five reported
109
+ `anysearch: still running at deadline`; four hit the expected official domain.
110
+
111
+ A controlled rerun of the same five queries after this change:
112
+
113
+ | Query | Before | After | Official after |
114
+ | --- | ---: | ---: | --- |
115
+ | Node.js 24 发布说明 | 3502 ms | 3012 ms | yes |
116
+ | PostgreSQL 18 发布说明 | 3504 ms | 3002 ms | yes |
117
+ | React 19 发布说明 | 3503 ms | 3003 ms | yes |
118
+ | Python 3.14 发布说明 | 3506 ms | 2978 ms | yes |
119
+ | Rust 1.90 发布说明 | 3502 ms | 2565 ms | yes |
120
+
121
+ All five now hit the expected official host. AnySearch still appears as a
122
+ deadline failure on a few runs; that is the `wait` budget doing its job, not a
123
+ blocking wait. The user-visible tail is capped at about 3 s for these queries.
124
+
125
+ ### Full 30-query follow-up
126
+
127
+ A plugin-only run with the official lookups enabled produced:
128
+
129
+ | Metric | Original clean sweep | Follow-up |
130
+ | --- | ---: | ---: |
131
+ | Answered | 30/30 | 30/30 |
132
+ | Official-domain hit | 28/30 | 29/30 |
133
+ | Mean unique domains | 5.83 | 5.67 |
134
+ | Mean latency | 3008 ms | 3768 ms |
135
+ | p50 | 2791 ms | 4000 ms |
136
+ | p95 | 5188 ms | 5003 ms |
137
+
138
+ The follow-up's **mean got worse**, not better. The run happened under more
139
+ variable upstream latency, and the new official lookup adds work to release
140
+ queries; the 5 s cap improves the p95 but does not make the average as fast as
141
+ the original clean network window. The remaining official miss is the arXiv
142
+ query, unrelated to the three issues above. Raw data:
143
+ [`results-2026-10-26-plugin-after.json`](../bench/compare30/results-2026-10-26-plugin-after.json).
144
+
145
+ The five-query table is the targeted result for the reported Chinese tail; the
146
+ 30-query table is the honest full-run result, including the regression in mean.
147
+
148
+ ## 4. Do we need a vector database?
149
+
150
+ No, not to fix these issues. A vector database would help exploratory semantic
151
+ queries, but it would add an indexing pipeline, embedding model, freshness
152
+ policy and a new hosting cost. It would also make the keyless promise harder:
153
+ the current project can run without an API key and without owning a crawl.
154
+
155
+ What is worth borrowing from the retrieval literature:
156
+
157
+ - **HNSW** shows how a graph index makes approximate nearest-neighbour search
158
+ logarithmic in practice; it is one low-latency option Exa does not claim to
159
+ use: [Malkov & Yashunin](https://doi.org/10.1109/TPAMI.2018.2889473).
160
+ - **ColBERT** shows that late interaction can preserve quality while reducing
161
+ the amount of expensive query-document interaction:
162
+ [Khattab & Zaharia](https://doi.org/10.1145/3397271.3401075).
163
+ - **Matryoshka representation learning** is directly relevant to Exa's
164
+ truncated-embedding approach:
165
+ [Kusupati et al.](https://arxiv.org/abs/2205.13147).
166
+ - **The Tail at Scale** is the basis for not waiting for every fan-out leg:
167
+ [Dean & Barroso](https://dl.acm.org/doi/10.1145/2408776.2408794).
168
+
169
+ The plugin deliberately does **not** claim to be a semantic search engine. Its
170
+ advantage is keyless, multi-source coverage and explicit failure reporting; its
171
+ cost is that it cannot match a single-index provider's tail latency on every
172
+ query.
173
+
174
+ ## 5. What remains
175
+
176
+ - AnySearch can still be the slow leg. The 3 s CJK budget bounds the damage,
177
+ but a query may return fewer than eight rows from the remaining engines.
178
+ - Keyless Firecrawl can still return `429` immediately; use a key if you need
179
+ repeatability, or treat it as a fallback only.
180
+ - Exa remains faster in the clean sweep. The public comparison should continue
181
+ to say so.
@@ -23,7 +23,7 @@ Only `0.2.1-rc.2` is declared, because that is the version actually exercised.
23
23
  ## Environment
24
24
 
25
25
  - macOS (arm64), DeepSeek Harness **0.2.1-rc.2** (`dsh --version`), Node 22.18+.
26
- - Runtime artifacts: `lib/` is committed to this repository (78 files, 912 KiB),
26
+ - Runtime artifacts: `lib/` is committed to this repository (82 files, 932 KiB),
27
27
  so the fixed commit's tree contains `lib/index.js` and `lib/index.d.ts`.
28
28
  - Disposable profile for the destructive steps: a fresh `DSH_HOME` created with
29
29
  `mktemp -d`; the real `~/.dsh` was never written to. Bundled pnpm 11.28.5.
package/lib/config.d.ts CHANGED
@@ -48,6 +48,7 @@ export declare const Config: z<Schemastery.ObjectS<NoInfer<{
48
48
  cacheFile: z<string, string, "volatile-defined">;
49
49
  anysearchApiKey: z<string, string, "volatile-defined">;
50
50
  anysearchDomain: z<string, string, "volatile-defined">;
51
+ firecrawlApiKey: z<string, string, "volatile-defined">;
51
52
  baseCooldownMs: z<number, number, "volatile-defined">;
52
53
  maxCooldownMs: z<number, number, "volatile-defined">;
53
54
  domainSources: z<NoInfer<string | string[]>, NoInfer<string | string[]>, "volatile-defined">;
@@ -79,6 +80,7 @@ export declare const Config: z<Schemastery.ObjectS<NoInfer<{
79
80
  cacheFile: z<string, string, "volatile-defined">;
80
81
  anysearchApiKey: z<string, string, "volatile-defined">;
81
82
  anysearchDomain: z<string, string, "volatile-defined">;
83
+ firecrawlApiKey: z<string, string, "volatile-defined">;
82
84
  baseCooldownMs: z<number, number, "volatile-defined">;
83
85
  maxCooldownMs: z<number, number, "volatile-defined">;
84
86
  domainSources: z<NoInfer<string | string[]>, NoInfer<string | string[]>, "volatile-defined">;
@@ -139,6 +141,8 @@ export interface ResolvedConfig {
139
141
  anysearchApiKey: string;
140
142
  /** AnySearch vertical domain, or an empty string for the open web. */
141
143
  anysearchDomain: string;
144
+ /** Optional Firecrawl credential for the quota-limited fallback. */
145
+ firecrawlApiKey: string;
142
146
  /** Ranking knobs. */
143
147
  rank: RankOptions;
144
148
  /** Cache lifetime in milliseconds. */
package/lib/config.js CHANGED
@@ -112,6 +112,10 @@ export const Config = z.object({
112
112
  .string()
113
113
  .default('')
114
114
  .description('只搜某一类专业内容(如 finance 财经、academic 学术、code 代码)。要专查某类才填;日常搜索留空。').volatile(),
115
+ firecrawlApiKey: z
116
+ .string()
117
+ .default('')
118
+ .description('Firecrawl 通行证(可选):留空仍可匿名调用兜底源,但匿名配额很紧,连续查询会返回 429;填自己的 key 后 Firecrawl 兜底更稳定。不填不影响主搜索引擎。').volatile(),
115
119
  baseCooldownMs: z.natural().default(300000).description('搜索引擎说“你查得太频繁了”后,先歇多久(毫秒)。歇太久会少一个引擎,太短易被再拒;保持 300000(5 分钟)即可。').volatile(),
116
120
  maxCooldownMs: z.natural().default(1800000).description('它反复被拒时,最长让它歇多久(毫秒),也就是等待一步步加长的封顶值。一般不用改;保持 1800000(30 分钟)即可。').volatile(),
117
121
  domainSources: z
@@ -190,6 +194,7 @@ export function resolveConfig(config, callLimit) {
190
194
  fallbackOn: [...settings.fallbackOn],
191
195
  anysearchApiKey: settings.anysearchApiKey.trim(),
192
196
  anysearchDomain: settings.anysearchDomain.trim(),
197
+ firecrawlApiKey: settings.firecrawlApiKey.trim(),
193
198
  domainSourceIds: domainSources.length > 0 ? domainSources : [...DEFAULT_VERTICAL_IDS],
194
199
  domainLimit: settings.domainLimit,
195
200
  domainTimeoutMs: settings.domainTimeoutMs,
package/lib/engines.d.ts CHANGED
@@ -22,12 +22,24 @@ import type { EngineHealth } from './health.ts';
22
22
  * runs as a first choice or only when no primary answered.
23
23
  */
24
24
  export declare const HOSTS: readonly HostSpec[];
25
+ /**
26
+ * Whether a tier may release before its soft deadline.
27
+ *
28
+ * `limit` and `needed` are intentionally separate: eight rows from one engine
29
+ * are still a single-engine answer, while eight rows from two engines are a
30
+ * genuine cross-engine result. Counting every region here is what prevents a
31
+ * slow lead engine from holding a query open after another region supplied
32
+ * enough rows.
33
+ */
34
+ export declare function tierReleaseReady(available: number, contributors: number, limit: number, needed: number): boolean;
25
35
  /** Options for {@link search}: the seam's budget plus the plugin's runtime. */
26
36
  export interface EngineRunOptions extends SearchOptions {
27
37
  /** AnySearch Bearer token; empty or absent means anonymous access. */
28
38
  anysearchApiKey?: string;
29
39
  /** AnySearch vertical domain, such as `finance` or `finance.us_stock`. */
30
40
  anysearchDomain?: string;
41
+ /** Optional Firecrawl Bearer token; absent keeps the keyless fallback. */
42
+ firecrawlApiKey?: string;
31
43
  /** Ranking knobs, falling back to the documented defaults. */
32
44
  rank?: RankOptions;
33
45
  /** Failure classes that may trigger the fallback tier; absent means all. */
@@ -95,13 +107,13 @@ export interface SearchTrace {
95
107
  /**
96
108
  * Apply per-run configuration to an engine definition.
97
109
  *
98
- * Only AnySearch is configurable today: it authenticates with a Bearer token and
99
- * can be pointed at one of its vertical domains. Returning a copy keeps the
100
- * exported `HOSTS` catalogue static, so a stale credential can never leak into
101
- * the next run through a mutated descriptor.
110
+ * AnySearch can be pointed at one of its vertical domains; both AnySearch and
111
+ * Firecrawl authenticate with an optional Bearer token. Returning a copy keeps
112
+ * the exported `HOSTS` catalogue static, so a stale credential can never leak
113
+ * into the next run through a mutated descriptor.
102
114
  * @param host - the engine definition from `HOSTS`.
103
- * @param options - the run options carrying the credential and domain.
115
+ * @param options - the run options carrying credentials and the domain.
104
116
  * @returns the engine definition to use for this run.
105
117
  */
106
- export declare function resolveHost(host: HostSpec, options: Pick<EngineRunOptions, 'anysearchApiKey' | 'anysearchDomain'>): HostSpec;
118
+ export declare function resolveHost(host: HostSpec, options: Pick<EngineRunOptions, 'anysearchApiKey' | 'anysearchDomain' | 'firecrawlApiKey'>): HostSpec;
107
119
  export declare function search(query: string, signal: AbortSignal | undefined, options: EngineRunOptions): Promise<SearchOutcome>;
package/lib/engines.js CHANGED
@@ -179,7 +179,10 @@ function ddgWindow(timeRange) {
179
179
  */
180
180
  /** Soft deadline per wait policy, in milliseconds. */
181
181
  const SOFT_DEADLINE = {
182
- auto: (ascii) => (ascii ? 6000 : 3500),
182
+ // Auto is the interactive path: cap a slow fan-out leg instead of letting
183
+ // the caller absorb the slowest engine. `all` keeps the completeness escape
184
+ // hatch for callers that explicitly want every engine.
185
+ auto: (ascii) => (ascii ? 5000 : 3000),
183
186
  all: () => 10000,
184
187
  fast: () => 3500,
185
188
  };
@@ -250,6 +253,18 @@ function allowsFallback(details, fallbackOn) {
250
253
  const allowed = new Set(fallbackOn);
251
254
  return details.every((detail) => allowed.has(detail.kind));
252
255
  }
256
+ /**
257
+ * Whether a tier may release before its soft deadline.
258
+ *
259
+ * `limit` and `needed` are intentionally separate: eight rows from one engine
260
+ * are still a single-engine answer, while eight rows from two engines are a
261
+ * genuine cross-engine result. Counting every region here is what prevents a
262
+ * slow lead engine from holding a query open after another region supplied
263
+ * enough rows.
264
+ */
265
+ export function tierReleaseReady(available, contributors, limit, needed) {
266
+ return available >= limit && contributors >= needed;
267
+ }
253
268
  /**
254
269
  * Ask one engine, rotating to the next key when the failure blames the key.
255
270
  *
@@ -319,14 +334,22 @@ async function runTier(hosts, query, signal, timeoutMs, failures, failureDetails
319
334
  const rawCounts = new Array(hosts.length).fill(0);
320
335
  const answered = () => outcomes.filter((outcome) => outcome instanceof SearchError || outcome !== undefined);
321
336
  const available = () => outcomes.reduce((total, outcome) => total + (outcome instanceof SearchError || outcome === undefined ? 0 : outcome.sources.length), 0);
322
- const contributors = () => outcomes.filter((outcome, index) => counters(hosts[index]) && !(outcome instanceof SearchError) && outcome !== undefined && outcome.sources.length > 0).length;
337
+ // A contributor is any engine that returned rows, not only an engine in the
338
+ // leading region. The lead still decides ordering and the `leadSettled`
339
+ // shortcut; it must not also veto an early release when a healthy global
340
+ // engine has already supplied useful rows. This is the fan-out tail issue:
341
+ // a slow AnySearch request should not hold the whole Chinese query open
342
+ // after CN Bing and a global engine have enough rows between them.
343
+ const contributors = () => outcomes.filter((outcome) => !(outcome instanceof SearchError) && outcome !== undefined && outcome.sources.length > 0).length;
323
344
  const leadTotal = hosts.filter(counters).length || hosts.length;
324
345
  const needed = Math.min(limits.needed, leadTotal);
325
346
  // Nothing more can arrive from the leading region once every one of its
326
347
  // engines settled, so a failed engine there must not hold the tier open until
327
348
  // the soft deadline.
328
349
  const leadSettled = () => outcomes.filter((outcome, index) => counters(hosts[index]) && outcome !== undefined).length >= leadTotal;
329
- const done = () => answered().length === hosts.length || leadSettled() || (available() >= limits.limit && contributors() >= needed);
350
+ const done = () => answered().length === hosts.length ||
351
+ leadSettled() ||
352
+ tierReleaseReady(available(), contributors(), limits.limit, needed);
330
353
  let releasedEarly = false;
331
354
  let closed = false;
332
355
  let released = () => { };
@@ -405,15 +428,19 @@ async function runTier(hosts, query, signal, timeoutMs, failures, failureDetails
405
428
  /**
406
429
  * Apply per-run configuration to an engine definition.
407
430
  *
408
- * Only AnySearch is configurable today: it authenticates with a Bearer token and
409
- * can be pointed at one of its vertical domains. Returning a copy keeps the
410
- * exported `HOSTS` catalogue static, so a stale credential can never leak into
411
- * the next run through a mutated descriptor.
431
+ * AnySearch can be pointed at one of its vertical domains; both AnySearch and
432
+ * Firecrawl authenticate with an optional Bearer token. Returning a copy keeps
433
+ * the exported `HOSTS` catalogue static, so a stale credential can never leak
434
+ * into the next run through a mutated descriptor.
412
435
  * @param host - the engine definition from `HOSTS`.
413
- * @param options - the run options carrying the credential and domain.
436
+ * @param options - the run options carrying credentials and the domain.
414
437
  * @returns the engine definition to use for this run.
415
438
  */
416
439
  export function resolveHost(host, options) {
440
+ if (host.id === 'firecrawl') {
441
+ const firecrawlKey = (options.firecrawlApiKey ?? '').trim();
442
+ return firecrawlKey.length === 0 ? host : { ...host, headers: { authorization: `Bearer ${firecrawlKey}` } };
443
+ }
417
444
  if (host.id !== 'anysearch')
418
445
  return host;
419
446
  const key = (options.anysearchApiKey ?? '').trim();
@@ -453,7 +480,11 @@ export async function search(query, signal, options) {
453
480
  const allow = options.hostIds !== undefined && options.hostIds.length > 0 ? new Set(options.hostIds) : undefined;
454
481
  const hosts = HOSTS.filter((host) => allow === undefined || allow.has(host.id))
455
482
  .filter((host) => host.viaProxy !== true || proxy !== undefined)
456
- .map((host) => resolveHost(host, options.anysearchDomain === undefined ? {} : { anysearchDomain: options.anysearchDomain }));
483
+ .map((host) => resolveHost(host, {
484
+ ...(options.anysearchApiKey === undefined ? {} : { anysearchApiKey: options.anysearchApiKey }),
485
+ ...(options.anysearchDomain === undefined ? {} : { anysearchDomain: options.anysearchDomain }),
486
+ ...(options.firecrawlApiKey === undefined ? {} : { firecrawlApiKey: options.firecrawlApiKey }),
487
+ }));
457
488
  // Skip engines that are cooling down, but never end up with no engine at all:
458
489
  // a throttled engine is still better than no search.
459
490
  const now = Date.now();
@@ -475,9 +506,13 @@ export async function search(query, signal, options) {
475
506
  const fallbackOn = options.fallbackOn;
476
507
  const ascii = !CJK.test(query);
477
508
  const lead = ascii ? 'global' : 'cn';
478
- const wanted = usable.filter((host) => leadsQuery(host, lead)).length;
479
509
  const softMs = options.softMs ?? SOFT_DEADLINE[wait](ascii);
480
- const needed = wait === 'all' ? usable.length : wait === 'fast' ? 1 : wanted > 0 ? wanted : Math.min(2, usable.length);
510
+ // `auto` is a latency policy, not a completeness policy: two independent
511
+ // contributors are enough to make fusion meaningful. Waiting for every lead
512
+ // engine made one slow AnySearch leg hold an otherwise complete answer open
513
+ // for the full soft deadline. `all` still asks for every engine, and `fast`
514
+ // keeps its first-answer behavior.
515
+ const needed = wait === 'all' ? usable.length : wait === 'fast' ? 1 : Math.min(2, usable.length);
481
516
  const limits = { limit, softMs, needed, lead: wait === 'all' ? 'all' : lead, timeRange };
482
517
  // Ranking does not drop rows; the engine layer does. `timeRange` is passed
483
518
  // into the rank options only so undated rows are scored lower while a window
package/lib/index.d.ts CHANGED
@@ -97,6 +97,7 @@ declare const _default: typeof apply & {
97
97
  cacheFile: import("@deepseek-ai/schemastery").default<string, string, "volatile-defined">;
98
98
  anysearchApiKey: import("@deepseek-ai/schemastery").default<string, string, "volatile-defined">;
99
99
  anysearchDomain: import("@deepseek-ai/schemastery").default<string, string, "volatile-defined">;
100
+ firecrawlApiKey: import("@deepseek-ai/schemastery").default<string, string, "volatile-defined">;
100
101
  baseCooldownMs: import("@deepseek-ai/schemastery").default<number, number, "volatile-defined">;
101
102
  maxCooldownMs: import("@deepseek-ai/schemastery").default<number, number, "volatile-defined">;
102
103
  domainSources: import("@deepseek-ai/schemastery").default<NoInfer<string | string[]>, NoInfer<string | string[]>, "volatile-defined">;
@@ -128,6 +129,7 @@ declare const _default: typeof apply & {
128
129
  cacheFile: import("@deepseek-ai/schemastery").default<string, string, "volatile-defined">;
129
130
  anysearchApiKey: import("@deepseek-ai/schemastery").default<string, string, "volatile-defined">;
130
131
  anysearchDomain: import("@deepseek-ai/schemastery").default<string, string, "volatile-defined">;
132
+ firecrawlApiKey: import("@deepseek-ai/schemastery").default<string, string, "volatile-defined">;
131
133
  baseCooldownMs: import("@deepseek-ai/schemastery").default<number, number, "volatile-defined">;
132
134
  maxCooldownMs: import("@deepseek-ai/schemastery").default<number, number, "volatile-defined">;
133
135
  domainSources: import("@deepseek-ai/schemastery").default<NoInfer<string | string[]>, NoInfer<string | string[]>, "volatile-defined">;
package/lib/index.js CHANGED
@@ -27,7 +27,7 @@ import { registerEngineTestBridge, registerUpdateBridge } from "./bridge.js";
27
27
  import { registerAdvancedSearchTool, undatedWindowNote } from "./advanced-search.js";
28
28
  import { searchVerticals } from "./verticals.js";
29
29
  import { routePackageQuery } from "./query-router.js";
30
- import { fetchPostgresSupport, isPostgresVersionQuery } from "./official-support.js";
30
+ import { mergeOfficial, officialLookup } from "./official-sources.js";
31
31
  import { chromeFeedNote, chromeWindow, fetchChromeReleases, filterChromeChannel, hasNonStableChromePosts, isChromeSecurityQuery, } from "./official-feed.js";
32
32
  import { registerSearchPrompt } from "./system-prompt.js";
33
33
  import { registerUnifiedTool } from "./tool-suite.js";
@@ -147,6 +147,7 @@ export function apply(ctx, config = {}) {
147
147
  rank: resolved.rank,
148
148
  ...(resolved.anysearchApiKey.length === 0 ? {} : { anysearchApiKey: resolved.anysearchApiKey }),
149
149
  ...(resolved.anysearchDomain.length === 0 ? {} : { anysearchDomain: resolved.anysearchDomain }),
150
+ ...(resolved.firecrawlApiKey.length === 0 ? {} : { firecrawlApiKey: resolved.firecrawlApiKey }),
150
151
  cache,
151
152
  health,
152
153
  ...(signal === undefined ? {} : { signal }),
@@ -236,6 +237,7 @@ export function apply(ctx, config = {}) {
236
237
  ...(live.proxy === undefined ? {} : { proxy: live.proxy }),
237
238
  ...(live.anysearchApiKey.length === 0 ? {} : { anysearchApiKey: live.anysearchApiKey }),
238
239
  ...(live.anysearchDomain.length === 0 ? {} : { anysearchDomain: live.anysearchDomain }),
240
+ ...(live.firecrawlApiKey.length === 0 ? {} : { firecrawlApiKey: live.firecrawlApiKey }),
239
241
  hostIds: [engine],
240
242
  });
241
243
  };
@@ -254,27 +256,26 @@ export function apply(ctx, config = {}) {
254
256
  const asked = typeof request.maxResults === 'number' && Number.isInteger(request.maxResults) && request.maxResults > 0
255
257
  ? request.maxResults
256
258
  : undefined;
259
+ const live = resolveConfig(section, undefined);
260
+ // Start an official lookup before the fan-out and await it afterwards.
261
+ // It then costs max(search, lookup), not search + lookup, which matters
262
+ // for a version query where the first result should be authoritative but
263
+ // the ordinary engines still need to fill the remaining rows.
264
+ const official = officialLookup(query, {
265
+ timeoutMs: Math.min(live.timeoutMs, 2_500),
266
+ ...(live.proxy === undefined ? {} : { proxy: live.proxy }),
267
+ ...(signal === undefined ? {} : { signal }),
268
+ });
257
269
  const outcome = await runQuery(query, asked, signal);
258
- // A version/EOL question about PostgreSQL is better answered by the
259
- // project's own support table than by whatever an engine ranks first.
270
+ // A version/EOL question about PostgreSQL or Rust is better answered by
271
+ // the project's own release facts than by whatever an engine ranks first.
260
272
  // The lookup is query-scoped and failure-tolerant; it never replaces the
261
273
  // ordinary rows when the page is unavailable.
262
- let sources = outcome.sources;
263
- if (isPostgresVersionQuery(query)) {
264
- const live = resolveConfig(section, undefined);
265
- const official = await fetchPostgresSupport({
266
- timeoutMs: Math.min(live.timeoutMs, 8_000),
267
- ...(live.proxy === undefined ? {} : { proxy: live.proxy }),
268
- ...(signal === undefined ? {} : { signal }),
269
- });
270
- if (official !== undefined) {
271
- sources = [official, ...sources.filter((source) => source.url !== official.url)];
272
- }
273
- }
274
+ const sources = mergeOfficial(outcome.sources, await official);
274
275
  if (sources.length === 0) {
275
276
  throw new Error(`no search engine returned results for ${JSON.stringify(query)} (${outcome.failures.join('; ')})`);
276
277
  }
277
- const cap = Math.min(asked ?? resolveConfig(section, undefined).limit, resolveConfig(section, undefined).limit);
278
+ const cap = Math.min(asked ?? live.limit, live.limit);
278
279
  return { sources: sources.slice(0, cap), truncated: outcome.truncated || sources.length > cap };
279
280
  },
280
281
  });
@@ -0,0 +1,73 @@
1
+ /**
2
+ * Official Rust release notes for an explicitly versioned release query.
3
+ *
4
+ * Search engines often return the Rust home page or a third-party changelog for
5
+ * a query such as `Rust 1.90 发布说明`, even when the official announcement is
6
+ * indexed. The release announcement has a deterministic shape once its date is
7
+ * known: `https://blog.rust-lang.org/YYYY/MM/DD/Rust-X.Y.Z/`. GitHub's public
8
+ * release endpoint is key-free and exposes that date, so this module can add
9
+ * the official page as a high-priority row without hard-coding release dates.
10
+ *
11
+ * The lookup is deliberately narrow. Only a Rust query that names a numeric
12
+ * version *and* asks for release notes is enriched; a generic Rust tutorial
13
+ * keeps the normal engine ranking.
14
+ *
15
+ * @module @fangchenghao/dsh-web-search/official-release
16
+ */
17
+ import type { Source } from './types.ts';
18
+ /** Public GitHub release endpoint for the Rust repository. */
19
+ export declare const RUST_RELEASE_API = "https://api.github.com/repos/rust-lang/rust/releases/tags";
20
+ /** Official Rust announcement host. */
21
+ export declare const RUST_BLOG_HOST = "blog.rust-lang.org";
22
+ /** Inputs accepted by {@link fetchRustRelease}. */
23
+ export interface RustReleaseOptions {
24
+ /** Tunnel URL from the plugin's `proxy` setting, when configured. */
25
+ proxy?: string;
26
+ /** Per-request deadline in milliseconds. */
27
+ timeoutMs?: number;
28
+ /** Caller cancellation. */
29
+ signal?: AbortSignal;
30
+ /** Test seam; defaults to the platform `fetch`. */
31
+ fetch?: typeof fetch;
32
+ }
33
+ /** A version such as `1.90`, normalized to a release tag like `1.90.0`. */
34
+ export interface RustVersion {
35
+ /** User-facing version, e.g. `1.90.0`. */
36
+ readonly version: string;
37
+ /** GitHub tag, normally identical to {@link version}. */
38
+ readonly tag: string;
39
+ }
40
+ /** Extract a Rust version from a release query. */
41
+ export declare function rustReleaseVersion(query: string): RustVersion | undefined;
42
+ /** True for a query asking for a specific Rust release's notes. */
43
+ export declare function isRustReleaseQuery(query: string): boolean;
44
+ /**
45
+ * Build the canonical announcement URL from a GitHub release timestamp.
46
+ *
47
+ * Rust publishes the post on the UTC date in the release metadata. Using UTC
48
+ * avoids shifting the path when the machine is in a negative-offset timezone.
49
+ */
50
+ export declare function rustBlogUrl(version: string, publishedAt: string): string | undefined;
51
+ /**
52
+ * Parse a GitHub release response into one official-blog source.
53
+ *
54
+ * A malformed response, a missing publication date or a tag that does not
55
+ * match the requested version returns `undefined`; the caller then keeps the
56
+ * ordinary search rows.
57
+ *
58
+ * @param payload - GitHub JSON response body.
59
+ * @param requested - normalized version from the query.
60
+ * @returns the official source, or `undefined`.
61
+ */
62
+ export declare function parseRustRelease(payload: string, requested: RustVersion): Source | undefined;
63
+ /**
64
+ * Fetch the official Rust release page for a query.
65
+ *
66
+ * Failure is contained: an unavailable API, rate limit or malformed body
67
+ * returns `undefined`, and the caller keeps its ordinary search rows.
68
+ *
69
+ * @param query - the original user query.
70
+ * @param options - proxy, deadline, cancellation and fetch seam.
71
+ * @returns the official blog source, or `undefined`.
72
+ */
73
+ export declare function fetchRustRelease(query: string, options?: RustReleaseOptions): Promise<Source | undefined>;
@@ -0,0 +1,177 @@
1
+ import { proxyGet } from "./proxy.js";
2
+ import { text } from "./parsers.js";
3
+ /** Public GitHub release endpoint for the Rust repository. */
4
+ export const RUST_RELEASE_API = 'https://api.github.com/repos/rust-lang/rust/releases/tags';
5
+ /** Official Rust announcement host. */
6
+ export const RUST_BLOG_HOST = 'blog.rust-lang.org';
7
+ const DEFAULT_TIMEOUT_MS = 6_000;
8
+ /** Extract a Rust version from a release query. */
9
+ export function rustReleaseVersion(query) {
10
+ if (typeof query !== 'string' || !/\brust\b/iu.test(query))
11
+ return undefined;
12
+ const match = /\brust(?:\s+(?:release|version|v))?\s*([0-9]{1,2}\.[0-9]{1,2}(?:\.[0-9]{1,2})?)\b/iu.exec(query);
13
+ if (match?.[1] === undefined)
14
+ return undefined;
15
+ const raw = match[1];
16
+ const version = raw.split('.').length === 2 ? `${raw}.0` : raw;
17
+ return { version, tag: version };
18
+ }
19
+ /** True for a query asking for a specific Rust release's notes. */
20
+ export function isRustReleaseQuery(query) {
21
+ if (typeof query !== 'string')
22
+ return false;
23
+ if (rustReleaseVersion(query) === undefined)
24
+ return false;
25
+ return /(?:release|releases|changelog|notes?|announcement|发布|发行|版本|更新|说明|日志|公告)/iu.test(query);
26
+ }
27
+ /** Trim release Markdown/HTML into a bounded snippet. */
28
+ function releaseSnippet(body, version) {
29
+ const plain = text(typeof body === 'string' ? body : '')
30
+ .replace(/\[([^\]]+)\]\([^)]*\)/gu, '$1')
31
+ .replace(/https?:\/\/\S+/gu, ' ')
32
+ .replace(/\s+/gu, ' ')
33
+ .trim();
34
+ const prefix = `Official Rust ${version} release notes. `;
35
+ if (plain.length === 0)
36
+ return prefix.trim();
37
+ const budget = Math.max(80, 420 - prefix.length);
38
+ return `${prefix}${plain.length > budget ? `${plain.slice(0, budget).trimEnd()}…` : plain}`;
39
+ }
40
+ /**
41
+ * Build the canonical announcement URL from a GitHub release timestamp.
42
+ *
43
+ * Rust publishes the post on the UTC date in the release metadata. Using UTC
44
+ * avoids shifting the path when the machine is in a negative-offset timezone.
45
+ */
46
+ export function rustBlogUrl(version, publishedAt) {
47
+ const date = new Date(publishedAt);
48
+ if (Number.isNaN(date.getTime()))
49
+ return undefined;
50
+ const yyyy = String(date.getUTCFullYear()).padStart(4, '0');
51
+ const mm = String(date.getUTCMonth() + 1).padStart(2, '0');
52
+ const dd = String(date.getUTCDate()).padStart(2, '0');
53
+ return `https://${RUST_BLOG_HOST}/${yyyy}/${mm}/${dd}/Rust-${version}/`;
54
+ }
55
+ /**
56
+ * Parse a GitHub release response into one official-blog source.
57
+ *
58
+ * A malformed response, a missing publication date or a tag that does not
59
+ * match the requested version returns `undefined`; the caller then keeps the
60
+ * ordinary search rows.
61
+ *
62
+ * @param payload - GitHub JSON response body.
63
+ * @param requested - normalized version from the query.
64
+ * @returns the official source, or `undefined`.
65
+ */
66
+ export function parseRustRelease(payload, requested) {
67
+ if (typeof payload !== 'string' || payload.trim().length === 0)
68
+ return undefined;
69
+ let body;
70
+ try {
71
+ body = JSON.parse(payload);
72
+ }
73
+ catch {
74
+ return undefined;
75
+ }
76
+ if (typeof body !== 'object' || body === null)
77
+ return undefined;
78
+ const record = body;
79
+ const publishedAt = typeof record['published_at'] === 'string' ? record['published_at'] : '';
80
+ const url = rustBlogUrl(requested.version, publishedAt);
81
+ if (url === undefined)
82
+ return undefined;
83
+ const tag = typeof record['tag_name'] === 'string' ? record['tag_name'] : '';
84
+ if (tag.length > 0 && tag !== requested.tag && tag !== requested.version)
85
+ return undefined;
86
+ const title = `Announcing Rust ${requested.version} - Rust Blog`;
87
+ const snippet = releaseSnippet(record['body'], requested.version);
88
+ return {
89
+ url,
90
+ title,
91
+ snippet,
92
+ ...(Number.isNaN(new Date(publishedAt).getTime()) ? {} : { publishedAt: new Date(publishedAt).toISOString() }),
93
+ };
94
+ }
95
+ /** One direct GET with its own abort budget. */
96
+ async function directJson(url, headers, options, timeoutMs) {
97
+ const controller = new AbortController();
98
+ const timer = setTimeout(() => controller.abort(), timeoutMs);
99
+ const abort = () => controller.abort();
100
+ options.signal?.addEventListener('abort', abort, { once: true });
101
+ try {
102
+ const send = options.fetch ?? fetch;
103
+ const response = await send(url, { headers, signal: controller.signal });
104
+ if (!response.ok)
105
+ throw new Error(`HTTP ${response.status}`);
106
+ return await response.text();
107
+ }
108
+ finally {
109
+ clearTimeout(timer);
110
+ options.signal?.removeEventListener('abort', abort);
111
+ }
112
+ }
113
+ /**
114
+ * Read the GitHub release endpoint.
115
+ *
116
+ * Measurement showed this API is often fast directly and slow through the
117
+ * configured tunnel (a successful proxy request took ~4 s while the direct
118
+ * request took <1 s). Try direct first with a short exploratory budget, then
119
+ * spend the rest of the caller's deadline on the tunnel when direct fails.
120
+ * A test seam bypasses the tunnel so unit tests stay deterministic.
121
+ */
122
+ async function fetchJson(url, options, timeoutMs) {
123
+ const headers = {
124
+ accept: 'application/vnd.github+json',
125
+ 'x-github-api-version': '2022-11-28',
126
+ 'user-agent': 'Mozilla/5.0 (compatible; dsh-web-search; +https://github.com/fangchenghao/dsh-web-search)',
127
+ };
128
+ const started = Date.now();
129
+ if (options.fetch !== undefined)
130
+ return directJson(url, headers, options, timeoutMs);
131
+ try {
132
+ return await directJson(url, headers, options, Math.min(timeoutMs, 1_500));
133
+ }
134
+ catch (error) {
135
+ if (options.proxy === undefined || options.proxy.length === 0)
136
+ throw error;
137
+ const remaining = timeoutMs - (Date.now() - started);
138
+ if (remaining < 250)
139
+ throw error;
140
+ const response = await proxyGet(url, {
141
+ proxy: options.proxy,
142
+ timeoutMs: remaining,
143
+ headers,
144
+ ...(options.signal === undefined ? {} : { signal: options.signal }),
145
+ });
146
+ if (response.status < 200 || response.status >= 300)
147
+ throw new Error(`HTTP ${response.status}`);
148
+ return response.text;
149
+ }
150
+ }
151
+ /**
152
+ * Fetch the official Rust release page for a query.
153
+ *
154
+ * Failure is contained: an unavailable API, rate limit or malformed body
155
+ * returns `undefined`, and the caller keeps its ordinary search rows.
156
+ *
157
+ * @param query - the original user query.
158
+ * @param options - proxy, deadline, cancellation and fetch seam.
159
+ * @returns the official blog source, or `undefined`.
160
+ */
161
+ export async function fetchRustRelease(query, options = {}) {
162
+ if (!isRustReleaseQuery(query))
163
+ return undefined;
164
+ const requested = rustReleaseVersion(query);
165
+ if (requested === undefined)
166
+ return undefined;
167
+ const timeoutMs = typeof options.timeoutMs === 'number' && Number.isFinite(options.timeoutMs) && options.timeoutMs > 0
168
+ ? Math.min(options.timeoutMs, DEFAULT_TIMEOUT_MS)
169
+ : DEFAULT_TIMEOUT_MS;
170
+ try {
171
+ const payload = await fetchJson(`${RUST_RELEASE_API}/${requested.tag}`, options, timeoutMs);
172
+ return parseRustRelease(payload, requested);
173
+ }
174
+ catch {
175
+ return undefined;
176
+ }
177
+ }
@@ -0,0 +1,40 @@
1
+ /**
2
+ * Route a versioned query to the narrow official lookup that can answer it.
3
+ *
4
+ * Keep this as a separate seam rather than putting network calls inside
5
+ * `engines.search()`: the engine layer stays a pure keyless fan-out, while the
6
+ * provider, MCP server and public benchmark can all enrich the same result with
7
+ * the project's own release facts.
8
+ *
9
+ * @module @fangchenghao/dsh-web-search/official-sources
10
+ */
11
+ import type { Source } from './types.ts';
12
+ /** Options shared by the official lookup implementations. */
13
+ export interface OfficialLookupOptions {
14
+ /** Tunnel URL from the plugin's `proxy` setting, when configured. */
15
+ proxy?: string;
16
+ /** Per-request deadline in milliseconds. */
17
+ timeoutMs?: number;
18
+ /** Caller cancellation. */
19
+ signal?: AbortSignal;
20
+ /** Test seam; defaults to the platform `fetch`. */
21
+ fetch?: typeof fetch;
22
+ }
23
+ /** Scoped npm package name in an explicitly npm-scoped query. */
24
+ export declare function scopedNpmPackage(query: string): string | undefined;
25
+ /**
26
+ * Start the applicable official lookup, or return `undefined` when the query
27
+ * has no official resolver.
28
+ *
29
+ * Returning the promise instead of awaiting it lets a caller overlap the lookup
30
+ * with the ordinary search fan-out.
31
+ */
32
+ export declare function officialLookup(query: string, options?: OfficialLookupOptions): Promise<Source | undefined> | undefined;
33
+ /**
34
+ * Put official rows first and remove duplicate URLs.
35
+ *
36
+ * The order of `official` is preserved; ordinary rows keep their relative
37
+ * order. A source with no usable URL is discarded rather than allowed to
38
+ * displace a real result.
39
+ */
40
+ export declare function mergeOfficial(sources: readonly Source[], official: Source | undefined): Source[];
@@ -0,0 +1,68 @@
1
+ import { fetchPostgresSupport, isPostgresVersionQuery } from "./official-support.js";
2
+ import { fetchRustRelease, isRustReleaseQuery } from "./official-release.js";
3
+ import { searchVerticals } from "./verticals.js";
4
+ /** Scoped npm package name in an explicitly npm-scoped query. */
5
+ export function scopedNpmPackage(query) {
6
+ if (typeof query !== 'string' || !/\bnpm(?:js)?\b/iu.test(query))
7
+ return undefined;
8
+ const match = /(?:^|\s)(@[a-z0-9][a-z0-9._-]*\/[a-z0-9][a-z0-9._-]*)(?=\s|$)/iu.exec(query.trim());
9
+ return match?.[1];
10
+ }
11
+ /** Resolve one scoped npm package through the existing npm vertical. */
12
+ async function fetchNpmPackage(name, options) {
13
+ try {
14
+ const result = await searchVerticals(name, {
15
+ ids: ['npm'],
16
+ limit: 1,
17
+ timeoutMs: typeof options.timeoutMs === 'number' && Number.isFinite(options.timeoutMs) && options.timeoutMs > 0
18
+ ? Math.min(options.timeoutMs, 8_000)
19
+ : 8_000,
20
+ ...(options.proxy === undefined ? {} : { proxy: options.proxy }),
21
+ ...(options.signal === undefined ? {} : { signal: options.signal }),
22
+ });
23
+ return result.sources[0];
24
+ }
25
+ catch {
26
+ return undefined;
27
+ }
28
+ }
29
+ /**
30
+ * Start the applicable official lookup, or return `undefined` when the query
31
+ * has no official resolver.
32
+ *
33
+ * Returning the promise instead of awaiting it lets a caller overlap the lookup
34
+ * with the ordinary search fan-out.
35
+ */
36
+ export function officialLookup(query, options = {}) {
37
+ if (isPostgresVersionQuery(query)) {
38
+ return fetchPostgresSupport(options);
39
+ }
40
+ if (isRustReleaseQuery(query)) {
41
+ return fetchRustRelease(query, options);
42
+ }
43
+ const npm = scopedNpmPackage(query);
44
+ if (npm !== undefined) {
45
+ return fetchNpmPackage(npm, options);
46
+ }
47
+ return undefined;
48
+ }
49
+ /**
50
+ * Put official rows first and remove duplicate URLs.
51
+ *
52
+ * The order of `official` is preserved; ordinary rows keep their relative
53
+ * order. A source with no usable URL is discarded rather than allowed to
54
+ * displace a real result.
55
+ */
56
+ export function mergeOfficial(sources, official) {
57
+ if (official === undefined || official.url.length === 0)
58
+ return [...sources];
59
+ const seen = new Set([official.url]);
60
+ const merged = [official];
61
+ for (const source of sources) {
62
+ if (source.url.length === 0 || seen.has(source.url))
63
+ continue;
64
+ seen.add(source.url);
65
+ merged.push(source);
66
+ }
67
+ return merged;
68
+ }
package/mcp/server.mjs CHANGED
@@ -17,6 +17,7 @@
17
17
  * Environment:
18
18
  * DSH_WEB_SEARCH_PROXY / HTTPS_PROXY / HTTP_PROXY optional tunnel
19
19
  * ANYSEARCH_API_KEY optional AnySearch key
20
+ * FIRECRAWL_API_KEY optional Firecrawl key
20
21
  * DSH_WEB_SEARCH_MCP_LIMIT default result limit (8)
21
22
  *
22
23
  * The server writes protocol messages to stdout and diagnostics to stderr.
@@ -31,6 +32,7 @@ import { search } from '../lib/engines.js';
31
32
  import { createFetchProvider } from '../lib/fetch.js';
32
33
  import { searchVerticals } from '../lib/verticals.js';
33
34
  import { createResearchDefinition, formatResearchValue } from '../lib/research.js';
35
+ import { mergeOfficial, officialLookup } from '../lib/official-sources.js';
34
36
  import { wrapUntrustedContent } from '../lib/untrusted.js';
35
37
 
36
38
  const PACKAGE = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8'));
@@ -44,6 +46,7 @@ const MAX_TEXT = 80_000;
44
46
  const env = process.env;
45
47
  const proxy = env.DSH_WEB_SEARCH_PROXY ?? env.HTTPS_PROXY ?? env.HTTP_PROXY ?? undefined;
46
48
  const anysearchApiKey = env.ANYSEARCH_API_KEY ?? env.DSH_WEB_SEARCH_ANYSEARCH_KEY ?? '';
49
+ const firecrawlApiKey = env.FIRECRAWL_API_KEY ?? env.DSH_WEB_SEARCH_FIRECRAWL_KEY ?? '';
47
50
  const defaultLimit = (() => {
48
51
  const raw = Number(env.DSH_WEB_SEARCH_MCP_LIMIT);
49
52
  return Number.isInteger(raw) && raw > 0 ? Math.min(raw, MAX_LIMIT) : DEFAULT_LIMIT;
@@ -85,6 +88,7 @@ function searchOptions(limit, timeoutMs) {
85
88
  wait: 'auto',
86
89
  ...(proxy === undefined ? {} : { proxy }),
87
90
  ...(anysearchApiKey.length === 0 ? {} : { anysearchApiKey }),
91
+ ...(firecrawlApiKey.length === 0 ? {} : { firecrawlApiKey }),
88
92
  };
89
93
  }
90
94
 
@@ -171,8 +175,13 @@ async function callTool(name, args) {
171
175
  if (query.length === 0) throw new Error('search requires a non-empty query');
172
176
  const limit = intArg(args.limit, defaultLimit, 1, MAX_LIMIT);
173
177
  const timeoutMs = intArg(args.timeoutMs, 12_000, 1_000, 30_000);
178
+ const official = officialLookup(query, {
179
+ timeoutMs: Math.min(timeoutMs, 2_500),
180
+ ...(proxy === undefined ? {} : { proxy }),
181
+ });
174
182
  const outcome = await search(query, undefined, searchOptions(limit, timeoutMs));
175
- return formatSources(query, outcome.sources, { failures: outcome.failures });
183
+ const sources = mergeOfficial(outcome.sources, await official);
184
+ return formatSources(query, sources.slice(0, limit), { failures: outcome.failures });
176
185
  }
177
186
  if (name === 'fetch') {
178
187
  const url = typeof args.url === 'string' ? args.url.trim() : '';
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@fffangchenghao/dsh-web-search",
3
- "version": "0.21.0",
4
- "description": "DSH bundle: key-free web search, page fetch and one web_tools action tool for verticals, platforms and cited research; eight engines fused with RRF, 34 vertical sources, official-source checks, a replayable benchmark and a dependency-free MCP server.",
3
+ "version": "0.22.0",
4
+ "description": "DSH bundle: key-free web search, page fetch and one web_tools action tool for verticals, platforms and cited research; eight engines fused with RRF, 34 vertical sources, official release facts, a replayable benchmark, an optional Firecrawl key and a dependency-free MCP server.",
5
5
  "type": "module",
6
6
  "main": "lib/index.js",
7
7
  "types": "lib/index.d.ts",
@@ -32,7 +32,8 @@
32
32
  "docs/quickstart.md",
33
33
  "docs/cross-agent.md",
34
34
  "docs/comparison.md",
35
- "docs/images/web-tools-demo.svg"
35
+ "docs/images/web-tools-demo.svg",
36
+ "docs/latency-and-sources.md"
36
37
  ],
37
38
  "scripts": {
38
39
  "build": "tsc -p tsconfig.json",