@fffangchenghao/dsh-web-search 0.13.11 → 0.14.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,23 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.14.0
4
+
5
+ Distribution release. No runtime behaviour changed; the search and fetch paths are
6
+ the same as 0.13.11.
7
+
8
+ ### Added
9
+ - A banner, a pipeline diagram, a benchmark chart and a real terminal capture in
10
+ `docs/images/`, referenced by both READMEs through absolute raw URLs so they
11
+ render on GitHub and npm alike.
12
+ - `docs/rrf-ranking.md`: how the fusion works, why each default exists, the
13
+ baseline numbers, the 81-configuration sweep with its confidence interval, and
14
+ the limits the benchmark makes visible.
15
+ - A "Why this one" section and an honest comparison table in both READMEs,
16
+ including what the larger competing plugins do better.
17
+
18
+ ### Changed
19
+ - The `files` list now ships `docs/rrf-ranking.md`.
20
+
3
21
  ## 0.13.11
4
22
 
5
23
  ### Fixed
package/README.md CHANGED
@@ -5,6 +5,44 @@
5
5
  - [中文说明](README.zh-CN.md)
6
6
  - [Changelog](CHANGELOG.md) · [Why a plugin's settings go missing: the four-layer chain](docs/plugin-settings-form.md)
7
7
 
8
+ <p align="center"><img src="https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/banner.png" alt="dsh-web-search — key-free web search and page fetch for DeepSeek Harness" width="100%"></p>
9
+
10
+ <p align="center">
11
+ <a href="https://www.npmjs.com/package/@fffangchenghao/dsh-web-search"><img src="https://img.shields.io/npm/v/@fffangchenghao/dsh-web-search.svg" alt="npm version"></a>
12
+ <img src="https://img.shields.io/badge/tests-361%20passing-brightgreen" alt="tests">
13
+ <img src="https://img.shields.io/badge/engines-6%2C%20zero%20API%20key-brightgreen" alt="engines">
14
+ <img src="https://img.shields.io/badge/node-%3E%3D22.18-blue" alt="node">
15
+ <img src="https://img.shields.io/badge/dsh-%3E%3D0.2.1--rc.2%20%3C0.3.0-blueviolet" alt="dsh">
16
+ <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue" alt="license"></a>
17
+ </p>
18
+
19
+ ## Why this one
20
+
21
+ - **No API key, six engines.** AnySearch, CN Bing, DuckDuckGo HTML, www Bing, DuckDuckGo Lite and 360 Search are queried concurrently; the plugin works out of the box.
22
+ - **The ranking is measured, not asserted.** Cross-engine reciprocal-rank fusion with authority and freshness boosts, a 20-query offline benchmark, and an 81-configuration parameter sweep — the only search plugin here that publishes both its metric and its confidence intervals. See [How the ranking works](docs/rrf-ranking.md).
23
+ - **Fetching admits failure.** Binary documents and empty 200 responses are rejected by name (`SEARCH_UNSUPPORTED`, `SEARCH_EMPTY`) instead of being returned as noise; the direct → tunnel → render → extract ladder shares one 25 s budget and stops when there is no time left. The budget is per request — there is a regression test for exactly that.
24
+ - **361 tests across 26 files**, a committed `lib/`, declared `engines.dsh` / `engines.node`, and a [SECURITY.md](SECURITY.md) that lists every process, host and file the plugin touches.
25
+ - **Bilingual documentation.** This README and [中文说明](README.zh-CN.md) are maintained side by side.
26
+
27
+ ![Ranking baseline and parameter sweep](https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/bench.png)
28
+
29
+ The benchmark is offline and deterministic: 20 recorded queries, precision/recall/MRR@8, per-language and per-query output, so a regression cannot hide in an average. `node scripts/bench.mjs` reproduces the screenshot.
30
+
31
+ ## Where it stands
32
+
33
+ The web-search corner of the DSH plugin ecosystem is crowded, and most rivals are bigger than this one. An honest snapshot (2026-10-23; check the linked projects for current numbers):
34
+
35
+ | | this plugin | [dsh-free-search](https://github.com/DDDMUC/dsh-free-search) | [modsearch](https://github.com/liustack/modsearch) | [anysearch-dsh](https://github.com/anysearch-team/anysearch-dsh) |
36
+ |---|---|---|---|---|
37
+ | Key-free engines | 6 | ~30 | 7 channels (+X) | 1 official API |
38
+ | Dedicated vertical sources | 8 | 41 | — | — |
39
+ | Model-facing tools | 5 | ~10 | 3 | 5 |
40
+ | Test suite | 26 files / 361 cases | none | 29 files | 6 files |
41
+ | Ranking benchmark | 20 queries + 81-config sweep | — | — | — |
42
+ | Store status | source contract satisfied, awaiting review | unlisted | no entry | blocked (review-required) |
43
+
44
+ If you need maximum source breadth, use `dsh-free-search`; if you need a cross-harness search bridge, use `modsearch`; if you want the official AnySearch path only, use `anysearch-dsh`. This plugin is the one that treats ranking and failure semantics as the product, and measures them.
45
+
8
46
  > The latest version and install command live on [npm](https://www.npmjs.com/package/@fffangchenghao/dsh-web-search); no version number is hard-coded here, so the docs cannot drift from the release. What follows starts with what this means for you; the technical detail is collected in the appendix.
9
47
 
10
48
  ## In one sentence
package/README.zh-CN.md CHANGED
@@ -5,6 +5,44 @@
5
5
  - [English README](README.md)
6
6
  - [更新日志](CHANGELOG.md) · [设置页为什么不显示:四层判定链](docs/plugin-settings-form.md)
7
7
 
8
+ <p align="center"><img src="https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/banner.png" alt="dsh-web-search — 给 DeepSeek Harness 的免 key 联网搜索与网页抓取" width="100%"></p>
9
+
10
+ <p align="center">
11
+ <a href="https://www.npmjs.com/package/@fffangchenghao/dsh-web-search"><img src="https://img.shields.io/npm/v/@fffangchenghao/dsh-web-search.svg" alt="npm 版本"></a>
12
+ <img src="https://img.shields.io/badge/tests-361%20passing-brightgreen" alt="测试">
13
+ <img src="https://img.shields.io/badge/engines-6%2C%20zero%20API%20key-brightgreen" alt="引擎">
14
+ <img src="https://img.shields.io/badge/node-%3E%3D22.18-blue" alt="node">
15
+ <img src="https://img.shields.io/badge/dsh-%3E%3D0.2.1--rc.2%20%3C0.3.0-blueviolet" alt="dsh">
16
+ <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue" alt="许可证"></a>
17
+ </p>
18
+
19
+ ## 为什么选这个
20
+
21
+ - **不要 API key,六个引擎。** AnySearch、必应中国、DuckDuckGo HTML、必应国际、DuckDuckGo Lite、360 搜索并发查询,装好就能用。
22
+ - **排序是量出来的,不是写出来的。** 跨引擎 RRF 融合,叠加权威度与时效加权;20 条离线基准 + 81 组参数扫描,是这类插件里唯一同时公开指标和置信区间的。方法见 [排序是怎么做的](docs/rrf-ranking.md)。
23
+ - **抓取会承认失败。** 二进制文档和「200 但正文为空」会被明确拒绝(`SEARCH_UNSUPPORTED`、`SEARCH_EMPTY`),而不是当噪音返回;direct → tunnel → render → extract 四级阶梯共享 25 秒预算,预算不足就不再起新的一级。预算是「每次请求」而不是「插件启动时」,这点有回归测试守着。
24
+ - **26 个测试文件、361 个用例**,`lib/` 产物入库,声明了 `engines.dsh` / `engines.node`,[SECURITY.md](SECURITY.md) 逐项列出插件会碰的进程、主机和文件。
25
+ - **双语文档。** 本文件和 [English README](README.md) 同步维护。
26
+
27
+ ![排序基线参数扫描](https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/bench.png)
28
+
29
+ 基准是离线、可复现的:20 条录制的查询,precision/recall/MRR@8,按语言、按类别、逐条输出,回归问题藏不进平均值里。运行 `node scripts/bench.mjs` 就能复现这张图。
30
+
31
+ ## 它在同类里的位置
32
+
33
+ DSH 插件生态里的「联网搜索」这一块已经很挤,大多数竞品也比我们大。下面是一张诚实的快照(2026-10-23;最新数字请点开对应项目查看):
34
+
35
+ | | 本插件 | [dsh-free-search](https://github.com/DDDMUC/dsh-free-search) | [modsearch](https://github.com/liustack/modsearch) | [anysearch-dsh](https://github.com/anysearch-team/anysearch-dsh) |
36
+ |---|---|---|---|---|
37
+ | 免 key 引擎 | 6 | ~30 | 7 条通道(含 X) | 1 个官方 API |
38
+ | 专门数据源 | 8 | 41 | — | — |
39
+ | 面向模型的工具 | 5 | ~10 | 3 | 5 |
40
+ | 测试 | 26 文件 / 361 用例 | 无 | 29 文件 | 6 文件 |
41
+ | 排序基准 | 20 查询 + 81 组扫描 | — | — | — |
42
+ | 商店状态 | 源码契约已满足,待复核 | unlisted | 无条目 | blocked(review-required) |
43
+
44
+ 要最全的源,用 `dsh-free-search`;要跨 harness 的搜索桥,用 `modsearch`;只要官方 AnySearch 通道,用 `anysearch-dsh`。本插件把「排序」和「失败语义」当产品来做,并且把它们量出来。
45
+
8
46
  > 最新版本与安装命令以 [npm 上的 latest](https://www.npmjs.com/package/@fffangchenghao/dsh-web-search) 为准(这里不写死版本号,免得文档和发布不同步)。下面先讲「它跟我有什么关系」,技术细节放在后面的附录。
9
47
 
10
48
  ## 一句话说明
@@ -0,0 +1,129 @@
1
+ # How the ranking works, and how it is measured
2
+
3
+ Most "web search plugins" for an agent harness are thin wrappers: call one
4
+ engine, return its rows. That is enough to prove the model can reach the web —
5
+ and it is why result quality is usually discussed as a feeling.
6
+
7
+ This plugin treats the eight rows the Host shows the model as the product. That
8
+ means the ranking has to be explainable, and the explanation has to survive a
9
+ measurement. This page is both.
10
+
11
+ ![Search pipeline](images/pipeline.png)
12
+
13
+ ## The pipeline
14
+
15
+ 1. **Fan out.** Every enabled engine is queried concurrently with its own
16
+ deadline (`timeoutMs`, default 15 s) and its own cooldown state. An engine
17
+ that just failed is skipped instead of being retried on every query.
18
+ 2. **Normalise.** Each parser returns `{title, url, snippet, position, votes}`
19
+ with the engine's own result order preserved. URLs are canonicalised before
20
+ de-duplication so `?utm_source=…` cannot smuggle the same page in twice.
21
+ 3. **Interleave.** Engines are merged by weighted round-robin, not by
22
+ concatenation: an engine with weight 2 contributes two slots per round. This
23
+ keeps one verbose engine from filling all eight rows before a second engine
24
+ is even looked at. `anysearch` carries weight 2 because its precision was
25
+ measured highest on both Chinese and English queries.
26
+ 4. **Fuse.** Each engine's list is scored with reciprocal-rank fusion:
27
+
28
+ ```
29
+ vote = 1 / (rrfConstant + position) # rrfConstant default 60
30
+ score = Σ votes
31
+ + authorityBoost * isOfficialDomain
32
+ + freshnessBoost * freshness(parsedDate, freshnessDays)
33
+ ```
34
+
35
+ RRF is used instead of score normalisation because engines report scores on
36
+ incomparable scales (or none at all); ranks are the only signal every engine
37
+ provides honestly.
38
+ 5. **Region policy.** A CJK query leads with the engines that answer Chinese
39
+ sources (`cn.bing`, `anysearch`); an ASCII query leads with the global set
40
+ (`ddg-html`, `www.bing`, `anysearch`). The other region's engines still
41
+ contribute, so a Chinese query can still surface an English original.
42
+ 6. **Cut.** The fused list is truncated to `limit` (default 8, clamped 1–20).
43
+ Only the rows that survive are returned; the model never sees the raw
44
+ per-engine lists.
45
+
46
+ ## Why these exact knobs
47
+
48
+ Each knob exists because a specific failure was observed:
49
+
50
+ | knob | default | what it prevents |
51
+ |---|---|---|
52
+ | `rrfConstant` | 60 | a single engine's first result dominating the fusion |
53
+ | `authorityBoost` | 0.25 | documentation on `kubernetes.io` losing to a SEO farm |
54
+ | `freshnessBoost` / `freshnessDays` | 0.2 / 30 | last year's release notes outranking this month's |
55
+ | region split | on | English pages burying the Chinese documentation a Chinese query asked for |
56
+ | engine weight | `anysearch` = 2 | a precise engine receiving the same airtime as a noisy one |
57
+
58
+ ## How it is measured
59
+
60
+ `bench/` holds 20 fixed queries — 10 Chinese, 10 English, five each in docs /
61
+ release / package / news — each annotated with 1–3 expected domains. Encoding
62
+ the expectation in domains rather than exact URLs is deliberate: result titles
63
+ and snippets change constantly, domains are the part the ranking is supposed to
64
+ get right.
65
+
66
+ ```
67
+ node scripts/bench.mjs # offline: replay recorded fixtures, no network
68
+ node scripts/bench.mjs --record # refresh fixtures from the live engines
69
+ node scripts/bench.mjs --json # machine-readable output for version diffs
70
+ ```
71
+
72
+ The bench reports precision@8, recall@8 and MRR@8 overall, per language, per
73
+ category and per engine, plus the worst query. Offline replay makes the number
74
+ comparable across commits; `--record` is the only mode that touches the network.
75
+
76
+ ![Baseline and sweep](images/bench.png)
77
+
78
+ Baseline (v0.13.0, 20 queries, k=8):
79
+
80
+ | metric | overall | Chinese | English |
81
+ |---|---:|---:|---:|
82
+ | P@8 | 0.3625 | 0.3250 | 0.4000 |
83
+ | R@8 | 0.7417 | 0.7333 | 0.7500 |
84
+ | MRR@8 | 0.7600 | 0.6700 | 0.8500 |
85
+
86
+ One query (`zh-linux-612`) returns zero expected domains; that is visible in
87
+ the output on purpose rather than averaged away.
88
+
89
+ ## Parameter sweep
90
+
91
+ `scripts/sweep.mjs` runs an 81-configuration grid over the same fixtures and
92
+ reports marginal means plus a paired bootstrap against the shipped defaults:
93
+
94
+ ```
95
+ node scripts/sweep.mjs # full grid (~2 min, offline)
96
+ node scripts/sweep.mjs --quick # 27-config subset
97
+ node scripts/sweep.mjs --json # includes the per-config CI data
98
+ ```
99
+
100
+ Marginal MRR@8 (all other knobs at defaults):
101
+
102
+ | `rrfConstant` | MRR@8 | `authorityBoost` | MRR@8 | `freshnessBoost` | MRR@8 |
103
+ |---:|---:|---:|---:|---:|---:|
104
+ | 40 | 0.7712 | 0.1 | 0.7733 | 0.1 | 0.7681 |
105
+ | **60** | **0.7604** | **0.25** | **0.7645** | **0.2** | **0.7598** |
106
+ | 80 | 0.7456 | 0.4 | 0.7395 | 0.35 | 0.7494 |
107
+
108
+ The best grid point (`rrf=40, authority=0.1, freshness=0.1`) measures MRR 0.8042
109
+ against the default 0.7600 — a gain of +0.0442 — but its 95 % paired-bootstrap
110
+ interval is `[-0.0038, +0.1229]`, which straddles zero. The entire gain comes
111
+ from two of twenty queries. That is a promising direction, not a proven win, so
112
+ the defaults did not change. The sweep is kept as a script precisely so this
113
+ question can be re-asked after the query set grows.
114
+
115
+ The same discipline applied to engine weights: dropping `cn.bing` to weight 0.25
116
+ and raising `ddg-html` to 2 measured MRR 0.7708 with a lower bound of 0.0000 —
117
+ inside noise, so it was not adopted.
118
+
119
+ ## Known limits
120
+
121
+ - **Relative Chinese time words.** A query like "上周发布的那版" carries its time
122
+ constraint in the words themselves. The freshness boost can only act on dates
123
+ it can parse out of snippets; when an engine's snippet omits the date, the
124
+ ranker cannot recover it. This is documented rather than papered over.
125
+ - **Snippet quality bounds ranking.** If no engine returns the expected domain
126
+ in its top 20, fusion cannot invent it; `bench/` makes that visible per query.
127
+ - **The fixture set is 20 queries.** Small enough to run on every change, too
128
+ small to certify a general ranking. The per-query output is kept so the number
129
+ cannot hide a single catastrophic query.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@fffangchenghao/dsh-web-search",
3
- "version": "0.13.11",
3
+ "version": "0.14.0",
4
4
  "description": "DSH bundle: a key-free web search provider. Leads with the anonymous AnySearch API and falls back to public result pages (DuckDuckGo, Bing, 360), merged by reciprocal-rank fusion with a bilingual region policy, an engine cooldown and a TTL cache.",
5
5
  "type": "module",
6
6
  "main": "lib/index.js",
@@ -24,7 +24,8 @@
24
24
  "CHANGELOG.md",
25
25
  "LICENSE",
26
26
  "SECURITY.md",
27
- "docs/store-evidence.md"
27
+ "docs/store-evidence.md",
28
+ "docs/rrf-ranking.md"
28
29
  ],
29
30
  "scripts": {
30
31
  "build": "tsc -p tsconfig.json",