@fffangchenghao/dsh-web-search 0.24.0 → 0.24.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,33 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.24.1
4
+
5
+ Documentation and metadata patch; no search or fetch behavior changed.
6
+
7
+ ### Fixed
8
+ - **README and docs no longer imply macOS-only proxy discovery.** The English
9
+ and Chinese READMEs, architecture note, and troubleshooting guide now describe
10
+ macOS `scutil` and Windows Internet Settings as the two system-proxy sources,
11
+ with PowerShell checks for environment variables, registry values and port
12
+ reachability.
13
+ - **The test commands match the cross-platform script.** Docs use `npm test`
14
+ instead of an unquoted shell glob; CI documentation now names Ubuntu, macOS
15
+ and Windows.
16
+ - **The troubleshooting guide no longer says `lib/` is gitignored.** It is
17
+ committed, and the guide now says so.
18
+ - **The "no version is hard-coded" claim was removed.** The ecosystem table is
19
+ explicitly a dated snapshot, not a live leaderboard.
20
+ - **README links point at files that exist in the npm package.** The package now
21
+ ships `docs/architecture.md`, `docs/engines.md`, `docs/plugin-settings-form.md`
22
+ and `docs/troubleshooting.md`; the benchmark link goes to GitHub instead.
23
+ - **Package metadata and Settings copy were refreshed.** The npm description
24
+ names Windows/macOS proxy support; keywords include `windows`, `keyless`,
25
+ `arxiv` and `web-fetch`; the Settings title/description now names eight
26
+ engines and 34 sources instead of Bing/AnySearch.
27
+
28
+ **Test suite: 690 tests across 56 files** (689 pass, 1 skipped); this patch does
29
+ not change runtime code.
30
+
3
31
  ## 0.24.0
4
32
 
5
33
  ### Added
package/README.md CHANGED
@@ -37,7 +37,7 @@ The benchmark is offline and deterministic: 20 recorded queries, precision/recal
37
37
 
38
38
  ## Where it stands
39
39
 
40
- The web-search corner of the DSH plugin ecosystem is crowded. A fresh snapshot
40
+ The web-search corner of the DSH plugin ecosystem is crowded. A dated snapshot
41
41
  (2026-10-25; stars and npm downloads are adoption signals, not quality scores):
42
42
 
43
43
  | | this plugin | [dsh-free-search](https://github.com/DDDMUC/dsh-free-search) | [dsh-web-search-pro](https://github.com/anweat/dsh-web-search-pro) | [modsearch](https://github.com/liustack/modsearch) | [anysearch-dsh](https://github.com/anysearch-team/anysearch-dsh) |
@@ -63,9 +63,9 @@ read is:
63
63
  - **Key-free + official-source correctness:** its strongest niche. The official
64
64
  Chrome, PostgreSQL and AI Act paths and the 34-source vertical layer are not
65
65
  common in the other plugins.
66
- - **Adoption and maturity:** bottom third of this sample. The public npm release
67
- is 0.24.0, matching this tree, but the GitHub repository is four days old
68
- with no stars and little independent adoption.
66
+ - **Adoption and maturity:** bottom third of this sample. At the 2026-10-25
67
+ snapshot the repository was four days old, had no stars and little
68
+ independent adoption; the current npm release is 0.24.1.
69
69
  - **Tool-surface economy:** now competitive. 0.20.x exposes one `web_tools`
70
70
  entry with eight actions, close to `dsh-web-search-pro`'s two tools with 20
71
71
  actions; the legacy eight-name surface remains opt-in for old prompts.
@@ -77,7 +77,7 @@ is the compact, fully key-free option for search plus vertical lookup, research
77
77
  packs and official-source correction. It is not the most popular, and it should
78
78
  not claim to be the largest.
79
79
 
80
- > The latest version and install command live on [npm](https://www.npmjs.com/package/@fffangchenghao/dsh-web-search); no version number is hard-coded here, so the docs cannot drift from the release. What follows starts with what this means for you; the technical detail is collected in the appendix.
80
+ > The latest version and install command live on [npm](https://www.npmjs.com/package/@fffangchenghao/dsh-web-search); the comparison table is a dated snapshot, not a live leaderboard. What follows starts with what this means for you; the technical detail is collected in the appendix.
81
81
 
82
82
  ## In one sentence
83
83
 
@@ -308,7 +308,7 @@ Each line is the plain version first, the technical note second. The technical n
308
308
 
309
309
  13. **It works on a restricted network.**
310
310
  With a tunnel running and set as the system proxy, nothing needs configuring. With the tunnel off, it falls back to direct requests instead of failing.
311
- *Technical: proxy sources in order — the `proxy` setting, then `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY`, then macOS `scutil --proxy`; HTTP proxies only (`CONNECT`), not `socks5://`.*
311
+ *Technical: proxy sources in order — the `proxy` setting, then `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY`, then the OS system proxy (macOS `scutil`, Windows Internet Settings); HTTP proxies only (`CONNECT`), not `socks5://`.*
312
312
 
313
313
  14. **Every setting is editable in the UI.**
314
314
  Thirty controls in six groups; a save takes effect on the next search, apart from `cachePersist` and `cacheFile` (cache wiring at mount) and `toolSurface` (the tool registry is built at mount).
@@ -590,7 +590,7 @@ Optionally, add a config row:
590
590
 
591
591
  ### Step 4: restart completely
592
592
 
593
- After changing code or profile config, **quit DSH entirely and open it again** (on macOS, quit from the menu or press `Cmd+Q`; closing the window is not enough). Modules are evaluated once per process, and a running process never sees a file you edited underneath it.
593
+ After changing code or profile config, **quit DSH entirely and open it again** (on macOS press `Cmd+Q`; on Windows quit from the tray or menu; closing the window is not enough). Modules are evaluated once per process, and a running process never sees a file you edited underneath it.
594
594
 
595
595
  Values saved from the Settings page apply immediately; only `cachePersist` and `cacheFile` need the plugin to reload.
596
596
 
@@ -1067,9 +1067,9 @@ Conclusions:
1067
1067
 
1068
1068
  ### Proxy
1069
1069
 
1070
- Node's global `fetch` reads neither the macOS system proxy nor `HTTPS_PROXY` — unless the process started with `NODE_USE_ENV_PROXY=1`, which cannot be switched on from inside a running process. So when a proxy is needed, the plugin speaks to it directly: send `CONNECT`, upgrade with TLS for `https:` targets, write the request by hand, and decode `Content-Length`, chunked, gzip, deflate and brotli, following up to four redirects. `307`/`308` keep the method and body; any other `3xx` downgrades a `POST` to a bodyless `GET`.
1070
+ Node's global `fetch` reads neither the OS system proxy (macOS `scutil` / Windows Internet Settings) nor `HTTPS_PROXY` — unless the process started with `NODE_USE_ENV_PROXY=1`, which cannot be switched on from inside a running process. So when a proxy is needed, the plugin speaks to it directly: send `CONNECT`, upgrade with TLS for `https:` targets, write the request by hand, and decode `Content-Length`, chunked, gzip, deflate and brotli, following up to four redirects. `307`/`308` keep the method and body; any other `3xx` downgrades a `POST` to a bodyless `GET`.
1071
1071
 
1072
- Resolution order: the `proxy` setting → the environment (`DSH_WEB_SEARCH_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, `HTTP_PROXY`, either casing) → macOS `scutil --proxy`, adopted only if a 400 ms TCP probe connects, and cached for 60 seconds. HTTP proxies only; replace `socks5://` with the client's HTTP/mixed port. Proxying is scoped to this plugin — the rest of the app keeps its ordinary network path.
1072
+ Resolution order: the `proxy` setting → the environment (`DSH_WEB_SEARCH_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, `HTTP_PROXY`, either casing) → the OS system proxy (macOS `scutil --proxy` / Windows Internet Settings), adopted only if a 400 ms TCP probe connects, and cached for 60 seconds. HTTP proxies only; replace `socks5://` with the client's HTTP/mixed port. Proxying is scoped to this plugin — the rest of the app keeps its ordinary network path.
1073
1073
 
1074
1074
  ### Known limitations
1075
1075
 
@@ -1133,16 +1133,24 @@ Resolution order: the `proxy` setting → the environment (`DSH_WEB_SEARCH_PROXY
1133
1133
  ### Development and tests
1134
1134
 
1135
1135
  ```sh
1136
- node --test test/*.test.ts # the whole offline suite (one live case is skipped by default)
1136
+ npm test # the whole offline suite (one live case is skipped by default)
1137
1137
  node --test test/proxy.test.ts # one file
1138
1138
  npm run typecheck # tsc --noEmit
1139
1139
  npm run build # compile src/ to lib/
1140
1140
  npm run check # typecheck + tests
1141
1141
  ```
1142
1142
 
1143
- The suite runs offline by default and passes in full; the one live paging check is skipped unless `READ_PAGE_LIVE=1`: parser tests replay captured fixtures from `test/fixtures/` (normal and empty responses for the default eight and the 26 opt-in sources), transport tests open no sockets, proxy tests cover `parseProxy`, `proxyFromEnv` and the pure response framer, time-window tests pin the date with an injected `now`, and the newer suites cover Markdown extraction (`markdown.test.ts`), batch search (`search-many.test.ts`), chunked reading (`read-page.test.ts`), the disk snapshot (`cache-persist.test.ts`), optional JS rendering (`render-js.test.ts`) and benchmark scoring/replay (`bench-metrics.test.ts`, `bench-replay.test.ts`). One live-network test in `read-page.test.ts` is skipped by default; run it with `READ_PAGE_LIVE=1 node --test test/read-page.test.ts`.
1143
+ The suite runs offline by default and passes in full; the one live paging check is skipped unless `READ_PAGE_LIVE=1`: parser tests replay captured fixtures from `test/fixtures/` (normal and empty responses for the default eight and the 26 opt-in sources), transport tests open no sockets, proxy tests cover `parseProxy`, `proxyFromEnv` and the pure response framer, time-window tests pin the date with an injected `now`, and the newer suites cover Markdown extraction (`markdown.test.ts`), batch search (`search-many.test.ts`), chunked reading (`read-page.test.ts`), the disk snapshot (`cache-persist.test.ts`), optional JS rendering (`render-js.test.ts`) and benchmark scoring/replay (`bench-metrics.test.ts`, `bench-replay.test.ts`). One live-network test in `read-page.test.ts` is skipped by default; run it with:
1144
1144
 
1145
- CI (`.github/workflows/ci.yml`) runs `npm ci`, `npm run typecheck`, `npm test` and `npm run build` on Node 24 for Ubuntu and macOS, then checks that the build emitted `lib/index.js` and `lib/index.d.ts` and that both `apply` and `Config` import.
1145
+ ```sh
1146
+ # macOS / Linux
1147
+ READ_PAGE_LIVE=1 node --test test/read-page.test.ts
1148
+
1149
+ # Windows PowerShell
1150
+ $env:READ_PAGE_LIVE=1; node --test test/read-page.test.ts
1151
+ ```
1152
+
1153
+ CI (`.github/workflows/ci.yml`) runs `npm ci`, `npm run typecheck`, `npm test` and `npm run build` on Node 24 for Ubuntu, macOS and Windows, then checks that the build emitted `lib/index.js` and `lib/index.d.ts` and that both `apply` and `Config` import.
1146
1154
 
1147
1155
  After editing `src/*.ts`, build and restart — otherwise you are still running the old code in `lib/`.
1148
1156
 
@@ -1153,7 +1161,7 @@ After editing `src/*.ts`, build and restart — otherwise you are still running
1153
1161
  - [docs/architecture.md](docs/architecture.md): the module map and the data flow from the `web` service to a source list.
1154
1162
  - [docs/engines.md](docs/engines.md): each engine's request shape, parser quirks and measured results.
1155
1163
  - [docs/troubleshooting.md](docs/troubleshooting.md): no results, proxy not detected, provider not selected, cooldown stuck.
1156
- - [bench/README.md](bench/README.md): how the 20-query retrieval benchmark records fixtures, scores them and compares versions.
1164
+ - [bench/README.md](https://github.com/fangchenghao/dsh-web-search/blob/main/bench/README.md): how the 20-query retrieval benchmark records fixtures, scores them and compares versions.
1157
1165
  - [README.zh-CN.md](README.zh-CN.md): the Chinese guide, section for section.
1158
1166
 
1159
1167
  MIT © 2026 fangchenghao. See [LICENSE](LICENSE).
package/README.zh-CN.md CHANGED
@@ -36,7 +36,7 @@
36
36
 
37
37
  ## 它在同类里的位置
38
38
 
39
- DSH 插件生态里的「联网搜索」已经很挤。下面是一张 2026-10-25 的快照
39
+ DSH 插件生态里的「联网搜索」已经很挤。下面是一张 2026-10-25 的历史快照
40
40
  (stars 和 npm 下载量是采用度信号,不是质量分数):
41
41
 
42
42
  | | 本插件 | [dsh-free-search](https://github.com/DDDMUC/dsh-free-search) | [dsh-web-search-pro](https://github.com/anweat/dsh-web-search-pro) | [modsearch](https://github.com/liustack/modsearch) | [anysearch-dsh](https://github.com/anysearch-team/anysearch-dsh) |
@@ -58,8 +58,7 @@ DSH 插件生态里的「联网搜索」已经很挤。下面是一张 2026-10-2
58
58
  明确的安全边界、只有 1 个运行时依赖,同时体积远小于 Playwright 重型方案。
59
59
  - **免 key + 官方源纠错:最强卖点。** Chrome、PostgreSQL、AI Act 的官方路径
60
60
  和 34 个垂直源,不是其他插件普遍具备的。
61
- - **采用度与成熟度:这个样本里的后段。** 公开 npm 已是 0.24.0,与这棵树一致;但 GitHub 仓库创建 4 天、0 stars,
62
- 还缺少独立采用。
61
+ - **采用度与成熟度:这个样本里的后段。** 2026-10-25 快照时 GitHub 仓库刚创建 4 天、0 stars,还缺少独立采用;当前 npm 版本是 0.24.1。
63
62
  - **工具面的经济性:现在有竞争力。** 0.20.x 只暴露一个 `web_tools` + 8 个
64
63
  action,接近 `dsh-web-search-pro` 的 2 工具 + 20 动作;旧的八个工具名保留为
65
64
  可选兼容模式。
@@ -69,7 +68,7 @@ DSH 插件生态里的「联网搜索」已经很挤。下面是一张 2026-10-2
69
68
  搜索 + 垂直查询 + 研究包 + 官方源纠错」的轻量选项。它不是最流行的,也不该
70
69
  自称最大。
71
70
 
72
- > 最新版本与安装命令以 [npm 上的 latest](https://www.npmjs.com/package/@fffangchenghao/dsh-web-search) 为准(这里不写死版本号,免得文档和发布不同步)。下面先讲「它跟我有什么关系」,技术细节放在后面的附录。
71
+ > 最新版本与安装命令以 [npm 上的 latest](https://www.npmjs.com/package/@fffangchenghao/dsh-web-search) 为准;上面的对比表是历史快照,不是实时排行榜。下面先讲「它跟我有什么关系」,技术细节放在后面的附录。
73
72
 
74
73
  ## 一句话说明
75
74
 
@@ -300,7 +299,7 @@ Harness 里有个叫 `web` 的**服务**,它自己不会搜索,只负责「
300
299
 
301
300
  13. **受限网络下也能用。**
302
301
  梯子开着并且设为系统代理时,不用配置任何东西;关掉梯子会自动退回直连,不会因此报错。
303
- *技术补充:代理来源依次是配置里的 `proxy`、环境变量 `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY`、macOS 的 `scutil --proxy`;只支持 HTTP 代理(`CONNECT`),不支持 `socks5://`。*
302
+ *技术补充:代理来源依次是配置里的 `proxy`、环境变量 `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY`、操作系统系统代理(macOS 的 `scutil`、Windows 的 Internet 设置);只支持 HTTP 代理(`CONNECT`),不支持 `socks5://`。*
304
303
 
305
304
  14. **所有设置都能在界面上改,不用编辑文件。**
306
305
  设置页有 30 个控件,分六组;保存后下一次搜索就生效;只有 `cachePersist` / `cacheFile`(决定缓存接线)和 `toolSurface`(工具注册表在挂载时构建)要重载插件。
@@ -558,7 +557,7 @@ pnpm install
558
557
 
559
558
  ### 第四步:完全重启
560
559
 
561
- 改完代码或 profile 配置后,**完全退出 DSH 再打开**(macOS 上不是只关窗口,要从菜单退出,或按 `Cmd+Q`)。原因是模块在每个进程里只求值一次,热改文件不会被已运行的进程看见。
560
+ 改完代码或 profile 配置后,**完全退出 DSH 再打开**(macOS 按 `Cmd+Q`;Windows 从托盘或菜单退出,不能只关窗口)。原因是模块在每个进程里只求值一次,热改文件不会被已运行的进程看见。
562
561
 
563
562
  设置页里大部分值不需要重启,例外情况见下一节。
564
563
 
@@ -1006,9 +1005,9 @@ read_page 调用
1006
1005
 
1007
1006
  ### 代理
1008
1007
 
1009
- Node 的全局 `fetch` 既不读 macOS 系统代理,也不读 `HTTPS_PROXY`——除非进程启动时带了 `NODE_USE_ENV_PROXY=1`,而这个开关在进程跑起来之后打不开。所以需要代理时,插件自己连:发 `CONNECT`、对 `https:` 目标做 TLS 升级、手写请求、自己解析 `Content-Length` / chunked / gzip / deflate / brotli,最多跟 4 次跳转。`307`/`308` 保留方法和请求体,其他 `3xx` 把 `POST` 降级成无 body 的 `GET`。
1008
+ Node 的全局 `fetch` 既不读操作系统系统代理(macOS 的 `scutil`、Windows 的 Internet 设置),也不读 `HTTPS_PROXY`——除非进程启动时带了 `NODE_USE_ENV_PROXY=1`,而这个开关在进程跑起来之后打不开。所以需要代理时,插件自己连:发 `CONNECT`、对 `https:` 目标做 TLS 升级、手写请求、自己解析 `Content-Length` / chunked / gzip / deflate / brotli,最多跟 4 次跳转。`307`/`308` 保留方法和请求体,其他 `3xx` 把 `POST` 降级成无 body 的 `GET`。
1010
1009
 
1011
- 代理来源顺序:配置里的 `proxy` → 环境变量 `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY`(大小写都认)→ macOS `scutil --proxy`(带 400 ms TCP 探测,结果缓存 60 秒)。只支持 HTTP 代理;`socks5://` 请换成客户端的 HTTP/mixed 端口。代理只作用于本插件,应用其他流量走原来的路。
1010
+ 代理来源顺序:配置里的 `proxy` → 环境变量 `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY`(大小写都认)→ 操作系统系统代理(macOS `scutil --proxy` / Windows Internet 设置,带 400 ms TCP 探测,结果缓存 60 秒)。只支持 HTTP 代理;`socks5://` 请换成客户端的 HTTP/mixed 端口。代理只作用于本插件,应用其他流量走原来的路。
1012
1011
 
1013
1012
  ### 已知限制
1014
1013
 
@@ -1072,16 +1071,24 @@ Node 的全局 `fetch` 既不读 macOS 系统代理,也不读 `HTTPS_PROXY`—
1072
1071
  ### 开发与测试
1073
1072
 
1074
1073
  ```sh
1075
- node --test test/*.test.ts # 全部离线测试(联网用例默认跳过)
1074
+ npm test # 全部离线测试(联网用例默认跳过)
1076
1075
  node --test test/proxy.test.ts # 单个文件
1077
1076
  npm run typecheck # tsc --noEmit
1078
1077
  npm run build # 编译 src/ 到 lib/
1079
1078
  npm run check # typecheck + 测试
1080
1079
  ```
1081
1080
 
1082
- 测试默认全程离线、全部通过;唯一需要联网的分段验证默认跳过(`READ_PAGE_LIVE=1` 时才跑):解析器测试回放 `test/fixtures/` 里抓下来的真实响应(含默认八个与 26 个按需源的真实响应),传输测试不开 socket,代理测试只验 `parseProxy`、`proxyFromEnv` 和纯函数式的响应分帧,时间窗测试用注入的 `now` 固定日期;新增的几组覆盖 Markdown 正文提取(`markdown.test.ts`)、批量查询(`search-many.test.ts`)、分段续读(`read-page.test.ts`)、硬盘快照(`cache-persist.test.ts`)、可选 JS 渲染(`render-js.test.ts`)和基准计分/重放(`bench-metrics.test.ts`、`bench-replay.test.ts`)。其中 `read-page.test.ts` 有一条真实联网用例默认跳过,联网验证时用 `READ_PAGE_LIVE=1 node --test test/read-page.test.ts` 运行。
1081
+ 测试默认全程离线、全部通过;唯一需要联网的分段验证默认跳过(`READ_PAGE_LIVE=1` 时才跑):解析器测试回放 `test/fixtures/` 里抓下来的真实响应(含默认八个与 26 个按需源的真实响应),传输测试不开 socket,代理测试只验 `parseProxy`、`proxyFromEnv` 和纯函数式的响应分帧,时间窗测试用注入的 `now` 固定日期;新增的几组覆盖 Markdown 正文提取(`markdown.test.ts`)、批量查询(`search-many.test.ts`)、分段续读(`read-page.test.ts`)、硬盘快照(`cache-persist.test.ts`)、可选 JS 渲染(`render-js.test.ts`)和基准计分/重放(`bench-metrics.test.ts`、`bench-replay.test.ts`)。其中 `read-page.test.ts` 有一条真实联网用例默认跳过;联网验证时:
1083
1082
 
1084
- CI(`.github/workflows/ci.yml`)在 Node 24 的 Ubuntu 与 macOS 上跑 `npm ci`、`npm run typecheck`、`npm test`、`npm run build`,并检查构建确实产出了 `lib/index.js` 和 `lib/index.d.ts`,且 `apply` 与 `Config` 都可导入。
1083
+ ```sh
1084
+ # macOS / Linux
1085
+ READ_PAGE_LIVE=1 node --test test/read-page.test.ts
1086
+
1087
+ # Windows PowerShell
1088
+ $env:READ_PAGE_LIVE=1; node --test test/read-page.test.ts
1089
+ ```
1090
+
1091
+ CI(`.github/workflows/ci.yml`)在 Node 24 的 Ubuntu、macOS 与 Windows 上跑 `npm ci`、`npm run typecheck`、`npm test`、`npm run build`,并检查构建确实产出了 `lib/index.js` 和 `lib/index.d.ts`,且 `apply` 与 `Config` 都可导入。
1085
1092
 
1086
1093
  改完 `src/*.ts` 记得构建并重启,否则跑的还是 `lib/` 里的旧代码。
1087
1094
 
@@ -1092,7 +1099,7 @@ CI(`.github/workflows/ci.yml`)在 Node 24 的 Ubuntu 与 macOS 上跑 `npm c
1092
1099
  - [docs/architecture.md](docs/architecture.md):模块划分和从 `web` 服务到来源列表的数据流。
1093
1100
  - [docs/engines.md](docs/engines.md):每个引擎的请求形状、解析怪癖与实测结论。
1094
1101
  - [docs/troubleshooting.md](docs/troubleshooting.md):没有结果、代理没探测到、提供方没被选中、冷却卡住。
1095
- - [bench/README.md](bench/README.md):20 条查询的检索基准怎么录制夹具、怎么计分、怎么比较版本。
1102
+ - [bench/README.md](https://github.com/fangchenghao/dsh-web-search/blob/main/bench/README.md):20 条查询的检索基准怎么录制夹具、怎么计分、怎么比较版本。
1096
1103
  - [README.md](README.md):英文版,与中文版逐节对应。
1097
1104
 
1098
1105
  MIT © 2026 fangchenghao,见 [LICENSE](LICENSE)。
package/client.js CHANGED
@@ -74,14 +74,14 @@
74
74
  var ENTRY = 'web-search-bing';
75
75
  var PACKAGE_NAME = 'dsh-web-search-bing';
76
76
  /** Keep in step with package.json; compared with the npm registry. */
77
- var VERSION = '0.24.0';
77
+ var VERSION = '0.24.1';
78
78
  /** Snapshot used while no configuration service is reachable. */
79
79
  var EMPTY_STATE = { status: 'unavailable', value: undefined, base: undefined, user: undefined, revision: undefined, writable: false };
80
80
 
81
81
  // English copy. Every label the page renders lives here.
82
82
  var en = {
83
- title: 'Web search (Bing)',
84
- description: 'Edit the key-free search provider: engine allow-list, wait policy, proxy, result budget, ranking, cache, cooldowns and dedicated sources. Model tools: web_search, web_fetch and one unified web_tools action tool (legacy eight-name mode is opt-in).',
83
+ title: 'Web Search',
84
+ description: 'Key-free search, page fetch and research: eight engines fused with RRF, 34 dedicated sources, fetch ladder, proxy, cache and cooldown settings.',
85
85
  jumpTo: 'Jump to',
86
86
  latest: 'Latest on npm',
87
87
  upToDate: 'up to date',
@@ -191,7 +191,7 @@
191
191
 
192
192
  // Simplified Chinese copy.
193
193
  var zh = {
194
- title: '网页搜索(Bing)',
194
+ title: '网页搜索',
195
195
  jumpTo: '跳到',
196
196
  latest: 'npm 最新版',
197
197
  autoUpdateHint: '定时自动更新需要 profile 更新器(dsh-autoupdater);「安装更新」按钮不依赖它:它会请 Host 插件管理器安装 npm 包,装完需重启。link:/file: 安装会被拒绝。',
@@ -205,7 +205,7 @@
205
205
  updateUnavailable: '当前部署没有内置插件管理器;请用 profile 的包管理器更新。',
206
206
  upToDate: '已是最新',
207
207
  updateAvailable: '有新版本',
208
- description: '编辑免密钥搜索提供方:引擎白名单、等待策略、代理、结果上限、排序、缓存、冷却与专用检索源。模型工具:web_search、web_fetch,以及一个统一的 web_tools action 工具(旧版八个工具名可选)。',
208
+ description: '免密钥的搜索、抓取与研究:8 个引擎 RRF 融合、34 个专用源、抓取梯队、代理、缓存与冷却设置。',
209
209
  'group.engines': '引擎与传输',
210
210
  'group.ranking': '结果与排序',
211
211
  'group.fetch': '抓取',
@@ -0,0 +1,181 @@
1
+ # Architecture
2
+
3
+ How a `web_search` call becomes a list of sources, and which module owns which
4
+ decision. Everything described here exists in `src/*.ts`; the compiled output in
5
+ `lib/` is what `package.json` points at.
6
+
7
+ - [Engines and their quirks](engines.md)
8
+ - [Troubleshooting](troubleshooting.md)
9
+ - [README](../README.md) · [中文说明](../README.zh-CN.md)
10
+
11
+ ## Module map
12
+
13
+ | Module | Owns | Exports worth knowing |
14
+ | --- | --- | --- |
15
+ | `src/types.ts` | Every contract: `Source`, `HostSpec`, `EngineAnswer`, `SearchOptions`, `SearchOutcome`, `RankOptions`, `CacheOptions`, `HealthOptions`, `SearchErrorCode` | types only, no runtime code |
16
+ | `src/errors.ts` | The one error class and the `Retry-After` parser | `SearchError`, `parseRetryAfter` |
17
+ | `src/config.ts` | The schemastery schema, the defaults and their normalisation | `Config`, `resolveConfig`, `PluginConfig`, `ResolvedConfig` |
18
+ | `src/index.ts` | The plugin entry: `apply` registers the provider with the `web` seam and owns one cache and one health tracker per mount | `apply`, `name`, `inject`, `PROVIDER_ID` |
19
+ | `src/engines.ts` | The engine registry, the region policy, tier scheduling and the cache/cooldown integration | `HOSTS`, `search` |
20
+ | `src/transport.ts` | One engine request, direct or tunnelled, classified into a `SearchError` | `searchHost` |
21
+ | `src/proxy.ts` | Proxy detection and the hand-written `CONNECT`/TLS transport | `parseProxy`, `proxyFromEnv`, `proxyFromSystem`, `resolveProxy`, `proxyGet`, `readResponse` |
22
+ | `src/parsers.ts` | The five payload parsers plus the URL, entity and echoed-query helpers | `resolveHref`, `parseDdgHtml`, `parseDdgLite`, `parseBing`, `parseSo`, `parseAnySearch`, `echoesQuery`, `text` |
23
+ | `src/rank.ts` | Dedupe, weighted interleave, weight-as-admission RRF (votes over merged position), region merge | `dedupe`, `interleave`, `fuse`, `mergeRegions`, `RRF_CONSTANT`, `DEFAULT_AUTHORITY_DOMAINS` |
24
+ | `src/cache.ts` | TTL + LRU result cache with an injectable clock | `cacheKey`, `SearchCache` |
25
+ | `src/health.ts` | Per-engine cooldown, exponential backoff, `Retry-After` override | `EngineHealth` |
26
+
27
+ Dependency direction is one-way and acyclic: `index → engines → {transport,
28
+ rank, cache, health, parsers, proxy}`, every module → `types`, and `proxy` /
29
+ `transport` / `engines` → `errors`. `parsers`, `rank`, `cache` and `health` never
30
+ touch the network; `proxy` never parses a result page.
31
+
32
+ ## Data flow
33
+
34
+ 1. **Seam call.** The harness calls the provider registered by `apply`
35
+ (`src/index.ts`) with `(request, signal)`. `request.query` must be a non-empty
36
+ string; `request.maxResults`, when a positive integer, caps the budget.
37
+ 2. **Config.** The loader has already validated the config section against
38
+ `Config`; `resolveConfig` turns it into the engine layer's shape. `limit`
39
+ becomes `min(maxResults ?? config.limit, config.limit)`.
40
+ 3. **Search.** `search(query, signal, options)` in `src/engines.ts`:
41
+ - resolve the proxy (`useProxy: false` forces direct; otherwise
42
+ `resolveProxy(config.proxy)`);
43
+ - filter `HOSTS` by `hostIds`, and drop `viaProxy` engines when no proxy is
44
+ available;
45
+ - split engines into usable / cooling, but never end up with zero engines —
46
+ if everything is cooling, everything is used;
47
+ - compute the leading region from the query script (`global` for ASCII, `cn`
48
+ for CJK);
49
+ - look up the cache with a key built from the normalised query, the budget,
50
+ the wait policy, the lead region and the sorted engine ids.
51
+ 4. **Tier run.** Engines of the leading region first, then the rest, split into
52
+ `primary` and `fallback`. `runTier` asks every engine of a tier concurrently
53
+ through `searchHost` and releases on the first of:
54
+ - every engine settled;
55
+ - every engine of the leading region settled (success *or* failure), so a
56
+ failed engine cannot hold the tier open until the deadline;
57
+ - `available() >= limit` sources from at least `needed` contributors across
58
+ **all** regions (two for `auto`), so one slow lead engine cannot keep a
59
+ complete cross-engine answer waiting;
60
+ - the soft deadline (`auto`: 5 s ASCII / 3 s CJK, `fast`: 3.5 s, `all`: 10 s).
61
+
62
+ Engines still running at release are aborted and reported as
63
+ `aborted once enough results had arrived` or `still running at deadline`.
64
+ 5. **Per engine.** `searchHost` (`src/transport.ts`) builds the URL from
65
+ `host.build(query)` and picks a route: a domestic engine goes direct first
66
+ and is retried through the tunnel only if that fails; a `viaProxy` engine
67
+ tunnels immediately. Direct requests use global `fetch` with
68
+ `AbortSignal.timeout` combined with the caller's signal; tunnelled requests
69
+ use `proxyGet`.
70
+ 6. **Classify.** A non-2xx status becomes `SEARCH_THROTTLED` (429),
71
+ `SEARCH_BLOCKED` (403 and everything else in the 4xx range),
72
+ `SEARCH_NETWORK` (5xx and transport failures) or `SEARCH_TIMEOUT` (a deadline
73
+ that expired). A JSON API validates itself; every scraped page must pass
74
+ `echoesQuery` or it is `SEARCH_BLOCKED`. A parser that throws is
75
+ `SEARCH_PARSE` and is **not** retryable. Cancellation is rethrown, never
76
+ returned.
77
+ 7. **Parse.** `host.parse(payload, url)` returns `Source[]` with absolute
78
+ `http(s)` URLs and no empty fields.
79
+ 8. **Health.** Each settled engine updates the tracker: success clears its
80
+ streak, failure may start or extend a cooldown.
81
+ 9. **Fuse.** `mergeRegions(answers, lead, limit, rank)` fuses the leading group
82
+ with RRF and fills the budget; the other region is fused separately and only
83
+ appended (deduped against the head) when the head came up short. If the
84
+ primary tier produced nothing, the fallback tier runs and the same merge is
85
+ applied. Weight is an *admission* share, not a score multiplier: each engine's
86
+ list is walked in a weighted round robin to give every candidate a 1-based
87
+ position, and the score is `votes / (rrfConstant + position)`, with `votes`
88
+ counting the distinct engines that returned the source.
89
+ 10. **Return.** A non-empty result is cached, then returned as
90
+ `{ sources, failures, proxy }`. `index.ts` re-slices to the budget and sets
91
+ `truncated`, or throws an `Error` that lists every engine's failure reason.
92
+
93
+ ## Frames and invariants
94
+
95
+ - **Failures are values.** One dead engine degrades into the next one; only
96
+ cancellation escapes as a rejection.
97
+ - **Parsers never throw on markup drift.** They return `[]`, and the transport
98
+ records a parse failure.
99
+ - **Empty fields are omitted, not emptied.** A source never carries `title: ''`.
100
+ - **Determinism.** Fusion ties resolve on engine index, then rank, then URL, so
101
+ the same answers always produce the same order.
102
+ - **No credentials.** Nothing in the flow reads or sends an API key.
103
+
104
+ ## Failure codes
105
+
106
+ | Code | Produced by | Cools the engine down |
107
+ | --- | --- | --- |
108
+ | `SEARCH_THROTTLED` | HTTP 429 | yes |
109
+ | `SEARCH_BLOCKED` | HTTP 403, other 4xx, echoed-query mismatch | yes |
110
+ | `SEARCH_NETWORK` | HTTP 5xx, transport error | yes |
111
+ | `SEARCH_TIMEOUT` | a request that hit its deadline | yes |
112
+ | `SEARCH_PARSE` | a parser that threw | no — retrying markup cannot help |
113
+ | `SEARCH_ABORTED` | the caller cancelled | no |
114
+
115
+ The cooldown length comes from `Retry-After` when the service sent one (delta
116
+ seconds or an HTTP date, read from the final redirect hop), otherwise
117
+ `baseCooldownMs × 2^(failures−1)`, capped at `maxCooldownMs`.
118
+
119
+ ## Cache key
120
+
121
+ `cacheKey` builds `lead|wait|limit|sorted engine ids|normalised query`. The query
122
+ is trimmed, lowercased and has internal whitespace collapsed; engine order never
123
+ matters because the ids are sorted. The engine ids are the ones the run would use
124
+ *before* the cooldown filter, so an engine entering or leaving a cooldown does not
125
+ invalidate an entry: otherwise every throttle would turn the next identical query
126
+ into a cache miss. Only successful, non-empty results are stored. A hit returns a copy and skips every request. `ttlMs: 0` disables the
127
+ cache entirely; past `maxEntries`, the least recently used entry is evicted.
128
+
129
+ ## Proxy transport
130
+
131
+ `proxyGet` opens an HTTP `CONNECT` tunnel to the target host and port, upgrades
132
+ the socket with TLS when the target is `https:`, writes the request itself
133
+ (`accept-encoding: identity`, `connection: close`) and decodes the response:
134
+ `Transfer-Encoding: chunked`, `Content-Length`, `gzip`, `deflate` and `br`.
135
+ Redirects are followed hop by hop, re-opening the tunnel each time, up to four
136
+ hops. `307`/`308` preserve the method and body; any other `3xx` downgrades a
137
+ `POST` to a bodyless `GET`. The final hop's status, headers (lowercased), body
138
+ and URL are returned, which is how the caller can read `Retry-After`.
139
+
140
+ Proxy selection is explicit config → environment (`DSH_WEB_SEARCH_PROXY`,
141
+ `HTTPS_PROXY`, `ALL_PROXY`, `HTTP_PROXY`, both casings) → the OS system proxy
142
+ (macOS `scutil --proxy`, Windows Internet Settings) gated by a 400 ms TCP probe,
143
+ cached for 60 seconds. Only `http://` proxies are
144
+ dialled from detection, because the tunnel itself is a plain `CONNECT`; an
145
+ `https://` value elsewhere in the environment is skipped rather than selected.
146
+
147
+ ## Build and runtime layout
148
+
149
+ - `src/**/*.ts` is the source of truth; `npm run build` compiles it to `lib/`
150
+ (`tsconfig.json`, `rootDir: src`, `outDir: lib`).
151
+ - `package.json` points `main` and `types` at `lib/index.js` /
152
+ `lib/index.d.ts`, so a checkout must be built before it can be mounted.
153
+ - Relative imports carry the `.ts` extension and `rewriteRelativeImportExtensions`
154
+ rewrites them on emit; the source is also runnable directly by Node's type
155
+ stripping, which is how the tests run (`npm test`; it passes a quoted glob so
156
+ cmd.exe works too).
157
+ - `types.ts` is types only, so it emits nothing; every other module emits a
158
+ normal ES module.
159
+ - `index.js`, `engines.js` and `proxy.js` are the pre-rewrite JavaScript
160
+ implementation. They are not imported by `src/` and are not part of the
161
+ TypeScript build.
162
+
163
+ ## Route memory and engine health
164
+
165
+ `EngineHealth` does two jobs. It keeps the failure cooldown (an engine that
166
+ returned a retryable block rests for 5–30 minutes) and, since 0.21.0, a
167
+ ten-minute route preference. A direct request that ends in `403`/`429` and then
168
+ succeeds through the tunnel records `proxy` for that engine; the next request
169
+ tries the tunnel first, and the direct route stays as a fallback. A later direct
170
+ success clears the preference. The transport test drives this through a real
171
+ HTTP `CONNECT` proxy rather than mocking `proxyGet`.
172
+
173
+ ## MCP runner
174
+
175
+ `mcp/server.mjs` is a dependency-free stdio JSON-RPC process. It imports the
176
+ same `lib/engines.js`, `lib/fetch.js`, `lib/research.js` and
177
+ `lib/verticals.js` as the DSH plugin, exposes four tools (`search`, `fetch`,
178
+ `research`, `domain_search`), wraps every output in the untrusted-content
179
+ boundary and never reads a DSH profile. It is not a second search
180
+ implementation; a ranking or transport fix reaches both runtimes after
181
+ `npm run build`.
@@ -6,6 +6,8 @@ whether a provider answers at all, how long it takes, how many distinct domains
6
6
  it returns, and whether a page on a known official domain appears in the top 8.
7
7
  The failures are part of the result.
8
8
 
9
+ > **Version note (2026-11-02):** 0.24.0 added Windows system-proxy discovery, and 0.24.1 is a documentation/metadata patch. Neither changes the search, ranking or source-adapter path measured below, so the 0.23.0 numbers remain the latest comparison.
10
+
9
11
  ## Method
10
12
 
11
13
  - **Corpus:** 30 queries in `bench/compare30/queries.json` — 13 release queries
@@ -0,0 +1,203 @@
1
+ # Engines
2
+
3
+ What each engine is asked for, how its answer is parsed, and what was actually
4
+ measured on this network. The registry is `HOSTS` in `src/engines.ts`; the
5
+ parsers are in `src/parsers.ts`; the shared request path is
6
+ `src/transport.ts`.
7
+
8
+ - [Architecture](architecture.md)
9
+ - [Troubleshooting](troubleshooting.md)
10
+ - [README](../README.md) · [中文说明](../README.zh-CN.md)
11
+
12
+ ## The shared request
13
+
14
+ Every engine gets the same base headers, built in `src/transport.ts`:
15
+
16
+ ```
17
+ user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15
18
+ accept-language: zh-CN,zh;q=0.9,en;q=0.8
19
+ accept: application/json (JSON POST engines)
20
+ accept: text/html,application/xhtml+xml (scraped pages)
21
+ content-type: application/json (JSON POST engines only)
22
+ ```
23
+
24
+ - **Route.** `anysearch`, `cn.bing` and `so` go direct first and are retried
25
+ through the tunnel only if the direct attempt fails. `ddg-html`, `www.bing` and
26
+ `ddg-lite` are `viaProxy`: they tunnel from the start, and they are filtered out
27
+ of a run when no proxy is available rather than failing slowly.
28
+ - **Deadline.** `timeoutMs` per attempt (default 15 000 ms). Direct requests use
29
+ `AbortSignal.timeout` combined with the caller's signal; tunnelled requests use
30
+ the socket timeouts in `src/proxy.ts`.
31
+ - **Redirects.** Direct requests let `fetch` follow them; tunnelled requests
32
+ follow up to four hops themselves.
33
+ - **Echo check.** Every scraped page must contain one query token of at least
34
+ three characters (`echoesQuery`) or it is treated as a rate-limit page. JSON
35
+ engines set `skipEcho` because their envelope validates itself.
36
+
37
+ ## `anysearch` — anonymous JSON API
38
+
39
+ | | |
40
+ | --- | --- |
41
+ | Request | `POST https://api.anysearch.com/v1/search` |
42
+ | Body | `{"query":"<query>","max_results":20}` |
43
+ | Region / weight | `both` / 2 (two slots per fusion round) |
44
+ | Tier | primary |
45
+
46
+ The response is an envelope, not markup:
47
+
48
+ ```json
49
+ { "code": 0, "message": "success", "request_id": "…",
50
+ "data": { "results": [{ "title": "…", "url": "…", "snippet": "…", "content": "…" }], "metadata": { } } }
51
+ ```
52
+
53
+ `parseAnySearch` requires `code === 0` and an array at `data.results`; anything
54
+ else — a non-zero code, bad JSON, a missing array — yields `[]` rather than an
55
+ exception. Rows are dropped when `url` does not resolve to an absolute `http(s)`
56
+ URL, and empty `title`/`snippet` fields are omitted. The extra `content` field is
57
+ deliberately ignored: the seam wants a citable title/link/snippet, and the body
58
+ belongs to `web_fetch`.
59
+
60
+ A live capture (see [fixtures](../test/fixtures/README.md)) shows the envelope
61
+ and rows above, plus the service's error shape: an empty query is answered with
62
+ HTTP 400 and `{"code":-1,"message":"Query is required."}`.
63
+
64
+ The endpoint is metered: anonymous calls return at most 10 results, and the
65
+ provider's free tier with a key is documented upstream as 1 000 requests per day.
66
+ That is why the engine asks for 20 rows (`ANYSEARCH_MAX_RESULTS`) but the merge
67
+ sees at most what the service sends.
68
+
69
+ ## `ddg-html` — DuckDuckGo HTML
70
+
71
+ | | |
72
+ | --- | --- |
73
+ | Request | `GET https://html.duckduckgo.com/html/?q=<urlencoded query>` |
74
+ | Region / weight | `global` / 1 |
75
+ | Tier / route | primary / tunnel only |
76
+
77
+ The richest English endpoint of the three DuckDuckGo faces here. `parseDdgHtml`
78
+ matches anchors whose `class` is `result__a` (single or double quotes, any
79
+ attribute order), takes the anchor text as the title, and looks for an element
80
+ with `class="result__snippet"` between this anchor and the next one as the
81
+ snippet. Hrefs may be direct or wrapped as `/l/?uddg=<percent-encoded>`, which
82
+ `resolveHref` unwraps.
83
+
84
+ Measured: the live capture on 2026-10-22 serves **direct** hrefs, not the `uddg`
85
+ wrapper that the original implementation was written against. The unwrapper is
86
+ kept because the wrapper still appears, and it is pinned by a synthetic fixture.
87
+
88
+ ## `cn.bing` — Bing China
89
+
90
+ | | |
91
+ | --- | --- |
92
+ | Request | `GET https://cn.bing.com/search?q=<urlencoded query>` |
93
+ | Region / weight | `cn` / 1 |
94
+ | Tier / route | primary / direct first, tunnel retry |
95
+
96
+ The Chinese workhorse: the `cn` host targets the mainland index, mixed
97
+ Chinese/English queries work, and the result URLs are direct links. No market
98
+ parameters are sent.
99
+
100
+ ## `www.bing` — Bing international (fallback)
101
+
102
+ | | |
103
+ | --- | --- |
104
+ | Request | `GET https://www.bing.com/search?q=<urlencoded>&setmkt=en-US&setlang=en&cc=US` |
105
+ | Region / weight | `global` / 1 |
106
+ | Tier / route | fallback / tunnel only |
107
+
108
+ `parseBing` splits the page on `<li class="b_algo"`, takes the first
109
+ `<h2><a href>` pair as title and URL, and the first `<p>` as the snippet.
110
+
111
+ Links are frequently Bing click trackers — `/ck/a?...&u=a1<base64url>` — where
112
+ the `&` arrives HTML-escaped as `&amp;`. `resolveHref` decodes `&amp;`/`&#38;`
113
+ first, base64url-decodes the payload, and rejects the tracker if no real target
114
+ comes out.
115
+
116
+ **Why it is only a fallback.** Measured on stopword-heavy English queries it
117
+ matched a single token and answered with unrelated pages: `how to set up postgres
118
+ logical replication` returned jigsaw-puzzle sites for "set", while the engines
119
+ above stayed on topic. It is worth reaching only when nothing else answered.
120
+
121
+ ## `so` — 360 search
122
+
123
+ | | |
124
+ | --- | --- |
125
+ | Request | `GET https://www.so.com/s?q=<urlencoded query>` |
126
+ | Region / weight | `cn` / 1 |
127
+ | Tier / route | primary / direct first, tunnel retry |
128
+
129
+ `parseSo` splits on `class="res-list"`, finds the heading `<h3 class="res-title…">`
130
+ (the served class value carries a trailing space, and the wrapper is a `<li>` for
131
+ ordinary results and a `<div>` for rich ones, so the element name is not matched),
132
+ then reads the **`data-mdurl`** attribute of the first anchor after it. The
133
+ `href` is a 360 click tracker, which is not a citable source: a result without
134
+ `data-mdurl` is skipped. The snippet comes from `class="res-list-summary"`.
135
+
136
+ ## `ddg-lite` — DuckDuckGo Lite (fallback)
137
+
138
+ | | |
139
+ | --- | --- |
140
+ | Request | `GET https://lite.duckduckgo.com/lite/?q=<urlencoded query>` |
141
+ | Region / weight | `global` / 1 |
142
+ | Tier / route | fallback / tunnel only |
143
+
144
+ `parseDdgLite` accepts single- or double-quoted attributes, because this page
145
+ serves `class='result-link'` where Bing serves double quotes. Each hit is
146
+ examined up to the next hit, which is where its sibling
147
+ `<td class='result-snippet'>` lives.
148
+
149
+ ## Parser regressions
150
+
151
+ Six failures were paid for once and are now pinned by tests and fixtures:
152
+
153
+ 1. a Bing `&amp;u=a1<base64url>` tracker must decode to the real target, not a
154
+ `bing.com/ck/a` URL;
155
+ 2. a DuckDuckGo `/l/?uddg=<urlencoded>` wrapper must unwrap to the target;
156
+ 3. a single-quoted `class='result-link'` must still be parsed;
157
+ 4. 360's `data-mdurl` must be used, not the click tracker;
158
+ 5. an AnySearch envelope with `code !== 0` must yield `[]`;
159
+ 6. `echoesQuery` must reject a rate-limit/captcha page served for a different
160
+ query.
161
+
162
+ Fixtures: [test/fixtures/README.md](../test/fixtures/README.md) lists every file,
163
+ the exact URL it came from, the query and the capture date (2026-10-22). Two
164
+ files are synthetic and marked as such: the `uddg` wrapper and Bing's
165
+ rate-limited reply, whose block page could not be provoked on demand.
166
+
167
+ ## What was measured
168
+
169
+ Timings on this network, with a tunnel on:
170
+
171
+ | Situation | Time |
172
+ | --- | --- |
173
+ | CJK query, `auto` | 1.4–2.3 s |
174
+ | CJK query, domestic engines only (no AnySearch) | 0.2–0.7 s |
175
+ | ASCII query, `auto` | 2.6–3.8 s |
176
+
177
+ Coverage for an English query: 16–30 sources across 12–21 distinct domains, and
178
+ the first 8 slots are 8/8 on-topic foreign documentation. The A/B evidence for
179
+ leading CJK queries with AnySearch is in the [README](../README.md#ab-evidence-cjk-queries-need-anysearch-too).
180
+
181
+ Admission is weighted, scoring is not: with weights 2/1/1 over lists of 10, 10
182
+ and 4 results at `limit: 8`, the 8 slots split 4/2/2 between the engines. The
183
+ previous rule multiplied every row of an engine by its weight, which let
184
+ AnySearch fill all 8 slots and starved the Chinese engines completely (8/0/0).
185
+
186
+ Block pages seen in practice:
187
+
188
+ - 360 served its `访问异常页面` with a captcha under rapid unattended requests;
189
+ - another domestic engine (Shenma, not part of this registry) served an `x5sec`
190
+ punish page during the same measurements;
191
+ - Bing answered repeated rapid requests with a "trending" feed belonging to
192
+ somebody else's search — HTTP 200, so only the echoed-query check catches it.
193
+
194
+ That evidence is why block pages and echoed-query mismatches cool an engine down
195
+ instead of being retried on every search.
196
+
197
+ ## What could not be made to work
198
+
199
+ - **Google.** A tunnelled `www.google.com/search` returns a JavaScript shell
200
+ (94 KB, 1–3 links); headless Chrome is redirected to `google.com.hk` and gets a
201
+ reCAPTCHA page. `gbv=1`, a consent cookie and a mobile user agent all failed.
202
+ - **Public SearXNG instances.** Eight were tried; all answered 429, 403 or
203
+ something that was not JSON.
@@ -6,6 +6,8 @@ This note answers three questions from the public 30-query comparison:
6
6
  2. Can the AnySearch tail and the Chinese Rust miss be fixed?
7
7
  3. Does the fix require a vector database or a different retrieval model?
8
8
 
9
+ > **Version note (2026-11-02):** 0.24.0 added Windows system-proxy discovery, and 0.24.1 is a documentation/metadata patch. The search path is unchanged, so these 0.23.0 measurements still describe 0.24.x.
10
+
9
11
  The short answer: Exa is a different architecture, not a faster RRF loop. This
10
12
  plugin can borrow tail-tolerance and source-specific facts, but copying Exa's
11
13
  semantic index would be a different product. The changes below improve the
@@ -0,0 +1,160 @@
1
+ # 让 DSH 插件的配置出现在「设置」里:四层判定链
2
+
3
+ > 结论先行:一个插件配置项要能在 Settings 里编辑,需要**同时**满足四层条件。
4
+ > 任何一层不满足,症状都完全一样——**设置页空白**。这就是它难 debug 的原因。
5
+ > 本文的 file:line 均来自 `DeepSeek Harness.app/Contents/Resources/app.asar`(2026-10 版本)。
6
+
7
+ ## 症状
8
+
9
+ 设置页显示插件标题和描述,但下面没有控件;或显示「当前部署尚未提供此插件的设置」。
10
+ 插件本身加载正常(工具能用),日志里可能有一行 `warning: 1 entry did not activate`。
11
+
12
+ ## 四层判定链
13
+
14
+ ### 第 1 层:插件必须真正激活(`fiber.state === 2`)
15
+
16
+ `@deepseek-ai/dsh-settings/lib/index.js` 的 `describe()` 开头:
17
+
18
+ ```js
19
+ const schema = this.schema(entry);
20
+ if (schema === undefined || entry.fiber === undefined
21
+ || entry.fiber.runtime === null || entry.fiber.state !== 2) return [];
22
+ ```
23
+
24
+ 而 `schema(entry)` 是:
25
+
26
+ ```js
27
+ schema(entry) {
28
+ const schema = entry.fiber?.runtime?.Config;
29
+ return schema !== undefined && 'toJSON' in schema ? schema : undefined;
30
+ }
31
+ ```
32
+
33
+ **要点:schema 挂在「运行时对象」上,不是模块命名空间上。**
34
+
35
+ Loader 的 `unwrapExports`(`cordis-plugin-loader/lib/index.js:669`):
36
+
37
+ ```js
38
+ exports = exports.default ?? exports;
39
+ ```
40
+
41
+ | 插件导出写法 | 结果 |
42
+ |---|---|
43
+ | `export function apply` + `export { Config }`(无 default) | ✅ `plugin` = 命名空间,`plugin.Config` 命中 |
44
+ | `export default Object.assign(apply, { Config, inject })` | ✅ `plugin` = default,`Config` 命中 |
45
+ | `export default apply` 再另写 `export const Config` | ❌ default 存在 → 命名空间被丢弃 → `runtime.Config` 为 undefined |
46
+
47
+ **函数式插件若用 default 导出,必须把 `Config` 和 `inject` 一起挂上去**;只挂 `Config` 会得到
48
+ `Error: cannot get property "web" without inject`——Cordis 从它挂载的那个对象读 `inject`。
49
+
50
+ ### 第 2 层:配置校验必须通过
51
+
52
+ Loader 只校验一次,校验的是 **`entry.options.config`(patch 里的原始值)**:
53
+
54
+ ```js
55
+ // cordis/lib/index.js:1355 → :1345 → :956
56
+ const result = runtime.Config['~standard'].validate(config);
57
+ if (result.issues) throw new ValidationError(result.issues);
58
+ return result.value;
59
+ ```
60
+
61
+ **校验后的 `result.value` 里,每个 volatile 字段都是 `{ get }` 持有器,不是值。**
62
+
63
+ 最容易踩的坑:**别把已经解析过的 config 再喂回 schema**。
64
+
65
+ ```ts
66
+ // ❌ 在 apply 内部这样写,必炸
67
+ const settings = Config(config as PluginConfig);
68
+ // ValidationError: $.timeoutMs expected number but got [object Object]
69
+ ```
70
+
71
+ 这条错误消息**没有** `invalid config:` 前缀,说明它是 schemastery 抛的(你自己的代码),
72
+ 却被 Host 报成「entry did not activate」,极易误判成插件加载失败。
73
+
74
+ 正确做法:先解包再校验,或干脆只在校验时用 plain 值、输出时优先用调用方的持有器。
75
+
76
+ ### 第 3 层:至少一个字段带 `meta.volatile`
77
+
78
+ `lib/types/schema.js:43`:
79
+
80
+ ```js
81
+ export function volatileForm(schema) {
82
+ if (schema.meta.volatile) return plainSchema(schema);
83
+ if (schema.type === 'object') {
84
+ // 只保留 volatileForm !== undefined 的子字段
85
+ return Object.keys(dict).length === 0 ? undefined : z.object(dict);
86
+ }
87
+ return undefined;
88
+ }
89
+ ```
90
+
91
+ `describe()` 里 `if (form === undefined) return [];`——**一个 volatile 字段都没有,整个命名空间被丢弃**。
92
+
93
+ ### 第 4 层:消费方必须 `.get()`
94
+
95
+ 一旦标了 `.volatile()`,schema 解析结果是 `{ get: fn }`:
96
+
97
+ ```js
98
+ z.object({ timeoutMs: z.number().default(15000).volatile() })({})
99
+ // → { timeoutMs: { get: [Function], [Symbol(cosmokit.volatile.write)]: [Function] } }
100
+ // .get() === 15000
101
+ ```
102
+
103
+ 所以读取处必须解包:
104
+
105
+ ```ts
106
+ const read = (v: unknown) => {
107
+ const holder = v as { get?: unknown } | null;
108
+ return typeof holder?.get === 'function' ? (holder.get as () => unknown)() : v;
109
+ };
110
+ ```
111
+
112
+ 数组也一样——`{}` 只是持有器的 **JSON 形态**(`get` 是函数、writer 是 symbol)。不标数组反而会让
113
+ 设置页保存时报 `Config field "hosts" is not volatile`(`dsh-settings/lib/types/index.js:351`)。
114
+
115
+ ## 最小 checklist(新插件抄这个)
116
+
117
+ ```ts
118
+ export const Config = z.object({
119
+ someScalar: z.string().default('').description('…').volatile(), // 标量标 volatile
120
+ someArray: z.array(z.string()).default([]).description('…').volatile(), // 数组也标
121
+ });
122
+
123
+ export function apply(ctx, config) {
124
+ // config 里已是持有器:读的时候必须 .get()
125
+ const value = config.someScalar.get();
126
+ }
127
+
128
+ export default Object.assign(apply, { Config, inject }); // Config + inject 一起挂
129
+ ```
130
+
131
+ 保存/生效语义:**只有 volatile 字段变化**才走 live commit(原地改持有器、不重挂载,
132
+ `cordis-plugin-loader/lib/index.js:382-435`);其他字段变化会整个插件重挂载。因此
133
+ 如果插件在 `apply` 里缓存了配置快照,volatile 的实时改动对缓存那份不生效——要么每次调用重读,
134
+ 要么接受「需重挂载」。
135
+
136
+ ## 诊断顺序(不要跳步)
137
+
138
+ 1. **先确认插件激活**:看有没有 `did not activate`,有就先修激活,别动设置页;
139
+ 2. **确认 `runtime.Config` 存在**:`describe()` 第一行就依赖它;
140
+ 3. **确认 patch 校验通过**:注意报错有没有 `invalid config:` 前缀,没有就是你自己代码抛的;
141
+ 4. **确认命名空间被收录**:字段有没有 `meta.volatile`;
142
+ 5. **确认客户端读的是正确的 ns 名**:`entry.options.id`(通常等于 patch 里的 `id`);
143
+ 6. 最后才怀疑客户端渲染。
144
+
145
+ ## 这次踩过的四个误判(供警惕)
146
+
147
+ | 当时的判断 | 为什么看起来合理 | 实际 |
148
+ |---|---|---|
149
+ | 「客户端没 inject `configForms`」 | 页面确实什么都没渲染 | 真实存在但非根因 |
150
+ | 「表单是异步投影,读太早」 | 加订阅后报错消失 | 真实存在但非根因 |
151
+ | 「字段缺 `.volatile()`」 | 官方插件都有这个标记 | 真实存在但非根因 |
152
+ | 「要重装/换安装方式」 | 别的插件能显示,我们是手装的 | 无关 |
153
+
154
+ **根因是第 2 层**:`apply` 内部二次校验已解析的 config,抛在 plugin 内 → 表现为「未激活」。
155
+ 四层都修对之后页面才出来——所以每次只修一层,症状看起来完全没变。
156
+
157
+ **教训:先定位「命名空间有没有被 Host 收录」,再改客户端。** 客户端加一行诊断
158
+ (把 `settings.describe()` 返回的 ns 列表打印出来)能在 10 分钟内省掉上面四个误判。
159
+
160
+ > 同一份内容也作为 skill 存放在 `~/.dsh/skills/dsh-plugin-settings-form/SKILL.md`(自包含,触发时一次性加载)。改这里时请同步改那边。
@@ -29,7 +29,7 @@ Only `0.2.1-rc.2` is declared, because that is the version actually exercised.
29
29
  `mktemp -d`; the real `~/.dsh` was never written to. Bundled pnpm 11.28.5.
30
30
  - **Windows CI:** GitHub Actions `windows-latest`, Node 24. Install, typecheck,
31
31
  the full test suite, build and the entry-point import all pass. Run:
32
- https://github.com/fangchenghao/dsh-web-search/actions/runs/40354223874
32
+ https://github.com/fangchenghao/dsh-web-search/actions/runs/41387433033
33
33
  (the matrix also runs on macOS and Ubuntu). This is a CI environment, not a
34
34
  claim about every Windows terminal or proxy client.
35
35
 
@@ -0,0 +1,225 @@
1
+ # Troubleshooting
2
+
3
+ Symptom first, cause second, fix third. Every command below is meant to be run
4
+ from the repository root.
5
+
6
+ - [Architecture](architecture.md)
7
+ - [Engines](engines.md)
8
+ - [README](../README.md) · [中文说明](../README.zh-CN.md)
9
+
10
+ ## `web_search` fails with "no search engine returned results"
11
+
12
+ That message is the provider's own, and the interesting part is in the
13
+ parentheses: one `id: reason` entry per engine that was tried. Read it before
14
+ changing anything.
15
+
16
+ ```
17
+ no search engine returned results for "…" (anysearch: HTTP 429; cn.bing: page does not match the query (possibly rate limited); …)
18
+ ```
19
+
20
+ | Reason in the list | Meaning | What to do |
21
+ | --- | --- | --- |
22
+ | `no results` | the engine answered 2xx and parsed to zero sources | markup may have changed; check that engine's parser against a fresh capture |
23
+ | `page does not match the query (possibly rate limited)` | the echoed-query check failed: a block/trending page for someone else's query | let the cooldown pass, or search a different query |
24
+ | `HTTP 403` / `HTTP 429` | the engine decided this client is a robot or is throttling it | cooldown applies; slow down |
25
+ | `still running at deadline` | the soft deadline released the tier first | `wait: all`, a larger `timeoutMs`, or accept it |
26
+ | `aborted once enough results had arrived` | the tier had enough and stopped waiting | not an error |
27
+ | an `ECONNRESET`/`ENOTFOUND` style message | transport failure | check the tunnel, see below |
28
+
29
+ If *every* reason mentions the tunnel or a timeout, and the `global` engines are
30
+ missing from the list entirely, the run had no proxy: `viaProxy` engines
31
+ (`ddg-html`, `www.bing`, `ddg-lite`) are filtered out rather than failing slowly.
32
+ See [proxy not detected](#proxy-is-not-detected).
33
+
34
+ To narrow a failure down, restrict the run to one engine and watch the reason
35
+ change:
36
+
37
+ ```yaml
38
+ - id: web-search-bing
39
+ name: dsh-web-search-bing
40
+ config:
41
+ hosts: [anysearch]
42
+ ```
43
+
44
+ ## The provider is not selected
45
+
46
+ `web_search` still behaves like the built-in provider → `web.searchProvider` is
47
+ not `bing`. The provider id is fixed (`PROVIDER_ID = 'bing'`) and `available()`
48
+ returns `true` unconditionally, so selection is purely configuration — there is
49
+ no credential or health gate to blame.
50
+
51
+ ```yaml
52
+ - id: web
53
+ name: "@deepseek-ai/dsh-web"
54
+ config:
55
+ searchProvider: bing
56
+ fetchProvider: http
57
+ ```
58
+
59
+ Check that the plugin is actually mounted and registers itself. This snippet
60
+ loads the built entry, calls `apply` with a stub context and prints the provider
61
+ id — no network access:
62
+
63
+ ```sh
64
+ NODE=~/.dsh/dsh-runtimes/dsh-primary-runtime/dependencies/node/bin/node
65
+ $NODE -e "
66
+ import('file://$HOME/.dsh/plugins/dsh-web-search-bing/lib/index.js').then((m) => {
67
+ let p; m.apply({ web: { registerSearchProvider: (x) => { p = x } } }, m.Config({}));
68
+ console.log(p.id, p.available());
69
+ });"
70
+ ```
71
+
72
+ Expected output: `bing true`. If the import fails with `ERR_MODULE_NOT_FOUND`,
73
+ `lib/` has not been built — see [below](#lib-is-missing).
74
+
75
+ ## `lib/` is missing
76
+
77
+ `package.json` points `main` and `types` at `lib/`. The compiled `lib/` is
78
+ **committed** (DSH STORE scans the fixed Git tree), so a fresh checkout already
79
+ contains it. After changing `src/`, rebuild it:
80
+
81
+ ```sh
82
+ npm run build
83
+ ```
84
+
85
+ `prepublishOnly` runs the same build for a published package.
86
+
87
+ ## Code changes have no effect
88
+
89
+ Two separate reasons, usually both:
90
+
91
+ 1. ESM modules are evaluated once per process, so a running DSH keeps the code it
92
+ loaded. Restart the host.
93
+ 2. The host loads `lib/`, not `src/`. Rebuild before restarting.
94
+
95
+ ```sh
96
+ npm run build
97
+ ```
98
+
99
+ ## Proxy is not detected
100
+
101
+ Resolution order is: explicit `proxy` config → environment → the OS system
102
+ proxy (macOS `scutil`, Windows Internet Settings) → direct. The decision is
103
+ cached for 60 seconds, so a change can take up to a minute (or a restart) to be
104
+ honoured.
105
+
106
+ 1. Is a proxy configured at all?
107
+
108
+ macOS/Linux:
109
+
110
+ ```sh
111
+ env | grep -i proxy
112
+ ```
113
+
114
+ Windows PowerShell:
115
+
116
+ ```powershell
117
+ Get-ChildItem Env: | Where-Object Name -match 'proxy'
118
+ ```
119
+
120
+ `DSH_WEB_SEARCH_PROXY`, `HTTPS_PROXY`, `ALL_PROXY` and `HTTP_PROXY` are read
121
+ in both casings. A value counts only if it parses as an **`http://`** proxy;
122
+ `socks5://` and `https://` values are skipped (the tunnel is a plain
123
+ `CONNECT`, so an `https://` proxy would be dialled in cleartext).
124
+ `socks5://` users must point at the client's HTTP/mixed port instead.
125
+
126
+ 2. Is the system proxy set, and is that port actually listening?
127
+
128
+ macOS:
129
+
130
+ ```sh
131
+ scutil --proxy
132
+ nc -z -G 1 127.0.0.1 7890 && echo reachable || echo refused
133
+ ```
134
+
135
+ Windows PowerShell:
136
+
137
+ ```powershell
138
+ Get-ItemProperty 'HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings' |
139
+ Select-Object ProxyEnable, ProxyServer
140
+ Test-NetConnection 127.0.0.1 -Port 7890
141
+ ```
142
+
143
+ The system proxy is only adopted when the port accepts a TCP connection,
144
+ within 400 ms. A stale `ProxyEnable = 1` / `HTTPSEnable : 1` with nothing
145
+ listening is ignored on purpose, so the search degrades to a direct request
146
+ instead of stalling.
147
+
148
+ 3. Is `NO_PROXY` in the way?
149
+
150
+ A loopback-only list (`localhost,127.0.0.1`, a common shell default) is
151
+ skipped: it cannot cover a public search host and must not be read as "no
152
+ proxy". A wildcard (`NO_PROXY=*`) is honoured and disables proxying.
153
+
154
+ 4. Fix it explicitly instead of relying on detection:
155
+
156
+ ```yaml
157
+ - id: web-search-bing
158
+ name: dsh-web-search-bing
159
+ config:
160
+ proxy: "http://127.0.0.1:7890"
161
+ ```
162
+
163
+ Under TUN/enhanced mode the kernel takes over TCP transparently; this module is
164
+ not involved and the behaviour matches a direct request, so a working TUN setup
165
+ needs no proxy configuration at all.
166
+
167
+ ## An engine is always skipped with "cooling down"
168
+
169
+ The failure list says both how long and why:
170
+ `cooling down for another 2m 10s after SEARCH_THROTTLED`. Durations are
171
+ `Retry-After` when the service sent one, otherwise `baseCooldownMs` doubled per
172
+ consecutive failure, capped at `maxCooldownMs` (5 min → 30 min by default).
173
+
174
+ - `SEARCH_THROTTLED`, `SEARCH_BLOCKED`, `SEARCH_TIMEOUT`, `SEARCH_NETWORK` cool
175
+ the engine down.
176
+ - `SEARCH_PARSE` and `SEARCH_ABORTED` never do: markup drift and caller
177
+ cancellation cannot be fixed by waiting.
178
+
179
+ Check what the service actually said (a 429 with `Retry-After` explains a long
180
+ rest):
181
+
182
+ ```sh
183
+ curl -sSI --max-time 10 'https://www.so.com/s?q=test' | head -20
184
+ ```
185
+
186
+ A cooldown never empties the engine set: if every engine is resting, the run
187
+ uses them anyway. To clear the state early, restart the host — the cache and the
188
+ health tracker are per mount, so a restart starts from a clean slate.
189
+
190
+ ## Results do not change after editing the config
191
+
192
+ The result cache answers identical logical requests for `cacheTtlMs` (default
193
+ 10 minutes). The key includes the lead region, the `wait` policy, the budget, the
194
+ participating engine ids and the normalised query, so most config changes do
195
+ produce a new key — but a query repeat within the TTL is served from memory. Set
196
+ `cacheTtlMs: 0` while experimenting.
197
+
198
+ ## Results have no `publishedAt`
199
+
200
+ Not a bug. No engine currently supplies a date: the scraped pages do not expose
201
+ one in a stable field, and AnySearch's rows carry `title`, `url`, `snippet` and
202
+ `content` but no date. Sources therefore omit `publishedAt`, and the freshness
203
+ boost in the ranker stays inactive until an engine starts providing it.
204
+
205
+ ## Google never shows up
206
+
207
+ Expected, and not configurable. Google is unreachable from this network: a
208
+ tunnelled `www.google.com/search` returns a JavaScript shell (94 KB, 1–3 links),
209
+ and headless Chrome is redirected to `google.com.hk` with a reCAPTCHA page.
210
+ `gbv=1`, a consent cookie and a mobile user agent were all tried. See
211
+ [engines.md](engines.md#what-could-not-be-made-to-work).
212
+
213
+ ## Typecheck fails after an edit
214
+
215
+ Node strips types instead of transforming them, so the source must remain
216
+ erasable:
217
+
218
+ - no `enum`, no `namespace`, no constructor parameter properties;
219
+ - relative imports carry the `.ts` extension (`from './types.ts'`);
220
+ - type-only imports use `import type`.
221
+
222
+ ```sh
223
+ npm run typecheck # tsc --noEmit over src/**/*.ts
224
+ npm run check # typecheck + the whole test suite
225
+ ```
package/locales/en.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "meta": {
3
- "title": "Web Search (Bing)",
4
- "description": "Edits the key-free Bing and AnySearch provider: engine allow-list, wait policy, proxy, result budget, ranking, cache and cooldown settings."
3
+ "title": "Web Search",
4
+ "description": "Key-free search, page fetch and research: eight engines fused with RRF, 34 dedicated sources, fetch ladder, proxy, cache and cooldown settings."
5
5
  }
6
6
  }
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "meta": {
3
- "title": "网页搜索(Bing)",
4
- "description": "编辑免密钥的 Bing 与 AnySearch 搜索提供方:引擎白名单、等待策略、代理、结果上限、排序、缓存与冷却设置。"
3
+ "title": "网页搜索",
4
+ "description": "免密钥的搜索、抓取与研究:8 个引擎 RRF 融合、34 个专用源、抓取梯队、代理、缓存与冷却设置。"
5
5
  }
6
6
  }
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@fffangchenghao/dsh-web-search",
3
- "version": "0.24.0",
4
- "description": "DSH bundle: key-free web search, page fetch and one web_tools action tool for verticals, platforms and cited research; eight engines fused with RRF, 34 vertical sources, arXiv research lookups and readable papers, official release facts, a replayable benchmark, an optional Firecrawl key and a dependency-free MCP server.",
3
+ "version": "0.24.1",
4
+ "description": "DSH bundle: key-free web search, page fetch and research for DeepSeek Harness; eight engines fused with RRF, 34 vertical sources, arXiv, Windows/macOS system-proxy support and a dependency-free MCP server.",
5
5
  "type": "module",
6
6
  "main": "lib/index.js",
7
7
  "types": "lib/index.d.ts",
@@ -33,7 +33,11 @@
33
33
  "docs/cross-agent.md",
34
34
  "docs/comparison.md",
35
35
  "docs/images/web-tools-demo.svg",
36
- "docs/latency-and-sources.md"
36
+ "docs/latency-and-sources.md",
37
+ "docs/plugin-settings-form.md",
38
+ "docs/troubleshooting.md",
39
+ "docs/architecture.md",
40
+ "docs/engines.md"
37
41
  ],
38
42
  "scripts": {
39
43
  "build": "tsc -p tsconfig.json",
@@ -53,7 +57,11 @@
53
57
  "anysearch",
54
58
  "research",
55
59
  "web-tools",
56
- "mcp"
60
+ "mcp",
61
+ "windows",
62
+ "keyless",
63
+ "arxiv",
64
+ "page-fetch"
57
65
  ],
58
66
  "license": "MIT",
59
67
  "dsh": {