@fffangchenghao/dsh-web-search 0.24.0 → 0.24.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +48 -0
- package/README.md +24 -16
- package/README.zh-CN.md +22 -15
- package/client.js +5 -5
- package/docs/architecture.md +181 -0
- package/docs/comparison.md +2 -0
- package/docs/engines.md +203 -0
- package/docs/images/bench.png +0 -0
- package/docs/images/bench.svg +84 -0
- package/docs/latency-and-sources.md +2 -0
- package/docs/plugin-settings-form.md +160 -0
- package/docs/rrf-ranking.md +1 -1
- package/docs/store-evidence.md +1 -1
- package/docs/troubleshooting.md +225 -0
- package/locales/en.json +2 -2
- package/locales/zh-CN.json +2 -2
- package/package.json +14 -4
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,53 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.24.2
|
|
4
|
+
|
|
5
|
+
Documentation and image patch; no runtime behavior changed.
|
|
6
|
+
|
|
7
|
+
### Fixed
|
|
8
|
+
- **The benchmark chart no longer claims to be v0.15.0.** `docs/images/bench.png`
|
|
9
|
+
was regenerated from the current offline replay and labelled v0.24.x. The
|
|
10
|
+
numbers are unchanged (P@8 0.4375, R@8 0.7917, MRR@8 0.8071) because the
|
|
11
|
+
ranking path did not change; the chart now says so without presenting an old
|
|
12
|
+
release as the current one.
|
|
13
|
+
- **Baseline labels now say "current" everywhere.** README, Chinese README and
|
|
14
|
+
`docs/rrf-ranking.md` still explain that the tier-aware replay protocol was
|
|
15
|
+
introduced in 0.15.0, but no longer call the current baseline a "0.15.0
|
|
16
|
+
baseline".
|
|
17
|
+
- **The chart has an editable source.** `docs/images/bench.svg` is checked in
|
|
18
|
+
next to the PNG. Both files are shipped in the npm package so the relative
|
|
19
|
+
image link in `docs/rrf-ranking.md` resolves there too.
|
|
20
|
+
|
|
21
|
+
**Test suite: 690 tests across 56 files** (689 pass, 1 skipped).
|
|
22
|
+
|
|
23
|
+
## 0.24.1
|
|
24
|
+
|
|
25
|
+
Documentation and metadata patch; no search or fetch behavior changed.
|
|
26
|
+
|
|
27
|
+
### Fixed
|
|
28
|
+
- **README and docs no longer imply macOS-only proxy discovery.** The English
|
|
29
|
+
and Chinese READMEs, architecture note, and troubleshooting guide now describe
|
|
30
|
+
macOS `scutil` and Windows Internet Settings as the two system-proxy sources,
|
|
31
|
+
with PowerShell checks for environment variables, registry values and port
|
|
32
|
+
reachability.
|
|
33
|
+
- **The test commands match the cross-platform script.** Docs use `npm test`
|
|
34
|
+
instead of an unquoted shell glob; CI documentation now names Ubuntu, macOS
|
|
35
|
+
and Windows.
|
|
36
|
+
- **The troubleshooting guide no longer says `lib/` is gitignored.** It is
|
|
37
|
+
committed, and the guide now says so.
|
|
38
|
+
- **The "no version is hard-coded" claim was removed.** The ecosystem table is
|
|
39
|
+
explicitly a dated snapshot, not a live leaderboard.
|
|
40
|
+
- **README links point at files that exist in the npm package.** The package now
|
|
41
|
+
ships `docs/architecture.md`, `docs/engines.md`, `docs/plugin-settings-form.md`
|
|
42
|
+
and `docs/troubleshooting.md`; the benchmark link goes to GitHub instead.
|
|
43
|
+
- **Package metadata and Settings copy were refreshed.** The npm description
|
|
44
|
+
names Windows/macOS proxy support; keywords include `windows`, `keyless`,
|
|
45
|
+
`arxiv` and `web-fetch`; the Settings title/description now names eight
|
|
46
|
+
engines and 34 sources instead of Bing/AnySearch.
|
|
47
|
+
|
|
48
|
+
**Test suite: 690 tests across 56 files** (689 pass, 1 skipped); this patch does
|
|
49
|
+
not change runtime code.
|
|
50
|
+
|
|
3
51
|
## 0.24.0
|
|
4
52
|
|
|
5
53
|
### Added
|
package/README.md
CHANGED
|
@@ -37,7 +37,7 @@ The benchmark is offline and deterministic: 20 recorded queries, precision/recal
|
|
|
37
37
|
|
|
38
38
|
## Where it stands
|
|
39
39
|
|
|
40
|
-
The web-search corner of the DSH plugin ecosystem is crowded. A
|
|
40
|
+
The web-search corner of the DSH plugin ecosystem is crowded. A dated snapshot
|
|
41
41
|
(2026-10-25; stars and npm downloads are adoption signals, not quality scores):
|
|
42
42
|
|
|
43
43
|
| | this plugin | [dsh-free-search](https://github.com/DDDMUC/dsh-free-search) | [dsh-web-search-pro](https://github.com/anweat/dsh-web-search-pro) | [modsearch](https://github.com/liustack/modsearch) | [anysearch-dsh](https://github.com/anysearch-team/anysearch-dsh) |
|
|
@@ -63,9 +63,9 @@ read is:
|
|
|
63
63
|
- **Key-free + official-source correctness:** its strongest niche. The official
|
|
64
64
|
Chrome, PostgreSQL and AI Act paths and the 34-source vertical layer are not
|
|
65
65
|
common in the other plugins.
|
|
66
|
-
- **Adoption and maturity:** bottom third of this sample.
|
|
67
|
-
|
|
68
|
-
|
|
66
|
+
- **Adoption and maturity:** bottom third of this sample. At the 2026-10-25
|
|
67
|
+
snapshot the repository was four days old, had no stars and little
|
|
68
|
+
independent adoption; the current npm release is 0.24.2.
|
|
69
69
|
- **Tool-surface economy:** now competitive. 0.20.x exposes one `web_tools`
|
|
70
70
|
entry with eight actions, close to `dsh-web-search-pro`'s two tools with 20
|
|
71
71
|
actions; the legacy eight-name surface remains opt-in for old prompts.
|
|
@@ -77,7 +77,7 @@ is the compact, fully key-free option for search plus vertical lookup, research
|
|
|
77
77
|
packs and official-source correction. It is not the most popular, and it should
|
|
78
78
|
not claim to be the largest.
|
|
79
79
|
|
|
80
|
-
> The latest version and install command live on [npm](https://www.npmjs.com/package/@fffangchenghao/dsh-web-search);
|
|
80
|
+
> The latest version and install command live on [npm](https://www.npmjs.com/package/@fffangchenghao/dsh-web-search); the comparison table is a dated snapshot, not a live leaderboard. What follows starts with what this means for you; the technical detail is collected in the appendix.
|
|
81
81
|
|
|
82
82
|
## In one sentence
|
|
83
83
|
|
|
@@ -308,7 +308,7 @@ Each line is the plain version first, the technical note second. The technical n
|
|
|
308
308
|
|
|
309
309
|
13. **It works on a restricted network.**
|
|
310
310
|
With a tunnel running and set as the system proxy, nothing needs configuring. With the tunnel off, it falls back to direct requests instead of failing.
|
|
311
|
-
*Technical: proxy sources in order — the `proxy` setting, then `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY`, then macOS `scutil
|
|
311
|
+
*Technical: proxy sources in order — the `proxy` setting, then `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY`, then the OS system proxy (macOS `scutil`, Windows Internet Settings); HTTP proxies only (`CONNECT`), not `socks5://`.*
|
|
312
312
|
|
|
313
313
|
14. **Every setting is editable in the UI.**
|
|
314
314
|
Thirty controls in six groups; a save takes effect on the next search, apart from `cachePersist` and `cacheFile` (cache wiring at mount) and `toolSurface` (the tool registry is built at mount).
|
|
@@ -340,7 +340,7 @@ Each line is the plain version first, the technical note second. The technical n
|
|
|
340
340
|
|
|
341
341
|
21. **Search quality has a repeatable benchmark.**
|
|
342
342
|
Ranking changes are no longer judged by feel: a fixed set of 20 queries produces the same scores every time.
|
|
343
|
-
*Technical: `bench/` holds 20 queries (10 Chinese, 10 English), five in each of docs / release / package / news, each annotated with 1–3 expected domains. `node scripts/bench.mjs` replays recorded fixtures offline, never touches the network, and prints precision@8, recall@8, MRR@8 plus each engine's contribution; `--record` re-records live (1.5 s between queries) to refresh the fixtures; `--json` gives machine-readable output for version comparisons, and past results live in `bench/results/`.
|
|
343
|
+
*Technical: `bench/` holds 20 queries (10 Chinese, 10 English), five in each of docs / release / package / news, each annotated with 1–3 expected domains. `node scripts/bench.mjs` replays recorded fixtures offline, never touches the network, and prints precision@8, recall@8, MRR@8 plus each engine's contribution; `--record` re-records live (1.5 s between queries) to refresh the fixtures; `--json` gives machine-readable output for version comparisons, and past results live in `bench/results/`. Current baseline (re-run on the 0.24.x line; tier-aware replay, so the recorded fallback engines only count when production would run them): P@8 `0.4375`, R@8 `0.7917`, MRR@8 `0.8071`; Chinese `0.4000 / 0.8167 / 0.7976`, English `0.4750 / 0.7667 / 0.8167`. A separate five-query long-Chinese diagnostic (`bench/queries-long.json`, recorded with `--queries ... --fixtures ...`) scores `0.4500 / 1.0000 / 0.7667`, with every query hitting an expected domain; it is kept separate so the 20-query baseline stays comparable. The controlled experiments and known gaps are in the appendix.*
|
|
344
344
|
|
|
345
345
|
22. **Research, in one call.**
|
|
346
346
|
For "what do the official docs and two other sources actually say about X", `research` runs up to three searches, opens the leading pages through the same fetch ladder, and returns one evidence pack in which every excerpt sits under its URL. A page that will not load is reported by URL and reason instead of aborting the pack; bot challenges and login redirects are rejected rather than quoted as evidence.
|
|
@@ -590,7 +590,7 @@ Optionally, add a config row:
|
|
|
590
590
|
|
|
591
591
|
### Step 4: restart completely
|
|
592
592
|
|
|
593
|
-
After changing code or profile config, **quit DSH entirely and open it again** (on macOS
|
|
593
|
+
After changing code or profile config, **quit DSH entirely and open it again** (on macOS press `Cmd+Q`; on Windows quit from the tray or menu; closing the window is not enough). Modules are evaluated once per process, and a running process never sees a file you edited underneath it.
|
|
594
594
|
|
|
595
595
|
Values saved from the Settings page apply immediately; only `cachePersist` and `cacheFile` need the plugin to reload.
|
|
596
596
|
|
|
@@ -792,7 +792,7 @@ No. `package.json` does not list Playwright, and the plugin never downloads or i
|
|
|
792
792
|
|
|
793
793
|
### How do you know search quality improved?
|
|
794
794
|
|
|
795
|
-
From the fixed query set in `bench/`, not from intuition. There are 20 queries (10 Chinese, 10 English), five in each of docs / release / package / news, each annotated with 1–3 expected domains. `node scripts/bench.mjs` replays recorded fixtures offline and prints P@8 / R@8 / MRR@8. The
|
|
795
|
+
From the fixed query set in `bench/`, not from intuition. There are 20 queries (10 Chinese, 10 English), five in each of docs / release / package / news, each annotated with 1–3 expected domains. `node scripts/bench.mjs` replays recorded fixtures offline and prints P@8 / R@8 / MRR@8. The current 20-query baseline (tier-aware replay, protocol introduced in 0.15.0) is **0.4375 / 0.7917 / 0.8071**; Chinese 0.4000 / 0.8167 / 0.7976, English 0.4750 / 0.7667 / 0.8167. Run it before and after a ranking change and the direction is a number. Twenty queries are not statistical significance; news recall is only 0.467 and the per-query table still prints every miss — the appendix says so.
|
|
796
796
|
|
|
797
797
|
### Why do the results differ from what I get searching by hand?
|
|
798
798
|
|
|
@@ -1040,7 +1040,7 @@ Measured on `https://docs.python.org/3/library/asyncio-task.html`: starting at `
|
|
|
1040
1040
|
|
|
1041
1041
|
Metrics: P@8 = matched rows in the top 8 ÷ 8 (fixed denominator; a short list is not excused); R@8 = distinct expected domains found in the top 8 ÷ all expected domains; MRR@8 = the mean reciprocal rank of the first match over 20 queries, with a miss scoring 0. Domain matching is one-way from subdomain to parent: `docs.python.org` matches expected `python.org`, but the reverse does not.
|
|
1042
1042
|
|
|
1043
|
-
**
|
|
1043
|
+
**Current baseline** (v0.24.x replay; tier-aware protocol introduced in 0.15.0):
|
|
1044
1044
|
|
|
1045
1045
|
| Scope | P@8 | R@8 | MRR@8 |
|
|
1046
1046
|
| --- | --- | --- | --- |
|
|
@@ -1067,9 +1067,9 @@ Conclusions:
|
|
|
1067
1067
|
|
|
1068
1068
|
### Proxy
|
|
1069
1069
|
|
|
1070
|
-
Node's global `fetch` reads neither the
|
|
1070
|
+
Node's global `fetch` reads neither the OS system proxy (macOS `scutil` / Windows Internet Settings) nor `HTTPS_PROXY` — unless the process started with `NODE_USE_ENV_PROXY=1`, which cannot be switched on from inside a running process. So when a proxy is needed, the plugin speaks to it directly: send `CONNECT`, upgrade with TLS for `https:` targets, write the request by hand, and decode `Content-Length`, chunked, gzip, deflate and brotli, following up to four redirects. `307`/`308` keep the method and body; any other `3xx` downgrades a `POST` to a bodyless `GET`.
|
|
1071
1071
|
|
|
1072
|
-
Resolution order: the `proxy` setting → the environment (`DSH_WEB_SEARCH_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, `HTTP_PROXY`, either casing) → macOS `scutil --proxy
|
|
1072
|
+
Resolution order: the `proxy` setting → the environment (`DSH_WEB_SEARCH_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, `HTTP_PROXY`, either casing) → the OS system proxy (macOS `scutil --proxy` / Windows Internet Settings), adopted only if a 400 ms TCP probe connects, and cached for 60 seconds. HTTP proxies only; replace `socks5://` with the client's HTTP/mixed port. Proxying is scoped to this plugin — the rest of the app keeps its ordinary network path.
|
|
1073
1073
|
|
|
1074
1074
|
### Known limitations
|
|
1075
1075
|
|
|
@@ -1133,16 +1133,24 @@ Resolution order: the `proxy` setting → the environment (`DSH_WEB_SEARCH_PROXY
|
|
|
1133
1133
|
### Development and tests
|
|
1134
1134
|
|
|
1135
1135
|
```sh
|
|
1136
|
-
|
|
1136
|
+
npm test # the whole offline suite (one live case is skipped by default)
|
|
1137
1137
|
node --test test/proxy.test.ts # one file
|
|
1138
1138
|
npm run typecheck # tsc --noEmit
|
|
1139
1139
|
npm run build # compile src/ to lib/
|
|
1140
1140
|
npm run check # typecheck + tests
|
|
1141
1141
|
```
|
|
1142
1142
|
|
|
1143
|
-
The suite runs offline by default and passes in full; the one live paging check is skipped unless `READ_PAGE_LIVE=1`: parser tests replay captured fixtures from `test/fixtures/` (normal and empty responses for the default eight and the 26 opt-in sources), transport tests open no sockets, proxy tests cover `parseProxy`, `proxyFromEnv` and the pure response framer, time-window tests pin the date with an injected `now`, and the newer suites cover Markdown extraction (`markdown.test.ts`), batch search (`search-many.test.ts`), chunked reading (`read-page.test.ts`), the disk snapshot (`cache-persist.test.ts`), optional JS rendering (`render-js.test.ts`) and benchmark scoring/replay (`bench-metrics.test.ts`, `bench-replay.test.ts`). One live-network test in `read-page.test.ts` is skipped by default; run it with
|
|
1143
|
+
The suite runs offline by default and passes in full; the one live paging check is skipped unless `READ_PAGE_LIVE=1`: parser tests replay captured fixtures from `test/fixtures/` (normal and empty responses for the default eight and the 26 opt-in sources), transport tests open no sockets, proxy tests cover `parseProxy`, `proxyFromEnv` and the pure response framer, time-window tests pin the date with an injected `now`, and the newer suites cover Markdown extraction (`markdown.test.ts`), batch search (`search-many.test.ts`), chunked reading (`read-page.test.ts`), the disk snapshot (`cache-persist.test.ts`), optional JS rendering (`render-js.test.ts`) and benchmark scoring/replay (`bench-metrics.test.ts`, `bench-replay.test.ts`). One live-network test in `read-page.test.ts` is skipped by default; run it with:
|
|
1144
1144
|
|
|
1145
|
-
|
|
1145
|
+
```sh
|
|
1146
|
+
# macOS / Linux
|
|
1147
|
+
READ_PAGE_LIVE=1 node --test test/read-page.test.ts
|
|
1148
|
+
|
|
1149
|
+
# Windows PowerShell
|
|
1150
|
+
$env:READ_PAGE_LIVE=1; node --test test/read-page.test.ts
|
|
1151
|
+
```
|
|
1152
|
+
|
|
1153
|
+
CI (`.github/workflows/ci.yml`) runs `npm ci`, `npm run typecheck`, `npm test` and `npm run build` on Node 24 for Ubuntu, macOS and Windows, then checks that the build emitted `lib/index.js` and `lib/index.d.ts` and that both `apply` and `Config` import.
|
|
1146
1154
|
|
|
1147
1155
|
After editing `src/*.ts`, build and restart — otherwise you are still running the old code in `lib/`.
|
|
1148
1156
|
|
|
@@ -1153,7 +1161,7 @@ After editing `src/*.ts`, build and restart — otherwise you are still running
|
|
|
1153
1161
|
- [docs/architecture.md](docs/architecture.md): the module map and the data flow from the `web` service to a source list.
|
|
1154
1162
|
- [docs/engines.md](docs/engines.md): each engine's request shape, parser quirks and measured results.
|
|
1155
1163
|
- [docs/troubleshooting.md](docs/troubleshooting.md): no results, proxy not detected, provider not selected, cooldown stuck.
|
|
1156
|
-
- [bench/README.md](bench/README.md): how the 20-query retrieval benchmark records fixtures, scores them and compares versions.
|
|
1164
|
+
- [bench/README.md](https://github.com/fangchenghao/dsh-web-search/blob/main/bench/README.md): how the 20-query retrieval benchmark records fixtures, scores them and compares versions.
|
|
1157
1165
|
- [README.zh-CN.md](README.zh-CN.md): the Chinese guide, section for section.
|
|
1158
1166
|
|
|
1159
1167
|
MIT © 2026 fangchenghao. See [LICENSE](LICENSE).
|
package/README.zh-CN.md
CHANGED
|
@@ -36,7 +36,7 @@
|
|
|
36
36
|
|
|
37
37
|
## 它在同类里的位置
|
|
38
38
|
|
|
39
|
-
DSH 插件生态里的「联网搜索」已经很挤。下面是一张 2026-10-25
|
|
39
|
+
DSH 插件生态里的「联网搜索」已经很挤。下面是一张 2026-10-25 的历史快照
|
|
40
40
|
(stars 和 npm 下载量是采用度信号,不是质量分数):
|
|
41
41
|
|
|
42
42
|
| | 本插件 | [dsh-free-search](https://github.com/DDDMUC/dsh-free-search) | [dsh-web-search-pro](https://github.com/anweat/dsh-web-search-pro) | [modsearch](https://github.com/liustack/modsearch) | [anysearch-dsh](https://github.com/anysearch-team/anysearch-dsh) |
|
|
@@ -58,8 +58,7 @@ DSH 插件生态里的「联网搜索」已经很挤。下面是一张 2026-10-2
|
|
|
58
58
|
明确的安全边界、只有 1 个运行时依赖,同时体积远小于 Playwright 重型方案。
|
|
59
59
|
- **免 key + 官方源纠错:最强卖点。** Chrome、PostgreSQL、AI Act 的官方路径
|
|
60
60
|
和 34 个垂直源,不是其他插件普遍具备的。
|
|
61
|
-
- **采用度与成熟度:这个样本里的后段。**
|
|
62
|
-
还缺少独立采用。
|
|
61
|
+
- **采用度与成熟度:这个样本里的后段。** 2026-10-25 快照时 GitHub 仓库刚创建 4 天、0 stars,还缺少独立采用;当前 npm 版本是 0.24.2。
|
|
63
62
|
- **工具面的经济性:现在有竞争力。** 0.20.x 只暴露一个 `web_tools` + 8 个
|
|
64
63
|
action,接近 `dsh-web-search-pro` 的 2 工具 + 20 动作;旧的八个工具名保留为
|
|
65
64
|
可选兼容模式。
|
|
@@ -69,7 +68,7 @@ DSH 插件生态里的「联网搜索」已经很挤。下面是一张 2026-10-2
|
|
|
69
68
|
搜索 + 垂直查询 + 研究包 + 官方源纠错」的轻量选项。它不是最流行的,也不该
|
|
70
69
|
自称最大。
|
|
71
70
|
|
|
72
|
-
> 最新版本与安装命令以 [npm 上的 latest](https://www.npmjs.com/package/@fffangchenghao/dsh-web-search)
|
|
71
|
+
> 最新版本与安装命令以 [npm 上的 latest](https://www.npmjs.com/package/@fffangchenghao/dsh-web-search) 为准;上面的对比表是历史快照,不是实时排行榜。下面先讲「它跟我有什么关系」,技术细节放在后面的附录。
|
|
73
72
|
|
|
74
73
|
## 一句话说明
|
|
75
74
|
|
|
@@ -300,7 +299,7 @@ Harness 里有个叫 `web` 的**服务**,它自己不会搜索,只负责「
|
|
|
300
299
|
|
|
301
300
|
13. **受限网络下也能用。**
|
|
302
301
|
梯子开着并且设为系统代理时,不用配置任何东西;关掉梯子会自动退回直连,不会因此报错。
|
|
303
|
-
*技术补充:代理来源依次是配置里的 `proxy`、环境变量 `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY
|
|
302
|
+
*技术补充:代理来源依次是配置里的 `proxy`、环境变量 `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY`、操作系统系统代理(macOS 的 `scutil`、Windows 的 Internet 设置);只支持 HTTP 代理(`CONNECT`),不支持 `socks5://`。*
|
|
304
303
|
|
|
305
304
|
14. **所有设置都能在界面上改,不用编辑文件。**
|
|
306
305
|
设置页有 30 个控件,分六组;保存后下一次搜索就生效;只有 `cachePersist` / `cacheFile`(决定缓存接线)和 `toolSurface`(工具注册表在挂载时构建)要重载插件。
|
|
@@ -332,7 +331,7 @@ Harness 里有个叫 `web` 的**服务**,它自己不会搜索,只负责「
|
|
|
332
331
|
|
|
333
332
|
21. **搜索质量有了可重复的基准。**
|
|
334
333
|
改排序不再靠感觉:仓库里有一组固定的 20 条查询,每次改完都能跑出同一套分数。
|
|
335
|
-
*技术补充:`bench/` 下 20 条查询(中文 10、英文 10),四类各 5 条(文档 / 版本发布 / 包与库 / 新闻),每条标注 1–3 个期望域名。`node scripts/bench.mjs` 离线读夹具重放,不联网,输出 precision@8、recall@8、MRR@8 与每个引擎的贡献;`--record` 联网重新录制(每条限速 1.5 秒),用于更新夹具;`--json` 输出机器可读结果,便于版本间对比,历史结果存在 `bench/results/`。当前基线(0.15.0
|
|
334
|
+
*技术补充:`bench/` 下 20 条查询(中文 10、英文 10),四类各 5 条(文档 / 版本发布 / 包与库 / 新闻),每条标注 1–3 个期望域名。`node scripts/bench.mjs` 离线读夹具重放,不联网,输出 precision@8、recall@8、MRR@8 与每个引擎的贡献;`--record` 联网重新录制(每条限速 1.5 秒),用于更新夹具;`--json` 输出机器可读结果,便于版本间对比,历史结果存在 `bench/results/`。当前基线(0.24.x 重放,沿用 0.15.0 的 tier 回放协议):P@8 `0.4375`、R@8 `0.7917`、MRR@8 `0.8071`;中文 `0.4000 / 0.8167 / 0.7976`,英文 `0.4750 / 0.7667 / 0.8167`。另有五条中文长查询的独立诊断集(`bench/queries-long.json`,用 `--queries ... --fixtures ...` 录制):`0.4500 / 1.0000 / 0.7667`,五条都命中期望域名;它与 20 条基线分开保存,避免改变基线的可比性。对照实验和已知短板见附录。*
|
|
336
335
|
|
|
337
336
|
22. **一次调用完成小调研。**
|
|
338
337
|
问「官方文档和另外两个来源到底怎么说」时,`research` 先跑最多三条查询,再用与 `web_fetch` 相同的抓取阶梯打开靠前的页面,最后返回一份证据包:每段摘录下面就是它的 URL。打不开的页面按 URL 和原因列出,不会让整份证据包失败;人机验证页和登录跳转页会被拒绝,不会被当作证据引用。
|
|
@@ -558,7 +557,7 @@ pnpm install
|
|
|
558
557
|
|
|
559
558
|
### 第四步:完全重启
|
|
560
559
|
|
|
561
|
-
改完代码或 profile 配置后,**完全退出 DSH 再打开**(macOS
|
|
560
|
+
改完代码或 profile 配置后,**完全退出 DSH 再打开**(macOS 按 `Cmd+Q`;Windows 从托盘或菜单退出,不能只关窗口)。原因是模块在每个进程里只求值一次,热改文件不会被已运行的进程看见。
|
|
562
561
|
|
|
563
562
|
设置页里大部分值不需要重启,例外情况见下一节。
|
|
564
563
|
|
|
@@ -731,7 +730,7 @@ pnpm install
|
|
|
731
730
|
|
|
732
731
|
### 你们怎么知道搜索质量变好了?
|
|
733
732
|
|
|
734
|
-
靠 `bench/` 里的固定查询集,而不是拍脑袋。20 条查询(中文 10、英文 10),文档 / 版本发布 / 包与库 / 新闻四类各 5 条,每条标注 1–3 个期望域名。`node scripts/bench.mjs` 离线重放录好的夹具,输出 P@8 / R@8 / MRR@8
|
|
733
|
+
靠 `bench/` 里的固定查询集,而不是拍脑袋。20 条查询(中文 10、英文 10),文档 / 版本发布 / 包与库 / 新闻四类各 5 条,每条标注 1–3 个期望域名。`node scripts/bench.mjs` 离线重放录好的夹具,输出 P@8 / R@8 / MRR@8。当前基线(沿用 0.15.0 的 tier 回放协议)是 **0.4375 / 0.7917 / 0.8071**;中文 0.4000 / 0.8167 / 0.7976,英文 0.4750 / 0.7667 / 0.8167。改排序前后各跑一次,数字是变好还是变坏一目了然。样本只有 20 条,不能当统计显著性;新闻类 recall 只有 0.467,逐条输出会把每次没命中都打出来——这些短板都写在附录里。
|
|
735
734
|
|
|
736
735
|
### 为什么搜索结果和我手动搜的不一样?
|
|
737
736
|
|
|
@@ -979,7 +978,7 @@ read_page 调用
|
|
|
979
978
|
|
|
980
979
|
指标:P@8 = 前 8 条里命中期望域名的行数 ÷ 8(固定分母,返回不足 8 条也不放宽);R@8 = 前 8 条覆盖的期望域名 ÷ 全部期望域名;MRR@8 = 第一条命中的排名倒数,20 条取平均,没命中记 0。域名匹配允许「子域命中父域」单向成立:`docs.python.org` 命中期望的 `python.org`,反过来不算。
|
|
981
980
|
|
|
982
|
-
|
|
981
|
+
**当前基线**(0.24.x 重放;沿用 0.15.0 的 tier 回放协议):
|
|
983
982
|
|
|
984
983
|
| 范围 | P@8 | R@8 | MRR@8 |
|
|
985
984
|
| --- | --- | --- | --- |
|
|
@@ -1006,9 +1005,9 @@ read_page 调用
|
|
|
1006
1005
|
|
|
1007
1006
|
### 代理
|
|
1008
1007
|
|
|
1009
|
-
Node 的全局 `fetch`
|
|
1008
|
+
Node 的全局 `fetch` 既不读操作系统系统代理(macOS 的 `scutil`、Windows 的 Internet 设置),也不读 `HTTPS_PROXY`——除非进程启动时带了 `NODE_USE_ENV_PROXY=1`,而这个开关在进程跑起来之后打不开。所以需要代理时,插件自己连:发 `CONNECT`、对 `https:` 目标做 TLS 升级、手写请求、自己解析 `Content-Length` / chunked / gzip / deflate / brotli,最多跟 4 次跳转。`307`/`308` 保留方法和请求体,其他 `3xx` 把 `POST` 降级成无 body 的 `GET`。
|
|
1010
1009
|
|
|
1011
|
-
代理来源顺序:配置里的 `proxy` → 环境变量 `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY`(大小写都认)→ macOS `scutil --proxy
|
|
1010
|
+
代理来源顺序:配置里的 `proxy` → 环境变量 `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY`(大小写都认)→ 操作系统系统代理(macOS `scutil --proxy` / Windows Internet 设置,带 400 ms TCP 探测,结果缓存 60 秒)。只支持 HTTP 代理;`socks5://` 请换成客户端的 HTTP/mixed 端口。代理只作用于本插件,应用其他流量走原来的路。
|
|
1012
1011
|
|
|
1013
1012
|
### 已知限制
|
|
1014
1013
|
|
|
@@ -1072,16 +1071,24 @@ Node 的全局 `fetch` 既不读 macOS 系统代理,也不读 `HTTPS_PROXY`—
|
|
|
1072
1071
|
### 开发与测试
|
|
1073
1072
|
|
|
1074
1073
|
```sh
|
|
1075
|
-
|
|
1074
|
+
npm test # 全部离线测试(联网用例默认跳过)
|
|
1076
1075
|
node --test test/proxy.test.ts # 单个文件
|
|
1077
1076
|
npm run typecheck # tsc --noEmit
|
|
1078
1077
|
npm run build # 编译 src/ 到 lib/
|
|
1079
1078
|
npm run check # typecheck + 测试
|
|
1080
1079
|
```
|
|
1081
1080
|
|
|
1082
|
-
测试默认全程离线、全部通过;唯一需要联网的分段验证默认跳过(`READ_PAGE_LIVE=1` 时才跑):解析器测试回放 `test/fixtures/` 里抓下来的真实响应(含默认八个与 26 个按需源的真实响应),传输测试不开 socket,代理测试只验 `parseProxy`、`proxyFromEnv` 和纯函数式的响应分帧,时间窗测试用注入的 `now` 固定日期;新增的几组覆盖 Markdown 正文提取(`markdown.test.ts`)、批量查询(`search-many.test.ts`)、分段续读(`read-page.test.ts`)、硬盘快照(`cache-persist.test.ts`)、可选 JS 渲染(`render-js.test.ts`)和基准计分/重放(`bench-metrics.test.ts`、`bench-replay.test.ts`)。其中 `read-page.test.ts`
|
|
1081
|
+
测试默认全程离线、全部通过;唯一需要联网的分段验证默认跳过(`READ_PAGE_LIVE=1` 时才跑):解析器测试回放 `test/fixtures/` 里抓下来的真实响应(含默认八个与 26 个按需源的真实响应),传输测试不开 socket,代理测试只验 `parseProxy`、`proxyFromEnv` 和纯函数式的响应分帧,时间窗测试用注入的 `now` 固定日期;新增的几组覆盖 Markdown 正文提取(`markdown.test.ts`)、批量查询(`search-many.test.ts`)、分段续读(`read-page.test.ts`)、硬盘快照(`cache-persist.test.ts`)、可选 JS 渲染(`render-js.test.ts`)和基准计分/重放(`bench-metrics.test.ts`、`bench-replay.test.ts`)。其中 `read-page.test.ts` 有一条真实联网用例默认跳过;联网验证时:
|
|
1083
1082
|
|
|
1084
|
-
|
|
1083
|
+
```sh
|
|
1084
|
+
# macOS / Linux
|
|
1085
|
+
READ_PAGE_LIVE=1 node --test test/read-page.test.ts
|
|
1086
|
+
|
|
1087
|
+
# Windows PowerShell
|
|
1088
|
+
$env:READ_PAGE_LIVE=1; node --test test/read-page.test.ts
|
|
1089
|
+
```
|
|
1090
|
+
|
|
1091
|
+
CI(`.github/workflows/ci.yml`)在 Node 24 的 Ubuntu、macOS 与 Windows 上跑 `npm ci`、`npm run typecheck`、`npm test`、`npm run build`,并检查构建确实产出了 `lib/index.js` 和 `lib/index.d.ts`,且 `apply` 与 `Config` 都可导入。
|
|
1085
1092
|
|
|
1086
1093
|
改完 `src/*.ts` 记得构建并重启,否则跑的还是 `lib/` 里的旧代码。
|
|
1087
1094
|
|
|
@@ -1092,7 +1099,7 @@ CI(`.github/workflows/ci.yml`)在 Node 24 的 Ubuntu 与 macOS 上跑 `npm c
|
|
|
1092
1099
|
- [docs/architecture.md](docs/architecture.md):模块划分和从 `web` 服务到来源列表的数据流。
|
|
1093
1100
|
- [docs/engines.md](docs/engines.md):每个引擎的请求形状、解析怪癖与实测结论。
|
|
1094
1101
|
- [docs/troubleshooting.md](docs/troubleshooting.md):没有结果、代理没探测到、提供方没被选中、冷却卡住。
|
|
1095
|
-
- [bench/README.md](bench/README.md):20 条查询的检索基准怎么录制夹具、怎么计分、怎么比较版本。
|
|
1102
|
+
- [bench/README.md](https://github.com/fangchenghao/dsh-web-search/blob/main/bench/README.md):20 条查询的检索基准怎么录制夹具、怎么计分、怎么比较版本。
|
|
1096
1103
|
- [README.md](README.md):英文版,与中文版逐节对应。
|
|
1097
1104
|
|
|
1098
1105
|
MIT © 2026 fangchenghao,见 [LICENSE](LICENSE)。
|
package/client.js
CHANGED
|
@@ -74,14 +74,14 @@
|
|
|
74
74
|
var ENTRY = 'web-search-bing';
|
|
75
75
|
var PACKAGE_NAME = 'dsh-web-search-bing';
|
|
76
76
|
/** Keep in step with package.json; compared with the npm registry. */
|
|
77
|
-
var VERSION = '0.24.
|
|
77
|
+
var VERSION = '0.24.2';
|
|
78
78
|
/** Snapshot used while no configuration service is reachable. */
|
|
79
79
|
var EMPTY_STATE = { status: 'unavailable', value: undefined, base: undefined, user: undefined, revision: undefined, writable: false };
|
|
80
80
|
|
|
81
81
|
// English copy. Every label the page renders lives here.
|
|
82
82
|
var en = {
|
|
83
|
-
title: 'Web
|
|
84
|
-
description: '
|
|
83
|
+
title: 'Web Search',
|
|
84
|
+
description: 'Key-free search, page fetch and research: eight engines fused with RRF, 34 dedicated sources, fetch ladder, proxy, cache and cooldown settings.',
|
|
85
85
|
jumpTo: 'Jump to',
|
|
86
86
|
latest: 'Latest on npm',
|
|
87
87
|
upToDate: 'up to date',
|
|
@@ -191,7 +191,7 @@
|
|
|
191
191
|
|
|
192
192
|
// Simplified Chinese copy.
|
|
193
193
|
var zh = {
|
|
194
|
-
title: '
|
|
194
|
+
title: '网页搜索',
|
|
195
195
|
jumpTo: '跳到',
|
|
196
196
|
latest: 'npm 最新版',
|
|
197
197
|
autoUpdateHint: '定时自动更新需要 profile 更新器(dsh-autoupdater);「安装更新」按钮不依赖它:它会请 Host 插件管理器安装 npm 包,装完需重启。link:/file: 安装会被拒绝。',
|
|
@@ -205,7 +205,7 @@
|
|
|
205
205
|
updateUnavailable: '当前部署没有内置插件管理器;请用 profile 的包管理器更新。',
|
|
206
206
|
upToDate: '已是最新',
|
|
207
207
|
updateAvailable: '有新版本',
|
|
208
|
-
description: '
|
|
208
|
+
description: '免密钥的搜索、抓取与研究:8 个引擎 RRF 融合、34 个专用源、抓取梯队、代理、缓存与冷却设置。',
|
|
209
209
|
'group.engines': '引擎与传输',
|
|
210
210
|
'group.ranking': '结果与排序',
|
|
211
211
|
'group.fetch': '抓取',
|
|
@@ -0,0 +1,181 @@
|
|
|
1
|
+
# Architecture
|
|
2
|
+
|
|
3
|
+
How a `web_search` call becomes a list of sources, and which module owns which
|
|
4
|
+
decision. Everything described here exists in `src/*.ts`; the compiled output in
|
|
5
|
+
`lib/` is what `package.json` points at.
|
|
6
|
+
|
|
7
|
+
- [Engines and their quirks](engines.md)
|
|
8
|
+
- [Troubleshooting](troubleshooting.md)
|
|
9
|
+
- [README](../README.md) · [中文说明](../README.zh-CN.md)
|
|
10
|
+
|
|
11
|
+
## Module map
|
|
12
|
+
|
|
13
|
+
| Module | Owns | Exports worth knowing |
|
|
14
|
+
| --- | --- | --- |
|
|
15
|
+
| `src/types.ts` | Every contract: `Source`, `HostSpec`, `EngineAnswer`, `SearchOptions`, `SearchOutcome`, `RankOptions`, `CacheOptions`, `HealthOptions`, `SearchErrorCode` | types only, no runtime code |
|
|
16
|
+
| `src/errors.ts` | The one error class and the `Retry-After` parser | `SearchError`, `parseRetryAfter` |
|
|
17
|
+
| `src/config.ts` | The schemastery schema, the defaults and their normalisation | `Config`, `resolveConfig`, `PluginConfig`, `ResolvedConfig` |
|
|
18
|
+
| `src/index.ts` | The plugin entry: `apply` registers the provider with the `web` seam and owns one cache and one health tracker per mount | `apply`, `name`, `inject`, `PROVIDER_ID` |
|
|
19
|
+
| `src/engines.ts` | The engine registry, the region policy, tier scheduling and the cache/cooldown integration | `HOSTS`, `search` |
|
|
20
|
+
| `src/transport.ts` | One engine request, direct or tunnelled, classified into a `SearchError` | `searchHost` |
|
|
21
|
+
| `src/proxy.ts` | Proxy detection and the hand-written `CONNECT`/TLS transport | `parseProxy`, `proxyFromEnv`, `proxyFromSystem`, `resolveProxy`, `proxyGet`, `readResponse` |
|
|
22
|
+
| `src/parsers.ts` | The five payload parsers plus the URL, entity and echoed-query helpers | `resolveHref`, `parseDdgHtml`, `parseDdgLite`, `parseBing`, `parseSo`, `parseAnySearch`, `echoesQuery`, `text` |
|
|
23
|
+
| `src/rank.ts` | Dedupe, weighted interleave, weight-as-admission RRF (votes over merged position), region merge | `dedupe`, `interleave`, `fuse`, `mergeRegions`, `RRF_CONSTANT`, `DEFAULT_AUTHORITY_DOMAINS` |
|
|
24
|
+
| `src/cache.ts` | TTL + LRU result cache with an injectable clock | `cacheKey`, `SearchCache` |
|
|
25
|
+
| `src/health.ts` | Per-engine cooldown, exponential backoff, `Retry-After` override | `EngineHealth` |
|
|
26
|
+
|
|
27
|
+
Dependency direction is one-way and acyclic: `index → engines → {transport,
|
|
28
|
+
rank, cache, health, parsers, proxy}`, every module → `types`, and `proxy` /
|
|
29
|
+
`transport` / `engines` → `errors`. `parsers`, `rank`, `cache` and `health` never
|
|
30
|
+
touch the network; `proxy` never parses a result page.
|
|
31
|
+
|
|
32
|
+
## Data flow
|
|
33
|
+
|
|
34
|
+
1. **Seam call.** The harness calls the provider registered by `apply`
|
|
35
|
+
(`src/index.ts`) with `(request, signal)`. `request.query` must be a non-empty
|
|
36
|
+
string; `request.maxResults`, when a positive integer, caps the budget.
|
|
37
|
+
2. **Config.** The loader has already validated the config section against
|
|
38
|
+
`Config`; `resolveConfig` turns it into the engine layer's shape. `limit`
|
|
39
|
+
becomes `min(maxResults ?? config.limit, config.limit)`.
|
|
40
|
+
3. **Search.** `search(query, signal, options)` in `src/engines.ts`:
|
|
41
|
+
- resolve the proxy (`useProxy: false` forces direct; otherwise
|
|
42
|
+
`resolveProxy(config.proxy)`);
|
|
43
|
+
- filter `HOSTS` by `hostIds`, and drop `viaProxy` engines when no proxy is
|
|
44
|
+
available;
|
|
45
|
+
- split engines into usable / cooling, but never end up with zero engines —
|
|
46
|
+
if everything is cooling, everything is used;
|
|
47
|
+
- compute the leading region from the query script (`global` for ASCII, `cn`
|
|
48
|
+
for CJK);
|
|
49
|
+
- look up the cache with a key built from the normalised query, the budget,
|
|
50
|
+
the wait policy, the lead region and the sorted engine ids.
|
|
51
|
+
4. **Tier run.** Engines of the leading region first, then the rest, split into
|
|
52
|
+
`primary` and `fallback`. `runTier` asks every engine of a tier concurrently
|
|
53
|
+
through `searchHost` and releases on the first of:
|
|
54
|
+
- every engine settled;
|
|
55
|
+
- every engine of the leading region settled (success *or* failure), so a
|
|
56
|
+
failed engine cannot hold the tier open until the deadline;
|
|
57
|
+
- `available() >= limit` sources from at least `needed` contributors across
|
|
58
|
+
**all** regions (two for `auto`), so one slow lead engine cannot keep a
|
|
59
|
+
complete cross-engine answer waiting;
|
|
60
|
+
- the soft deadline (`auto`: 5 s ASCII / 3 s CJK, `fast`: 3.5 s, `all`: 10 s).
|
|
61
|
+
|
|
62
|
+
Engines still running at release are aborted and reported as
|
|
63
|
+
`aborted once enough results had arrived` or `still running at deadline`.
|
|
64
|
+
5. **Per engine.** `searchHost` (`src/transport.ts`) builds the URL from
|
|
65
|
+
`host.build(query)` and picks a route: a domestic engine goes direct first
|
|
66
|
+
and is retried through the tunnel only if that fails; a `viaProxy` engine
|
|
67
|
+
tunnels immediately. Direct requests use global `fetch` with
|
|
68
|
+
`AbortSignal.timeout` combined with the caller's signal; tunnelled requests
|
|
69
|
+
use `proxyGet`.
|
|
70
|
+
6. **Classify.** A non-2xx status becomes `SEARCH_THROTTLED` (429),
|
|
71
|
+
`SEARCH_BLOCKED` (403 and everything else in the 4xx range),
|
|
72
|
+
`SEARCH_NETWORK` (5xx and transport failures) or `SEARCH_TIMEOUT` (a deadline
|
|
73
|
+
that expired). A JSON API validates itself; every scraped page must pass
|
|
74
|
+
`echoesQuery` or it is `SEARCH_BLOCKED`. A parser that throws is
|
|
75
|
+
`SEARCH_PARSE` and is **not** retryable. Cancellation is rethrown, never
|
|
76
|
+
returned.
|
|
77
|
+
7. **Parse.** `host.parse(payload, url)` returns `Source[]` with absolute
|
|
78
|
+
`http(s)` URLs and no empty fields.
|
|
79
|
+
8. **Health.** Each settled engine updates the tracker: success clears its
|
|
80
|
+
streak, failure may start or extend a cooldown.
|
|
81
|
+
9. **Fuse.** `mergeRegions(answers, lead, limit, rank)` fuses the leading group
|
|
82
|
+
with RRF and fills the budget; the other region is fused separately and only
|
|
83
|
+
appended (deduped against the head) when the head came up short. If the
|
|
84
|
+
primary tier produced nothing, the fallback tier runs and the same merge is
|
|
85
|
+
applied. Weight is an *admission* share, not a score multiplier: each engine's
|
|
86
|
+
list is walked in a weighted round robin to give every candidate a 1-based
|
|
87
|
+
position, and the score is `votes / (rrfConstant + position)`, with `votes`
|
|
88
|
+
counting the distinct engines that returned the source.
|
|
89
|
+
10. **Return.** A non-empty result is cached, then returned as
|
|
90
|
+
`{ sources, failures, proxy }`. `index.ts` re-slices to the budget and sets
|
|
91
|
+
`truncated`, or throws an `Error` that lists every engine's failure reason.
|
|
92
|
+
|
|
93
|
+
## Frames and invariants
|
|
94
|
+
|
|
95
|
+
- **Failures are values.** One dead engine degrades into the next one; only
|
|
96
|
+
cancellation escapes as a rejection.
|
|
97
|
+
- **Parsers never throw on markup drift.** They return `[]`, and the transport
|
|
98
|
+
records a parse failure.
|
|
99
|
+
- **Empty fields are omitted, not emptied.** A source never carries `title: ''`.
|
|
100
|
+
- **Determinism.** Fusion ties resolve on engine index, then rank, then URL, so
|
|
101
|
+
the same answers always produce the same order.
|
|
102
|
+
- **No credentials.** Nothing in the flow reads or sends an API key.
|
|
103
|
+
|
|
104
|
+
## Failure codes
|
|
105
|
+
|
|
106
|
+
| Code | Produced by | Cools the engine down |
|
|
107
|
+
| --- | --- | --- |
|
|
108
|
+
| `SEARCH_THROTTLED` | HTTP 429 | yes |
|
|
109
|
+
| `SEARCH_BLOCKED` | HTTP 403, other 4xx, echoed-query mismatch | yes |
|
|
110
|
+
| `SEARCH_NETWORK` | HTTP 5xx, transport error | yes |
|
|
111
|
+
| `SEARCH_TIMEOUT` | a request that hit its deadline | yes |
|
|
112
|
+
| `SEARCH_PARSE` | a parser that threw | no — retrying markup cannot help |
|
|
113
|
+
| `SEARCH_ABORTED` | the caller cancelled | no |
|
|
114
|
+
|
|
115
|
+
The cooldown length comes from `Retry-After` when the service sent one (delta
|
|
116
|
+
seconds or an HTTP date, read from the final redirect hop), otherwise
|
|
117
|
+
`baseCooldownMs × 2^(failures−1)`, capped at `maxCooldownMs`.
|
|
118
|
+
|
|
119
|
+
## Cache key
|
|
120
|
+
|
|
121
|
+
`cacheKey` builds `lead|wait|limit|sorted engine ids|normalised query`. The query
|
|
122
|
+
is trimmed, lowercased and has internal whitespace collapsed; engine order never
|
|
123
|
+
matters because the ids are sorted. The engine ids are the ones the run would use
|
|
124
|
+
*before* the cooldown filter, so an engine entering or leaving a cooldown does not
|
|
125
|
+
invalidate an entry: otherwise every throttle would turn the next identical query
|
|
126
|
+
into a cache miss. Only successful, non-empty results are stored. A hit returns a copy and skips every request. `ttlMs: 0` disables the
|
|
127
|
+
cache entirely; past `maxEntries`, the least recently used entry is evicted.
|
|
128
|
+
|
|
129
|
+
## Proxy transport
|
|
130
|
+
|
|
131
|
+
`proxyGet` opens an HTTP `CONNECT` tunnel to the target host and port, upgrades
|
|
132
|
+
the socket with TLS when the target is `https:`, writes the request itself
|
|
133
|
+
(`accept-encoding: identity`, `connection: close`) and decodes the response:
|
|
134
|
+
`Transfer-Encoding: chunked`, `Content-Length`, `gzip`, `deflate` and `br`.
|
|
135
|
+
Redirects are followed hop by hop, re-opening the tunnel each time, up to four
|
|
136
|
+
hops. `307`/`308` preserve the method and body; any other `3xx` downgrades a
|
|
137
|
+
`POST` to a bodyless `GET`. The final hop's status, headers (lowercased), body
|
|
138
|
+
and URL are returned, which is how the caller can read `Retry-After`.
|
|
139
|
+
|
|
140
|
+
Proxy selection is explicit config → environment (`DSH_WEB_SEARCH_PROXY`,
|
|
141
|
+
`HTTPS_PROXY`, `ALL_PROXY`, `HTTP_PROXY`, both casings) → the OS system proxy
|
|
142
|
+
(macOS `scutil --proxy`, Windows Internet Settings) gated by a 400 ms TCP probe,
|
|
143
|
+
cached for 60 seconds. Only `http://` proxies are
|
|
144
|
+
dialled from detection, because the tunnel itself is a plain `CONNECT`; an
|
|
145
|
+
`https://` value elsewhere in the environment is skipped rather than selected.
|
|
146
|
+
|
|
147
|
+
## Build and runtime layout
|
|
148
|
+
|
|
149
|
+
- `src/**/*.ts` is the source of truth; `npm run build` compiles it to `lib/`
|
|
150
|
+
(`tsconfig.json`, `rootDir: src`, `outDir: lib`).
|
|
151
|
+
- `package.json` points `main` and `types` at `lib/index.js` /
|
|
152
|
+
`lib/index.d.ts`, so a checkout must be built before it can be mounted.
|
|
153
|
+
- Relative imports carry the `.ts` extension and `rewriteRelativeImportExtensions`
|
|
154
|
+
rewrites them on emit; the source is also runnable directly by Node's type
|
|
155
|
+
stripping, which is how the tests run (`npm test`; it passes a quoted glob so
|
|
156
|
+
cmd.exe works too).
|
|
157
|
+
- `types.ts` is types only, so it emits nothing; every other module emits a
|
|
158
|
+
normal ES module.
|
|
159
|
+
- `index.js`, `engines.js` and `proxy.js` are the pre-rewrite JavaScript
|
|
160
|
+
implementation. They are not imported by `src/` and are not part of the
|
|
161
|
+
TypeScript build.
|
|
162
|
+
|
|
163
|
+
## Route memory and engine health
|
|
164
|
+
|
|
165
|
+
`EngineHealth` does two jobs. It keeps the failure cooldown (an engine that
|
|
166
|
+
returned a retryable block rests for 5–30 minutes) and, since 0.21.0, a
|
|
167
|
+
ten-minute route preference. A direct request that ends in `403`/`429` and then
|
|
168
|
+
succeeds through the tunnel records `proxy` for that engine; the next request
|
|
169
|
+
tries the tunnel first, and the direct route stays as a fallback. A later direct
|
|
170
|
+
success clears the preference. The transport test drives this through a real
|
|
171
|
+
HTTP `CONNECT` proxy rather than mocking `proxyGet`.
|
|
172
|
+
|
|
173
|
+
## MCP runner
|
|
174
|
+
|
|
175
|
+
`mcp/server.mjs` is a dependency-free stdio JSON-RPC process. It imports the
|
|
176
|
+
same `lib/engines.js`, `lib/fetch.js`, `lib/research.js` and
|
|
177
|
+
`lib/verticals.js` as the DSH plugin, exposes four tools (`search`, `fetch`,
|
|
178
|
+
`research`, `domain_search`), wraps every output in the untrusted-content
|
|
179
|
+
boundary and never reads a DSH profile. It is not a second search
|
|
180
|
+
implementation; a ranking or transport fix reaches both runtimes after
|
|
181
|
+
`npm run build`.
|
package/docs/comparison.md
CHANGED
|
@@ -6,6 +6,8 @@ whether a provider answers at all, how long it takes, how many distinct domains
|
|
|
6
6
|
it returns, and whether a page on a known official domain appears in the top 8.
|
|
7
7
|
The failures are part of the result.
|
|
8
8
|
|
|
9
|
+
> **Version note (2026-11-02):** 0.24.0 added Windows system-proxy discovery, and 0.24.1 and 0.24.2 are documentation/metadata patches. These patches do not change the search, ranking or source-adapter path measured below, so the 0.23.0 numbers remain the latest comparison.
|
|
10
|
+
|
|
9
11
|
## Method
|
|
10
12
|
|
|
11
13
|
- **Corpus:** 30 queries in `bench/compare30/queries.json` — 13 release queries
|