@fffangchenghao/dsh-web-search 0.24.2 → 0.24.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,77 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.24.4
4
+
5
+ Second independent audit, this time with `Vendor2/GPT-6-astra` at reasoning
6
+ effort `xhigh`. Search and ranking inputs are unchanged except where stated.
7
+
8
+ ### Fixed
9
+ - **A failed proxy-first route no longer removes the direct fallback.** After a
10
+ tunnel error, `searchHost` now retries the same engine directly instead of
11
+ treating `undefined` as "no alternate route".
12
+ - **Platform searches have one deadline for direct, alternate and fallback.**
13
+ A fallback that ignores its signal can no longer hang the whole `Promise.all`;
14
+ caller cancellation still propagates as `SEARCH_ABORTED`.
15
+ - **Already-cancelled requests do not start network work.** The official
16
+ release/support readers and the proxy transport check the signal before the
17
+ first request; the research tool also refuses to start a search after cancel.
18
+ - **Scoped npm lookups verify the package identity.** The official resolver asks
19
+ for several search rows and accepts only an exact name or `name@version`
20
+ title instead of trusting the first fuzzy hit.
21
+ - **OpenAlex abstracts keep repeated words.** Every inverted-index position is
22
+ rebuilt, not only the first.
23
+ - **arXiv date windows preserve explicit field queries.** `ti:`, `cat:`, boolean
24
+ expressions and quotes are passed through; only the sort parameters are added.
25
+ - **Tracker unwrapping is host/path scoped.** A normal URL containing `uddg=` or
26
+ `u=a1...` is no longer rewritten as if it came from Bing or DuckDuckGo.
27
+ - **Exa same-line `Highlights:` text is kept.** The inline value now becomes the
28
+ snippet instead of being discarded.
29
+ - **The search-cache key is an unambiguous JSON tuple.** Optional field values
30
+ and the query can no longer collide through delimiter prefixes.
31
+ - **Research enforces its 45 s budget across search and fetch**, reports search
32
+ failures in the evidence pack, and hard-caps the selected excerpt.
33
+ - **`read_page` chunk index/count follows surrogate-safe boundaries**, so a
34
+ cut emoji cannot make pagination repeat or skip a chunk.
35
+ - **A cache snapshot that cannot fit an empty envelope is not written**, leaving
36
+ the previous file intact rather than replacing it with over-budget data.
37
+
38
+ **Test suite: 711 tests across 56 files** (710 pass, 1 skipped).
39
+
40
+ ## 0.24.3
41
+
42
+ Audit fixes from an independent Claude Fable 5.1 review (reasoning effort
43
+ `xhigh`). Search endpoints and ranking inputs are unchanged, except where noted.
44
+
45
+ ### Fixed
46
+ - **A fractional engine weight silently dropped rows.** `fuse()` used the raw
47
+ weight as an array index, so a weight such as `2.5` produced non-integer row
48
+ positions and skipped every odd slot. It now uses the same floored weight as
49
+ `interleave()`.
50
+ - **Windows `ProxyEnable` was compared as a string.** `0x00000000` looked
51
+ enabled, so a disabled system proxy could be selected. The DWORD is now parsed
52
+ as a hexadecimal number.
53
+ - **IPv6 proxy targets used bracketed hostnames for SNI and TCP probes.** They
54
+ are now unbracketed before `net.connect`/`tls.connect`; `CONNECT` still writes
55
+ the bracketed authority. `https://` proxy URLs are rejected explicitly instead
56
+ of being handed to a plaintext CONNECT tunnel that cannot speak TLS.
57
+ - **Fetch bodies were unbounded before `maxBytes` was applied.** The direct path
58
+ now reads at most the byte budget from the response stream and cancels the
59
+ rest; the tunnel path caps the raw response and decompressed output. A
60
+ decompression bomb can no longer expand without a ceiling.
61
+ - **The Settings bridge accepted any Host on a loopback socket.** It now requires
62
+ a loopback Host name (DNS-rebinding defense), a JSON content type, and catches
63
+ body-read errors instead of leaving an async handler rejected.
64
+ - **Markdown rendering had three denial-of-service/forgery edges.** Raw-text
65
+ close tags reuse one lowercase copy instead of re-lowercasing the page per
66
+ `<script>`; element nesting is capped at 512 so recursive pruning/rendering
67
+ cannot overflow the stack; and stash placeholders carry a per-render nonce so
68
+ page text cannot forge a code placeholder.
69
+ - **The truncation notice counted outside `fetchMaxLength`.** The notice is now
70
+ part of the character budget, so `fetchMaxLength` is a real upper bound.
71
+ `clamp()` now describes its byte-level cut accurately.
72
+
73
+ **Test suite: 698 tests across 56 files** (697 pass, 1 skipped).
74
+
3
75
  ## 0.24.2
4
76
 
5
77
  Documentation and image patch; no runtime behavior changed.
package/README.md CHANGED
@@ -10,7 +10,7 @@
10
10
 
11
11
  <p align="center">
12
12
  <a href="https://www.npmjs.com/package/@fffangchenghao/dsh-web-search"><img src="https://img.shields.io/npm/v/@fffangchenghao/dsh-web-search.svg" alt="npm version"></a>
13
- <img src="https://img.shields.io/badge/tests-690%20tests-brightgreen" alt="tests">
13
+ <img src="https://img.shields.io/badge/tests-711%20tests-brightgreen" alt="tests">
14
14
  <img src="https://img.shields.io/badge/engines-8%2C%20zero%20API%20key-brightgreen" alt="engines">
15
15
  <img src="https://img.shields.io/badge/node-%3E%3D22.18-blue" alt="node">
16
16
  <img src="https://img.shields.io/badge/dsh-%3E%3D0.2.1--rc.2%20%3C0.3.0-blueviolet" alt="dsh">
@@ -28,7 +28,7 @@
28
28
  - **Measured in public, including the losses.** A 30-query bilingual comparison records availability, latency, unique domains, official-domain presence and every failure against keyless Firecrawl and anonymous Exa. The follow-up documents the Exa architecture difference, the 3 s CJK budget and the Rust official-source lookup. Exa is faster here; this plugin returns more distinct domains. Both test arms were keyless. The method and raw results are in [docs/comparison.md](docs/comparison.md) and a [public gist](https://gist.github.com/fangchenghao/3db9838fa98928435c1c2df38202c373).
29
29
  - **Fetching admits failure.** Binary documents and empty 200 responses are rejected by name (`SEARCH_UNSUPPORTED`, `SEARCH_EMPTY`) instead of being returned as noise; arXiv PDF links are the deliberate exception and are read through the HTML full-text view; the direct → tunnel → render → extract ladder shares one 25 s budget and stops when there is no time left. The budget is per request — there is a regression test for exactly that.
30
30
  - **Windows is a first-class host.** Pure ESM, no native addon or install script; the system proxy is discovered from Windows Internet Settings, and the full suite runs on a `windows-latest` CI runner. See [Install on Windows](#windows).
31
- - **690 tests across 56 files**, a committed `lib/`, declared `engines.dsh` / `engines.node`, and a [SECURITY.md](SECURITY.md) that lists every process, host and file the plugin touches.
31
+ - **711 tests across 56 files**, a committed `lib/`, declared `engines.dsh` / `engines.node`, and a [SECURITY.md](SECURITY.md) that lists every process, host and file the plugin touches.
32
32
  - **Bilingual documentation.** This README and [中文说明](README.zh-CN.md) are maintained side by side.
33
33
 
34
34
  ![Ranking baseline and parameter sweep](https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/bench.png)
@@ -45,7 +45,7 @@ The web-search corner of the DSH plugin ecosystem is crowded. A dated snapshot
45
45
  | Key-free path | 8 engines, all key-free | ~30-engine free/paid chain | 5 defaults (seam, Exa, DDG, Bing, Jina) | Firecrawl + Keenable keyless defaults | 1 anonymous AnySearch API |
46
46
  | Dedicated vertical sources | 34 (8 on by default) | 41 | platform registry, not a vertical catalog | web + X, no vertical catalog | vertical API behind AnySearch |
47
47
  | Model-facing tools | 1 (`web_tools`, 8 actions) + 4 MCP tools | ~10 | 2 (`web_index` / `web_call`, 20 actions) | native search/fetch + skill | 5 |
48
- | Tests | 56 files / 690 cases, locally verified | no test script in `package.json` | 774 unit + 94 bench, self-reported | Vitest configured; count not published | Vitest configured; count not published |
48
+ | Tests | 56 files / 711 cases, locally verified | no test script in `package.json` | 774 unit + 94 bench, self-reported | Vitest configured; count not published | Vitest configured; count not published |
49
49
  | GitHub stars | 0 | 620 | 77 | 836 | 663 |
50
50
  | npm downloads, last month | 1,087 | 52,436 | 6,420 | 43,151 | 9,251 |
51
51
  | npm unpacked size | 1.2 MB | 0.70 MB | 3.05 MB | 0.51 MB | 0.27 MB |
@@ -56,7 +56,7 @@ read is:
56
56
  - **Feature depth:** upper-middle. `dsh-free-search` has more engines and
57
57
  verticals; `dsh-web-search-pro` has a deeper evidence/ranking pipeline and far
58
58
  more platform backends; `modsearch` is the stronger cross-host bridge.
59
- - **Engineering and verification:** near the top. 690 local tests, a replayable
59
+ - **Engineering and verification:** near the top. 711 local tests, a replayable
60
60
  ranking benchmark, declared security boundaries and one runtime dependency
61
61
  put it alongside the best-maintained entries, while remaining much smaller
62
62
  than the Playwright-heavy option.
@@ -65,7 +65,7 @@ read is:
65
65
  common in the other plugins.
66
66
  - **Adoption and maturity:** bottom third of this sample. At the 2026-10-25
67
67
  snapshot the repository was four days old, had no stars and little
68
- independent adoption; the current npm release is 0.24.2.
68
+ independent adoption; the current npm release is 0.24.4.
69
69
  - **Tool-surface economy:** now competitive. 0.20.x exposes one `web_tools`
70
70
  entry with eight actions, close to `dsh-web-search-pro`'s two tools with 20
71
71
  actions; the legacy eight-name surface remains opt-in for old prompts.
package/README.zh-CN.md CHANGED
@@ -9,7 +9,7 @@
9
9
 
10
10
  <p align="center">
11
11
  <a href="https://www.npmjs.com/package/@fffangchenghao/dsh-web-search"><img src="https://img.shields.io/npm/v/@fffangchenghao/dsh-web-search.svg" alt="npm 版本"></a>
12
- <img src="https://img.shields.io/badge/tests-690%20tests-brightgreen" alt="测试">
12
+ <img src="https://img.shields.io/badge/tests-711%20tests-brightgreen" alt="测试">
13
13
  <img src="https://img.shields.io/badge/engines-8%2C%20zero%20API%20key-brightgreen" alt="引擎">
14
14
  <img src="https://img.shields.io/badge/node-%3E%3D22.18-blue" alt="node">
15
15
  <img src="https://img.shields.io/badge/dsh-%3E%3D0.2.1--rc.2%20%3C0.3.0-blueviolet" alt="dsh">
@@ -27,7 +27,7 @@
27
27
  - **公开测量,也公开输的地方。** 30 条中英查询对比记录可用性、延迟、独立域名、官方域名命中与每一条失败;对比对象是免 key Firecrawl 和匿名 Exa。这里 Exa 更快,本插件返回的独立域名更多;两条对比路径都未使用 key。方法和原始结果在 [docs/comparison.md](docs/comparison.md);Exa 架构差异、中文 3 秒预算和 Rust 官方源见 [延迟与官方源](docs/latency-and-sources.md)。
28
28
  - **抓取会承认失败。** 二进制文档和「200 但正文为空」会被明确拒绝(`SEARCH_UNSUPPORTED`、`SEARCH_EMPTY`),而不是当噪音返回;arXiv 的 PDF 链接是有意保留的例外,会改读 HTML 全文;direct → tunnel → render → extract 四级阶梯共享 25 秒预算,预算不足就不再起新的一级。预算是「每次请求」而不是「插件启动时」,这点有回归测试守着。
29
29
  - **Windows 是一等宿主。** 纯 ESM,没有原生扩展或安装脚本;会自动读取 Windows 的 Internet 设置代理,完整测试套件也在 `windows-latest` CI 上跑通。安装见[Windows 小节](#windows)。
30
- - **56 个测试文件、690 个用例**,`lib/` 产物入库,声明了 `engines.dsh` / `engines.node`,[SECURITY.md](SECURITY.md) 逐项列出插件会碰的进程、主机和文件。
30
+ - **56 个测试文件、711 个用例**,`lib/` 产物入库,声明了 `engines.dsh` / `engines.node`,[SECURITY.md](SECURITY.md) 逐项列出插件会碰的进程、主机和文件。
31
31
  - **双语文档。** 本文件和 [English README](README.md) 同步维护。
32
32
 
33
33
  ![排序基线参数扫描](https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/bench.png)
@@ -44,7 +44,7 @@ DSH 插件生态里的「联网搜索」已经很挤。下面是一张 2026-10-2
44
44
  | 免 key 路径 | 8 个引擎,全部免 key | ~30 个免费/付费引擎链 | 默认 5 个(seam、Exa、DDG、Bing、Jina) | Firecrawl + Keenable 免 key 默认 | 1 个 AnySearch 匿名 API |
45
45
  | 专门数据源 | 34 个(默认开 8 个) | 41 个 | 平台注册表,不是垂直源目录 | 网页 + X,无垂直源目录 | AnySearch 的垂直 API |
46
46
  | 面向模型的工具 | 1 个(`web_tools`,8 个 action)+ 4 个 MCP 工具 | ~10 个 | 2 个(`web_index` / `web_call`,20 个动作) | 原生搜索/抓取 + skill | 5 个 |
47
- | 测试 | 56 文件 / 690 用例,本地实测 | `package.json` 无测试脚本 | 774 单元 + 94 bench,自报 | 配了 Vitest,未公布数量 | 配了 Vitest,未公布数量 |
47
+ | 测试 | 56 文件 / 698 用例,本地实测 | `package.json` 无测试脚本 | 774 单元 + 94 bench,自报 | 配了 Vitest,未公布数量 | 配了 Vitest,未公布数量 |
48
48
  | GitHub stars | 0 | 620 | 77 | 836 | 663 |
49
49
  | npm 近一月下载 | 1,087 | 52,436 | 6,420 | 43,151 | 9,251 |
50
50
  | npm 解包大小 | 1.2 MB | 0.70 MB | 3.05 MB | 0.51 MB | 0.27 MB |
@@ -54,11 +54,11 @@ DSH 插件生态里的「联网搜索」已经很挤。下面是一张 2026-10-2
54
54
  - **功能深度:中上。** `dsh-free-search` 引擎和垂直源更多;
55
55
  `dsh-web-search-pro` 的证据/排序管线和平台后端更深;`modsearch` 的跨宿主
56
56
  能力更强。
57
- - **工程质量与可验证性:接近第一梯队。** 690 个本地测试、可重放的排序基准、
57
+ - **工程质量与可验证性:接近第一梯队。** 711 个本地测试、可重放的排序基准、
58
58
  明确的安全边界、只有 1 个运行时依赖,同时体积远小于 Playwright 重型方案。
59
59
  - **免 key + 官方源纠错:最强卖点。** Chrome、PostgreSQL、AI Act 的官方路径
60
60
  和 34 个垂直源,不是其他插件普遍具备的。
61
- - **采用度与成熟度:这个样本里的后段。** 2026-10-25 快照时 GitHub 仓库刚创建 4 天、0 stars,还缺少独立采用;当前 npm 版本是 0.24.2。
61
+ - **采用度与成熟度:这个样本里的后段。** 2026-10-25 快照时 GitHub 仓库刚创建 4 天、0 stars,还缺少独立采用;当前 npm 版本是 0.24.4。
62
62
  - **工具面的经济性:现在有竞争力。** 0.20.x 只暴露一个 `web_tools` + 8 个
63
63
  action,接近 `dsh-web-search-pro` 的 2 工具 + 20 动作;旧的八个工具名保留为
64
64
  可选兼容模式。
package/SECURITY.md CHANGED
@@ -43,16 +43,22 @@ Outbound TCP/HTTPS only, plus one local system query described below.
43
43
  On Windows the system proxy is read locally from `HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings`; that registry read is not network traffic.
44
44
 
45
45
  `web_fetch` and `read_page` reach **any** HTTP(S) URL the model asks for; that is
46
- their purpose. `engine_test` sends its probe query to the engines it tests, just
46
+ their purpose. There is no private-address block: DSH's local agent already has
47
+ the host's permissions, and blocking `localhost` would break reading a local
48
+ development server. A deployment that treats the model as untrusted should put
49
+ the network boundary in the Host, not in this plugin. `engine_test` sends its probe query to the engines it tests, just
47
50
  like a normal search; `advanced_search` is an ordinary search with a per-call
48
51
  date window, except a windowed Chrome security/release query may read the
49
- official Chrome Releases feed above. The Settings page's engine test uses an **inbound**,
50
- loopback-only route on the host's own `webServer` (`POST
51
- /api/dsh-web-search/engine-test`); the plugin never opens a listening socket
52
- itself and the route refuses non-POST, non-loopback and cross-origin requests.
52
+ official Chrome Releases feed above. The Settings page's engine test and **Install update** button use **inbound**
53
+ routes on the host's own `webServer` (`POST /api/dsh-web-search/engine-test`
54
+ and `POST /api/dsh-web-search/update`); the plugin never opens a listening
55
+ socket itself. Both routes accept POST only, require a loopback socket and a
56
+ loopback `Host` (so DNS rebinding cannot reach them), require
57
+ `application/json`, and reject cross-origin callers.
53
58
 
54
59
  With `useProxy` enabled they speak to the configured proxy
55
- (default `127.0.0.1:7890`, auto-detected) using HTTP `CONNECT`; a 400 ms TCP
60
+ (default `127.0.0.1:7890`, auto-detected) using HTTP `CONNECT`; `https://`
61
+ proxy URLs are rejected because the tunnel is plaintext to the proxy. A 400 ms TCP
56
62
  connect first checks that the proxy port is listening so a stale setting cannot
57
63
  stall a search. The plugin never contacts an analytics, telemetry, or update
58
64
  endpoint.
@@ -129,7 +135,7 @@ no cookies. Fetched research bodies exist only in process memory for
129
135
  | one `platform_search` | 40 s tool budget; 8 s per direct platform attempt |
130
136
  | cache | TTL and entry count are bounded and both configurable, `0` disables |
131
137
  | system proxy probe | 2 s, then the child is killed |
132
- | engine test bridge request | loopback only, POST only, 16 KiB JSON body |
138
+ | settings bridge request | loopback socket and Host, POST + JSON only, 16 KiB body |
133
139
  | one engine test | 8 s per engine, three probes in flight, no cache or cooldown writes |
134
140
  | one advanced_search | 30 s tool budget, same engine deadlines as web_search |
135
141
 
package/client.js CHANGED
@@ -74,7 +74,7 @@
74
74
  var ENTRY = 'web-search-bing';
75
75
  var PACKAGE_NAME = 'dsh-web-search-bing';
76
76
  /** Keep in step with package.json; compared with the npm registry. */
77
- var VERSION = '0.24.2';
77
+ var VERSION = '0.24.4';
78
78
  /** Snapshot used while no configuration service is reachable. */
79
79
  var EMPTY_STATE = { status: 'unavailable', value: undefined, base: undefined, user: undefined, revision: undefined, writable: false };
80
80
 
@@ -6,7 +6,7 @@ whether a provider answers at all, how long it takes, how many distinct domains
6
6
  it returns, and whether a page on a known official domain appears in the top 8.
7
7
  The failures are part of the result.
8
8
 
9
- > **Version note (2026-11-02):** 0.24.0 added Windows system-proxy discovery, and 0.24.1 and 0.24.2 are documentation/metadata patches. These patches do not change the search, ranking or source-adapter path measured below, so the 0.23.0 numbers remain the latest comparison.
9
+ > **Version note (2026-11-02):** 0.24.0 added Windows system-proxy discovery, and 0.24.1 and 0.24.2 are documentation/metadata patches; 0.24.3/0.24.4 fix fetch/proxy/markdown/cache reliability without changing the measured search path. These patches do not change the search, ranking or source-adapter path measured below, so the 0.23.0 numbers remain the latest comparison.
10
10
 
11
11
  ## Method
12
12
 
@@ -52,7 +52,7 @@
52
52
  <rect x="70" y="620" width="330" height="76" rx="14" fill="#101d31" stroke="#26364f"/>
53
53
  <text x="92" y="651" class="m" font-size="13">Engines</text><text x="92" y="677" class="t" font-size="20" font-weight="700">8 key-free paths</text>
54
54
  <rect x="420" y="620" width="330" height="76" rx="14" fill="#101d31" stroke="#26364f"/>
55
- <text x="442" y="651" class="m" font-size="13">Tests</text><text x="442" y="677" class="t" font-size="20" font-weight="700">690 local cases</text>
55
+ <text x="442" y="651" class="m" font-size="13">Tests</text><text x="442" y="677" class="t" font-size="20" font-weight="700">711 local cases</text>
56
56
  <rect x="770" y="620" width="360" height="76" rx="14" fill="#101d31" stroke="#26364f"/>
57
57
  <text x="792" y="651" class="m" font-size="13">Cross-agent</text><text x="792" y="677" class="t" font-size="20" font-weight="700">MCP stdio server</text>
58
58
  </g>
@@ -6,7 +6,7 @@ This note answers three questions from the public 30-query comparison:
6
6
  2. Can the AnySearch tail and the Chinese Rust miss be fixed?
7
7
  3. Does the fix require a vector database or a different retrieval model?
8
8
 
9
- > **Version note (2026-11-02):** 0.24.0 added Windows system-proxy discovery, and 0.24.1 and 0.24.2 are documentation/metadata patches. The search path is unchanged, so these 0.23.0 measurements still describe 0.24.x.
9
+ > **Version note (2026-11-02):** 0.24.0 added Windows system-proxy discovery, and 0.24.1 and 0.24.2 are documentation/metadata patches; 0.24.3/0.24.4 fix fetch/proxy/markdown/cache reliability without changing the measured search path. The search path is unchanged, so these 0.23.0 measurements still describe 0.24.x.
10
10
 
11
11
  The short answer: Exa is a different architecture, not a faster RRF loop. This
12
12
  plugin can borrow tail-tolerance and source-specific facts, but copying Exa's
@@ -29,7 +29,7 @@ Only `0.2.1-rc.2` is declared, because that is the version actually exercised.
29
29
  `mktemp -d`; the real `~/.dsh` was never written to. Bundled pnpm 11.28.5.
30
30
  - **Windows CI:** GitHub Actions `windows-latest`, Node 24. Install, typecheck,
31
31
  the full test suite, build and the entry-point import all pass. Run:
32
- https://github.com/fangchenghao/dsh-web-search/actions/runs/41389877558
32
+ https://github.com/fangchenghao/dsh-web-search/actions/runs/44230164601
33
33
  (the matrix also runs on macOS and Ubuntu). This is a CI environment, not a
34
34
  claim about every Windows terminal or proxy client.
35
35
 
package/lib/bridge.js CHANGED
@@ -23,6 +23,30 @@ function hostOf(value) {
23
23
  return undefined;
24
24
  }
25
25
  }
26
+ /** Whether a Host hostname is a loopback literal. */
27
+ function isLoopbackHostname(hostname) {
28
+ const value = hostname.replace(/\.$/u, '');
29
+ return value === 'localhost'
30
+ || value === '::1'
31
+ || value === '[::1]'
32
+ || /^127(?:\.\d{1,3}){3}$/u.test(value);
33
+ }
34
+ /** Host header hostname, without a port; undefined for a malformed header. */
35
+ function hostnameOf(value) {
36
+ if (typeof value !== 'string' || value.length === 0)
37
+ return undefined;
38
+ try {
39
+ return new URL(`http://${value}`).hostname.toLowerCase();
40
+ }
41
+ catch {
42
+ return undefined;
43
+ }
44
+ }
45
+ /** Whether a POST declares the JSON body this bridge parses. */
46
+ function isJsonRequest(request) {
47
+ const value = request.headers['content-type'];
48
+ return typeof value === 'string' && /^application\/(?:[a-z0-9.+-]+\+)?json\s*(?:;|$)/iu.test(value);
49
+ }
26
50
  /**
27
51
  * Refuse cross-site and non-loopback callers.
28
52
  *
@@ -33,6 +57,11 @@ function hostOf(value) {
33
57
  function isAllowed(request) {
34
58
  if (!isLoopback(request.socket.remoteAddress))
35
59
  return false;
60
+ // A loopback socket is not enough: DNS rebinding makes a remote page
61
+ // same-origin while its Host/Origin still name the attacker's domain.
62
+ const hostname = hostnameOf(request.headers.host);
63
+ if (hostname === undefined || !isLoopbackHostname(hostname))
64
+ return false;
36
65
  if (request.headers['sec-fetch-site'] === 'cross-site')
37
66
  return false;
38
67
  const origin = request.headers.origin;
@@ -161,7 +190,18 @@ export function registerUpdateBridge(ctx) {
161
190
  writeJson(response, 403, { ok: false, code: 'forbidden', message: 'loopback settings only' });
162
191
  return;
163
192
  }
164
- const body = await readJson(request);
193
+ if (!isJsonRequest(request)) {
194
+ writeJson(response, 415, { ok: false, code: 'unsupported-media-type', message: 'application/json required' });
195
+ return;
196
+ }
197
+ let body;
198
+ try {
199
+ body = await readJson(request);
200
+ }
201
+ catch {
202
+ writeJson(response, 400, { ok: false, code: 'invalid-json', message: 'could not read the JSON body' });
203
+ return;
204
+ }
165
205
  if (body === undefined || typeof body !== 'object' || body === null || Array.isArray(body)) {
166
206
  writeJson(response, 400, { ok: false, code: 'invalid-json', message: 'body must be a JSON object' });
167
207
  return;
@@ -267,7 +307,18 @@ export function registerEngineTestBridge(ctx, runTest) {
267
307
  writeJson(response, 403, { ok: false, code: 'forbidden', message: 'loopback settings only' });
268
308
  return;
269
309
  }
270
- const body = await readJson(request);
310
+ if (!isJsonRequest(request)) {
311
+ writeJson(response, 415, { ok: false, code: 'unsupported-media-type', message: 'application/json required' });
312
+ return;
313
+ }
314
+ let body;
315
+ try {
316
+ body = await readJson(request);
317
+ }
318
+ catch {
319
+ writeJson(response, 400, { ok: false, code: 'invalid-json', message: 'could not read the JSON body' });
320
+ return;
321
+ }
271
322
  if (body === undefined) {
272
323
  writeJson(response, 400, { ok: false, code: 'invalid-json', message: 'malformed or oversized JSON body' });
273
324
  return;
@@ -228,7 +228,15 @@ let tempCounter = 0;
228
228
  */
229
229
  export function writeCacheFile(file, entries, now, options = {}) {
230
230
  const io = options.io ?? nodeCacheFileIo;
231
- const { data } = serializeCache(entries, now, options.maxBytes);
231
+ const snapshot = serializeCache(entries, now, options.maxBytes);
232
+ const budget = typeof options.maxBytes === 'number' && Number.isFinite(options.maxBytes) && options.maxBytes > 0
233
+ ? Math.floor(options.maxBytes)
234
+ : DEFAULT_CACHE_FILE_MAX_BYTES;
235
+ // The empty envelope itself can exceed a tiny custom budget. Refuse the
236
+ // write rather than replacing a valid old snapshot with an over-budget one.
237
+ if (snapshot.bytes > budget)
238
+ return false;
239
+ const { data } = snapshot;
232
240
  tempCounter += 1;
233
241
  const temp = `${file}.${process.pid}.${tempCounter}.tmp`;
234
242
  try {
package/lib/cache.js CHANGED
@@ -23,12 +23,21 @@ const DEFAULT_PERSIST_THROTTLE_MS = 2_000;
23
23
  */
24
24
  export function cacheKey(parts) {
25
25
  const query = parts.query.trim().toLowerCase().replace(/\s+/gu, ' ');
26
- const engines = [...parts.engines].sort().join('+');
27
- const window = parts.timeRange === undefined || parts.timeRange === 'off' ? '' : `${parts.timeRange}|`;
28
- const safe = parts.safeSearch === undefined || parts.safeSearch === 'off' ? '' : `safe=${parts.safeSearch}|`;
29
- const fallback = parts.fallbackOn === undefined ? '' : `fallback=${[...parts.fallbackOn].sort().join(',')}|`;
30
- const blocked = parts.blockedDomains === undefined ? '' : `blocked=${[...parts.blockedDomains].sort().join(',')}|`;
31
- return `${parts.lead}|${parts.wait}|${parts.limit}|${engines}|${window}${safe}${fallback}${blocked}${query}`;
26
+ // A JSON tuple is unambiguous and keeps every optional value in its own
27
+ // slot. The old delimiter join let `safeSearch: 'off'` + a query containing
28
+ // `safe=strict|` collide with `safeSearch: 'strict'` + a shorter query.
29
+ return JSON.stringify([
30
+ 'v2',
31
+ parts.lead,
32
+ parts.wait,
33
+ parts.limit,
34
+ [...parts.engines].sort(),
35
+ parts.timeRange === undefined || parts.timeRange === 'off' ? null : parts.timeRange,
36
+ parts.safeSearch === undefined || parts.safeSearch === 'off' ? null : parts.safeSearch,
37
+ parts.fallbackOn === undefined ? null : [...parts.fallbackOn].sort(),
38
+ parts.blockedDomains === undefined || parts.blockedDomains.length === 0 ? null : [...parts.blockedDomains].sort(),
39
+ query,
40
+ ]);
32
41
  }
33
42
  /**
34
43
  * Copy the array and each source.
package/lib/fetch.js CHANGED
@@ -74,15 +74,67 @@ function mimeOf(contentType) {
74
74
  return (contentType ?? '').split(';')[0]?.trim() ?? '';
75
75
  }
76
76
  /**
77
- * Cut a body at the byte budget without splitting a surrogate pair.
77
+ * Cut a decoded body at the byte budget.
78
+ *
79
+ * The cut can land inside a multi-byte code point; `TextDecoder` replaces that
80
+ * final partial sequence rather than leaving a lone surrogate. The budget is a
81
+ * byte budget, not a character budget, so the result may be shorter.
82
+ *
78
83
  * @param text - the decoded body.
79
84
  * @param maxBytes - the byte budget.
80
85
  * @returns the body and whether it was cut.
81
86
  */
82
87
  function clamp(text, maxBytes) {
83
- if (Buffer.byteLength(text, 'utf8') <= maxBytes)
88
+ const bytes = Buffer.from(text, 'utf8');
89
+ if (bytes.length <= maxBytes)
84
90
  return { content: text, truncated: false };
85
- return { content: Buffer.from(text, 'utf8').subarray(0, maxBytes).toString('utf8'), truncated: true };
91
+ const content = new TextDecoder('utf-8').decode(bytes.subarray(0, Math.max(0, maxBytes))).replace(/\uFFFD+$/u, '');
92
+ return { content, truncated: true };
93
+ }
94
+ /**
95
+ * Read a fetch body without buffering more than the byte budget.
96
+ *
97
+ * `response.text()` materialises the whole body first, so a 1 MiB clamp after
98
+ * it has already downloaded 500 MiB. This reader stops at the budget and
99
+ * cancels the stream.
100
+ *
101
+ * @param response - the fetch response.
102
+ * @param maxBytes - maximum UTF-8 bytes to retain.
103
+ * @returns decoded text and whether the body was cut.
104
+ */
105
+ async function readTextBounded(response, maxBytes) {
106
+ if (response.body === null)
107
+ return { content: '', truncated: false };
108
+ const reader = response.body.getReader();
109
+ const chunks = [];
110
+ let total = 0;
111
+ let truncated = false;
112
+ try {
113
+ for (;;) {
114
+ const { done, value } = await reader.read();
115
+ if (done)
116
+ break;
117
+ if (value === undefined)
118
+ continue;
119
+ const left = maxBytes - total;
120
+ if (value.byteLength > left) {
121
+ if (left > 0)
122
+ chunks.push(value.subarray(0, left));
123
+ total += Math.max(0, left);
124
+ truncated = true;
125
+ await reader.cancel();
126
+ break;
127
+ }
128
+ chunks.push(value);
129
+ total += value.byteLength;
130
+ }
131
+ }
132
+ finally {
133
+ reader.releaseLock();
134
+ }
135
+ const bytes = Buffer.concat(chunks.map((chunk) => Buffer.from(chunk)));
136
+ const content = new TextDecoder('utf-8').decode(bytes).replace(/\uFFFD+$/u, '');
137
+ return { content, truncated };
86
138
  }
87
139
  /**
88
140
  * Combine a caller signal with the per-attempt deadline.
@@ -165,15 +217,14 @@ export async function directFetch(url, timeoutMs, signal, userAgent, maxBytes) {
165
217
  let target = url;
166
218
  let last;
167
219
  for (let hop = 0; hop <= REDIRECT_HOPS; hop += 1) {
168
- const left = Math.max(500, timeoutMs - (Date.now() - startedAt));
220
+ const left = Math.max(1, timeoutMs - (Date.now() - startedAt));
169
221
  const response = await fetch(target, {
170
222
  signal: deadline(left, signal),
171
223
  redirect: 'follow',
172
224
  headers: { 'user-agent': userAgent, accept: 'text/html,application/xhtml+xml,application/json;q=0.9,*/*;q=0.8' },
173
225
  });
174
226
  const contentType = response.headers.get('content-type');
175
- const text = await response.text();
176
- const { content, truncated } = clamp(text, maxBytes);
227
+ const { content, truncated } = await readTextBounded(response, maxBytes);
177
228
  last = {
178
229
  url: response.url === '' ? target : response.url,
179
230
  statusCode: response.status,
@@ -196,11 +247,12 @@ async function tunnelFetch(url, proxy, timeoutMs, signal, userAgent, maxBytes) {
196
247
  let target = url;
197
248
  let last;
198
249
  for (let hop = 0; hop <= REDIRECT_HOPS; hop += 1) {
199
- const left = Math.max(500, timeoutMs - (Date.now() - startedAt));
250
+ const left = Math.max(1, timeoutMs - (Date.now() - startedAt));
200
251
  const response = await proxyGet(target, {
201
252
  proxy,
202
253
  headers: { 'user-agent': userAgent, accept: 'text/html,application/xhtml+xml,application/json;q=0.9,*/*;q=0.8' },
203
254
  timeoutMs: left,
255
+ maxBytes: Math.max(1, maxBytes),
204
256
  ...(signal === undefined ? {} : { signal }),
205
257
  });
206
258
  const { content, truncated } = clamp(response.text, maxBytes);
package/lib/index.js CHANGED
@@ -266,7 +266,7 @@ export function apply(ctx, config = {}) {
266
266
  timeoutMs: Math.min(live.timeoutMs, 2_500),
267
267
  ...(live.proxy === undefined ? {} : { proxy: live.proxy }),
268
268
  ...(signal === undefined ? {} : { signal }),
269
- });
269
+ })?.catch(() => undefined);
270
270
  // An explicit `arXiv` query gets the preprint index in parallel with the
271
271
  // ordinary fan-out. This is separate from the official-release lookup:
272
272
  // a survey query needs arXiv's own search, not a release announcement.
@@ -275,7 +275,7 @@ export function apply(ctx, config = {}) {
275
275
  timeoutMs: Math.min(live.timeoutMs, 3_500),
276
276
  ...(live.proxy === undefined ? {} : { proxy: live.proxy }),
277
277
  ...(signal === undefined ? {} : { signal }),
278
- });
278
+ }).catch(() => []);
279
279
  const outcome = await runQuery(query, asked, signal);
280
280
  // A version/EOL question about PostgreSQL or Rust is better answered by
281
281
  // the project's own release facts than by whatever an engine ranks first.
package/lib/markdown.js CHANGED
@@ -20,6 +20,14 @@
20
20
  */
21
21
  /** Default `fetchMaxLength`: about 20k characters of prose. */
22
22
  export const DEFAULT_FETCH_MAX_LENGTH = 20_000;
23
+ /**
24
+ * Maximum element nesting kept by the parser.
25
+ *
26
+ * The tree builder itself is iterative, but pruning and rendering recurse. A
27
+ * hostile page can nest thousands of `<div>`s; refusing to grow past this depth
28
+ * keeps every later pass within the JavaScript stack.
29
+ */
30
+ const MAX_DOM_DEPTH = 512;
23
31
  /** Named entities that appear in real prose; unknown names are left alone. */
24
32
  const ENTITIES = {
25
33
  amp: '&',
@@ -188,6 +196,12 @@ function parseAttributes(raw) {
188
196
  function parseHtml(html) {
189
197
  const root = element('#root', {});
190
198
  const stack = [root];
199
+ /** Opening tags dropped at MAX_DOM_DEPTH, so their closing tags can be ignored too. */
200
+ let flattened = 0;
201
+ // `toLowerCase()` would change indices for a few non-ASCII strings. Keep a
202
+ // reusable lowercase copy only when its length still maps 1:1 to the input.
203
+ const lowered = html.toLowerCase();
204
+ const lower = lowered.length === html.length ? lowered : undefined;
191
205
  const pushText = (value) => {
192
206
  if (value.length === 0)
193
207
  return;
@@ -214,6 +228,10 @@ function parseHtml(html) {
214
228
  }
215
229
  };
216
230
  const closeTag = (name) => {
231
+ if (flattened > 0) {
232
+ flattened -= 1;
233
+ return;
234
+ }
217
235
  for (let index = stack.length - 1; index >= 1; index -= 1) {
218
236
  const open = stack[index];
219
237
  if (open === undefined)
@@ -230,6 +248,13 @@ function parseHtml(html) {
230
248
  const top = stack[stack.length - 1];
231
249
  if (top === undefined)
232
250
  return;
251
+ // Deeper markup is flattened into the current element. Count the dropped
252
+ // opener so its closing tag does not pop one of the real ancestors.
253
+ if (stack.length >= MAX_DOM_DEPTH) {
254
+ if (!selfClosing && !VOID_TAGS.has(name))
255
+ flattened += 1;
256
+ return;
257
+ }
233
258
  if (top.tag === 'p' && BLOCK_TAGS.has(name))
234
259
  stack.pop();
235
260
  if (name === 'li')
@@ -306,7 +331,7 @@ function parseHtml(html) {
306
331
  const name = (match[0] ?? '').toLowerCase();
307
332
  index = end + 1;
308
333
  if (RAW_TEXT_TAGS.has(name)) {
309
- const closeAt = html.toLowerCase().indexOf(`</${name}`, index);
334
+ const closeAt = (lower ?? html.toLowerCase()).indexOf(`</${name}`, index);
310
335
  if (closeAt === -1) {
311
336
  index = html.length;
312
337
  continue;
@@ -515,13 +540,13 @@ function pickMain(root) {
515
540
  /** Replace a code fragment with a placeholder so clean-up cannot touch it. */
516
541
  function stashCode(ctx, text) {
517
542
  ctx.stash.push(text);
518
- return `\uE000${ctx.stash.length - 1}\uE001`;
543
+ return `${ctx.stashToken}${ctx.stash.length - 1}\uE001`;
519
544
  }
520
545
  /** Put stashed code fragments back after whitespace clean-up. */
521
546
  function restoreCode(text, ctx) {
522
547
  let out = text;
523
548
  for (let index = 0; index < ctx.stash.length; index += 1) {
524
- out = out.split(`\uE000${index}\uE001`).join(ctx.stash[index] ?? '');
549
+ out = out.split(`${ctx.stashToken}${index}\uE001`).join(ctx.stash[index] ?? '');
525
550
  }
526
551
  return out;
527
552
  }
@@ -877,6 +902,7 @@ export function htmlToMarkdown(html, options = {}) {
877
902
  format: options.format ?? 'markdown',
878
903
  baseUrl: options.baseUrl,
879
904
  stash: [],
905
+ stashToken: `\uE000${Math.random().toString(36).slice(2, 10)}:`,
880
906
  };
881
907
  return restoreCode(cleanup(renderElement(node, ctx)), ctx).trim();
882
908
  };
@@ -905,13 +931,17 @@ export function truncateFetchText(content, maxLength) {
905
931
  const limit = Number.isFinite(maxLength) && maxLength >= 1 ? Math.floor(maxLength) : DEFAULT_FETCH_MAX_LENGTH;
906
932
  if (content.length <= limit)
907
933
  return { content, truncated: false };
908
- let cut = content.slice(0, limit);
934
+ const fullNotice = `\n\n[truncated at ${limit} chars; original length ${content.length} chars; the rest is not shown]`;
935
+ // The notice is part of the configured budget, not an allowance on top of
936
+ // it. For a tiny budget, shorten the notice rather than exceeding the limit.
937
+ const notice = fullNotice.length >= limit ? fullNotice.slice(0, Math.max(0, limit - 1)) : fullNotice;
938
+ const budget = Math.max(1, limit - notice.length);
939
+ let cut = content.slice(0, budget);
909
940
  const last = cut.charCodeAt(cut.length - 1);
910
941
  // Never leave half of a surrogate pair at the cut point.
911
942
  if (last >= 0xd800 && last <= 0xdbff)
912
943
  cut = cut.slice(0, -1);
913
944
  cut = cut.replace(/\s+$/, '');
914
- const notice = `\n\n[truncated at ${limit} chars; original length ${content.length} chars; the rest is not shown]`;
915
945
  return { content: `${cut}${notice}`, truncated: true };
916
946
  }
917
947
  /**
@@ -94,6 +94,10 @@ export function parseRustRelease(payload, requested) {
94
94
  }
95
95
  /** One direct GET with its own abort budget. */
96
96
  async function directJson(url, headers, options, timeoutMs) {
97
+ if (options.signal?.aborted === true) {
98
+ const reason = options.signal.reason;
99
+ throw reason instanceof Error ? reason : new DOMException('The operation was aborted', 'AbortError');
100
+ }
97
101
  const controller = new AbortController();
98
102
  const timer = setTimeout(() => controller.abort(), timeoutMs);
99
103
  const abort = () => controller.abort();
@@ -132,6 +136,8 @@ async function fetchJson(url, options, timeoutMs) {
132
136
  return await directJson(url, headers, options, Math.min(timeoutMs, 1_500));
133
137
  }
134
138
  catch (error) {
139
+ if (options.signal?.aborted === true)
140
+ throw error;
135
141
  if (options.proxy === undefined || options.proxy.length === 0)
136
142
  throw error;
137
143
  const remaining = timeoutMs - (Date.now() - started);