@fffangchenghao/dsh-web-search 0.19.1 → 0.19.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,27 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.19.3
4
+
5
+ ### Fixed
6
+ - **Firecrawl works again.** It was marked `viaProxy`, and the keyless endpoint
7
+ rejects the tunnel's exit IP (`HTTP 403: Unfortunately, your IP address looks
8
+ like a VPN or proxy`). A direct request still answers, so Firecrawl is now
9
+ direct-first; the transport keeps its tunnel retry for networks where direct
10
+ is blocked. Verified live: `hostIds: ['firecrawl']` returns Kubernetes
11
+ Service rows again. One regression test pins the direct-first route.
12
+
13
+ ## 0.19.2
14
+
15
+ ### Fixed
16
+ - **`web_fetch` no longer returns only a page's footer.** The reading-root picker
17
+ preferred the densest container, and a short comment form can be far denser
18
+ than an article wrapped in `<div>` markup. On PostgreSQL's documentation that
19
+ returned 307 characters (`Submit correction…`) instead of the chapter. The
20
+ picker now falls back to the largest container that holds a real share of the
21
+ page text, and then to the body, before accepting a fragment. The same URL now
22
+ returns the chapter introduction and body (6,078 characters in the text
23
+ format); GitHub and PyTorch pages keep their content. One regression test.
24
+
3
25
  ## 0.19.1
4
26
 
5
27
  ### Changed
package/README.md CHANGED
@@ -9,7 +9,7 @@
9
9
 
10
10
  <p align="center">
11
11
  <a href="https://www.npmjs.com/package/@fffangchenghao/dsh-web-search"><img src="https://img.shields.io/npm/v/@fffangchenghao/dsh-web-search.svg" alt="npm version"></a>
12
- <img src="https://img.shields.io/badge/tests-498%20passing-brightgreen" alt="tests">
12
+ <img src="https://img.shields.io/badge/tests-500%20passing-brightgreen" alt="tests">
13
13
  <img src="https://img.shields.io/badge/engines-8%2C%20zero%20API%20key-brightgreen" alt="engines">
14
14
  <img src="https://img.shields.io/badge/node-%3E%3D22.18-blue" alt="node">
15
15
  <img src="https://img.shields.io/badge/dsh-%3E%3D0.2.1--rc.2%20%3C0.3.0-blueviolet" alt="dsh">
@@ -21,7 +21,7 @@
21
21
  - **No API key, eight engines.** AnySearch, CN Bing, DuckDuckGo HTML, www Bing, DuckDuckGo Lite and 360 Search are scraped directly; Exa joins the primary tier through its keyless MCP endpoint, and Firecrawl's search API is the measured fallback. The plugin works out of the box.
22
22
  - **The ranking is measured, not asserted.** Cross-engine reciprocal-rank fusion with authority and freshness boosts, a 20-query offline benchmark, and an 81-configuration parameter sweep — the only search plugin here that publishes both its metric and its confidence intervals. See [How the ranking works](docs/rrf-ranking.md).
23
23
  - **Fetching admits failure.** Binary documents and empty 200 responses are rejected by name (`SEARCH_UNSUPPORTED`, `SEARCH_EMPTY`) instead of being returned as noise; the direct → tunnel → render → extract ladder shares one 25 s budget and stops when there is no time left. The budget is per request — there is a regression test for exactly that.
24
- - **499 tests across 34 files**, a committed `lib/`, declared `engines.dsh` / `engines.node`, and a [SECURITY.md](SECURITY.md) that lists every process, host and file the plugin touches.
24
+ - **501 tests across 34 files**, a committed `lib/`, declared `engines.dsh` / `engines.node`, and a [SECURITY.md](SECURITY.md) that lists every process, host and file the plugin touches.
25
25
  - **Bilingual documentation.** This README and [中文说明](README.zh-CN.md) are maintained side by side.
26
26
 
27
27
  ![Ranking baseline and parameter sweep](https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/bench.png)
@@ -37,7 +37,7 @@ The web-search corner of the DSH plugin ecosystem is crowded, and most rivals ar
37
37
  | Key-free engines | 8 | ~30 | 7 channels (+X) | 1 official API |
38
38
  | Dedicated vertical sources | 34 (8 on by default) | 41 | — | — |
39
39
  | Model-facing tools | 6 | ~10 | 3 | 5 |
40
- | Test suite | 34 files / 499 cases | none | 29 files | 6 files |
40
+ | Test suite | 34 files / 501 cases | none | 29 files | 6 files |
41
41
  | Ranking benchmark | 20 queries + 81-config sweep | — | — | — |
42
42
  | Store status | source contract satisfied, awaiting review | unlisted | no entry | blocked (review-required) |
43
43
 
@@ -226,7 +226,7 @@ Each line is the plain version first, the technical note second. The technical n
226
226
 
227
227
  1. **Ask several search engines at once.**
228
228
  Four primaries run together (AnySearch, DuckDuckGo HTML, Bing China, Exa). If none of them returns anything, Bing International, 360, DuckDuckGo Lite and Firecrawl step in. Eight sources in total.
229
- *Technical: `tier: primary` and `tier: fallback` in `HOSTS`; 360 is a fallback because of its verification wall; the three `viaProxy` engines (`ddg-html`, `www.bing`, `ddg-lite`, `exa`, `firecrawl`) only join when a proxy is detected.*
229
+ *Technical: `tier: primary` and `tier: fallback` in `HOSTS`; 360 is a fallback because of its verification wall; the `viaProxy` engines (`ddg-html`, `www.bing`, `ddg-lite`, `exa`) only join when a proxy is detected; Firecrawl stays direct-first because its keyless endpoint rejects the tunnel's exit IP as a VPN.*
230
230
 
231
231
  2. **No API key at all.**
232
232
  AnySearch answers anonymously; the domestic engines and DuckDuckGo are scraped from public result pages. No sign-up needed.
package/README.zh-CN.md CHANGED
@@ -9,7 +9,7 @@
9
9
 
10
10
  <p align="center">
11
11
  <a href="https://www.npmjs.com/package/@fffangchenghao/dsh-web-search"><img src="https://img.shields.io/npm/v/@fffangchenghao/dsh-web-search.svg" alt="npm 版本"></a>
12
- <img src="https://img.shields.io/badge/tests-498%20passing-brightgreen" alt="测试">
12
+ <img src="https://img.shields.io/badge/tests-500%20passing-brightgreen" alt="测试">
13
13
  <img src="https://img.shields.io/badge/engines-8%2C%20zero%20API%20key-brightgreen" alt="引擎">
14
14
  <img src="https://img.shields.io/badge/node-%3E%3D22.18-blue" alt="node">
15
15
  <img src="https://img.shields.io/badge/dsh-%3E%3D0.2.1--rc.2%20%3C0.3.0-blueviolet" alt="dsh">
@@ -21,7 +21,7 @@
21
21
  - **不要 API key,八个引擎。** AnySearch、必应中国、DuckDuckGo HTML、必应国际、DuckDuckGo Lite、360 搜索直接抓取;Exa 走它的免密钥 MCP 通道加入主引擎,Firecrawl 的搜索 API 作为实测后的兜底。装好就能用。
22
22
  - **排序是量出来的,不是写出来的。** 跨引擎 RRF 融合,叠加权威度与时效加权;20 条离线基准 + 81 组参数扫描,是这类插件里唯一同时公开指标和置信区间的。方法见 [排序是怎么做的](docs/rrf-ranking.md)。
23
23
  - **抓取会承认失败。** 二进制文档和「200 但正文为空」会被明确拒绝(`SEARCH_UNSUPPORTED`、`SEARCH_EMPTY`),而不是当噪音返回;direct → tunnel → render → extract 四级阶梯共享 25 秒预算,预算不足就不再起新的一级。预算是「每次请求」而不是「插件启动时」,这点有回归测试守着。
24
- - **34 个测试文件、499 个用例**,`lib/` 产物入库,声明了 `engines.dsh` / `engines.node`,[SECURITY.md](SECURITY.md) 逐项列出插件会碰的进程、主机和文件。
24
+ - **34 个测试文件、501 个用例**,`lib/` 产物入库,声明了 `engines.dsh` / `engines.node`,[SECURITY.md](SECURITY.md) 逐项列出插件会碰的进程、主机和文件。
25
25
  - **双语文档。** 本文件和 [English README](README.md) 同步维护。
26
26
 
27
27
  ![排序基线参数扫描](https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/bench.png)
@@ -37,7 +37,7 @@ DSH 插件生态里的「联网搜索」这一块已经很挤,大多数竞品
37
37
  | 免 key 引擎 | 8 | ~30 | 7 条通道(含 X) | 1 个官方 API |
38
38
  | 专门数据源 | 34(默认开 8 个)| 41 | — | — |
39
39
  | 面向模型的工具 | 6 | ~10 | 3 | 5 |
40
- | 测试 | 34 文件 / 499 用例 | 无 | 29 文件 | 6 文件 |
40
+ | 测试 | 34 文件 / 501 用例 | 无 | 29 文件 | 6 文件 |
41
41
  | 排序基准 | 20 查询 + 81 组扫描 | — | — | — |
42
42
  | 商店状态 | 源码契约已满足,待复核 | unlisted | 无条目 | blocked(review-required) |
43
43
 
package/client.js CHANGED
@@ -47,7 +47,7 @@
47
47
  var ENTRY = 'web-search-bing';
48
48
  var PACKAGE_NAME = 'dsh-web-search-bing';
49
49
  /** Keep in step with package.json; compared with the npm registry. */
50
- var VERSION = '0.19.1';
50
+ var VERSION = '0.19.3';
51
51
  /** Snapshot used while no configuration service is reachable. */
52
52
  var EMPTY_STATE = { status: 'unavailable', value: undefined, base: undefined, user: undefined, revision: undefined, writable: false };
53
53
 
package/lib/engines.js CHANGED
@@ -78,7 +78,10 @@ export const HOSTS = [
78
78
  tier: 'fallback',
79
79
  region: 'both',
80
80
  weight: 1,
81
- viaProxy: true,
81
+ // Direct first: the tunnel's exit IP is rejected as a VPN by the keyless
82
+ // endpoint (`HTTP 403 … looks like a VPN or proxy`), while a direct request
83
+ // still answers. The transport still retries through the tunnel when the
84
+ // direct attempt fails, so deployments that need the tunnel lose nothing.
82
85
  method: 'POST',
83
86
  skipEcho: true,
84
87
  build: () => 'https://api.firecrawl.dev/v1/search',
package/lib/markdown.js CHANGED
@@ -423,6 +423,21 @@ function densestContainer(body) {
423
423
  });
424
424
  return best;
425
425
  }
426
+ /** Largest container by visible text, used when density picks a stub. */
427
+ function largestContainer(body) {
428
+ let best = body;
429
+ let bestCharacters = characterCount(body);
430
+ walkElements(body, (node) => {
431
+ if (!CONTAINER_TAGS.has(node.tag))
432
+ return;
433
+ const characters = characterCount(node);
434
+ if (characters > bestCharacters) {
435
+ best = node;
436
+ bestCharacters = characters;
437
+ }
438
+ });
439
+ return best;
440
+ }
426
441
  /** Pick the reading root: article, main, then the densest container. */
427
442
  function pickMain(root) {
428
443
  const body = findElement(root, (node) => node.tag === 'body') ?? root;
@@ -452,10 +467,17 @@ function pickMain(root) {
452
467
  }
453
468
  if (characterCount(densest) >= floor)
454
469
  return densest;
455
- // Nothing reaches the floor: return whichever of the three holds the most.
456
- const ranked = [article, main, densest].filter((node) => node !== undefined);
470
+ // Density alone can crown a short, dense stub — a comment form or a footer —
471
+ // while the prose sits in a container with more markup and fewer characters
472
+ // per tag. Prefer a container that still holds a real share of the page text,
473
+ // then fall back to the body when every candidate is a fragment.
474
+ const largest = largestContainer(body);
475
+ if (characterCount(largest) >= floor)
476
+ return largest;
477
+ const ranked = [article, main, densest, largest].filter((node) => node !== undefined);
457
478
  ranked.sort((left, right) => characterCount(right) - characterCount(left));
458
- return ranked[0] ?? body;
479
+ const best = ranked[0] ?? body;
480
+ return characterCount(best) * 2 < characterCount(body) ? body : best;
459
481
  }
460
482
  /** Replace a code fragment with a placeholder so clean-up cannot touch it. */
461
483
  function stashCode(ctx, text) {
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@fffangchenghao/dsh-web-search",
3
- "version": "0.19.1",
3
+ "version": "0.19.3",
4
4
  "description": "DSH bundle: a key-free web search provider. Leads with the anonymous AnySearch API and falls back to public result pages (DuckDuckGo, Bing, 360), merged by reciprocal-rank fusion with a bilingual region policy, an engine cooldown and a TTL cache.",
5
5
  "type": "module",
6
6
  "main": "lib/index.js",