@fffangchenghao/dsh-web-search 0.24.3 → 0.24.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,42 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.24.4
4
+
5
+ Second independent audit, this time with `Vendor2/GPT-6-astra` at reasoning
6
+ effort `xhigh`. Search and ranking inputs are unchanged except where stated.
7
+
8
+ ### Fixed
9
+ - **A failed proxy-first route no longer removes the direct fallback.** After a
10
+ tunnel error, `searchHost` now retries the same engine directly instead of
11
+ treating `undefined` as "no alternate route".
12
+ - **Platform searches have one deadline for direct, alternate and fallback.**
13
+ A fallback that ignores its signal can no longer hang the whole `Promise.all`;
14
+ caller cancellation still propagates as `SEARCH_ABORTED`.
15
+ - **Already-cancelled requests do not start network work.** The official
16
+ release/support readers and the proxy transport check the signal before the
17
+ first request; the research tool also refuses to start a search after cancel.
18
+ - **Scoped npm lookups verify the package identity.** The official resolver asks
19
+ for several search rows and accepts only an exact name or `name@version`
20
+ title instead of trusting the first fuzzy hit.
21
+ - **OpenAlex abstracts keep repeated words.** Every inverted-index position is
22
+ rebuilt, not only the first.
23
+ - **arXiv date windows preserve explicit field queries.** `ti:`, `cat:`, boolean
24
+ expressions and quotes are passed through; only the sort parameters are added.
25
+ - **Tracker unwrapping is host/path scoped.** A normal URL containing `uddg=` or
26
+ `u=a1...` is no longer rewritten as if it came from Bing or DuckDuckGo.
27
+ - **Exa same-line `Highlights:` text is kept.** The inline value now becomes the
28
+ snippet instead of being discarded.
29
+ - **The search-cache key is an unambiguous JSON tuple.** Optional field values
30
+ and the query can no longer collide through delimiter prefixes.
31
+ - **Research enforces its 45 s budget across search and fetch**, reports search
32
+ failures in the evidence pack, and hard-caps the selected excerpt.
33
+ - **`read_page` chunk index/count follows surrogate-safe boundaries**, so a
34
+ cut emoji cannot make pagination repeat or skip a chunk.
35
+ - **A cache snapshot that cannot fit an empty envelope is not written**, leaving
36
+ the previous file intact rather than replacing it with over-budget data.
37
+
38
+ **Test suite: 711 tests across 56 files** (710 pass, 1 skipped).
39
+
3
40
  ## 0.24.3
4
41
 
5
42
  Audit fixes from an independent Claude Fable 5.1 review (reasoning effort
package/README.md CHANGED
@@ -10,7 +10,7 @@
10
10
 
11
11
  <p align="center">
12
12
  <a href="https://www.npmjs.com/package/@fffangchenghao/dsh-web-search"><img src="https://img.shields.io/npm/v/@fffangchenghao/dsh-web-search.svg" alt="npm version"></a>
13
- <img src="https://img.shields.io/badge/tests-698%20tests-brightgreen" alt="tests">
13
+ <img src="https://img.shields.io/badge/tests-711%20tests-brightgreen" alt="tests">
14
14
  <img src="https://img.shields.io/badge/engines-8%2C%20zero%20API%20key-brightgreen" alt="engines">
15
15
  <img src="https://img.shields.io/badge/node-%3E%3D22.18-blue" alt="node">
16
16
  <img src="https://img.shields.io/badge/dsh-%3E%3D0.2.1--rc.2%20%3C0.3.0-blueviolet" alt="dsh">
@@ -28,7 +28,7 @@
28
28
  - **Measured in public, including the losses.** A 30-query bilingual comparison records availability, latency, unique domains, official-domain presence and every failure against keyless Firecrawl and anonymous Exa. The follow-up documents the Exa architecture difference, the 3 s CJK budget and the Rust official-source lookup. Exa is faster here; this plugin returns more distinct domains. Both test arms were keyless. The method and raw results are in [docs/comparison.md](docs/comparison.md) and a [public gist](https://gist.github.com/fangchenghao/3db9838fa98928435c1c2df38202c373).
29
29
  - **Fetching admits failure.** Binary documents and empty 200 responses are rejected by name (`SEARCH_UNSUPPORTED`, `SEARCH_EMPTY`) instead of being returned as noise; arXiv PDF links are the deliberate exception and are read through the HTML full-text view; the direct → tunnel → render → extract ladder shares one 25 s budget and stops when there is no time left. The budget is per request — there is a regression test for exactly that.
30
30
  - **Windows is a first-class host.** Pure ESM, no native addon or install script; the system proxy is discovered from Windows Internet Settings, and the full suite runs on a `windows-latest` CI runner. See [Install on Windows](#windows).
31
- - **698 tests across 56 files**, a committed `lib/`, declared `engines.dsh` / `engines.node`, and a [SECURITY.md](SECURITY.md) that lists every process, host and file the plugin touches.
31
+ - **711 tests across 56 files**, a committed `lib/`, declared `engines.dsh` / `engines.node`, and a [SECURITY.md](SECURITY.md) that lists every process, host and file the plugin touches.
32
32
  - **Bilingual documentation.** This README and [中文说明](README.zh-CN.md) are maintained side by side.
33
33
 
34
34
  ![Ranking baseline and parameter sweep](https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/bench.png)
@@ -45,7 +45,7 @@ The web-search corner of the DSH plugin ecosystem is crowded. A dated snapshot
45
45
  | Key-free path | 8 engines, all key-free | ~30-engine free/paid chain | 5 defaults (seam, Exa, DDG, Bing, Jina) | Firecrawl + Keenable keyless defaults | 1 anonymous AnySearch API |
46
46
  | Dedicated vertical sources | 34 (8 on by default) | 41 | platform registry, not a vertical catalog | web + X, no vertical catalog | vertical API behind AnySearch |
47
47
  | Model-facing tools | 1 (`web_tools`, 8 actions) + 4 MCP tools | ~10 | 2 (`web_index` / `web_call`, 20 actions) | native search/fetch + skill | 5 |
48
- | Tests | 56 files / 698 cases, locally verified | no test script in `package.json` | 774 unit + 94 bench, self-reported | Vitest configured; count not published | Vitest configured; count not published |
48
+ | Tests | 56 files / 711 cases, locally verified | no test script in `package.json` | 774 unit + 94 bench, self-reported | Vitest configured; count not published | Vitest configured; count not published |
49
49
  | GitHub stars | 0 | 620 | 77 | 836 | 663 |
50
50
  | npm downloads, last month | 1,087 | 52,436 | 6,420 | 43,151 | 9,251 |
51
51
  | npm unpacked size | 1.2 MB | 0.70 MB | 3.05 MB | 0.51 MB | 0.27 MB |
@@ -56,7 +56,7 @@ read is:
56
56
  - **Feature depth:** upper-middle. `dsh-free-search` has more engines and
57
57
  verticals; `dsh-web-search-pro` has a deeper evidence/ranking pipeline and far
58
58
  more platform backends; `modsearch` is the stronger cross-host bridge.
59
- - **Engineering and verification:** near the top. 698 local tests, a replayable
59
+ - **Engineering and verification:** near the top. 711 local tests, a replayable
60
60
  ranking benchmark, declared security boundaries and one runtime dependency
61
61
  put it alongside the best-maintained entries, while remaining much smaller
62
62
  than the Playwright-heavy option.
@@ -65,7 +65,7 @@ read is:
65
65
  common in the other plugins.
66
66
  - **Adoption and maturity:** bottom third of this sample. At the 2026-10-25
67
67
  snapshot the repository was four days old, had no stars and little
68
- independent adoption; the current npm release is 0.24.3.
68
+ independent adoption; the current npm release is 0.24.4.
69
69
  - **Tool-surface economy:** now competitive. 0.20.x exposes one `web_tools`
70
70
  entry with eight actions, close to `dsh-web-search-pro`'s two tools with 20
71
71
  actions; the legacy eight-name surface remains opt-in for old prompts.
package/README.zh-CN.md CHANGED
@@ -9,7 +9,7 @@
9
9
 
10
10
  <p align="center">
11
11
  <a href="https://www.npmjs.com/package/@fffangchenghao/dsh-web-search"><img src="https://img.shields.io/npm/v/@fffangchenghao/dsh-web-search.svg" alt="npm 版本"></a>
12
- <img src="https://img.shields.io/badge/tests-698%20tests-brightgreen" alt="测试">
12
+ <img src="https://img.shields.io/badge/tests-711%20tests-brightgreen" alt="测试">
13
13
  <img src="https://img.shields.io/badge/engines-8%2C%20zero%20API%20key-brightgreen" alt="引擎">
14
14
  <img src="https://img.shields.io/badge/node-%3E%3D22.18-blue" alt="node">
15
15
  <img src="https://img.shields.io/badge/dsh-%3E%3D0.2.1--rc.2%20%3C0.3.0-blueviolet" alt="dsh">
@@ -27,7 +27,7 @@
27
27
  - **公开测量,也公开输的地方。** 30 条中英查询对比记录可用性、延迟、独立域名、官方域名命中与每一条失败;对比对象是免 key Firecrawl 和匿名 Exa。这里 Exa 更快,本插件返回的独立域名更多;两条对比路径都未使用 key。方法和原始结果在 [docs/comparison.md](docs/comparison.md);Exa 架构差异、中文 3 秒预算和 Rust 官方源见 [延迟与官方源](docs/latency-and-sources.md)。
28
28
  - **抓取会承认失败。** 二进制文档和「200 但正文为空」会被明确拒绝(`SEARCH_UNSUPPORTED`、`SEARCH_EMPTY`),而不是当噪音返回;arXiv 的 PDF 链接是有意保留的例外,会改读 HTML 全文;direct → tunnel → render → extract 四级阶梯共享 25 秒预算,预算不足就不再起新的一级。预算是「每次请求」而不是「插件启动时」,这点有回归测试守着。
29
29
  - **Windows 是一等宿主。** 纯 ESM,没有原生扩展或安装脚本;会自动读取 Windows 的 Internet 设置代理,完整测试套件也在 `windows-latest` CI 上跑通。安装见[Windows 小节](#windows)。
30
- - **56 个测试文件、698 个用例**,`lib/` 产物入库,声明了 `engines.dsh` / `engines.node`,[SECURITY.md](SECURITY.md) 逐项列出插件会碰的进程、主机和文件。
30
+ - **56 个测试文件、711 个用例**,`lib/` 产物入库,声明了 `engines.dsh` / `engines.node`,[SECURITY.md](SECURITY.md) 逐项列出插件会碰的进程、主机和文件。
31
31
  - **双语文档。** 本文件和 [English README](README.md) 同步维护。
32
32
 
33
33
  ![排序基线参数扫描](https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/bench.png)
@@ -54,11 +54,11 @@ DSH 插件生态里的「联网搜索」已经很挤。下面是一张 2026-10-2
54
54
  - **功能深度:中上。** `dsh-free-search` 引擎和垂直源更多;
55
55
  `dsh-web-search-pro` 的证据/排序管线和平台后端更深;`modsearch` 的跨宿主
56
56
  能力更强。
57
- - **工程质量与可验证性:接近第一梯队。** 698 个本地测试、可重放的排序基准、
57
+ - **工程质量与可验证性:接近第一梯队。** 711 个本地测试、可重放的排序基准、
58
58
  明确的安全边界、只有 1 个运行时依赖,同时体积远小于 Playwright 重型方案。
59
59
  - **免 key + 官方源纠错:最强卖点。** Chrome、PostgreSQL、AI Act 的官方路径
60
60
  和 34 个垂直源,不是其他插件普遍具备的。
61
- - **采用度与成熟度:这个样本里的后段。** 2026-10-25 快照时 GitHub 仓库刚创建 4 天、0 stars,还缺少独立采用;当前 npm 版本是 0.24.3。
61
+ - **采用度与成熟度:这个样本里的后段。** 2026-10-25 快照时 GitHub 仓库刚创建 4 天、0 stars,还缺少独立采用;当前 npm 版本是 0.24.4。
62
62
  - **工具面的经济性:现在有竞争力。** 0.20.x 只暴露一个 `web_tools` + 8 个
63
63
  action,接近 `dsh-web-search-pro` 的 2 工具 + 20 动作;旧的八个工具名保留为
64
64
  可选兼容模式。
package/client.js CHANGED
@@ -74,7 +74,7 @@
74
74
  var ENTRY = 'web-search-bing';
75
75
  var PACKAGE_NAME = 'dsh-web-search-bing';
76
76
  /** Keep in step with package.json; compared with the npm registry. */
77
- var VERSION = '0.24.3';
77
+ var VERSION = '0.24.4';
78
78
  /** Snapshot used while no configuration service is reachable. */
79
79
  var EMPTY_STATE = { status: 'unavailable', value: undefined, base: undefined, user: undefined, revision: undefined, writable: false };
80
80
 
@@ -6,7 +6,7 @@ whether a provider answers at all, how long it takes, how many distinct domains
6
6
  it returns, and whether a page on a known official domain appears in the top 8.
7
7
  The failures are part of the result.
8
8
 
9
- > **Version note (2026-11-02):** 0.24.0 added Windows system-proxy discovery, and 0.24.1 and 0.24.2 are documentation/metadata patches; 0.24.3 fixes fetch/proxy/markdown reliability without changing the measured search path. These patches do not change the search, ranking or source-adapter path measured below, so the 0.23.0 numbers remain the latest comparison.
9
+ > **Version note (2026-11-02):** 0.24.0 added Windows system-proxy discovery, and 0.24.1 and 0.24.2 are documentation/metadata patches; 0.24.3/0.24.4 fix fetch/proxy/markdown/cache reliability without changing the measured search path. These patches do not change the search, ranking or source-adapter path measured below, so the 0.23.0 numbers remain the latest comparison.
10
10
 
11
11
  ## Method
12
12
 
@@ -52,7 +52,7 @@
52
52
  <rect x="70" y="620" width="330" height="76" rx="14" fill="#101d31" stroke="#26364f"/>
53
53
  <text x="92" y="651" class="m" font-size="13">Engines</text><text x="92" y="677" class="t" font-size="20" font-weight="700">8 key-free paths</text>
54
54
  <rect x="420" y="620" width="330" height="76" rx="14" fill="#101d31" stroke="#26364f"/>
55
- <text x="442" y="651" class="m" font-size="13">Tests</text><text x="442" y="677" class="t" font-size="20" font-weight="700">698 local cases</text>
55
+ <text x="442" y="651" class="m" font-size="13">Tests</text><text x="442" y="677" class="t" font-size="20" font-weight="700">711 local cases</text>
56
56
  <rect x="770" y="620" width="360" height="76" rx="14" fill="#101d31" stroke="#26364f"/>
57
57
  <text x="792" y="651" class="m" font-size="13">Cross-agent</text><text x="792" y="677" class="t" font-size="20" font-weight="700">MCP stdio server</text>
58
58
  </g>
@@ -6,7 +6,7 @@ This note answers three questions from the public 30-query comparison:
6
6
  2. Can the AnySearch tail and the Chinese Rust miss be fixed?
7
7
  3. Does the fix require a vector database or a different retrieval model?
8
8
 
9
- > **Version note (2026-11-02):** 0.24.0 added Windows system-proxy discovery, and 0.24.1 and 0.24.2 are documentation/metadata patches; 0.24.3 fixes fetch/proxy/markdown reliability without changing the measured search path. The search path is unchanged, so these 0.23.0 measurements still describe 0.24.x.
9
+ > **Version note (2026-11-02):** 0.24.0 added Windows system-proxy discovery, and 0.24.1 and 0.24.2 are documentation/metadata patches; 0.24.3/0.24.4 fix fetch/proxy/markdown/cache reliability without changing the measured search path. The search path is unchanged, so these 0.23.0 measurements still describe 0.24.x.
10
10
 
11
11
  The short answer: Exa is a different architecture, not a faster RRF loop. This
12
12
  plugin can borrow tail-tolerance and source-specific facts, but copying Exa's
@@ -29,7 +29,7 @@ Only `0.2.1-rc.2` is declared, because that is the version actually exercised.
29
29
  `mktemp -d`; the real `~/.dsh` was never written to. Bundled pnpm 11.28.5.
30
30
  - **Windows CI:** GitHub Actions `windows-latest`, Node 24. Install, typecheck,
31
31
  the full test suite, build and the entry-point import all pass. Run:
32
- https://github.com/fangchenghao/dsh-web-search/actions/runs/44131883819
32
+ https://github.com/fangchenghao/dsh-web-search/actions/runs/44230164601
33
33
  (the matrix also runs on macOS and Ubuntu). This is a CI environment, not a
34
34
  claim about every Windows terminal or proxy client.
35
35
 
@@ -228,7 +228,15 @@ let tempCounter = 0;
228
228
  */
229
229
  export function writeCacheFile(file, entries, now, options = {}) {
230
230
  const io = options.io ?? nodeCacheFileIo;
231
- const { data } = serializeCache(entries, now, options.maxBytes);
231
+ const snapshot = serializeCache(entries, now, options.maxBytes);
232
+ const budget = typeof options.maxBytes === 'number' && Number.isFinite(options.maxBytes) && options.maxBytes > 0
233
+ ? Math.floor(options.maxBytes)
234
+ : DEFAULT_CACHE_FILE_MAX_BYTES;
235
+ // The empty envelope itself can exceed a tiny custom budget. Refuse the
236
+ // write rather than replacing a valid old snapshot with an over-budget one.
237
+ if (snapshot.bytes > budget)
238
+ return false;
239
+ const { data } = snapshot;
232
240
  tempCounter += 1;
233
241
  const temp = `${file}.${process.pid}.${tempCounter}.tmp`;
234
242
  try {
package/lib/cache.js CHANGED
@@ -23,12 +23,21 @@ const DEFAULT_PERSIST_THROTTLE_MS = 2_000;
23
23
  */
24
24
  export function cacheKey(parts) {
25
25
  const query = parts.query.trim().toLowerCase().replace(/\s+/gu, ' ');
26
- const engines = [...parts.engines].sort().join('+');
27
- const window = parts.timeRange === undefined || parts.timeRange === 'off' ? '' : `${parts.timeRange}|`;
28
- const safe = parts.safeSearch === undefined || parts.safeSearch === 'off' ? '' : `safe=${parts.safeSearch}|`;
29
- const fallback = parts.fallbackOn === undefined ? '' : `fallback=${[...parts.fallbackOn].sort().join(',')}|`;
30
- const blocked = parts.blockedDomains === undefined ? '' : `blocked=${[...parts.blockedDomains].sort().join(',')}|`;
31
- return `${parts.lead}|${parts.wait}|${parts.limit}|${engines}|${window}${safe}${fallback}${blocked}${query}`;
26
+ // A JSON tuple is unambiguous and keeps every optional value in its own
27
+ // slot. The old delimiter join let `safeSearch: 'off'` + a query containing
28
+ // `safe=strict|` collide with `safeSearch: 'strict'` + a shorter query.
29
+ return JSON.stringify([
30
+ 'v2',
31
+ parts.lead,
32
+ parts.wait,
33
+ parts.limit,
34
+ [...parts.engines].sort(),
35
+ parts.timeRange === undefined || parts.timeRange === 'off' ? null : parts.timeRange,
36
+ parts.safeSearch === undefined || parts.safeSearch === 'off' ? null : parts.safeSearch,
37
+ parts.fallbackOn === undefined ? null : [...parts.fallbackOn].sort(),
38
+ parts.blockedDomains === undefined || parts.blockedDomains.length === 0 ? null : [...parts.blockedDomains].sort(),
39
+ query,
40
+ ]);
32
41
  }
33
42
  /**
34
43
  * Copy the array and each source.
@@ -94,6 +94,10 @@ export function parseRustRelease(payload, requested) {
94
94
  }
95
95
  /** One direct GET with its own abort budget. */
96
96
  async function directJson(url, headers, options, timeoutMs) {
97
+ if (options.signal?.aborted === true) {
98
+ const reason = options.signal.reason;
99
+ throw reason instanceof Error ? reason : new DOMException('The operation was aborted', 'AbortError');
100
+ }
97
101
  const controller = new AbortController();
98
102
  const timer = setTimeout(() => controller.abort(), timeoutMs);
99
103
  const abort = () => controller.abort();
@@ -132,6 +136,8 @@ async function fetchJson(url, options, timeoutMs) {
132
136
  return await directJson(url, headers, options, Math.min(timeoutMs, 1_500));
133
137
  }
134
138
  catch (error) {
139
+ if (options.signal?.aborted === true)
140
+ throw error;
135
141
  if (options.proxy === undefined || options.proxy.length === 0)
136
142
  throw error;
137
143
  const remaining = timeoutMs - (Date.now() - started);
@@ -13,14 +13,16 @@ async function fetchNpmPackage(name, options) {
13
13
  try {
14
14
  const result = await searchVerticals(name, {
15
15
  ids: ['npm'],
16
- limit: 1,
16
+ // Search, not a registry exact lookup: ask for a few rows and verify the
17
+ // package identity instead of trusting the first fuzzy result.
18
+ limit: 10,
17
19
  timeoutMs: typeof options.timeoutMs === 'number' && Number.isFinite(options.timeoutMs) && options.timeoutMs > 0
18
20
  ? Math.min(options.timeoutMs, 8_000)
19
21
  : 8_000,
20
22
  ...(options.proxy === undefined ? {} : { proxy: options.proxy }),
21
23
  ...(options.signal === undefined ? {} : { signal: options.signal }),
22
24
  });
23
- return result.sources[0];
25
+ return result.sources.find((source) => source.title === name || source.title?.startsWith(`${name}@`) === true);
24
26
  }
25
27
  catch {
26
28
  return undefined;
@@ -75,6 +75,10 @@ export function postgresSupportSource(releases) {
75
75
  }
76
76
  /** Read the page over the configured tunnel, or directly when none is set. */
77
77
  async function fetchText(url, options, timeoutMs) {
78
+ if (options.signal?.aborted === true) {
79
+ const reason = options.signal.reason;
80
+ throw reason instanceof Error ? reason : new DOMException('The operation was aborted', 'AbortError');
81
+ }
78
82
  if (options.proxy !== undefined && options.proxy.length > 0) {
79
83
  const response = await proxyGet(url, {
80
84
  proxy: options.proxy,
package/lib/parsers.js CHANGED
@@ -82,38 +82,52 @@ export function resolveHref(raw, baseUrl) {
82
82
  if (typeof raw !== 'string' || raw.length === 0)
83
83
  return undefined;
84
84
  // Attribute values are HTML-escaped, so a Bing click tracker arrives as
85
- // `...&amp;u=a1<base64url>`; decode before looking for the payload.
86
- let candidate = raw.replace(/&amp;/gi, '&').replace(/&#0*38;/g, '&');
87
- const wrapped = /[?&]u=a1([A-Za-z0-9_-]+)/.exec(candidate);
88
- if (wrapped) {
89
- try {
90
- candidate = Buffer.from(wrapped[1].replace(/-/g, '+').replace(/_/g, '/'), 'base64').toString('utf8');
91
- }
92
- catch {
93
- return undefined;
85
+ // `...&amp;u=a1<base64url>`; decode before parsing.
86
+ const candidate = raw.replace(/&amp;/gi, '&').replace(/&#0*38;/g, '&');
87
+ let parsed;
88
+ try {
89
+ parsed = new URL(candidate, baseUrl);
90
+ }
91
+ catch {
92
+ return undefined;
93
+ }
94
+ // Only the search engine's own wrapper is unwrapped. A normal page whose
95
+ // query string happens to contain `uddg` or `u=a1...` must keep its URL.
96
+ const host = parsed.hostname.toLowerCase();
97
+ let target;
98
+ if ((host === 'bing.com' || host === 'www.bing.com') && parsed.pathname === '/ck/a') {
99
+ const payload = parsed.searchParams.get('u');
100
+ const match = payload === null ? null : /^a1([A-Za-z0-9_-]+={0,2})$/u.exec(payload);
101
+ if (match !== null) {
102
+ try {
103
+ target = Buffer.from(match[1].replace(/=+$/u, '').replace(/-/g, '+').replace(/_/g, '/'), 'base64').toString('utf8');
104
+ }
105
+ catch {
106
+ return undefined;
107
+ }
94
108
  }
95
109
  }
96
- else {
97
- // DuckDuckGo wraps outbound links as `/l/?uddg=<percent-encoded>`.
98
- const uddg = /[?&]uddg=([^&]+)/.exec(candidate);
99
- if (uddg) {
110
+ else if ((host === 'duckduckgo.com' || host.endsWith('.duckduckgo.com')) && parsed.pathname === '/l/') {
111
+ const wrapped = parsed.searchParams.get('uddg');
112
+ if (wrapped !== null && wrapped.length > 0) {
100
113
  try {
101
- candidate = decodeURIComponent(uddg[1]);
114
+ target = decodeURIComponent(wrapped);
102
115
  }
103
116
  catch {
104
117
  return undefined;
105
118
  }
106
119
  }
107
120
  }
108
- try {
109
- const url = new URL(candidate, baseUrl);
110
- if (url.protocol !== 'http:' && url.protocol !== 'https:')
121
+ if (target !== undefined) {
122
+ try {
123
+ const unwrapped = new URL(target);
124
+ return unwrapped.protocol === 'http:' || unwrapped.protocol === 'https:' ? unwrapped.toString() : undefined;
125
+ }
126
+ catch {
111
127
  return undefined;
112
- return url.toString();
113
- }
114
- catch {
115
- return undefined;
128
+ }
116
129
  }
130
+ return parsed.protocol === 'http:' || parsed.protocol === 'https:' ? parsed.toString() : undefined;
117
131
  }
118
132
  /**
119
133
  * Parse a 360 (so.com) results page. The real URL lives in `data-mdurl`;
@@ -457,6 +471,9 @@ export function parseExaMcp(payload, baseUrl) {
457
471
  }
458
472
  else if (/^highlights\s*:/i.test(line)) {
459
473
  inHighlights = true;
474
+ const inline = line.replace(/^highlights\s*:/i, '').trim();
475
+ if (inline.length > 0)
476
+ highlights.push(inline);
460
477
  }
461
478
  else if (/^(published|author)\s*:/i.test(line)) {
462
479
  inHighlights = false;
package/lib/platforms.js CHANGED
@@ -267,6 +267,26 @@ function deadline(timeoutMs, signal) {
267
267
  const timer = AbortSignal.timeout(timeoutMs);
268
268
  return signal === undefined ? timer : AbortSignal.any([signal, timer]);
269
269
  }
270
+ /** Race a promise against an abort signal, observing both outcomes. */
271
+ function withAbort(promise, signal) {
272
+ if (signal.aborted) {
273
+ return Promise.reject(signal.reason instanceof Error ? signal.reason : new Error('platform search was aborted'));
274
+ }
275
+ return new Promise((resolve, reject) => {
276
+ const onAbort = () => {
277
+ signal.removeEventListener('abort', onAbort);
278
+ reject(signal.reason instanceof Error ? signal.reason : new Error('platform search was aborted'));
279
+ };
280
+ signal.addEventListener('abort', onAbort, { once: true });
281
+ promise.then((value) => {
282
+ signal.removeEventListener('abort', onAbort);
283
+ resolve(value);
284
+ }, (error) => {
285
+ signal.removeEventListener('abort', onAbort);
286
+ reject(error);
287
+ });
288
+ });
289
+ }
270
290
  /** Turn a non-2xx status into a reason. */
271
291
  function statusError(status) {
272
292
  return new Error(status === 0 ? 'no response' : `HTTP ${status}`);
@@ -828,7 +848,11 @@ async function fallbackAnswer(spec, query, limit, options, directReason) {
828
848
  }
829
849
  const fallbackQuery = `site:${spec.site} ${query}`;
830
850
  try {
831
- const result = await options.fallbackSearch(fallbackQuery, limit, options.signal);
851
+ if (options.signal?.aborted === true) {
852
+ const reason = options.signal.reason;
853
+ throw reason instanceof Error ? reason : new Error('platform search was aborted');
854
+ }
855
+ const result = await withAbort(options.fallbackSearch(fallbackQuery, limit, options.signal), options.signal ?? new AbortController().signal);
832
856
  // A search engine may ignore the site: operator and return generic pages.
833
857
  // A platform result must at least live on that platform's domain.
834
858
  const allowed = normaliseSources(result.sources, limit * 2)
@@ -875,7 +899,7 @@ async function fallbackAnswer(spec, query, limit, options, directReason) {
875
899
  * @param timeoutMs - per-platform deadline.
876
900
  * @returns the platform answer.
877
901
  */
878
- async function searchPlatform(spec, query, limit, options, timeoutMs) {
902
+ async function searchPlatformInner(spec, query, limit, options, timeoutMs) {
879
903
  if (spec.fallbackOnly === true || spec.build === undefined || spec.parse === undefined) {
880
904
  if (options.fallbackSearch === undefined) {
881
905
  return {
@@ -927,6 +951,45 @@ async function searchPlatform(spec, query, limit, options, timeoutMs) {
927
951
  }
928
952
  return fallbackAnswer(spec, query, limit, options, directReason);
929
953
  }
954
+ /**
955
+ * Run one platform under a single deadline that also covers the fallback.
956
+ *
957
+ * `requestText` has its own per-request timeout, but an alternate endpoint or
958
+ * an injected fallback search can otherwise get a fresh budget, and a fallback
959
+ * that ignores its signal can hang the whole `Promise.all` forever.
960
+ *
961
+ * @param spec - platform being served.
962
+ * @param query - original query, already trimmed.
963
+ * @param limit - per-platform budget.
964
+ * @param options - caller options.
965
+ * @param timeoutMs - total deadline for this platform.
966
+ * @returns the platform answer; timeout becomes a platform failure unless the
967
+ * caller itself cancelled.
968
+ */
969
+ async function searchPlatform(spec, query, limit, options, timeoutMs) {
970
+ const controller = new AbortController();
971
+ const timer = setTimeout(() => controller.abort(new Error(`platform ${spec.id} timed out after ${timeoutMs}ms`)), timeoutMs);
972
+ const combined = options.signal === undefined
973
+ ? controller.signal
974
+ : AbortSignal.any([options.signal, controller.signal]);
975
+ const scoped = { ...options, signal: combined };
976
+ try {
977
+ return await withAbort(searchPlatformInner(spec, query, limit, scoped, timeoutMs), combined);
978
+ }
979
+ catch (error) {
980
+ if (options.signal?.aborted === true)
981
+ throw error;
982
+ return {
983
+ id: spec.id,
984
+ sources: [],
985
+ via: spec.fallbackOnly === true ? 'fallback' : 'api',
986
+ failure: failureText(spec.id, messageOf(error)),
987
+ };
988
+ }
989
+ finally {
990
+ clearTimeout(timer);
991
+ }
992
+ }
930
993
  /**
931
994
  * Search the requested platforms and merge their answers.
932
995
  *
package/lib/proxy.js CHANGED
@@ -486,6 +486,10 @@ export function readResponse(buffer, maxBytes = DEFAULT_MAX_BYTES) {
486
486
  * @returns `{ status, text, url, headers }` for the final hop.
487
487
  */
488
488
  export async function proxyGet(url, options) {
489
+ if (options.signal?.aborted === true) {
490
+ const reason = options.signal.reason;
491
+ throw reason instanceof Error ? reason : new DOMException('The operation was aborted', 'AbortError');
492
+ }
489
493
  const proxy = parseProxy(options.proxy);
490
494
  const timeoutMs = options.timeoutMs !== undefined && Number.isFinite(options.timeoutMs) && options.timeoutMs > 0 ? options.timeoutMs : DEFAULT_TIMEOUT_MS;
491
495
  const redirects = options.redirects !== undefined && Number.isInteger(options.redirects) ? options.redirects : DEFAULT_REDIRECTS;
package/lib/read-page.js CHANGED
@@ -401,20 +401,39 @@ export function createReadPageDefinition(name, readSettings, options = {}) {
401
401
  if (parsed.offset > totalLength) {
402
402
  throw new Error(`read_page: offset ${parsed.offset} is past the end of ${parsed.url} (totalLength ${totalLength}); use offset 0 to start over or ${totalLength} to read the tail`);
403
403
  }
404
- // Keep surrogate pairs intact at the cut point. `nextOffset` follows the
405
- // adjusted end, so the concatenated chunks still equal the full body.
406
- let end = Math.min(totalLength, parsed.offset + parsed.maxLength);
407
- if (end > parsed.offset && end < totalLength) {
408
- const last = page.content.charCodeAt(end - 1);
409
- if (last >= 0xd800 && last <= 0xdbff)
410
- end -= 1;
404
+ // Walk the real page boundaries. A surrogate pair at a cut point moves
405
+ // that boundary one code unit back, so the chunk count and index cannot
406
+ // be derived from `ceil(totalLength / maxLength)` without drifting.
407
+ const pageEnd = (start) => {
408
+ let end = Math.min(totalLength, start + parsed.maxLength);
409
+ if (end > start && end < totalLength) {
410
+ const last = page.content.charCodeAt(end - 1);
411
+ if (last >= 0xd800 && last <= 0xdbff)
412
+ end -= 1;
413
+ }
414
+ // A one-character budget on a surrogate pair must still advance.
415
+ return end > start ? end : Math.min(totalLength, start + 1);
416
+ };
417
+ let count = 0;
418
+ let index = 0;
419
+ let cursor = 0;
420
+ while (cursor < totalLength) {
421
+ const boundary = pageEnd(cursor);
422
+ count += 1;
423
+ if (parsed.offset >= cursor && parsed.offset < boundary)
424
+ index = count;
425
+ cursor = boundary;
426
+ }
427
+ if (totalLength === 0) {
428
+ count = 1;
429
+ index = 1;
430
+ }
431
+ else if (parsed.offset === totalLength) {
432
+ index = count;
411
433
  }
434
+ const end = pageEnd(parsed.offset);
412
435
  const content = page.content.slice(parsed.offset, end);
413
436
  const nextOffset = end < totalLength ? end : null;
414
- const count = totalLength === 0 ? 1 : Math.ceil(totalLength / parsed.maxLength);
415
- const index = totalLength === 0
416
- ? 1
417
- : Math.min(count, Math.floor(parsed.offset / parsed.maxLength) + 1);
418
437
  const value = {
419
438
  url: parsed.url,
420
439
  content,
package/lib/research.d.ts CHANGED
@@ -87,6 +87,11 @@ export interface ResearchValue {
87
87
  }>;
88
88
  /** Fetch failures, one entry per URL. */
89
89
  failures: ResearchFetchFailure[];
90
+ /** Search errors, one entry per failed query; omitted when none failed. */
91
+ searchFailures?: Array<{
92
+ query: string;
93
+ reason: string;
94
+ }>;
90
95
  /** Number of search rows seen before the fetch budget cut them. */
91
96
  found: number;
92
97
  /** True when the search row list was cut by the fetch budget. */
package/lib/research.js CHANGED
@@ -169,9 +169,13 @@ export function selectExcerpt(text, tokens, max) {
169
169
  const add = (index) => {
170
170
  const window = windows[index];
171
171
  if (window === undefined || picked.has(index))
172
- return;
172
+ return false;
173
+ const addition = window.length + 2;
174
+ if (used + addition > max)
175
+ return false;
173
176
  picked.add(index);
174
- used += window.length + 2;
177
+ used += addition;
178
+ return true;
175
179
  };
176
180
  // Regulations often put the effective date in the final article, after the
177
181
  // definitions. A pure relevance sort can spend the budget on the definitions
@@ -233,10 +237,12 @@ export function selectExcerpt(text, tokens, max) {
233
237
  }
234
238
  const selected = [...picked].sort((a, b) => a - b).map((index) => windows[index] ?? '');
235
239
  const excerpt = selected.join('\n\n…\n\n');
236
- // Do not truncate from the end here: when tailNeeded is true, the end is the
237
- // effective date the caller asked for. The selected windows are already
238
- // bounded to roughly max plus one window.
239
- return excerpt;
240
+ // The tail/legal/opening safeguards must not bypass the caller's excerpt
241
+ // budget. If a single window still overshoots, cut it explicitly; losing a
242
+ // suffix is better than returning a multi-thousand-character excerpt.
243
+ if (excerpt.length <= max)
244
+ return excerpt;
245
+ return max > 0 ? `${excerpt.slice(0, Math.max(0, max - 1)).trimEnd()}…` : '';
240
246
  }
241
247
  /** Walk `worker` over items with at most `limit` in flight, preserving order. */
242
248
  async function mapLimit(items, limit, worker) {
@@ -254,6 +260,26 @@ async function mapLimit(items, limit, worker) {
254
260
  await Promise.all(runners);
255
261
  return results;
256
262
  }
263
+ /** Race a promise against a signal while observing both outcomes. */
264
+ function withAbort(promise, signal) {
265
+ if (signal.aborted) {
266
+ return Promise.reject(signal.reason instanceof Error ? signal.reason : new Error('research was aborted'));
267
+ }
268
+ return new Promise((resolve, reject) => {
269
+ const onAbort = () => {
270
+ signal.removeEventListener('abort', onAbort);
271
+ reject(signal.reason instanceof Error ? signal.reason : new Error('research was aborted'));
272
+ };
273
+ signal.addEventListener('abort', onAbort, { once: true });
274
+ promise.then((value) => {
275
+ signal.removeEventListener('abort', onAbort);
276
+ resolve(value);
277
+ }, (error) => {
278
+ signal.removeEventListener('abort', onAbort);
279
+ reject(error);
280
+ });
281
+ });
282
+ }
257
283
  /** Words that carry no retrieval signal in a research query. */
258
284
  const QUERY_STOPWORDS = new Set([
259
285
  'the', 'a', 'an', 'and', 'or', 'of', 'to', 'in', 'on', 'for', 'with', 'from',
@@ -607,6 +633,15 @@ function outputSchema() {
607
633
  required: ['url', 'reason'],
608
634
  },
609
635
  },
636
+ searchFailures: {
637
+ type: 'array',
638
+ items: {
639
+ type: 'object',
640
+ additionalProperties: false,
641
+ properties: { query: { type: 'string' }, reason: { type: 'string' } },
642
+ required: ['query', 'reason'],
643
+ },
644
+ },
610
645
  found: { type: 'integer' },
611
646
  truncated: { type: 'boolean' },
612
647
  },
@@ -644,10 +679,20 @@ export function formatResearchValue(value) {
644
679
  '## Found but not fetched',
645
680
  ...value.sources.map((source) => `- ${source.title !== undefined && source.title.length > 0 ? `${source.title} — ` : ''}${source.url}`),
646
681
  ];
682
+ const searchFailures = value.searchFailures === undefined || value.searchFailures.length === 0
683
+ ? []
684
+ : ['## Search failures', ...value.searchFailures.map((failure) => `- ${JSON.stringify(failure.query)}: ${failure.reason}`)];
647
685
  const failures = value.failures.length === 0
648
686
  ? []
649
687
  : ['## Fetch failures', ...value.failures.map((failure) => `- ${failure.url}: ${failure.reason}`)];
650
- return [...head, '', ...sections, ...(notFetched.length === 0 ? [] : ['', ...notFetched]), ...(failures.length === 0 ? [] : ['', ...failures])]
688
+ return [
689
+ ...head,
690
+ '',
691
+ ...sections,
692
+ ...(notFetched.length === 0 ? [] : ['', ...notFetched]),
693
+ ...(searchFailures.length === 0 ? [] : ['', ...searchFailures]),
694
+ ...(failures.length === 0 ? [] : ['', ...failures]),
695
+ ]
651
696
  .join('\n')
652
697
  .trimEnd();
653
698
  }
@@ -673,15 +718,26 @@ export function createResearchDefinition(name, search, fetchPage, options = {})
673
718
  const parsed = parseArgs(args);
674
719
  const signal = exec.signal;
675
720
  const startedAt = Date.now();
721
+ // The tool's declared 45 s budget must cover the searches too. Passing
722
+ // only the host signal let a slow search consume the whole 60 s host
723
+ // timeout before the page ladder ever saw `remaining()`.
724
+ const deadline = AbortSignal.timeout(RESEARCH_BUDGET_MS);
725
+ const workSignal = signal === undefined ? deadline : AbortSignal.any([signal, deadline]);
676
726
  const remaining = () => Math.max(0, RESEARCH_BUDGET_MS - (Date.now() - startedAt));
677
727
  const cancelled = () => signal !== undefined && signal.aborted;
678
728
  const searchLimit = Math.max(5, parsed.maxSources);
679
729
  const queriesUsed = [...parsed.queries];
680
730
  const runSearch = async (query) => {
681
731
  try {
682
- return await search(query, searchLimit, signal);
732
+ if (workSignal.aborted === true) {
733
+ const reason = workSignal.reason;
734
+ throw reason instanceof Error ? reason : new Error('research was aborted');
735
+ }
736
+ return await withAbort(search(query, searchLimit, workSignal), workSignal);
683
737
  }
684
738
  catch (error) {
739
+ if (signal?.aborted === true)
740
+ throw error;
685
741
  return {
686
742
  sources: [],
687
743
  failures: [error instanceof Error ? error.message : String(error)],
@@ -691,12 +747,15 @@ export function createResearchDefinition(name, search, fetchPage, options = {})
691
747
  const outcomes = await mapLimit(parsed.queries, 2, runSearch);
692
748
  if (cancelled())
693
749
  throw new Error('research was cancelled');
750
+ const searchFailures = parsed.queries.flatMap((query, index) => (outcomes[index]?.failures ?? []).map((reason) => ({ query, reason })));
694
751
  let candidates = interleave(outcomes.map((outcome) => outcome.sources));
695
752
  const hints = officialHints(parsed.queries);
696
753
  // Long questions often need the issuing body's own site, which the open
697
754
  // engine may not rank. Spend unused query slots on a site-specific query
698
755
  // instead of letting the evidence pack stop at an aggregator.
699
756
  for (const hint of hints) {
757
+ if (remaining() === 0)
758
+ break;
700
759
  // A host match is enough for most issuers, but a generic W3C page does
701
760
  // not cover a specific success criterion; let the targeted query run.
702
761
  // A generic host match is enough for most issuers, but not for the
@@ -713,6 +772,8 @@ export function createResearchDefinition(name, search, fetchPage, options = {})
713
772
  const extra = await runSearch(hinted);
714
773
  if (cancelled())
715
774
  throw new Error('research was cancelled');
775
+ for (const reason of extra.failures)
776
+ searchFailures.push({ query: hinted, reason });
716
777
  // A site: operator is advisory on some engines; keep only rows that
717
778
  // really belong to the issuer's domain.
718
779
  outcomes.push({
@@ -801,11 +862,13 @@ export function createResearchDefinition(name, search, fetchPage, options = {})
801
862
  failures.push({ url: source.url, reason: 'no time left in the research budget' });
802
863
  return;
803
864
  }
804
- const bound = signal === undefined
805
- ? AbortSignal.timeout(budget)
806
- : AbortSignal.any([signal, AbortSignal.timeout(budget)]);
865
+ const bound = AbortSignal.any([workSignal, AbortSignal.timeout(budget)]);
807
866
  try {
808
- const fetched = await fetchPage(source.url, bound);
867
+ if (bound.aborted === true) {
868
+ const reason = bound.reason;
869
+ throw reason instanceof Error ? reason : new Error('research budget exhausted');
870
+ }
871
+ const fetched = await withAbort(fetchPage(source.url, bound), bound);
809
872
  if (looksLikeHtmlDocument(fetched.text)) {
810
873
  failures.push({ url: source.url, reason: 'the page returned a raw HTML document, not readable text' });
811
874
  continue;
@@ -888,6 +951,7 @@ export function createResearchDefinition(name, search, fetchPage, options = {})
888
951
  ...(source.title === undefined || source.title.length === 0 ? {} : { title: source.title }),
889
952
  })),
890
953
  failures,
954
+ ...(searchFailures.length === 0 ? {} : { searchFailures }),
891
955
  found: candidates.length,
892
956
  truncated: candidates.length > parsed.maxSources,
893
957
  };
package/lib/transport.js CHANGED
@@ -199,9 +199,10 @@ async function attemptWithRoute(host, query, signal, timeoutMs, proxy, timeRange
199
199
  }
200
200
  if (signal?.aborted === true || proxy === undefined)
201
201
  return first;
202
+ // When the failed first route was the proxy, the alternate is `undefined`,
203
+ // which means direct — not "no alternate". The old guard here removed the
204
+ // direct fallback exactly when a proxy-first route failed.
202
205
  const alternateRoute = firstRoute === proxy ? undefined : proxy;
203
- if (alternateRoute === undefined)
204
- return first;
205
206
  const retry = await attempt(host, url, signal, timeoutMs, alternateRoute, query);
206
207
  if (retry.ok && retry.route !== undefined)
207
208
  health?.noteRouteSuccess(host.id, retry.route);
package/lib/verticals.js CHANGED
@@ -152,10 +152,13 @@ function openAlexAbstract(value) {
152
152
  for (const [word, positions] of Object.entries(value)) {
153
153
  if (!Array.isArray(positions) || word.trim().length === 0)
154
154
  continue;
155
- const position = positions.find((item) => typeof item === 'number' && Number.isFinite(item));
156
- if (position === undefined)
157
- continue;
158
- words.push({ word: word.trim(), position });
155
+ // The inverted index maps one word to every occurrence. Keeping only the
156
+ // first position silently deleted repeated words from the abstract.
157
+ for (const item of positions) {
158
+ if (typeof item === 'number' && Number.isSafeInteger(item) && item >= 0) {
159
+ words.push({ word: word.trim(), position: item });
160
+ }
161
+ }
159
162
  }
160
163
  words.sort((a, b) => a.position - b.position);
161
164
  return words.map((entry) => entry.word).join(' ');
@@ -1654,8 +1657,11 @@ export async function searchVerticals(query, options = {}) {
1654
1657
  const answers = await Promise.all(specs.map(async (spec) => {
1655
1658
  try {
1656
1659
  const built = spec.build(trimmed, limit);
1660
+ // arXiv's builder already encodes fields/booleans correctly. A date
1661
+ // window only asks arXiv to sort newest first; replacing its
1662
+ // `search_query` mangled `ti:`/`cat:` queries into a quoted phrase.
1657
1663
  const url = spec.id === 'arxiv' && options.timeRange !== undefined && options.timeRange !== 'off'
1658
- ? `${built.replace(/search_query=[^&]*/u, `search_query=${encodeURIComponent(`all:"${trimmed}"`)}`)}&sortBy=submittedDate&sortOrder=descending`
1664
+ ? `${built}&sortBy=submittedDate&sortOrder=descending`
1659
1665
  : built;
1660
1666
  const payload = await fetchPayload(spec, url, options, timeoutMs);
1661
1667
  const sources = spec.parse(payload, url, limit);
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@fffangchenghao/dsh-web-search",
3
- "version": "0.24.3",
3
+ "version": "0.24.4",
4
4
  "description": "DSH bundle: key-free web search, page fetch and research for DeepSeek Harness; eight engines fused with RRF, 34 vertical sources, arXiv, Windows/macOS system-proxy support and a dependency-free MCP server.",
5
5
  "type": "module",
6
6
  "main": "lib/index.js",