@fffangchenghao/dsh-web-search 0.15.0 → 0.16.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,25 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.16.0
4
+
5
+ ### Added
6
+ - **Key rotation.** `anysearchApiKey` now accepts several keys separated by
7
+ commas, spaces or newlines. They are used round-robin; a 401 retires the key
8
+ for a day, a 429 rests it for 60 s doubling to 30 min, a 403 for ten minutes,
9
+ and a success clears its strikes. A throttled or rejected key retries the same
10
+ request on the next key once, in both search and page extraction. Pools are
11
+ cached per Settings value, so editing the field starts clean.
12
+ - **Settings page polish.** A sticky jump bar scrolls to any of the five groups,
13
+ and a line under the title compares the installed version with npm's latest
14
+ (hidden when the registry is unreachable rather than guessed).
15
+ - **Portable skill.** `skills/dsh-web-search/SKILL.md` states when to search,
16
+ fetch, or use `research`, and how to cite; it can be copied into any harness
17
+ that reads `SKILL.md` from disk.
18
+
19
+ ### Changed
20
+ - The AnySearch user agent reports `dsh-web-search/0.15`.
21
+ - Both READMEs document the rotation, the jump bar and the skill file.
22
+
3
23
  ## 0.15.0
4
24
 
5
25
  ### Added
package/README.md CHANGED
@@ -9,7 +9,7 @@
9
9
 
10
10
  <p align="center">
11
11
  <a href="https://www.npmjs.com/package/@fffangchenghao/dsh-web-search"><img src="https://img.shields.io/npm/v/@fffangchenghao/dsh-web-search.svg" alt="npm version"></a>
12
- <img src="https://img.shields.io/badge/tests-466%20passing-brightgreen" alt="tests">
12
+ <img src="https://img.shields.io/badge/tests-474%20passing-brightgreen" alt="tests">
13
13
  <img src="https://img.shields.io/badge/engines-8%2C%20zero%20API%20key-brightgreen" alt="engines">
14
14
  <img src="https://img.shields.io/badge/node-%3E%3D22.18-blue" alt="node">
15
15
  <img src="https://img.shields.io/badge/dsh-%3E%3D0.2.1--rc.2%20%3C0.3.0-blueviolet" alt="dsh">
@@ -21,7 +21,7 @@
21
21
  - **No API key, eight engines.** AnySearch, CN Bing, DuckDuckGo HTML, www Bing, DuckDuckGo Lite and 360 Search are scraped directly; Exa joins the primary tier through its keyless MCP endpoint, and Firecrawl's search API is the measured fallback. The plugin works out of the box.
22
22
  - **The ranking is measured, not asserted.** Cross-engine reciprocal-rank fusion with authority and freshness boosts, a 20-query offline benchmark, and an 81-configuration parameter sweep — the only search plugin here that publishes both its metric and its confidence intervals. See [How the ranking works](docs/rrf-ranking.md).
23
23
  - **Fetching admits failure.** Binary documents and empty 200 responses are rejected by name (`SEARCH_UNSUPPORTED`, `SEARCH_EMPTY`) instead of being returned as noise; the direct → tunnel → render → extract ladder shares one 25 s budget and stops when there is no time left. The budget is per request — there is a regression test for exactly that.
24
- - **466 tests across 28 files**, a committed `lib/`, declared `engines.dsh` / `engines.node`, and a [SECURITY.md](SECURITY.md) that lists every process, host and file the plugin touches.
24
+ - **474 tests across 29 files**, a committed `lib/`, declared `engines.dsh` / `engines.node`, and a [SECURITY.md](SECURITY.md) that lists every process, host and file the plugin touches.
25
25
  - **Bilingual documentation.** This README and [中文说明](README.zh-CN.md) are maintained side by side.
26
26
 
27
27
  ![Ranking baseline and parameter sweep](https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/bench.png)
@@ -37,7 +37,7 @@ The web-search corner of the DSH plugin ecosystem is crowded, and most rivals ar
37
37
  | Key-free engines | 8 | ~30 | 7 channels (+X) | 1 official API |
38
38
  | Dedicated vertical sources | 34 (8 on by default) | 41 | — | — |
39
39
  | Model-facing tools | 6 | ~10 | 3 | 5 |
40
- | Test suite | 28 files / 466 cases | none | 29 files | 6 files |
40
+ | Test suite | 29 files / 474 cases | none | 29 files | 6 files |
41
41
  | Ranking benchmark | 20 queries + 81-config sweep | — | — | — |
42
42
  | Store status | source contract satisfied, awaiting review | unlisted | no entry | blocked (review-required) |
43
43
 
@@ -417,7 +417,7 @@ If the Settings card appears but search still fails, start with [docs/troublesho
417
417
 
418
418
  ## Using the Settings page
419
419
 
420
- Open Settings → "Web search (Bing)". Twenty-six controls, arranged in five groups. Each group owns one kind of decision:
420
+ Open Settings → "Web search (Bing)". Twenty-six controls, arranged in five groups. A sticky jump bar at the top scrolls to any group, and a line under the title compares the installed version with the latest one on npm (it stays hidden when the registry cannot be reached, so it never guesses). Each group owns one kind of decision:
421
421
 
422
422
  | Group | Controls | What it decides |
423
423
  | --- | --- | --- |
@@ -435,7 +435,7 @@ Open Settings → "Web search (Bing)". Twenty-six controls, arranged in five gro
435
435
  | How long to wait | `wait` | `auto` | `auto` follows the query language; `all` waits for everyone; `fast` takes the first answer |
436
436
  | Wait limit (ms) | `timeoutMs` | `15000` | Per-engine deadline, at least 500 |
437
437
  | Proxy (relay) | `proxy` | empty (detect) | e.g. `http://127.0.0.1:7890`; empty reads the environment, then the system proxy |
438
- | AnySearch key | `anysearchApiKey` | empty (anonymous) | Optional. Sent only to `api.anysearch.com`; raises limits and unlocks page extraction |
438
+ | AnySearch key | `anysearchApiKey` | empty (anonymous) | Optional. Sent only to `api.anysearch.com`; raises limits and unlocks page extraction. Several keys may be comma-separated and are rotated, with a throttled or rejected one resting automatically |
439
439
  | Vertical domain | `anysearchDomain` | empty (general) | Typical values: `finance` / `academic` / `security` / `legal` / `code`; a dotted value such as `finance.us_stock` selects a sub-domain |
440
440
 
441
441
  ### Results and ranking (7)
@@ -498,6 +498,10 @@ Three things to keep in mind:
498
498
  - **Ranking changes can be hidden by the cache**: the cache key holds `lead|wait|limit|engine set|time window|query`, not the ranking knobs. Ask the same question within ten minutes and you get the old order — and with persistence on, restarting DSH does not clear it. Rephrase it, or wait ten minutes. Changing the window changes the key, so an old entry cannot interfere.
499
499
  - **`web_fetch` cannot continue from the middle**: it makes one request per URL, so if the tail is missing, raise `fetchMaxLength` or use `read_page` to continue from the break; fetching a narrower page still costs the least context.
500
500
 
501
+ ## Using the skill from another harness
502
+
503
+ `skills/dsh-web-search/SKILL.md` is a plain skill file: it states when to search, when to open pages, when to use `research`, and how to cite. Copy the folder into any harness that reads `SKILL.md` files from disk (for example `~/.claude/skills/`). The plugin itself is a DSH plugin, so another harness still needs its own search and fetch tools; the file is the part that transfers.
504
+
501
505
  ## FAQ
502
506
 
503
507
  ### Do I need an API key?
package/README.zh-CN.md CHANGED
@@ -9,7 +9,7 @@
9
9
 
10
10
  <p align="center">
11
11
  <a href="https://www.npmjs.com/package/@fffangchenghao/dsh-web-search"><img src="https://img.shields.io/npm/v/@fffangchenghao/dsh-web-search.svg" alt="npm 版本"></a>
12
- <img src="https://img.shields.io/badge/tests-466%20passing-brightgreen" alt="测试">
12
+ <img src="https://img.shields.io/badge/tests-474%20passing-brightgreen" alt="测试">
13
13
  <img src="https://img.shields.io/badge/engines-8%2C%20zero%20API%20key-brightgreen" alt="引擎">
14
14
  <img src="https://img.shields.io/badge/node-%3E%3D22.18-blue" alt="node">
15
15
  <img src="https://img.shields.io/badge/dsh-%3E%3D0.2.1--rc.2%20%3C0.3.0-blueviolet" alt="dsh">
@@ -21,7 +21,7 @@
21
21
  - **不要 API key,八个引擎。** AnySearch、必应中国、DuckDuckGo HTML、必应国际、DuckDuckGo Lite、360 搜索直接抓取;Exa 走它的免密钥 MCP 通道加入主引擎,Firecrawl 的搜索 API 作为实测后的兜底。装好就能用。
22
22
  - **排序是量出来的,不是写出来的。** 跨引擎 RRF 融合,叠加权威度与时效加权;20 条离线基准 + 81 组参数扫描,是这类插件里唯一同时公开指标和置信区间的。方法见 [排序是怎么做的](docs/rrf-ranking.md)。
23
23
  - **抓取会承认失败。** 二进制文档和「200 但正文为空」会被明确拒绝(`SEARCH_UNSUPPORTED`、`SEARCH_EMPTY`),而不是当噪音返回;direct → tunnel → render → extract 四级阶梯共享 25 秒预算,预算不足就不再起新的一级。预算是「每次请求」而不是「插件启动时」,这点有回归测试守着。
24
- - **28 个测试文件、466 个用例**,`lib/` 产物入库,声明了 `engines.dsh` / `engines.node`,[SECURITY.md](SECURITY.md) 逐项列出插件会碰的进程、主机和文件。
24
+ - **29 个测试文件、474 个用例**,`lib/` 产物入库,声明了 `engines.dsh` / `engines.node`,[SECURITY.md](SECURITY.md) 逐项列出插件会碰的进程、主机和文件。
25
25
  - **双语文档。** 本文件和 [English README](README.md) 同步维护。
26
26
 
27
27
  ![排序基线参数扫描](https://raw.githubusercontent.com/fangchenghao/dsh-web-search/main/docs/images/bench.png)
@@ -37,7 +37,7 @@ DSH 插件生态里的「联网搜索」这一块已经很挤,大多数竞品
37
37
  | 免 key 引擎 | 8 | ~30 | 7 条通道(含 X) | 1 个官方 API |
38
38
  | 专门数据源 | 34(默认开 8 个)| 41 | — | — |
39
39
  | 面向模型的工具 | 6 | ~10 | 3 | 5 |
40
- | 测试 | 28 文件 / 466 用例 | 无 | 29 文件 | 6 文件 |
40
+ | 测试 | 29 文件 / 474 用例 | 无 | 29 文件 | 6 文件 |
41
41
  | 排序基准 | 20 查询 + 81 组扫描 | — | — | — |
42
42
  | 商店状态 | 源码契约已满足,待复核 | unlisted | 无条目 | blocked(review-required) |
43
43
 
@@ -417,7 +417,7 @@ pnpm install
417
417
 
418
418
  ## 设置页怎么用
419
419
 
420
- 打开 设置 → 「网页搜索(Bing)」。页面上一共 26 个控件,按五组排列。每组只管一类事:
420
+ 打开 设置 → 「网页搜索(Bing)」。页面上一共 26 个控件,按五组排列。顶部有一条随滚动固定的跳转栏,点组名直接跳过去;标题下还会拿当前版本和 npm 上的最新版比一下(连不上 npm 时这行不显示,不会瞎猜)。每组只管一类事:
421
421
 
422
422
  | 组 | 控件数 | 这组管什么 |
423
423
  | --- | --- | --- |
@@ -435,7 +435,7 @@ pnpm install
435
435
  | 等多久 | `wait` | `auto` | `auto` 按提问语言等;`all` 等全部;`fast` 谁先回用谁 |
436
436
  | 等待上限(毫秒) | `timeoutMs` | `15000` | 单个引擎最多等多久,最小 500 |
437
437
  | 代理(中转站) | `proxy` | 空(自动探测) | 如 `http://127.0.0.1:7890`;留空先读环境变量再读系统代理 |
438
- | AnySearch 通行证 | `anysearchApiKey` | 空(匿名) | 可选项。只发给 `api.anysearch.com`,用于提高额度、解锁正文抽取 |
438
+ | AnySearch 通行证 | `anysearchApiKey` | 空(匿名) | 可选项。只发给 `api.anysearch.com`,用于提高额度、解锁正文抽取。可以用逗号分隔填多个 key,插件轮流使用;某个被限流或拒绝会自动休息 |
439
439
  | 专业领域 | `anysearchDomain` | 空(通用) | 常见取值 `finance` / `academic` / `security` / `legal` / `code`,可带子域如 `finance.us_stock` |
440
440
 
441
441
  ### 结果与排序(7 个)
package/client.js CHANGED
@@ -46,6 +46,8 @@
46
46
  /** Loader entry id the Host projects this plugin's Config under. */
47
47
  var ENTRY = 'web-search-bing';
48
48
  var PACKAGE_NAME = 'dsh-web-search-bing';
49
+ /** Keep in step with package.json; compared with the npm registry. */
50
+ var VERSION = '0.16.0';
49
51
  /** Snapshot used while no configuration service is reachable. */
50
52
  var EMPTY_STATE = { status: 'unavailable', value: undefined, base: undefined, user: undefined, revision: undefined, writable: false };
51
53
 
@@ -53,6 +55,10 @@
53
55
  var en = {
54
56
  title: 'Web search (Bing)',
55
57
  description: 'Edit the key-free search provider: engine allow-list, wait policy, proxy, result budget, ranking, cache, cooldowns and dedicated sources.',
58
+ jumpTo: 'Jump to',
59
+ latest: 'Latest on npm',
60
+ upToDate: 'up to date',
61
+ updateAvailable: 'update available',
56
62
  'group.engines': 'Engines and transport',
57
63
  'group.ranking': 'Results and ranking',
58
64
  'group.cache': 'Cache and cooldowns',
@@ -118,7 +124,7 @@
118
124
  'field.cacheFile': 'Cache file',
119
125
  'hint.cacheFile': 'Empty uses ~/.dsh/cache/dsh-web-search/cache.json',
120
126
  'field.anysearchApiKey': 'AnySearch key',
121
- 'hint.anysearchApiKey': 'Leave empty to stay anonymous at a lower rate limit; only sent to api.anysearch.com',
127
+ 'hint.anysearchApiKey': 'Leave empty to stay anonymous at a lower rate limit; only sent to api.anysearch.com. Separate several keys with commas and the plugin rotates them.',
122
128
  'field.anysearchDomain': 'Vertical domain',
123
129
  'hint.anysearchDomain': 'Restrict to one vertical (finance, academic, code…); leave empty normally',
124
130
  'field.baseCooldownMs': 'Cooldown (ms)',
@@ -137,6 +143,10 @@
137
143
  // Simplified Chinese copy.
138
144
  var zh = {
139
145
  title: '网页搜索(Bing)',
146
+ jumpTo: '跳到',
147
+ latest: 'npm 最新版',
148
+ upToDate: '已是最新',
149
+ updateAvailable: '有新版本',
140
150
  description: '编辑免密钥搜索提供方:引擎白名单、等待策略、代理、结果上限、排序、缓存、冷却与专用检索源。',
141
151
  'group.engines': '引擎与传输',
142
152
  'group.ranking': '结果与排序',
@@ -203,7 +213,7 @@
203
213
  'field.cacheFile': '缓存文件路径',
204
214
  'hint.cacheFile': '留空使用 ~/.dsh/cache/dsh-web-search/cache.json',
205
215
  'field.anysearchApiKey': 'AnySearch 通行证',
206
- 'hint.anysearchApiKey': '没有就留空,仍可用但速率较低;只发给 api.anysearch.com',
216
+ 'hint.anysearchApiKey': '没有就留空,仍可用但速率较低;只发给 api.anysearch.com。有多个 key 用逗号分隔,插件会自动轮换。',
207
217
  'field.anysearchDomain': '专业领域',
208
218
  'hint.anysearchDomain': '只搜某类专业内容(finance/academic/code 等);日常留空',
209
219
  'field.baseCooldownMs': '被拒后歇多久',
@@ -466,6 +476,27 @@
466
476
  var errorPair = React.useState({});
467
477
  var errors = errorPair[0];
468
478
  var setErrors = errorPair[1];
479
+ var latestPair = React.useState(null);
480
+ var latest = latestPair[0];
481
+ var setLatest = latestPair[1];
482
+
483
+ // Ask the npm registry once per mount. A failure (offline, proxy
484
+ // down, registry error) hides the line instead of showing an
485
+ // invented version; the page must never lie about an update.
486
+ React.useEffect(function () {
487
+ var alive = true;
488
+ try {
489
+ fetch('https://registry.npmjs.org/@fffangchenghao%2Fdsh-web-search/latest', { headers: { accept: 'application/json' } })
490
+ .then(function (response) { return response.ok ? response.json() : null; })
491
+ .then(function (body) {
492
+ if (alive && body !== null && typeof body.version === 'string') setLatest(body.version);
493
+ })
494
+ .catch(function () {});
495
+ } catch (ignored) {
496
+ /* fetch unavailable: the page simply omits the line */
497
+ }
498
+ return function () { alive = false; };
499
+ }, []);
469
500
 
470
501
  React.useEffect(function () {
471
502
  setDraft(null);
@@ -526,6 +557,14 @@
526
557
  h('h3', { key: 'title', style: { margin: 0, fontSize: '16px', lineHeight: 1.4, color: 'var(--dsw-alias-label-primary)' } }, t('title')),
527
558
  h('p', { key: 'description', style: { margin: 0, fontSize: '13px', lineHeight: 1.5, color: 'var(--dsw-alias-label-tertiary)' } }, t('description'))
528
559
  ];
560
+ if (latest !== null) {
561
+ var newer = latest !== VERSION;
562
+ children.push(h('p', {
563
+ key: 'latest',
564
+ style: { margin: 0, fontSize: '12px', color: newer ? 'var(--dsw-alias-state-warn-primary)' : 'var(--dsw-alias-label-tertiary)' },
565
+ role: 'status'
566
+ }, t('latest') + ': ' + latest + (newer ? ' — ' + t('updateAvailable') : ' — ' + t('upToDate'))));
567
+ }
529
568
 
530
569
  if (!ready) {
531
570
  if (!surface.hasService()) {
@@ -545,8 +584,36 @@
545
584
  children.push(h('p', { key: 'readonly', style: { margin: 0, fontSize: '12px', color: 'var(--dsw-alias-label-tertiary)' } }, t('readOnly')));
546
585
  }
547
586
 
587
+ var jump = GROUPS.map(function (group) {
588
+ return h('a', {
589
+ key: group.key,
590
+ href: '#dsh-ws-group-' + group.key,
591
+ onClick: function (event) {
592
+ event.preventDefault();
593
+ try {
594
+ var target = document.getElementById('dsh-ws-group-' + group.key);
595
+ if (target !== null) target.scrollIntoView({ behavior: 'smooth', block: 'start' });
596
+ } catch (ignored) {
597
+ /* older engines still jump through the href */
598
+ }
599
+ },
600
+ style: { fontSize: '12px', color: 'var(--dsw-alias-label-secondary)', textDecoration: 'none', whiteSpace: 'nowrap' }
601
+ }, t('group.' + group.key));
602
+ });
603
+ children.push(h('nav', {
604
+ key: 'toc',
605
+ 'aria-label': t('jumpTo'),
606
+ style: {
607
+ position: 'sticky', top: 0, zIndex: 1,
608
+ display: 'flex', alignItems: 'center', gap: '10px', flexWrap: 'wrap',
609
+ padding: '6px 0',
610
+ background: 'var(--dsw-alias-bg-base)',
611
+ borderBottom: '1px solid var(--dsw-alias-border-l1)'
612
+ }
613
+ }, [h('span', { key: 'label', style: { fontSize: '11px', color: 'var(--dsw-alias-label-tertiary)' } }, t('jumpTo'))].concat(jump)));
614
+
548
615
  children.push(h('div', { key: 'groups', style: { display: 'flex', flexDirection: 'column', gap: '12px' } }, GROUPS.map(function (group) {
549
- return h('section', { key: group.key, style: groupStyle }, [
616
+ return h('section', { key: group.key, id: 'dsh-ws-group-' + group.key, style: groupStyle }, [
550
617
  h('h4', { key: 'title', style: { margin: 0, fontSize: '13px', lineHeight: 1.4, color: 'var(--dsw-alias-label-primary)' } }, t('group.' + group.key)),
551
618
  h('div', { key: 'grid', style: gridStyle }, group.fields.map(function (field) {
552
619
  var key = field.key;
package/lib/config.js CHANGED
@@ -92,7 +92,7 @@ export const Config = z.object({
92
92
  anysearchApiKey: z
93
93
  .string()
94
94
  .default('')
95
- .description('AnySearch 通行证(API Key):没有就留空,能用但慢;填了更稳且只发 api.anysearch.com。').volatile(),
95
+ .description('AnySearch 通行证(API Key):留空则匿名(慢一些);填了更稳且只发 api.anysearch.com。有多个 key 用逗号分隔,插件会轮流使用,某个被限流就自动换下一个。').volatile(),
96
96
  anysearchDomain: z
97
97
  .string()
98
98
  .default('')
package/lib/engines.js CHANGED
@@ -5,6 +5,7 @@ import { parseAnySearch, parseBing, parseDdgHtml, parseDdgLite, parseExaMcp, par
5
5
  import { mergeRegions } from "./rank.js";
6
6
  import { cacheKey } from "./cache.js";
7
7
  import { searchHost } from "./transport.js";
8
+ import { keyFailureKind, keyPoolFor } from "./keys.js";
8
9
  /** How many rows to ask AnySearch for; the merge picks from them. */
9
10
  // AnySearch caps max_results at 10: asking for more returns 10 with HTTP
10
11
  // 200, so the constant matches the real ceiling instead of over-asking.
@@ -172,6 +173,51 @@ const SOFT_DEADLINE = {
172
173
  function leadsQuery(host, lead) {
173
174
  return host.region === 'both' || host.region === lead;
174
175
  }
176
+ /**
177
+ * Ask one engine, rotating to the next key when the failure blames the key.
178
+ *
179
+ * A 401 or 429 from the one credentialed engine is the only failure a second
180
+ * attempt can fix; an engine that timed out or lost the network will fail the
181
+ * same way with a different token. At most two keys are tried per engine per
182
+ * tier, which bounds the added latency to one extra request.
183
+ * @param host - the engine definition.
184
+ * @param query - the user query.
185
+ * @param signal - caller cancellation.
186
+ * @param timeoutMs - per-engine deadline.
187
+ * @param proxy - tunnel URL, or `undefined`.
188
+ * @param timeRange - active publication window.
189
+ * @param pool - configured keys, possibly empty.
190
+ * @param domain - AnySearch vertical domain, if configured.
191
+ * @returns the last attempt's outcome.
192
+ */
193
+ async function searchHostWithKeys(host, query, signal, timeoutMs, proxy, timeRange, pool, domain) {
194
+ if (host.id !== 'anysearch' || pool.size === 0) {
195
+ return searchHost(host, query, signal, timeoutMs, proxy, timeRange);
196
+ }
197
+ let key = pool.pick();
198
+ let last;
199
+ for (let attempt = 0; attempt < 2 && key !== undefined; attempt += 1) {
200
+ const configured = resolveHost(host, {
201
+ anysearchApiKey: key,
202
+ ...(domain === undefined ? {} : { anysearchDomain: domain }),
203
+ });
204
+ const outcome = await searchHost(configured, query, signal, timeoutMs, proxy, timeRange);
205
+ if (outcome.ok) {
206
+ pool.reward(key);
207
+ return outcome;
208
+ }
209
+ last = outcome;
210
+ const kind = keyFailureKind(outcome.error);
211
+ if (kind === undefined || signal?.aborted === true)
212
+ return outcome;
213
+ pool.penalize(key, kind);
214
+ const next = pool.pick();
215
+ if (next === undefined || next === key)
216
+ return outcome;
217
+ key = next;
218
+ }
219
+ return last;
220
+ }
175
221
  /**
176
222
  * Run one tier concurrently and return the answers that arrived in time.
177
223
  * @param hosts - engines of this tier, in priority order.
@@ -184,7 +230,7 @@ function leadsQuery(host, lead) {
184
230
  * @param health - cooldown tracker to update, when one is configured.
185
231
  * @returns one answer per engine that produced sources.
186
232
  */
187
- async function runTier(hosts, query, signal, timeoutMs, failures, proxy, limits, health) {
233
+ async function runTier(hosts, query, signal, timeoutMs, failures, proxy, limits, health, pool, anysearchDomain) {
188
234
  const counters = limits.lead === 'all' || !hosts.some((host) => leadsQuery(host, limits.lead))
189
235
  ? () => true
190
236
  : (host) => leadsQuery(host, limits.lead);
@@ -215,7 +261,7 @@ async function runTier(hosts, query, signal, timeoutMs, failures, proxy, limits,
215
261
  releasedEarly = early;
216
262
  released();
217
263
  };
218
- const running = hosts.map((host, index) => searchHost(host, query, controller.signal, timeoutMs, proxy, limits.timeRange).then((outcome) => {
264
+ const running = hosts.map((host, index) => searchHostWithKeys(host, query, controller.signal, timeoutMs, proxy, limits.timeRange, pool, anysearchDomain).then((outcome) => {
219
265
  if (outcome.ok) {
220
266
  outcomes[index] = { host, sources: outcome.sources };
221
267
  health?.noteSuccess(host.id);
@@ -317,10 +363,12 @@ export async function search(query, signal, options) {
317
363
  const wait = options.wait ?? 'auto';
318
364
  const timeRange = options.timeRange ?? 'off';
319
365
  const proxy = options.useProxy === false ? undefined : await resolveProxy(options.proxy);
366
+ const pool = keyPoolFor(options.anysearchApiKey);
367
+ const anysearchDomain = (options.anysearchDomain ?? '').trim();
320
368
  const allow = options.hostIds !== undefined && options.hostIds.length > 0 ? new Set(options.hostIds) : undefined;
321
369
  const hosts = HOSTS.filter((host) => allow === undefined || allow.has(host.id))
322
370
  .filter((host) => host.viaProxy !== true || proxy !== undefined)
323
- .map((host) => resolveHost(host, options));
371
+ .map((host) => resolveHost(host, options.anysearchDomain === undefined ? {} : { anysearchDomain: options.anysearchDomain }));
324
372
  // Skip engines that are cooling down, but never end up with no engine at all:
325
373
  // a throttled engine is still better than no search.
326
374
  const now = Date.now();
@@ -385,13 +433,13 @@ export async function search(query, signal, options) {
385
433
  const primary = ordered.filter((host) => host.tier === 'primary');
386
434
  const fallback = ordered.filter((host) => host.tier === 'fallback');
387
435
  let tier = 'none';
388
- let answers = applyWindow(await runTier(primary, query, signal, timeoutMs, failures, proxy, limits, options.health));
436
+ let answers = applyWindow(await runTier(primary, query, signal, timeoutMs, failures, proxy, limits, options.health, pool, anysearchDomain));
389
437
  if (answers.length > 0)
390
438
  tier = 'primary';
391
439
  let sources = answers.length === 0 ? [] : preferWindow(mergeRegions(answers, lead, limit, rank), timeRange);
392
440
  if (sources.length === 0 && fallback.length > 0) {
393
441
  tier = 'fallback';
394
- answers = applyWindow(await runTier(fallback, query, signal, timeoutMs, failures, proxy, limits, options.health));
442
+ answers = applyWindow(await runTier(fallback, query, signal, timeoutMs, failures, proxy, limits, options.health, pool, anysearchDomain));
395
443
  sources = answers.length === 0 ? [] : preferWindow(mergeRegions(answers, lead, limit, rank), timeRange);
396
444
  }
397
445
  if (signal?.aborted === true)
package/lib/fetch.js CHANGED
@@ -33,6 +33,7 @@
33
33
  * @module @fangchenghao/dsh-web-search/fetch
34
34
  */
35
35
  import { SearchError } from "./errors.js";
36
+ import { keyFailureKind, keyPoolFor } from "./keys.js";
36
37
  import { DEFAULT_FETCH_MAX_LENGTH, renderFetchText, truncateFetchText } from "./markdown.js";
37
38
  import { proxyGet, resolveProxy } from "./proxy.js";
38
39
  import { DEFAULT_RENDER_TIMEOUT_MS, describeRenderError, detectShellPage, loadPlaywrightRenderer, readableTextLength, redactRenderReason, renderWithPlaywright, } from "./render.js";
@@ -155,6 +156,45 @@ async function extractFetch(url, apiKey, timeoutMs, signal) {
155
156
  const title = typeof payload.data.title === 'string' && payload.data.title.length > 0 ? `# ${payload.data.title}\n\n` : '';
156
157
  return { url, statusCode: 200, body: { kind: 'text', content: `${title}${payload.data.content}` }, truncated: false };
157
158
  }
159
+ /**
160
+ * Run AnySearch extraction, rotating to the next key when the failure blames
161
+ * the key rather than the page.
162
+ *
163
+ * A 401 means this key will never work again; a 429 means its quota needs a
164
+ * rest. Both are key-level facts the next fetch must remember, which is why the
165
+ * pool outlives this call instead of living in the provider closure.
166
+ * @param extract - transport hook.
167
+ * @param url - target page.
168
+ * @param pool - configured keys.
169
+ * @param budgetMs - remaining overall budget for this rung.
170
+ * @param signal - caller cancellation.
171
+ * @returns the extraction outcome.
172
+ */
173
+ async function extractWithKeys(extract, url, pool, budgetMs, signal) {
174
+ let key = pool.pick();
175
+ if (key === undefined)
176
+ throw new SearchError('every configured AnySearch key is resting', 'SEARCH_THROTTLED', true);
177
+ let last;
178
+ for (let attempt = 0; attempt < 2 && key !== undefined; attempt += 1) {
179
+ try {
180
+ const outcome = await extract(url, key, budgetMs, signal);
181
+ pool.reward(key);
182
+ return outcome;
183
+ }
184
+ catch (error) {
185
+ last = error;
186
+ const kind = keyFailureKind(error instanceof SearchError ? error : { message: error instanceof Error ? error.message : String(error) });
187
+ if (kind === undefined || signal?.aborted === true)
188
+ throw error;
189
+ pool.penalize(key, kind);
190
+ const next = pool.pick();
191
+ if (next === undefined || next === key)
192
+ throw error;
193
+ key = next;
194
+ }
195
+ }
196
+ throw last;
197
+ }
158
198
  /**
159
199
  * Build the fetch provider.
160
200
  * @param options - timeouts, tunnel and extraction settings.
@@ -174,7 +214,7 @@ export function createFetchProvider(options = {}, overrides = {}) {
174
214
  const timeoutMs = options.timeoutMs ?? DEFAULT_TIMEOUT_MS;
175
215
  const maxBytes = options.maxBytes ?? DEFAULT_MAX_BYTES;
176
216
  const userAgent = options.userAgent ?? USER_AGENT;
177
- const apiKey = (options.anysearchApiKey ?? '').trim();
217
+ const keyPool = keyPoolFor(options.anysearchApiKey);
178
218
  /**
179
219
  * Resolve the two volatile Settings fields per request. Reading them here
180
220
  * rather than at construction is what lets a saved edit apply without a
@@ -430,12 +470,12 @@ export function createFetchProvider(options = {}, overrides = {}) {
430
470
  if (!step.done)
431
471
  failures.push(`render: ${step.report.reason}`);
432
472
  }
433
- if (apiKey.length > 0) {
473
+ if (keyPool.size > 0) {
434
474
  try {
435
475
  const extractBudget = Math.min(timeoutMs, remaining());
436
476
  if (extractBudget < MIN_RUNG_MS)
437
477
  throw new Error('no time left in the fetch budget');
438
- const extracted = await deps.extract(target, apiKey, extractBudget, signal);
478
+ const extracted = await extractWithKeys(deps.extract, target, keyPool, extractBudget, signal);
439
479
  // Say where the body came from when the browser was tried first.
440
480
  const report = lastReport !== undefined && lastReport.attempted && !lastReport.applied
441
481
  ? makeReport(lastReport.mode, true, false, `${lastReport.reason}; the result came from AnySearch extraction`)
package/lib/keys.d.ts ADDED
@@ -0,0 +1,101 @@
1
+ /**
2
+ * Credential rotation for the one optional key this plugin understands.
3
+ *
4
+ * AnySearch is the only engine with a credential, and it is optional: without a
5
+ * key the plugin stays anonymous at a lower rate limit. A user who does hold
6
+ * several keys should not have to edit Settings when one runs out mid-day, so
7
+ * the plugin accepts a list and rotates through it. A key that answers 401 is
8
+ * retired until Settings change; a key that answers 429 rests on an exponential
9
+ * schedule, exactly like a throttled engine.
10
+ *
11
+ * The pool lives outside the search run because its whole point is memory: the
12
+ * second request must know that the first key ran out. Pools are cached by the
13
+ * raw Settings string, so editing the field creates a fresh pool and reverting
14
+ * it restores the old one.
15
+ *
16
+ * @module @fangchenghao/dsh-web-search/keys
17
+ */
18
+ /** Why a key was set aside. */
19
+ export type KeyFailureKind = 'unauthorized' | 'throttled' | 'blocked';
20
+ /** The part of an error this module classifies. */
21
+ export interface KeyFailureLike {
22
+ code?: string;
23
+ message?: string;
24
+ }
25
+ /** Tunable cooldowns. */
26
+ export interface KeyPoolOptions {
27
+ /** Clock, for tests. */
28
+ now?: () => number;
29
+ /** How long a rejected key is retired, in ms. */
30
+ unauthorizedCooldownMs?: number;
31
+ /** First throttle rest, doubling per consecutive throttle. */
32
+ throttleBaseMs?: number;
33
+ /** Ceiling for the throttle rest. */
34
+ throttleMaxMs?: number;
35
+ /** How long a forbidden key rests. */
36
+ blockedCooldownMs?: number;
37
+ }
38
+ /**
39
+ * Split a Settings value into usable keys.
40
+ * @param raw - comma/newline/space separated keys, or a single key.
41
+ * @returns distinct keys, in the order given.
42
+ */
43
+ export declare function parseApiKeys(raw: unknown): string[];
44
+ /**
45
+ * Decide whether an error blames the key rather than the engine.
46
+ * @param error - classified search or fetch failure.
47
+ * @returns the kind of key failure, or `undefined` when the key is not at fault.
48
+ */
49
+ export declare function keyFailureKind(error: KeyFailureLike | undefined): KeyFailureKind | undefined;
50
+ /**
51
+ * A rotating set of keys with per-key cooldowns.
52
+ *
53
+ * `pick` never returns a resting key; when every key rests it returns
54
+ * `undefined` and callers fall back to anonymous access. That is deliberate: a
55
+ * rejected key retried immediately is how a temporary 429 becomes a longer
56
+ * ban.
57
+ */
58
+ export declare class KeyPool {
59
+ readonly keys: readonly string[];
60
+ private readonly states;
61
+ private readonly now;
62
+ private readonly unauthorizedMs;
63
+ private readonly throttleBaseMs;
64
+ private readonly throttleMaxMs;
65
+ private readonly blockedMs;
66
+ private cursor;
67
+ constructor(keys: readonly string[], options?: KeyPoolOptions);
68
+ /** Number of configured keys. */
69
+ get size(): number;
70
+ /** True when every configured key is resting. */
71
+ allCooling(now?: number): boolean;
72
+ /** Milliseconds until this key may be used again; 0 means now. */
73
+ cooldownLeft(key: string, now?: number): number;
74
+ /**
75
+ * Take the next available key, round-robin.
76
+ * @param now - clock override.
77
+ * @returns a usable key, or `undefined` when none is configured or all rest.
78
+ */
79
+ pick(now?: number): string | undefined;
80
+ /**
81
+ * Record that a key worked: clear its strikes and any rest.
82
+ * @param key - the key that answered.
83
+ */
84
+ reward(key: string): void;
85
+ /**
86
+ * Rest a key after a failure it caused.
87
+ * @param key - the rejected key.
88
+ * @param kind - why it was rejected.
89
+ * @param now - clock override.
90
+ * @returns milliseconds until the key may be tried again.
91
+ */
92
+ penalize(key: string, kind: KeyFailureKind, now?: number): number;
93
+ }
94
+ /**
95
+ * Get or create the pool for a Settings value.
96
+ * @param raw - the `anysearchApiKey` field.
97
+ * @returns a stable pool for this key list.
98
+ */
99
+ export declare function keyPoolFor(raw: string | undefined): KeyPool;
100
+ /** Drop cached pools; used by tests and after a Settings save. */
101
+ export declare function resetKeyPools(): void;
package/lib/keys.js ADDED
@@ -0,0 +1,163 @@
1
+ /**
2
+ * Credential rotation for the one optional key this plugin understands.
3
+ *
4
+ * AnySearch is the only engine with a credential, and it is optional: without a
5
+ * key the plugin stays anonymous at a lower rate limit. A user who does hold
6
+ * several keys should not have to edit Settings when one runs out mid-day, so
7
+ * the plugin accepts a list and rotates through it. A key that answers 401 is
8
+ * retired until Settings change; a key that answers 429 rests on an exponential
9
+ * schedule, exactly like a throttled engine.
10
+ *
11
+ * The pool lives outside the search run because its whole point is memory: the
12
+ * second request must know that the first key ran out. Pools are cached by the
13
+ * raw Settings string, so editing the field creates a fresh pool and reverting
14
+ * it restores the old one.
15
+ *
16
+ * @module @fangchenghao/dsh-web-search/keys
17
+ */
18
+ /** Seconds in a day, used for the 401 rest. */
19
+ const DAY_MS = 86_400_000;
20
+ /** Keys separated by anything a person might type. */
21
+ const SEPARATOR = /[\s,;]+/u;
22
+ /** Placeholder-looking values users paste from documentation. */
23
+ const PLACEHOLDER = /^(?:your[-_ ]?key|api[-_ ]?key|token|none|null|undefined|x{3,})$/iu;
24
+ /**
25
+ * Split a Settings value into usable keys.
26
+ * @param raw - comma/newline/space separated keys, or a single key.
27
+ * @returns distinct keys, in the order given.
28
+ */
29
+ export function parseApiKeys(raw) {
30
+ if (typeof raw !== 'string')
31
+ return [];
32
+ const seen = new Set();
33
+ const keys = [];
34
+ for (const part of raw.split(SEPARATOR)) {
35
+ const key = part.trim();
36
+ if (key.length === 0 || PLACEHOLDER.test(key) || seen.has(key))
37
+ continue;
38
+ seen.add(key);
39
+ keys.push(key);
40
+ }
41
+ return keys;
42
+ }
43
+ /**
44
+ * Decide whether an error blames the key rather than the engine.
45
+ * @param error - classified search or fetch failure.
46
+ * @returns the kind of key failure, or `undefined` when the key is not at fault.
47
+ */
48
+ export function keyFailureKind(error) {
49
+ if (error === undefined)
50
+ return undefined;
51
+ if (error.code === 'SEARCH_THROTTLED')
52
+ return 'throttled';
53
+ const message = error.message ?? '';
54
+ if (/HTTP 401|invalid api key|api key (?:is )?invalid|unauthori[sz]ed/iu.test(message))
55
+ return 'unauthorized';
56
+ if (/HTTP 403|forbidden/iu.test(message))
57
+ return 'blocked';
58
+ return undefined;
59
+ }
60
+ /**
61
+ * A rotating set of keys with per-key cooldowns.
62
+ *
63
+ * `pick` never returns a resting key; when every key rests it returns
64
+ * `undefined` and callers fall back to anonymous access. That is deliberate: a
65
+ * rejected key retried immediately is how a temporary 429 becomes a longer
66
+ * ban.
67
+ */
68
+ export class KeyPool {
69
+ keys;
70
+ states = new Map();
71
+ now;
72
+ unauthorizedMs;
73
+ throttleBaseMs;
74
+ throttleMaxMs;
75
+ blockedMs;
76
+ cursor = 0;
77
+ constructor(keys, options = {}) {
78
+ this.keys = [...keys];
79
+ this.now = options.now ?? Date.now;
80
+ this.unauthorizedMs = options.unauthorizedCooldownMs ?? DAY_MS;
81
+ this.throttleBaseMs = options.throttleBaseMs ?? 60_000;
82
+ this.throttleMaxMs = options.throttleMaxMs ?? 1_800_000;
83
+ this.blockedMs = options.blockedCooldownMs ?? 600_000;
84
+ }
85
+ /** Number of configured keys. */
86
+ get size() {
87
+ return this.keys.length;
88
+ }
89
+ /** True when every configured key is resting. */
90
+ allCooling(now = this.now()) {
91
+ return this.keys.length > 0 && this.keys.every((key) => this.cooldownLeft(key, now) > 0);
92
+ }
93
+ /** Milliseconds until this key may be used again; 0 means now. */
94
+ cooldownLeft(key, now = this.now()) {
95
+ const state = this.states.get(key);
96
+ return state === undefined ? 0 : Math.max(0, state.cooldownUntil - now);
97
+ }
98
+ /**
99
+ * Take the next available key, round-robin.
100
+ * @param now - clock override.
101
+ * @returns a usable key, or `undefined` when none is configured or all rest.
102
+ */
103
+ pick(now = this.now()) {
104
+ const count = this.keys.length;
105
+ if (count === 0)
106
+ return undefined;
107
+ for (let offset = 0; offset < count; offset += 1) {
108
+ const index = (this.cursor + offset) % count;
109
+ const key = this.keys[index];
110
+ if (this.cooldownLeft(key, now) === 0) {
111
+ this.cursor = (index + 1) % count;
112
+ return key;
113
+ }
114
+ }
115
+ return undefined;
116
+ }
117
+ /**
118
+ * Record that a key worked: clear its strikes and any rest.
119
+ * @param key - the key that answered.
120
+ */
121
+ reward(key) {
122
+ this.states.set(key, { cooldownUntil: 0, strikes: 0 });
123
+ }
124
+ /**
125
+ * Rest a key after a failure it caused.
126
+ * @param key - the rejected key.
127
+ * @param kind - why it was rejected.
128
+ * @param now - clock override.
129
+ * @returns milliseconds until the key may be tried again.
130
+ */
131
+ penalize(key, kind, now = this.now()) {
132
+ const previous = this.states.get(key);
133
+ const strikes = kind === 'throttled' ? (previous?.strikes ?? 0) + 1 : (previous?.strikes ?? 0);
134
+ const rest = kind === 'unauthorized'
135
+ ? this.unauthorizedMs
136
+ : kind === 'blocked'
137
+ ? this.blockedMs
138
+ : Math.min(this.throttleMaxMs, this.throttleBaseMs * 2 ** Math.max(0, strikes - 1));
139
+ this.states.set(key, { cooldownUntil: now + rest, strikes });
140
+ return rest;
141
+ }
142
+ }
143
+ /** Pools keyed by the exact Settings value, so an edit starts clean. */
144
+ const pools = new Map();
145
+ /**
146
+ * Get or create the pool for a Settings value.
147
+ * @param raw - the `anysearchApiKey` field.
148
+ * @returns a stable pool for this key list.
149
+ */
150
+ export function keyPoolFor(raw) {
151
+ const keys = parseApiKeys(raw ?? '');
152
+ const signature = keys.join('\n');
153
+ const existing = pools.get(signature);
154
+ if (existing !== undefined)
155
+ return existing;
156
+ const created = new KeyPool(keys);
157
+ pools.set(signature, created);
158
+ return created;
159
+ }
160
+ /** Drop cached pools; used by tests and after a Settings save. */
161
+ export function resetKeyPools() {
162
+ pools.clear();
163
+ }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@fffangchenghao/dsh-web-search",
3
- "version": "0.15.0",
3
+ "version": "0.16.0",
4
4
  "description": "DSH bundle: a key-free web search provider. Leads with the anonymous AnySearch API and falls back to public result pages (DuckDuckGo, Bing, 360), merged by reciprocal-rank fusion with a bilingual region policy, an engine cooldown and a TTL cache.",
5
5
  "type": "module",
6
6
  "main": "lib/index.js",
@@ -16,6 +16,7 @@
16
16
  },
17
17
  "files": [
18
18
  "lib",
19
+ "skills",
19
20
  "client.js",
20
21
  "locales/*.json",
21
22
  "cordis.patch.yml",
@@ -0,0 +1,47 @@
1
+ ---
2
+ name: dsh-web-search
3
+ description: Search the live web and read the pages behind the results when the DSH web-search plugin provides web_search, research, domain_search, search_many, read_page or web_fetch. Use it for current facts, version numbers, release notes, package metadata, or any answer that needs a citation, instead of guessing.
4
+ ---
5
+
6
+ # Live web research with dsh-web-search
7
+
8
+ The host exposes some or all of these tools. Use the narrowest one that answers
9
+ the question, and cite the URLs you actually read.
10
+
11
+ | Tool | Use it when |
12
+ | --- | --- |
13
+ | `web_search` | You need candidate URLs. One call returns up to eight de-duplicated rows with title, link and snippet. |
14
+ | `research` | The answer needs several sources or citations. One call searches up to three queries, opens the leading pages and returns an evidence pack with a URL under every excerpt. |
15
+ | `domain_search` | The question is about a package, repository, paper, Q&A thread or reference entry, and a structured record beats a web page. |
16
+ | `search_many` | You have 2–5 distinct questions that should run together instead of one at a time. |
17
+ | `read_page` | A page is longer than one fetch and you need the next chunk (pass the `nextOffset` from the previous call). |
18
+ | `web_fetch` | You already have the URL and need its readable text. |
19
+
20
+ ## How to ask
21
+
22
+ - Put the **distinctive words** in the query: an error string, a version, an
23
+ API name. "python requests timeout parameter" beats "how do I fix my request".
24
+ - Do not paste whole paragraphs; a search query is a few keywords, not a
25
+ document.
26
+ - One `web_search` per fact is usually enough. If the first page of rows does
27
+ not contain the answer, change a word rather than repeating the same query.
28
+ - Prefer official documentation, release notes and the project's own repository
29
+ over aggregator pages. The plugin already boosts known authority domains, but
30
+ it cannot read the page for you.
31
+
32
+ ## How to answer
33
+
34
+ - Quote the page, not the snippet: a snippet can be stale or truncated.
35
+ - For a factual claim, link the URL you read, not the search results page.
36
+ - When a source contradicts another, say so and show both links rather than
37
+ picking the more convenient one.
38
+ - If every fetch fails, report the URLs and the failure reasons; do not invent
39
+ content from the snippet.
40
+
41
+ ## Installing this skill elsewhere
42
+
43
+ This folder follows the plain `SKILL.md` convention, so it can be copied into a
44
+ harness that reads skills from disk (for example `~/.claude/skills/` or a
45
+ project's `skills/` directory). The plugin itself is a DSH plugin; other
46
+ harnesses need their own way to expose the tools above, but the guidance still
47
+ applies whenever equivalent search and fetch tools exist.