@240xu/dsh-websearch 2.7.3 → 2.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -115,6 +115,30 @@ pnpm install
115
115
 
116
116
  环境变量兜底:未在 credentials 配置时回退读同名环境变量;`DSH_UNIFIED_SEARCH_BACKENDS` 可逗号分隔强制指定启用集合。
117
117
 
118
+ ## v2.8.0 查全查多 | What's new in v2.8.0
119
+
120
+ 面向「查的多一点、查的全一点」的检索深度升级,全部 feature-gated、默认关:
121
+
122
+ ### 1. multiQuery 多查询 RRF 融合(`multiQueryEnabled`,默认关)
123
+
124
+ - 门控启发式(刻意从严):查询长度 > 60 字符,或含 `vs / and / 比较 / 对比 / 和` 分隔词,才触发;简单查询单扇出,行为与旧版完全一致(arXiv:2404.01037:盲目多查询会劣化)。
125
+ - 触发后派生 ≤3 个子查询(原查询永远参与融合且保持全权重),各自走完整 11 后端扇出,再用 RRF(k=60)融合:出现在多个子查询结果里的条目被提升,单列表独占条目按位次衰减;URL 去重、首见元数据保留。
126
+ - 遥测行合并各变体并截断至 12 条,避免淹没结果。
127
+
128
+ ### 2. deepCoverage 深度覆盖(`deepCoverage`,默认关)
129
+
130
+ - 每后端请求条数上调至 `ceil(maxResults × 1.5)`,全局去重后仍按 maxResults 截断——牺牲带宽换覆盖面。
131
+ - 后端能力适配(feature-detect,不破坏旧参数):Tavily 自动切 `search_depth: advanced`(2 credits,多 snippet/URL);SearXNG 类目扩为 `general,news,it`(Search API `categories` 逗号列表)。
132
+ - Exa:`inferExaCategory` 从查询推断 data category(github / research paper / news,仅高置信注入)。
133
+
134
+ ### 3. 运维可观测(P2)
135
+
136
+ - `GET /api/websearch/history` 响应新增 `backends` 字段:每后端最近真实成败/延迟/熔断状态(`{id, ok, ms, at, failCount, cooled}`,无错误串无凭证),兜底冗余度可观测。
137
+
138
+ ### 4. per-backend 超时上限(P2)
139
+
140
+ - `ddg` / `searxng` 专属超时上限 5s(`min(5s, backendTimeoutMs)`,全局默认仍 30s、配置更短则以配置为准)——不可达后端不再拖住整个扇出尾。实测见 docs/real-call-2.8.0.md。
141
+
118
142
  ## v2.7.3 兼容修复 | What's new in v2.7.3
119
143
 
120
144
  - **DSH 0.1.7 客户端兼容(P0)**:settingsScope 客户端服务在 0.1.7 被移除;现改为 feature-detect——0.1.5(或装 dsh-settings-scope-shim)行为不变,0.1.7 无 shim 时设置卡片优雅停用(console.warn 指引),客户端不再静默全死。
@@ -216,7 +240,9 @@ v2.7.0 缓存/熔断/历史三项的依据与刻意取舍(含查证过的反
216
240
  1. **熔断器(Circuit Breaker)**:出处 Michael T. Nygard《Release It!》(Pragmatic Bookshelf, 2007) 与 Microsoft Azure Architecture Center「Circuit Breaker pattern」https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker 。经典状态机是 Closed → Open → Half-Open(半开态放行少量探测请求)。**本插件刻意省略半开态**:冷却到期即视为 Closed(下一次真实搜索就是探测),失败则立即重新开窗——因为这里的"请求"是一次多后端扇出中的单个后端调用,探测请求必然真实发生且失败成本只是一次软降级,无需额外探测计数器;同时保留 fail-open 兜底(全部冷却时放行),比经典模式更保守。
217
241
  2. **缓存失效(TTL-only vs 事件驱动)**:Cloudflare「Retention vs Freshness (TTL)」https://developers.cloudflare.com/cache/concepts/retention-vs-freshness/ 与「Revalidation」https://developers.cloudflare.com/cache/concepts/revalidation/ 。搜索结果没有可靠的原站变更信号可订阅(网页索引持续变化),事件驱动失效无锚点;TTL-only + 把 TTL 做成设置项(默认 900s)+ key 纳入后端组合,是"可接受的最终一致性"。取舍:900s 内索引更新不可见——对搜索场景可接受,且用户可调低。
218
242
  3. **原子写(tmp + renameSync)**:POSIX 上 rename(2) 原子替换(Node 文档:newPath 已存在时覆盖);Windows 上 libuv 将 fs.rename 映射为 MoveFileExW(MOVEFILE_REPLACE_EXISTING)(libuv#283 https://github.com/joyent/libuv/issues/283 ,LWN 讨论 https://lwn.net/Articles/682988/ 指出 MS 文档不承诺该调用的原子性)。**结论:POSIX 双端原子;Windows 实践上等价但官方文档不背书**——因此本插件在写入端额外做了兜底:读端永远校验 JSON 完整性、坏条目按 miss 处理并删除,即使 Windows 上出现半写状态也不会返回坏数据。
219
- 4. **搜索历史隐私**:Article 29 Working Party 意见书 WP148 https://ec.europa.eu/justice/article-29/documentation/opinion-recommendation/files/2008/wp148_en.pdf 认定查询日志(query + 时间 + 来源标识)属个人数据;EFF 六条自保建议 https://w2.eff.org/Privacy/search/searchtips.pdf 的第一条是"别在搜索词里放 PII"——说明 query 天然可能携带 PII(人名、账号、地址)。本插件的取舍:历史只存 {query, time, resultCount, backendsOk/Total}(无 IP、无 cookie、无结果 URL)、环形 50 条自动滚动即短保留、本机存储不出网、提供 historyEnabled 开关与 POST clear 端点。**未做 query 脱敏**:脱敏会破坏"查看最近搜了什么"的核心用途,且本机单用户场景下该数据本就在 DSH 会话日志里存在;若未来历史跨设备同步,脱敏/加密是上线前置条件。
243
+ 4b. **查全查多(v2.8.0)**:Tavily Search API(`search_depth: advanced` 2 credits、`max_results` 0-20、`include_answer`、topic=news 时间加权)https://docs.tavily.com/api-reference/endpoint/search ;Exa Search API(type 分类检索、category 枚举、livecrawl;注意 2026 版 coding-agent 指南声明 neural 为 legacy 术语、且告诫勿臆造 category 值——本插件只注入 OpenAPI 枚举内的高置信值)https://exa.ai/docs/reference/search-api-guide-for-coding-agents 与 https://github.com/exa-labs/openapi-spec/blob/master/exa-openapi-spec.yaml ;SearXNG Search API(`categories` 逗号多类目、`language`)https://docs.searxng.org/dev/search_api.html ;Brave Search API(`freshness`、`result_filter`,本轮仅调研未接线)https://api-dashboard.search.brave.com/app/documentation 。多查询融合:RRF 出自 Cormack et al. 2009,RAG-Fusion 实现 https://github.com/Raudaschl/rag-fusion ;HyDE https://arxiv.org/abs/2212.10496 ;**门控依据** arXiv:2404.01037(盲目多查询劣化)——因此 multiQuery 默认关、启发式从严、原查询全权重参与。
244
+ 4c. **per-backend 超时**:不可达主机的 TCP 连接失败不受应用层超时保护,只能靠 abort 计时器兜底;ddg(HTML 抓取)与 SearXNG(公共实例无 SLA)是拖尾主力,5s 上限取「P50 成功延迟的一个数量级」。
245
+ 4d. **搜索历史隐私**:Article 29 Working Party 意见书 WP148 https://ec.europa.eu/justice/article-29/documentation/opinion-recommendation/files/2008/wp148_en.pdf 认定查询日志(query + 时间 + 来源标识)属个人数据;EFF 六条自保建议 https://w2.eff.org/Privacy/search/searchtips.pdf 的第一条是"别在搜索词里放 PII"——说明 query 天然可能携带 PII(人名、账号、地址)。本插件的取舍:历史只存 {query, time, resultCount, backendsOk/Total}(无 IP、无 cookie、无结果 URL)、环形 50 条自动滚动即短保留、本机存储不出网、提供 historyEnabled 开关与 POST clear 端点。**未做 query 脱敏**:脱敏会破坏"查看最近搜了什么"的核心用途,且本机单用户场景下该数据本就在 DSH 会话日志里存在;若未来历史跨设备同步,脱敏/加密是上线前置条件。
220
246
 
221
247
  ## License | 许可
222
248
 
@@ -13,6 +13,22 @@ import { WebError } from "@deepseek-ai/dsh-web";
13
13
  const EXA_ENDPOINT = "https://mcp.exa.ai/mcp";
14
14
  const EXA_TOOL = "web_search_exa";
15
15
 
16
+ /**
17
+ * Infer an Exa data category from the query (v2.8.0). Only high-confidence
18
+ * patterns map — Exa rejects unknown categories with a 400, so this returns
19
+ * undefined unless a signal is explicit. Valid values per the Exa OpenAPI
20
+ * enum: github / "research paper" / news / pdf / company / people /
21
+ * "personal site" / "financial report".
22
+ */
23
+ export function inferExaCategory(query) {
24
+ const q = typeof query === "string" ? query.toLowerCase() : "";
25
+ if (!q) return undefined;
26
+ if (/\bgithub\b|\brepo\b|repository|开源项目|源码/.test(q)) return "github";
27
+ if (/\barxiv\b|\bpaper\b|论文|研究文献|preprint/.test(q)) return "research paper";
28
+ if (/\bnews\b|新闻|最新消息|动态/.test(q)) return "news";
29
+ return undefined;
30
+ }
31
+
16
32
  export const exaBackend = {
17
33
  id: "exa",
18
34
  requiresCredential: false,
@@ -26,6 +42,10 @@ export const exaBackend = {
26
42
  query: request.query,
27
43
  numResults: request.maxResults ?? 8,
28
44
  };
45
+ // v2.8.0: category inference (feature-detect — only sent when confident;
46
+ // if the endpoint rejects it the fan-out demotes softly).
47
+ const category = inferExaCategory(request.query);
48
+ if (category) args.category = category;
29
49
  hooks.recordRequest?.({
30
50
  endpoint: EXA_ENDPOINT,
31
51
  toolName: EXA_TOOL,
@@ -33,7 +33,9 @@ export const searxngBackend = {
33
33
  url.pathname = url.pathname.replace(/\/+$/,"") + "/search";
34
34
  url.searchParams.set("q", request.query);
35
35
  url.searchParams.set("format", "json");
36
- url.searchParams.set("categories", "general");
36
+ // deepCoverage widens the category sweep (docs.searxng.org Search API:
37
+ // categories is a comma-separated list) — breadth over one category.
38
+ url.searchParams.set("categories", request.deepCoverage === true ? "general,news,it" : "general");
37
39
  // Language: settings/request filter wins, else historical "en" default.
38
40
  const filters = normalizeFilters(request.filters);
39
41
  url.searchParams.set("language", filters.lang ?? "en");
@@ -31,7 +31,12 @@ export const tavilyBackend = {
31
31
 
32
32
  const baseURL = backendOpts.baseURL ?? TAVILY_ENDPOINT;
33
33
  const maxResults = Math.min(request.maxResults ?? 8, 20);
34
- const searchDepth = backendOpts.searchDepth ?? "basic";
34
+ // deepCoverage (provider sets request.deepCoverage) upgrades the depth:
35
+ // advanced = highest relevance, multiple snippets per URL (2 API credits).
36
+ // An explicit operator setting keeps precedence unless it was left basic.
37
+ const searchDepth = request.deepCoverage === true && backendOpts.searchDepth !== "advanced"
38
+ ? "advanced"
39
+ : (backendOpts.searchDepth ?? "basic");
35
40
 
36
41
  const body = {
37
42
  query: request.query,
package/lib/cache.js CHANGED
@@ -33,13 +33,14 @@ export const CACHE_DEFAULTS = {
33
33
  * mix must never serve a stale result produced by a different combination
34
34
  * (arch-review P1-1). Pure function; exported for tests and host reuse.
35
35
  */
36
- export function cacheKeyFor({ query, filters, maxResults, backends }) {
36
+ export function cacheKeyFor({ query, filters, maxResults, backends, mode }) {
37
37
  const backendSet = Array.isArray(backends) ? [...backends].sort().join(",") : "";
38
38
  const canonical = JSON.stringify({
39
39
  q: typeof query === "string" ? query : "",
40
40
  f: filters ?? {},
41
41
  n: Number(maxResults) || 0,
42
42
  b: backendSet,
43
+ m: typeof mode === "string" ? mode : "",
43
44
  });
44
45
  return createHash("sha256").update(canonical).digest("hex").slice(0, 32);
45
46
  }
package/lib/history.js CHANGED
@@ -115,13 +115,37 @@ export function isTrustedApiRequest(req) {
115
115
  }
116
116
  }
117
117
 
118
+ /**
119
+ * Project the module-level backend health map into a small observability
120
+ * payload (v2.8.0 P2): per-backend last real success/failure, latency and
121
+ * breaker state — no error strings, no credentials, safe for the GUI.
122
+ */
123
+ export function toBackendObservability(healthMap, nowMs = Date.now()) {
124
+ const out = [];
125
+ if (healthMap instanceof Map) {
126
+ for (const [id, e] of healthMap) {
127
+ out.push({
128
+ id,
129
+ ok: e?.ok === true,
130
+ ms: Number.isFinite(e?.ms) ? e.ms : null,
131
+ at: Number.isFinite(e?.at) ? e.at : null,
132
+ failCount: Number.isFinite(e?.failCount) ? e.failCount : 0,
133
+ cooled: Number.isFinite(e?.cooledUntil) && e.cooledUntil > nowMs,
134
+ });
135
+ }
136
+ }
137
+ return out;
138
+ }
139
+
118
140
  /**
119
141
  * Register the read-only history routes on the (late-mounted) webServer.
120
- * GET /api/websearch/history -> { ok, entries }
142
+ * GET /api/websearch/history -> { ok, entries, backends }
143
+ * backends: per-backend last real outcome/latency/breaker state (v2.8.0),
144
+ * so fallback redundancy is observable without a separate endpoint.
121
145
  * POST /api/websearch/history/clear -> { ok, cleared: true }
122
146
  * Both routes sit behind isTrustedApiRequest (loopback Host + same-site).
123
147
  */
124
- export function registerHistoryRoutes(ws, { getStore }) {
148
+ export function registerHistoryRoutes(ws, { getStore, getBackendHealth }) {
125
149
  if (!ws || typeof ws.register !== "function") return;
126
150
  const send = (res, code, body) => {
127
151
  res.writeHead(code, { "content-type": "application/json" });
@@ -144,7 +168,11 @@ export function registerHistoryRoutes(ws, { getStore }) {
144
168
  return;
145
169
  }
146
170
  const store = getStore();
147
- send(res, 200, { ok: true, entries: store ? store.list() : [] });
171
+ send(res, 200, {
172
+ ok: true,
173
+ entries: store ? store.list() : [],
174
+ backends: toBackendObservability(typeof getBackendHealth === "function" ? getBackendHealth() : getBackendHealth),
175
+ });
148
176
  }),
149
177
  });
150
178
  ws.register({
package/lib/index.js CHANGED
@@ -169,6 +169,12 @@ export const Config = z.object({
169
169
  historyEnabled: z.boolean().default(true)
170
170
  .description("记录搜索历史【历史】:只追加,最多 50 条;可在设置 → 搜索 中查看与清空"),
171
171
 
172
+ // v2.8.0 查全查多
173
+ multiQueryEnabled: z.boolean().default(false)
174
+ .description("多查询融合【查全】:复杂查询(长度>60 或含 vs/and/比较/对比/和)自动派生 2-3 个子查询,各扇出后用 RRF(k=60) 融合;默认关"),
175
+ deepCoverage: z.boolean().default(false)
176
+ .description("深度覆盖【查全】:每后端请求条数上调 50% 再全局去重截断,牺牲带宽换覆盖面;Tavily 自动切 advanced 深度,SearXNG 扩展多类目;默认关"),
177
+
172
178
  // Enable toggles (one per backend)
173
179
  enableExa: z.boolean().default(true).description(DEFAULT_LABELS.exa),
174
180
  enableParallel: z.boolean().default(true).description(DEFAULT_LABELS.parallel),
@@ -326,6 +332,9 @@ export function resolveOptions(ctx, config) {
326
332
  ? Math.max(0, Math.floor(Number(config?.breakerCooldownMs) || 0)) // 0 = disabled
327
333
  : BREAKER_DEFAULTS.cooldownMs,
328
334
  historyEnabled: config?.historyEnabled !== false,
335
+ // v2.8.0 coverage / multiQuery
336
+ multiQueryEnabled: config?.multiQueryEnabled === true,
337
+ deepCoverage: config?.deepCoverage === true,
329
338
  filters: {
330
339
  recency: typeof config?.recency === "string" && config.recency !== "any" ? config.recency : undefined,
331
340
  lang: language.length > 0 ? language : undefined,
@@ -423,6 +432,10 @@ export function apply(ctx, config) {
423
432
  getBackendHealth: getLatestBackendHealth,
424
433
  });
425
434
  // v2.7.0 search history: read-only GET + explicit clear POST.
426
- registerHistoryRoutes(webCtx.webServer, { getStore: () => historyStore });
435
+ // v2.8.0: GET also carries per-backend last-outcome observability.
436
+ registerHistoryRoutes(webCtx.webServer, {
437
+ getStore: () => historyStore,
438
+ getBackendHealth: getLatestBackendHealth,
439
+ });
427
440
  });
428
441
  }
@@ -0,0 +1,48 @@
1
+ // multiquery.js — v2.8.0 query-variant derivation (pure functions, zero deps).
2
+ //
3
+ // Gated multi-query: arXiv:2404.01037 (Wang et al.) shows BLIND multi-query
4
+ // can degrade retrieval — so variants are only derived for queries that look
5
+ // genuinely multi-faceted, and the original query always participates in the
6
+ // fusion with full weight. Splitting is purely mechanical (at separator
7
+ // words/punctuation); entity text is never rewritten.
8
+ //
9
+ // Separators per the v2.8 spec: "vs", "and", 比较, 对比, 和 — plus sentence
10
+ // punctuation for the long-query case.
11
+
12
+ const SEPARATOR_SPLIT_RE = /\s+(?:vs\.?|and)\s+|\s*(?:比较|对比|和)\s*|\s*[、;;。!?!?|]+\s*/i;
13
+ const COMPLEX_MARK_RE = /(?:\bvs\.?\b|\band\b|比较|对比|和)/i;
14
+
15
+ /** Sub-query hard cap (original + 2 variants max). */
16
+ export const MAX_SUB_QUERIES = 3;
17
+
18
+ /**
19
+ * Heuristic gate: does this query look multi-faceted enough to justify
20
+ * multi-query fan-out? Long queries (>60 chars) or queries containing a
21
+ * separator word qualify. Returns false for anything else — blind
22
+ * multi-query degrades (arXiv:2404.01037), so the default is NO.
23
+ */
24
+ export function isComplexQuery(query) {
25
+ if (typeof query !== "string") return false;
26
+ const q = query.trim();
27
+ if (q.length === 0) return false;
28
+ if (q.length > 60) return true;
29
+ return COMPLEX_MARK_RE.test(q);
30
+ }
31
+
32
+ /**
33
+ * Derive 2-3 sub-query variants from a complex query: the original query
34
+ * (always first, full weight in fusion) plus up to two mechanical splits at
35
+ * separator boundaries. Never rewrites entity text; returns [query] when no
36
+ * usable split exists.
37
+ */
38
+ export function deriveSubQueries(query) {
39
+ const base = typeof query === "string" ? query.trim() : "";
40
+ if (base.length === 0) return [];
41
+ const parts = base
42
+ .split(SEPARATOR_SPLIT_RE)
43
+ .map((p) => p.trim())
44
+ .filter((p) => p.length >= 2);
45
+ if (parts.length < 2) return [base];
46
+ const variants = [base, parts[0], parts.slice(1).join(" ")];
47
+ return [...new Set(variants.map((v) => v.trim()).filter(Boolean))].slice(0, MAX_SUB_QUERIES);
48
+ }
package/lib/provider.js CHANGED
@@ -25,9 +25,27 @@ import { normalizeFilters } from "./util/filters.js";
25
25
  import { shapeQuery, resultHeader, TELEMETRY_TAG } from "./prompting.js";
26
26
  import { cacheKeyFor } from "./cache.js";
27
27
  import { foldBreaker, isCooled, cooledAnnotation } from "./breaker.js";
28
+ import { rrfFuse } from "./rrf.js";
29
+ import { isComplexQuery, deriveSubQueries } from "./multiquery.js";
28
30
 
29
31
  const PROVIDER_ID = "unified";
30
32
 
33
+ // Per-backend timeout ceilings (v2.8.0, P2): ddg/scrapes and self-hosted
34
+ // SearXNG instances are the slowest to fail when unreachable — a 30s global
35
+ // default lets one dead backend stall the whole fan-out tail. The effective
36
+ // timeout is min(override, backendTimeoutMs), so an operator's shorter global
37
+ // timeout always wins.
38
+ const BACKEND_TIMEOUT_OVERRIDES_MS = {
39
+ ddg: 5000,
40
+ searxng: 5000,
41
+ };
42
+
43
+ /** Effective per-backend timeout = min(override ?? ∞, global). Exported for tests. */
44
+ export function effectiveBackendTimeoutMs(id, backendTimeoutMs) {
45
+ const override = BACKEND_TIMEOUT_OVERRIDES_MS[id];
46
+ return Number.isFinite(override) ? Math.min(override, backendTimeoutMs) : backendTimeoutMs;
47
+ }
48
+
31
49
  // Module-level per-backend health telemetry (id -> entry), updated on every
32
50
  // backend completion during a real search. Purely passive: nothing writes here
33
51
  // outside search(), so the read-only /api/unified-search/health route can
@@ -177,14 +195,25 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
177
195
  const concurrency = opts.concurrency ?? 6;
178
196
  const backendTimeoutMs = opts.backendTimeoutMs ?? 30000;
179
197
 
180
- // v2.7.0 disk cache: keyed on (shaped query + filters + maxResults).
181
- // A hit short-circuits the fan-out entirely; sources are byte-identical
182
- // to the first search and content carries a "cache hit, age Ns" note.
183
- // Cache failures never break the search (cache API never throws).
198
+ // v2.8.0 coverage / multiQuery gates (both default OFF; arXiv:2404.01037
199
+ // warns blind multi-query degrades, so the gate is deliberately strict).
200
+ const coverage = opts.deepCoverage === true;
201
+ // Per-backend widening: fetch more per backend, dedupe down afterwards.
202
+ const backendMaxResults = coverage ? Math.ceil(maxResults * 1.5) : maxResults;
203
+ const multi = opts.multiQueryEnabled === true && isComplexQuery(effReq.query);
204
+ const variants = multi ? deriveSubQueries(effReq.query) : [effReq.query];
205
+ // Cache-key mode marker: different coverage/multiQuery settings must not
206
+ // serve each other's cached results.
207
+ const cacheMode = (multi ? "multi" : "single") + (coverage ? "+coverage" : "");
208
+
209
+ // v2.7.0 disk cache: keyed on (shaped query + filters + maxResults +
210
+ // backends + mode). A hit short-circuits the fan-out entirely; sources
211
+ // are byte-identical to the first search and content carries a
212
+ // "cache hit, age Ns" note. Cache failures never break the search.
184
213
  const cacheEnabled = opts.cacheEnabled === true && cache;
185
214
  let cacheKey = null;
186
215
  if (cacheEnabled) {
187
- cacheKey = cacheKeyFor({ query: effReq.query, filters, maxResults, backends: opts.enabledBackends });
216
+ cacheKey = cacheKeyFor({ query: effReq.query, filters, maxResults, backends: opts.enabledBackends, mode: cacheMode });
188
217
  const hit = cache.get(cacheKey);
189
218
  if (hit && Array.isArray(hit.value?.sources) && hit.value.sources.length > 0) {
190
219
  // Defense in depth: the fresh path drops non-http(s) schemes before
@@ -219,7 +248,7 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
219
248
  // backend inside its 60s cooldown window is skipped (annotated in
220
249
  // telemetry as "id ⏸cooled"). Fail-open: if EVERY candidate is cooled,
221
250
  // run them all anyway — the breaker must never make search worse than
222
- // no breaker.
251
+ // no breaker. The pick is shared by every query variant.
223
252
  const breakerCfg = opts.breakerEnabled === false ? null : {
224
253
  threshold: opts.breakerThreshold,
225
254
  cooldownMs: opts.breakerCooldownMs,
@@ -255,98 +284,109 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
255
284
  );
256
285
  }
257
286
 
258
- // Settings-level default filters, overridden by per-request filters
259
- // (normalizeFilters drops unknown/invalid values so typos degrade to
260
- // "no filter" instead of breaking the search).
261
-
262
- const telemetry = [];
263
- // Concurrency limiter
264
- const limit = createLimiter(concurrency);
265
-
266
- // Run one backend under the per-backend timeout. The internal timer's
267
- // controller is merged into the caller's signal; when the INTERNAL timer
268
- // fired (caller did not abort), a backend abort-shaped failure is
269
- // reclassified as WEB_PROVIDER_ERROR so timeouts surface as real
270
- // failures instead of being silently demoted like user cancellations.
271
- const runBackend = ({ id, backend, beOpts, hooks }) => {
272
- const startedAtMs = Date.now();
273
- // Trust boundary: baseURL is operator-configurable and API keys are sent to it.
274
- // Enforce http(s); plain http only for private hosts (self-hosted SearXNG).
275
- // Reject embedded credentials. Failure is a per-backend soft error.
276
- if (beOpts.baseURL !== undefined) {
277
- let bu = null;
278
- try { bu = new URL(beOpts.baseURL); } catch { bu = null; }
279
- const privHost = bu ? /^(localhost|127(\.\d{1,3}){3}|10(\.\d{1,3}){3}|192\.168(\.\d{1,3}){3}|172\.(1[6-9]|2\d|3[01])(\.\d{1,3}){3})$/.test(bu.hostname) : false;
280
- const schemeOk = bu && (bu.protocol === "https:" || (bu.protocol === "http:" && privHost));
281
- if (!schemeOk || bu.username || bu.password) {
282
- throw new WebError(`backend "${id}" has an invalid baseURL: https required (plain http only for private hosts), credentials-in-URL rejected`, "WEB_PROVIDER_ERROR");
287
+ // Run ONE query variant through the full fan-out: breaker-aware backend
288
+ // execution under the per-variant limiter, then PHASE 1 collect. The
289
+ // internal abort/timeout semantics are unchanged from v2.7.
290
+ const fanoutVariant = async (variantReq) => {
291
+ const telemetry = [];
292
+ const limit = createLimiter(concurrency);
293
+ const runBackend = ({ id, backend, beOpts, hooks }) => {
294
+ const startedAtMs = Date.now();
295
+ // Trust boundary: baseURL is operator-configurable and API keys are sent to it.
296
+ // Enforce http(s); plain http only for private hosts (self-hosted SearXNG).
297
+ // Reject embedded credentials. Failure is a per-backend soft error.
298
+ if (beOpts.baseURL !== undefined) {
299
+ let bu = null;
300
+ try { bu = new URL(beOpts.baseURL); } catch { bu = null; }
301
+ const privHost = bu ? /^(localhost|127(\.\d{1,3}){3}|10(\.\d{1,3}){3}|192\.168(\.\d{1,3}){3}|172\.(1[6-9]|2\d|3[01])(\.\d{1,3}){3})$/.test(bu.hostname) : false;
302
+ const schemeOk = bu && (bu.protocol === "https:" || (bu.protocol === "http:" && privHost));
303
+ if (!schemeOk || bu.username || bu.password) {
304
+ throw new WebError(`backend "${id}" has an invalid baseURL: https required (plain http only for private hosts), credentials-in-URL rejected`, "WEB_PROVIDER_ERROR");
305
+ }
283
306
  }
284
- }
285
- const controller = new AbortController();
286
- const mergedSignal = signal
287
- ? AbortSignal.any([signal, controller.signal])
288
- : controller.signal;
289
- const timeoutId = setTimeout(() => controller.abort(), backendTimeoutMs);
290
- return backend
291
- .search(effReq, mergedSignal, beOpts, hooks)
292
- .then(
293
- (value) => {
294
- telemetry.push({ id, ok: true, ms: Date.now() - startedAtMs, count: value?.sources?.length ?? 0 });
295
- recordBackendHealth(id, { ok: true, at: Date.now(), ms: Date.now() - startedAtMs, count: value?.sources?.length ?? 0 }, breakerCfg);
296
- return value;
297
- },
298
- (err) => {
299
- const msg = String(err instanceof Error ? err.message : err).slice(0, 80);
300
- telemetry.push({ id, ok: false, ms: Date.now() - startedAtMs, error: msg });
301
- recordBackendHealth(id, { ok: false, at: Date.now(), ms: Date.now() - startedAtMs, error: msg }, breakerCfg);
307
+ const controller = new AbortController();
308
+ const mergedSignal = signal
309
+ ? AbortSignal.any([signal, controller.signal])
310
+ : controller.signal;
311
+ const effectiveTimeoutMs = effectiveBackendTimeoutMs(id, backendTimeoutMs);
312
+ const timeoutId = setTimeout(() => controller.abort(), effectiveTimeoutMs);
313
+ return backend
314
+ .search(variantReq, mergedSignal, beOpts, hooks)
315
+ .then(
316
+ (value) => {
317
+ telemetry.push({ id, ok: true, ms: Date.now() - startedAtMs, count: value?.sources?.length ?? 0 });
318
+ recordBackendHealth(id, { ok: true, at: Date.now(), ms: Date.now() - startedAtMs, count: value?.sources?.length ?? 0 }, breakerCfg);
319
+ return value;
320
+ },
321
+ (err) => {
322
+ const msg = String(err instanceof Error ? err.message : err).slice(0, 80);
323
+ telemetry.push({ id, ok: false, ms: Date.now() - startedAtMs, error: msg });
324
+ recordBackendHealth(id, { ok: false, at: Date.now(), ms: Date.now() - startedAtMs, error: msg }, breakerCfg);
325
+ throw err;
326
+ },
327
+ )
328
+ .catch((err) => {
329
+ if (controller.signal.aborted && !(signal && signal.aborted)) {
330
+ throw new WebError(
331
+ `backend "${id}" timed out after ${effectiveTimeoutMs}ms`,
332
+ "WEB_PROVIDER_ERROR",
333
+ { cause: err instanceof Error ? err : undefined },
334
+ );
335
+ }
302
336
  throw err;
303
- },
304
- )
305
- .catch((err) => {
306
- if (controller.signal.aborted && !(signal && signal.aborted)) {
307
- throw new WebError(
308
- `backend "${id}" timed out after ${backendTimeoutMs}ms`,
309
- "WEB_PROVIDER_ERROR",
310
- { cause: err instanceof Error ? err : undefined },
311
- );
312
- }
313
- throw err;
314
- })
315
- .finally(() => clearTimeout(timeoutId));
316
- };
317
-
318
- const results = await Promise.allSettled(
319
- eligible.map(({ id, backend, beOpts }) => {
320
- const hooks = {
321
- recordRequest: (req) => recordBackendRequest(ctx, id, req),
322
- recordOutcome: (out) => recordBackendOutcome(ctx, id, out),
323
- };
324
- return limit(() => runBackend({ id, backend, beOpts, hooks }));
325
- }),
326
- );
327
-
328
- throwIfSearchAborted(signal);
337
+ })
338
+ .finally(() => clearTimeout(timeoutId));
339
+ };
329
340
 
330
- // PHASE 1 collect: fold the settled fan-out into raw sources (fan-out
331
- // priority order), first backend content, and per-backend failures.
332
- const { collected, mergedContent, hardAborted, hardError, failures } =
333
- collectFanoutResults(results, eligible, signal);
341
+ const results = await Promise.allSettled(
342
+ eligible.map(({ id, backend, beOpts }) => {
343
+ const hooks = {
344
+ recordRequest: (req) => recordBackendRequest(ctx, id, req),
345
+ recordOutcome: (out) => recordBackendOutcome(ctx, id, out),
346
+ };
347
+ return limit(() => runBackend({ id, backend, beOpts, hooks }));
348
+ }),
349
+ );
350
+ throwIfSearchAborted(signal);
351
+ const fold = collectFanoutResults(results, eligible, signal);
352
+ return { ...fold, telemetry };
353
+ };
334
354
 
335
- // Caller-cancellation check (hard abort fires before we yield results).
336
- if (signal?.aborted === true || hardAborted) {
337
- throw searchAborted(signal);
355
+ // Fan out per variant (sequential — the limiter already parallelizes
356
+ // backends within a variant), dedupe/rank each list, then fuse.
357
+ const perVariant = [];
358
+ const allTelemetry = [];
359
+ const allFailures = [];
360
+ let hardError = null;
361
+ for (const variantQuery of variants) {
362
+ const variantReq = {
363
+ ...effReq,
364
+ query: variantQuery,
365
+ maxResults: backendMaxResults,
366
+ ...(coverage ? { deepCoverage: true } : {}),
367
+ };
368
+ const fold = await fanoutVariant(variantReq);
369
+ allTelemetry.push(...fold.telemetry);
370
+ allFailures.push(...fold.failures);
371
+ if (fold.hardError !== null && hardError === null) hardError = fold.hardError;
372
+ // Caller-cancellation check (hard abort fires before we yield results).
373
+ if (signal?.aborted === true || fold.hardAborted) {
374
+ throw searchAborted(signal);
375
+ }
376
+ // PHASE 2 dedupe / PHASE 3 rerank (per variant list, pre-fusion).
377
+ const strategy = opts.dedupStrategy === "url+title" ? "url+title" : "url";
378
+ const deduped = dedupeSources(fold.collected, strategy);
379
+ const ordered = opts.rerank === true ? rerankSources(deduped, variantQuery) : deduped;
380
+ perVariant.push({ query: variantQuery, ordered, mergedContent: fold.mergedContent });
338
381
  }
339
382
 
340
- // PHASE 2 dedupe / PHASE 3 rerank: "url" preserves the historical
341
- // URL-key behavior; "url+title"
342
- // additionally collapses same-story syndicated copies across backends.
343
- const strategy = opts.dedupStrategy === "url+title" ? "url+title" : "url";
344
- const deduped = dedupeSources(collected, strategy);
345
- // Optional deterministic relevance rerank (stable ties keep fan-out order).
346
- const ordered =
347
- opts.rerank === true
348
- ? rerankSources(deduped, typeof request?.query === "string" ? request.query : "")
349
- : deduped;
383
+ // PHASE 4 fuse: multi-query mode fuses the per-variant ranked lists with
384
+ // RRF (k=60); single-query mode keeps the historical behavior (the one
385
+ // list, unchanged ordering).
386
+ const orderedLists = perVariant.filter((v) => v.ordered.length > 0).map((v) => v.ordered);
387
+ const ordered = multi
388
+ ? rrfFuse(orderedLists).fused
389
+ : (orderedLists[0] ?? []);
350
390
 
351
391
  // If we got ANY sources, return them even if some backends failed.
352
392
  if (ordered.length > 0) {
@@ -355,7 +395,7 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
355
395
  // 置于结果块最前,要求逐条引用来源 URL、优先官方/一手与较新来源;
356
396
  // 语言按查询文本自动选择(含 CJK 用中文头)。可用
357
397
  // opts.resultHeader === false 关闭。
358
- let outContent = mergedContent;
398
+ let outContent = perVariant.find((v) => v.mergedContent !== undefined)?.mergedContent;
359
399
  if (opts.resultHeader !== false && (sources.length > 0 || outContent !== undefined)) {
360
400
  const header = resultHeader(typeof request?.query === "string" && /[\u4e00-\u9fff]/.test(request.query) ? "zh" : "en");
361
401
  outContent = outContent === undefined ? header : header + "\n\n" + outContent;
@@ -369,18 +409,22 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
369
409
  query: request?.query,
370
410
  time: Date.now(),
371
411
  resultCount: sources.length,
372
- backendsOk: telemetry.filter((t) => t.ok).length,
373
- backendsTotal: telemetry.length,
412
+ backendsOk: allTelemetry.filter((t) => t.ok).length,
413
+ backendsTotal: allTelemetry.length,
374
414
  });
375
- if (opts.resultTelemetry !== false && (telemetry.length > 0 || cooled.size > 0)) {
415
+ if (opts.resultTelemetry !== false && (allTelemetry.length > 0 || cooled.size > 0)) {
376
416
  const cooledParts = [];
377
417
  for (const [cid, centry] of cooled) {
378
418
  const ann = cooledAnnotation(centry, Date.now()) || "⏸cooled";
379
419
  cooledParts.push(cid + " " + ann);
380
420
  }
381
- const parts = telemetry
421
+ // Multi-query fans every variant through every backend; cap the line
422
+ // so the telemetry note cannot dwarf the results themselves.
423
+ const parts = allTelemetry
424
+ .slice(0, 12)
382
425
  .map((t) => t.id + (t.ok ? " ✓" : " ✗") + t.ms + "ms" + (t.ok ? "/" + t.count : (t.error ? " (" + String(t.error).slice(0, 40) + ")" : "")))
383
426
  .concat(cooledParts);
427
+ if (allTelemetry.length > 12) parts.push(`… +${allTelemetry.length - 12}`);
384
428
  const line = "[websearch backends]" + TELEMETRY_TAG + " " + parts.join(" · ");
385
429
  outContent = outContent === undefined ? line : outContent + "\n\n" + line;
386
430
  }
@@ -393,17 +437,17 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
393
437
 
394
438
  // All enabled backends failed - synthesize one meaningful error that
395
439
  // names each backend's own reason so the caller (model or user) can act.
396
- const detail = failures.slice(0, 4)
440
+ const detail = allFailures.slice(0, 4)
397
441
  .map((f) => f.id + ": " + String(f.err instanceof Error ? f.err.message : f.err).slice(0, 120))
398
442
  .join("; ");
399
443
  const allKeyMissing =
400
- failures.length > 0 &&
401
- failures.every((f) => f.err instanceof WebError && (f.err.code === "WEB_PROVIDER_UNAVAILABLE" || f.err.code === "WEB_PROVIDER_CREDENTIAL_MISSING"));
444
+ allFailures.length > 0 &&
445
+ allFailures.every((f) => f.err instanceof WebError && (f.err.code === "WEB_PROVIDER_UNAVAILABLE" || f.err.code === "WEB_PROVIDER_CREDENTIAL_MISSING"));
402
446
  const hint = allKeyMissing
403
447
  ? " (every enabled backend is missing its API key - add keys in Settings > Web Search, or enable a keyless backend)"
404
448
  : "";
405
449
  throw new WebError(
406
- `unified search: all enabled backends failed (${failures.length}) - ${detail}` + hint,
450
+ `unified search: all enabled backends failed (${allFailures.length}) - ${detail}` + hint,
407
451
  "WEB_PROVIDER_ERROR",
408
452
  { cause: hardError ?? undefined },
409
453
  );
package/lib/rrf.js ADDED
@@ -0,0 +1,56 @@
1
+ // rrf.js — v2.8.0 Reciprocal Rank Fusion (pure functions, zero deps).
2
+ //
3
+ // RRF merges multiple ranked result lists into one: each list contributes
4
+ // score 1/(k + rank) to every item it contains (k=60 is the canonical value
5
+ // from Cormack et al. 2009, popularized by RAG-Fusion
6
+ // https://github.com/Raudaschl/rag-fusion). Items are deduped by URL; the
7
+ // first-seen entry wins for metadata, scores accumulate across lists.
8
+ //
9
+ // Why RRF over naive concatenation: it needs no score calibration across
10
+ // backends/queries (only ranks), and demotes one-off hits while promoting
11
+ // items that appear in several lists.
12
+
13
+ /** Canonical RRF constant. */
14
+ export const RRF_K = 60;
15
+
16
+ /** Stable key for dedupe (URL string; missing/odd URLs fall back to title). */
17
+ function itemKey(item) {
18
+ if (item && typeof item.url === "string" && item.url.length > 0) return item.url;
19
+ if (item && typeof item.title === "string") return `t:${item.title}`;
20
+ return "";
21
+ }
22
+
23
+ /**
24
+ * Fuse ranked lists of search sources with Reciprocal Rank Fusion.
25
+ *
26
+ * @param {Array<Array<object>>} lists each list is already ranked best-first
27
+ * @param {{k?: number}} [opts] k constant (default 60)
28
+ * @returns {{ fused: object[], scores: Map<string, number> }}
29
+ * fused: sources sorted by accumulated RRF score desc (ties keep first
30
+ * appearance order); sources without a usable key are dropped.
31
+ */
32
+ export function rrfFuse(lists, { k = RRF_K } = {}) {
33
+ const kk = Number(k) > 0 ? Number(k) : RRF_K;
34
+ const scores = new Map(); // key -> accumulated score
35
+ const firstSeen = new Map(); // key -> source object (first metadata wins)
36
+ const order = []; // keys in first-appearance order (stable ties)
37
+
38
+ for (const list of Array.isArray(lists) ? lists : []) {
39
+ if (!Array.isArray(list)) continue;
40
+ for (let rank = 0; rank < list.length; rank++) {
41
+ const item = list[rank];
42
+ const key = itemKey(item);
43
+ if (!key) continue;
44
+ scores.set(key, (scores.get(key) ?? 0) + 1 / (kk + rank));
45
+ if (!firstSeen.has(key)) {
46
+ firstSeen.set(key, item);
47
+ order.push(key);
48
+ }
49
+ }
50
+ }
51
+
52
+ const fused = [...order]
53
+ .sort((a, b) => (scores.get(b) ?? 0) - (scores.get(a) ?? 0))
54
+ .map((key) => firstSeen.get(key));
55
+ return { fused, scores };
56
+ }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@240xu/dsh-websearch",
3
- "version": "2.7.3",
3
+ "version": "2.8.0",
4
4
  "description": "Unified web search provider for DeepSeek Harness (ctx.web): 11 backends — Exa + Parallel + DDG + SearXNG (keyless), DeepSeek + Anthropic + OpenAI + Brave + Tavily + Serper + Mojeek (key-gated via Settings credentials). Registers provider id \"unified\" for zero-config web_search. 统一网页搜索聚合:11 后端并发扇出、去重合并,零依赖。See CHANGELOG.md for version history.",
5
5
  "type": "module",
6
6
  "main": "lib/index.js",