@240xu/dsh-websearch 2.7.3 → 2.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +27 -1
- package/lib/backends/exa.js +20 -0
- package/lib/backends/searxng.js +3 -1
- package/lib/backends/tavily.js +6 -1
- package/lib/cache.js +2 -1
- package/lib/history.js +31 -3
- package/lib/index.js +14 -1
- package/lib/multiquery.js +48 -0
- package/lib/provider.js +145 -101
- package/lib/rrf.js +56 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -115,6 +115,30 @@ pnpm install
|
|
|
115
115
|
|
|
116
116
|
环境变量兜底:未在 credentials 配置时回退读同名环境变量;`DSH_UNIFIED_SEARCH_BACKENDS` 可逗号分隔强制指定启用集合。
|
|
117
117
|
|
|
118
|
+
## v2.8.0 查全查多 | What's new in v2.8.0
|
|
119
|
+
|
|
120
|
+
面向「查的多一点、查的全一点」的检索深度升级,全部 feature-gated、默认关:
|
|
121
|
+
|
|
122
|
+
### 1. multiQuery 多查询 RRF 融合(`multiQueryEnabled`,默认关)
|
|
123
|
+
|
|
124
|
+
- 门控启发式(刻意从严):查询长度 > 60 字符,或含 `vs / and / 比较 / 对比 / 和` 分隔词,才触发;简单查询单扇出,行为与旧版完全一致(arXiv:2404.01037:盲目多查询会劣化)。
|
|
125
|
+
- 触发后派生 ≤3 个子查询(原查询永远参与融合且保持全权重),各自走完整 11 后端扇出,再用 RRF(k=60)融合:出现在多个子查询结果里的条目被提升,单列表独占条目按位次衰减;URL 去重、首见元数据保留。
|
|
126
|
+
- 遥测行合并各变体并截断至 12 条,避免淹没结果。
|
|
127
|
+
|
|
128
|
+
### 2. deepCoverage 深度覆盖(`deepCoverage`,默认关)
|
|
129
|
+
|
|
130
|
+
- 每后端请求条数上调至 `ceil(maxResults × 1.5)`,全局去重后仍按 maxResults 截断——牺牲带宽换覆盖面。
|
|
131
|
+
- 后端能力适配(feature-detect,不破坏旧参数):Tavily 自动切 `search_depth: advanced`(2 credits,多 snippet/URL);SearXNG 类目扩为 `general,news,it`(Search API `categories` 逗号列表)。
|
|
132
|
+
- Exa:`inferExaCategory` 从查询推断 data category(github / research paper / news,仅高置信注入)。
|
|
133
|
+
|
|
134
|
+
### 3. 运维可观测(P2)
|
|
135
|
+
|
|
136
|
+
- `GET /api/websearch/history` 响应新增 `backends` 字段:每后端最近真实成败/延迟/熔断状态(`{id, ok, ms, at, failCount, cooled}`,无错误串无凭证),兜底冗余度可观测。
|
|
137
|
+
|
|
138
|
+
### 4. per-backend 超时上限(P2)
|
|
139
|
+
|
|
140
|
+
- `ddg` / `searxng` 专属超时上限 5s(`min(5s, backendTimeoutMs)`,全局默认仍 30s、配置更短则以配置为准)——不可达后端不再拖住整个扇出尾。实测见 docs/real-call-2.8.0.md。
|
|
141
|
+
|
|
118
142
|
## v2.7.3 兼容修复 | What's new in v2.7.3
|
|
119
143
|
|
|
120
144
|
- **DSH 0.1.7 客户端兼容(P0)**:settingsScope 客户端服务在 0.1.7 被移除;现改为 feature-detect——0.1.5(或装 dsh-settings-scope-shim)行为不变,0.1.7 无 shim 时设置卡片优雅停用(console.warn 指引),客户端不再静默全死。
|
|
@@ -216,7 +240,9 @@ v2.7.0 缓存/熔断/历史三项的依据与刻意取舍(含查证过的反
|
|
|
216
240
|
1. **熔断器(Circuit Breaker)**:出处 Michael T. Nygard《Release It!》(Pragmatic Bookshelf, 2007) 与 Microsoft Azure Architecture Center「Circuit Breaker pattern」https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker 。经典状态机是 Closed → Open → Half-Open(半开态放行少量探测请求)。**本插件刻意省略半开态**:冷却到期即视为 Closed(下一次真实搜索就是探测),失败则立即重新开窗——因为这里的"请求"是一次多后端扇出中的单个后端调用,探测请求必然真实发生且失败成本只是一次软降级,无需额外探测计数器;同时保留 fail-open 兜底(全部冷却时放行),比经典模式更保守。
|
|
217
241
|
2. **缓存失效(TTL-only vs 事件驱动)**:Cloudflare「Retention vs Freshness (TTL)」https://developers.cloudflare.com/cache/concepts/retention-vs-freshness/ 与「Revalidation」https://developers.cloudflare.com/cache/concepts/revalidation/ 。搜索结果没有可靠的原站变更信号可订阅(网页索引持续变化),事件驱动失效无锚点;TTL-only + 把 TTL 做成设置项(默认 900s)+ key 纳入后端组合,是"可接受的最终一致性"。取舍:900s 内索引更新不可见——对搜索场景可接受,且用户可调低。
|
|
218
242
|
3. **原子写(tmp + renameSync)**:POSIX 上 rename(2) 原子替换(Node 文档:newPath 已存在时覆盖);Windows 上 libuv 将 fs.rename 映射为 MoveFileExW(MOVEFILE_REPLACE_EXISTING)(libuv#283 https://github.com/joyent/libuv/issues/283 ,LWN 讨论 https://lwn.net/Articles/682988/ 指出 MS 文档不承诺该调用的原子性)。**结论:POSIX 双端原子;Windows 实践上等价但官方文档不背书**——因此本插件在写入端额外做了兜底:读端永远校验 JSON 完整性、坏条目按 miss 处理并删除,即使 Windows 上出现半写状态也不会返回坏数据。
|
|
219
|
-
|
|
243
|
+
4b. **查全查多(v2.8.0)**:Tavily Search API(`search_depth: advanced` 2 credits、`max_results` 0-20、`include_answer`、topic=news 时间加权)https://docs.tavily.com/api-reference/endpoint/search ;Exa Search API(type 分类检索、category 枚举、livecrawl;注意 2026 版 coding-agent 指南声明 neural 为 legacy 术语、且告诫勿臆造 category 值——本插件只注入 OpenAPI 枚举内的高置信值)https://exa.ai/docs/reference/search-api-guide-for-coding-agents 与 https://github.com/exa-labs/openapi-spec/blob/master/exa-openapi-spec.yaml ;SearXNG Search API(`categories` 逗号多类目、`language`)https://docs.searxng.org/dev/search_api.html ;Brave Search API(`freshness`、`result_filter`,本轮仅调研未接线)https://api-dashboard.search.brave.com/app/documentation 。多查询融合:RRF 出自 Cormack et al. 2009,RAG-Fusion 实现 https://github.com/Raudaschl/rag-fusion ;HyDE https://arxiv.org/abs/2212.10496 ;**门控依据** arXiv:2404.01037(盲目多查询劣化)——因此 multiQuery 默认关、启发式从严、原查询全权重参与。
|
|
244
|
+
4c. **per-backend 超时**:不可达主机的 TCP 连接失败不受应用层超时保护,只能靠 abort 计时器兜底;ddg(HTML 抓取)与 SearXNG(公共实例无 SLA)是拖尾主力,5s 上限取「P50 成功延迟的一个数量级」。
|
|
245
|
+
4d. **搜索历史隐私**:Article 29 Working Party 意见书 WP148 https://ec.europa.eu/justice/article-29/documentation/opinion-recommendation/files/2008/wp148_en.pdf 认定查询日志(query + 时间 + 来源标识)属个人数据;EFF 六条自保建议 https://w2.eff.org/Privacy/search/searchtips.pdf 的第一条是"别在搜索词里放 PII"——说明 query 天然可能携带 PII(人名、账号、地址)。本插件的取舍:历史只存 {query, time, resultCount, backendsOk/Total}(无 IP、无 cookie、无结果 URL)、环形 50 条自动滚动即短保留、本机存储不出网、提供 historyEnabled 开关与 POST clear 端点。**未做 query 脱敏**:脱敏会破坏"查看最近搜了什么"的核心用途,且本机单用户场景下该数据本就在 DSH 会话日志里存在;若未来历史跨设备同步,脱敏/加密是上线前置条件。
|
|
220
246
|
|
|
221
247
|
## License | 许可
|
|
222
248
|
|
package/lib/backends/exa.js
CHANGED
|
@@ -13,6 +13,22 @@ import { WebError } from "@deepseek-ai/dsh-web";
|
|
|
13
13
|
const EXA_ENDPOINT = "https://mcp.exa.ai/mcp";
|
|
14
14
|
const EXA_TOOL = "web_search_exa";
|
|
15
15
|
|
|
16
|
+
/**
|
|
17
|
+
* Infer an Exa data category from the query (v2.8.0). Only high-confidence
|
|
18
|
+
* patterns map — Exa rejects unknown categories with a 400, so this returns
|
|
19
|
+
* undefined unless a signal is explicit. Valid values per the Exa OpenAPI
|
|
20
|
+
* enum: github / "research paper" / news / pdf / company / people /
|
|
21
|
+
* "personal site" / "financial report".
|
|
22
|
+
*/
|
|
23
|
+
export function inferExaCategory(query) {
|
|
24
|
+
const q = typeof query === "string" ? query.toLowerCase() : "";
|
|
25
|
+
if (!q) return undefined;
|
|
26
|
+
if (/\bgithub\b|\brepo\b|repository|开源项目|源码/.test(q)) return "github";
|
|
27
|
+
if (/\barxiv\b|\bpaper\b|论文|研究文献|preprint/.test(q)) return "research paper";
|
|
28
|
+
if (/\bnews\b|新闻|最新消息|动态/.test(q)) return "news";
|
|
29
|
+
return undefined;
|
|
30
|
+
}
|
|
31
|
+
|
|
16
32
|
export const exaBackend = {
|
|
17
33
|
id: "exa",
|
|
18
34
|
requiresCredential: false,
|
|
@@ -26,6 +42,10 @@ export const exaBackend = {
|
|
|
26
42
|
query: request.query,
|
|
27
43
|
numResults: request.maxResults ?? 8,
|
|
28
44
|
};
|
|
45
|
+
// v2.8.0: category inference (feature-detect — only sent when confident;
|
|
46
|
+
// if the endpoint rejects it the fan-out demotes softly).
|
|
47
|
+
const category = inferExaCategory(request.query);
|
|
48
|
+
if (category) args.category = category;
|
|
29
49
|
hooks.recordRequest?.({
|
|
30
50
|
endpoint: EXA_ENDPOINT,
|
|
31
51
|
toolName: EXA_TOOL,
|
package/lib/backends/searxng.js
CHANGED
|
@@ -33,7 +33,9 @@ export const searxngBackend = {
|
|
|
33
33
|
url.pathname = url.pathname.replace(/\/+$/,"") + "/search";
|
|
34
34
|
url.searchParams.set("q", request.query);
|
|
35
35
|
url.searchParams.set("format", "json");
|
|
36
|
-
|
|
36
|
+
// deepCoverage widens the category sweep (docs.searxng.org Search API:
|
|
37
|
+
// categories is a comma-separated list) — breadth over one category.
|
|
38
|
+
url.searchParams.set("categories", request.deepCoverage === true ? "general,news,it" : "general");
|
|
37
39
|
// Language: settings/request filter wins, else historical "en" default.
|
|
38
40
|
const filters = normalizeFilters(request.filters);
|
|
39
41
|
url.searchParams.set("language", filters.lang ?? "en");
|
package/lib/backends/tavily.js
CHANGED
|
@@ -31,7 +31,12 @@ export const tavilyBackend = {
|
|
|
31
31
|
|
|
32
32
|
const baseURL = backendOpts.baseURL ?? TAVILY_ENDPOINT;
|
|
33
33
|
const maxResults = Math.min(request.maxResults ?? 8, 20);
|
|
34
|
-
|
|
34
|
+
// deepCoverage (provider sets request.deepCoverage) upgrades the depth:
|
|
35
|
+
// advanced = highest relevance, multiple snippets per URL (2 API credits).
|
|
36
|
+
// An explicit operator setting keeps precedence unless it was left basic.
|
|
37
|
+
const searchDepth = request.deepCoverage === true && backendOpts.searchDepth !== "advanced"
|
|
38
|
+
? "advanced"
|
|
39
|
+
: (backendOpts.searchDepth ?? "basic");
|
|
35
40
|
|
|
36
41
|
const body = {
|
|
37
42
|
query: request.query,
|
package/lib/cache.js
CHANGED
|
@@ -33,13 +33,14 @@ export const CACHE_DEFAULTS = {
|
|
|
33
33
|
* mix must never serve a stale result produced by a different combination
|
|
34
34
|
* (arch-review P1-1). Pure function; exported for tests and host reuse.
|
|
35
35
|
*/
|
|
36
|
-
export function cacheKeyFor({ query, filters, maxResults, backends }) {
|
|
36
|
+
export function cacheKeyFor({ query, filters, maxResults, backends, mode }) {
|
|
37
37
|
const backendSet = Array.isArray(backends) ? [...backends].sort().join(",") : "";
|
|
38
38
|
const canonical = JSON.stringify({
|
|
39
39
|
q: typeof query === "string" ? query : "",
|
|
40
40
|
f: filters ?? {},
|
|
41
41
|
n: Number(maxResults) || 0,
|
|
42
42
|
b: backendSet,
|
|
43
|
+
m: typeof mode === "string" ? mode : "",
|
|
43
44
|
});
|
|
44
45
|
return createHash("sha256").update(canonical).digest("hex").slice(0, 32);
|
|
45
46
|
}
|
package/lib/history.js
CHANGED
|
@@ -115,13 +115,37 @@ export function isTrustedApiRequest(req) {
|
|
|
115
115
|
}
|
|
116
116
|
}
|
|
117
117
|
|
|
118
|
+
/**
|
|
119
|
+
* Project the module-level backend health map into a small observability
|
|
120
|
+
* payload (v2.8.0 P2): per-backend last real success/failure, latency and
|
|
121
|
+
* breaker state — no error strings, no credentials, safe for the GUI.
|
|
122
|
+
*/
|
|
123
|
+
export function toBackendObservability(healthMap, nowMs = Date.now()) {
|
|
124
|
+
const out = [];
|
|
125
|
+
if (healthMap instanceof Map) {
|
|
126
|
+
for (const [id, e] of healthMap) {
|
|
127
|
+
out.push({
|
|
128
|
+
id,
|
|
129
|
+
ok: e?.ok === true,
|
|
130
|
+
ms: Number.isFinite(e?.ms) ? e.ms : null,
|
|
131
|
+
at: Number.isFinite(e?.at) ? e.at : null,
|
|
132
|
+
failCount: Number.isFinite(e?.failCount) ? e.failCount : 0,
|
|
133
|
+
cooled: Number.isFinite(e?.cooledUntil) && e.cooledUntil > nowMs,
|
|
134
|
+
});
|
|
135
|
+
}
|
|
136
|
+
}
|
|
137
|
+
return out;
|
|
138
|
+
}
|
|
139
|
+
|
|
118
140
|
/**
|
|
119
141
|
* Register the read-only history routes on the (late-mounted) webServer.
|
|
120
|
-
* GET /api/websearch/history -> { ok, entries }
|
|
142
|
+
* GET /api/websearch/history -> { ok, entries, backends }
|
|
143
|
+
* backends: per-backend last real outcome/latency/breaker state (v2.8.0),
|
|
144
|
+
* so fallback redundancy is observable without a separate endpoint.
|
|
121
145
|
* POST /api/websearch/history/clear -> { ok, cleared: true }
|
|
122
146
|
* Both routes sit behind isTrustedApiRequest (loopback Host + same-site).
|
|
123
147
|
*/
|
|
124
|
-
export function registerHistoryRoutes(ws, { getStore }) {
|
|
148
|
+
export function registerHistoryRoutes(ws, { getStore, getBackendHealth }) {
|
|
125
149
|
if (!ws || typeof ws.register !== "function") return;
|
|
126
150
|
const send = (res, code, body) => {
|
|
127
151
|
res.writeHead(code, { "content-type": "application/json" });
|
|
@@ -144,7 +168,11 @@ export function registerHistoryRoutes(ws, { getStore }) {
|
|
|
144
168
|
return;
|
|
145
169
|
}
|
|
146
170
|
const store = getStore();
|
|
147
|
-
send(res, 200, {
|
|
171
|
+
send(res, 200, {
|
|
172
|
+
ok: true,
|
|
173
|
+
entries: store ? store.list() : [],
|
|
174
|
+
backends: toBackendObservability(typeof getBackendHealth === "function" ? getBackendHealth() : getBackendHealth),
|
|
175
|
+
});
|
|
148
176
|
}),
|
|
149
177
|
});
|
|
150
178
|
ws.register({
|
package/lib/index.js
CHANGED
|
@@ -169,6 +169,12 @@ export const Config = z.object({
|
|
|
169
169
|
historyEnabled: z.boolean().default(true)
|
|
170
170
|
.description("记录搜索历史【历史】:只追加,最多 50 条;可在设置 → 搜索 中查看与清空"),
|
|
171
171
|
|
|
172
|
+
// v2.8.0 查全查多
|
|
173
|
+
multiQueryEnabled: z.boolean().default(false)
|
|
174
|
+
.description("多查询融合【查全】:复杂查询(长度>60 或含 vs/and/比较/对比/和)自动派生 2-3 个子查询,各扇出后用 RRF(k=60) 融合;默认关"),
|
|
175
|
+
deepCoverage: z.boolean().default(false)
|
|
176
|
+
.description("深度覆盖【查全】:每后端请求条数上调 50% 再全局去重截断,牺牲带宽换覆盖面;Tavily 自动切 advanced 深度,SearXNG 扩展多类目;默认关"),
|
|
177
|
+
|
|
172
178
|
// Enable toggles (one per backend)
|
|
173
179
|
enableExa: z.boolean().default(true).description(DEFAULT_LABELS.exa),
|
|
174
180
|
enableParallel: z.boolean().default(true).description(DEFAULT_LABELS.parallel),
|
|
@@ -326,6 +332,9 @@ export function resolveOptions(ctx, config) {
|
|
|
326
332
|
? Math.max(0, Math.floor(Number(config?.breakerCooldownMs) || 0)) // 0 = disabled
|
|
327
333
|
: BREAKER_DEFAULTS.cooldownMs,
|
|
328
334
|
historyEnabled: config?.historyEnabled !== false,
|
|
335
|
+
// v2.8.0 coverage / multiQuery
|
|
336
|
+
multiQueryEnabled: config?.multiQueryEnabled === true,
|
|
337
|
+
deepCoverage: config?.deepCoverage === true,
|
|
329
338
|
filters: {
|
|
330
339
|
recency: typeof config?.recency === "string" && config.recency !== "any" ? config.recency : undefined,
|
|
331
340
|
lang: language.length > 0 ? language : undefined,
|
|
@@ -423,6 +432,10 @@ export function apply(ctx, config) {
|
|
|
423
432
|
getBackendHealth: getLatestBackendHealth,
|
|
424
433
|
});
|
|
425
434
|
// v2.7.0 search history: read-only GET + explicit clear POST.
|
|
426
|
-
|
|
435
|
+
// v2.8.0: GET also carries per-backend last-outcome observability.
|
|
436
|
+
registerHistoryRoutes(webCtx.webServer, {
|
|
437
|
+
getStore: () => historyStore,
|
|
438
|
+
getBackendHealth: getLatestBackendHealth,
|
|
439
|
+
});
|
|
427
440
|
});
|
|
428
441
|
}
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
// multiquery.js — v2.8.0 query-variant derivation (pure functions, zero deps).
|
|
2
|
+
//
|
|
3
|
+
// Gated multi-query: arXiv:2404.01037 (Wang et al.) shows BLIND multi-query
|
|
4
|
+
// can degrade retrieval — so variants are only derived for queries that look
|
|
5
|
+
// genuinely multi-faceted, and the original query always participates in the
|
|
6
|
+
// fusion with full weight. Splitting is purely mechanical (at separator
|
|
7
|
+
// words/punctuation); entity text is never rewritten.
|
|
8
|
+
//
|
|
9
|
+
// Separators per the v2.8 spec: "vs", "and", 比较, 对比, 和 — plus sentence
|
|
10
|
+
// punctuation for the long-query case.
|
|
11
|
+
|
|
12
|
+
const SEPARATOR_SPLIT_RE = /\s+(?:vs\.?|and)\s+|\s*(?:比较|对比|和)\s*|\s*[、;;。!?!?|]+\s*/i;
|
|
13
|
+
const COMPLEX_MARK_RE = /(?:\bvs\.?\b|\band\b|比较|对比|和)/i;
|
|
14
|
+
|
|
15
|
+
/** Sub-query hard cap (original + 2 variants max). */
|
|
16
|
+
export const MAX_SUB_QUERIES = 3;
|
|
17
|
+
|
|
18
|
+
/**
|
|
19
|
+
* Heuristic gate: does this query look multi-faceted enough to justify
|
|
20
|
+
* multi-query fan-out? Long queries (>60 chars) or queries containing a
|
|
21
|
+
* separator word qualify. Returns false for anything else — blind
|
|
22
|
+
* multi-query degrades (arXiv:2404.01037), so the default is NO.
|
|
23
|
+
*/
|
|
24
|
+
export function isComplexQuery(query) {
|
|
25
|
+
if (typeof query !== "string") return false;
|
|
26
|
+
const q = query.trim();
|
|
27
|
+
if (q.length === 0) return false;
|
|
28
|
+
if (q.length > 60) return true;
|
|
29
|
+
return COMPLEX_MARK_RE.test(q);
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
/**
|
|
33
|
+
* Derive 2-3 sub-query variants from a complex query: the original query
|
|
34
|
+
* (always first, full weight in fusion) plus up to two mechanical splits at
|
|
35
|
+
* separator boundaries. Never rewrites entity text; returns [query] when no
|
|
36
|
+
* usable split exists.
|
|
37
|
+
*/
|
|
38
|
+
export function deriveSubQueries(query) {
|
|
39
|
+
const base = typeof query === "string" ? query.trim() : "";
|
|
40
|
+
if (base.length === 0) return [];
|
|
41
|
+
const parts = base
|
|
42
|
+
.split(SEPARATOR_SPLIT_RE)
|
|
43
|
+
.map((p) => p.trim())
|
|
44
|
+
.filter((p) => p.length >= 2);
|
|
45
|
+
if (parts.length < 2) return [base];
|
|
46
|
+
const variants = [base, parts[0], parts.slice(1).join(" ")];
|
|
47
|
+
return [...new Set(variants.map((v) => v.trim()).filter(Boolean))].slice(0, MAX_SUB_QUERIES);
|
|
48
|
+
}
|
package/lib/provider.js
CHANGED
|
@@ -25,9 +25,27 @@ import { normalizeFilters } from "./util/filters.js";
|
|
|
25
25
|
import { shapeQuery, resultHeader, TELEMETRY_TAG } from "./prompting.js";
|
|
26
26
|
import { cacheKeyFor } from "./cache.js";
|
|
27
27
|
import { foldBreaker, isCooled, cooledAnnotation } from "./breaker.js";
|
|
28
|
+
import { rrfFuse } from "./rrf.js";
|
|
29
|
+
import { isComplexQuery, deriveSubQueries } from "./multiquery.js";
|
|
28
30
|
|
|
29
31
|
const PROVIDER_ID = "unified";
|
|
30
32
|
|
|
33
|
+
// Per-backend timeout ceilings (v2.8.0, P2): ddg/scrapes and self-hosted
|
|
34
|
+
// SearXNG instances are the slowest to fail when unreachable — a 30s global
|
|
35
|
+
// default lets one dead backend stall the whole fan-out tail. The effective
|
|
36
|
+
// timeout is min(override, backendTimeoutMs), so an operator's shorter global
|
|
37
|
+
// timeout always wins.
|
|
38
|
+
const BACKEND_TIMEOUT_OVERRIDES_MS = {
|
|
39
|
+
ddg: 5000,
|
|
40
|
+
searxng: 5000,
|
|
41
|
+
};
|
|
42
|
+
|
|
43
|
+
/** Effective per-backend timeout = min(override ?? ∞, global). Exported for tests. */
|
|
44
|
+
export function effectiveBackendTimeoutMs(id, backendTimeoutMs) {
|
|
45
|
+
const override = BACKEND_TIMEOUT_OVERRIDES_MS[id];
|
|
46
|
+
return Number.isFinite(override) ? Math.min(override, backendTimeoutMs) : backendTimeoutMs;
|
|
47
|
+
}
|
|
48
|
+
|
|
31
49
|
// Module-level per-backend health telemetry (id -> entry), updated on every
|
|
32
50
|
// backend completion during a real search. Purely passive: nothing writes here
|
|
33
51
|
// outside search(), so the read-only /api/unified-search/health route can
|
|
@@ -177,14 +195,25 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
|
|
|
177
195
|
const concurrency = opts.concurrency ?? 6;
|
|
178
196
|
const backendTimeoutMs = opts.backendTimeoutMs ?? 30000;
|
|
179
197
|
|
|
180
|
-
// v2.
|
|
181
|
-
//
|
|
182
|
-
|
|
183
|
-
//
|
|
198
|
+
// v2.8.0 coverage / multiQuery gates (both default OFF; arXiv:2404.01037
|
|
199
|
+
// warns blind multi-query degrades, so the gate is deliberately strict).
|
|
200
|
+
const coverage = opts.deepCoverage === true;
|
|
201
|
+
// Per-backend widening: fetch more per backend, dedupe down afterwards.
|
|
202
|
+
const backendMaxResults = coverage ? Math.ceil(maxResults * 1.5) : maxResults;
|
|
203
|
+
const multi = opts.multiQueryEnabled === true && isComplexQuery(effReq.query);
|
|
204
|
+
const variants = multi ? deriveSubQueries(effReq.query) : [effReq.query];
|
|
205
|
+
// Cache-key mode marker: different coverage/multiQuery settings must not
|
|
206
|
+
// serve each other's cached results.
|
|
207
|
+
const cacheMode = (multi ? "multi" : "single") + (coverage ? "+coverage" : "");
|
|
208
|
+
|
|
209
|
+
// v2.7.0 disk cache: keyed on (shaped query + filters + maxResults +
|
|
210
|
+
// backends + mode). A hit short-circuits the fan-out entirely; sources
|
|
211
|
+
// are byte-identical to the first search and content carries a
|
|
212
|
+
// "cache hit, age Ns" note. Cache failures never break the search.
|
|
184
213
|
const cacheEnabled = opts.cacheEnabled === true && cache;
|
|
185
214
|
let cacheKey = null;
|
|
186
215
|
if (cacheEnabled) {
|
|
187
|
-
cacheKey = cacheKeyFor({ query: effReq.query, filters, maxResults, backends: opts.enabledBackends });
|
|
216
|
+
cacheKey = cacheKeyFor({ query: effReq.query, filters, maxResults, backends: opts.enabledBackends, mode: cacheMode });
|
|
188
217
|
const hit = cache.get(cacheKey);
|
|
189
218
|
if (hit && Array.isArray(hit.value?.sources) && hit.value.sources.length > 0) {
|
|
190
219
|
// Defense in depth: the fresh path drops non-http(s) schemes before
|
|
@@ -219,7 +248,7 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
|
|
|
219
248
|
// backend inside its 60s cooldown window is skipped (annotated in
|
|
220
249
|
// telemetry as "id ⏸cooled"). Fail-open: if EVERY candidate is cooled,
|
|
221
250
|
// run them all anyway — the breaker must never make search worse than
|
|
222
|
-
// no breaker.
|
|
251
|
+
// no breaker. The pick is shared by every query variant.
|
|
223
252
|
const breakerCfg = opts.breakerEnabled === false ? null : {
|
|
224
253
|
threshold: opts.breakerThreshold,
|
|
225
254
|
cooldownMs: opts.breakerCooldownMs,
|
|
@@ -255,98 +284,109 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
|
|
|
255
284
|
);
|
|
256
285
|
}
|
|
257
286
|
|
|
258
|
-
//
|
|
259
|
-
//
|
|
260
|
-
//
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
let bu = null;
|
|
278
|
-
try { bu = new URL(beOpts.baseURL); } catch { bu = null; }
|
|
279
|
-
const privHost = bu ? /^(localhost|127(\.\d{1,3}){3}|10(\.\d{1,3}){3}|192\.168(\.\d{1,3}){3}|172\.(1[6-9]|2\d|3[01])(\.\d{1,3}){3})$/.test(bu.hostname) : false;
|
|
280
|
-
const schemeOk = bu && (bu.protocol === "https:" || (bu.protocol === "http:" && privHost));
|
|
281
|
-
if (!schemeOk || bu.username || bu.password) {
|
|
282
|
-
throw new WebError(`backend "${id}" has an invalid baseURL: https required (plain http only for private hosts), credentials-in-URL rejected`, "WEB_PROVIDER_ERROR");
|
|
287
|
+
// Run ONE query variant through the full fan-out: breaker-aware backend
|
|
288
|
+
// execution under the per-variant limiter, then PHASE 1 collect. The
|
|
289
|
+
// internal abort/timeout semantics are unchanged from v2.7.
|
|
290
|
+
const fanoutVariant = async (variantReq) => {
|
|
291
|
+
const telemetry = [];
|
|
292
|
+
const limit = createLimiter(concurrency);
|
|
293
|
+
const runBackend = ({ id, backend, beOpts, hooks }) => {
|
|
294
|
+
const startedAtMs = Date.now();
|
|
295
|
+
// Trust boundary: baseURL is operator-configurable and API keys are sent to it.
|
|
296
|
+
// Enforce http(s); plain http only for private hosts (self-hosted SearXNG).
|
|
297
|
+
// Reject embedded credentials. Failure is a per-backend soft error.
|
|
298
|
+
if (beOpts.baseURL !== undefined) {
|
|
299
|
+
let bu = null;
|
|
300
|
+
try { bu = new URL(beOpts.baseURL); } catch { bu = null; }
|
|
301
|
+
const privHost = bu ? /^(localhost|127(\.\d{1,3}){3}|10(\.\d{1,3}){3}|192\.168(\.\d{1,3}){3}|172\.(1[6-9]|2\d|3[01])(\.\d{1,3}){3})$/.test(bu.hostname) : false;
|
|
302
|
+
const schemeOk = bu && (bu.protocol === "https:" || (bu.protocol === "http:" && privHost));
|
|
303
|
+
if (!schemeOk || bu.username || bu.password) {
|
|
304
|
+
throw new WebError(`backend "${id}" has an invalid baseURL: https required (plain http only for private hosts), credentials-in-URL rejected`, "WEB_PROVIDER_ERROR");
|
|
305
|
+
}
|
|
283
306
|
}
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
307
|
+
const controller = new AbortController();
|
|
308
|
+
const mergedSignal = signal
|
|
309
|
+
? AbortSignal.any([signal, controller.signal])
|
|
310
|
+
: controller.signal;
|
|
311
|
+
const effectiveTimeoutMs = effectiveBackendTimeoutMs(id, backendTimeoutMs);
|
|
312
|
+
const timeoutId = setTimeout(() => controller.abort(), effectiveTimeoutMs);
|
|
313
|
+
return backend
|
|
314
|
+
.search(variantReq, mergedSignal, beOpts, hooks)
|
|
315
|
+
.then(
|
|
316
|
+
(value) => {
|
|
317
|
+
telemetry.push({ id, ok: true, ms: Date.now() - startedAtMs, count: value?.sources?.length ?? 0 });
|
|
318
|
+
recordBackendHealth(id, { ok: true, at: Date.now(), ms: Date.now() - startedAtMs, count: value?.sources?.length ?? 0 }, breakerCfg);
|
|
319
|
+
return value;
|
|
320
|
+
},
|
|
321
|
+
(err) => {
|
|
322
|
+
const msg = String(err instanceof Error ? err.message : err).slice(0, 80);
|
|
323
|
+
telemetry.push({ id, ok: false, ms: Date.now() - startedAtMs, error: msg });
|
|
324
|
+
recordBackendHealth(id, { ok: false, at: Date.now(), ms: Date.now() - startedAtMs, error: msg }, breakerCfg);
|
|
325
|
+
throw err;
|
|
326
|
+
},
|
|
327
|
+
)
|
|
328
|
+
.catch((err) => {
|
|
329
|
+
if (controller.signal.aborted && !(signal && signal.aborted)) {
|
|
330
|
+
throw new WebError(
|
|
331
|
+
`backend "${id}" timed out after ${effectiveTimeoutMs}ms`,
|
|
332
|
+
"WEB_PROVIDER_ERROR",
|
|
333
|
+
{ cause: err instanceof Error ? err : undefined },
|
|
334
|
+
);
|
|
335
|
+
}
|
|
302
336
|
throw err;
|
|
303
|
-
}
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
if (controller.signal.aborted && !(signal && signal.aborted)) {
|
|
307
|
-
throw new WebError(
|
|
308
|
-
`backend "${id}" timed out after ${backendTimeoutMs}ms`,
|
|
309
|
-
"WEB_PROVIDER_ERROR",
|
|
310
|
-
{ cause: err instanceof Error ? err : undefined },
|
|
311
|
-
);
|
|
312
|
-
}
|
|
313
|
-
throw err;
|
|
314
|
-
})
|
|
315
|
-
.finally(() => clearTimeout(timeoutId));
|
|
316
|
-
};
|
|
317
|
-
|
|
318
|
-
const results = await Promise.allSettled(
|
|
319
|
-
eligible.map(({ id, backend, beOpts }) => {
|
|
320
|
-
const hooks = {
|
|
321
|
-
recordRequest: (req) => recordBackendRequest(ctx, id, req),
|
|
322
|
-
recordOutcome: (out) => recordBackendOutcome(ctx, id, out),
|
|
323
|
-
};
|
|
324
|
-
return limit(() => runBackend({ id, backend, beOpts, hooks }));
|
|
325
|
-
}),
|
|
326
|
-
);
|
|
327
|
-
|
|
328
|
-
throwIfSearchAborted(signal);
|
|
337
|
+
})
|
|
338
|
+
.finally(() => clearTimeout(timeoutId));
|
|
339
|
+
};
|
|
329
340
|
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
341
|
+
const results = await Promise.allSettled(
|
|
342
|
+
eligible.map(({ id, backend, beOpts }) => {
|
|
343
|
+
const hooks = {
|
|
344
|
+
recordRequest: (req) => recordBackendRequest(ctx, id, req),
|
|
345
|
+
recordOutcome: (out) => recordBackendOutcome(ctx, id, out),
|
|
346
|
+
};
|
|
347
|
+
return limit(() => runBackend({ id, backend, beOpts, hooks }));
|
|
348
|
+
}),
|
|
349
|
+
);
|
|
350
|
+
throwIfSearchAborted(signal);
|
|
351
|
+
const fold = collectFanoutResults(results, eligible, signal);
|
|
352
|
+
return { ...fold, telemetry };
|
|
353
|
+
};
|
|
334
354
|
|
|
335
|
-
//
|
|
336
|
-
|
|
337
|
-
|
|
355
|
+
// Fan out per variant (sequential — the limiter already parallelizes
|
|
356
|
+
// backends within a variant), dedupe/rank each list, then fuse.
|
|
357
|
+
const perVariant = [];
|
|
358
|
+
const allTelemetry = [];
|
|
359
|
+
const allFailures = [];
|
|
360
|
+
let hardError = null;
|
|
361
|
+
for (const variantQuery of variants) {
|
|
362
|
+
const variantReq = {
|
|
363
|
+
...effReq,
|
|
364
|
+
query: variantQuery,
|
|
365
|
+
maxResults: backendMaxResults,
|
|
366
|
+
...(coverage ? { deepCoverage: true } : {}),
|
|
367
|
+
};
|
|
368
|
+
const fold = await fanoutVariant(variantReq);
|
|
369
|
+
allTelemetry.push(...fold.telemetry);
|
|
370
|
+
allFailures.push(...fold.failures);
|
|
371
|
+
if (fold.hardError !== null && hardError === null) hardError = fold.hardError;
|
|
372
|
+
// Caller-cancellation check (hard abort fires before we yield results).
|
|
373
|
+
if (signal?.aborted === true || fold.hardAborted) {
|
|
374
|
+
throw searchAborted(signal);
|
|
375
|
+
}
|
|
376
|
+
// PHASE 2 dedupe / PHASE 3 rerank (per variant list, pre-fusion).
|
|
377
|
+
const strategy = opts.dedupStrategy === "url+title" ? "url+title" : "url";
|
|
378
|
+
const deduped = dedupeSources(fold.collected, strategy);
|
|
379
|
+
const ordered = opts.rerank === true ? rerankSources(deduped, variantQuery) : deduped;
|
|
380
|
+
perVariant.push({ query: variantQuery, ordered, mergedContent: fold.mergedContent });
|
|
338
381
|
}
|
|
339
382
|
|
|
340
|
-
// PHASE
|
|
341
|
-
//
|
|
342
|
-
//
|
|
343
|
-
const
|
|
344
|
-
const
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
opts.rerank === true
|
|
348
|
-
? rerankSources(deduped, typeof request?.query === "string" ? request.query : "")
|
|
349
|
-
: deduped;
|
|
383
|
+
// PHASE 4 fuse: multi-query mode fuses the per-variant ranked lists with
|
|
384
|
+
// RRF (k=60); single-query mode keeps the historical behavior (the one
|
|
385
|
+
// list, unchanged ordering).
|
|
386
|
+
const orderedLists = perVariant.filter((v) => v.ordered.length > 0).map((v) => v.ordered);
|
|
387
|
+
const ordered = multi
|
|
388
|
+
? rrfFuse(orderedLists).fused
|
|
389
|
+
: (orderedLists[0] ?? []);
|
|
350
390
|
|
|
351
391
|
// If we got ANY sources, return them even if some backends failed.
|
|
352
392
|
if (ordered.length > 0) {
|
|
@@ -355,7 +395,7 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
|
|
|
355
395
|
// 置于结果块最前,要求逐条引用来源 URL、优先官方/一手与较新来源;
|
|
356
396
|
// 语言按查询文本自动选择(含 CJK 用中文头)。可用
|
|
357
397
|
// opts.resultHeader === false 关闭。
|
|
358
|
-
let outContent = mergedContent;
|
|
398
|
+
let outContent = perVariant.find((v) => v.mergedContent !== undefined)?.mergedContent;
|
|
359
399
|
if (opts.resultHeader !== false && (sources.length > 0 || outContent !== undefined)) {
|
|
360
400
|
const header = resultHeader(typeof request?.query === "string" && /[\u4e00-\u9fff]/.test(request.query) ? "zh" : "en");
|
|
361
401
|
outContent = outContent === undefined ? header : header + "\n\n" + outContent;
|
|
@@ -369,18 +409,22 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
|
|
|
369
409
|
query: request?.query,
|
|
370
410
|
time: Date.now(),
|
|
371
411
|
resultCount: sources.length,
|
|
372
|
-
backendsOk:
|
|
373
|
-
backendsTotal:
|
|
412
|
+
backendsOk: allTelemetry.filter((t) => t.ok).length,
|
|
413
|
+
backendsTotal: allTelemetry.length,
|
|
374
414
|
});
|
|
375
|
-
if (opts.resultTelemetry !== false && (
|
|
415
|
+
if (opts.resultTelemetry !== false && (allTelemetry.length > 0 || cooled.size > 0)) {
|
|
376
416
|
const cooledParts = [];
|
|
377
417
|
for (const [cid, centry] of cooled) {
|
|
378
418
|
const ann = cooledAnnotation(centry, Date.now()) || "⏸cooled";
|
|
379
419
|
cooledParts.push(cid + " " + ann);
|
|
380
420
|
}
|
|
381
|
-
|
|
421
|
+
// Multi-query fans every variant through every backend; cap the line
|
|
422
|
+
// so the telemetry note cannot dwarf the results themselves.
|
|
423
|
+
const parts = allTelemetry
|
|
424
|
+
.slice(0, 12)
|
|
382
425
|
.map((t) => t.id + (t.ok ? " ✓" : " ✗") + t.ms + "ms" + (t.ok ? "/" + t.count : (t.error ? " (" + String(t.error).slice(0, 40) + ")" : "")))
|
|
383
426
|
.concat(cooledParts);
|
|
427
|
+
if (allTelemetry.length > 12) parts.push(`… +${allTelemetry.length - 12}`);
|
|
384
428
|
const line = "[websearch backends]" + TELEMETRY_TAG + " " + parts.join(" · ");
|
|
385
429
|
outContent = outContent === undefined ? line : outContent + "\n\n" + line;
|
|
386
430
|
}
|
|
@@ -393,17 +437,17 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
|
|
|
393
437
|
|
|
394
438
|
// All enabled backends failed - synthesize one meaningful error that
|
|
395
439
|
// names each backend's own reason so the caller (model or user) can act.
|
|
396
|
-
const detail =
|
|
440
|
+
const detail = allFailures.slice(0, 4)
|
|
397
441
|
.map((f) => f.id + ": " + String(f.err instanceof Error ? f.err.message : f.err).slice(0, 120))
|
|
398
442
|
.join("; ");
|
|
399
443
|
const allKeyMissing =
|
|
400
|
-
|
|
401
|
-
|
|
444
|
+
allFailures.length > 0 &&
|
|
445
|
+
allFailures.every((f) => f.err instanceof WebError && (f.err.code === "WEB_PROVIDER_UNAVAILABLE" || f.err.code === "WEB_PROVIDER_CREDENTIAL_MISSING"));
|
|
402
446
|
const hint = allKeyMissing
|
|
403
447
|
? " (every enabled backend is missing its API key - add keys in Settings > Web Search, or enable a keyless backend)"
|
|
404
448
|
: "";
|
|
405
449
|
throw new WebError(
|
|
406
|
-
`unified search: all enabled backends failed (${
|
|
450
|
+
`unified search: all enabled backends failed (${allFailures.length}) - ${detail}` + hint,
|
|
407
451
|
"WEB_PROVIDER_ERROR",
|
|
408
452
|
{ cause: hardError ?? undefined },
|
|
409
453
|
);
|
package/lib/rrf.js
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
// rrf.js — v2.8.0 Reciprocal Rank Fusion (pure functions, zero deps).
|
|
2
|
+
//
|
|
3
|
+
// RRF merges multiple ranked result lists into one: each list contributes
|
|
4
|
+
// score 1/(k + rank) to every item it contains (k=60 is the canonical value
|
|
5
|
+
// from Cormack et al. 2009, popularized by RAG-Fusion
|
|
6
|
+
// https://github.com/Raudaschl/rag-fusion). Items are deduped by URL; the
|
|
7
|
+
// first-seen entry wins for metadata, scores accumulate across lists.
|
|
8
|
+
//
|
|
9
|
+
// Why RRF over naive concatenation: it needs no score calibration across
|
|
10
|
+
// backends/queries (only ranks), and demotes one-off hits while promoting
|
|
11
|
+
// items that appear in several lists.
|
|
12
|
+
|
|
13
|
+
/** Canonical RRF constant. */
|
|
14
|
+
export const RRF_K = 60;
|
|
15
|
+
|
|
16
|
+
/** Stable key for dedupe (URL string; missing/odd URLs fall back to title). */
|
|
17
|
+
function itemKey(item) {
|
|
18
|
+
if (item && typeof item.url === "string" && item.url.length > 0) return item.url;
|
|
19
|
+
if (item && typeof item.title === "string") return `t:${item.title}`;
|
|
20
|
+
return "";
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
/**
|
|
24
|
+
* Fuse ranked lists of search sources with Reciprocal Rank Fusion.
|
|
25
|
+
*
|
|
26
|
+
* @param {Array<Array<object>>} lists each list is already ranked best-first
|
|
27
|
+
* @param {{k?: number}} [opts] k constant (default 60)
|
|
28
|
+
* @returns {{ fused: object[], scores: Map<string, number> }}
|
|
29
|
+
* fused: sources sorted by accumulated RRF score desc (ties keep first
|
|
30
|
+
* appearance order); sources without a usable key are dropped.
|
|
31
|
+
*/
|
|
32
|
+
export function rrfFuse(lists, { k = RRF_K } = {}) {
|
|
33
|
+
const kk = Number(k) > 0 ? Number(k) : RRF_K;
|
|
34
|
+
const scores = new Map(); // key -> accumulated score
|
|
35
|
+
const firstSeen = new Map(); // key -> source object (first metadata wins)
|
|
36
|
+
const order = []; // keys in first-appearance order (stable ties)
|
|
37
|
+
|
|
38
|
+
for (const list of Array.isArray(lists) ? lists : []) {
|
|
39
|
+
if (!Array.isArray(list)) continue;
|
|
40
|
+
for (let rank = 0; rank < list.length; rank++) {
|
|
41
|
+
const item = list[rank];
|
|
42
|
+
const key = itemKey(item);
|
|
43
|
+
if (!key) continue;
|
|
44
|
+
scores.set(key, (scores.get(key) ?? 0) + 1 / (kk + rank));
|
|
45
|
+
if (!firstSeen.has(key)) {
|
|
46
|
+
firstSeen.set(key, item);
|
|
47
|
+
order.push(key);
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
const fused = [...order]
|
|
53
|
+
.sort((a, b) => (scores.get(b) ?? 0) - (scores.get(a) ?? 0))
|
|
54
|
+
.map((key) => firstSeen.get(key));
|
|
55
|
+
return { fused, scores };
|
|
56
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@240xu/dsh-websearch",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.8.0",
|
|
4
4
|
"description": "Unified web search provider for DeepSeek Harness (ctx.web): 11 backends — Exa + Parallel + DDG + SearXNG (keyless), DeepSeek + Anthropic + OpenAI + Brave + Tavily + Serper + Mojeek (key-gated via Settings credentials). Registers provider id \"unified\" for zero-config web_search. 统一网页搜索聚合:11 后端并发扇出、去重合并,零依赖。See CHANGELOG.md for version history.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "lib/index.js",
|