@240xu/dsh-websearch 2.7.3 → 2.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +33 -1
- package/cordis.patch.yml +10 -0
- package/lib/backends/exa.js +20 -0
- package/lib/backends/searxng.js +3 -1
- package/lib/backends/tavily.js +6 -1
- package/lib/cache.js +2 -1
- package/lib/history.js +31 -3
- package/lib/index.js +65 -29
- package/lib/multiquery.js +48 -0
- package/lib/provider.js +145 -101
- package/lib/rrf.js +56 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -115,6 +115,36 @@ pnpm install
|
|
|
115
115
|
|
|
116
116
|
环境变量兜底:未在 credentials 配置时回退读同名环境变量;`DSH_UNIFIED_SEARCH_BACKENDS` 可逗号分隔强制指定启用集合。
|
|
117
117
|
|
|
118
|
+
## v2.8.1 SettingsForms 适配 | What's new in v2.8.1
|
|
119
|
+
|
|
120
|
+
- **0.2.0 设置面板表单**:Config 标记 `meta.volatile`——0.2.0 SettingsForms 从插件 Config 自动生成表单,但 volatileForm 只保留 volatile 字段(此前 websearch 表单为空);现整棵 Config 成为 0.2.0 表单。裁决与考证见 dsh-plugin-hub/docs/reviews/cross-settings-forms.md。
|
|
121
|
+
- 删除 0.2.0 线上与可见 UI 自相矛盾的 "no installSection" console.warn(0.1.5 路径不变)。
|
|
122
|
+
- 补齐全部裸字段(×ApiKeyEnv/×BaseURL/×Model)的 description——0.2.0 表单渲染唯一用户可见文案来自 meta.description。
|
|
123
|
+
|
|
124
|
+
## v2.8.0 查全查多 | What's new in v2.8.0
|
|
125
|
+
|
|
126
|
+
面向「查的多一点、查的全一点」的检索深度升级,全部 feature-gated、默认关:
|
|
127
|
+
|
|
128
|
+
### 1. multiQuery 多查询 RRF 融合(`multiQueryEnabled`,默认关)
|
|
129
|
+
|
|
130
|
+
- 门控启发式(刻意从严):查询长度 > 60 字符,或含 `vs / and / 比较 / 对比 / 和` 分隔词,才触发;简单查询单扇出,行为与旧版完全一致(arXiv:2404.01037:盲目多查询会劣化)。
|
|
131
|
+
- 触发后派生 ≤3 个子查询(原查询永远参与融合且保持全权重),各自走完整 11 后端扇出,再用 RRF(k=60)融合:出现在多个子查询结果里的条目被提升,单列表独占条目按位次衰减;URL 去重、首见元数据保留。
|
|
132
|
+
- 遥测行合并各变体并截断至 12 条,避免淹没结果。
|
|
133
|
+
|
|
134
|
+
### 2. deepCoverage 深度覆盖(`deepCoverage`,默认关)
|
|
135
|
+
|
|
136
|
+
- 每后端请求条数上调至 `ceil(maxResults × 1.5)`,全局去重后仍按 maxResults 截断——牺牲带宽换覆盖面。
|
|
137
|
+
- 后端能力适配(feature-detect,不破坏旧参数):Tavily 自动切 `search_depth: advanced`(2 credits,多 snippet/URL);SearXNG 类目扩为 `general,news,it`(Search API `categories` 逗号列表)。
|
|
138
|
+
- Exa:`inferExaCategory` 从查询推断 data category(github / research paper / news,仅高置信注入)。
|
|
139
|
+
|
|
140
|
+
### 3. 运维可观测(P2)
|
|
141
|
+
|
|
142
|
+
- `GET /api/websearch/history` 响应新增 `backends` 字段:每后端最近真实成败/延迟/熔断状态(`{id, ok, ms, at, failCount, cooled}`,无错误串无凭证),兜底冗余度可观测。
|
|
143
|
+
|
|
144
|
+
### 4. per-backend 超时上限(P2)
|
|
145
|
+
|
|
146
|
+
- `ddg` / `searxng` 专属超时上限 5s(`min(5s, backendTimeoutMs)`,全局默认仍 30s、配置更短则以配置为准)——不可达后端不再拖住整个扇出尾。实测见 docs/real-call-2.8.0.md。
|
|
147
|
+
|
|
118
148
|
## v2.7.3 兼容修复 | What's new in v2.7.3
|
|
119
149
|
|
|
120
150
|
- **DSH 0.1.7 客户端兼容(P0)**:settingsScope 客户端服务在 0.1.7 被移除;现改为 feature-detect——0.1.5(或装 dsh-settings-scope-shim)行为不变,0.1.7 无 shim 时设置卡片优雅停用(console.warn 指引),客户端不再静默全死。
|
|
@@ -216,7 +246,9 @@ v2.7.0 缓存/熔断/历史三项的依据与刻意取舍(含查证过的反
|
|
|
216
246
|
1. **熔断器(Circuit Breaker)**:出处 Michael T. Nygard《Release It!》(Pragmatic Bookshelf, 2007) 与 Microsoft Azure Architecture Center「Circuit Breaker pattern」https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker 。经典状态机是 Closed → Open → Half-Open(半开态放行少量探测请求)。**本插件刻意省略半开态**:冷却到期即视为 Closed(下一次真实搜索就是探测),失败则立即重新开窗——因为这里的"请求"是一次多后端扇出中的单个后端调用,探测请求必然真实发生且失败成本只是一次软降级,无需额外探测计数器;同时保留 fail-open 兜底(全部冷却时放行),比经典模式更保守。
|
|
217
247
|
2. **缓存失效(TTL-only vs 事件驱动)**:Cloudflare「Retention vs Freshness (TTL)」https://developers.cloudflare.com/cache/concepts/retention-vs-freshness/ 与「Revalidation」https://developers.cloudflare.com/cache/concepts/revalidation/ 。搜索结果没有可靠的原站变更信号可订阅(网页索引持续变化),事件驱动失效无锚点;TTL-only + 把 TTL 做成设置项(默认 900s)+ key 纳入后端组合,是"可接受的最终一致性"。取舍:900s 内索引更新不可见——对搜索场景可接受,且用户可调低。
|
|
218
248
|
3. **原子写(tmp + renameSync)**:POSIX 上 rename(2) 原子替换(Node 文档:newPath 已存在时覆盖);Windows 上 libuv 将 fs.rename 映射为 MoveFileExW(MOVEFILE_REPLACE_EXISTING)(libuv#283 https://github.com/joyent/libuv/issues/283 ,LWN 讨论 https://lwn.net/Articles/682988/ 指出 MS 文档不承诺该调用的原子性)。**结论:POSIX 双端原子;Windows 实践上等价但官方文档不背书**——因此本插件在写入端额外做了兜底:读端永远校验 JSON 完整性、坏条目按 miss 处理并删除,即使 Windows 上出现半写状态也不会返回坏数据。
|
|
219
|
-
|
|
249
|
+
4b. **查全查多(v2.8.0)**:Tavily Search API(`search_depth: advanced` 2 credits、`max_results` 0-20、`include_answer`、topic=news 时间加权)https://docs.tavily.com/api-reference/endpoint/search ;Exa Search API(type 分类检索、category 枚举、livecrawl;注意 2026 版 coding-agent 指南声明 neural 为 legacy 术语、且告诫勿臆造 category 值——本插件只注入 OpenAPI 枚举内的高置信值)https://exa.ai/docs/reference/search-api-guide-for-coding-agents 与 https://github.com/exa-labs/openapi-spec/blob/master/exa-openapi-spec.yaml ;SearXNG Search API(`categories` 逗号多类目、`language`)https://docs.searxng.org/dev/search_api.html ;Brave Search API(`freshness`、`result_filter`,本轮仅调研未接线)https://api-dashboard.search.brave.com/app/documentation 。多查询融合:RRF 出自 Cormack et al. 2009,RAG-Fusion 实现 https://github.com/Raudaschl/rag-fusion ;HyDE https://arxiv.org/abs/2212.10496 ;**门控依据** arXiv:2404.01037(盲目多查询劣化)——因此 multiQuery 默认关、启发式从严、原查询全权重参与。
|
|
250
|
+
4c. **per-backend 超时**:不可达主机的 TCP 连接失败不受应用层超时保护,只能靠 abort 计时器兜底;ddg(HTML 抓取)与 SearXNG(公共实例无 SLA)是拖尾主力,5s 上限取「P50 成功延迟的一个数量级」。
|
|
251
|
+
4d. **搜索历史隐私**:Article 29 Working Party 意见书 WP148 https://ec.europa.eu/justice/article-29/documentation/opinion-recommendation/files/2008/wp148_en.pdf 认定查询日志(query + 时间 + 来源标识)属个人数据;EFF 六条自保建议 https://w2.eff.org/Privacy/search/searchtips.pdf 的第一条是"别在搜索词里放 PII"——说明 query 天然可能携带 PII(人名、账号、地址)。本插件的取舍:历史只存 {query, time, resultCount, backendsOk/Total}(无 IP、无 cookie、无结果 URL)、环形 50 条自动滚动即短保留、本机存储不出网、提供 historyEnabled 开关与 POST clear 端点。**未做 query 脱敏**:脱敏会破坏"查看最近搜了什么"的核心用途,且本机单用户场景下该数据本就在 DSH 会话日志里存在;若未来历史跨设备同步,脱敏/加密是上线前置条件。
|
|
220
252
|
|
|
221
253
|
## License | 许可
|
|
222
254
|
|
package/cordis.patch.yml
CHANGED
|
@@ -1,3 +1,13 @@
|
|
|
1
|
+
# Bundle patch layer:
|
|
2
|
+
# 1. insert the plugin row (registers the unified search provider)
|
|
3
|
+
# 2. wire the built-in web_search tool to the unified provider (the plugin's
|
|
4
|
+
# whole reason to exist — previously a manual cordis.patch.yml edit, now
|
|
5
|
+
# automatic; revert = remove this row or disable the plugin, the base
|
|
6
|
+
# default (deepseek-official) is one yaml line away).
|
|
1
7
|
- insert:
|
|
2
8
|
- id: dsh-websearch
|
|
3
9
|
name: '@240xu/dsh-websearch'
|
|
10
|
+
- id: web
|
|
11
|
+
name: '@deepseek-ai/dsh-web'
|
|
12
|
+
config:
|
|
13
|
+
searchProvider: unified
|
package/lib/backends/exa.js
CHANGED
|
@@ -13,6 +13,22 @@ import { WebError } from "@deepseek-ai/dsh-web";
|
|
|
13
13
|
const EXA_ENDPOINT = "https://mcp.exa.ai/mcp";
|
|
14
14
|
const EXA_TOOL = "web_search_exa";
|
|
15
15
|
|
|
16
|
+
/**
|
|
17
|
+
* Infer an Exa data category from the query (v2.8.0). Only high-confidence
|
|
18
|
+
* patterns map — Exa rejects unknown categories with a 400, so this returns
|
|
19
|
+
* undefined unless a signal is explicit. Valid values per the Exa OpenAPI
|
|
20
|
+
* enum: github / "research paper" / news / pdf / company / people /
|
|
21
|
+
* "personal site" / "financial report".
|
|
22
|
+
*/
|
|
23
|
+
export function inferExaCategory(query) {
|
|
24
|
+
const q = typeof query === "string" ? query.toLowerCase() : "";
|
|
25
|
+
if (!q) return undefined;
|
|
26
|
+
if (/\bgithub\b|\brepo\b|repository|开源项目|源码/.test(q)) return "github";
|
|
27
|
+
if (/\barxiv\b|\bpaper\b|论文|研究文献|preprint/.test(q)) return "research paper";
|
|
28
|
+
if (/\bnews\b|新闻|最新消息|动态/.test(q)) return "news";
|
|
29
|
+
return undefined;
|
|
30
|
+
}
|
|
31
|
+
|
|
16
32
|
export const exaBackend = {
|
|
17
33
|
id: "exa",
|
|
18
34
|
requiresCredential: false,
|
|
@@ -26,6 +42,10 @@ export const exaBackend = {
|
|
|
26
42
|
query: request.query,
|
|
27
43
|
numResults: request.maxResults ?? 8,
|
|
28
44
|
};
|
|
45
|
+
// v2.8.0: category inference (feature-detect — only sent when confident;
|
|
46
|
+
// if the endpoint rejects it the fan-out demotes softly).
|
|
47
|
+
const category = inferExaCategory(request.query);
|
|
48
|
+
if (category) args.category = category;
|
|
29
49
|
hooks.recordRequest?.({
|
|
30
50
|
endpoint: EXA_ENDPOINT,
|
|
31
51
|
toolName: EXA_TOOL,
|
package/lib/backends/searxng.js
CHANGED
|
@@ -33,7 +33,9 @@ export const searxngBackend = {
|
|
|
33
33
|
url.pathname = url.pathname.replace(/\/+$/,"") + "/search";
|
|
34
34
|
url.searchParams.set("q", request.query);
|
|
35
35
|
url.searchParams.set("format", "json");
|
|
36
|
-
|
|
36
|
+
// deepCoverage widens the category sweep (docs.searxng.org Search API:
|
|
37
|
+
// categories is a comma-separated list) — breadth over one category.
|
|
38
|
+
url.searchParams.set("categories", request.deepCoverage === true ? "general,news,it" : "general");
|
|
37
39
|
// Language: settings/request filter wins, else historical "en" default.
|
|
38
40
|
const filters = normalizeFilters(request.filters);
|
|
39
41
|
url.searchParams.set("language", filters.lang ?? "en");
|
package/lib/backends/tavily.js
CHANGED
|
@@ -31,7 +31,12 @@ export const tavilyBackend = {
|
|
|
31
31
|
|
|
32
32
|
const baseURL = backendOpts.baseURL ?? TAVILY_ENDPOINT;
|
|
33
33
|
const maxResults = Math.min(request.maxResults ?? 8, 20);
|
|
34
|
-
|
|
34
|
+
// deepCoverage (provider sets request.deepCoverage) upgrades the depth:
|
|
35
|
+
// advanced = highest relevance, multiple snippets per URL (2 API credits).
|
|
36
|
+
// An explicit operator setting keeps precedence unless it was left basic.
|
|
37
|
+
const searchDepth = request.deepCoverage === true && backendOpts.searchDepth !== "advanced"
|
|
38
|
+
? "advanced"
|
|
39
|
+
: (backendOpts.searchDepth ?? "basic");
|
|
35
40
|
|
|
36
41
|
const body = {
|
|
37
42
|
query: request.query,
|
package/lib/cache.js
CHANGED
|
@@ -33,13 +33,14 @@ export const CACHE_DEFAULTS = {
|
|
|
33
33
|
* mix must never serve a stale result produced by a different combination
|
|
34
34
|
* (arch-review P1-1). Pure function; exported for tests and host reuse.
|
|
35
35
|
*/
|
|
36
|
-
export function cacheKeyFor({ query, filters, maxResults, backends }) {
|
|
36
|
+
export function cacheKeyFor({ query, filters, maxResults, backends, mode }) {
|
|
37
37
|
const backendSet = Array.isArray(backends) ? [...backends].sort().join(",") : "";
|
|
38
38
|
const canonical = JSON.stringify({
|
|
39
39
|
q: typeof query === "string" ? query : "",
|
|
40
40
|
f: filters ?? {},
|
|
41
41
|
n: Number(maxResults) || 0,
|
|
42
42
|
b: backendSet,
|
|
43
|
+
m: typeof mode === "string" ? mode : "",
|
|
43
44
|
});
|
|
44
45
|
return createHash("sha256").update(canonical).digest("hex").slice(0, 32);
|
|
45
46
|
}
|
package/lib/history.js
CHANGED
|
@@ -115,13 +115,37 @@ export function isTrustedApiRequest(req) {
|
|
|
115
115
|
}
|
|
116
116
|
}
|
|
117
117
|
|
|
118
|
+
/**
|
|
119
|
+
* Project the module-level backend health map into a small observability
|
|
120
|
+
* payload (v2.8.0 P2): per-backend last real success/failure, latency and
|
|
121
|
+
* breaker state — no error strings, no credentials, safe for the GUI.
|
|
122
|
+
*/
|
|
123
|
+
export function toBackendObservability(healthMap, nowMs = Date.now()) {
|
|
124
|
+
const out = [];
|
|
125
|
+
if (healthMap instanceof Map) {
|
|
126
|
+
for (const [id, e] of healthMap) {
|
|
127
|
+
out.push({
|
|
128
|
+
id,
|
|
129
|
+
ok: e?.ok === true,
|
|
130
|
+
ms: Number.isFinite(e?.ms) ? e.ms : null,
|
|
131
|
+
at: Number.isFinite(e?.at) ? e.at : null,
|
|
132
|
+
failCount: Number.isFinite(e?.failCount) ? e.failCount : 0,
|
|
133
|
+
cooled: Number.isFinite(e?.cooledUntil) && e.cooledUntil > nowMs,
|
|
134
|
+
});
|
|
135
|
+
}
|
|
136
|
+
}
|
|
137
|
+
return out;
|
|
138
|
+
}
|
|
139
|
+
|
|
118
140
|
/**
|
|
119
141
|
* Register the read-only history routes on the (late-mounted) webServer.
|
|
120
|
-
* GET /api/websearch/history -> { ok, entries }
|
|
142
|
+
* GET /api/websearch/history -> { ok, entries, backends }
|
|
143
|
+
* backends: per-backend last real outcome/latency/breaker state (v2.8.0),
|
|
144
|
+
* so fallback redundancy is observable without a separate endpoint.
|
|
121
145
|
* POST /api/websearch/history/clear -> { ok, cleared: true }
|
|
122
146
|
* Both routes sit behind isTrustedApiRequest (loopback Host + same-site).
|
|
123
147
|
*/
|
|
124
|
-
export function registerHistoryRoutes(ws, { getStore }) {
|
|
148
|
+
export function registerHistoryRoutes(ws, { getStore, getBackendHealth }) {
|
|
125
149
|
if (!ws || typeof ws.register !== "function") return;
|
|
126
150
|
const send = (res, code, body) => {
|
|
127
151
|
res.writeHead(code, { "content-type": "application/json" });
|
|
@@ -144,7 +168,11 @@ export function registerHistoryRoutes(ws, { getStore }) {
|
|
|
144
168
|
return;
|
|
145
169
|
}
|
|
146
170
|
const store = getStore();
|
|
147
|
-
send(res, 200, {
|
|
171
|
+
send(res, 200, {
|
|
172
|
+
ok: true,
|
|
173
|
+
entries: store ? store.list() : [],
|
|
174
|
+
backends: toBackendObservability(typeof getBackendHealth === "function" ? getBackendHealth() : getBackendHealth),
|
|
175
|
+
});
|
|
148
176
|
}),
|
|
149
177
|
});
|
|
150
178
|
ws.register({
|
package/lib/index.js
CHANGED
|
@@ -169,6 +169,12 @@ export const Config = z.object({
|
|
|
169
169
|
historyEnabled: z.boolean().default(true)
|
|
170
170
|
.description("记录搜索历史【历史】:只追加,最多 50 条;可在设置 → 搜索 中查看与清空"),
|
|
171
171
|
|
|
172
|
+
// v2.8.0 查全查多
|
|
173
|
+
multiQueryEnabled: z.boolean().default(false)
|
|
174
|
+
.description("多查询融合【查全】:复杂查询(长度>60 或含 vs/and/比较/对比/和)自动派生 2-3 个子查询,各扇出后用 RRF(k=60) 融合;默认关"),
|
|
175
|
+
deepCoverage: z.boolean().default(false)
|
|
176
|
+
.description("深度覆盖【查全】:每后端请求条数上调 50% 再全局去重截断,牺牲带宽换覆盖面;Tavily 自动切 advanced 深度,SearXNG 扩展多类目;默认关"),
|
|
177
|
+
|
|
172
178
|
// Enable toggles (one per backend)
|
|
173
179
|
enableExa: z.boolean().default(true).description(DEFAULT_LABELS.exa),
|
|
174
180
|
enableParallel: z.boolean().default(true).description(DEFAULT_LABELS.parallel),
|
|
@@ -183,30 +189,48 @@ export const Config = z.object({
|
|
|
183
189
|
enableOpenai: z.boolean().default(false).description(DEFAULT_LABELS.openai),
|
|
184
190
|
|
|
185
191
|
// Credential refs (env-var names; secrets live in the credentials service)
|
|
186
|
-
braveApiKeyEnv: z.string().role("credential-ref").default(DEFAULT_KEY_ENV.brave)
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
192
|
+
braveApiKeyEnv: z.string().role("credential-ref").default(DEFAULT_KEY_ENV.brave)
|
|
193
|
+
.description("Brave API Key 的凭证引用(填环境变量名,实际密钥存凭证库)"),
|
|
194
|
+
tavilyApiKeyEnv: z.string().role("credential-ref").default(DEFAULT_KEY_ENV.tavily)
|
|
195
|
+
.description("Tavily API Key 的凭证引用(填环境变量名,实际密钥存凭证库)"),
|
|
196
|
+
serperApiKeyEnv: z.string().role("credential-ref").default(DEFAULT_KEY_ENV.serper)
|
|
197
|
+
.description("Serper API Key 的凭证引用(填环境变量名,实际密钥存凭证库)"),
|
|
198
|
+
mojeekApiKeyEnv: z.string().role("credential-ref").default(DEFAULT_KEY_ENV.mojeek)
|
|
199
|
+
.description("Mojeek API Key 的凭证引用(填环境变量名,实际密钥存凭证库)"),
|
|
200
|
+
deepseekApiKeyEnv: z.string().role("credential-ref").default(DEFAULT_KEY_ENV.deepseek)
|
|
201
|
+
.description("Deepseek API Key 的凭证引用(填环境变量名,实际密钥存凭证库)"),
|
|
202
|
+
anthropicApiKeyEnv: z.string().role("credential-ref").default(DEFAULT_KEY_ENV.anthropic)
|
|
203
|
+
.description("Anthropic API Key 的凭证引用(填环境变量名,实际密钥存凭证库)"),
|
|
204
|
+
openaiApiKeyEnv: z.string().role("credential-ref").default(DEFAULT_KEY_ENV.openai)
|
|
205
|
+
.description("Openai API Key 的凭证引用(填环境变量名,实际密钥存凭证库)"),
|
|
193
206
|
searxngApiKeyEnv: z.string().role("credential-ref").default("SEARXNG_API_KEY")
|
|
194
|
-
.description("SearXNG 私有实例的 API Key
|
|
207
|
+
.description("SearXNG 私有实例的 API Key 凭证引用(公共实例留空)"),
|
|
195
208
|
|
|
196
209
|
// Base URLs (defaults from built-in registry)
|
|
197
|
-
braveBaseURL: z.string().default(DEFAULT_BASE_URLS.brave)
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
210
|
+
braveBaseURL: z.string().default(DEFAULT_BASE_URLS.brave)
|
|
211
|
+
.description("Brave API 端点(默认官方;自托管/代理实例改此项)"),
|
|
212
|
+
tavilyBaseURL: z.string().default(DEFAULT_BASE_URLS.tavily)
|
|
213
|
+
.description("Tavily API 端点(默认官方;自托管/代理实例改此项)"),
|
|
214
|
+
serperBaseURL: z.string().default(DEFAULT_BASE_URLS.serper)
|
|
215
|
+
.description("Serper.dev API 端点(默认官方;自托管/代理实例改此项)"),
|
|
216
|
+
mojeekBaseURL: z.string().default(DEFAULT_BASE_URLS.mojeek)
|
|
217
|
+
.description("Mojeek API 端点(默认官方;自托管/代理实例改此项)"),
|
|
218
|
+
searxngBaseURL: z.string().default(DEFAULT_BASE_URLS.searxng)
|
|
219
|
+
.description("SearXNG API 端点(默认官方;自托管/代理实例改此项)"),
|
|
220
|
+
deepseekBaseURL: z.string().default(DEFAULT_BASE_URLS.deepseek)
|
|
221
|
+
.description("DeepSeek API 端点(默认官方;自托管/代理实例改此项)"),
|
|
222
|
+
anthropicBaseURL: z.string().default(DEFAULT_BASE_URLS.anthropic)
|
|
223
|
+
.description("Anthropic API 端点(默认官方;自托管/代理实例改此项)"),
|
|
224
|
+
openaiBaseURL: z.string().default(DEFAULT_BASE_URLS.openai)
|
|
225
|
+
.description("OpenAI API 端点(默认官方;自托管/代理实例改此项)"),
|
|
205
226
|
|
|
206
227
|
// Models
|
|
207
|
-
deepseekModel: z.string().default(DEFAULT_MODELS.deepseek)
|
|
208
|
-
|
|
209
|
-
|
|
228
|
+
deepseekModel: z.string().default(DEFAULT_MODELS.deepseek)
|
|
229
|
+
.description("DeepSeek 后端使用的模型名"),
|
|
230
|
+
anthropicModel: z.string().default(DEFAULT_MODELS.anthropic)
|
|
231
|
+
.description("Anthropic 后端使用的模型名"),
|
|
232
|
+
openaiModel: z.string().default(DEFAULT_MODELS.openai)
|
|
233
|
+
.description("OpenAI 后端使用的模型名"),
|
|
210
234
|
tavilySearchDepth: z.union(["basic", "advanced"]).default("basic")
|
|
211
235
|
.description("Tavily 搜索深度"),
|
|
212
236
|
|
|
@@ -215,6 +239,15 @@ export const Config = z.object({
|
|
|
215
239
|
.description("显式启用后端列表(优先于上方开关;留空则用开关)"),
|
|
216
240
|
});
|
|
217
241
|
|
|
242
|
+
// 0.2.0 SettingsForms adapter (cross-settings-forms.md ruling): the 0.2.0
|
|
243
|
+
// settings service auto-generates its form from entry.fiber.runtime.Config —
|
|
244
|
+
// but volatileForm (RT dsh-settings lib/index.js:122-138) keeps ONLY fields
|
|
245
|
+
// with meta.volatile; a Config without any yields an empty form. Marking the
|
|
246
|
+
// top object makes the whole Config the form. Direct meta assignment (not
|
|
247
|
+
// .volatile(), which returns a NEW schema and throws on re-wrap) is
|
|
248
|
+
// equivalent and idempotent on both the 0.1.5 and 0.2.0 schemastery lines.
|
|
249
|
+
Config.meta = { ...Config.meta, volatile: true };
|
|
250
|
+
|
|
218
251
|
/**
|
|
219
252
|
* Per-field user-language labels (fe-ui W1 remainder): the settings renderer
|
|
220
253
|
* falls back to camelCase keys when no label is present, so attach one to
|
|
@@ -326,6 +359,9 @@ export function resolveOptions(ctx, config) {
|
|
|
326
359
|
? Math.max(0, Math.floor(Number(config?.breakerCooldownMs) || 0)) // 0 = disabled
|
|
327
360
|
: BREAKER_DEFAULTS.cooldownMs,
|
|
328
361
|
historyEnabled: config?.historyEnabled !== false,
|
|
362
|
+
// v2.8.0 coverage / multiQuery
|
|
363
|
+
multiQueryEnabled: config?.multiQueryEnabled === true,
|
|
364
|
+
deepCoverage: config?.deepCoverage === true,
|
|
329
365
|
filters: {
|
|
330
366
|
recency: typeof config?.recency === "string" && config.recency !== "any" ? config.recency : undefined,
|
|
331
367
|
lang: language.length > 0 ? language : undefined,
|
|
@@ -359,6 +395,7 @@ export function apply(ctx, config) {
|
|
|
359
395
|
ctx.inject(["settings"], (settingsCtx) => {
|
|
360
396
|
const settings = settingsCtx.settings;
|
|
361
397
|
if (typeof settings?.installSection === "function") {
|
|
398
|
+
// 0.1.5 SettingsProvider line: explicit section install (unchanged).
|
|
362
399
|
settings.installSection(ctx, NAMESPACE, Config, config, {
|
|
363
400
|
setSource: (source) => {
|
|
364
401
|
current = source;
|
|
@@ -367,15 +404,10 @@ export function apply(ctx, config) {
|
|
|
367
404
|
});
|
|
368
405
|
return;
|
|
369
406
|
}
|
|
370
|
-
//
|
|
371
|
-
//
|
|
372
|
-
//
|
|
373
|
-
//
|
|
374
|
-
console.warn(
|
|
375
|
-
"[dsh-websearch] settings service has no installSection (DSH >= 0.1.7 SettingsForms line); " +
|
|
376
|
-
"the Settings panel section is not installed. Configure the unified-search namespace via cordis config " +
|
|
377
|
-
"or the client settings card; migration to the SettingsForms API is tracked for a future release.",
|
|
378
|
-
);
|
|
407
|
+
// 0.2.0 SettingsForms line: no installSection; the settings form is
|
|
408
|
+
// auto-generated from the volatile-marked Config above — no action and no
|
|
409
|
+
// warning here (the form IS the section; a warning would contradict the
|
|
410
|
+
// visible UI). Config values still resolve via cordis config.
|
|
379
411
|
});
|
|
380
412
|
|
|
381
413
|
// v2.7.0 stores: disk cache + append-only search history, both under
|
|
@@ -423,6 +455,10 @@ export function apply(ctx, config) {
|
|
|
423
455
|
getBackendHealth: getLatestBackendHealth,
|
|
424
456
|
});
|
|
425
457
|
// v2.7.0 search history: read-only GET + explicit clear POST.
|
|
426
|
-
|
|
458
|
+
// v2.8.0: GET also carries per-backend last-outcome observability.
|
|
459
|
+
registerHistoryRoutes(webCtx.webServer, {
|
|
460
|
+
getStore: () => historyStore,
|
|
461
|
+
getBackendHealth: getLatestBackendHealth,
|
|
462
|
+
});
|
|
427
463
|
});
|
|
428
464
|
}
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
// multiquery.js — v2.8.0 query-variant derivation (pure functions, zero deps).
|
|
2
|
+
//
|
|
3
|
+
// Gated multi-query: arXiv:2404.01037 (Wang et al.) shows BLIND multi-query
|
|
4
|
+
// can degrade retrieval — so variants are only derived for queries that look
|
|
5
|
+
// genuinely multi-faceted, and the original query always participates in the
|
|
6
|
+
// fusion with full weight. Splitting is purely mechanical (at separator
|
|
7
|
+
// words/punctuation); entity text is never rewritten.
|
|
8
|
+
//
|
|
9
|
+
// Separators per the v2.8 spec: "vs", "and", 比较, 对比, 和 — plus sentence
|
|
10
|
+
// punctuation for the long-query case.
|
|
11
|
+
|
|
12
|
+
const SEPARATOR_SPLIT_RE = /\s+(?:vs\.?|and)\s+|\s*(?:比较|对比|和)\s*|\s*[、;;。!?!?|]+\s*/i;
|
|
13
|
+
const COMPLEX_MARK_RE = /(?:\bvs\.?\b|\band\b|比较|对比|和)/i;
|
|
14
|
+
|
|
15
|
+
/** Sub-query hard cap (original + 2 variants max). */
|
|
16
|
+
export const MAX_SUB_QUERIES = 3;
|
|
17
|
+
|
|
18
|
+
/**
|
|
19
|
+
* Heuristic gate: does this query look multi-faceted enough to justify
|
|
20
|
+
* multi-query fan-out? Long queries (>60 chars) or queries containing a
|
|
21
|
+
* separator word qualify. Returns false for anything else — blind
|
|
22
|
+
* multi-query degrades (arXiv:2404.01037), so the default is NO.
|
|
23
|
+
*/
|
|
24
|
+
export function isComplexQuery(query) {
|
|
25
|
+
if (typeof query !== "string") return false;
|
|
26
|
+
const q = query.trim();
|
|
27
|
+
if (q.length === 0) return false;
|
|
28
|
+
if (q.length > 60) return true;
|
|
29
|
+
return COMPLEX_MARK_RE.test(q);
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
/**
|
|
33
|
+
* Derive 2-3 sub-query variants from a complex query: the original query
|
|
34
|
+
* (always first, full weight in fusion) plus up to two mechanical splits at
|
|
35
|
+
* separator boundaries. Never rewrites entity text; returns [query] when no
|
|
36
|
+
* usable split exists.
|
|
37
|
+
*/
|
|
38
|
+
export function deriveSubQueries(query) {
|
|
39
|
+
const base = typeof query === "string" ? query.trim() : "";
|
|
40
|
+
if (base.length === 0) return [];
|
|
41
|
+
const parts = base
|
|
42
|
+
.split(SEPARATOR_SPLIT_RE)
|
|
43
|
+
.map((p) => p.trim())
|
|
44
|
+
.filter((p) => p.length >= 2);
|
|
45
|
+
if (parts.length < 2) return [base];
|
|
46
|
+
const variants = [base, parts[0], parts.slice(1).join(" ")];
|
|
47
|
+
return [...new Set(variants.map((v) => v.trim()).filter(Boolean))].slice(0, MAX_SUB_QUERIES);
|
|
48
|
+
}
|
package/lib/provider.js
CHANGED
|
@@ -25,9 +25,27 @@ import { normalizeFilters } from "./util/filters.js";
|
|
|
25
25
|
import { shapeQuery, resultHeader, TELEMETRY_TAG } from "./prompting.js";
|
|
26
26
|
import { cacheKeyFor } from "./cache.js";
|
|
27
27
|
import { foldBreaker, isCooled, cooledAnnotation } from "./breaker.js";
|
|
28
|
+
import { rrfFuse } from "./rrf.js";
|
|
29
|
+
import { isComplexQuery, deriveSubQueries } from "./multiquery.js";
|
|
28
30
|
|
|
29
31
|
const PROVIDER_ID = "unified";
|
|
30
32
|
|
|
33
|
+
// Per-backend timeout ceilings (v2.8.0, P2): ddg/scrapes and self-hosted
|
|
34
|
+
// SearXNG instances are the slowest to fail when unreachable — a 30s global
|
|
35
|
+
// default lets one dead backend stall the whole fan-out tail. The effective
|
|
36
|
+
// timeout is min(override, backendTimeoutMs), so an operator's shorter global
|
|
37
|
+
// timeout always wins.
|
|
38
|
+
const BACKEND_TIMEOUT_OVERRIDES_MS = {
|
|
39
|
+
ddg: 5000,
|
|
40
|
+
searxng: 5000,
|
|
41
|
+
};
|
|
42
|
+
|
|
43
|
+
/** Effective per-backend timeout = min(override ?? ∞, global). Exported for tests. */
|
|
44
|
+
export function effectiveBackendTimeoutMs(id, backendTimeoutMs) {
|
|
45
|
+
const override = BACKEND_TIMEOUT_OVERRIDES_MS[id];
|
|
46
|
+
return Number.isFinite(override) ? Math.min(override, backendTimeoutMs) : backendTimeoutMs;
|
|
47
|
+
}
|
|
48
|
+
|
|
31
49
|
// Module-level per-backend health telemetry (id -> entry), updated on every
|
|
32
50
|
// backend completion during a real search. Purely passive: nothing writes here
|
|
33
51
|
// outside search(), so the read-only /api/unified-search/health route can
|
|
@@ -177,14 +195,25 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
|
|
|
177
195
|
const concurrency = opts.concurrency ?? 6;
|
|
178
196
|
const backendTimeoutMs = opts.backendTimeoutMs ?? 30000;
|
|
179
197
|
|
|
180
|
-
// v2.
|
|
181
|
-
//
|
|
182
|
-
|
|
183
|
-
//
|
|
198
|
+
// v2.8.0 coverage / multiQuery gates (both default OFF; arXiv:2404.01037
|
|
199
|
+
// warns blind multi-query degrades, so the gate is deliberately strict).
|
|
200
|
+
const coverage = opts.deepCoverage === true;
|
|
201
|
+
// Per-backend widening: fetch more per backend, dedupe down afterwards.
|
|
202
|
+
const backendMaxResults = coverage ? Math.ceil(maxResults * 1.5) : maxResults;
|
|
203
|
+
const multi = opts.multiQueryEnabled === true && isComplexQuery(effReq.query);
|
|
204
|
+
const variants = multi ? deriveSubQueries(effReq.query) : [effReq.query];
|
|
205
|
+
// Cache-key mode marker: different coverage/multiQuery settings must not
|
|
206
|
+
// serve each other's cached results.
|
|
207
|
+
const cacheMode = (multi ? "multi" : "single") + (coverage ? "+coverage" : "");
|
|
208
|
+
|
|
209
|
+
// v2.7.0 disk cache: keyed on (shaped query + filters + maxResults +
|
|
210
|
+
// backends + mode). A hit short-circuits the fan-out entirely; sources
|
|
211
|
+
// are byte-identical to the first search and content carries a
|
|
212
|
+
// "cache hit, age Ns" note. Cache failures never break the search.
|
|
184
213
|
const cacheEnabled = opts.cacheEnabled === true && cache;
|
|
185
214
|
let cacheKey = null;
|
|
186
215
|
if (cacheEnabled) {
|
|
187
|
-
cacheKey = cacheKeyFor({ query: effReq.query, filters, maxResults, backends: opts.enabledBackends });
|
|
216
|
+
cacheKey = cacheKeyFor({ query: effReq.query, filters, maxResults, backends: opts.enabledBackends, mode: cacheMode });
|
|
188
217
|
const hit = cache.get(cacheKey);
|
|
189
218
|
if (hit && Array.isArray(hit.value?.sources) && hit.value.sources.length > 0) {
|
|
190
219
|
// Defense in depth: the fresh path drops non-http(s) schemes before
|
|
@@ -219,7 +248,7 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
|
|
|
219
248
|
// backend inside its 60s cooldown window is skipped (annotated in
|
|
220
249
|
// telemetry as "id ⏸cooled"). Fail-open: if EVERY candidate is cooled,
|
|
221
250
|
// run them all anyway — the breaker must never make search worse than
|
|
222
|
-
// no breaker.
|
|
251
|
+
// no breaker. The pick is shared by every query variant.
|
|
223
252
|
const breakerCfg = opts.breakerEnabled === false ? null : {
|
|
224
253
|
threshold: opts.breakerThreshold,
|
|
225
254
|
cooldownMs: opts.breakerCooldownMs,
|
|
@@ -255,98 +284,109 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
|
|
|
255
284
|
);
|
|
256
285
|
}
|
|
257
286
|
|
|
258
|
-
//
|
|
259
|
-
//
|
|
260
|
-
//
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
let bu = null;
|
|
278
|
-
try { bu = new URL(beOpts.baseURL); } catch { bu = null; }
|
|
279
|
-
const privHost = bu ? /^(localhost|127(\.\d{1,3}){3}|10(\.\d{1,3}){3}|192\.168(\.\d{1,3}){3}|172\.(1[6-9]|2\d|3[01])(\.\d{1,3}){3})$/.test(bu.hostname) : false;
|
|
280
|
-
const schemeOk = bu && (bu.protocol === "https:" || (bu.protocol === "http:" && privHost));
|
|
281
|
-
if (!schemeOk || bu.username || bu.password) {
|
|
282
|
-
throw new WebError(`backend "${id}" has an invalid baseURL: https required (plain http only for private hosts), credentials-in-URL rejected`, "WEB_PROVIDER_ERROR");
|
|
287
|
+
// Run ONE query variant through the full fan-out: breaker-aware backend
|
|
288
|
+
// execution under the per-variant limiter, then PHASE 1 collect. The
|
|
289
|
+
// internal abort/timeout semantics are unchanged from v2.7.
|
|
290
|
+
const fanoutVariant = async (variantReq) => {
|
|
291
|
+
const telemetry = [];
|
|
292
|
+
const limit = createLimiter(concurrency);
|
|
293
|
+
const runBackend = ({ id, backend, beOpts, hooks }) => {
|
|
294
|
+
const startedAtMs = Date.now();
|
|
295
|
+
// Trust boundary: baseURL is operator-configurable and API keys are sent to it.
|
|
296
|
+
// Enforce http(s); plain http only for private hosts (self-hosted SearXNG).
|
|
297
|
+
// Reject embedded credentials. Failure is a per-backend soft error.
|
|
298
|
+
if (beOpts.baseURL !== undefined) {
|
|
299
|
+
let bu = null;
|
|
300
|
+
try { bu = new URL(beOpts.baseURL); } catch { bu = null; }
|
|
301
|
+
const privHost = bu ? /^(localhost|127(\.\d{1,3}){3}|10(\.\d{1,3}){3}|192\.168(\.\d{1,3}){3}|172\.(1[6-9]|2\d|3[01])(\.\d{1,3}){3})$/.test(bu.hostname) : false;
|
|
302
|
+
const schemeOk = bu && (bu.protocol === "https:" || (bu.protocol === "http:" && privHost));
|
|
303
|
+
if (!schemeOk || bu.username || bu.password) {
|
|
304
|
+
throw new WebError(`backend "${id}" has an invalid baseURL: https required (plain http only for private hosts), credentials-in-URL rejected`, "WEB_PROVIDER_ERROR");
|
|
305
|
+
}
|
|
283
306
|
}
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
307
|
+
const controller = new AbortController();
|
|
308
|
+
const mergedSignal = signal
|
|
309
|
+
? AbortSignal.any([signal, controller.signal])
|
|
310
|
+
: controller.signal;
|
|
311
|
+
const effectiveTimeoutMs = effectiveBackendTimeoutMs(id, backendTimeoutMs);
|
|
312
|
+
const timeoutId = setTimeout(() => controller.abort(), effectiveTimeoutMs);
|
|
313
|
+
return backend
|
|
314
|
+
.search(variantReq, mergedSignal, beOpts, hooks)
|
|
315
|
+
.then(
|
|
316
|
+
(value) => {
|
|
317
|
+
telemetry.push({ id, ok: true, ms: Date.now() - startedAtMs, count: value?.sources?.length ?? 0 });
|
|
318
|
+
recordBackendHealth(id, { ok: true, at: Date.now(), ms: Date.now() - startedAtMs, count: value?.sources?.length ?? 0 }, breakerCfg);
|
|
319
|
+
return value;
|
|
320
|
+
},
|
|
321
|
+
(err) => {
|
|
322
|
+
const msg = String(err instanceof Error ? err.message : err).slice(0, 80);
|
|
323
|
+
telemetry.push({ id, ok: false, ms: Date.now() - startedAtMs, error: msg });
|
|
324
|
+
recordBackendHealth(id, { ok: false, at: Date.now(), ms: Date.now() - startedAtMs, error: msg }, breakerCfg);
|
|
325
|
+
throw err;
|
|
326
|
+
},
|
|
327
|
+
)
|
|
328
|
+
.catch((err) => {
|
|
329
|
+
if (controller.signal.aborted && !(signal && signal.aborted)) {
|
|
330
|
+
throw new WebError(
|
|
331
|
+
`backend "${id}" timed out after ${effectiveTimeoutMs}ms`,
|
|
332
|
+
"WEB_PROVIDER_ERROR",
|
|
333
|
+
{ cause: err instanceof Error ? err : undefined },
|
|
334
|
+
);
|
|
335
|
+
}
|
|
302
336
|
throw err;
|
|
303
|
-
}
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
if (controller.signal.aborted && !(signal && signal.aborted)) {
|
|
307
|
-
throw new WebError(
|
|
308
|
-
`backend "${id}" timed out after ${backendTimeoutMs}ms`,
|
|
309
|
-
"WEB_PROVIDER_ERROR",
|
|
310
|
-
{ cause: err instanceof Error ? err : undefined },
|
|
311
|
-
);
|
|
312
|
-
}
|
|
313
|
-
throw err;
|
|
314
|
-
})
|
|
315
|
-
.finally(() => clearTimeout(timeoutId));
|
|
316
|
-
};
|
|
317
|
-
|
|
318
|
-
const results = await Promise.allSettled(
|
|
319
|
-
eligible.map(({ id, backend, beOpts }) => {
|
|
320
|
-
const hooks = {
|
|
321
|
-
recordRequest: (req) => recordBackendRequest(ctx, id, req),
|
|
322
|
-
recordOutcome: (out) => recordBackendOutcome(ctx, id, out),
|
|
323
|
-
};
|
|
324
|
-
return limit(() => runBackend({ id, backend, beOpts, hooks }));
|
|
325
|
-
}),
|
|
326
|
-
);
|
|
327
|
-
|
|
328
|
-
throwIfSearchAborted(signal);
|
|
337
|
+
})
|
|
338
|
+
.finally(() => clearTimeout(timeoutId));
|
|
339
|
+
};
|
|
329
340
|
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
341
|
+
const results = await Promise.allSettled(
|
|
342
|
+
eligible.map(({ id, backend, beOpts }) => {
|
|
343
|
+
const hooks = {
|
|
344
|
+
recordRequest: (req) => recordBackendRequest(ctx, id, req),
|
|
345
|
+
recordOutcome: (out) => recordBackendOutcome(ctx, id, out),
|
|
346
|
+
};
|
|
347
|
+
return limit(() => runBackend({ id, backend, beOpts, hooks }));
|
|
348
|
+
}),
|
|
349
|
+
);
|
|
350
|
+
throwIfSearchAborted(signal);
|
|
351
|
+
const fold = collectFanoutResults(results, eligible, signal);
|
|
352
|
+
return { ...fold, telemetry };
|
|
353
|
+
};
|
|
334
354
|
|
|
335
|
-
//
|
|
336
|
-
|
|
337
|
-
|
|
355
|
+
// Fan out per variant (sequential — the limiter already parallelizes
|
|
356
|
+
// backends within a variant), dedupe/rank each list, then fuse.
|
|
357
|
+
const perVariant = [];
|
|
358
|
+
const allTelemetry = [];
|
|
359
|
+
const allFailures = [];
|
|
360
|
+
let hardError = null;
|
|
361
|
+
for (const variantQuery of variants) {
|
|
362
|
+
const variantReq = {
|
|
363
|
+
...effReq,
|
|
364
|
+
query: variantQuery,
|
|
365
|
+
maxResults: backendMaxResults,
|
|
366
|
+
...(coverage ? { deepCoverage: true } : {}),
|
|
367
|
+
};
|
|
368
|
+
const fold = await fanoutVariant(variantReq);
|
|
369
|
+
allTelemetry.push(...fold.telemetry);
|
|
370
|
+
allFailures.push(...fold.failures);
|
|
371
|
+
if (fold.hardError !== null && hardError === null) hardError = fold.hardError;
|
|
372
|
+
// Caller-cancellation check (hard abort fires before we yield results).
|
|
373
|
+
if (signal?.aborted === true || fold.hardAborted) {
|
|
374
|
+
throw searchAborted(signal);
|
|
375
|
+
}
|
|
376
|
+
// PHASE 2 dedupe / PHASE 3 rerank (per variant list, pre-fusion).
|
|
377
|
+
const strategy = opts.dedupStrategy === "url+title" ? "url+title" : "url";
|
|
378
|
+
const deduped = dedupeSources(fold.collected, strategy);
|
|
379
|
+
const ordered = opts.rerank === true ? rerankSources(deduped, variantQuery) : deduped;
|
|
380
|
+
perVariant.push({ query: variantQuery, ordered, mergedContent: fold.mergedContent });
|
|
338
381
|
}
|
|
339
382
|
|
|
340
|
-
// PHASE
|
|
341
|
-
//
|
|
342
|
-
//
|
|
343
|
-
const
|
|
344
|
-
const
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
opts.rerank === true
|
|
348
|
-
? rerankSources(deduped, typeof request?.query === "string" ? request.query : "")
|
|
349
|
-
: deduped;
|
|
383
|
+
// PHASE 4 fuse: multi-query mode fuses the per-variant ranked lists with
|
|
384
|
+
// RRF (k=60); single-query mode keeps the historical behavior (the one
|
|
385
|
+
// list, unchanged ordering).
|
|
386
|
+
const orderedLists = perVariant.filter((v) => v.ordered.length > 0).map((v) => v.ordered);
|
|
387
|
+
const ordered = multi
|
|
388
|
+
? rrfFuse(orderedLists).fused
|
|
389
|
+
: (orderedLists[0] ?? []);
|
|
350
390
|
|
|
351
391
|
// If we got ANY sources, return them even if some backends failed.
|
|
352
392
|
if (ordered.length > 0) {
|
|
@@ -355,7 +395,7 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
|
|
|
355
395
|
// 置于结果块最前,要求逐条引用来源 URL、优先官方/一手与较新来源;
|
|
356
396
|
// 语言按查询文本自动选择(含 CJK 用中文头)。可用
|
|
357
397
|
// opts.resultHeader === false 关闭。
|
|
358
|
-
let outContent = mergedContent;
|
|
398
|
+
let outContent = perVariant.find((v) => v.mergedContent !== undefined)?.mergedContent;
|
|
359
399
|
if (opts.resultHeader !== false && (sources.length > 0 || outContent !== undefined)) {
|
|
360
400
|
const header = resultHeader(typeof request?.query === "string" && /[\u4e00-\u9fff]/.test(request.query) ? "zh" : "en");
|
|
361
401
|
outContent = outContent === undefined ? header : header + "\n\n" + outContent;
|
|
@@ -369,18 +409,22 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
|
|
|
369
409
|
query: request?.query,
|
|
370
410
|
time: Date.now(),
|
|
371
411
|
resultCount: sources.length,
|
|
372
|
-
backendsOk:
|
|
373
|
-
backendsTotal:
|
|
412
|
+
backendsOk: allTelemetry.filter((t) => t.ok).length,
|
|
413
|
+
backendsTotal: allTelemetry.length,
|
|
374
414
|
});
|
|
375
|
-
if (opts.resultTelemetry !== false && (
|
|
415
|
+
if (opts.resultTelemetry !== false && (allTelemetry.length > 0 || cooled.size > 0)) {
|
|
376
416
|
const cooledParts = [];
|
|
377
417
|
for (const [cid, centry] of cooled) {
|
|
378
418
|
const ann = cooledAnnotation(centry, Date.now()) || "⏸cooled";
|
|
379
419
|
cooledParts.push(cid + " " + ann);
|
|
380
420
|
}
|
|
381
|
-
|
|
421
|
+
// Multi-query fans every variant through every backend; cap the line
|
|
422
|
+
// so the telemetry note cannot dwarf the results themselves.
|
|
423
|
+
const parts = allTelemetry
|
|
424
|
+
.slice(0, 12)
|
|
382
425
|
.map((t) => t.id + (t.ok ? " ✓" : " ✗") + t.ms + "ms" + (t.ok ? "/" + t.count : (t.error ? " (" + String(t.error).slice(0, 40) + ")" : "")))
|
|
383
426
|
.concat(cooledParts);
|
|
427
|
+
if (allTelemetry.length > 12) parts.push(`… +${allTelemetry.length - 12}`);
|
|
384
428
|
const line = "[websearch backends]" + TELEMETRY_TAG + " " + parts.join(" · ");
|
|
385
429
|
outContent = outContent === undefined ? line : outContent + "\n\n" + line;
|
|
386
430
|
}
|
|
@@ -393,17 +437,17 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cac
|
|
|
393
437
|
|
|
394
438
|
// All enabled backends failed - synthesize one meaningful error that
|
|
395
439
|
// names each backend's own reason so the caller (model or user) can act.
|
|
396
|
-
const detail =
|
|
440
|
+
const detail = allFailures.slice(0, 4)
|
|
397
441
|
.map((f) => f.id + ": " + String(f.err instanceof Error ? f.err.message : f.err).slice(0, 120))
|
|
398
442
|
.join("; ");
|
|
399
443
|
const allKeyMissing =
|
|
400
|
-
|
|
401
|
-
|
|
444
|
+
allFailures.length > 0 &&
|
|
445
|
+
allFailures.every((f) => f.err instanceof WebError && (f.err.code === "WEB_PROVIDER_UNAVAILABLE" || f.err.code === "WEB_PROVIDER_CREDENTIAL_MISSING"));
|
|
402
446
|
const hint = allKeyMissing
|
|
403
447
|
? " (every enabled backend is missing its API key - add keys in Settings > Web Search, or enable a keyless backend)"
|
|
404
448
|
: "";
|
|
405
449
|
throw new WebError(
|
|
406
|
-
`unified search: all enabled backends failed (${
|
|
450
|
+
`unified search: all enabled backends failed (${allFailures.length}) - ${detail}` + hint,
|
|
407
451
|
"WEB_PROVIDER_ERROR",
|
|
408
452
|
{ cause: hardError ?? undefined },
|
|
409
453
|
);
|
package/lib/rrf.js
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
// rrf.js — v2.8.0 Reciprocal Rank Fusion (pure functions, zero deps).
|
|
2
|
+
//
|
|
3
|
+
// RRF merges multiple ranked result lists into one: each list contributes
|
|
4
|
+
// score 1/(k + rank) to every item it contains (k=60 is the canonical value
|
|
5
|
+
// from Cormack et al. 2009, popularized by RAG-Fusion
|
|
6
|
+
// https://github.com/Raudaschl/rag-fusion). Items are deduped by URL; the
|
|
7
|
+
// first-seen entry wins for metadata, scores accumulate across lists.
|
|
8
|
+
//
|
|
9
|
+
// Why RRF over naive concatenation: it needs no score calibration across
|
|
10
|
+
// backends/queries (only ranks), and demotes one-off hits while promoting
|
|
11
|
+
// items that appear in several lists.
|
|
12
|
+
|
|
13
|
+
/** Canonical RRF constant. */
|
|
14
|
+
export const RRF_K = 60;
|
|
15
|
+
|
|
16
|
+
/** Stable key for dedupe (URL string; missing/odd URLs fall back to title). */
|
|
17
|
+
function itemKey(item) {
|
|
18
|
+
if (item && typeof item.url === "string" && item.url.length > 0) return item.url;
|
|
19
|
+
if (item && typeof item.title === "string") return `t:${item.title}`;
|
|
20
|
+
return "";
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
/**
|
|
24
|
+
* Fuse ranked lists of search sources with Reciprocal Rank Fusion.
|
|
25
|
+
*
|
|
26
|
+
* @param {Array<Array<object>>} lists each list is already ranked best-first
|
|
27
|
+
* @param {{k?: number}} [opts] k constant (default 60)
|
|
28
|
+
* @returns {{ fused: object[], scores: Map<string, number> }}
|
|
29
|
+
* fused: sources sorted by accumulated RRF score desc (ties keep first
|
|
30
|
+
* appearance order); sources without a usable key are dropped.
|
|
31
|
+
*/
|
|
32
|
+
export function rrfFuse(lists, { k = RRF_K } = {}) {
|
|
33
|
+
const kk = Number(k) > 0 ? Number(k) : RRF_K;
|
|
34
|
+
const scores = new Map(); // key -> accumulated score
|
|
35
|
+
const firstSeen = new Map(); // key -> source object (first metadata wins)
|
|
36
|
+
const order = []; // keys in first-appearance order (stable ties)
|
|
37
|
+
|
|
38
|
+
for (const list of Array.isArray(lists) ? lists : []) {
|
|
39
|
+
if (!Array.isArray(list)) continue;
|
|
40
|
+
for (let rank = 0; rank < list.length; rank++) {
|
|
41
|
+
const item = list[rank];
|
|
42
|
+
const key = itemKey(item);
|
|
43
|
+
if (!key) continue;
|
|
44
|
+
scores.set(key, (scores.get(key) ?? 0) + 1 / (kk + rank));
|
|
45
|
+
if (!firstSeen.has(key)) {
|
|
46
|
+
firstSeen.set(key, item);
|
|
47
|
+
order.push(key);
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
const fused = [...order]
|
|
53
|
+
.sort((a, b) => (scores.get(b) ?? 0) - (scores.get(a) ?? 0))
|
|
54
|
+
.map((key) => firstSeen.get(key));
|
|
55
|
+
return { fused, scores };
|
|
56
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@240xu/dsh-websearch",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.8.1",
|
|
4
4
|
"description": "Unified web search provider for DeepSeek Harness (ctx.web): 11 backends — Exa + Parallel + DDG + SearXNG (keyless), DeepSeek + Anthropic + OpenAI + Brave + Tavily + Serper + Mojeek (key-gated via Settings credentials). Registers provider id \"unified\" for zero-config web_search. 统一网页搜索聚合:11 后端并发扇出、去重合并,零依赖。See CHANGELOG.md for version history.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "lib/index.js",
|