@240xu/dsh-websearch 2.4.0 → 2.7.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +57 -1
- package/lib/breaker.js +65 -0
- package/lib/cache.js +151 -0
- package/lib/client.js +42 -1
- package/lib/health.js +103 -0
- package/lib/history.js +163 -0
- package/lib/index.js +137 -11
- package/lib/prompting.js +105 -0
- package/lib/provider.js +184 -47
- package/package.json +5 -2
package/README.md
CHANGED
|
@@ -115,6 +115,53 @@ pnpm install
|
|
|
115
115
|
|
|
116
116
|
环境变量兜底:未在 credentials 配置时回退读同名环境变量;`DSH_UNIFIED_SEARCH_BACKENDS` 可逗号分隔强制指定启用集合。
|
|
117
117
|
|
|
118
|
+
## v2.7.3 兼容修复 | What's new in v2.7.3
|
|
119
|
+
|
|
120
|
+
- **DSH 0.1.7 客户端兼容(P0)**:settingsScope 客户端服务在 0.1.7 被移除;现改为 feature-detect——0.1.5(或装 dsh-settings-scope-shim)行为不变,0.1.7 无 shim 时设置卡片优雅停用(console.warn 指引),客户端不再静默全死。
|
|
121
|
+
- **DSH 0.1.7 服务端(P1)**:SettingsForms 无 installSection,原 typeof 守卫静默跳过;现输出一次性 console.warn(搜索功能经 cordis config 不受影响),SettingsForms 迁移已立牌。
|
|
122
|
+
- **engines**:声明 `node >= 20.3`(AbortSignal.any 实测需求,Node 18/20.0-20.2 首次搜索即 TypeError)。
|
|
123
|
+
- description 瘦身(版本历史移回本文件/CHANGELOG.md)。
|
|
124
|
+
- **Windows 双端(P2)**:缓存/历史目录回退改用 `os.homedir()`(此前 `process.env.HOME` 在 Windows 通常未设 → 相对路径 → 缓存静默失效)。
|
|
125
|
+
|
|
126
|
+
## v2.7.2 补丁 | What's new in v2.7.2
|
|
127
|
+
|
|
128
|
+
- 设置项 label:6 个新设置项在 schema 层挂用户语言 label(含单位尾注,如「缓存有效期(秒)」),宿主渲染不再回退 camelCase 键名;description 同步以人话名称开头(真机渲染行为归视觉待办 V5)。
|
|
129
|
+
|
|
130
|
+
## v2.7.1 评审修复 | What's new in v2.7.1
|
|
131
|
+
|
|
132
|
+
- **缓存 key 加入 enabledBackends 选择集**(arch-review P1-1):切换后端组合后立即拿到与新配置一致的结果,不再吃旧组合缓存。
|
|
133
|
+
- **缓存命中路径复用 URL scheme 白名单**(纵深防御):被篡改的缓存文件里的 javascript:/data: 条目在命中时同样被过滤。
|
|
134
|
+
- **history 并发写串行化**:record() 经进程内 promise 链串行,杜绝并发搜索完成时的丢失更新;单条写入仍为 tmp+rename 原子替换。
|
|
135
|
+
- **history 路由信任围栏**:两条 /api/websearch/* 路由要求回环 Host(防 DNS rebinding)+ same-site 浏览器上下文(防跨站 text/plain CSRF);无头客户端(curl)放行只读 GET。
|
|
136
|
+
- **熔断冷却时长支持 0**:`breakerCooldownMs=0` 显式关闭冷却(此前静默回退 60s)。
|
|
137
|
+
- **设置文案**:6 个新设置项改用户语言并按【缓存】/【熔断】/【历史】分组,单位(秒/毫秒)写入说明。
|
|
138
|
+
- **互操作**:新增 window 事件 `dsh-websearch:open-settings`(devkit 命令联动);收到时经 devkit toast 提示设置路径(宿主无分区跳转 API)。
|
|
139
|
+
|
|
140
|
+
## v2.7.0 新增 | What's new in v2.7.0
|
|
141
|
+
|
|
142
|
+
三大增量功能,全部经设置开关控制、默认为保守值,且不改变既有结果语义(v2.6.0 提示词原样保留):
|
|
143
|
+
|
|
144
|
+
### 1. 搜索结果磁盘缓存(默认开)
|
|
145
|
+
|
|
146
|
+
- 相同 **(整形后查询 + filters + maxResults + 启用后端集合)** 在 TTL 内直接命中缓存,不打任何后端;`sources` 与首次搜索完全一致。
|
|
147
|
+
- 命中时 `content` 追加一行 `[websearch cache] cache hit, age Ns`。
|
|
148
|
+
- 存储位置:`$DSH_HOME/cache/websearch/results/`,单条目一个 JSON 文件,tmp+rename 原子写,超出上限按最旧淘汰;任何 fs 错误都被吞掉,缓存坏了绝不影响搜索。
|
|
149
|
+
- 设置:`cacheEnabled`(默认 true)、`cacheTtl`(秒,默认 900)。
|
|
150
|
+
|
|
151
|
+
### 2. 后端熔断器(默认开,保守阈值)
|
|
152
|
+
|
|
153
|
+
- 同一后端**连续失败 3 次**进入 **60s 冷却**;冷却期内 eligible 选择直接跳过该后端,遥测行标注 `id ⏸cooled Ns`。
|
|
154
|
+
- 状态与 `recordBackendHealth` 共用(健康条目新增 `failCount` / `cooledUntil` 字段),任意一次成功立即复位。
|
|
155
|
+
- **Fail-open 兜底**:若所有候选后端都处于冷却期,则全部放行照常扇出——熔断器永远只会让搜索更快,不会让它更差。
|
|
156
|
+
- 设置:`breakerEnabled`(默认 true)、`breakerThreshold`(默认 3)、`breakerCooldownMs`(默认 60000)。
|
|
157
|
+
|
|
158
|
+
### 3. 搜索历史 API(默认开,只追加 / 只读暴露)
|
|
159
|
+
|
|
160
|
+
- 每次搜索(含缓存命中)追加一条记录,环形上限 50 条,新在前;文件 `$DSH_HOME/cache/websearch/history.json`,写前原子替换。
|
|
161
|
+
- `GET /api/websearch/history` → `{ ok, entries: [{ query, time, resultCount, backendsOk, backendsTotal }] }`(只读,无凭证、无 URL)。
|
|
162
|
+
- `POST /api/websearch/history/clear` → `{ ok, cleared: true }` 显式清空。
|
|
163
|
+
- 设置:`historyEnabled`(默认 true)。
|
|
164
|
+
|
|
118
165
|
## Design | 设计
|
|
119
166
|
|
|
120
167
|
- **One provider, no ambiguity**: a single `registeredSearchProvider({id:"unified"})` — the `dsh-web` seam's selection rule picks it unambiguously, and `search()` caps `maxResults` itself.
|
|
@@ -160,7 +207,16 @@ tests/
|
|
|
160
207
|
node --test tests/
|
|
161
208
|
```
|
|
162
209
|
|
|
163
|
-
|
|
210
|
+
v2.7.0 起共 76+ tests pass(`node --test "tests/*.test.js" "test/*.test.mjs"`)。
|
|
211
|
+
|
|
212
|
+
## 权威依据与取舍 | Authoritative sources & design trade-offs
|
|
213
|
+
|
|
214
|
+
v2.7.0 缓存/熔断/历史三项的依据与刻意取舍(含查证过的反例):
|
|
215
|
+
|
|
216
|
+
1. **熔断器(Circuit Breaker)**:出处 Michael T. Nygard《Release It!》(Pragmatic Bookshelf, 2007) 与 Microsoft Azure Architecture Center「Circuit Breaker pattern」https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker 。经典状态机是 Closed → Open → Half-Open(半开态放行少量探测请求)。**本插件刻意省略半开态**:冷却到期即视为 Closed(下一次真实搜索就是探测),失败则立即重新开窗——因为这里的"请求"是一次多后端扇出中的单个后端调用,探测请求必然真实发生且失败成本只是一次软降级,无需额外探测计数器;同时保留 fail-open 兜底(全部冷却时放行),比经典模式更保守。
|
|
217
|
+
2. **缓存失效(TTL-only vs 事件驱动)**:Cloudflare「Retention vs Freshness (TTL)」https://developers.cloudflare.com/cache/concepts/retention-vs-freshness/ 与「Revalidation」https://developers.cloudflare.com/cache/concepts/revalidation/ 。搜索结果没有可靠的原站变更信号可订阅(网页索引持续变化),事件驱动失效无锚点;TTL-only + 把 TTL 做成设置项(默认 900s)+ key 纳入后端组合,是"可接受的最终一致性"。取舍:900s 内索引更新不可见——对搜索场景可接受,且用户可调低。
|
|
218
|
+
3. **原子写(tmp + renameSync)**:POSIX 上 rename(2) 原子替换(Node 文档:newPath 已存在时覆盖);Windows 上 libuv 将 fs.rename 映射为 MoveFileExW(MOVEFILE_REPLACE_EXISTING)(libuv#283 https://github.com/joyent/libuv/issues/283 ,LWN 讨论 https://lwn.net/Articles/682988/ 指出 MS 文档不承诺该调用的原子性)。**结论:POSIX 双端原子;Windows 实践上等价但官方文档不背书**——因此本插件在写入端额外做了兜底:读端永远校验 JSON 完整性、坏条目按 miss 处理并删除,即使 Windows 上出现半写状态也不会返回坏数据。
|
|
219
|
+
4. **搜索历史隐私**:Article 29 Working Party 意见书 WP148 https://ec.europa.eu/justice/article-29/documentation/opinion-recommendation/files/2008/wp148_en.pdf 认定查询日志(query + 时间 + 来源标识)属个人数据;EFF 六条自保建议 https://w2.eff.org/Privacy/search/searchtips.pdf 的第一条是"别在搜索词里放 PII"——说明 query 天然可能携带 PII(人名、账号、地址)。本插件的取舍:历史只存 {query, time, resultCount, backendsOk/Total}(无 IP、无 cookie、无结果 URL)、环形 50 条自动滚动即短保留、本机存储不出网、提供 historyEnabled 开关与 POST clear 端点。**未做 query 脱敏**:脱敏会破坏"查看最近搜了什么"的核心用途,且本机单用户场景下该数据本就在 DSH 会话日志里存在;若未来历史跨设备同步,脱敏/加密是上线前置条件。
|
|
164
220
|
|
|
165
221
|
## License | 许可
|
|
166
222
|
|
package/lib/breaker.js
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
// breaker.js — v2.7.0 per-backend circuit breaker (pure functions, zero deps).
|
|
2
|
+
//
|
|
3
|
+
// State lives ON the backend health entry (the same object recordBackendHealth
|
|
4
|
+
// in provider.js maintains), as two extra fields:
|
|
5
|
+
// failCount — consecutive failures (reset to 0 on any success)
|
|
6
|
+
// cooledUntil — epoch ms until which the backend is skipped; 0 = closed
|
|
7
|
+
// Breaker opens when failCount reaches threshold and stays open for
|
|
8
|
+
// cooldownMs. Failure modes are deliberately conservative: unknown state or
|
|
9
|
+
// missing fields never trip; a fully-cooled fan-out falls back to running
|
|
10
|
+
// every backend (fail-open) so the breaker can never make search WORSE than
|
|
11
|
+
// no breaker.
|
|
12
|
+
|
|
13
|
+
/** Default breaker tuning (settings can override; cooldownMs 0 = disabled). */
|
|
14
|
+
export const BREAKER_DEFAULTS = {
|
|
15
|
+
threshold: 3,
|
|
16
|
+
cooldownMs: 60_000,
|
|
17
|
+
};
|
|
18
|
+
|
|
19
|
+
/**
|
|
20
|
+
* Fold one backend outcome into the breaker fields of a health entry.
|
|
21
|
+
* Pure: returns the new fields; caller merges into the entry.
|
|
22
|
+
*
|
|
23
|
+
* @param {object|undefined} prev previous entry (may be undefined)
|
|
24
|
+
* @param {boolean} ok did this call succeed?
|
|
25
|
+
* @param {number} nowMs
|
|
26
|
+
* @param {{threshold?: number, cooldownMs?: number}} cfg
|
|
27
|
+
* @returns {{ failCount: number, cooledUntil: number }}
|
|
28
|
+
*/
|
|
29
|
+
export function foldBreaker(prev, ok, nowMs, cfg = {}) {
|
|
30
|
+
const threshold = Math.max(1, Math.floor(Number(cfg.threshold) || BREAKER_DEFAULTS.threshold));
|
|
31
|
+
// cooldownMs === 0 explicitly disables the cooldown window (cooledUntil = now
|
|
32
|
+
// expires instantly). Only null/undefined falls back to the default — an
|
|
33
|
+
// explicit 0 must not silently become 60s (arch-review P2 note).
|
|
34
|
+
const cooldownMs = cfg.cooldownMs != null
|
|
35
|
+
? Math.max(0, Math.floor(Number(cfg.cooldownMs) || 0))
|
|
36
|
+
: BREAKER_DEFAULTS.cooldownMs;
|
|
37
|
+
if (ok) return { failCount: 0, cooledUntil: 0 };
|
|
38
|
+
const failCount = (Number.isFinite(prev?.failCount) ? prev.failCount : 0) + 1;
|
|
39
|
+
if (failCount >= threshold) {
|
|
40
|
+
return { failCount, cooledUntil: nowMs + cooldownMs };
|
|
41
|
+
}
|
|
42
|
+
// Keep an existing open window if the backend was already open (failure
|
|
43
|
+
// while open extends nothing, but must not silently close it).
|
|
44
|
+
return { failCount, cooledUntil: Number.isFinite(prev?.cooledUntil) ? prev.cooledUntil : 0 };
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
/**
|
|
48
|
+
* Is the backend currently in its cooldown window?
|
|
49
|
+
* @param {object|undefined} entry health entry (failCount/cooledUntil optional)
|
|
50
|
+
* @param {number} nowMs
|
|
51
|
+
*/
|
|
52
|
+
export function isCooled(entry, nowMs = Date.now()) {
|
|
53
|
+
const cooledUntil = entry?.cooledUntil;
|
|
54
|
+
return Number.isFinite(cooledUntil) && cooledUntil > nowMs;
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
/**
|
|
58
|
+
* Human telemetry annotation for a cooled backend, e.g. "⏸cooled 42s".
|
|
59
|
+
* Empty string when not cooled.
|
|
60
|
+
*/
|
|
61
|
+
export function cooledAnnotation(entry, nowMs = Date.now()) {
|
|
62
|
+
if (!isCooled(entry, nowMs)) return "";
|
|
63
|
+
const restS = Math.max(0, Math.ceil((entry.cooledUntil - nowMs) / 1000));
|
|
64
|
+
return `⏸cooled ${restS}s`;
|
|
65
|
+
}
|
package/lib/cache.js
ADDED
|
@@ -0,0 +1,151 @@
|
|
|
1
|
+
// cache.js — v2.7.0 disk cache for unified search results (zero dependencies).
|
|
2
|
+
//
|
|
3
|
+
// Cache key: sha256 over the canonical JSON of {query (already shaped),
|
|
4
|
+
// filters, maxResults} — exactly the dimensions that change a search outcome.
|
|
5
|
+
// Entries live as one JSON file per key under <dir>, written atomically
|
|
6
|
+
// (tmp file + rename) so a crash mid-write never leaves a corrupt cache.
|
|
7
|
+
// Eviction: when the entry count exceeds maxEntries, the oldest-by-mtime
|
|
8
|
+
// files are deleted first. Every fs error is swallowed by the caller-facing
|
|
9
|
+
// API (get/set never throw) — a broken cache must never break a search.
|
|
10
|
+
|
|
11
|
+
import { createHash } from "node:crypto";
|
|
12
|
+
import {
|
|
13
|
+
existsSync,
|
|
14
|
+
mkdirSync,
|
|
15
|
+
readdirSync,
|
|
16
|
+
readFileSync,
|
|
17
|
+
renameSync,
|
|
18
|
+
statSync,
|
|
19
|
+
unlinkSync,
|
|
20
|
+
writeFileSync,
|
|
21
|
+
} from "node:fs";
|
|
22
|
+
import path from "node:path";
|
|
23
|
+
|
|
24
|
+
/** Defaults (overridable per instance; settings feed these from index.js). */
|
|
25
|
+
export const CACHE_DEFAULTS = {
|
|
26
|
+
ttlMs: 900_000, // 900s
|
|
27
|
+
maxEntries: 200,
|
|
28
|
+
};
|
|
29
|
+
|
|
30
|
+
/**
|
|
31
|
+
* Compute the deterministic cache key for a search.
|
|
32
|
+
* Includes the enabledBackends selection set (sorted): switching the backend
|
|
33
|
+
* mix must never serve a stale result produced by a different combination
|
|
34
|
+
* (arch-review P1-1). Pure function; exported for tests and host reuse.
|
|
35
|
+
*/
|
|
36
|
+
export function cacheKeyFor({ query, filters, maxResults, backends }) {
|
|
37
|
+
const backendSet = Array.isArray(backends) ? [...backends].sort().join(",") : "";
|
|
38
|
+
const canonical = JSON.stringify({
|
|
39
|
+
q: typeof query === "string" ? query : "",
|
|
40
|
+
f: filters ?? {},
|
|
41
|
+
n: Number(maxResults) || 0,
|
|
42
|
+
b: backendSet,
|
|
43
|
+
});
|
|
44
|
+
return createHash("sha256").update(canonical).digest("hex").slice(0, 32);
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
/** Age of a record in whole seconds (bounded >= 0). */
|
|
48
|
+
function ageSeconds(record, nowMs) {
|
|
49
|
+
return Math.max(0, Math.floor((nowMs - (record?.time ?? 0)) / 1000));
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
/**
|
|
53
|
+
* Create a disk-backed result cache.
|
|
54
|
+
*
|
|
55
|
+
* @param {object} opts
|
|
56
|
+
* @param {string} opts.dir cache directory (created lazily)
|
|
57
|
+
* @param {number} [opts.ttlMs] entry TTL in ms (default 900_000)
|
|
58
|
+
* @param {number} [opts.maxEntries] entry cap (default 200)
|
|
59
|
+
* @returns {{ get, set, clear }} get/set/clear never throw
|
|
60
|
+
*/
|
|
61
|
+
export function createSearchCache({ dir, ttlMs = CACHE_DEFAULTS.ttlMs, maxEntries = CACHE_DEFAULTS.maxEntries }) {
|
|
62
|
+
// ttlMs may be a number or () => number (settings can change at runtime).
|
|
63
|
+
const ttlOf = typeof ttlMs === "function" ? ttlMs : () => Number(ttlMs) || 0;
|
|
64
|
+
const cap = Math.max(1, Math.floor(Number(maxEntries) || CACHE_DEFAULTS.maxEntries));
|
|
65
|
+
|
|
66
|
+
const fileFor = (key) => path.join(dir, `${key}.json`);
|
|
67
|
+
|
|
68
|
+
function ensureDir() {
|
|
69
|
+
mkdirSync(dir, { recursive: true });
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
/**
|
|
73
|
+
* Atomic write: tmp sibling + rename. Atomic on POSIX (rename(2) replaces
|
|
74
|
+
* atomically) and on Windows via libuv's MoveFileExW(MOVEFILE_REPLACE_
|
|
75
|
+
* EXISTING) — see README "权威依据与取舍" for the Windows caveat.
|
|
76
|
+
*/
|
|
77
|
+
function atomicWrite(file, data) {
|
|
78
|
+
const tmp = `${file}.${process.pid}.tmp`;
|
|
79
|
+
writeFileSync(tmp, data, "utf8");
|
|
80
|
+
renameSync(tmp, file);
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
/** List current entries as [{key, file, mtime}] sorted oldest first. */
|
|
84
|
+
function listEntries() {
|
|
85
|
+
if (!existsSync(dir)) return [];
|
|
86
|
+
const out = [];
|
|
87
|
+
for (const name of readdirSync(dir)) {
|
|
88
|
+
if (!name.endsWith(".json") || name.includes(".tmp")) continue;
|
|
89
|
+
const file = path.join(dir, name);
|
|
90
|
+
try {
|
|
91
|
+
out.push({ key: name.slice(0, -5), file, mtime: statSync(file).mtimeMs });
|
|
92
|
+
} catch {
|
|
93
|
+
// raced delete — skip
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
return out.sort((a, b) => a.mtime - b.mtime);
|
|
97
|
+
}
|
|
98
|
+
|
|
99
|
+
function evict() {
|
|
100
|
+
const entries = listEntries();
|
|
101
|
+
for (let i = 0; i < entries.length - cap; i++) {
|
|
102
|
+
try { unlinkSync(entries[i].file); } catch { /* raced */ }
|
|
103
|
+
}
|
|
104
|
+
}
|
|
105
|
+
|
|
106
|
+
return {
|
|
107
|
+
/**
|
|
108
|
+
* Look up a cached result.
|
|
109
|
+
* @returns {{ value: object, age: number } | null} age in seconds; null on miss/expiry
|
|
110
|
+
*/
|
|
111
|
+
get(key, nowMs = Date.now()) {
|
|
112
|
+
try {
|
|
113
|
+
const file = fileFor(key);
|
|
114
|
+
if (!existsSync(file)) return null;
|
|
115
|
+
const record = JSON.parse(readFileSync(file, "utf8"));
|
|
116
|
+
if (!record || typeof record !== "object" || !record.value) return null;
|
|
117
|
+
const age = ageSeconds(record, nowMs);
|
|
118
|
+
const ttl = Math.max(0, Number(ttlOf()) || 0);
|
|
119
|
+
if (age * 1000 > ttl) {
|
|
120
|
+
try { unlinkSync(file); } catch { /* raced */ }
|
|
121
|
+
return null;
|
|
122
|
+
}
|
|
123
|
+
return { value: record.value, age };
|
|
124
|
+
} catch {
|
|
125
|
+
return null;
|
|
126
|
+
}
|
|
127
|
+
},
|
|
128
|
+
|
|
129
|
+
/** Store a result (best-effort; fs errors are swallowed). */
|
|
130
|
+
set(key, value, nowMs = Date.now()) {
|
|
131
|
+
try {
|
|
132
|
+
ensureDir();
|
|
133
|
+
atomicWrite(fileFor(key), JSON.stringify({ key, time: nowMs, value }));
|
|
134
|
+
evict();
|
|
135
|
+
} catch {
|
|
136
|
+
// cache write failures are non-fatal by design
|
|
137
|
+
}
|
|
138
|
+
},
|
|
139
|
+
|
|
140
|
+
/** Drop every cached entry (best-effort). */
|
|
141
|
+
clear() {
|
|
142
|
+
try {
|
|
143
|
+
for (const e of listEntries()) {
|
|
144
|
+
try { unlinkSync(e.file); } catch { /* raced */ }
|
|
145
|
+
}
|
|
146
|
+
} catch {
|
|
147
|
+
// nothing to do
|
|
148
|
+
}
|
|
149
|
+
},
|
|
150
|
+
};
|
|
151
|
+
}
|
package/lib/client.js
CHANGED
|
@@ -543,10 +543,23 @@ window.__ModuleLoader__.load({
|
|
|
543
543
|
] });
|
|
544
544
|
}
|
|
545
545
|
|
|
546
|
-
/**
|
|
546
|
+
/**
|
|
547
|
+
* Plugin entry: register the card into the settings plugins page.
|
|
548
|
+
* Compatibility (compat-audit P0): the settingsScope client service was
|
|
549
|
+
* removed in DSH 0.1.7. It stays in `inject` so the fiber still
|
|
550
|
+
* activates when the shim supplies it; every access is feature-detected.
|
|
551
|
+
*/
|
|
547
552
|
const inject = ["slots", "locale", "connection", "remote", "settingsScope"];
|
|
548
553
|
function apply(ctx) {
|
|
549
554
|
ctx.effect(() => ctx.locale.register(NS, { zh, en }), "websearch: copy dictionaries");
|
|
555
|
+
const hasSettingsScope = ctx.get("settingsScope") !== undefined && typeof ctx.settingsScope?.bind === "function";
|
|
556
|
+
if (!hasSettingsScope) {
|
|
557
|
+
// DSH >= 0.1.7 without dsh-settings-scope-shim: the settings card
|
|
558
|
+
// cannot bind its scope. Stop wiring the card instead of hard-
|
|
559
|
+
// crashing the client fiber; the breadcrumb aids diagnosis.
|
|
560
|
+
console.warn("[dsh-websearch] settingsScope service unavailable (DSH >= 0.1.7); unified-search settings card disabled. Configure via server settings/cordis config or install dsh-settings-scope-shim.");
|
|
561
|
+
return;
|
|
562
|
+
}
|
|
550
563
|
const controller = new UnifiedSearchCardController(
|
|
551
564
|
ctx.settingsScope.bind({ namespace: PLUGIN_NS }),
|
|
552
565
|
ctx.get("connection"),
|
|
@@ -562,6 +575,34 @@ window.__ModuleLoader__.load({
|
|
|
562
575
|
inject: () => controller.inject(),
|
|
563
576
|
}, UnifiedSearchSection));
|
|
564
577
|
controller.refreshAllCredentials();
|
|
578
|
+
|
|
579
|
+
// Interop: devkit (and any third-party plugin) dispatches the window
|
|
580
|
+
// event "dsh-websearch:open-settings" to surface this plugin's
|
|
581
|
+
// settings. The host exposes no section-jump API, so: if devkit's
|
|
582
|
+
// toast is available, point the user at the path; otherwise log.
|
|
583
|
+
// (fe-ui C3 keeps devkit toast as the ecosystem standard notifier.)
|
|
584
|
+
const onOpenSettings = () => {
|
|
585
|
+
const hint = "统一搜索设置位于:设置 → 左侧栏「搜索 / Web Search」";
|
|
586
|
+
try {
|
|
587
|
+
if (window.__dshDevkit && typeof window.__dshDevkit.toast === "function") {
|
|
588
|
+
window.__dshDevkit.toast(hint, { kind: "info" });
|
|
589
|
+
return;
|
|
590
|
+
}
|
|
591
|
+
} catch (e) { /* fall through */ }
|
|
592
|
+
console.info("[dsh-websearch] " + hint);
|
|
593
|
+
};
|
|
594
|
+
// Probe contract (devkit 0.1.1): ready flag + ack event, so the
|
|
595
|
+
// palette can tell "listener present" from "gracefully absent".
|
|
596
|
+
try {
|
|
597
|
+
window.__dshWebsearchSettingsReady = true;
|
|
598
|
+
window.addEventListener("dsh-websearch:open-settings", (ev) => {
|
|
599
|
+
onOpenSettings(ev);
|
|
600
|
+
try {
|
|
601
|
+
window.dispatchEvent(new CustomEvent("dsh-websearch:open-settings:ack"));
|
|
602
|
+
} catch (e) { /* CustomEvent unavailable */ }
|
|
603
|
+
});
|
|
604
|
+
ctx.effect(() => () => window.removeEventListener("dsh-websearch:open-settings", onOpenSettings), "websearch: open-settings listener");
|
|
605
|
+
} catch (e) { /* non-window env */ }
|
|
565
606
|
}
|
|
566
607
|
|
|
567
608
|
exports.apply = apply;
|
package/lib/health.js
ADDED
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
// health.js — read-only health surface for the unified-search plugin.
|
|
2
|
+
//
|
|
3
|
+
// GET /api/unified-search/health reports which backends are enabled / keyless
|
|
4
|
+
// and the LAST per-backend telemetry captured during a real search. It never
|
|
5
|
+
// probes anything by itself (quota safety) and never echoes credential
|
|
6
|
+
// material: error strings are truncated to 40-120 chars (matching the
|
|
7
|
+
// provider's own telemetry truncation) and run through a defensive redaction
|
|
8
|
+
// pass for token-like fragments.
|
|
9
|
+
|
|
10
|
+
const MAX_ERROR_CHARS = 120;
|
|
11
|
+
|
|
12
|
+
/** Token-like fragments that must never appear in a health payload. */
|
|
13
|
+
const SECRET_PATTERNS = [
|
|
14
|
+
/sk-[A-Za-z0-9_-]{8,}/g,
|
|
15
|
+
/Bearer\s+[A-Za-z0-9._-]{8,}/gi,
|
|
16
|
+
/(api[_-]?key|token|secret|password)\s*[=:]\s*\S{6,}/gi,
|
|
17
|
+
];
|
|
18
|
+
|
|
19
|
+
/**
|
|
20
|
+
* Truncate + redact an error string for safe display.
|
|
21
|
+
* Returns "" for falsy input; length bounded to 40-120 chars.
|
|
22
|
+
*/
|
|
23
|
+
export function sanitizeError(err) {
|
|
24
|
+
if (err === null || err === undefined) return "";
|
|
25
|
+
let s = String(err instanceof Error ? err.message : err);
|
|
26
|
+
for (const re of SECRET_PATTERNS) s = s.replace(re, "[redacted]");
|
|
27
|
+
s = s.replace(/\s+/g, " ").trim();
|
|
28
|
+
if (s.length > MAX_ERROR_CHARS) s = s.slice(0, MAX_ERROR_CHARS);
|
|
29
|
+
return s;
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
/**
|
|
33
|
+
* Build the health report body (pure — no ctx, no I/O; unit-testable).
|
|
34
|
+
*
|
|
35
|
+
* @param {object} deps
|
|
36
|
+
* @param {Function} deps.resolveOptions () => resolved options snapshot
|
|
37
|
+
* @param {object} deps.backends keyed backend registry (BACKENDS)
|
|
38
|
+
* @param {Map|Function} deps.getBackendHealth telemetry map or () => map
|
|
39
|
+
* entries: { ok: boolean, at: ms, ms: duration, error?: string,
|
|
40
|
+
* lastOkAt?: ms|null }
|
|
41
|
+
* @returns {object} JSON-serializable health payload
|
|
42
|
+
*/
|
|
43
|
+
export function buildHealthReport({ resolveOptions, backends, getBackendHealth }) {
|
|
44
|
+
const opts = resolveOptions();
|
|
45
|
+
const healthMap = typeof getBackendHealth === "function" ? getBackendHealth() : getBackendHealth;
|
|
46
|
+
const tel = healthMap instanceof Map ? healthMap : new Map();
|
|
47
|
+
|
|
48
|
+
const enabledSet = new Set(
|
|
49
|
+
(opts.enabledBackends ?? []).filter((id) => backends[id] !== undefined),
|
|
50
|
+
);
|
|
51
|
+
|
|
52
|
+
const list = Object.keys(backends).map((id) => {
|
|
53
|
+
const beOpts = opts.backends?.[id] ?? {};
|
|
54
|
+
const enabled = enabledSet.has(id);
|
|
55
|
+
const keyless = beOpts.requiresKey !== true;
|
|
56
|
+
const t = tel.get(id);
|
|
57
|
+
return {
|
|
58
|
+
id,
|
|
59
|
+
enabled,
|
|
60
|
+
keyless,
|
|
61
|
+
// reachable: true/false from the latest search telemetry; null = never probed.
|
|
62
|
+
reachable: t ? t.ok === true : null,
|
|
63
|
+
lastOk: t && typeof t.lastOkAt === "number" ? t.lastOkAt : null,
|
|
64
|
+
lastError: t && t.ok !== true ? sanitizeError(t.error) || "unknown error" : null,
|
|
65
|
+
};
|
|
66
|
+
});
|
|
67
|
+
|
|
68
|
+
const enabledList = list.filter((b) => b.enabled);
|
|
69
|
+
return {
|
|
70
|
+
ok: true,
|
|
71
|
+
enabled: enabledList.length,
|
|
72
|
+
keyless: enabledList.filter((b) => b.keyless).length,
|
|
73
|
+
backends: list,
|
|
74
|
+
updatedAt: Date.now(),
|
|
75
|
+
};
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
/**
|
|
79
|
+
* Register the read-only health route on the (late-mounted) webServer.
|
|
80
|
+
* GET only; never probes backends; response is JSON with no credentials.
|
|
81
|
+
*/
|
|
82
|
+
export function registerHealthRoute(ws, deps) {
|
|
83
|
+
if (!ws || typeof ws.register !== "function") return;
|
|
84
|
+
ws.register({
|
|
85
|
+
kind: "exact",
|
|
86
|
+
path: "/api/unified-search/health",
|
|
87
|
+
handler: (req, res) => {
|
|
88
|
+
const send = (code, body) => {
|
|
89
|
+
res.writeHead(code, { "content-type": "application/json" });
|
|
90
|
+
res.end(JSON.stringify(body));
|
|
91
|
+
};
|
|
92
|
+
if (req.method !== "GET" && req.method !== "HEAD") {
|
|
93
|
+
send(405, { ok: false, error: "method not allowed (GET only)" });
|
|
94
|
+
return;
|
|
95
|
+
}
|
|
96
|
+
try {
|
|
97
|
+
send(200, buildHealthReport(deps));
|
|
98
|
+
} catch (err) {
|
|
99
|
+
send(500, { ok: false, error: sanitizeError(err) || "health report failed" });
|
|
100
|
+
}
|
|
101
|
+
},
|
|
102
|
+
});
|
|
103
|
+
}
|
package/lib/history.js
ADDED
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
// history.js — v2.7.0 append-only search history (zero deps).
|
|
2
|
+
//
|
|
3
|
+
// Ring buffer of the most recent maxEntries searches, persisted as one JSON
|
|
4
|
+
// file under the plugin's cache dir. Writes are atomic (tmp + rename) and
|
|
5
|
+
// append-only: existing entries are never rewritten, only prepended; the
|
|
6
|
+
// oldest entry beyond the cap is dropped. Exposure is strictly read-only via
|
|
7
|
+
// GET /api/websearch/history; POST /api/websearch/history/clear empties it.
|
|
8
|
+
|
|
9
|
+
import { existsSync, mkdirSync, readFileSync, renameSync, unlinkSync, writeFileSync } from "node:fs";
|
|
10
|
+
import path from "node:path";
|
|
11
|
+
|
|
12
|
+
/** Ring-buffer cap (spec: 50). */
|
|
13
|
+
export const HISTORY_MAX_ENTRIES = 50;
|
|
14
|
+
|
|
15
|
+
/** Sanitize a history entry to the exact public shape (no credentials/URLs). */
|
|
16
|
+
function toPublicEntry(entry) {
|
|
17
|
+
return {
|
|
18
|
+
query: String(entry?.query ?? "").slice(0, 500),
|
|
19
|
+
time: Number.isFinite(entry?.time) ? entry.time : Date.now(),
|
|
20
|
+
resultCount: Math.max(0, Math.floor(Number(entry?.resultCount) || 0)),
|
|
21
|
+
backendsOk: Math.max(0, Math.floor(Number(entry?.backendsOk) || 0)),
|
|
22
|
+
backendsTotal: Math.max(0, Math.floor(Number(entry?.backendsTotal) || 0)),
|
|
23
|
+
};
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
/**
|
|
27
|
+
* Create an append-only history store.
|
|
28
|
+
*
|
|
29
|
+
* @param {object} opts
|
|
30
|
+
* @param {string} opts.dir directory holding history.json (created lazily)
|
|
31
|
+
* @param {number} [opts.maxEntries] ring cap (default 50)
|
|
32
|
+
* @returns {{ record, list, clear }}
|
|
33
|
+
*/
|
|
34
|
+
export function createHistoryStore({ dir, maxEntries = HISTORY_MAX_ENTRIES }) {
|
|
35
|
+
const cap = Math.max(1, Math.floor(Number(maxEntries) || HISTORY_MAX_ENTRIES));
|
|
36
|
+
const file = () => path.join(dir, "history.json");
|
|
37
|
+
// Serialize every write through a promise chain: record() is called from
|
|
38
|
+
// concurrent search completions; without the chain, two interleaved
|
|
39
|
+
// read-modify-write cycles lose updates (arch-review P2-2). The body stays
|
|
40
|
+
// synchronous (atomic tmp+rename), so the chain is the only lock needed
|
|
41
|
+
// within this process; cross-process writers are out of scope and each
|
|
42
|
+
// write is still atomic for readers.
|
|
43
|
+
let writeChain = Promise.resolve();
|
|
44
|
+
|
|
45
|
+
function readAll() {
|
|
46
|
+
try {
|
|
47
|
+
if (!existsSync(file())) return [];
|
|
48
|
+
const parsed = JSON.parse(readFileSync(file(), "utf8"));
|
|
49
|
+
return Array.isArray(parsed) ? parsed : [];
|
|
50
|
+
} catch {
|
|
51
|
+
return [];
|
|
52
|
+
}
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
function atomicWriteAll(list) {
|
|
56
|
+
mkdirSync(dir, { recursive: true });
|
|
57
|
+
const f = file();
|
|
58
|
+
const tmp = `${f}.${process.pid}.tmp`;
|
|
59
|
+
writeFileSync(tmp, JSON.stringify(list, null, 2), "utf8");
|
|
60
|
+
renameSync(tmp, f);
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
return {
|
|
64
|
+
/** Append one search (best-effort; fs errors are swallowed). Newest first. */
|
|
65
|
+
record(entry) {
|
|
66
|
+
writeChain = writeChain.then(() => {
|
|
67
|
+
try {
|
|
68
|
+
const list = readAll();
|
|
69
|
+
list.unshift(toPublicEntry(entry));
|
|
70
|
+
if (list.length > cap) list.length = cap;
|
|
71
|
+
atomicWriteAll(list);
|
|
72
|
+
} catch {
|
|
73
|
+
// history write failures are non-fatal by design
|
|
74
|
+
}
|
|
75
|
+
});
|
|
76
|
+
return writeChain;
|
|
77
|
+
},
|
|
78
|
+
|
|
79
|
+
/** Read the current history, newest first (defensive copy). */
|
|
80
|
+
list() {
|
|
81
|
+
return readAll().slice(0, cap).map(toPublicEntry);
|
|
82
|
+
},
|
|
83
|
+
|
|
84
|
+
/** Empty the history (removes the file; best-effort). */
|
|
85
|
+
clear() {
|
|
86
|
+
try {
|
|
87
|
+
if (existsSync(file())) unlinkSync(file());
|
|
88
|
+
} catch {
|
|
89
|
+
// nothing to do
|
|
90
|
+
}
|
|
91
|
+
},
|
|
92
|
+
};
|
|
93
|
+
}
|
|
94
|
+
|
|
95
|
+
/**
|
|
96
|
+
* Minimal trust fence for plugin-registered routes (the host /api RPC fence
|
|
97
|
+
* does not cover webServer.register paths): require a loopback Host (blocks
|
|
98
|
+
* DNS rebinding) and a same-site browser context (sec-fetch-site same-origin
|
|
99
|
+
* / none, or an Origin whose host matches the request Host — blocks
|
|
100
|
+
* cross-site CSRF including text/plain preflight-less POSTs).
|
|
101
|
+
* Returns null when trusted, otherwise a short rejection reason.
|
|
102
|
+
*/
|
|
103
|
+
export function isTrustedApiRequest(req) {
|
|
104
|
+
const host = String(req?.headers?.host ?? "");
|
|
105
|
+
const hostOk = /^(localhost|127\.\d{1,3}\.\d{1,3}\.\d{1,3}|\[::1\])(:\d+)?$/.test(host);
|
|
106
|
+
if (!hostOk) return "untrusted host";
|
|
107
|
+
const site = req?.headers?.["sec-fetch-site"];
|
|
108
|
+
if (site === "same-origin" || site === "none") return null;
|
|
109
|
+
const origin = req?.headers?.origin;
|
|
110
|
+
if (!origin) return site !== undefined ? "cross-site request" : null;
|
|
111
|
+
try {
|
|
112
|
+
return new URL(origin).host === host ? null : "cross-origin request";
|
|
113
|
+
} catch {
|
|
114
|
+
return "invalid origin";
|
|
115
|
+
}
|
|
116
|
+
}
|
|
117
|
+
|
|
118
|
+
/**
|
|
119
|
+
* Register the read-only history routes on the (late-mounted) webServer.
|
|
120
|
+
* GET /api/websearch/history -> { ok, entries }
|
|
121
|
+
* POST /api/websearch/history/clear -> { ok, cleared: true }
|
|
122
|
+
* Both routes sit behind isTrustedApiRequest (loopback Host + same-site).
|
|
123
|
+
*/
|
|
124
|
+
export function registerHistoryRoutes(ws, { getStore }) {
|
|
125
|
+
if (!ws || typeof ws.register !== "function") return;
|
|
126
|
+
const send = (res, code, body) => {
|
|
127
|
+
res.writeHead(code, { "content-type": "application/json" });
|
|
128
|
+
res.end(JSON.stringify(body));
|
|
129
|
+
};
|
|
130
|
+
const guarded = (handler) => (req, res) => {
|
|
131
|
+
const fence = isTrustedApiRequest(req);
|
|
132
|
+
if (fence) {
|
|
133
|
+
send(res, 403, { ok: false, error: `untrusted request (${fence})` });
|
|
134
|
+
return;
|
|
135
|
+
}
|
|
136
|
+
handler(req, res);
|
|
137
|
+
};
|
|
138
|
+
ws.register({
|
|
139
|
+
kind: "exact",
|
|
140
|
+
path: "/api/websearch/history",
|
|
141
|
+
handler: guarded((req, res) => {
|
|
142
|
+
if (req.method !== "GET" && req.method !== "HEAD") {
|
|
143
|
+
send(res, 405, { ok: false, error: "method not allowed (GET only)" });
|
|
144
|
+
return;
|
|
145
|
+
}
|
|
146
|
+
const store = getStore();
|
|
147
|
+
send(res, 200, { ok: true, entries: store ? store.list() : [] });
|
|
148
|
+
}),
|
|
149
|
+
});
|
|
150
|
+
ws.register({
|
|
151
|
+
kind: "exact",
|
|
152
|
+
path: "/api/websearch/history/clear",
|
|
153
|
+
handler: guarded((req, res) => {
|
|
154
|
+
if (req.method !== "POST") {
|
|
155
|
+
send(res, 405, { ok: false, error: "method not allowed (POST only)" });
|
|
156
|
+
return;
|
|
157
|
+
}
|
|
158
|
+
const store = getStore();
|
|
159
|
+
store?.clear();
|
|
160
|
+
send(res, 200, { ok: true, cleared: true });
|
|
161
|
+
}),
|
|
162
|
+
});
|
|
163
|
+
}
|
package/lib/index.js
CHANGED
|
@@ -10,10 +10,12 @@
|
|
|
10
10
|
*/
|
|
11
11
|
import z from "@deepseek-ai/schemastery";
|
|
12
12
|
import { credentialRef } from "@deepseek-ai/dsh-credentials";
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
13
|
+
// NOTE (2026-09-12, dsh 0.1.5 compat): installSettingsSection +
|
|
14
|
+
// settingsNamespace were removed from @deepseek-ai/dsh-settings in the 0.1.5
|
|
15
|
+
// line (only SettingsProvider/SettingsConflictError/redactSecrets remain).
|
|
16
|
+
// Settings sections are now installed through the `settings` service, as the
|
|
17
|
+
// stock plugins do (see dsh-permission-presets). NAMESPACE was always the
|
|
18
|
+
// plain string "unified-search". No import needed anymore.
|
|
17
19
|
import { launchEnvironmentOf } from "@deepseek-ai/dsh-launch-environment";
|
|
18
20
|
import { WebError } from "@deepseek-ai/dsh-web";
|
|
19
21
|
|
|
@@ -31,7 +33,13 @@ import { tavilyBackend } from "./backends/tavily.js";
|
|
|
31
33
|
import { serperBackend } from "./backends/serper.js";
|
|
32
34
|
import { searxngBackend } from "./backends/searxng.js";
|
|
33
35
|
import { mojeekBackend } from "./backends/mojeek.js";
|
|
34
|
-
import { createUnifiedSearchProvider } from "./provider.js";
|
|
36
|
+
import { createUnifiedSearchProvider, getLatestBackendHealth } from "./provider.js";
|
|
37
|
+
import { registerHealthRoute } from "./health.js";
|
|
38
|
+
import { createSearchCache, CACHE_DEFAULTS } from "./cache.js";
|
|
39
|
+
import { createHistoryStore, registerHistoryRoutes, HISTORY_MAX_ENTRIES } from "./history.js";
|
|
40
|
+
import { BREAKER_DEFAULTS } from "./breaker.js";
|
|
41
|
+
import os from "node:os";
|
|
42
|
+
import path from "node:path";
|
|
35
43
|
|
|
36
44
|
/** Cordis plugin name used by loader diagnostics. */
|
|
37
45
|
export const name = "unified-search";
|
|
@@ -39,7 +47,7 @@ export const name = "unified-search";
|
|
|
39
47
|
export const inject = ["web"];
|
|
40
48
|
|
|
41
49
|
/** Settings namespace key. */
|
|
42
|
-
const NAMESPACE =
|
|
50
|
+
const NAMESPACE = "unified-search";
|
|
43
51
|
|
|
44
52
|
/** Full backend registry. Order = default fan-out priority. */
|
|
45
53
|
const BACKENDS = {
|
|
@@ -144,6 +152,23 @@ export const Config = z.object({
|
|
|
144
152
|
resultTelemetry: z.boolean().default(true)
|
|
145
153
|
.description("在搜索结果尾部附一行后端健康遥测(状态+耗时),模型可据此自诊断"),
|
|
146
154
|
|
|
155
|
+
// v2.7.0 disk cache / circuit breaker / search history
|
|
156
|
+
// W1 (fe-ui review): labels speak user language with a group prefix
|
|
157
|
+
// (【缓存】/【熔断】/【历史】) instead of camelCase keys; units are
|
|
158
|
+
// spelled out (秒 / 毫秒) and grouping follows the field order below.
|
|
159
|
+
cacheEnabled: z.boolean().default(true)
|
|
160
|
+
.description("启用搜索结果缓存【缓存】:相同查询+过滤+条数在有效期内直接命中,不再请求后端"),
|
|
161
|
+
cacheTtl: z.number().step(30).min(30).max(86400).default(900)
|
|
162
|
+
.description("缓存有效期【缓存】(单位:秒):默认 900 秒;不想用缓存请直接关闭上一项"),
|
|
163
|
+
breakerEnabled: z.boolean().default(true)
|
|
164
|
+
.description("后端熔断器【熔断】:连续失败达阈值后暂停该后端一段时间;全部暂停时自动放行,不会让搜索更差"),
|
|
165
|
+
breakerThreshold: z.number().step(1).min(1).max(20).default(BREAKER_DEFAULTS.threshold)
|
|
166
|
+
.description("熔断阈值【熔断】:连续失败多少次后触发(默认 3 次)"),
|
|
167
|
+
breakerCooldownMs: z.number().step(1000).min(0).max(3600000).default(BREAKER_DEFAULTS.cooldownMs)
|
|
168
|
+
.description("熔断冷却时长【熔断】(单位:毫秒):默认 60000;设为 0 表示关闭冷却(熔断仅计数)"),
|
|
169
|
+
historyEnabled: z.boolean().default(true)
|
|
170
|
+
.description("记录搜索历史【历史】:只追加,最多 50 条;可在设置 → 搜索 中查看与清空"),
|
|
171
|
+
|
|
147
172
|
// Enable toggles (one per backend)
|
|
148
173
|
enableExa: z.boolean().default(true).description(DEFAULT_LABELS.exa),
|
|
149
174
|
enableParallel: z.boolean().default(true).description(DEFAULT_LABELS.parallel),
|
|
@@ -190,6 +215,30 @@ export const Config = z.object({
|
|
|
190
215
|
.description("显式启用后端列表(优先于上方开关;留空则用开关)"),
|
|
191
216
|
});
|
|
192
217
|
|
|
218
|
+
/**
|
|
219
|
+
* Per-field user-language labels (fe-ui W1 remainder): the settings renderer
|
|
220
|
+
* falls back to camelCase keys when no label is present, so attach one to
|
|
221
|
+
* each v2.7 descriptor (meta.label, plus a plain .label for renderers that
|
|
222
|
+
* read the top level). Units live in the label tail (秒 / 毫秒). Whether the
|
|
223
|
+
* host actually renders them is visual-check V5; the schema layer now
|
|
224
|
+
* provides the field regardless.
|
|
225
|
+
*/
|
|
226
|
+
const FIELD_LABELS = {
|
|
227
|
+
cacheEnabled: "启用搜索结果缓存",
|
|
228
|
+
cacheTtl: "缓存有效期(秒)",
|
|
229
|
+
breakerEnabled: "后端熔断器",
|
|
230
|
+
breakerThreshold: "熔断阈值(连续失败次数)",
|
|
231
|
+
breakerCooldownMs: "熔断冷却时长(毫秒)",
|
|
232
|
+
historyEnabled: "记录搜索历史",
|
|
233
|
+
};
|
|
234
|
+
for (const [key, label] of Object.entries(FIELD_LABELS)) {
|
|
235
|
+
const sch = Config.dict?.[key];
|
|
236
|
+
if (sch) {
|
|
237
|
+
sch.meta = { ...sch.meta, label };
|
|
238
|
+
sch.label = label;
|
|
239
|
+
}
|
|
240
|
+
}
|
|
241
|
+
|
|
193
242
|
/**
|
|
194
243
|
* Project one resolved settings section into the options snapshot the provider
|
|
195
244
|
* serves its NEXT search with. Key resolution: credentialRef -> credentials
|
|
@@ -268,6 +317,15 @@ export function resolveOptions(ctx, config) {
|
|
|
268
317
|
dedupStrategy: config?.dedupStrategy === "url+title" ? "url+title" : "url",
|
|
269
318
|
rerank: config?.rerank === true,
|
|
270
319
|
resultTelemetry: config?.resultTelemetry !== false,
|
|
320
|
+
// v2.7.0 cache / breaker / history
|
|
321
|
+
cacheEnabled: config?.cacheEnabled === true,
|
|
322
|
+
cacheTtlS: Math.max(30, Math.floor(Number(config?.cacheTtl) || CACHE_DEFAULTS.ttlMs / 1000)),
|
|
323
|
+
breakerEnabled: config?.breakerEnabled !== false,
|
|
324
|
+
breakerThreshold: Math.max(1, Math.floor(Number(config?.breakerThreshold) || BREAKER_DEFAULTS.threshold)),
|
|
325
|
+
breakerCooldownMs: config?.breakerCooldownMs != null
|
|
326
|
+
? Math.max(0, Math.floor(Number(config?.breakerCooldownMs) || 0)) // 0 = disabled
|
|
327
|
+
: BREAKER_DEFAULTS.cooldownMs,
|
|
328
|
+
historyEnabled: config?.historyEnabled !== false,
|
|
271
329
|
filters: {
|
|
272
330
|
recency: typeof config?.recency === "string" && config.recency !== "any" ? config.recency : undefined,
|
|
273
331
|
lang: language.length > 0 ? language : undefined,
|
|
@@ -277,6 +335,19 @@ export function resolveOptions(ctx, config) {
|
|
|
277
335
|
};
|
|
278
336
|
}
|
|
279
337
|
|
|
338
|
+
/**
|
|
339
|
+
* Resolve the plugin's persistent store directory (cache + history).
|
|
340
|
+
* Pure and exported for tests: precedence is DSH_HOME > <home>/.dsh, where
|
|
341
|
+
* home falls back to os.homedir() — NOT process.env.HOME, which is usually
|
|
342
|
+
* unset on Windows and would silently disable cache/history (compat-audit
|
|
343
|
+
* P2-7).
|
|
344
|
+
* @param {{DSH_HOME?: string, HOME?: string}} env
|
|
345
|
+
* @param {string} [fallbackHome] injected os.homedir() for tests
|
|
346
|
+
*/
|
|
347
|
+
export function resolveStoreDir(env = process.env, fallbackHome = os.homedir()) {
|
|
348
|
+
return path.join(env.DSH_HOME || path.join(fallbackHome, ".dsh"), "cache", "websearch");
|
|
349
|
+
}
|
|
350
|
+
|
|
280
351
|
/**
|
|
281
352
|
* Register the unified search provider with ctx.web. Installs a settings
|
|
282
353
|
* section so users can configure enabled backends / numResults / base URLs /
|
|
@@ -285,18 +356,73 @@ export function resolveOptions(ctx, config) {
|
|
|
285
356
|
export function apply(ctx, config) {
|
|
286
357
|
let current = () => config;
|
|
287
358
|
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
359
|
+
ctx.inject(["settings"], (settingsCtx) => {
|
|
360
|
+
const settings = settingsCtx.settings;
|
|
361
|
+
if (typeof settings?.installSection === "function") {
|
|
362
|
+
settings.installSection(ctx, NAMESPACE, Config, config, {
|
|
363
|
+
setSource: (source) => {
|
|
364
|
+
current = source;
|
|
365
|
+
},
|
|
366
|
+
onChange: () => {},
|
|
367
|
+
});
|
|
368
|
+
return;
|
|
369
|
+
}
|
|
370
|
+
// DSH 0.1.7+: the settings service is SettingsForms and has no
|
|
371
|
+
// installSection. The Config schema still resolves through cordis config
|
|
372
|
+
// (this file reads `current()`), so search keeps working; surface the gap
|
|
373
|
+
// once instead of silently swallowing it (compat-audit P1).
|
|
374
|
+
console.warn(
|
|
375
|
+
"[dsh-websearch] settings service has no installSection (DSH >= 0.1.7 SettingsForms line); " +
|
|
376
|
+
"the Settings panel section is not installed. Configure the unified-search namespace via cordis config " +
|
|
377
|
+
"or the client settings card; migration to the SettingsForms API is tracked for a future release.",
|
|
378
|
+
);
|
|
379
|
+
});
|
|
380
|
+
|
|
381
|
+
// v2.7.0 stores: disk cache + append-only search history, both under
|
|
382
|
+
// DSH_HOME/cache/websearch (created lazily; every fs error is swallowed so
|
|
383
|
+
// a broken store never breaks a search).
|
|
384
|
+
// DSH_HOME wins when set; otherwise the OS home directory. os.homedir()
|
|
385
|
+
// (not process.env.HOME) is the cross-platform fallback — HOME is usually
|
|
386
|
+
// unset on Windows, which previously made storeDir relative to the current
|
|
387
|
+
// working directory and silently disabled cache/history (compat-audit P2-7).
|
|
388
|
+
const storeDir = resolveStoreDir();
|
|
389
|
+
const searchCache = createSearchCache({
|
|
390
|
+
dir: path.join(storeDir, "results"),
|
|
391
|
+
ttlMs: () => resolveOptions(ctx, current()).cacheTtlS * 1000,
|
|
392
|
+
maxEntries: CACHE_DEFAULTS.maxEntries,
|
|
293
393
|
});
|
|
394
|
+
const historyStore = createHistoryStore({ dir: storeDir, maxEntries: HISTORY_MAX_ENTRIES });
|
|
294
395
|
|
|
295
396
|
const provider = createUnifiedSearchProvider({
|
|
296
397
|
ctx,
|
|
297
398
|
resolveOptions: () => resolveOptions(ctx, current()),
|
|
298
399
|
backends: BACKENDS,
|
|
400
|
+
cache: {
|
|
401
|
+
get: (key) => searchCache.get(key),
|
|
402
|
+
set: (key, value) => searchCache.set(key, value),
|
|
403
|
+
},
|
|
404
|
+
history: {
|
|
405
|
+
record: (entry) => {
|
|
406
|
+
if (resolveOptions(ctx, current()).historyEnabled !== true) return;
|
|
407
|
+
historyStore.record(entry);
|
|
408
|
+
},
|
|
409
|
+
},
|
|
299
410
|
});
|
|
300
411
|
|
|
301
412
|
ctx.web.registerSearchProvider(provider);
|
|
413
|
+
|
|
414
|
+
// Read-only health route: webServer is an optional, late-mounted host
|
|
415
|
+
// service (same pattern as dsh-chat-import's panel routes). Headless / CI
|
|
416
|
+
// profiles never mount it and the callback simply never runs. The route
|
|
417
|
+
// only reads the settings snapshot + last per-search telemetry — it never
|
|
418
|
+
// probes backends and never touches credential values.
|
|
419
|
+
ctx.inject(["webServer"], (webCtx) => {
|
|
420
|
+
registerHealthRoute(webCtx.webServer, {
|
|
421
|
+
resolveOptions: () => resolveOptions(ctx, current()),
|
|
422
|
+
backends: BACKENDS,
|
|
423
|
+
getBackendHealth: getLatestBackendHealth,
|
|
424
|
+
});
|
|
425
|
+
// v2.7.0 search history: read-only GET + explicit clear POST.
|
|
426
|
+
registerHistoryRoutes(webCtx.webServer, { getStore: () => historyStore });
|
|
427
|
+
});
|
|
302
428
|
}
|
package/lib/prompting.js
ADDED
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* dsh-websearch — 系统性搜索提示词(纯函数,零依赖)。
|
|
3
|
+
*
|
|
4
|
+
* 依据权威资料(2024-2026 官方文档与一手论文)实现三项必做:
|
|
5
|
+
* 1. 查询整形(query shaping):后端扇出前对口语化长查询做确定性归一,
|
|
6
|
+
* 并导出给宿主/代理层使用的查询构造指引。
|
|
7
|
+
* 依据:Tavily Best Practices(agent 风格 ≤1500 字符查询、子查询分解)
|
|
8
|
+
* https://docs.tavily.com/documentation/best-practices/best-practices-search
|
|
9
|
+
* Exa Search Best Practices(语义丰富、描述性查询)
|
|
10
|
+
* https://exa.ai/docs/search-best-practices
|
|
11
|
+
* 2. 结果呈现头(result-presentation header):模型看到的结果块以「来源
|
|
12
|
+
* 甄别 + 引用要求」开头,要求每条事实注明来源链接、优先官方/一手来源、
|
|
13
|
+
* 时效敏感主题优先较新结果。
|
|
14
|
+
* 依据:Anthropic Web Search Tool(结果块置于问题之前、引用来源)
|
|
15
|
+
* https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool
|
|
16
|
+
* OpenAI Prompt Engineering(分解 + 持久性指令)
|
|
17
|
+
* https://platform.openai.com/docs/guides/prompt-engineering
|
|
18
|
+
* 3. 多面查询分解指引(导出常量,供宿主/代理层按需使用;不对本插件内
|
|
19
|
+
* 的单次请求盲目多查询——盲目多查询可能劣化,见 arXiv:2404.01037)。
|
|
20
|
+
* 依据:RAG-Fusion / 多查询变体 https://github.com/Raudaschl/rag-fusion
|
|
21
|
+
* Step-Back Prompting https://arxiv.org/abs/2310.06117
|
|
22
|
+
* HyDE https://arxiv.org/abs/2212.10496
|
|
23
|
+
* @module dsh-websearch/prompting
|
|
24
|
+
*/
|
|
25
|
+
|
|
26
|
+
/** 查询硬上限(Tavily:agent 查询应简短聚焦;>1500 字符视为跑题长文)。 */
|
|
27
|
+
export const QUERY_MAX_CHARS = 1500;
|
|
28
|
+
|
|
29
|
+
/** 去口语化前缀(中英常见寒暄/请求词,确定性剥离,不改写语义)。 */
|
|
30
|
+
const FILLER_PREFIX_RE =
|
|
31
|
+
/^(?:please|pls|kindly|could you(?: please)?|can you(?: please)?|help me(?: to)?|search for|search|look up|google|find me|find|i(?:'d| would)? like to know|tell me|what is|who is|when is|where is|"|「|『)\s*/i;
|
|
32
|
+
const FILLER_PREFIX_RE_ZH =
|
|
33
|
+
/^(?:请|麻烦|帮我?|帮忙|帮我查一?下|查一?下|查询|搜索一下?|搜一下?|查一下?|帮我找|找一下?|我想知道|请问)/;
|
|
34
|
+
|
|
35
|
+
/**
|
|
36
|
+
* 确定性查询整形:剥离口语前缀(循环多次以处理叠加寒暄)、压缩空白、
|
|
37
|
+
* 截断到 QUERY_MAX_CHARS。绝不改写实体/版本号/日期。
|
|
38
|
+
* @param {string} query
|
|
39
|
+
* @returns {string}
|
|
40
|
+
*/
|
|
41
|
+
export function shapeQuery(query) {
|
|
42
|
+
if (typeof query !== "string") return "";
|
|
43
|
+
let q = query.replace(/\s+/g, " ").trim();
|
|
44
|
+
// 中文寒暄与英文寒暄交替剥离,直到稳定。
|
|
45
|
+
for (let i = 0; i < 6; i++) {
|
|
46
|
+
const before = q;
|
|
47
|
+
q = q.replace(FILLER_PREFIX_RE, "").trim();
|
|
48
|
+
q = q.replace(FILLER_PREFIX_RE_ZH, "").trim();
|
|
49
|
+
if (q === before) break;
|
|
50
|
+
}
|
|
51
|
+
if (q.length > QUERY_MAX_CHARS) q = q.slice(0, QUERY_MAX_CHARS).trim();
|
|
52
|
+
return q;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
/**
|
|
56
|
+
* 结果呈现头(英文)。置于模型可见结果块最前(Anthropic:结果先于问题
|
|
57
|
+
* 显著提升引用召回)。要求:逐条标注 [n]、优先官方/一手来源、时效敏感
|
|
58
|
+
* 主题优先较新结果、每条事实给出来源链接、与问题无关的条目忽略。
|
|
59
|
+
*/
|
|
60
|
+
export const RESULT_HEADER_EN =
|
|
61
|
+
"Web search results (merged from multiple engines, deduplicated by URL). " +
|
|
62
|
+
"Each entry: [n] title — URL. Prefer official / primary sources; prefer newer " +
|
|
63
|
+
"results for fast-moving topics; ignore entries whose snippet does not actually " +
|
|
64
|
+
"answer the question. Cite the URL for every fact you take from a result.";
|
|
65
|
+
|
|
66
|
+
/** 结果呈现头(中文)。 */
|
|
67
|
+
export const RESULT_HEADER_ZH =
|
|
68
|
+
"网页搜索结果(多引擎合并、已按 URL 去重)。每条格式:[n] 标题 — 链接。" +
|
|
69
|
+
"优先采用官方/一手来源;时效敏感主题优先较新结果;摘要与问题无关的条目直接忽略。" +
|
|
70
|
+
"引用结果中的任何事实时必须注明来源链接。";
|
|
71
|
+
|
|
72
|
+
/**
|
|
73
|
+
* 查询构造指引(导出给宿主/代理层注入 agent 系统提示用;本插件不发送)。
|
|
74
|
+
* EN 面向 agent,ZH 为同义说明。要点:一条聚焦查询、保留专名/版本/日期、
|
|
75
|
+
* 多面请求按维度拆成多条子查询、首轮结果不佳时先写 2-3 句假想答案再以其
|
|
76
|
+
* 关键短语重搜(HyDE)、持续检索直到每个维度都有答案。
|
|
77
|
+
*/
|
|
78
|
+
export const QUERY_GUIDANCE_EN =
|
|
79
|
+
"When searching the web: issue ONE concise, agent-style query (keep named " +
|
|
80
|
+
"entities, version numbers and dates; drop conversational filler). If the " +
|
|
81
|
+
"request has 2+ independent facets, issue separate focused queries per facet " +
|
|
82
|
+
"and merge findings. If the first round returns poor results, write a 2-3 " +
|
|
83
|
+
"sentence hypothetical ideal answer and re-query with its key phrases. " +
|
|
84
|
+
"Keep searching until every facet is answered or demonstrably unavailable.";
|
|
85
|
+
|
|
86
|
+
/** 查询构造指引(中文版)。 */
|
|
87
|
+
export const QUERY_GUIDANCE_ZH =
|
|
88
|
+
"联网检索时:一次发出一条简洁的 agent 风格查询(保留专名/版本号/日期,去掉口语寒暄)。" +
|
|
89
|
+
"若请求含 2 个以上独立维度,按维度分别发起聚焦查询并汇总结论。" +
|
|
90
|
+
"首轮结果不佳时,先写 2-3 句假想的理想答案,再用其中的关键短语重新检索。" +
|
|
91
|
+
"持续检索直到每个维度都有答案或确认不可得。";
|
|
92
|
+
|
|
93
|
+
/**
|
|
94
|
+
* 按语言取结果呈现头。
|
|
95
|
+
* @param {"zh"|"en"|string} [lang]
|
|
96
|
+
*/
|
|
97
|
+
export function resultHeader(lang) {
|
|
98
|
+
return String(lang || "").toLowerCase().startsWith("zh") ? RESULT_HEADER_ZH : RESULT_HEADER_EN;
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
/**
|
|
102
|
+
* 遥测行自标注(零成本修复):模型可能把后端失败遥测误读为结果来源;
|
|
103
|
+
* 显式声明「仅诊断,不是来源」。随结果块一并呈现。
|
|
104
|
+
*/
|
|
105
|
+
export const TELEMETRY_TAG = " (diagnostic only, not a source)";
|
package/lib/provider.js
CHANGED
|
@@ -22,9 +22,44 @@ import {
|
|
|
22
22
|
} from "./util/log.js";
|
|
23
23
|
import { dedupeSources, rerankSources } from "./util/rank.js";
|
|
24
24
|
import { normalizeFilters } from "./util/filters.js";
|
|
25
|
+
import { shapeQuery, resultHeader, TELEMETRY_TAG } from "./prompting.js";
|
|
26
|
+
import { cacheKeyFor } from "./cache.js";
|
|
27
|
+
import { foldBreaker, isCooled, cooledAnnotation } from "./breaker.js";
|
|
25
28
|
|
|
26
29
|
const PROVIDER_ID = "unified";
|
|
27
30
|
|
|
31
|
+
// Module-level per-backend health telemetry (id -> entry), updated on every
|
|
32
|
+
// backend completion during a real search. Purely passive: nothing writes here
|
|
33
|
+
// outside search(), so the read-only /api/unified-search/health route can
|
|
34
|
+
// report the LAST known reachability without firing probes or burning quota.
|
|
35
|
+
// Entry shape: { ok: boolean, at: ms, ms: duration, count?: number,
|
|
36
|
+
// error?: string (already truncated to 80), lastOkAt: ms|null }
|
|
37
|
+
const latestBackendHealth = new Map();
|
|
38
|
+
|
|
39
|
+
/** Read-only access for the health route (never mutate the returned map). */
|
|
40
|
+
export function getLatestBackendHealth() {
|
|
41
|
+
return latestBackendHealth;
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
function recordBackendHealth(id, entry, breakerCfg) {
|
|
45
|
+
const prev = latestBackendHealth.get(id);
|
|
46
|
+
const breaker = foldBreaker(prev, entry.ok === true, entry.at, breakerCfg ?? undefined);
|
|
47
|
+
latestBackendHealth.set(id, {
|
|
48
|
+
...entry,
|
|
49
|
+
lastOkAt: entry.ok === true ? entry.at : (prev && prev.lastOkAt) || null,
|
|
50
|
+
failCount: breaker.failCount,
|
|
51
|
+
cooledUntil: breaker.cooledUntil,
|
|
52
|
+
});
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
/**
|
|
56
|
+
* Read-only access to one backend's breaker state (failCount / cooledUntil).
|
|
57
|
+
*/
|
|
58
|
+
export function getBackendBreaker(id) {
|
|
59
|
+
const e = latestBackendHealth.get(id);
|
|
60
|
+
return e ? { failCount: e.failCount ?? 0, cooledUntil: e.cooledUntil ?? 0 } : { failCount: 0, cooledUntil: 0 };
|
|
61
|
+
}
|
|
62
|
+
|
|
28
63
|
/**
|
|
29
64
|
* Simple concurrency limiter using a semaphore pattern.
|
|
30
65
|
* No external dependency needed.
|
|
@@ -52,6 +87,47 @@ function createLimiter(concurrency) {
|
|
|
52
87
|
};
|
|
53
88
|
}
|
|
54
89
|
|
|
90
|
+
/**
|
|
91
|
+
* PHASE 1 (collect): fold settled fan-out results into the raw merge inputs.
|
|
92
|
+
* Sources keep fan-out priority order; dedupe/rank over the FULL set happens
|
|
93
|
+
* later (a mid-merge maxResults cap would silently drop later backends'
|
|
94
|
+
* entries before cross-backend fill-in and ranking could see them).
|
|
95
|
+
* URL scheme whitelist is enforced here — javascript:/data: payloads from
|
|
96
|
+
* compromised or poisoned backends never reach model/GUI consumers.
|
|
97
|
+
*/
|
|
98
|
+
function collectFanoutResults(results, eligible, signal) {
|
|
99
|
+
const collected = [];
|
|
100
|
+
let mergedContent;
|
|
101
|
+
let hardAborted = false;
|
|
102
|
+
let hardError = null;
|
|
103
|
+
const failures = [];
|
|
104
|
+
|
|
105
|
+
let backendIndex = 0;
|
|
106
|
+
for (const settled of results) {
|
|
107
|
+
const failedId = eligible[backendIndex] ? eligible[backendIndex].id : "unknown";
|
|
108
|
+
backendIndex++;
|
|
109
|
+
if (settled.status === "fulfilled") {
|
|
110
|
+
if (mergedContent === undefined && typeof settled.value?.content === "string" && settled.value.content.length > 0) {
|
|
111
|
+
mergedContent = settled.value.content;
|
|
112
|
+
}
|
|
113
|
+
for (const src of settled.value?.sources ?? []) {
|
|
114
|
+
if (src && typeof src.url === "string" && /^https?:\/\//i.test(src.url)) collected.push(src);
|
|
115
|
+
}
|
|
116
|
+
} else {
|
|
117
|
+
const err = settled.reason;
|
|
118
|
+
if (err instanceof WebError && err.code === "WEB_ABORTED") {
|
|
119
|
+
// Per-backend abort -> soft unless the caller signal fired.
|
|
120
|
+
if (signal?.aborted === true) hardAborted = true;
|
|
121
|
+
else failures.push({ id: failedId, err });
|
|
122
|
+
} else {
|
|
123
|
+
failures.push({ id: failedId, err });
|
|
124
|
+
if (hardError === null) hardError = err;
|
|
125
|
+
}
|
|
126
|
+
}
|
|
127
|
+
}
|
|
128
|
+
return { collected, mergedContent, hardAborted, hardError, failures };
|
|
129
|
+
}
|
|
130
|
+
|
|
55
131
|
/**
|
|
56
132
|
* Build the UnifiedSearchProvider.
|
|
57
133
|
*
|
|
@@ -60,7 +136,7 @@ function createLimiter(concurrency) {
|
|
|
60
136
|
* @param {Function} deps.resolveOptions () => resolved options snapshot
|
|
61
137
|
* @param {object} deps.backends keyed backend registry
|
|
62
138
|
*/
|
|
63
|
-
export function createUnifiedSearchProvider({ ctx, resolveOptions, backends }) {
|
|
139
|
+
export function createUnifiedSearchProvider({ ctx, resolveOptions, backends, cache, history }) {
|
|
64
140
|
return {
|
|
65
141
|
id: PROVIDER_ID,
|
|
66
142
|
|
|
@@ -92,20 +168,84 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends }) {
|
|
|
92
168
|
...(request?.filters ?? {}),
|
|
93
169
|
});
|
|
94
170
|
let effReq = { ...request, maxResults: Math.max(1, Number(request?.maxResults) || opts.numResults || 8) };
|
|
171
|
+
// 系统性查询整形(Tavily/Exa best practices):扇出前对口语化查询做
|
|
172
|
+
// 确定性归一 —— 剥离寒暄前缀、压缩空白、截断到 1500 字符;
|
|
173
|
+
// 专名/版本号/日期原样保留,不改写语义。
|
|
174
|
+
effReq.query = shapeQuery(effReq.query);
|
|
95
175
|
if (Object.keys(filters).length > 0) effReq = { ...effReq, filters };
|
|
96
176
|
const maxResults = effReq.maxResults;
|
|
97
177
|
const concurrency = opts.concurrency ?? 6;
|
|
98
178
|
const backendTimeoutMs = opts.backendTimeoutMs ?? 30000;
|
|
99
179
|
|
|
100
|
-
//
|
|
101
|
-
|
|
180
|
+
// v2.7.0 disk cache: keyed on (shaped query + filters + maxResults).
|
|
181
|
+
// A hit short-circuits the fan-out entirely; sources are byte-identical
|
|
182
|
+
// to the first search and content carries a "cache hit, age Ns" note.
|
|
183
|
+
// Cache failures never break the search (cache API never throws).
|
|
184
|
+
const cacheEnabled = opts.cacheEnabled === true && cache;
|
|
185
|
+
let cacheKey = null;
|
|
186
|
+
if (cacheEnabled) {
|
|
187
|
+
cacheKey = cacheKeyFor({ query: effReq.query, filters, maxResults, backends: opts.enabledBackends });
|
|
188
|
+
const hit = cache.get(cacheKey);
|
|
189
|
+
if (hit && Array.isArray(hit.value?.sources) && hit.value.sources.length > 0) {
|
|
190
|
+
// Defense in depth: the fresh path drops non-http(s) schemes before
|
|
191
|
+
// merge; a tampered cache file must not bypass that whitelist.
|
|
192
|
+
const safeSources = hit.value.sources.filter(
|
|
193
|
+
(src) => src && typeof src.url === "string" && /^https?:\/\//i.test(src.url),
|
|
194
|
+
);
|
|
195
|
+
if (safeSources.length === 0) {
|
|
196
|
+
// whole entry poisoned or filtered empty -> treat as a miss
|
|
197
|
+
cacheKey = null;
|
|
198
|
+
} else {
|
|
199
|
+
let hitContent = typeof hit.value.content === "string" ? hit.value.content : undefined;
|
|
200
|
+
const cacheNote = `[websearch cache] cache hit, age ${hit.age}s`;
|
|
201
|
+
hitContent = hitContent === undefined ? cacheNote : hitContent + "\n\n" + cacheNote;
|
|
202
|
+
history?.record({
|
|
203
|
+
query: request?.query,
|
|
204
|
+
time: Date.now(),
|
|
205
|
+
resultCount: hit.value.sources.length,
|
|
206
|
+
backendsOk: 0,
|
|
207
|
+
backendsTotal: 0,
|
|
208
|
+
});
|
|
209
|
+
return {
|
|
210
|
+
sources: safeSources,
|
|
211
|
+
...(hitContent !== undefined ? { content: hitContent } : {}),
|
|
212
|
+
truncated: hit.value.truncated === true,
|
|
213
|
+
};
|
|
214
|
+
}
|
|
215
|
+
}
|
|
216
|
+
}
|
|
217
|
+
|
|
218
|
+
// Pick enabled & available backends. With the breaker enabled, a
|
|
219
|
+
// backend inside its 60s cooldown window is skipped (annotated in
|
|
220
|
+
// telemetry as "id ⏸cooled"). Fail-open: if EVERY candidate is cooled,
|
|
221
|
+
// run them all anyway — the breaker must never make search worse than
|
|
222
|
+
// no breaker.
|
|
223
|
+
const breakerCfg = opts.breakerEnabled === false ? null : {
|
|
224
|
+
threshold: opts.breakerThreshold,
|
|
225
|
+
cooldownMs: opts.breakerCooldownMs,
|
|
226
|
+
};
|
|
227
|
+
const nowMs = Date.now();
|
|
228
|
+
const candidates = [];
|
|
102
229
|
for (const id of opts.enabledBackends) {
|
|
103
230
|
const be = backends[id];
|
|
104
231
|
if (!be) continue;
|
|
105
232
|
const beOpts = opts.backends?.[id] ?? {};
|
|
106
233
|
const avail = be.available(beOpts);
|
|
107
234
|
if (!avail) continue;
|
|
108
|
-
|
|
235
|
+
candidates.push({ id, backend: be, beOpts });
|
|
236
|
+
}
|
|
237
|
+
const cooled = new Map();
|
|
238
|
+
let eligible = candidates;
|
|
239
|
+
if (breakerCfg) {
|
|
240
|
+
const active = candidates.filter((c) => {
|
|
241
|
+
if (isCooled(latestBackendHealth.get(c.id), nowMs)) {
|
|
242
|
+
cooled.set(c.id, latestBackendHealth.get(c.id));
|
|
243
|
+
return false;
|
|
244
|
+
}
|
|
245
|
+
return true;
|
|
246
|
+
});
|
|
247
|
+
if (active.length > 0) eligible = active;
|
|
248
|
+
else cooled.clear(); // all cooled -> fail-open
|
|
109
249
|
}
|
|
110
250
|
|
|
111
251
|
if (eligible.length === 0) {
|
|
@@ -121,7 +261,7 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends }) {
|
|
|
121
261
|
|
|
122
262
|
const telemetry = [];
|
|
123
263
|
// Concurrency limiter
|
|
124
|
-
const limit = createLimiter(
|
|
264
|
+
const limit = createLimiter(concurrency);
|
|
125
265
|
|
|
126
266
|
// Run one backend under the per-backend timeout. The internal timer's
|
|
127
267
|
// controller is merged into the caller's signal; when the INTERNAL timer
|
|
@@ -152,10 +292,13 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends }) {
|
|
|
152
292
|
.then(
|
|
153
293
|
(value) => {
|
|
154
294
|
telemetry.push({ id, ok: true, ms: Date.now() - startedAtMs, count: value?.sources?.length ?? 0 });
|
|
295
|
+
recordBackendHealth(id, { ok: true, at: Date.now(), ms: Date.now() - startedAtMs, count: value?.sources?.length ?? 0 }, breakerCfg);
|
|
155
296
|
return value;
|
|
156
297
|
},
|
|
157
298
|
(err) => {
|
|
158
|
-
|
|
299
|
+
const msg = String(err instanceof Error ? err.message : err).slice(0, 80);
|
|
300
|
+
telemetry.push({ id, ok: false, ms: Date.now() - startedAtMs, error: msg });
|
|
301
|
+
recordBackendHealth(id, { ok: false, at: Date.now(), ms: Date.now() - startedAtMs, error: msg }, breakerCfg);
|
|
159
302
|
throw err;
|
|
160
303
|
},
|
|
161
304
|
)
|
|
@@ -184,50 +327,18 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends }) {
|
|
|
184
327
|
|
|
185
328
|
throwIfSearchAborted(signal);
|
|
186
329
|
|
|
187
|
-
//
|
|
188
|
-
//
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
const collected = [];
|
|
192
|
-
let mergedContent;
|
|
193
|
-
let hardAborted = false;
|
|
194
|
-
let hardError = null;
|
|
195
|
-
const failures = [];
|
|
196
|
-
|
|
197
|
-
let backendIndex = 0;
|
|
198
|
-
for (const settled of results) {
|
|
199
|
-
const failedId = eligible[backendIndex] ? eligible[backendIndex].id : "unknown";
|
|
200
|
-
backendIndex++;
|
|
201
|
-
if (settled.status === "fulfilled") {
|
|
202
|
-
if (mergedContent === undefined && typeof settled.value?.content === "string" && settled.value.content.length > 0) {
|
|
203
|
-
mergedContent = settled.value.content;
|
|
204
|
-
}
|
|
205
|
-
for (const src of settled.value?.sources ?? []) {
|
|
206
|
-
// Scheme whitelist: parsed URLs reach model/GUI consumers; drop
|
|
207
|
-
// javascript:/data: payloads from compromised or poisoned backends.
|
|
208
|
-
if (src && typeof src.url === "string" && !/^https?:\/\//i.test(src.url)) continue;
|
|
209
|
-
collected.push(src);
|
|
210
|
-
}
|
|
211
|
-
} else {
|
|
212
|
-
// settled.status === "rejected"
|
|
213
|
-
const err = settled.reason;
|
|
214
|
-
if (err instanceof WebError && err.code === "WEB_ABORTED") {
|
|
215
|
-
// Per-backend abort → soft unless caller signal fired.
|
|
216
|
-
if (signal?.aborted === true) hardAborted = true;
|
|
217
|
-
else failures.push({ id: failedId, err });
|
|
218
|
-
} else {
|
|
219
|
-
failures.push({ id: failedId, err });
|
|
220
|
-
if (hardError === null) hardError = err;
|
|
221
|
-
}
|
|
222
|
-
}
|
|
223
|
-
}
|
|
330
|
+
// PHASE 1 collect: fold the settled fan-out into raw sources (fan-out
|
|
331
|
+
// priority order), first backend content, and per-backend failures.
|
|
332
|
+
const { collected, mergedContent, hardAborted, hardError, failures } =
|
|
333
|
+
collectFanoutResults(results, eligible, signal);
|
|
224
334
|
|
|
225
335
|
// Caller-cancellation check (hard abort fires before we yield results).
|
|
226
336
|
if (signal?.aborted === true || hardAborted) {
|
|
227
337
|
throw searchAborted(signal);
|
|
228
338
|
}
|
|
229
339
|
|
|
230
|
-
//
|
|
340
|
+
// PHASE 2 dedupe / PHASE 3 rerank: "url" preserves the historical
|
|
341
|
+
// URL-key behavior; "url+title"
|
|
231
342
|
// additionally collapses same-story syndicated copies across backends.
|
|
232
343
|
const strategy = opts.dedupStrategy === "url+title" ? "url+title" : "url";
|
|
233
344
|
const deduped = dedupeSources(collected, strategy);
|
|
@@ -240,11 +351,37 @@ export function createUnifiedSearchProvider({ ctx, resolveOptions, backends }) {
|
|
|
240
351
|
// If we got ANY sources, return them even if some backends failed.
|
|
241
352
|
if (ordered.length > 0) {
|
|
242
353
|
const sources = ordered.slice(0, maxResults);
|
|
354
|
+
// 结果呈现头(Anthropic web-search / OpenAI grounding 官方指引):
|
|
355
|
+
// 置于结果块最前,要求逐条引用来源 URL、优先官方/一手与较新来源;
|
|
356
|
+
// 语言按查询文本自动选择(含 CJK 用中文头)。可用
|
|
357
|
+
// opts.resultHeader === false 关闭。
|
|
243
358
|
let outContent = mergedContent;
|
|
244
|
-
if (opts.
|
|
245
|
-
const
|
|
359
|
+
if (opts.resultHeader !== false && (sources.length > 0 || outContent !== undefined)) {
|
|
360
|
+
const header = resultHeader(typeof request?.query === "string" && /[\u4e00-\u9fff]/.test(request.query) ? "zh" : "en");
|
|
361
|
+
outContent = outContent === undefined ? header : header + "\n\n" + outContent;
|
|
362
|
+
}
|
|
363
|
+
// Store the pre-telemetry core (sources + headered content) so a cache
|
|
364
|
+
// hit reproduces the exact sources and the model-facing header.
|
|
365
|
+
if (cacheKey && cache) {
|
|
366
|
+
cache.set(cacheKey, { sources, content: outContent, truncated: ordered.length > maxResults });
|
|
367
|
+
}
|
|
368
|
+
history?.record({
|
|
369
|
+
query: request?.query,
|
|
370
|
+
time: Date.now(),
|
|
371
|
+
resultCount: sources.length,
|
|
372
|
+
backendsOk: telemetry.filter((t) => t.ok).length,
|
|
373
|
+
backendsTotal: telemetry.length,
|
|
374
|
+
});
|
|
375
|
+
if (opts.resultTelemetry !== false && (telemetry.length > 0 || cooled.size > 0)) {
|
|
376
|
+
const cooledParts = [];
|
|
377
|
+
for (const [cid, centry] of cooled) {
|
|
378
|
+
const ann = cooledAnnotation(centry, Date.now()) || "⏸cooled";
|
|
379
|
+
cooledParts.push(cid + " " + ann);
|
|
380
|
+
}
|
|
381
|
+
const parts = telemetry
|
|
246
382
|
.map((t) => t.id + (t.ok ? " ✓" : " ✗") + t.ms + "ms" + (t.ok ? "/" + t.count : (t.error ? " (" + String(t.error).slice(0, 40) + ")" : "")))
|
|
247
|
-
.
|
|
383
|
+
.concat(cooledParts);
|
|
384
|
+
const line = "[websearch backends]" + TELEMETRY_TAG + " " + parts.join(" · ");
|
|
248
385
|
outContent = outContent === undefined ? line : outContent + "\n\n" + line;
|
|
249
386
|
}
|
|
250
387
|
return {
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@240xu/dsh-websearch",
|
|
3
|
-
"version": "2.
|
|
4
|
-
"description": "Unified web search provider for DeepSeek Harness (ctx.web): 11 backends
|
|
3
|
+
"version": "2.7.3",
|
|
4
|
+
"description": "Unified web search provider for DeepSeek Harness (ctx.web): 11 backends — Exa + Parallel + DDG + SearXNG (keyless), DeepSeek + Anthropic + OpenAI + Brave + Tavily + Serper + Mojeek (key-gated via Settings credentials). Registers provider id \"unified\" for zero-config web_search. 统一网页搜索聚合:11 后端并发扇出、去重合并,零依赖。See CHANGELOG.md for version history.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "lib/index.js",
|
|
7
7
|
"exports": {
|
|
@@ -66,5 +66,8 @@
|
|
|
66
66
|
],
|
|
67
67
|
"platform": "web"
|
|
68
68
|
}
|
|
69
|
+
},
|
|
70
|
+
"engines": {
|
|
71
|
+
"node": ">=20.3"
|
|
69
72
|
}
|
|
70
73
|
}
|