@fffangchenghao/dsh-web-search 0.13.11 → 0.15.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,147 @@
1
+ # How the ranking works, and how it is measured
2
+
3
+ Most "web search plugins" for an agent harness are thin wrappers: call one
4
+ engine, return its rows. That is enough to prove the model can reach the web —
5
+ and it is why result quality is usually discussed as a feeling.
6
+
7
+ This plugin treats the eight rows the Host shows the model as the product. That
8
+ means the ranking has to be explainable, and the explanation has to survive a
9
+ measurement. This page is both.
10
+
11
+ ![Search pipeline](images/pipeline.png)
12
+
13
+ ## The pipeline
14
+
15
+ 1. **Fan out.** Every enabled engine is queried concurrently with its own
16
+ deadline (`timeoutMs`, default 15 s) and its own cooldown state. An engine
17
+ that just failed is skipped instead of being retried on every query. The
18
+ primary tier answers first; the fallback tier only runs when every primary
19
+ came back empty.
20
+ 2. **Normalise.** Each parser returns `{title, url, snippet, position, votes}`
21
+ with the engine's own result order preserved. URLs are canonicalised before
22
+ de-duplication so `?utm_source=…` cannot smuggle the same page in twice.
23
+ 3. **Interleave.** Engines are merged by weighted round-robin, not by
24
+ concatenation: an engine with weight 2 contributes two slots per round. This
25
+ keeps one verbose engine from filling all eight rows before a second engine
26
+ is even looked at. `anysearch` carries weight 2 because its precision was
27
+ measured highest on both Chinese and English queries.
28
+ 4. **Fuse.** Each engine's list is scored with reciprocal-rank fusion:
29
+
30
+ ```
31
+ vote = 1 / (rrfConstant + position) # rrfConstant default 60
32
+ score = Σ votes
33
+ + authorityBoost * isOfficialDomain
34
+ + freshnessBoost * freshness(parsedDate, freshnessDays)
35
+ ```
36
+
37
+ RRF is used instead of score normalisation because engines report scores on
38
+ incomparable scales (or none at all); ranks are the only signal every engine
39
+ provides honestly.
40
+ 5. **Region policy.** A CJK query leads with the engines that answer Chinese
41
+ sources (`cn.bing`, `anysearch`); an ASCII query leads with the global set
42
+ (`ddg-html`, `www.bing`, `anysearch`). The other region's engines still
43
+ contribute, so a Chinese query can still surface an English original.
44
+ 6. **Cut.** The fused list is truncated to `limit` (default 8, clamped 1–20).
45
+ Only the rows that survive are returned; the model never sees the raw
46
+ per-engine lists.
47
+
48
+ ## Why these exact knobs
49
+
50
+ Each knob exists because a specific failure was observed:
51
+
52
+ | knob | default | what it prevents |
53
+ |---|---|---|
54
+ | `rrfConstant` | 60 | a single engine's first result dominating the fusion |
55
+ | `authorityBoost` | 0.25 | documentation on `kubernetes.io` losing to a SEO farm |
56
+ | `freshnessBoost` / `freshnessDays` | 0.2 / 30 | last year's release notes outranking this month's |
57
+ | region split | on | English pages burying the Chinese documentation a Chinese query asked for |
58
+ | engine weight | `anysearch` = 2 | a precise engine receiving the same airtime as a noisy one |
59
+
60
+ ## How it is measured
61
+
62
+ `bench/` holds 20 fixed queries — 10 Chinese, 10 English, five each in docs /
63
+ release / package / news — each annotated with 1–3 expected domains. Encoding
64
+ the expectation in domains rather than exact URLs is deliberate: result titles
65
+ and snippets change constantly, domains are the part the ranking is supposed to
66
+ get right.
67
+
68
+ ```
69
+ node scripts/bench.mjs # offline: replay recorded fixtures, no network
70
+ node scripts/bench.mjs --record # refresh fixtures from the live engines
71
+ node scripts/bench.mjs --json # machine-readable output for version diffs
72
+ ```
73
+
74
+ The bench reports precision@8, recall@8 and MRR@8 overall, per language, per
75
+ category and per engine, plus the worst query. Offline replay makes the number
76
+ comparable across commits; `--record` is the only mode that touches the network.
77
+
78
+ ![Baseline and sweep](images/bench.png)
79
+
80
+ Baseline (v0.15.0, 20 queries, k=8, tier-aware replay):
81
+
82
+ | metric | overall | Chinese | English |
83
+ |---|---:|---:|---:|
84
+ | P@8 | 0.4375 | 0.4000 | 0.4750 |
85
+ | R@8 | 0.7917 | 0.8167 | 0.7667 |
86
+ | MRR@8 | 0.8071 | 0.7976 | 0.8167 |
87
+
88
+ Replay is tier-aware: it applies the same primary-then-fallback gate the live
89
+ `search()` applies, instead of ranking every recorded engine as if it had
90
+ answered. That changed the protocol in 0.15.0, so numbers from earlier releases
91
+ (0.7600 MRR) are not directly comparable to these.
92
+
93
+ ## Parameter sweep
94
+
95
+ `scripts/sweep.mjs` runs an 81-configuration grid over the same fixtures and
96
+ reports marginal means plus a paired bootstrap against the shipped defaults:
97
+
98
+ ```
99
+ node scripts/sweep.mjs # full grid (~2 min, offline)
100
+ node scripts/sweep.mjs --quick # 27-config subset
101
+ node scripts/sweep.mjs --json # includes the per-config CI data
102
+ ```
103
+
104
+ Marginal means over the whole grid (every other knob varies with it):
105
+
106
+ | `rrfConstant` | MRR@8 | `authorityBoost` | MRR@8 | `freshnessBoost` | MRR@8 |
107
+ |---:|---:|---:|---:|---:|---:|
108
+ | 40 | 0.8339 | 0.1 | 0.8275 | 0.1 | 0.8230 |
109
+ | **60** | **0.8278** | **0.25** | **0.8262** | **0.2** | **0.8245** |
110
+ | 80 | 0.8195 | 0.4 | 0.8275 | 0.35 | 0.8336 |
111
+
112
+ The best grid point (`rrf=40, authority=0.25, freshness=0.35`) measures MRR
113
+ 0.8583 against the default 0.8071 — a gain of +0.0512 — but its 95 %
114
+ paired-bootstrap interval `[0.0000, +0.1250]` touches zero, and the marginals
115
+ above differ from the default by less than the interval width. On a 20-query
116
+ corpus that is a direction, not a proven win, so the defaults did not change.
117
+ The sweep is kept as a script precisely so this question can be re-asked after
118
+ the query set grows.
119
+
120
+ The same discipline decided the new engines' tiers. On one recorded fixture
121
+ set, with the tier-aware replay:
122
+
123
+ | configuration | MRR@8 |
124
+ |---|---:|
125
+ | previous primaries only | 0.7738 |
126
+ | **previous primaries + Exa** | **0.8071** |
127
+ | previous primaries + Firecrawl | 0.7208 |
128
+ | previous primaries + both | 0.7542 |
129
+
130
+ Exa therefore leads; Firecrawl is parsed and tested but ships as the fallback,
131
+ available through `hostIds: ['firecrawl']` and used whenever every primary
132
+ returns nothing.
133
+
134
+ ## Known limits
135
+
136
+ - **Relative Chinese time words.** A query like "上周发布的那版" carries its time
137
+ constraint in the words themselves. The freshness boost can only act on dates
138
+ it can parse out of snippets; when an engine's snippet omits the date, the
139
+ ranker cannot recover it. This is documented rather than papered over.
140
+ - **Replay, not live search.** The bench replays recorded rows, so it measures
141
+ the ranker and the fusion, not today's engine availability. `--record` is the
142
+ only mode that touches the network.
143
+ - **Snippet quality bounds ranking.** If no engine returns the expected domain
144
+ in its top 20, fusion cannot invent it; `bench/` makes that visible per query.
145
+ - **The fixture set is 20 queries.** Small enough to run on every change, too
146
+ small to certify a general ranking. The per-query output is kept so the number
147
+ cannot hide a single catastrophic query.
package/lib/config.js CHANGED
@@ -21,7 +21,7 @@
21
21
  */
22
22
  import z from '@deepseek-ai/schemastery';
23
23
  import { DEFAULT_FETCH_MAX_LENGTH } from "./markdown.js";
24
- import { VERTICAL_IDS, isVerticalId } from "./verticals.js";
24
+ import { DEFAULT_VERTICAL_IDS, VERTICAL_IDS, isVerticalId } from "./verticals.js";
25
25
  /** Schema of the plugin's configuration section. */
26
26
  export const Config = z.object({
27
27
  hosts: z
@@ -101,8 +101,8 @@ export const Config = z.object({
101
101
  maxCooldownMs: z.natural().default(1800000).description('它反复被拒时,最长让它歇多久(毫秒),也就是等待一步步加长的封顶值。一般不用改;保持 1800000(30 分钟)即可。').volatile(),
102
102
  domainSources: z
103
103
  .union([z.array(z.string()), z.string()])
104
- .default([...VERTICAL_IDS])
105
- .description('默认查哪些专用源,逗号分隔;留空表示 github、npm 等全部八个源。')
104
+ .default([...DEFAULT_VERTICAL_IDS])
105
+ .description('默认查哪些专用源,逗号分隔;留空表示默认的 8 个(github、npm、pypi、arxiv、wikipedia、hn、stackexchange、crates)。其余源可用 sources 参数按需指定。')
106
106
  .volatile(),
107
107
  domainLimit: z.natural().min(1).max(20).default(5).description('domain_search 一次最多返回几条(1-20);默认 5 条,想多看可调到 10。').volatile(),
108
108
  domainTimeoutMs: z.natural().min(500).default(8000).description('每个专用源最多等多久(毫秒);网络慢可加到 15000,平时 8000 即可。').volatile(),
@@ -167,7 +167,7 @@ export function resolveConfig(config, callLimit) {
167
167
  timeRange: settings.timeRange,
168
168
  anysearchApiKey: settings.anysearchApiKey.trim(),
169
169
  anysearchDomain: settings.anysearchDomain.trim(),
170
- domainSourceIds: domainSources.length > 0 ? domainSources : [...VERTICAL_IDS],
170
+ domainSourceIds: domainSources.length > 0 ? domainSources : [...DEFAULT_VERTICAL_IDS],
171
171
  domainLimit: settings.domainLimit,
172
172
  domainTimeoutMs: settings.domainTimeoutMs,
173
173
  rank: {
@@ -1,5 +1,5 @@
1
1
  import { resolveProxy } from "./proxy.js";
2
- import { searchVerticals, isVerticalId, VERTICAL_IDS } from "./verticals.js";
2
+ import { DEFAULT_VERTICAL_IDS, searchVerticals, isVerticalId, VERTICAL_IDS } from "./verticals.js";
3
3
  /** The name this tool prefers. */
4
4
  export const DOMAIN_TOOL_NAME = 'domain_search';
5
5
  /** The name used when another plugin already registered `domain_search`. */
@@ -70,7 +70,7 @@ function parseArgs(args, settings) {
70
70
  if (query.length === 0)
71
71
  throw new Error('domain_search requires a non-empty query');
72
72
  const enabled = settings.sourceIds.filter(isVerticalId);
73
- let ids = enabled.length > 0 ? enabled : [...VERTICAL_IDS];
73
+ let ids = enabled.length > 0 ? enabled : [...DEFAULT_VERTICAL_IDS];
74
74
  if (Array.isArray(args.sources) && args.sources.length > 0) {
75
75
  const asked = args.sources.filter(isVerticalId);
76
76
  if (asked.length === 0) {
package/lib/engines.js CHANGED
@@ -1,7 +1,7 @@
1
1
  import { SearchError } from "./errors.js";
2
2
  import { filterByTimeRange, preferWindow } from "./freshness.js";
3
3
  import { resolveProxy } from "./proxy.js";
4
- import { parseAnySearch, parseBing, parseDdgHtml, parseDdgLite, parseSo } from "./parsers.js";
4
+ import { parseAnySearch, parseBing, parseDdgHtml, parseDdgLite, parseExaMcp, parseFirecrawl, parseSo } from "./parsers.js";
5
5
  import { mergeRegions } from "./rank.js";
6
6
  import { cacheKey } from "./cache.js";
7
7
  import { searchHost } from "./transport.js";
@@ -35,6 +35,48 @@ export const HOSTS = [
35
35
  body: (query) => JSON.stringify({ query, max_results: ANYSEARCH_MAX_RESULTS }),
36
36
  parse: parseAnySearch,
37
37
  },
38
+ {
39
+ id: 'exa',
40
+ // A semantic (embedding) index rather than a keyword page: it answers with
41
+ // the page itself plus highlights. The MCP endpoint accepts anonymous
42
+ // calls, so it costs no key. Measured on the same recorded fixtures with the
43
+ // tier-aware replay, MRR@8 0.8071 with Exa in the primary tier versus
44
+ // 0.7738 without it, and it removes the one zero-hit query.
45
+ tier: 'primary',
46
+ region: 'global',
47
+ weight: 1,
48
+ viaProxy: true,
49
+ method: 'POST',
50
+ skipEcho: true,
51
+ headers: { accept: 'application/json, text/event-stream' },
52
+ build: () => 'https://mcp.exa.ai/mcp',
53
+ body: (query) => JSON.stringify({
54
+ jsonrpc: '2.0',
55
+ id: 1,
56
+ method: 'tools/call',
57
+ params: { name: 'web_search_exa', arguments: { query, numResults: 10 } },
58
+ }),
59
+ parse: parseExaMcp,
60
+ },
61
+ {
62
+ id: 'firecrawl',
63
+ // The keyless tier of the Firecrawl search API. It returns extracted page
64
+ // Markdown rather than snippets, so only the opening prose is kept as the
65
+ // row's snippet; it handled a Chinese query with the same shape. It ships as
66
+ // a fallback because it diluted the fusion when it led: MRR@8 0.7208 with
67
+ // Firecrawl in the primary tier versus 0.8071 with Exa there instead, on the
68
+ // same fixtures (standalone 0.4625). `hostIds: ['firecrawl']` still forces
69
+ // it, and it answers whenever every primary came back empty.
70
+ tier: 'fallback',
71
+ region: 'both',
72
+ weight: 1,
73
+ viaProxy: true,
74
+ method: 'POST',
75
+ skipEcho: true,
76
+ build: () => 'https://api.firecrawl.dev/v1/search',
77
+ body: (query) => JSON.stringify({ query, limit: 10 }),
78
+ parse: parseFirecrawl,
79
+ },
38
80
  {
39
81
  id: 'ddg-html',
40
82
  tier: 'primary',
package/lib/index.d.ts CHANGED
@@ -138,12 +138,14 @@ export { resolveHref, parseDdgHtml, parseDdgLite, parseBing, parseSo, parseAnySe
138
138
  export { fuse, interleave, dedupe, mergeRegions } from './rank.ts';
139
139
  export { MS_PER_DAY, TIME_RANGE_MS, TIME_RANGES, parsePublishedAt, extractPublishedAt, publishedTimestamp, isWithinTimeRange, filterByTimeRange, } from './freshness.ts';
140
140
  export { parseProxy, proxyFromEnv, resolveProxy, proxyGet } from './proxy.ts';
141
- export { VERTICAL_IDS, VERTICALS, searchVerticals, verticalById, isVerticalId, parseGithub, parseNpm, parsePypi, parseArxiv, parseWikipedia, parseHn, parseStackExchange, parseCrates, } from './verticals.ts';
141
+ export { DEFAULT_VERTICAL_IDS, VERTICAL_IDS, VERTICALS, searchVerticals, verticalById, isVerticalId, parseGithub, parseNpm, parsePypi, parseArxiv, parseWikipedia, parseHn, parseStackExchange, parseCrates, } from './verticals.ts';
142
142
  export { DOMAIN_TOOL_NAME, DOMAIN_TOOL_FALLBACK_NAME, createDomainToolDefinition, registerDomainTool, formatDomainValue, } from './domain-tool.ts';
143
143
  export type { VerticalId, VerticalSpec, VerticalOutcome, VerticalSearchOptions } from './verticals.ts';
144
144
  export type { DomainToolSettings, DomainToolValue, DomainToolDefinition, ToolRegistryLike } from './domain-tool.ts';
145
145
  export { SEARCH_MANY_TOOL_NAME, SEARCH_MANY_FALLBACK_NAME, SEARCH_MANY_CONCURRENCY, SEARCH_MANY_DEFAULT_LIMIT, createSearchManyDefinition, registerSearchManyTool, pickSearchManyToolName, formatSearchManyValue, } from './search-many.ts';
146
146
  export type { SearchManyDefinition, SearchManyOutcome, SearchManyRegistration, SearchManyRegistryLike, SearchManyResult, SearchManyRunner, SearchManySource, SearchManyValue, } from './search-many.ts';
147
+ export { RESEARCH_TOOL_NAME, RESEARCH_TOOL_FALLBACK_NAME, RESEARCH_DEFAULT_SOURCES, RESEARCH_MAX_SOURCES, RESEARCH_MAX_QUERIES, RESEARCH_EXCERPT_CHARS, RESEARCH_BUDGET_MS, createResearchDefinition, registerResearchTool, pickResearchToolName, formatResearchValue, } from './research.ts';
148
+ export type { ResearchDefinition, ResearchFetcher, ResearchFetchedPage, ResearchFetchFailure, ResearchPage, ResearchRegistration, ResearchRegistryLike, ResearchSearcher, ResearchSearchOutcome, ResearchValue, } from './research.ts';
147
149
  export { READ_PAGE_TOOL_NAME, READ_PAGE_TOOL_FALLBACK_NAME, READ_PAGE_MIN_LENGTH, READ_PAGE_MAX_LENGTH, READ_PAGE_DEFAULT_LENGTH, READ_PAGE_INTERNAL_MAX_CHARS, READ_PAGE_CACHE_ENTRIES, READ_PAGE_PROVIDER_ID, createReadPageDefinition, registerReadPageTool, pickReadPageToolName, formatReadPageValue, } from './read-page.ts';
148
150
  export type { ReadPageDefinition, ReadPageOptions, ReadPageRegistration, ReadPageRegistryLike, ReadPageSettings, ReadPageValue, } from './read-page.ts';
149
151
  export type { Source, SearchOutcome, RankOptions, SearchOptions, HostSpec, Region, TimeRange, WaitPolicy } from './types.ts';
package/lib/index.js CHANGED
@@ -17,6 +17,7 @@ import { createFetchProvider } from "./fetch.js";
17
17
  import { registerDomainTool } from "./domain-tool.js";
18
18
  import { registerSearchManyTool } from "./search-many.js";
19
19
  import { registerReadPageTool } from "./read-page.js";
20
+ import { registerResearchTool } from "./research.js";
20
21
  export const name = 'dsh-web-search';
21
22
  /** The `ctx.web` service this plugin registers a search provider with. */
22
23
  export const inject = ['web'];
@@ -214,6 +215,33 @@ export function apply(ctx, config = {}) {
214
215
  if (paged !== undefined) {
215
216
  onDispose(ctx, paged.dispose);
216
217
  }
218
+ // Research: search -> fetch -> cite in one call. The fetch side reuses the
219
+ // same ladder as `web_fetch`, but through its own provider: the renderer
220
+ // load is cached per provider, and a research pack must not replace the
221
+ // page the host already fetched for the user.
222
+ const researchProvider = createFetchProvider({
223
+ id: 'http-proxy-research',
224
+ timeoutMs: mounted.timeoutMs,
225
+ format: 'markdown',
226
+ maxLength: 6_000,
227
+ renderJs: () => resolveConfig(section, undefined).renderJs,
228
+ renderTimeoutMs: () => resolveConfig(section, undefined).renderTimeoutMs,
229
+ ...(mounted.proxy === undefined ? {} : { proxy: mounted.proxy }),
230
+ ...(mounted.anysearchApiKey.length === 0 ? {} : { anysearchApiKey: mounted.anysearchApiKey }),
231
+ });
232
+ const researchFetch = async (url, signal) => {
233
+ const outcome = await researchProvider.fetch({ url }, signal);
234
+ return {
235
+ url: outcome.url,
236
+ status: outcome.statusCode,
237
+ text: outcome.body.kind === 'text' ? outcome.body.content : '',
238
+ truncated: outcome.truncated,
239
+ };
240
+ };
241
+ const research = registerResearchTool(tools, (query, limit, signal) => runQuery(query, limit, signal), researchFetch, { warn: warnTool });
242
+ if (research !== undefined) {
243
+ onDispose(ctx, research.dispose);
244
+ }
217
245
  }
218
246
  }
219
247
  catch (error) {
@@ -244,7 +272,8 @@ export { resolveHref, parseDdgHtml, parseDdgLite, parseBing, parseSo, parseAnySe
244
272
  export { fuse, interleave, dedupe, mergeRegions } from "./rank.js";
245
273
  export { MS_PER_DAY, TIME_RANGE_MS, TIME_RANGES, parsePublishedAt, extractPublishedAt, publishedTimestamp, isWithinTimeRange, filterByTimeRange, } from "./freshness.js";
246
274
  export { parseProxy, proxyFromEnv, resolveProxy, proxyGet } from "./proxy.js";
247
- export { VERTICAL_IDS, VERTICALS, searchVerticals, verticalById, isVerticalId, parseGithub, parseNpm, parsePypi, parseArxiv, parseWikipedia, parseHn, parseStackExchange, parseCrates, } from "./verticals.js";
275
+ export { DEFAULT_VERTICAL_IDS, VERTICAL_IDS, VERTICALS, searchVerticals, verticalById, isVerticalId, parseGithub, parseNpm, parsePypi, parseArxiv, parseWikipedia, parseHn, parseStackExchange, parseCrates, } from "./verticals.js";
248
276
  export { DOMAIN_TOOL_NAME, DOMAIN_TOOL_FALLBACK_NAME, createDomainToolDefinition, registerDomainTool, formatDomainValue, } from "./domain-tool.js";
249
277
  export { SEARCH_MANY_TOOL_NAME, SEARCH_MANY_FALLBACK_NAME, SEARCH_MANY_CONCURRENCY, SEARCH_MANY_DEFAULT_LIMIT, createSearchManyDefinition, registerSearchManyTool, pickSearchManyToolName, formatSearchManyValue, } from "./search-many.js";
278
+ export { RESEARCH_TOOL_NAME, RESEARCH_TOOL_FALLBACK_NAME, RESEARCH_DEFAULT_SOURCES, RESEARCH_MAX_SOURCES, RESEARCH_MAX_QUERIES, RESEARCH_EXCERPT_CHARS, RESEARCH_BUDGET_MS, createResearchDefinition, registerResearchTool, pickResearchToolName, formatResearchValue, } from "./research.js";
250
279
  export { READ_PAGE_TOOL_NAME, READ_PAGE_TOOL_FALLBACK_NAME, READ_PAGE_MIN_LENGTH, READ_PAGE_MAX_LENGTH, READ_PAGE_DEFAULT_LENGTH, READ_PAGE_INTERNAL_MAX_CHARS, READ_PAGE_CACHE_ENTRIES, READ_PAGE_PROVIDER_ID, createReadPageDefinition, registerReadPageTool, pickReadPageToolName, formatReadPageValue, } from "./read-page.js";
package/lib/parsers.d.ts CHANGED
@@ -90,3 +90,28 @@ export declare function parseAnySearch(payload: string, baseUrl: string): Source
90
90
  * @returns whether the payload contains a distinctive query token.
91
91
  */
92
92
  export declare function echoesQuery(payload: string, query: string): boolean;
93
+ /**
94
+ * Parse the Firecrawl `/v1/search` JSON API.
95
+ *
96
+ * The endpoint is credential-free and answers
97
+ * `{ success: true, data: [{ title, url, description }] }`, where `description`
98
+ * is the page's extracted Markdown rather than a search snippet. Only the
99
+ * opening prose is kept: a whole page in one snippet would push every other
100
+ * engine's result out of the model-facing row.
101
+ * @param payload - the response body.
102
+ * @param baseUrl - the endpoint URL, used to resolve relative links.
103
+ * @returns parsed sources in the served order; empty on drift.
104
+ */
105
+ export declare function parseFirecrawl(payload: string, baseUrl: string): Source[];
106
+ /**
107
+ * Parse an Exa MCP `web_search_exa` answer.
108
+ *
109
+ * The MCP server replies with one SSE frame whose `text` content lists results
110
+ * as `Title: … / URL: … / Published: … / Author: … / Highlights: …` blocks
111
+ * separated by `---`. The block is the only stable shape here, so it is parsed
112
+ * line-wise; anything unexpected yields fewer rows, never an exception.
113
+ * @param payload - the raw SSE body.
114
+ * @param baseUrl - the MCP endpoint, used to resolve relative links.
115
+ * @returns parsed sources in the served order; empty on drift.
116
+ */
117
+ export declare function parseExaMcp(payload: string, baseUrl: string): Source[];
package/lib/parsers.js CHANGED
@@ -354,3 +354,142 @@ export function echoesQuery(payload, query) {
354
354
  const lower = payload.toLowerCase();
355
355
  return tokens.some((token) => lower.includes(token));
356
356
  }
357
+ /**
358
+ * Parse the Firecrawl `/v1/search` JSON API.
359
+ *
360
+ * The endpoint is credential-free and answers
361
+ * `{ success: true, data: [{ title, url, description }] }`, where `description`
362
+ * is the page's extracted Markdown rather than a search snippet. Only the
363
+ * opening prose is kept: a whole page in one snippet would push every other
364
+ * engine's result out of the model-facing row.
365
+ * @param payload - the response body.
366
+ * @param baseUrl - the endpoint URL, used to resolve relative links.
367
+ * @returns parsed sources in the served order; empty on drift.
368
+ */
369
+ export function parseFirecrawl(payload, baseUrl) {
370
+ let body;
371
+ try {
372
+ body = JSON.parse(payload);
373
+ }
374
+ catch {
375
+ return [];
376
+ }
377
+ if (!isRecord(body) || body.success !== true)
378
+ return [];
379
+ const rows = body.data;
380
+ if (!Array.isArray(rows))
381
+ return [];
382
+ const sources = [];
383
+ for (const row of rows) {
384
+ if (!isRecord(row))
385
+ continue;
386
+ const url = resolveHref(typeof row.url === 'string' ? row.url : undefined, baseUrl);
387
+ if (url === undefined)
388
+ continue;
389
+ const title = typeof row.title === 'string' ? row.title.trim() : '';
390
+ const description = typeof row.description === 'string' ? row.description : '';
391
+ const snippet = excerpt(description, 240);
392
+ sources.push({
393
+ url,
394
+ ...(title ? { title } : {}),
395
+ ...(snippet ? { snippet } : {}),
396
+ });
397
+ }
398
+ return sources;
399
+ }
400
+ /**
401
+ * Parse an Exa MCP `web_search_exa` answer.
402
+ *
403
+ * The MCP server replies with one SSE frame whose `text` content lists results
404
+ * as `Title: … / URL: … / Published: … / Author: … / Highlights: …` blocks
405
+ * separated by `---`. The block is the only stable shape here, so it is parsed
406
+ * line-wise; anything unexpected yields fewer rows, never an exception.
407
+ * @param payload - the raw SSE body.
408
+ * @param baseUrl - the MCP endpoint, used to resolve relative links.
409
+ * @returns parsed sources in the served order; empty on drift.
410
+ */
411
+ export function parseExaMcp(payload, baseUrl) {
412
+ if (typeof payload !== 'string' || payload.length === 0)
413
+ return [];
414
+ const texts = [];
415
+ for (const line of payload.split(/\r?\n/)) {
416
+ if (!line.startsWith('data:'))
417
+ continue;
418
+ const chunk = line.slice(5).trim();
419
+ if (chunk.length === 0 || chunk === '[DONE]')
420
+ continue;
421
+ try {
422
+ const frame = JSON.parse(chunk);
423
+ if (!isRecord(frame))
424
+ continue;
425
+ const result = frame.result;
426
+ if (!isRecord(result))
427
+ continue;
428
+ const content = result.content;
429
+ if (!Array.isArray(content))
430
+ continue;
431
+ for (const item of content) {
432
+ if (isRecord(item) && item.type === 'text' && typeof item.text === 'string')
433
+ texts.push(item.text);
434
+ }
435
+ }
436
+ catch {
437
+ // A non-JSON frame is not fatal: the remaining frames may still parse.
438
+ }
439
+ }
440
+ if (texts.length === 0)
441
+ return [];
442
+ const sources = [];
443
+ for (const block of texts.join('\n').split(/\n-{3,}\n?/u)) {
444
+ const lines = block.split(/\r?\n/);
445
+ let title = '';
446
+ let url;
447
+ const highlights = [];
448
+ let inHighlights = false;
449
+ for (const line of lines) {
450
+ if (/^title\s*:/i.test(line)) {
451
+ title = line.replace(/^title\s*:/i, '').trim();
452
+ inHighlights = false;
453
+ }
454
+ else if (/^url\s*:/i.test(line)) {
455
+ url = resolveHref(line.replace(/^url\s*:/i, '').trim(), baseUrl);
456
+ inHighlights = false;
457
+ }
458
+ else if (/^highlights\s*:/i.test(line)) {
459
+ inHighlights = true;
460
+ }
461
+ else if (/^(published|author)\s*:/i.test(line)) {
462
+ inHighlights = false;
463
+ }
464
+ else if (inHighlights) {
465
+ highlights.push(line);
466
+ }
467
+ }
468
+ if (url === undefined)
469
+ continue;
470
+ const snippet = excerpt(highlights.join(' '), 240);
471
+ sources.push({
472
+ url,
473
+ ...(title ? { title } : {}),
474
+ ...(snippet ? { snippet } : {}),
475
+ });
476
+ }
477
+ return sources;
478
+ }
479
+ /**
480
+ * Flatten a Markdown-ish page extract into one snippet line.
481
+ * @param raw - the text to flatten.
482
+ * @param max - the character ceiling.
483
+ * @returns a single trimmed line, possibly empty.
484
+ */
485
+ function excerpt(raw, max) {
486
+ const flattened = raw
487
+ .replace(/!\[[^\]]*\]\([^)]*\)/g, ' ')
488
+ .replace(/\[([^\]]*)\]\([^)]*\)/g, '$1')
489
+ .replace(/^#{1,6}\s+/gm, '')
490
+ .replace(/\s+/g, ' ')
491
+ .trim();
492
+ if (flattened.length <= max)
493
+ return flattened;
494
+ return `${flattened.slice(0, max - 1).trimEnd()}…`;
495
+ }
@@ -0,0 +1,158 @@
1
+ /**
2
+ * The model-facing `research` tool: one call that searches, opens the best
3
+ * pages, and returns an evidence pack with the citations already attached.
4
+ *
5
+ * `web_search` answers "what links exist" and `web_fetch` answers "read this
6
+ * page". The step the model keeps doing by hand is the middle one: pick the
7
+ * two or three rows that matter, fetch them without timing out, and keep track
8
+ * of which sentence came from which URL. This tool performs exactly that step:
9
+ * it runs up to three searches through the normal engine path, de-duplicates
10
+ * the rows, fetches the leading pages through the same ladder `web_fetch` uses,
11
+ * and returns one Markdown document in which every excerpt sits under its URL.
12
+ *
13
+ * Failure is part of the contract. A page that will not load is reported by
14
+ * URL and reason, the remaining pages still arrive, and a total fetch failure
15
+ * still returns the search rows — a research call that found links but could
16
+ * not open them is still useful evidence.
17
+ *
18
+ * @module @fangchenghao/dsh-web-search/research
19
+ */
20
+ import type { Source } from './types.ts';
21
+ import type { ToolContent, ToolJsonSchema } from './domain-tool.ts';
22
+ /** The name this tool prefers. */
23
+ export declare const RESEARCH_TOOL_NAME = "research";
24
+ /** The name used when another plugin already registered `research`. */
25
+ export declare const RESEARCH_TOOL_FALLBACK_NAME = "web_research";
26
+ /** Pages fetched when the call names no budget. */
27
+ export declare const RESEARCH_DEFAULT_SOURCES = 3;
28
+ /** Smallest fetch budget a call may ask for. */
29
+ export declare const RESEARCH_MIN_SOURCES = 1;
30
+ /** Largest fetch budget a call may ask for. */
31
+ export declare const RESEARCH_MAX_SOURCES = 5;
32
+ /** Distinct queries accepted in one call. */
33
+ export declare const RESEARCH_MAX_QUERIES = 3;
34
+ /** Characters kept from each fetched page. */
35
+ export declare const RESEARCH_EXCERPT_CHARS = 1200;
36
+ /** Pages fetched at the same time. */
37
+ export declare const RESEARCH_PAGE_CONCURRENCY = 3;
38
+ /** Wall-clock budget for the whole call, below the tool's host timeout. */
39
+ export declare const RESEARCH_BUDGET_MS = 45000;
40
+ /** What one search reports to the tool. */
41
+ export interface ResearchSearchOutcome {
42
+ sources: Source[];
43
+ failures: string[];
44
+ }
45
+ /** Runs one query through the plugin's normal engine path. */
46
+ export type ResearchSearcher = (query: string, limit: number, signal?: AbortSignal) => Promise<ResearchSearchOutcome>;
47
+ /** What the fetch ladder reports for one URL. */
48
+ export interface ResearchFetchedPage {
49
+ url: string;
50
+ status: number;
51
+ text: string;
52
+ truncated: boolean;
53
+ }
54
+ /** Fetches one URL through the plugin's normal ladder. */
55
+ export type ResearchFetcher = (url: string, signal?: AbortSignal) => Promise<ResearchFetchedPage>;
56
+ /** One page in the evidence pack. */
57
+ export interface ResearchPage {
58
+ url: string;
59
+ title?: string;
60
+ status: number;
61
+ chars: number;
62
+ excerpt: string;
63
+ truncated: boolean;
64
+ }
65
+ /** A URL the call found but could not turn into evidence. */
66
+ export interface ResearchFetchFailure {
67
+ url: string;
68
+ reason: string;
69
+ }
70
+ /** What the tool returns to the model. */
71
+ export interface ResearchValue {
72
+ /** The query exactly as accepted (trimmed). */
73
+ query: string;
74
+ /** Every query the call actually ran. */
75
+ queries: string[];
76
+ /** Fetched pages, in the order they were chosen. */
77
+ pages: ResearchPage[];
78
+ /** Rows that were found but not fetched, with their titles when known. */
79
+ sources: Array<{
80
+ url: string;
81
+ title?: string;
82
+ }>;
83
+ /** Fetch failures, one entry per URL. */
84
+ failures: ResearchFetchFailure[];
85
+ /** Number of search rows seen before the fetch budget cut them. */
86
+ found: number;
87
+ /** True when the search row list was cut by the fetch budget. */
88
+ truncated: boolean;
89
+ }
90
+ /** A tool definition in the registry's native shape. */
91
+ export interface ResearchDefinition {
92
+ name: string;
93
+ description: string;
94
+ parameters: ToolJsonSchema;
95
+ output: {
96
+ schema: ToolJsonSchema;
97
+ render: (args: unknown, value: ResearchValue) => ToolContent[];
98
+ };
99
+ timeoutMs?: number;
100
+ isConcurrencySafe?: (args: unknown) => boolean;
101
+ execute: (args: unknown, exec: {
102
+ signal?: AbortSignal;
103
+ }) => Promise<ResearchValue>;
104
+ }
105
+ /** The slice of the tool registry this module needs. */
106
+ export interface ResearchRegistryLike {
107
+ register(definition: ResearchDefinition): unknown;
108
+ get?(name: string, scope?: unknown): unknown;
109
+ }
110
+ /** Construction seams. */
111
+ export interface ResearchOptions {
112
+ /** Registration diagnostics. */
113
+ warn?: (message: string) => void;
114
+ }
115
+ /** A successful registration plus the exact way to undo it. */
116
+ export interface ResearchRegistration {
117
+ name: string;
118
+ dispose: () => void;
119
+ }
120
+ /**
121
+ * Canonical form used to decide whether two rows are the same page.
122
+ *
123
+ * Search engines disagree about `www.`, trailing slashes and fragments; two
124
+ * rows that differ only in those are one page, and fetching both wastes the
125
+ * whole budget.
126
+ * @param raw - candidate URL.
127
+ * @returns a stable key, or `undefined` for a non-http(s) URL.
128
+ */
129
+ export declare function canonicalKey(raw: string): string | undefined;
130
+ /**
131
+ * Format the evidence pack the model reads.
132
+ * @param value - what the call produced.
133
+ * @returns a Markdown document with one section per fetched page.
134
+ */
135
+ export declare function formatResearchValue(value: ResearchValue): string;
136
+ /**
137
+ * Build the tool definition the host registry accepts.
138
+ * @param name - `research`, or the fallback when that name is taken.
139
+ * @param search - runs one query through the plugin's search path.
140
+ * @param fetchPage - fetches one URL through the plugin's fetch ladder.
141
+ * @returns a registry-ready definition.
142
+ */
143
+ export declare function createResearchDefinition(name: string, search: ResearchSearcher, fetchPage: ResearchFetcher): ResearchDefinition;
144
+ /**
145
+ * Pick the tool name this deployment can accept.
146
+ * @param tools - the live registry.
147
+ * @returns the preferred or fallback name.
148
+ */
149
+ export declare function pickResearchToolName(tools: ResearchRegistryLike): string;
150
+ /**
151
+ * Register the tool when a registry is available.
152
+ * @param tools - the live registry.
153
+ * @param search - executes one search through the plugin's normal path.
154
+ * @param fetchPage - fetches one URL through the plugin's fetch ladder.
155
+ * @param options - diagnostics.
156
+ * @returns the registered name and its disposer, or `undefined` when skipped.
157
+ */
158
+ export declare function registerResearchTool(tools: ResearchRegistryLike, search: ResearchSearcher, fetchPage: ResearchFetcher, options?: ResearchOptions): ResearchRegistration | undefined;