@fffangchenghao/dsh-web-search 0.13.11 → 0.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +51 -0
- package/README.md +67 -25
- package/README.zh-CN.md +65 -23
- package/SECURITY.md +4 -0
- package/client.js +2 -2
- package/docs/rrf-ranking.md +147 -0
- package/lib/config.js +4 -4
- package/lib/domain-tool.js +2 -2
- package/lib/engines.js +43 -1
- package/lib/index.d.ts +3 -1
- package/lib/index.js +30 -1
- package/lib/parsers.d.ts +25 -0
- package/lib/parsers.js +139 -0
- package/lib/research.d.ts +158 -0
- package/lib/research.js +393 -0
- package/lib/verticals.d.ts +240 -3
- package/lib/verticals.js +1185 -78
- package/package.json +3 -2
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
# How the ranking works, and how it is measured
|
|
2
|
+
|
|
3
|
+
Most "web search plugins" for an agent harness are thin wrappers: call one
|
|
4
|
+
engine, return its rows. That is enough to prove the model can reach the web —
|
|
5
|
+
and it is why result quality is usually discussed as a feeling.
|
|
6
|
+
|
|
7
|
+
This plugin treats the eight rows the Host shows the model as the product. That
|
|
8
|
+
means the ranking has to be explainable, and the explanation has to survive a
|
|
9
|
+
measurement. This page is both.
|
|
10
|
+
|
|
11
|
+

|
|
12
|
+
|
|
13
|
+
## The pipeline
|
|
14
|
+
|
|
15
|
+
1. **Fan out.** Every enabled engine is queried concurrently with its own
|
|
16
|
+
deadline (`timeoutMs`, default 15 s) and its own cooldown state. An engine
|
|
17
|
+
that just failed is skipped instead of being retried on every query. The
|
|
18
|
+
primary tier answers first; the fallback tier only runs when every primary
|
|
19
|
+
came back empty.
|
|
20
|
+
2. **Normalise.** Each parser returns `{title, url, snippet, position, votes}`
|
|
21
|
+
with the engine's own result order preserved. URLs are canonicalised before
|
|
22
|
+
de-duplication so `?utm_source=…` cannot smuggle the same page in twice.
|
|
23
|
+
3. **Interleave.** Engines are merged by weighted round-robin, not by
|
|
24
|
+
concatenation: an engine with weight 2 contributes two slots per round. This
|
|
25
|
+
keeps one verbose engine from filling all eight rows before a second engine
|
|
26
|
+
is even looked at. `anysearch` carries weight 2 because its precision was
|
|
27
|
+
measured highest on both Chinese and English queries.
|
|
28
|
+
4. **Fuse.** Each engine's list is scored with reciprocal-rank fusion:
|
|
29
|
+
|
|
30
|
+
```
|
|
31
|
+
vote = 1 / (rrfConstant + position) # rrfConstant default 60
|
|
32
|
+
score = Σ votes
|
|
33
|
+
+ authorityBoost * isOfficialDomain
|
|
34
|
+
+ freshnessBoost * freshness(parsedDate, freshnessDays)
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
RRF is used instead of score normalisation because engines report scores on
|
|
38
|
+
incomparable scales (or none at all); ranks are the only signal every engine
|
|
39
|
+
provides honestly.
|
|
40
|
+
5. **Region policy.** A CJK query leads with the engines that answer Chinese
|
|
41
|
+
sources (`cn.bing`, `anysearch`); an ASCII query leads with the global set
|
|
42
|
+
(`ddg-html`, `www.bing`, `anysearch`). The other region's engines still
|
|
43
|
+
contribute, so a Chinese query can still surface an English original.
|
|
44
|
+
6. **Cut.** The fused list is truncated to `limit` (default 8, clamped 1–20).
|
|
45
|
+
Only the rows that survive are returned; the model never sees the raw
|
|
46
|
+
per-engine lists.
|
|
47
|
+
|
|
48
|
+
## Why these exact knobs
|
|
49
|
+
|
|
50
|
+
Each knob exists because a specific failure was observed:
|
|
51
|
+
|
|
52
|
+
| knob | default | what it prevents |
|
|
53
|
+
|---|---|---|
|
|
54
|
+
| `rrfConstant` | 60 | a single engine's first result dominating the fusion |
|
|
55
|
+
| `authorityBoost` | 0.25 | documentation on `kubernetes.io` losing to a SEO farm |
|
|
56
|
+
| `freshnessBoost` / `freshnessDays` | 0.2 / 30 | last year's release notes outranking this month's |
|
|
57
|
+
| region split | on | English pages burying the Chinese documentation a Chinese query asked for |
|
|
58
|
+
| engine weight | `anysearch` = 2 | a precise engine receiving the same airtime as a noisy one |
|
|
59
|
+
|
|
60
|
+
## How it is measured
|
|
61
|
+
|
|
62
|
+
`bench/` holds 20 fixed queries — 10 Chinese, 10 English, five each in docs /
|
|
63
|
+
release / package / news — each annotated with 1–3 expected domains. Encoding
|
|
64
|
+
the expectation in domains rather than exact URLs is deliberate: result titles
|
|
65
|
+
and snippets change constantly, domains are the part the ranking is supposed to
|
|
66
|
+
get right.
|
|
67
|
+
|
|
68
|
+
```
|
|
69
|
+
node scripts/bench.mjs # offline: replay recorded fixtures, no network
|
|
70
|
+
node scripts/bench.mjs --record # refresh fixtures from the live engines
|
|
71
|
+
node scripts/bench.mjs --json # machine-readable output for version diffs
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
The bench reports precision@8, recall@8 and MRR@8 overall, per language, per
|
|
75
|
+
category and per engine, plus the worst query. Offline replay makes the number
|
|
76
|
+
comparable across commits; `--record` is the only mode that touches the network.
|
|
77
|
+
|
|
78
|
+

|
|
79
|
+
|
|
80
|
+
Baseline (v0.15.0, 20 queries, k=8, tier-aware replay):
|
|
81
|
+
|
|
82
|
+
| metric | overall | Chinese | English |
|
|
83
|
+
|---|---:|---:|---:|
|
|
84
|
+
| P@8 | 0.4375 | 0.4000 | 0.4750 |
|
|
85
|
+
| R@8 | 0.7917 | 0.8167 | 0.7667 |
|
|
86
|
+
| MRR@8 | 0.8071 | 0.7976 | 0.8167 |
|
|
87
|
+
|
|
88
|
+
Replay is tier-aware: it applies the same primary-then-fallback gate the live
|
|
89
|
+
`search()` applies, instead of ranking every recorded engine as if it had
|
|
90
|
+
answered. That changed the protocol in 0.15.0, so numbers from earlier releases
|
|
91
|
+
(0.7600 MRR) are not directly comparable to these.
|
|
92
|
+
|
|
93
|
+
## Parameter sweep
|
|
94
|
+
|
|
95
|
+
`scripts/sweep.mjs` runs an 81-configuration grid over the same fixtures and
|
|
96
|
+
reports marginal means plus a paired bootstrap against the shipped defaults:
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
node scripts/sweep.mjs # full grid (~2 min, offline)
|
|
100
|
+
node scripts/sweep.mjs --quick # 27-config subset
|
|
101
|
+
node scripts/sweep.mjs --json # includes the per-config CI data
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Marginal means over the whole grid (every other knob varies with it):
|
|
105
|
+
|
|
106
|
+
| `rrfConstant` | MRR@8 | `authorityBoost` | MRR@8 | `freshnessBoost` | MRR@8 |
|
|
107
|
+
|---:|---:|---:|---:|---:|---:|
|
|
108
|
+
| 40 | 0.8339 | 0.1 | 0.8275 | 0.1 | 0.8230 |
|
|
109
|
+
| **60** | **0.8278** | **0.25** | **0.8262** | **0.2** | **0.8245** |
|
|
110
|
+
| 80 | 0.8195 | 0.4 | 0.8275 | 0.35 | 0.8336 |
|
|
111
|
+
|
|
112
|
+
The best grid point (`rrf=40, authority=0.25, freshness=0.35`) measures MRR
|
|
113
|
+
0.8583 against the default 0.8071 — a gain of +0.0512 — but its 95 %
|
|
114
|
+
paired-bootstrap interval `[0.0000, +0.1250]` touches zero, and the marginals
|
|
115
|
+
above differ from the default by less than the interval width. On a 20-query
|
|
116
|
+
corpus that is a direction, not a proven win, so the defaults did not change.
|
|
117
|
+
The sweep is kept as a script precisely so this question can be re-asked after
|
|
118
|
+
the query set grows.
|
|
119
|
+
|
|
120
|
+
The same discipline decided the new engines' tiers. On one recorded fixture
|
|
121
|
+
set, with the tier-aware replay:
|
|
122
|
+
|
|
123
|
+
| configuration | MRR@8 |
|
|
124
|
+
|---|---:|
|
|
125
|
+
| previous primaries only | 0.7738 |
|
|
126
|
+
| **previous primaries + Exa** | **0.8071** |
|
|
127
|
+
| previous primaries + Firecrawl | 0.7208 |
|
|
128
|
+
| previous primaries + both | 0.7542 |
|
|
129
|
+
|
|
130
|
+
Exa therefore leads; Firecrawl is parsed and tested but ships as the fallback,
|
|
131
|
+
available through `hostIds: ['firecrawl']` and used whenever every primary
|
|
132
|
+
returns nothing.
|
|
133
|
+
|
|
134
|
+
## Known limits
|
|
135
|
+
|
|
136
|
+
- **Relative Chinese time words.** A query like "上周发布的那版" carries its time
|
|
137
|
+
constraint in the words themselves. The freshness boost can only act on dates
|
|
138
|
+
it can parse out of snippets; when an engine's snippet omits the date, the
|
|
139
|
+
ranker cannot recover it. This is documented rather than papered over.
|
|
140
|
+
- **Replay, not live search.** The bench replays recorded rows, so it measures
|
|
141
|
+
the ranker and the fusion, not today's engine availability. `--record` is the
|
|
142
|
+
only mode that touches the network.
|
|
143
|
+
- **Snippet quality bounds ranking.** If no engine returns the expected domain
|
|
144
|
+
in its top 20, fusion cannot invent it; `bench/` makes that visible per query.
|
|
145
|
+
- **The fixture set is 20 queries.** Small enough to run on every change, too
|
|
146
|
+
small to certify a general ranking. The per-query output is kept so the number
|
|
147
|
+
cannot hide a single catastrophic query.
|
package/lib/config.js
CHANGED
|
@@ -21,7 +21,7 @@
|
|
|
21
21
|
*/
|
|
22
22
|
import z from '@deepseek-ai/schemastery';
|
|
23
23
|
import { DEFAULT_FETCH_MAX_LENGTH } from "./markdown.js";
|
|
24
|
-
import { VERTICAL_IDS, isVerticalId } from "./verticals.js";
|
|
24
|
+
import { DEFAULT_VERTICAL_IDS, VERTICAL_IDS, isVerticalId } from "./verticals.js";
|
|
25
25
|
/** Schema of the plugin's configuration section. */
|
|
26
26
|
export const Config = z.object({
|
|
27
27
|
hosts: z
|
|
@@ -101,8 +101,8 @@ export const Config = z.object({
|
|
|
101
101
|
maxCooldownMs: z.natural().default(1800000).description('它反复被拒时,最长让它歇多久(毫秒),也就是等待一步步加长的封顶值。一般不用改;保持 1800000(30 分钟)即可。').volatile(),
|
|
102
102
|
domainSources: z
|
|
103
103
|
.union([z.array(z.string()), z.string()])
|
|
104
|
-
.default([...
|
|
105
|
-
.description('
|
|
104
|
+
.default([...DEFAULT_VERTICAL_IDS])
|
|
105
|
+
.description('默认查哪些专用源,逗号分隔;留空表示默认的 8 个(github、npm、pypi、arxiv、wikipedia、hn、stackexchange、crates)。其余源可用 sources 参数按需指定。')
|
|
106
106
|
.volatile(),
|
|
107
107
|
domainLimit: z.natural().min(1).max(20).default(5).description('domain_search 一次最多返回几条(1-20);默认 5 条,想多看可调到 10。').volatile(),
|
|
108
108
|
domainTimeoutMs: z.natural().min(500).default(8000).description('每个专用源最多等多久(毫秒);网络慢可加到 15000,平时 8000 即可。').volatile(),
|
|
@@ -167,7 +167,7 @@ export function resolveConfig(config, callLimit) {
|
|
|
167
167
|
timeRange: settings.timeRange,
|
|
168
168
|
anysearchApiKey: settings.anysearchApiKey.trim(),
|
|
169
169
|
anysearchDomain: settings.anysearchDomain.trim(),
|
|
170
|
-
domainSourceIds: domainSources.length > 0 ? domainSources : [...
|
|
170
|
+
domainSourceIds: domainSources.length > 0 ? domainSources : [...DEFAULT_VERTICAL_IDS],
|
|
171
171
|
domainLimit: settings.domainLimit,
|
|
172
172
|
domainTimeoutMs: settings.domainTimeoutMs,
|
|
173
173
|
rank: {
|
package/lib/domain-tool.js
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import { resolveProxy } from "./proxy.js";
|
|
2
|
-
import { searchVerticals, isVerticalId, VERTICAL_IDS } from "./verticals.js";
|
|
2
|
+
import { DEFAULT_VERTICAL_IDS, searchVerticals, isVerticalId, VERTICAL_IDS } from "./verticals.js";
|
|
3
3
|
/** The name this tool prefers. */
|
|
4
4
|
export const DOMAIN_TOOL_NAME = 'domain_search';
|
|
5
5
|
/** The name used when another plugin already registered `domain_search`. */
|
|
@@ -70,7 +70,7 @@ function parseArgs(args, settings) {
|
|
|
70
70
|
if (query.length === 0)
|
|
71
71
|
throw new Error('domain_search requires a non-empty query');
|
|
72
72
|
const enabled = settings.sourceIds.filter(isVerticalId);
|
|
73
|
-
let ids = enabled.length > 0 ? enabled : [...
|
|
73
|
+
let ids = enabled.length > 0 ? enabled : [...DEFAULT_VERTICAL_IDS];
|
|
74
74
|
if (Array.isArray(args.sources) && args.sources.length > 0) {
|
|
75
75
|
const asked = args.sources.filter(isVerticalId);
|
|
76
76
|
if (asked.length === 0) {
|
package/lib/engines.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { SearchError } from "./errors.js";
|
|
2
2
|
import { filterByTimeRange, preferWindow } from "./freshness.js";
|
|
3
3
|
import { resolveProxy } from "./proxy.js";
|
|
4
|
-
import { parseAnySearch, parseBing, parseDdgHtml, parseDdgLite, parseSo } from "./parsers.js";
|
|
4
|
+
import { parseAnySearch, parseBing, parseDdgHtml, parseDdgLite, parseExaMcp, parseFirecrawl, parseSo } from "./parsers.js";
|
|
5
5
|
import { mergeRegions } from "./rank.js";
|
|
6
6
|
import { cacheKey } from "./cache.js";
|
|
7
7
|
import { searchHost } from "./transport.js";
|
|
@@ -35,6 +35,48 @@ export const HOSTS = [
|
|
|
35
35
|
body: (query) => JSON.stringify({ query, max_results: ANYSEARCH_MAX_RESULTS }),
|
|
36
36
|
parse: parseAnySearch,
|
|
37
37
|
},
|
|
38
|
+
{
|
|
39
|
+
id: 'exa',
|
|
40
|
+
// A semantic (embedding) index rather than a keyword page: it answers with
|
|
41
|
+
// the page itself plus highlights. The MCP endpoint accepts anonymous
|
|
42
|
+
// calls, so it costs no key. Measured on the same recorded fixtures with the
|
|
43
|
+
// tier-aware replay, MRR@8 0.8071 with Exa in the primary tier versus
|
|
44
|
+
// 0.7738 without it, and it removes the one zero-hit query.
|
|
45
|
+
tier: 'primary',
|
|
46
|
+
region: 'global',
|
|
47
|
+
weight: 1,
|
|
48
|
+
viaProxy: true,
|
|
49
|
+
method: 'POST',
|
|
50
|
+
skipEcho: true,
|
|
51
|
+
headers: { accept: 'application/json, text/event-stream' },
|
|
52
|
+
build: () => 'https://mcp.exa.ai/mcp',
|
|
53
|
+
body: (query) => JSON.stringify({
|
|
54
|
+
jsonrpc: '2.0',
|
|
55
|
+
id: 1,
|
|
56
|
+
method: 'tools/call',
|
|
57
|
+
params: { name: 'web_search_exa', arguments: { query, numResults: 10 } },
|
|
58
|
+
}),
|
|
59
|
+
parse: parseExaMcp,
|
|
60
|
+
},
|
|
61
|
+
{
|
|
62
|
+
id: 'firecrawl',
|
|
63
|
+
// The keyless tier of the Firecrawl search API. It returns extracted page
|
|
64
|
+
// Markdown rather than snippets, so only the opening prose is kept as the
|
|
65
|
+
// row's snippet; it handled a Chinese query with the same shape. It ships as
|
|
66
|
+
// a fallback because it diluted the fusion when it led: MRR@8 0.7208 with
|
|
67
|
+
// Firecrawl in the primary tier versus 0.8071 with Exa there instead, on the
|
|
68
|
+
// same fixtures (standalone 0.4625). `hostIds: ['firecrawl']` still forces
|
|
69
|
+
// it, and it answers whenever every primary came back empty.
|
|
70
|
+
tier: 'fallback',
|
|
71
|
+
region: 'both',
|
|
72
|
+
weight: 1,
|
|
73
|
+
viaProxy: true,
|
|
74
|
+
method: 'POST',
|
|
75
|
+
skipEcho: true,
|
|
76
|
+
build: () => 'https://api.firecrawl.dev/v1/search',
|
|
77
|
+
body: (query) => JSON.stringify({ query, limit: 10 }),
|
|
78
|
+
parse: parseFirecrawl,
|
|
79
|
+
},
|
|
38
80
|
{
|
|
39
81
|
id: 'ddg-html',
|
|
40
82
|
tier: 'primary',
|
package/lib/index.d.ts
CHANGED
|
@@ -138,12 +138,14 @@ export { resolveHref, parseDdgHtml, parseDdgLite, parseBing, parseSo, parseAnySe
|
|
|
138
138
|
export { fuse, interleave, dedupe, mergeRegions } from './rank.ts';
|
|
139
139
|
export { MS_PER_DAY, TIME_RANGE_MS, TIME_RANGES, parsePublishedAt, extractPublishedAt, publishedTimestamp, isWithinTimeRange, filterByTimeRange, } from './freshness.ts';
|
|
140
140
|
export { parseProxy, proxyFromEnv, resolveProxy, proxyGet } from './proxy.ts';
|
|
141
|
-
export { VERTICAL_IDS, VERTICALS, searchVerticals, verticalById, isVerticalId, parseGithub, parseNpm, parsePypi, parseArxiv, parseWikipedia, parseHn, parseStackExchange, parseCrates, } from './verticals.ts';
|
|
141
|
+
export { DEFAULT_VERTICAL_IDS, VERTICAL_IDS, VERTICALS, searchVerticals, verticalById, isVerticalId, parseGithub, parseNpm, parsePypi, parseArxiv, parseWikipedia, parseHn, parseStackExchange, parseCrates, } from './verticals.ts';
|
|
142
142
|
export { DOMAIN_TOOL_NAME, DOMAIN_TOOL_FALLBACK_NAME, createDomainToolDefinition, registerDomainTool, formatDomainValue, } from './domain-tool.ts';
|
|
143
143
|
export type { VerticalId, VerticalSpec, VerticalOutcome, VerticalSearchOptions } from './verticals.ts';
|
|
144
144
|
export type { DomainToolSettings, DomainToolValue, DomainToolDefinition, ToolRegistryLike } from './domain-tool.ts';
|
|
145
145
|
export { SEARCH_MANY_TOOL_NAME, SEARCH_MANY_FALLBACK_NAME, SEARCH_MANY_CONCURRENCY, SEARCH_MANY_DEFAULT_LIMIT, createSearchManyDefinition, registerSearchManyTool, pickSearchManyToolName, formatSearchManyValue, } from './search-many.ts';
|
|
146
146
|
export type { SearchManyDefinition, SearchManyOutcome, SearchManyRegistration, SearchManyRegistryLike, SearchManyResult, SearchManyRunner, SearchManySource, SearchManyValue, } from './search-many.ts';
|
|
147
|
+
export { RESEARCH_TOOL_NAME, RESEARCH_TOOL_FALLBACK_NAME, RESEARCH_DEFAULT_SOURCES, RESEARCH_MAX_SOURCES, RESEARCH_MAX_QUERIES, RESEARCH_EXCERPT_CHARS, RESEARCH_BUDGET_MS, createResearchDefinition, registerResearchTool, pickResearchToolName, formatResearchValue, } from './research.ts';
|
|
148
|
+
export type { ResearchDefinition, ResearchFetcher, ResearchFetchedPage, ResearchFetchFailure, ResearchPage, ResearchRegistration, ResearchRegistryLike, ResearchSearcher, ResearchSearchOutcome, ResearchValue, } from './research.ts';
|
|
147
149
|
export { READ_PAGE_TOOL_NAME, READ_PAGE_TOOL_FALLBACK_NAME, READ_PAGE_MIN_LENGTH, READ_PAGE_MAX_LENGTH, READ_PAGE_DEFAULT_LENGTH, READ_PAGE_INTERNAL_MAX_CHARS, READ_PAGE_CACHE_ENTRIES, READ_PAGE_PROVIDER_ID, createReadPageDefinition, registerReadPageTool, pickReadPageToolName, formatReadPageValue, } from './read-page.ts';
|
|
148
150
|
export type { ReadPageDefinition, ReadPageOptions, ReadPageRegistration, ReadPageRegistryLike, ReadPageSettings, ReadPageValue, } from './read-page.ts';
|
|
149
151
|
export type { Source, SearchOutcome, RankOptions, SearchOptions, HostSpec, Region, TimeRange, WaitPolicy } from './types.ts';
|
package/lib/index.js
CHANGED
|
@@ -17,6 +17,7 @@ import { createFetchProvider } from "./fetch.js";
|
|
|
17
17
|
import { registerDomainTool } from "./domain-tool.js";
|
|
18
18
|
import { registerSearchManyTool } from "./search-many.js";
|
|
19
19
|
import { registerReadPageTool } from "./read-page.js";
|
|
20
|
+
import { registerResearchTool } from "./research.js";
|
|
20
21
|
export const name = 'dsh-web-search';
|
|
21
22
|
/** The `ctx.web` service this plugin registers a search provider with. */
|
|
22
23
|
export const inject = ['web'];
|
|
@@ -214,6 +215,33 @@ export function apply(ctx, config = {}) {
|
|
|
214
215
|
if (paged !== undefined) {
|
|
215
216
|
onDispose(ctx, paged.dispose);
|
|
216
217
|
}
|
|
218
|
+
// Research: search -> fetch -> cite in one call. The fetch side reuses the
|
|
219
|
+
// same ladder as `web_fetch`, but through its own provider: the renderer
|
|
220
|
+
// load is cached per provider, and a research pack must not replace the
|
|
221
|
+
// page the host already fetched for the user.
|
|
222
|
+
const researchProvider = createFetchProvider({
|
|
223
|
+
id: 'http-proxy-research',
|
|
224
|
+
timeoutMs: mounted.timeoutMs,
|
|
225
|
+
format: 'markdown',
|
|
226
|
+
maxLength: 6_000,
|
|
227
|
+
renderJs: () => resolveConfig(section, undefined).renderJs,
|
|
228
|
+
renderTimeoutMs: () => resolveConfig(section, undefined).renderTimeoutMs,
|
|
229
|
+
...(mounted.proxy === undefined ? {} : { proxy: mounted.proxy }),
|
|
230
|
+
...(mounted.anysearchApiKey.length === 0 ? {} : { anysearchApiKey: mounted.anysearchApiKey }),
|
|
231
|
+
});
|
|
232
|
+
const researchFetch = async (url, signal) => {
|
|
233
|
+
const outcome = await researchProvider.fetch({ url }, signal);
|
|
234
|
+
return {
|
|
235
|
+
url: outcome.url,
|
|
236
|
+
status: outcome.statusCode,
|
|
237
|
+
text: outcome.body.kind === 'text' ? outcome.body.content : '',
|
|
238
|
+
truncated: outcome.truncated,
|
|
239
|
+
};
|
|
240
|
+
};
|
|
241
|
+
const research = registerResearchTool(tools, (query, limit, signal) => runQuery(query, limit, signal), researchFetch, { warn: warnTool });
|
|
242
|
+
if (research !== undefined) {
|
|
243
|
+
onDispose(ctx, research.dispose);
|
|
244
|
+
}
|
|
217
245
|
}
|
|
218
246
|
}
|
|
219
247
|
catch (error) {
|
|
@@ -244,7 +272,8 @@ export { resolveHref, parseDdgHtml, parseDdgLite, parseBing, parseSo, parseAnySe
|
|
|
244
272
|
export { fuse, interleave, dedupe, mergeRegions } from "./rank.js";
|
|
245
273
|
export { MS_PER_DAY, TIME_RANGE_MS, TIME_RANGES, parsePublishedAt, extractPublishedAt, publishedTimestamp, isWithinTimeRange, filterByTimeRange, } from "./freshness.js";
|
|
246
274
|
export { parseProxy, proxyFromEnv, resolveProxy, proxyGet } from "./proxy.js";
|
|
247
|
-
export { VERTICAL_IDS, VERTICALS, searchVerticals, verticalById, isVerticalId, parseGithub, parseNpm, parsePypi, parseArxiv, parseWikipedia, parseHn, parseStackExchange, parseCrates, } from "./verticals.js";
|
|
275
|
+
export { DEFAULT_VERTICAL_IDS, VERTICAL_IDS, VERTICALS, searchVerticals, verticalById, isVerticalId, parseGithub, parseNpm, parsePypi, parseArxiv, parseWikipedia, parseHn, parseStackExchange, parseCrates, } from "./verticals.js";
|
|
248
276
|
export { DOMAIN_TOOL_NAME, DOMAIN_TOOL_FALLBACK_NAME, createDomainToolDefinition, registerDomainTool, formatDomainValue, } from "./domain-tool.js";
|
|
249
277
|
export { SEARCH_MANY_TOOL_NAME, SEARCH_MANY_FALLBACK_NAME, SEARCH_MANY_CONCURRENCY, SEARCH_MANY_DEFAULT_LIMIT, createSearchManyDefinition, registerSearchManyTool, pickSearchManyToolName, formatSearchManyValue, } from "./search-many.js";
|
|
278
|
+
export { RESEARCH_TOOL_NAME, RESEARCH_TOOL_FALLBACK_NAME, RESEARCH_DEFAULT_SOURCES, RESEARCH_MAX_SOURCES, RESEARCH_MAX_QUERIES, RESEARCH_EXCERPT_CHARS, RESEARCH_BUDGET_MS, createResearchDefinition, registerResearchTool, pickResearchToolName, formatResearchValue, } from "./research.js";
|
|
250
279
|
export { READ_PAGE_TOOL_NAME, READ_PAGE_TOOL_FALLBACK_NAME, READ_PAGE_MIN_LENGTH, READ_PAGE_MAX_LENGTH, READ_PAGE_DEFAULT_LENGTH, READ_PAGE_INTERNAL_MAX_CHARS, READ_PAGE_CACHE_ENTRIES, READ_PAGE_PROVIDER_ID, createReadPageDefinition, registerReadPageTool, pickReadPageToolName, formatReadPageValue, } from "./read-page.js";
|
package/lib/parsers.d.ts
CHANGED
|
@@ -90,3 +90,28 @@ export declare function parseAnySearch(payload: string, baseUrl: string): Source
|
|
|
90
90
|
* @returns whether the payload contains a distinctive query token.
|
|
91
91
|
*/
|
|
92
92
|
export declare function echoesQuery(payload: string, query: string): boolean;
|
|
93
|
+
/**
|
|
94
|
+
* Parse the Firecrawl `/v1/search` JSON API.
|
|
95
|
+
*
|
|
96
|
+
* The endpoint is credential-free and answers
|
|
97
|
+
* `{ success: true, data: [{ title, url, description }] }`, where `description`
|
|
98
|
+
* is the page's extracted Markdown rather than a search snippet. Only the
|
|
99
|
+
* opening prose is kept: a whole page in one snippet would push every other
|
|
100
|
+
* engine's result out of the model-facing row.
|
|
101
|
+
* @param payload - the response body.
|
|
102
|
+
* @param baseUrl - the endpoint URL, used to resolve relative links.
|
|
103
|
+
* @returns parsed sources in the served order; empty on drift.
|
|
104
|
+
*/
|
|
105
|
+
export declare function parseFirecrawl(payload: string, baseUrl: string): Source[];
|
|
106
|
+
/**
|
|
107
|
+
* Parse an Exa MCP `web_search_exa` answer.
|
|
108
|
+
*
|
|
109
|
+
* The MCP server replies with one SSE frame whose `text` content lists results
|
|
110
|
+
* as `Title: … / URL: … / Published: … / Author: … / Highlights: …` blocks
|
|
111
|
+
* separated by `---`. The block is the only stable shape here, so it is parsed
|
|
112
|
+
* line-wise; anything unexpected yields fewer rows, never an exception.
|
|
113
|
+
* @param payload - the raw SSE body.
|
|
114
|
+
* @param baseUrl - the MCP endpoint, used to resolve relative links.
|
|
115
|
+
* @returns parsed sources in the served order; empty on drift.
|
|
116
|
+
*/
|
|
117
|
+
export declare function parseExaMcp(payload: string, baseUrl: string): Source[];
|
package/lib/parsers.js
CHANGED
|
@@ -354,3 +354,142 @@ export function echoesQuery(payload, query) {
|
|
|
354
354
|
const lower = payload.toLowerCase();
|
|
355
355
|
return tokens.some((token) => lower.includes(token));
|
|
356
356
|
}
|
|
357
|
+
/**
|
|
358
|
+
* Parse the Firecrawl `/v1/search` JSON API.
|
|
359
|
+
*
|
|
360
|
+
* The endpoint is credential-free and answers
|
|
361
|
+
* `{ success: true, data: [{ title, url, description }] }`, where `description`
|
|
362
|
+
* is the page's extracted Markdown rather than a search snippet. Only the
|
|
363
|
+
* opening prose is kept: a whole page in one snippet would push every other
|
|
364
|
+
* engine's result out of the model-facing row.
|
|
365
|
+
* @param payload - the response body.
|
|
366
|
+
* @param baseUrl - the endpoint URL, used to resolve relative links.
|
|
367
|
+
* @returns parsed sources in the served order; empty on drift.
|
|
368
|
+
*/
|
|
369
|
+
export function parseFirecrawl(payload, baseUrl) {
|
|
370
|
+
let body;
|
|
371
|
+
try {
|
|
372
|
+
body = JSON.parse(payload);
|
|
373
|
+
}
|
|
374
|
+
catch {
|
|
375
|
+
return [];
|
|
376
|
+
}
|
|
377
|
+
if (!isRecord(body) || body.success !== true)
|
|
378
|
+
return [];
|
|
379
|
+
const rows = body.data;
|
|
380
|
+
if (!Array.isArray(rows))
|
|
381
|
+
return [];
|
|
382
|
+
const sources = [];
|
|
383
|
+
for (const row of rows) {
|
|
384
|
+
if (!isRecord(row))
|
|
385
|
+
continue;
|
|
386
|
+
const url = resolveHref(typeof row.url === 'string' ? row.url : undefined, baseUrl);
|
|
387
|
+
if (url === undefined)
|
|
388
|
+
continue;
|
|
389
|
+
const title = typeof row.title === 'string' ? row.title.trim() : '';
|
|
390
|
+
const description = typeof row.description === 'string' ? row.description : '';
|
|
391
|
+
const snippet = excerpt(description, 240);
|
|
392
|
+
sources.push({
|
|
393
|
+
url,
|
|
394
|
+
...(title ? { title } : {}),
|
|
395
|
+
...(snippet ? { snippet } : {}),
|
|
396
|
+
});
|
|
397
|
+
}
|
|
398
|
+
return sources;
|
|
399
|
+
}
|
|
400
|
+
/**
|
|
401
|
+
* Parse an Exa MCP `web_search_exa` answer.
|
|
402
|
+
*
|
|
403
|
+
* The MCP server replies with one SSE frame whose `text` content lists results
|
|
404
|
+
* as `Title: … / URL: … / Published: … / Author: … / Highlights: …` blocks
|
|
405
|
+
* separated by `---`. The block is the only stable shape here, so it is parsed
|
|
406
|
+
* line-wise; anything unexpected yields fewer rows, never an exception.
|
|
407
|
+
* @param payload - the raw SSE body.
|
|
408
|
+
* @param baseUrl - the MCP endpoint, used to resolve relative links.
|
|
409
|
+
* @returns parsed sources in the served order; empty on drift.
|
|
410
|
+
*/
|
|
411
|
+
export function parseExaMcp(payload, baseUrl) {
|
|
412
|
+
if (typeof payload !== 'string' || payload.length === 0)
|
|
413
|
+
return [];
|
|
414
|
+
const texts = [];
|
|
415
|
+
for (const line of payload.split(/\r?\n/)) {
|
|
416
|
+
if (!line.startsWith('data:'))
|
|
417
|
+
continue;
|
|
418
|
+
const chunk = line.slice(5).trim();
|
|
419
|
+
if (chunk.length === 0 || chunk === '[DONE]')
|
|
420
|
+
continue;
|
|
421
|
+
try {
|
|
422
|
+
const frame = JSON.parse(chunk);
|
|
423
|
+
if (!isRecord(frame))
|
|
424
|
+
continue;
|
|
425
|
+
const result = frame.result;
|
|
426
|
+
if (!isRecord(result))
|
|
427
|
+
continue;
|
|
428
|
+
const content = result.content;
|
|
429
|
+
if (!Array.isArray(content))
|
|
430
|
+
continue;
|
|
431
|
+
for (const item of content) {
|
|
432
|
+
if (isRecord(item) && item.type === 'text' && typeof item.text === 'string')
|
|
433
|
+
texts.push(item.text);
|
|
434
|
+
}
|
|
435
|
+
}
|
|
436
|
+
catch {
|
|
437
|
+
// A non-JSON frame is not fatal: the remaining frames may still parse.
|
|
438
|
+
}
|
|
439
|
+
}
|
|
440
|
+
if (texts.length === 0)
|
|
441
|
+
return [];
|
|
442
|
+
const sources = [];
|
|
443
|
+
for (const block of texts.join('\n').split(/\n-{3,}\n?/u)) {
|
|
444
|
+
const lines = block.split(/\r?\n/);
|
|
445
|
+
let title = '';
|
|
446
|
+
let url;
|
|
447
|
+
const highlights = [];
|
|
448
|
+
let inHighlights = false;
|
|
449
|
+
for (const line of lines) {
|
|
450
|
+
if (/^title\s*:/i.test(line)) {
|
|
451
|
+
title = line.replace(/^title\s*:/i, '').trim();
|
|
452
|
+
inHighlights = false;
|
|
453
|
+
}
|
|
454
|
+
else if (/^url\s*:/i.test(line)) {
|
|
455
|
+
url = resolveHref(line.replace(/^url\s*:/i, '').trim(), baseUrl);
|
|
456
|
+
inHighlights = false;
|
|
457
|
+
}
|
|
458
|
+
else if (/^highlights\s*:/i.test(line)) {
|
|
459
|
+
inHighlights = true;
|
|
460
|
+
}
|
|
461
|
+
else if (/^(published|author)\s*:/i.test(line)) {
|
|
462
|
+
inHighlights = false;
|
|
463
|
+
}
|
|
464
|
+
else if (inHighlights) {
|
|
465
|
+
highlights.push(line);
|
|
466
|
+
}
|
|
467
|
+
}
|
|
468
|
+
if (url === undefined)
|
|
469
|
+
continue;
|
|
470
|
+
const snippet = excerpt(highlights.join(' '), 240);
|
|
471
|
+
sources.push({
|
|
472
|
+
url,
|
|
473
|
+
...(title ? { title } : {}),
|
|
474
|
+
...(snippet ? { snippet } : {}),
|
|
475
|
+
});
|
|
476
|
+
}
|
|
477
|
+
return sources;
|
|
478
|
+
}
|
|
479
|
+
/**
|
|
480
|
+
* Flatten a Markdown-ish page extract into one snippet line.
|
|
481
|
+
* @param raw - the text to flatten.
|
|
482
|
+
* @param max - the character ceiling.
|
|
483
|
+
* @returns a single trimmed line, possibly empty.
|
|
484
|
+
*/
|
|
485
|
+
function excerpt(raw, max) {
|
|
486
|
+
const flattened = raw
|
|
487
|
+
.replace(/!\[[^\]]*\]\([^)]*\)/g, ' ')
|
|
488
|
+
.replace(/\[([^\]]*)\]\([^)]*\)/g, '$1')
|
|
489
|
+
.replace(/^#{1,6}\s+/gm, '')
|
|
490
|
+
.replace(/\s+/g, ' ')
|
|
491
|
+
.trim();
|
|
492
|
+
if (flattened.length <= max)
|
|
493
|
+
return flattened;
|
|
494
|
+
return `${flattened.slice(0, max - 1).trimEnd()}…`;
|
|
495
|
+
}
|
|
@@ -0,0 +1,158 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The model-facing `research` tool: one call that searches, opens the best
|
|
3
|
+
* pages, and returns an evidence pack with the citations already attached.
|
|
4
|
+
*
|
|
5
|
+
* `web_search` answers "what links exist" and `web_fetch` answers "read this
|
|
6
|
+
* page". The step the model keeps doing by hand is the middle one: pick the
|
|
7
|
+
* two or three rows that matter, fetch them without timing out, and keep track
|
|
8
|
+
* of which sentence came from which URL. This tool performs exactly that step:
|
|
9
|
+
* it runs up to three searches through the normal engine path, de-duplicates
|
|
10
|
+
* the rows, fetches the leading pages through the same ladder `web_fetch` uses,
|
|
11
|
+
* and returns one Markdown document in which every excerpt sits under its URL.
|
|
12
|
+
*
|
|
13
|
+
* Failure is part of the contract. A page that will not load is reported by
|
|
14
|
+
* URL and reason, the remaining pages still arrive, and a total fetch failure
|
|
15
|
+
* still returns the search rows — a research call that found links but could
|
|
16
|
+
* not open them is still useful evidence.
|
|
17
|
+
*
|
|
18
|
+
* @module @fangchenghao/dsh-web-search/research
|
|
19
|
+
*/
|
|
20
|
+
import type { Source } from './types.ts';
|
|
21
|
+
import type { ToolContent, ToolJsonSchema } from './domain-tool.ts';
|
|
22
|
+
/** The name this tool prefers. */
|
|
23
|
+
export declare const RESEARCH_TOOL_NAME = "research";
|
|
24
|
+
/** The name used when another plugin already registered `research`. */
|
|
25
|
+
export declare const RESEARCH_TOOL_FALLBACK_NAME = "web_research";
|
|
26
|
+
/** Pages fetched when the call names no budget. */
|
|
27
|
+
export declare const RESEARCH_DEFAULT_SOURCES = 3;
|
|
28
|
+
/** Smallest fetch budget a call may ask for. */
|
|
29
|
+
export declare const RESEARCH_MIN_SOURCES = 1;
|
|
30
|
+
/** Largest fetch budget a call may ask for. */
|
|
31
|
+
export declare const RESEARCH_MAX_SOURCES = 5;
|
|
32
|
+
/** Distinct queries accepted in one call. */
|
|
33
|
+
export declare const RESEARCH_MAX_QUERIES = 3;
|
|
34
|
+
/** Characters kept from each fetched page. */
|
|
35
|
+
export declare const RESEARCH_EXCERPT_CHARS = 1200;
|
|
36
|
+
/** Pages fetched at the same time. */
|
|
37
|
+
export declare const RESEARCH_PAGE_CONCURRENCY = 3;
|
|
38
|
+
/** Wall-clock budget for the whole call, below the tool's host timeout. */
|
|
39
|
+
export declare const RESEARCH_BUDGET_MS = 45000;
|
|
40
|
+
/** What one search reports to the tool. */
|
|
41
|
+
export interface ResearchSearchOutcome {
|
|
42
|
+
sources: Source[];
|
|
43
|
+
failures: string[];
|
|
44
|
+
}
|
|
45
|
+
/** Runs one query through the plugin's normal engine path. */
|
|
46
|
+
export type ResearchSearcher = (query: string, limit: number, signal?: AbortSignal) => Promise<ResearchSearchOutcome>;
|
|
47
|
+
/** What the fetch ladder reports for one URL. */
|
|
48
|
+
export interface ResearchFetchedPage {
|
|
49
|
+
url: string;
|
|
50
|
+
status: number;
|
|
51
|
+
text: string;
|
|
52
|
+
truncated: boolean;
|
|
53
|
+
}
|
|
54
|
+
/** Fetches one URL through the plugin's normal ladder. */
|
|
55
|
+
export type ResearchFetcher = (url: string, signal?: AbortSignal) => Promise<ResearchFetchedPage>;
|
|
56
|
+
/** One page in the evidence pack. */
|
|
57
|
+
export interface ResearchPage {
|
|
58
|
+
url: string;
|
|
59
|
+
title?: string;
|
|
60
|
+
status: number;
|
|
61
|
+
chars: number;
|
|
62
|
+
excerpt: string;
|
|
63
|
+
truncated: boolean;
|
|
64
|
+
}
|
|
65
|
+
/** A URL the call found but could not turn into evidence. */
|
|
66
|
+
export interface ResearchFetchFailure {
|
|
67
|
+
url: string;
|
|
68
|
+
reason: string;
|
|
69
|
+
}
|
|
70
|
+
/** What the tool returns to the model. */
|
|
71
|
+
export interface ResearchValue {
|
|
72
|
+
/** The query exactly as accepted (trimmed). */
|
|
73
|
+
query: string;
|
|
74
|
+
/** Every query the call actually ran. */
|
|
75
|
+
queries: string[];
|
|
76
|
+
/** Fetched pages, in the order they were chosen. */
|
|
77
|
+
pages: ResearchPage[];
|
|
78
|
+
/** Rows that were found but not fetched, with their titles when known. */
|
|
79
|
+
sources: Array<{
|
|
80
|
+
url: string;
|
|
81
|
+
title?: string;
|
|
82
|
+
}>;
|
|
83
|
+
/** Fetch failures, one entry per URL. */
|
|
84
|
+
failures: ResearchFetchFailure[];
|
|
85
|
+
/** Number of search rows seen before the fetch budget cut them. */
|
|
86
|
+
found: number;
|
|
87
|
+
/** True when the search row list was cut by the fetch budget. */
|
|
88
|
+
truncated: boolean;
|
|
89
|
+
}
|
|
90
|
+
/** A tool definition in the registry's native shape. */
|
|
91
|
+
export interface ResearchDefinition {
|
|
92
|
+
name: string;
|
|
93
|
+
description: string;
|
|
94
|
+
parameters: ToolJsonSchema;
|
|
95
|
+
output: {
|
|
96
|
+
schema: ToolJsonSchema;
|
|
97
|
+
render: (args: unknown, value: ResearchValue) => ToolContent[];
|
|
98
|
+
};
|
|
99
|
+
timeoutMs?: number;
|
|
100
|
+
isConcurrencySafe?: (args: unknown) => boolean;
|
|
101
|
+
execute: (args: unknown, exec: {
|
|
102
|
+
signal?: AbortSignal;
|
|
103
|
+
}) => Promise<ResearchValue>;
|
|
104
|
+
}
|
|
105
|
+
/** The slice of the tool registry this module needs. */
|
|
106
|
+
export interface ResearchRegistryLike {
|
|
107
|
+
register(definition: ResearchDefinition): unknown;
|
|
108
|
+
get?(name: string, scope?: unknown): unknown;
|
|
109
|
+
}
|
|
110
|
+
/** Construction seams. */
|
|
111
|
+
export interface ResearchOptions {
|
|
112
|
+
/** Registration diagnostics. */
|
|
113
|
+
warn?: (message: string) => void;
|
|
114
|
+
}
|
|
115
|
+
/** A successful registration plus the exact way to undo it. */
|
|
116
|
+
export interface ResearchRegistration {
|
|
117
|
+
name: string;
|
|
118
|
+
dispose: () => void;
|
|
119
|
+
}
|
|
120
|
+
/**
|
|
121
|
+
* Canonical form used to decide whether two rows are the same page.
|
|
122
|
+
*
|
|
123
|
+
* Search engines disagree about `www.`, trailing slashes and fragments; two
|
|
124
|
+
* rows that differ only in those are one page, and fetching both wastes the
|
|
125
|
+
* whole budget.
|
|
126
|
+
* @param raw - candidate URL.
|
|
127
|
+
* @returns a stable key, or `undefined` for a non-http(s) URL.
|
|
128
|
+
*/
|
|
129
|
+
export declare function canonicalKey(raw: string): string | undefined;
|
|
130
|
+
/**
|
|
131
|
+
* Format the evidence pack the model reads.
|
|
132
|
+
* @param value - what the call produced.
|
|
133
|
+
* @returns a Markdown document with one section per fetched page.
|
|
134
|
+
*/
|
|
135
|
+
export declare function formatResearchValue(value: ResearchValue): string;
|
|
136
|
+
/**
|
|
137
|
+
* Build the tool definition the host registry accepts.
|
|
138
|
+
* @param name - `research`, or the fallback when that name is taken.
|
|
139
|
+
* @param search - runs one query through the plugin's search path.
|
|
140
|
+
* @param fetchPage - fetches one URL through the plugin's fetch ladder.
|
|
141
|
+
* @returns a registry-ready definition.
|
|
142
|
+
*/
|
|
143
|
+
export declare function createResearchDefinition(name: string, search: ResearchSearcher, fetchPage: ResearchFetcher): ResearchDefinition;
|
|
144
|
+
/**
|
|
145
|
+
* Pick the tool name this deployment can accept.
|
|
146
|
+
* @param tools - the live registry.
|
|
147
|
+
* @returns the preferred or fallback name.
|
|
148
|
+
*/
|
|
149
|
+
export declare function pickResearchToolName(tools: ResearchRegistryLike): string;
|
|
150
|
+
/**
|
|
151
|
+
* Register the tool when a registry is available.
|
|
152
|
+
* @param tools - the live registry.
|
|
153
|
+
* @param search - executes one search through the plugin's normal path.
|
|
154
|
+
* @param fetchPage - fetches one URL through the plugin's fetch ladder.
|
|
155
|
+
* @param options - diagnostics.
|
|
156
|
+
* @returns the registered name and its disposer, or `undefined` when skipped.
|
|
157
|
+
*/
|
|
158
|
+
export declare function registerResearchTool(tools: ResearchRegistryLike, search: ResearchSearcher, fetchPage: ResearchFetcher, options?: ResearchOptions): ResearchRegistration | undefined;
|