dsh-all-search 0.2.10 → 0.2.12

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -4,13 +4,15 @@
4
4
 
5
5
  # dsh-all-search
6
6
 
7
- Adds an **AnySearch** web-search provider to DeepSeek Harness, registered into `ctx.web`. AnySearch is a single MCP gateway that aggregates exa / tavily / firecrawl / context7 behind **one API key**.
7
+ Adds a web-search provider to DeepSeek Harness, registered into `ctx.web`. AnySearch is a single MCP gateway that aggregates exa / tavily / firecrawl / context7 behind **one API key**. Optional **TinyFish** search adds free news and academic results, and its full-browser fetch can render JavaScript-heavy pages.
8
8
 
9
9
  When `api_key` is set, queries go to that gateway. When it is absent, the same `anysearch` provider stays available and searches via **Firecrawl** `POST /v1/search` with no `Authorization` header (keyless). Set optional `firecrawl_api_key` to send `Authorization: Bearer` and use your own quota. NAT/CI runners that share an egress IP share that keyless daily credit pool.
10
10
 
11
11
  With `firecrawl_api_key` and an AnySearch key, developer-intent queries (repo / issue / PR / commit / skill) are answered by the **Firecrawl Developer Index** first -- a semantic artifact index over READMEs, issues, PRs, OpenAPI specs and skills -- falling back to AnySearch when it fails or returns nothing.
12
12
 
13
- > Port of [pi-all-search](https://github.com/RealAlexandreAI/pi-all-search).
13
+ With `tinyfish_api_key`, news and academic queries route to TinyFish first (`domain_type=news` / `research_paper`), then fall back to AnySearch or Firecrawl. Set `tinyfish_fetch: true` to also register TinyFish as the fetch provider for full-browser page rendering.
14
+
15
+ > Port of [pi-all-search](https://github.com/RetiredPhysicist/pi-plugins/tree/main/packages/pi-all-search).
14
16
 
15
17
  [English](README.md) · [中文](README.zh.md)
16
18
 
@@ -38,14 +40,19 @@ The provider registers as `anysearch` on `ctx.web` -- the built-in `web_search`
38
40
  | `api_key` | - | AnySearch key. When set, queries use the AnySearch gateway. |
39
41
  | `base_url` | - | MCP endpoint override |
40
42
  | `firecrawl_api_key` | - | optional Firecrawl key (Bearer quota). Also enables the Developer Index branch for repo/issue/PR/skill queries when AnySearch is configured |
43
+ | `tinyfish_api_key` | - | optional TinyFish key. Enables free news/academic search and full-browser fetch. |
44
+ | `tinyfish_fetch` | - | register TinyFish as the fetch provider (default `false`; the host usually supplies a local fetch provider) |
41
45
 
42
46
  Without an AnySearch key the provider still reports `available() = true` and searches Firecrawl keyless. Hosts that share an egress IP (NAT, CI) share those daily credits; set `firecrawl_api_key` for a private quota.
43
47
 
48
+ Search routing with a TinyFish key: developer queries → Developer Index → TinyFish (news/academic) → AnySearch → Firecrawl. Every key is optional, so any subset of providers still answers queries.
49
+
44
50
  ## Privacy
45
51
 
46
52
  - Keys live only in your config file -- never logged.
47
53
  - With `api_key`, only your query and result count go to the AnySearch gateway.
48
54
  - Without `api_key`, the query goes to Firecrawl `/v1/search`.
55
+ - With `tinyfish_api_key`, TinyFish keeps every query and fetched URL under its `/usage` endpoint. Treat it as a non-private retrieval layer.
49
56
 
50
57
  ## Development
51
58
 
package/README.zh.md CHANGED
@@ -4,11 +4,13 @@
4
4
 
5
5
  # dsh-all-search
6
6
 
7
- 给 DeepSeek Harness 加一个 **AnySearch** 搜索 provider,注册进 `ctx.web`。AnySearch 是单 MCP 网关,一把 API key 聚合 exa / tavily / firecrawl / context7 等多家搜索。
7
+ 给 DeepSeek Harness 加一个网页搜索 provider,注册进 `ctx.web`。AnySearch 是单 MCP 网关,一把 API key 聚合 exa / tavily / firecrawl / context7 等多家搜索。可选的 **TinyFish** 补充免费新闻与学术搜索,其全浏览器 Fetch 能渲染 JavaScript 重的页面。
8
8
 
9
9
  配置了 api_key 时查询走 AnySearch 网关。没有 AnySearch key 时,同一个 anysearch provider 仍可用,改走 Firecrawl POST /v1/search,不带 Authorization(keyless)。若设置可选的 firecrawl_api_key,则发送 Authorization: Bearer 并使用你自己的配额。共享出口 IP 的 NAT/CI 会共用这份无 key 日额度。
10
10
 
11
- > 由 [pi-all-search](https://github.com/RealAlexandreAI/pi-all-search) 移植。
11
+ 设置 tinyfish_api_key 后,新闻与学术查询优先走 TinyFish(`domain_type=news` / `research_paper`),失败再回落 AnySearch 或 Firecrawl。设置 tinyfish_fetch: true 会把 TinyFish 注册为 fetch provider,用于全浏览器页面渲染。
12
+
13
+ > 由 [pi-all-search](https://github.com/RetiredPhysicist/pi-plugins/tree/main/packages/pi-all-search) 移植。
12
14
 
13
15
  [English](README.md) · [中文](README.zh.md)
14
16
 
@@ -36,6 +38,8 @@ provider 以 `anysearch` 注册到 `ctx.web`,内置的 `web_search` 工具会自
36
38
  | `api_key` | – | AnySearch key。配置后查询走 AnySearch 网关。 |
37
39
  | `base_url` | – | MCP 端点覆盖 |
38
40
  | `firecrawl_api_key` | – | 可选 Firecrawl key(Bearer 配额)。配置 AnySearch 时还启用 Developer Index 分支 |
41
+ | `tinyfish_api_key` | – | 可选 TinyFish key。启用免费新闻/学术搜索与全浏览器 Fetch |
42
+ | `tinyfish_fetch` | – | 把 TinyFish 注册为 fetch provider(默认 `false`,宿主通常自带本地 fetch provider) |
39
43
 
40
44
  没有 AnySearch key 时 provider 仍 available() = true,走 Firecrawl keyless。共享出口 IP 的 NAT/CI 会共用日额度;需要独立配额时设置 firecrawl_api_key。
41
45
 
@@ -44,6 +48,7 @@ provider 以 `anysearch` 注册到 `ctx.web`,内置的 `web_search` 工具会自
44
48
  - key 只存在于你的配置文件——不写日志
45
49
  - 有 api_key 时只向 AnySearch 网关发送查询词和结果数量
46
50
  - 没有 api_key 时查询发往 Firecrawl /v1/search
51
+ - 配置 tinyfish_api_key 后,TinyFish 会在其 `/usage` 端点保留每次查询与抓取的 URL,按非私密检索层对待
47
52
 
48
53
  ## 开发
49
54
 
package/dist/index.d.ts CHANGED
@@ -1,6 +1,6 @@
1
1
  import { Context } from '@deepseek-ai/cordis';
2
2
  import z from '@deepseek-ai/schemastery';
3
- import type { WebSearchProvider, WebSearchRequest, WebSearchResult } from '@deepseek-ai/dsh-web';
3
+ import type { WebFetchProvider, WebFetchRequest, WebFetchResult, WebSearchProvider, WebSearchRequest, WebSearchResult } from '@deepseek-ai/dsh-web';
4
4
  export declare const name = "all-search";
5
5
  export declare const inject: string[];
6
6
  export interface Config {
@@ -10,10 +10,23 @@ export interface Config {
10
10
  base_url?: string;
11
11
  /** Firecrawl API key — Bearer quota for /v1/search; also enables Developer Index when AnySearch is configured. */
12
12
  firecrawl_api_key?: string;
13
+ /** TinyFish API key — enables TinyFish search and the full-browser fetch provider. */
14
+ tinyfish_api_key?: string;
15
+ /**
16
+ * Register TinyFish as the fetch provider. Off by default: the host may
17
+ * already provide a local fetch provider, and two usable fetch providers
18
+ * raise WEB_PROVIDER_AMBIGUOUS. Enable when the host has no fetch provider
19
+ * or when `fetchProvider` is pinned to `tinyfish`.
20
+ */
21
+ tinyfish_fetch?: boolean;
13
22
  }
14
23
  export declare const Config: z<Config>;
15
24
  export declare const FIRECRAWL_SEARCH_URL = "https://api.firecrawl.dev/v1/search";
25
+ export declare const TINYFISH_SEARCH_URL = "https://api.search.tinyfish.ai";
26
+ export declare const TINYFISH_FETCH_URL = "https://api.fetch.tinyfish.ai";
16
27
  export declare function isDeveloperQuery(query: string): boolean;
28
+ export declare function isNewsQuery(query: string): boolean;
29
+ export declare function isAcademicQuery(query: string): boolean;
17
30
  export declare class AnysearchProvider implements WebSearchProvider {
18
31
  private readonly apiKey;
19
32
  private readonly baseUrl;
@@ -39,4 +52,25 @@ export declare class FirecrawlDevProvider implements WebSearchProvider {
39
52
  available(): boolean;
40
53
  search(request: WebSearchRequest, signal?: AbortSignal): Promise<WebSearchResult>;
41
54
  }
55
+ /** TinyFish search — free structured results, with news and academic modes. */
56
+ export declare class TinyFishSearchProvider implements WebSearchProvider {
57
+ private readonly apiKey;
58
+ readonly id = "tinyfish";
59
+ constructor(apiKey: string);
60
+ available(): boolean;
61
+ search(request: WebSearchRequest, signal?: AbortSignal): Promise<WebSearchResult>;
62
+ }
63
+ /**
64
+ * TinyFish fetch — full-browser rendering and clean extraction.
65
+ *
66
+ * The seam's fetch result carries only url/statusCode/body, so a per-URL
67
+ * failure is surfaced as a non-2xx status with the error text as the body.
68
+ */
69
+ export declare class TinyFishFetchProvider implements WebFetchProvider {
70
+ private readonly apiKey;
71
+ readonly id = "tinyfish";
72
+ constructor(apiKey: string);
73
+ available(): boolean;
74
+ fetch(request: WebFetchRequest, signal?: AbortSignal): Promise<WebFetchResult>;
75
+ }
42
76
  export declare function apply(ctx: Context, config: Config): void;
package/dist/index.js CHANGED
@@ -1,15 +1,20 @@
1
1
  // dsh-all-search — DeepSeek Harness (Cordis) plugin.
2
2
  //
3
- // Registers a single WebSearchProvider (id anysearch) with ctx.web. AnySearch
4
- // is a MCP gateway that aggregates exa / tavily / firecrawl / context7 behind
5
- // one API key. When that key is set, queries stay on the gateway (developer-
6
- // intent queries may try the Firecrawl Developer Index first). When the
7
- // AnySearch key is absent, the same wrapper stays available and searches via
8
- // Firecrawl POST /v1/search — no Authorization header unless firecrawl_api_key
9
- // is set (then Bearer, own quota).
3
+ // Registers one WebSearchProvider (id anysearch) and, when a TinyFish key is
4
+ // configured, one WebFetchProvider (id tinyfish) with ctx.web. AnySearch is a
5
+ // MCP gateway that aggregates exa / tavily / firecrawl / context7 behind one
6
+ // API key. When that key is set, queries stay on the gateway (developer-intent
7
+ // queries may try the Firecrawl Developer Index first). When the AnySearch key
8
+ // is absent, the same wrapper stays available and searches via Firecrawl
9
+ // POST /v1/search — no Authorization header unless firecrawl_api_key is set
10
+ // (then Bearer, own quota).
10
11
  //
11
- // One registerSearchProvider call: the dsh-web seam errors on multiple usable
12
- // providers (WEB_PROVIDER_AMBIGUOUS) unless one is configured explicitly.
12
+ // The dsh-web seam errors on multiple usable providers
13
+ // (WEB_PROVIDER_AMBIGUOUS) unless one is configured explicitly, so this plugin
14
+ // registers exactly one provider per capability. Search stays on the anysearch
15
+ // wrapper; TinyFish competes inside it rather than registering a second search
16
+ // provider. Fetch registers only when a TinyFish key exists, because the host
17
+ // already supplies a local fetch provider.
13
18
  //
14
19
  // Privacy: API keys live in the plugin config only; they are never logged.
15
20
  import z from '@deepseek-ai/schemastery';
@@ -19,13 +24,25 @@ export const Config = z.object({
19
24
  api_key: z.string().description('AnySearch API key (optional — without it, Firecrawl keyless search is used)'),
20
25
  base_url: z.string().description('AnySearch MCP endpoint override (default https://api.anysearch.com/mcp)'),
21
26
  firecrawl_api_key: z.string().description('Firecrawl API key (optional) — Bearer for Firecrawl quota; also enables Developer Index when AnySearch is configured'),
27
+ tinyfish_api_key: z.string().description('TinyFish API key (optional) — enables TinyFish search and full-browser fetch'),
28
+ tinyfish_fetch: z.boolean().description('Register TinyFish as the fetch provider (default false — the host usually supplies a local fetch provider)'),
22
29
  });
23
30
  const DEFAULT_BASE_URL = 'https://api.anysearch.com/mcp';
24
31
  export const FIRECRAWL_SEARCH_URL = 'https://api.firecrawl.dev/v1/search';
32
+ export const TINYFISH_SEARCH_URL = 'https://api.search.tinyfish.ai';
33
+ export const TINYFISH_FETCH_URL = 'https://api.fetch.tinyfish.ai';
25
34
  const DEVELOPER_QUERY_RE = /\b(repo|repos|repository|repositories|github|issue|issues|pull request|pull requests|commit|commits|branch|merge|openapi|readme|skill|skills|library|framework)\b/i;
35
+ const NEWS_QUERY_RE = /\b(news|latest|today|breaking|announced|announcement|update|released|release|happened|current|recent)\b/i;
36
+ const ACADEMIC_QUERY_RE = /\b(paper|papers|research|journal|doi|arxiv|scholar|academic|study|thesis|citation|citations)\b/i;
26
37
  export function isDeveloperQuery(query) {
27
38
  return DEVELOPER_QUERY_RE.test(query);
28
39
  }
40
+ export function isNewsQuery(query) {
41
+ return NEWS_QUERY_RE.test(query);
42
+ }
43
+ export function isAcademicQuery(query) {
44
+ return ACADEMIC_QUERY_RE.test(query);
45
+ }
29
46
  export class AnysearchProvider {
30
47
  apiKey;
31
48
  baseUrl;
@@ -153,33 +170,132 @@ export class FirecrawlDevProvider {
153
170
  };
154
171
  }
155
172
  }
173
+ /** TinyFish search — free structured results, with news and academic modes. */
174
+ export class TinyFishSearchProvider {
175
+ apiKey;
176
+ id = 'tinyfish';
177
+ constructor(apiKey) {
178
+ this.apiKey = apiKey;
179
+ }
180
+ available() {
181
+ return Boolean(this.apiKey);
182
+ }
183
+ async search(request, signal) {
184
+ const params = new URLSearchParams({ query: request.query });
185
+ if (isNewsQuery(request.query))
186
+ params.set('domain_type', 'news');
187
+ else if (isAcademicQuery(request.query))
188
+ params.set('domain_type', 'research_paper');
189
+ const resp = await fetch(`${TINYFISH_SEARCH_URL}?${params.toString()}`, {
190
+ headers: { 'X-API-Key': this.apiKey },
191
+ signal,
192
+ });
193
+ if (!resp.ok)
194
+ throw new Error(`TinyFish search HTTP ${resp.status}`);
195
+ const data = (await resp.json());
196
+ const sources = (data.results ?? []).map((r) => ({
197
+ url: r.url ?? '',
198
+ title: r.title ?? '',
199
+ snippet: r.snippet ?? '',
200
+ ...(r.date ? { publishedAt: String(r.date) } : {}),
201
+ }));
202
+ const truncated = request.maxResults !== undefined && sources.length > request.maxResults;
203
+ return { sources: truncated ? sources.slice(0, request.maxResults) : sources, truncated };
204
+ }
205
+ }
206
+ /**
207
+ * TinyFish fetch — full-browser rendering and clean extraction.
208
+ *
209
+ * The seam's fetch result carries only url/statusCode/body, so a per-URL
210
+ * failure is surfaced as a non-2xx status with the error text as the body.
211
+ */
212
+ export class TinyFishFetchProvider {
213
+ apiKey;
214
+ id = 'tinyfish';
215
+ constructor(apiKey) {
216
+ this.apiKey = apiKey;
217
+ }
218
+ available() {
219
+ return Boolean(this.apiKey);
220
+ }
221
+ async fetch(request, signal) {
222
+ const resp = await fetch(TINYFISH_FETCH_URL, {
223
+ method: 'POST',
224
+ headers: { 'Content-Type': 'application/json', 'X-API-Key': this.apiKey },
225
+ signal,
226
+ body: JSON.stringify({ urls: [request.url], format: 'markdown', per_url_timeout_ms: 45000 }),
227
+ });
228
+ if (!resp.ok)
229
+ throw new Error(`TinyFish fetch HTTP ${resp.status}`);
230
+ const data = (await resp.json());
231
+ const page = data.results?.find((r) => r.text);
232
+ if (page?.text !== undefined && page.text !== null) {
233
+ return {
234
+ url: page.final_url ?? page.url,
235
+ statusCode: 200,
236
+ body: { kind: 'text', content: page.text },
237
+ truncated: false,
238
+ };
239
+ }
240
+ const failure = data.errors?.[0];
241
+ return {
242
+ url: failure?.url ?? request.url,
243
+ statusCode: 502,
244
+ body: { kind: 'text', content: failure?.error ?? 'TinyFish fetch returned no content' },
245
+ truncated: false,
246
+ };
247
+ }
248
+ }
156
249
  export function apply(ctx, config) {
157
250
  const anysearch = new AnysearchProvider(config.api_key ?? '', config.base_url ?? DEFAULT_BASE_URL);
158
251
  const firecrawl = new FirecrawlSearchProvider(config.firecrawl_api_key);
159
252
  const devIndex = config.firecrawl_api_key ? new FirecrawlDevProvider(config.firecrawl_api_key) : null;
253
+ const tinyfish = config.tinyfish_api_key ? new TinyFishSearchProvider(config.tinyfish_api_key) : null;
160
254
  // Single provider to avoid dsh-web's WEB_PROVIDER_AMBIGUOUS on multiple
161
- // usable providers. AnySearch key → gateway (Dev Index first for developer
162
- // queries). No AnySearch key → Firecrawl /v1/search (keyless unless a
163
- // firecrawl_api_key is set).
255
+ // usable providers. Priority: Dev Index (developer queries) → TinyFish
256
+ // (news/academic) → AnySearch gateway → TinyFish → Firecrawl /v1/search.
164
257
  const wrapper = {
165
258
  id: 'anysearch',
166
- available: () => Boolean(config.api_key) || firecrawl.available(),
259
+ available: () => Boolean(config.api_key) || Boolean(config.tinyfish_api_key) || firecrawl.available(),
167
260
  async search(request, signal) {
168
- if (config.api_key) {
169
- if (devIndex && isDeveloperQuery(request.query)) {
170
- try {
171
- const dev = await devIndex.search(request, signal);
172
- if (dev.sources.length > 0)
173
- return dev;
174
- }
175
- catch {
176
- // fall through to AnySearch
177
- }
261
+ if (devIndex && isDeveloperQuery(request.query)) {
262
+ try {
263
+ const dev = await devIndex.search(request, signal);
264
+ if (dev.sources.length > 0)
265
+ return dev;
266
+ }
267
+ catch {
268
+ // fall through to the next provider
269
+ }
270
+ }
271
+ if (tinyfish && (isNewsQuery(request.query) || isAcademicQuery(request.query))) {
272
+ try {
273
+ const tf = await tinyfish.search(request, signal);
274
+ if (tf.sources.length > 0)
275
+ return tf;
276
+ }
277
+ catch {
278
+ // fall through to the next provider
178
279
  }
280
+ }
281
+ if (config.api_key) {
179
282
  return anysearch.search(request, signal);
180
283
  }
284
+ if (tinyfish) {
285
+ try {
286
+ const tf = await tinyfish.search(request, signal);
287
+ if (tf.sources.length > 0)
288
+ return tf;
289
+ }
290
+ catch {
291
+ // fall through to Firecrawl
292
+ }
293
+ }
181
294
  return firecrawl.search(request, signal);
182
295
  },
183
296
  };
184
297
  ctx.web.registerSearchProvider(wrapper);
298
+ if (config.tinyfish_fetch && config.tinyfish_api_key && typeof ctx.web.registerFetchProvider === 'function') {
299
+ ctx.web.registerFetchProvider(new TinyFishFetchProvider(config.tinyfish_api_key));
300
+ }
185
301
  }
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "dsh-all-search",
3
- "version": "0.2.10",
4
- "description": "DeepSeek Harness plugin: AnySearch-backed web search provider (aggregates exa/tavily/firecrawl via one MCP gateway) registered into ctx.web.",
3
+ "version": "0.2.12",
4
+ "description": "DeepSeek Harness plugin: AnySearch-backed web search provider with optional TinyFish search and full-browser fetch, registered into ctx.web.",
5
5
  "type": "module",
6
6
  "main": "dist/index.js",
7
7
  "exports": {
@@ -45,7 +45,8 @@
45
45
  },
46
46
  "repository": {
47
47
  "type": "git",
48
- "url": "https://github.com/RealAlexandreAI/dsh-all-search"
48
+ "url": "git+https://github.com/RetiredPhysicist/dsh-plugins.git",
49
+ "directory": "packages/dsh-all-search"
49
50
  },
50
51
  "engines": {
51
52
  "node": ">=20.0.0"