@juicesharp/rpiv-web-tools 1.7.0 → 1.8.0

package/README.md

@@ -11,16 +11,18 @@
 [![npm version](https://img.shields.io/npm/v/@juicesharp/rpiv-web-tools.svg)](https://www.npmjs.com/package/@juicesharp/rpiv-web-tools)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-Let the model search the web and read pages. `rpiv-web-tools` adds `web_search` and `web_fetch` tools to [Pi Agent](https://github.com/badlogic/pi-mono), backed by the Brave Search API, plus `/web-search-config` for interactive API-key setup.
+Let the model search the web and read pages. `rpiv-web-tools` adds `web_search` and `web_fetch` tools to [Pi Agent](https://github.com/badlogic/pi-mono) with pluggable providers (Brave, Tavily, Serper, Exa, Jina, Firecrawl), plus `/web-search-config` for interactive provider selection and API-key setup.
 
-![Brave Search API key prompt](https://raw.githubusercontent.com/juicesharp/rpiv-mono/main/packages/rpiv-web-tools/docs/config.jpg)
+![Provider selection prompt](https://raw.githubusercontent.com/juicesharp/rpiv-mono/main/packages/rpiv-web-tools/docs/config.jpg)
 
 ## Features
 
-- **Brave-backed search** - 1–10 ranked results per query with title and snippet.
-- **Read any URL** - fetch http/https pages, strip HTML to text, or get the raw HTML with `raw: true`.
+- **Six pluggable providers** - Brave, Tavily, Serper, Exa, Jina, Firecrawl. Pick one as the active backend; switch any time without losing the others' keys.
+- **Per-provider fetch strategy** - Brave and Serper read the URL directly and strip HTML to text; Tavily/Exa/Jina/Firecrawl use their native extraction endpoints (markdown for Jina/Firecrawl, plain text for Tavily/Exa).
+- **Read any URL** - fetch http/https pages with HTML-to-text extraction, or get the raw response with `raw: true` (honoured by Brave/Serper; extraction providers always return their parsed text).
 - **Large-page spillover** - oversized responses truncate inline and spill the full body to a temp file the model can read on demand.
-- **Interactive setup** - `/web-search-config` writes the key to `~/.config/rpiv-web-tools/config.json` (chmod 0600); env var `BRAVE_SEARCH_API_KEY` also works.
+- **SSRF guard** - refuses loopback, RFC 1918, link-local, and cloud-metadata addresses (`localhost`, `127.0.0.0/8`, `10.0.0.0/8`, `169.254.0.0/16`, `172.16.0.0/12`, `192.168.0.0/16`, `::1`, `fc00::/7`, `fe80::/10`).
+- **Interactive setup** - `/web-search-config` lists providers (active one first, configured ones marked) and writes to `~/.config/rpiv-web-tools/config.json` (chmod 0600); per-provider env vars also work and take precedence over persisted keys.
 
 ## Install
 
@@ -32,11 +34,11 @@ Then restart your Pi session.
 
 ## Tools
 
-- **`web_search`** - query the Brave Search API and return titled snippets.
+- **`web_search`** - query the active provider's search API and return titled snippets.
   1–10 results per call.
-- **`web_fetch`** - fetch an http/https URL, strip HTML to text (or return raw
-  HTML with `raw: true`), truncate large responses with a temp-file spill for
-  the full content.
+- **`web_fetch`** - fetch an http/https URL through the active provider's content path
+  (raw HTTP+htmlToText for Brave/Serper; native extraction for Tavily/Exa/Jina/Firecrawl),
+  truncate large responses with a temp-file spill for the full content.
 
 ### Schema - `web_search`
 
@@ -54,14 +56,14 @@ Returns:
   content: [{ type: "text", text: string }], // markdown list of "**title**\n url\n snippet"
   details: {
     query: string,
-    backend: "brave",
+    backend: "brave" | "tavily" | "serper" | "exa" | "jina" | "firecrawl",
     resultCount: number,
     results?: Array<{ title: string, url: string, snippet: string }>,
   }
 }
 ```
 
-Throws when `BRAVE_SEARCH_API_KEY` is unset or the Brave API returns a non-2xx response.
+Throws when the active provider's API key is unset (e.g. `EXA_API_KEY is not set`) or the provider's API returns a non-2xx response.
 
 ### Schema - `web_fetch`
 
@@ -88,20 +90,24 @@ Returns:
 }
 ```
 
-Throws on invalid URL, non-http(s) protocol, non-2xx response, or `image/` / `video/` / `audio/` content types.
+Throws on invalid URL, non-http(s) protocol, private/loopback hostnames (SSRF guard), non-2xx response, or `image/` / `video/` / `audio/` content types. Extraction providers (Tavily/Exa/Jina/Firecrawl) additionally throw when the API returns an empty body or a vendor-level failure (e.g. Firecrawl `success: false`, Tavily `failed_results`).
 
 ## Commands
 
-- **`/web-search-config`** - set the Brave API key interactively. Writes to
-  `~/.config/rpiv-web-tools/config.json` (chmod 0600). Pass `--show` to see
-  the current (masked) key and env var status.
+- **`/web-search-config`** - pick the active provider and set its API key interactively.
+  Providers already configured show `(configured)`; the active one is listed first with a `✓`.
+  Pressing Enter on an empty input keeps the existing key for the chosen provider while
+  persisting the provider switch. Pass `--show` to see all per-provider keys (masked) and env var status.
 
-## API key resolution
+## API key resolution (per active provider)
 
 First match wins:
 
-1. `BRAVE_SEARCH_API_KEY` environment variable
-2. `apiKey` field in `~/.config/rpiv-web-tools/config.json`
+1. The active provider's environment variable: `BRAVE_SEARCH_API_KEY`, `TAVILY_API_KEY`, `SERPER_API_KEY`, `EXA_API_KEY`, `JINA_API_KEY`, or `FIRECRAWL_API_KEY`
+2. `apiKeys.<provider>` field in `~/.config/rpiv-web-tools/config.json`
+3. Legacy `apiKey` field (Brave only — auto-migrated to the new shape on next save)
+
+The active provider is `config.provider` (set by `/web-search-config`); falls back to `brave` if absent.
 
 ## Executor guidance overrides
 
@@ -109,10 +115,14 @@ Override the `promptSnippet` / `promptGuidelines` the model sees for each tool b
 
 ```json
 {
-  "apiKey": "sk-...",
+  "provider": "exa",
+  "apiKeys": {
+    "exa": "sk-...",
+    "brave": "sk-..."
+  },
   "guidance": {
     "web_search": {
-      "promptSnippet": "Search Brave for current docs and library versions",
+      "promptSnippet": "Search the web for current docs and library versions",
       "promptGuidelines": [
         "Only call web_search when training-data answers may be stale.",
         "Always include a Sources: section with markdown hyperlinks."
@@ -127,9 +137,11 @@ Override the `promptSnippet` / `promptGuidelines` the model sees for each tool b
 
 Each field is independent: omit one and the built-in default is kept. Invalid values (empty string, wrong type, empty array) silently fall back to defaults. Changes take effect on the next Pi session start.
 
-## Security note: `web_fetch` reach
+## Security note: `web_fetch` host guard
+
+`web_fetch` refuses URLs targeting loopback (`localhost`, `127.0.0.0/8`, `::1`), RFC 1918 private ranges (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`), link-local (`169.254.0.0/16`, including cloud-metadata at `169.254.169.254`), and IPv6 unique-local / link-local (`fc00::/7`, `fe80::/10`). Attempts surface as `Refusing to fetch private/loopback address: <host>`. This blocks the most common SSRF class — direct-literal targeting of internal services or cloud-metadata endpoints — without preventing legitimate public-web fetches.
 
-`web_fetch` accepts any http/https URL the model passes, including loopback (`127.0.0.1`, `::1`) and private-range addresses (RFC 1918, link-local, cloud metadata at `169.254.169.254`). This is intentional — local dev servers and intranet docs are common legitimate targets in a single-user CLI. If you run Pi in an untrusted automation context where the model could be steered toward internal services, restrict the tool at the network layer (firewall, egress proxy) or fork to add a host filter; the package ships without one.
+The guard is host-literal only; it does NOT resolve DNS or validate redirects. A public hostname that resolves to a private IP, or a public URL that 302-redirects to one, will still reach the target. For untrusted automation environments, layer an egress proxy or firewall on top.
 
 ## License
 

package/index.ts

@@ -4,13 +4,16 @@
  * Registers the `web_search` and `web_fetch` tools, plus the
  * `/web-search-config` slash command. Body lives in `web-tools.ts`.
  *
- * Config persists at ~/.config/rpiv-web-tools/config.json. Env var
- * BRAVE_SEARCH_API_KEY wins over the config file.
+ * Config persists at ~/.config/rpiv-web-tools/config.json. Per-provider env
+ * vars (e.g. BRAVE_SEARCH_API_KEY, TAVILY_API_KEY) win over the config file.
  */
 
 import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
 import { registerWebFetchTool, registerWebSearchConfigCommand, registerWebSearchTool } from "./web-tools.js";
 
+export { createSearchProvider } from "./providers/factory.js";
+
+export type { FetchResponse, SearchProvider, SearchResponse, SearchResult } from "./providers/types.js";
 export {
 	DEFAULT_WEB_FETCH_GUIDELINES,
 	DEFAULT_WEB_FETCH_SNIPPET,

package/package.json

@@ -1,13 +1,21 @@
 {
   "name": "@juicesharp/rpiv-web-tools",
-  "version": "1.7.0",
-  "description": "Pi extension. Web search and fetch for the model, backed by the Brave Search API.",
+  "version": "1.8.0",
+  "description": "Pi extension. Web search and fetch for the model with pluggable providers (Brave, Tavily, Serper, Exa, Jina, Firecrawl).",
   "keywords": [
     "pi-package",
     "pi-extension",
     "rpiv",
     "web-search",
-    "brave"
+    "search",
+    "fetch",
+    "scrape",
+    "brave",
+    "tavily",
+    "serper",
+    "exa",
+    "jina",
+    "firecrawl"
   ],
   "type": "module",
   "license": "MIT",
@@ -30,6 +38,7 @@
   "files": [
     "index.ts",
     "web-tools.ts",
+    "providers/",
     "README.md",
     "LICENSE"
   ],

package/providers/brave.ts

@@ -0,0 +1,76 @@
+import { assertTextContentType, extractBodyAsText, fetchUrlOrThrow } from "./fetch-helpers.js";
+import type { FetchResponse, SearchProvider, SearchResponse, SearchResult } from "./types.js";
+
+const BRAVE_SEARCH_API_URL = "https://api.search.brave.com/res/v1/web/search";
+export const BRAVE_API_KEY_ENV_VAR = "BRAVE_SEARCH_API_KEY";
+export const BRAVE_PROVIDER_META = {
+	name: "brave",
+	label: "Brave",
+	envVar: BRAVE_API_KEY_ENV_VAR,
+} as const;
+
+interface BraveRawResponse {
+	web?: { results?: Array<{ title?: string; url?: string; description?: string }> };
+}
+
+function normalizeBraveResults(raw: BraveRawResponse): SearchResult[] {
+	return (raw.web?.results ?? []).map((r) => ({
+		title: r.title ?? "",
+		url: r.url ?? "",
+		snippet: r.description ?? "",
+	}));
+}
+
+export class BraveProvider implements SearchProvider {
+	readonly name = BRAVE_PROVIDER_META.name;
+	readonly label = BRAVE_PROVIDER_META.label;
+	readonly envVar = BRAVE_PROVIDER_META.envVar;
+
+	constructor(private readonly apiKey: string) {}
+
+	async search(query: string, maxResults: number, signal?: AbortSignal): Promise<SearchResponse> {
+		if (!this.apiKey) {
+			throw new Error(`${this.envVar} is not set. Run /web-search-config to configure, or export the env var.`);
+		}
+
+		const url = new URL(BRAVE_SEARCH_API_URL);
+		url.searchParams.set("q", query);
+		url.searchParams.set("count", String(maxResults));
+
+		const res = await fetch(url.toString(), {
+			method: "GET",
+			headers: {
+				Accept: "application/json",
+				"Accept-Encoding": "gzip",
+				"X-Subscription-Token": this.apiKey,
+			},
+			signal,
+		});
+
+		if (!res.ok) {
+			const text = await res.text();
+			throw new Error(`${this.label} Search API error (${res.status}): ${text}`);
+		}
+
+		const raw = (await res.json()) as BraveRawResponse;
+		return { query, results: normalizeBraveResults(raw) };
+	}
+
+	// No apiKey guard: Brave's fetch() wraps the built-in HTTP+htmlToText
+	// pipeline and does not call any vendor endpoint. Adding a guard would
+	// break the "use any provider for fetch" contract.
+	async fetch(url: string, raw: boolean, signal?: AbortSignal): Promise<FetchResponse> {
+		const res = await fetchUrlOrThrow(url, signal);
+		const contentType = res.headers.get("content-type") ?? "";
+		assertTextContentType(contentType);
+
+		const { text, title } = await extractBodyAsText(res, contentType, raw);
+		const contentLengthHeader = res.headers.get("content-length");
+		return {
+			text,
+			title,
+			contentType: contentType || undefined,
+			contentLength: contentLengthHeader ? Number(contentLengthHeader) : undefined,
+		};
+	}
+}

package/providers/exa.ts

@@ -0,0 +1,106 @@
+import type { FetchResponse, SearchProvider, SearchResponse, SearchResult } from "./types.js";
+
+const EXA_API_URL = "https://api.exa.ai/search";
+const EXA_CONTENTS_API_URL = "https://api.exa.ai/contents";
+export const EXA_API_KEY_ENV_VAR = "EXA_API_KEY";
+export const EXA_PROVIDER_META = {
+	name: "exa",
+	label: "Exa",
+	envVar: EXA_API_KEY_ENV_VAR,
+} as const;
+const EXA_MAX_SNIPPET_CHARACTERS = 300;
+const EXA_MAX_FETCH_CHARACTERS = 10000;
+
+interface ExaRawResult {
+	title?: string;
+	url?: string;
+	text?: string;
+}
+
+interface ExaRawResponse {
+	results?: ExaRawResult[];
+	error?: string;
+}
+
+function normalizeExaResults(results: ExaRawResult[]): SearchResult[] {
+	return results.map((r) => ({
+		title: r.title ?? "",
+		url: r.url ?? "",
+		snippet: r.text ?? "",
+	}));
+}
+
+export class ExaProvider implements SearchProvider {
+	readonly name = EXA_PROVIDER_META.name;
+	readonly label = EXA_PROVIDER_META.label;
+	readonly envVar = EXA_PROVIDER_META.envVar;
+
+	constructor(private readonly apiKey: string) {}
+
+	async search(query: string, maxResults: number, signal?: AbortSignal): Promise<SearchResponse> {
+		if (!this.apiKey) {
+			throw new Error(`${this.envVar} is not set. Run /web-search-config to configure, or export the env var.`);
+		}
+
+		const res = await fetch(EXA_API_URL, {
+			method: "POST",
+			headers: {
+				"Content-Type": "application/json",
+				"x-api-key": this.apiKey,
+			},
+			body: JSON.stringify({
+				query,
+				numResults: maxResults,
+				contents: {
+					text: { maxCharacters: EXA_MAX_SNIPPET_CHARACTERS },
+				},
+			}),
+			signal,
+		});
+
+		if (!res.ok) {
+			const text = await res.text();
+			throw new Error(`${this.label} Search API error (${res.status}): ${text}`);
+		}
+
+		const raw = (await res.json()) as ExaRawResponse;
+		return { query, results: normalizeExaResults(raw.results ?? []) };
+	}
+
+	async fetch(url: string, _raw: boolean, signal?: AbortSignal): Promise<FetchResponse> {
+		if (!this.apiKey) {
+			throw new Error(`${this.envVar} is not set. Run /web-search-config to configure, or export the env var.`);
+		}
+
+		const res = await fetch(EXA_CONTENTS_API_URL, {
+			method: "POST",
+			headers: {
+				"Content-Type": "application/json",
+				"x-api-key": this.apiKey,
+			},
+			body: JSON.stringify({
+				ids: [url],
+				text: { maxCharacters: EXA_MAX_FETCH_CHARACTERS },
+			}),
+			signal,
+		});
+
+		if (!res.ok) {
+			const text = await res.text();
+			throw new Error(`${this.label} Fetch API error (${res.status}): ${text}`);
+		}
+
+		const data = (await res.json()) as ExaRawResponse;
+		const result = data.results?.[0];
+
+		if (!result?.text) {
+			throw new Error(`${this.label} Fetch API error: no content returned for ${url}`);
+		}
+
+		return {
+			text: result.text,
+			title: result.title || undefined,
+			contentType: "text/plain",
+		};
+	}
+}

package/providers/factory.ts

@@ -0,0 +1,26 @@
+import { BraveProvider } from "./brave.js";
+import { ExaProvider } from "./exa.js";
+import { FirecrawlProvider } from "./firecrawl.js";
+import { JinaProvider } from "./jina.js";
+import { SerperProvider } from "./serper.js";
+import { TavilyProvider } from "./tavily.js";
+import type { SearchProvider } from "./types.js";
+
+export function createSearchProvider(name: string, apiKey: string): SearchProvider {
+	switch (name) {
+		case "brave":
+			return new BraveProvider(apiKey);
+		case "tavily":
+			return new TavilyProvider(apiKey);
+		case "serper":
+			return new SerperProvider(apiKey);
+		case "exa":
+			return new ExaProvider(apiKey);
+		case "jina":
+			return new JinaProvider(apiKey);
+		case "firecrawl":
+			return new FirecrawlProvider(apiKey);
+		default:
+			throw new Error(`Unknown search provider: "${name}"`);
+	}
+}

package/providers/fetch-helpers.ts

@@ -0,0 +1,117 @@
+/**
+ * Shared fetch helpers — HTTP client, content-type guards, and HTML-to-text
+ * extraction used by providers that wrap the built-in pipeline (Brave, Serper).
+ */
+
+// ---------------------------------------------------------------------------
+// Constants
+// ---------------------------------------------------------------------------
+
+const USER_AGENT = "Mozilla/5.0 (compatible; rpiv-pi/1.0)";
+const FETCH_ACCEPT_HEADER = "text/html,application/xhtml+xml,application/xml;q=0.9,text/plain;q=0.8,*/*;q=0.5";
+const BINARY_CONTENT_TYPE_PREFIXES = ["image/", "video/", "audio/"];
+const HTML_CONTENT_TYPE_TOKEN = "text/html";
+
+// ---------------------------------------------------------------------------
+// HTML-to-text extraction
+// ---------------------------------------------------------------------------
+
+const SCRIPT_BLOCK_REGEX = /<script[\s\S]*?<\/script>/gi;
+const STYLE_BLOCK_REGEX = /<style[\s\S]*?<\/style>/gi;
+const NOSCRIPT_BLOCK_REGEX = /<noscript[\s\S]*?<\/noscript>/gi;
+const BLOCK_CLOSER_REGEX =
+	/<\/(p|div|h[1-6]|li|tr|br|blockquote|pre|section|article|header|footer|nav|details|summary)>/gi;
+const SELF_CLOSING_BR_REGEX = /<br\s*\/?>/gi;
+const ANY_REMAINING_TAG_REGEX = /<[^>]+>/g;
+const TITLE_TAG_REGEX = /<title[^>]*>([\s\S]*?)<\/title>/i;
+const NUMERIC_HTML_ENTITY_REGEX = /&#(\d+);/g;
+const HORIZONTAL_WHITESPACE_RUN = /[ \t]+/g;
+const BLANK_LINE_RUN = /\n{3,}/g;
+
+function stripNonContentBlocks(html: string): string {
+	return html.replace(SCRIPT_BLOCK_REGEX, "").replace(STYLE_BLOCK_REGEX, "").replace(NOSCRIPT_BLOCK_REGEX, "");
+}
+
+function convertBlockTagsToNewlines(text: string): string {
+	return text.replace(BLOCK_CLOSER_REGEX, "\n").replace(SELF_CLOSING_BR_REGEX, "\n");
+}
+
+function stripRemainingTags(text: string): string {
+	return text.replace(ANY_REMAINING_TAG_REGEX, " ");
+}
+
+function decodeHtmlEntities(text: string): string {
+	return text
+		.replace(/&amp;/g, "&")
+		.replace(/&lt;/g, "<")
+		.replace(/&gt;/g, ">")
+		.replace(/&quot;/g, '"')
+		.replace(/&#39;/g, "'")
+		.replace(/&nbsp;/g, " ")
+		.replace(NUMERIC_HTML_ENTITY_REGEX, (_, code) => String.fromCharCode(Number(code)));
+}
+
+function collapseWhitespace(text: string): string {
+	return text.replace(HORIZONTAL_WHITESPACE_RUN, " ").replace(BLANK_LINE_RUN, "\n\n");
+}
+
+export function htmlToText(html: string): string {
+	let text = stripNonContentBlocks(html);
+	text = convertBlockTagsToNewlines(text);
+	text = stripRemainingTags(text);
+	text = decodeHtmlEntities(text);
+	text = collapseWhitespace(text);
+	return text.trim();
+}
+
+export function extractTitle(html: string): string | undefined {
+	const match = html.match(TITLE_TAG_REGEX);
+	if (!match) return undefined;
+	return match[1].replace(ANY_REMAINING_TAG_REGEX, "").trim() || undefined;
+}
+
+// ---------------------------------------------------------------------------
+// URL + content-type guards
+// ---------------------------------------------------------------------------
+
+export function isHtmlContentType(contentType: string): boolean {
+	return contentType.includes(HTML_CONTENT_TYPE_TOKEN);
+}
+
+export function assertTextContentType(contentType: string): void {
+	if (BINARY_CONTENT_TYPE_PREFIXES.some((prefix) => contentType.includes(prefix))) {
+		throw new Error(`Unsupported content type: ${contentType}. web_fetch supports text pages only.`);
+	}
+}
+
+// ---------------------------------------------------------------------------
+// HTTP fetch
+// ---------------------------------------------------------------------------
+
+export function buildFetchRequestInit(signal: AbortSignal | undefined): RequestInit {
+	return {
+		signal,
+		redirect: "follow",
+		headers: { "User-Agent": USER_AGENT, Accept: FETCH_ACCEPT_HEADER },
+	};
+}
+
+export async function fetchUrlOrThrow(url: string, signal: AbortSignal | undefined): Promise<Response> {
+	const res = await fetch(url, buildFetchRequestInit(signal));
+	if (!res.ok) {
+		throw new Error(`HTTP ${res.status} ${res.statusText} for ${url}`);
+	}
+	return res;
+}
+
+export async function extractBodyAsText(
+	res: Response,
+	contentType: string,
+	raw: boolean,
+): Promise<{ text: string; title?: string }> {
+	const body = await res.text();
+	if (!raw && isHtmlContentType(contentType)) {
+		return { text: htmlToText(body), title: extractTitle(body) };
+	}
+	return { text: body };
+}

package/providers/firecrawl.ts

@@ -0,0 +1,119 @@
+import type { FetchResponse, SearchProvider, SearchResponse, SearchResult } from "./types.js";
+
+const FIRECRAWL_API_URL = "https://api.firecrawl.dev/v1";
+export const FIRECRAWL_API_KEY_ENV_VAR = "FIRECRAWL_API_KEY";
+export const FIRECRAWL_PROVIDER_META = {
+	name: "firecrawl",
+	label: "Firecrawl",
+	envVar: FIRECRAWL_API_KEY_ENV_VAR,
+} as const;
+
+interface FirecrawlSearchResult {
+	title?: string;
+	url?: string;
+	description?: string;
+}
+
+interface FirecrawlSearchResponse {
+	success?: boolean;
+	data?: FirecrawlSearchResult[];
+	error?: string;
+}
+
+interface FirecrawlScrapeResponse {
+	success?: boolean;
+	data?: {
+		markdown?: string;
+		html?: string;
+		metadata?: {
+			title?: string;
+			description?: string;
+			language?: string;
+			statusCode?: number;
+		};
+	};
+	error?: string;
+}
+
+function normalizeFirecrawlResults(results: FirecrawlSearchResult[]): SearchResult[] {
+	return results.map((r) => ({
+		title: r.title ?? "",
+		url: r.url ?? "",
+		snippet: r.description ?? "",
+	}));
+}
+
+export class FirecrawlProvider implements SearchProvider {
+	readonly name = FIRECRAWL_PROVIDER_META.name;
+	readonly label = FIRECRAWL_PROVIDER_META.label;
+	readonly envVar = FIRECRAWL_PROVIDER_META.envVar;
+
+	constructor(private readonly apiKey: string) {}
+
+	async search(query: string, maxResults: number, signal?: AbortSignal): Promise<SearchResponse> {
+		if (!this.apiKey) {
+			throw new Error(`${this.envVar} is not set. Run /web-search-config to configure, or export the env var.`);
+		}
+
+		const res = await fetch(`${FIRECRAWL_API_URL}/search`, {
+			method: "POST",
+			headers: {
+				"Content-Type": "application/json",
+				Authorization: `Bearer ${this.apiKey}`,
+			},
+			body: JSON.stringify({
+				query,
+				limit: maxResults,
+			}),
+			signal,
+		});
+
+		if (!res.ok) {
+			const text = await res.text();
+			throw new Error(`${this.label} Search API error (${res.status}): ${text}`);
+		}
+
+		const raw = (await res.json()) as FirecrawlSearchResponse;
+		return { query, results: normalizeFirecrawlResults(raw.data ?? []) };
+	}
+
+	async fetch(url: string, _raw: boolean, signal?: AbortSignal): Promise<FetchResponse> {
+		if (!this.apiKey) {
+			throw new Error(`${this.envVar} is not set. Run /web-search-config to configure, or export the env var.`);
+		}
+
+		const res = await fetch(`${FIRECRAWL_API_URL}/scrape`, {
+			method: "POST",
+			headers: {
+				"Content-Type": "application/json",
+				Authorization: `Bearer ${this.apiKey}`,
+			},
+			body: JSON.stringify({
+				url,
+				formats: ["markdown"],
+			}),
+			signal,
+		});
+
+		if (!res.ok) {
+			const text = await res.text();
+			throw new Error(`${this.label} Fetch API error (${res.status}): ${text}`);
+		}
+
+		const raw = (await res.json()) as FirecrawlScrapeResponse;
+
+		if (!raw.success) {
+			throw new Error(`${this.label} Fetch API error: ${raw.error ?? "scrape failed"}`);
+		}
+
+		if (!raw.data?.markdown) {
+			throw new Error(`${this.label} Fetch API error: no content returned for ${url}`);
+		}
+
+		return {
+			text: raw.data.markdown,
+			title: raw.data.metadata?.title || undefined,
+			contentType: "text/markdown",
+		};
+	}
+}

package/providers/index.ts

@@ -0,0 +1,24 @@
+import { BRAVE_PROVIDER_META } from "./brave.js";
+import { EXA_PROVIDER_META } from "./exa.js";
+import { FIRECRAWL_PROVIDER_META } from "./firecrawl.js";
+import { JINA_PROVIDER_META } from "./jina.js";
+import { SERPER_PROVIDER_META } from "./serper.js";
+import { TAVILY_PROVIDER_META } from "./tavily.js";
+
+export { BRAVE_API_KEY_ENV_VAR, BRAVE_PROVIDER_META, BraveProvider } from "./brave.js";
+export { EXA_API_KEY_ENV_VAR, EXA_PROVIDER_META, ExaProvider } from "./exa.js";
+export { createSearchProvider } from "./factory.js";
+export { FIRECRAWL_API_KEY_ENV_VAR, FIRECRAWL_PROVIDER_META, FirecrawlProvider } from "./firecrawl.js";
+export { JINA_API_KEY_ENV_VAR, JINA_PROVIDER_META, JinaProvider } from "./jina.js";
+export { SERPER_API_KEY_ENV_VAR, SERPER_PROVIDER_META, SerperProvider } from "./serper.js";
+export { TAVILY_API_KEY_ENV_VAR, TAVILY_PROVIDER_META, TavilyProvider } from "./tavily.js";
+export type { FetchResponse, SearchProvider, SearchResponse, SearchResult } from "./types.js";
+
+export const PROVIDERS = [
+	BRAVE_PROVIDER_META,
+	TAVILY_PROVIDER_META,
+	SERPER_PROVIDER_META,
+	EXA_PROVIDER_META,
+	JINA_PROVIDER_META,
+	FIRECRAWL_PROVIDER_META,
+] as const;