pi-web-providers 1.0.0 → 2.0.0

package/README.md

@@ -12,11 +12,11 @@ off entirely.
 
 ## ✨ Features
 
-- **Multiple providers** — Claude, Codex, Exa, Gemini, Perplexity, Parallel,
-  Valyu
-- **Batched search and answers** — run several related queries in a single
+- **Multiple providers**: Claude, Cloudflare, Codex, Exa, Firecrawl,
+  Gemini, Perplexity, Parallel, [Tavily](https://tavily.com), Valyu
+- **Batched search and answers**: run several related queries in a single
   `web_search` or `web_answer` call and get grouped results back in one response
-- **Async contents prefetch** — optionally start background `web_contents`
+- **Async contents prefetch**: optionally start background `web_contents`
   extraction from `web_search` results and reuse the cached pages later
 
 ## 📦 Install
@@ -34,51 +34,72 @@ Run:
 ```
 
 This edits the global config file `~/.pi/agent/web-providers.json`. The
-settings UI mirrors the three sections below: tools, providers, and generic
-settings.
+settings UI mirrors the three sections below: tools, providers, and settings.
 
 Each tool can be routed to any compatible provider:
 
-| Provider       | search | contents | answer | research | Auth                   |
-| -------------- | :----: | :------: | :----: | :------: | ---------------------- |
-| **Claude**     |   ✔    |          |   ✔    |          | Local Claude Code auth |
-| **Codex**      |   ✔    |          |        |          | Local Codex CLI auth   |
-| **Exa**        |   ✔    |    ✔     |   ✔    |    ✔     | `EXA_API_KEY`          |
-| **Gemini**     |   ✔    |          |   ✔    |    ✔     | `GOOGLE_API_KEY`       |
-| **Perplexity** |   ✔    |          |   ✔    |    ✔     | `PERPLEXITY_API_KEY`   |
-| **Parallel**   |   ✔    |    ✔     |        |          | `PARALLEL_API_KEY`     |
-| **Valyu**      |   ✔    |    ✔     |   ✔    |    ✔     | `VALYU_API_KEY`        |
-
-See [`example-config.json`](example-config.json) for a full default
+| Provider       | search | contents | answer | research | Auth                                             |
+| -------------- | :----: | :------: | :----: | :------: | ------------------------------------------------ |
+| **Claude**     |   ✔    |          |   ✔    |          | Local Claude Code auth                           |
+| **Cloudflare** |        |    ✔     |        |          | `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID` |
+| **Codex**      |   ✔    |          |        |          | Local Codex CLI auth                             |
+| **Exa**        |   ✔    |    ✔     |   ✔    |    ✔     | `EXA_API_KEY`                                    |
+| **Firecrawl**  |   ✔    |    ✔     |        |          | `FIRECRAWL_API_KEY`                              |
+| **Gemini**     |   ✔    |          |   ✔    |    ✔     | `GOOGLE_API_KEY`                                 |
+| **Perplexity** |   ✔    |          |   ✔    |    ✔     | `PERPLEXITY_API_KEY`                             |
+| **Parallel**   |   ✔    |    ✔     |        |          | `PARALLEL_API_KEY`                               |
+| **Tavily**     |   ✔    |    ✔     |        |          | `TAVILY_API_KEY`                                 |
+| **Valyu**      |   ✔    |    ✔     |   ✔    |    ✔     | `VALYU_API_KEY`                                  |
+
+Advanced option: `custom` is a configurable adapter provider that can route
+any managed tool through a local wrapper command using a JSON stdin/stdout
+contract.
+
+See [`example-config.json`](example-config.json) for the minimal default
 configuration.
 
 ### Tools
 
-Each managed tool maps to one provider id or `null` for off under the top-level
-`tools` key. A tool is only exposed when it is mapped to a compatible provider
-and that provider is currently available. Tool-specific settings live under
-`toolSettings`; today this covers `toolSettings.search.prefetch`.
+Each managed tool maps to one provider id under the top-level `tools` key.
+Removing a tool mapping turns that tool off. A tool is only exposed when it is
+mapped to a compatible provider and that provider is currently available.
+Shared defaults and tool-specific settings live under `settings`; search-specific
+settings live under `settings.search`, and async research uses
+`settings.researchTimeoutMs`.
 
 #### `web_search`
 
-Find likely sources on the public web for up to 10 queries in a single call
-and return titles, URLs, and snippets grouped by query.
+Search the public web for up to 10 queries in one call. It returns grouped
+titles, URLs, and snippets for each query. Batch related queries when grouped
+comparison matters; use separate sibling `web_search` calls when independent
+results should arrive as soon as they are ready.
 
-| Parameter    | Type     | Default  | Description                                                          |
-| ------------ | -------- | -------- | -------------------------------------------------------------------- |
-| `queries`    | string[] | required | One or more search queries to run (max 10)                           |
-| `maxResults` | integer  | `5`      | Result count per query, clamped to `1–20`                            |
-| `options`    | object   | —        | Provider-specific search options plus local `prefetch` orchestration |
+<details>
+<summary><strong>Parameters and behavior</strong></summary>
+
+| Parameter    | Type     | Default  | Description                                                    |
+| ------------ | -------- | -------- | -------------------------------------------------------------- |
+| `queries`    | string[] | required | One or more search queries to run (max 10)                     |
+| `maxResults` | integer  | `5`      | Result count per query, clamped to `1–20`                      |
+| `options`    | object   | —        | Provider-specific search options and local `prefetch` settings |
 
 `web_search.options.prefetch` is local-only and not forwarded into the provider
 SDK. It accepts `provider`, `maxUrls`, `ttlMs`, and `contentsOptions`, and
 starts a background page-extraction workflow only when `prefetch.provider` is
 set. `/web-providers` can also persist default search prefetch settings under
-`toolSettings.search.prefetch`.
+`settings.search`.
+
+</details>
 
 #### `web_contents`
 
-Read and extract the main contents of one or more web pages.
+Read the main text from one or more web pages. It reuses cached pages when they
+match and fetches only missing or stale URLs. Batch related pages when they are
+meant to be read as one bundle; use separate sibling `web_contents` calls when
+each page can be acted on independently.
+
+<details>
+<summary><strong>Parameters and behavior</strong></summary>
 
 | Parameter | Type     | Default  | Description                          |
 | --------- | -------- | -------- | ------------------------------------ |
@@ -86,67 +107,61 @@ Read and extract the main contents of one or more web pages.
 | `options` | object   | —        | Provider-specific extraction options |
 
 `web_contents` reuses any matching cached pages already present in the local
-content store—whether they came from prefetch or an earlier read—and only
+in-memory cache—whether they came from prefetch or an earlier read—and only
 fetches missing or stale URLs.
 
+</details>
+
 #### `web_answer`
 
-Answer one or more questions using web-grounded evidence.
+Answer one or more questions using web-grounded evidence. When you ask more
+than one question, the response is grouped into per-question sections. Batch
+related questions when the answers belong together; split them into sibling
+calls when earlier independent answers can unblock the next step.
+
+<details>
+<summary><strong>Parameters and behavior</strong></summary>
 
 | Parameter | Type     | Default  | Description                                          |
 | --------- | -------- | -------- | ---------------------------------------------------- |
 | `queries` | string[] | required | One or more questions to answer in one call (max 10) |
 | `options` | object   | —        | Provider-specific options                            |
 
-Responses are grouped into per-question sections when more than one question is provided.
+Responses are grouped into per-question sections when more than one question is
+provided.
+
+</details>
 
 #### `web_research`
 
 Investigate a topic across web sources and produce a longer report.
+`web_research` is always asynchronous: it starts a background run, returns a
+short dispatch notice immediately, and later posts a completion message with a
+saved report path.
+
+<details>
+<summary><strong>Parameters and behavior</strong></summary>
 
 | Parameter | Type   | Default  | Description                |
 | --------- | ------ | -------- | -------------------------- |
 | `input`   | string | required | Research brief or question |
 | `options` | object | —        | Provider-specific options  |
 
-`options` are provider-native and provider-specific. Equivalent concepts can use
+`options` are provider-specific. Equivalent concepts can use
 different field names across SDKs—for example Perplexity uses `country`, Exa
 uses `userLocation`, and Valyu uses `countryCode`. Runtime `options` override
-provider-native config, but managed tool inputs and tool wiring stay fixed.
+provider config, but managed tool inputs and tool wiring stay fixed.
 
-<details>
-<summary><strong>Timeout, retry, and delivery modes</strong></summary>
-
-The extension accepts local control fields for robustness: `requestTimeoutMs`,
-`retryCount`, and `retryDelayMs` on request/response tools, plus
-`pollIntervalMs`, `timeoutMs`, `maxConsecutivePollErrors`, and `resumeId` on
-`web_research` for lifecycle-based research providers. These fields are handled
-by the extension and are not forwarded into the provider SDK call.
-
-- Exa and Valyu research support polling, overall deadlines, and resume IDs
-  but reject `requestTimeoutMs` and do not retry non-idempotent job creation.
-- Perplexity research runs in streaming foreground mode and only supports
-  `requestTimeoutMs`, `retryCount`, and `retryDelayMs`.
-
-Providers deliver results in one of three modes:
-
-- **Silent foreground** — no intermediate output; result returned when done.
-- **Streaming foreground** — progress updates while running, but the result is
-  still only usable after the tool finishes.
-- **Background research** — the provider runs in the background; if
-  interrupted, the run can be resumed later via `resumeId`.
+Unlike the other managed tools, `web_research` does not accept local timeout,
+retry, polling, or resume controls. Research has one opinionated execution
+style: pi starts it asynchronously, tracks it locally, and saves the final
+report under `.pi/artifacts/research/`.
 
 </details>
 
 ### Providers
 
-Every provider is a thin adapter around an official SDK. Each provider has an
-`enabled` toggle that controls whether it is eligible for tool mappings.
-Provider config is split into `native` settings (forwarded to the SDK) and
-`policy` settings (local overrides that take precedence over generic settings);
-legacy `defaults` blocks are still accepted when reading. Secret-like values
-can be literal strings, environment variable names (e.g., `EXA_API_KEY`), or
-shell commands prefixed with `!`.
+The built-in providers below are thin adapters around official SDKs.
 
 <details>
 <summary><strong>Claude</strong></summary>
@@ -154,19 +169,56 @@ shell commands prefixed with `!`.
 - SDK: `@anthropic-ai/claude-agent-sdk`
 - Uses Claude Code's built-in `WebSearch` and `WebFetch` tools behind a
   structured JSON adapter
-- Runs in **silent foreground** mode
 - Supports request-shaping `options` such as `model`, `thinking`, `effort`, and
   `maxTurns`
 - Great for search plus grounded answers if you already use Claude Code locally
 
 </details>
 
+<details>
+<summary><strong>Cloudflare</strong></summary>
+
+- SDK: `cloudflare`
+- Supports `web_contents` via Cloudflare Browser Rendering's `/markdown`
+  endpoint
+- Good for JavaScript-heavy pages that need a real browser render before
+  extraction
+- Supports provider-specific markdown options such as `gotoOptions`,
+  `waitForSelector`, `waitForTimeout`, `cacheTTL`, and request filtering
+
+**Setup**
+
+1. In the Cloudflare dashboard, create an API token.
+2. Grant it this permission:
+   - `Account | Browser Rendering | Edit`
+3. Scope it to the account you want to use.
+4. Copy that account's **Account ID** from the Cloudflare dashboard.
+5. Configure pi with both values:
+
+```json
+{
+  "tools": {
+    "contents": "cloudflare"
+  },
+  "providers": {
+    "cloudflare": {
+      "apiToken": "CLOUDFLARE_API_TOKEN",
+      "accountId": "CLOUDFLARE_ACCOUNT_ID"
+    }
+  }
+}
+```
+
+If Cloudflare returns `401 Authentication error`, the token permission, token
+scope, or account ID is usually wrong.
+
+</details>
+
 <details>
 <summary><strong>Codex</strong></summary>
 
 - SDK: `@openai/codex-sdk`
 - Runs in read-only mode with web search enabled
-- Runs in **silent foreground** mode
 - Supports request-shaping `web_search.options` such as `model`,
   `modelReasoningEffort`, and `webSearchMode`
 - Best if you already use the local Codex CLI and auth flow
@@ -177,22 +229,37 @@ shell commands prefixed with `!`.
 <summary><strong>Exa</strong></summary>
 
 - SDK: `exa-js`
-- Search, contents, and answer run in **silent foreground** mode
-- Research runs in **background research** mode and supports `resumeId`
+- Supports `web_search`, `web_contents`, `web_answer`, and `web_research`
+- `web_research` is exposed through pi's async research workflow
 - Neural, keyword, hybrid, and deep-research search modes
 - Inline text-content extraction on search results
 
 </details>
 
+<details>
+<summary><strong>Firecrawl</strong></summary>
+
+- SDK: `@mendable/firecrawl-js`
+- Supports `web_search` and `web_contents`
+- Search can optionally include Firecrawl scrape-backed result enrichment
+- Contents extraction uses Firecrawl scrape with markdown-first defaults
+- Supports provider-specific `options.search` such as `sources`, `categories`,
+  `location`, `timeout`, and `scrapeOptions`
+- Supports provider-specific `options.scrape` such as `formats`,
+  `onlyMainContent`, `waitFor`, `headers`, `location`, `mobile`, `proxy`, and
+  cache controls
+
+</details>
+
 <details>
 <summary><strong>Gemini</strong></summary>
 
 - SDK: `@google/genai`
-- Search and answer run in **silent foreground** mode
-- Research runs in **background research** mode and supports `resumeId`
+- Supports `web_search`, `web_answer`, and `web_research`
+- `web_research` is exposed through pi's async research workflow
 - Google Search grounding for answers
 - Deep-research agents via Google's Gemini API
-- Supports provider-native request options such as `model`, `config`,
+- Supports provider-specific request options such as `model`, `config`,
   `generation_config`, and `agent_config` depending on the tool
 
 </details>
@@ -201,8 +268,8 @@ shell commands prefixed with `!`.
 <summary><strong>Perplexity</strong></summary>
 
 - SDK: `@perplexity-ai/perplexity_ai`
-- `web_search` and `web_answer` run in **silent foreground** mode
-- `web_research` runs in **streaming foreground** mode (no `resumeId` support)
+- Supports `web_search`, `web_answer`, and `web_research`
+- `web_research` is exposed through pi's async research workflow
 - Uses Perplexity Search for `web_search`
 - Uses Sonar for `web_answer` and `sonar-deep-research` for `web_research`
 - Supports provider-specific `web_search.options` such as `country`,
@@ -214,10 +281,25 @@ shell commands prefixed with `!`.
 <summary><strong>Parallel</strong></summary>
 
 - SDK: `parallel-web`
-- Runs in **silent foreground** mode
 - Agentic and one-shot search modes
 - Page content extraction with excerpt and full-content toggles
-- Supports provider-native search and extraction options from the Parallel SDK
+- Supports provider-specific search and extraction options from the Parallel SDK
+
+</details>
+
+<details>
+<summary><strong>Tavily</strong></summary>
+
+- SDK: `@tavily/core`
+- Supports `web_search` via Tavily Search
+- Supports `web_contents` via Tavily Extract
+- Good for pairing LLM-oriented web search with lightweight page extraction
+- Supports provider-specific `options.search` such as `searchDepth`, `topic`,
+  `timeRange`, `includeRawContent`, `includeDomains`, `excludeDomains`,
+  `country`, `exactMatch`, and `includeFavicon`
+- Supports provider-specific `options.extract` such as `extractDepth`,
+  `format`, `includeImages`, `query`, `chunksPerSource`, and
+  `includeFavicon`
 
 </details>
 
@@ -225,36 +307,129 @@ shell commands prefixed with `!`.
 <summary><strong>Valyu</strong></summary>
 
 - SDK: `valyu-js`
-- Search, contents, and answer run in **silent foreground** mode
-- Research runs in **background research** mode and supports `resumeId`
+- Supports `web_search`, `web_contents`, `web_answer`, and `web_research`
+- `web_research` is exposed through pi's async research workflow
 - Web, proprietary, and news search types
-- Supports provider-native options such as `countryCode`, `responseLength`, and
+- Supports provider-specific options such as `countryCode`, `responseLength`, and
   search/source filters
 - Configurable response length for answers and research
 
 </details>
 
-### Generic settings
+### Custom provider
+
+The `custom` provider lets you bring your own wrapper command for any
+managed tool. Each capability can point at a different local command under
+`providers["custom"].options`.
+
+The repo includes actual wrapper examples under
+[`examples/custom/wrappers/`](examples/custom/wrappers/). They are
+small bash scripts that use `jq` for JSON handling. Each one uses a different
+backend pattern:
+
+- `codex --search exec` for `web_search`
+- Gemini API via `curl` for `web_contents`
+- `claude -p` for `web_answer`
+- Perplexity API via `curl` for `web_research`
+
+<details>
+<summary><strong>Configuration example</strong></summary>
+
+Copy the example wrappers into a local `./wrappers/` directory, then configure:
+
+```json
+{
+  "tools": {
+    "search": "custom",
+    "contents": "custom",
+    "answer": "custom",
+    "research": "custom"
+  },
+  "providers": {
+    "custom": {
+      "options": {
+        "search": {
+          "argv": ["bash", "./wrappers/codex-search.sh"]
+        },
+        "contents": {
+          "argv": ["bash", "./wrappers/gemini-contents.sh"]
+        },
+        "answer": {
+          "argv": ["bash", "./wrappers/claude-answer.sh"]
+        },
+        "research": {
+          "argv": ["bash", "./wrappers/perplexity-research.sh"]
+        }
+      }
+    }
+  }
+}
+```
+
+Those example wrappers deliberately use different local CLIs and APIs so you
+can see several wrapper styles in one setup without extra glue code.
+
+Each capability can also set an optional `cwd` and `env` block. Use `cwd` when
+one wrapper must run from a specific directory. Use `env` for per-command
+variables; each value can be a literal string, an environment variable name, or
+`!command`.
 
-The `genericSettings` block sets shared execution defaults that apply to all
-providers unless overridden in a provider's `policy` block:
+`web_research` uses the same async workflow as every other research provider:
+pi starts the wrapper in the background, tracks the job locally, and writes the
+final report to a file when it finishes.
 
-| Field                              | Default    | Description                                    |
-| ---------------------------------- | ---------- | ---------------------------------------------- |
-| `requestTimeoutMs`                 | `30000`    | Maximum time for a single provider request     |
-| `retryCount`                       | `3`        | Retries for transient failures                 |
-| `retryDelayMs`                     | `2000`     | Initial delay before retrying                  |
-| `researchPollIntervalMs`           | `3000`     | How often to poll long-running research jobs   |
-| `researchTimeoutMs`                | `21600000` | Overall deadline for research before returning |
-| `researchMaxConsecutivePollErrors` | `3`        | Consecutive poll failures before stopping      |
+Wrapper contract:
 
-## 🛠️ Development
+- `stdin`: one JSON request object with `capability` plus the per-call managed
+  inputs (`query`, `urls`, `input`, `maxResults`, `options`, `cwd`)
+- `stdout`: one JSON response object
+  - `search`: `{ "results": [{ "title", "url", "snippet" }] }`
+  - `contents`: `{ "answers": [{ "url", "content"?: "...", "summary"?: unknown, "metadata"?: {}, "error"?: "..." }] }`
+  - `answer` / `research`: `{ "text": "...", "summary"?: "...", "itemCount"?: 1, "metadata"?: {} }`
+- `stderr`: optional progress lines
+- exit code `0`: success
+- non-zero exit code: failure
+
+</details>
+
+See [`examples/custom/README.md`](examples/custom/README.md) for a
+copy-and-pasteable setup, and see
+[`examples/custom/wrappers/`](examples/custom/wrappers/) for the actual
+wrapper files.
+
+### Settings
+
+The `settings` block holds shared execution defaults that apply to all
+providers unless overridden in a provider's own `settings` block:
+
+| Field               | Default   | Description                                                 |
+| ------------------- | --------- | ----------------------------------------------------------- |
+| `requestTimeoutMs`  | `30000`   | Maximum time for a single provider request                  |
+| `retryCount`        | `3`       | Retries for transient failures                              |
+| `retryDelayMs`      | `2000`    | Initial delay before retrying                               |
+| `researchTimeoutMs` | `1800000` | Maximum total time for an async `web_research` job (30 min) |
+
+## 🔎 Live smoke tests
+
+Use the opt-in live smoke runner to validate the configured providers with the
+same config-resolution and execution path the extension uses at runtime:
+
+```bash
+npm run smoke:live
+```
+
+Optional filters:
 
 ```bash
-npm run check
-npm test
+npm run smoke:live -- --provider gemini
+npm run smoke:live -- --tool contents
+npm run smoke:live -- --include-research
 ```
 
+The default run exercises `search`, `contents`, and `answer`. Research probes
+are excluded unless you pass `--include-research`, because they are slower and
+may incur higher provider cost.
+
 ## 📄 License
 
 [MIT](LICENSE)