pi-web-providers 1.1.0 → 2.0.0

package/README.md

@@ -12,11 +12,11 @@ off entirely.
 
 ## ✨ Features
 
-- **Multiple providers** — Claude, Codex, Custom CLI, Exa, Gemini,
-  Perplexity, Parallel, Valyu
-- **Batched search and answers** — run several related queries in a single
+- **Multiple providers**: Claude, Cloudflare, Codex, Exa, Firecrawl,
+  Gemini, Perplexity, Parallel, [Tavily](https://tavily.com), Valyu
+- **Batched search and answers**: run several related queries in a single
   `web_search` or `web_answer` call and get grouped results back in one response
-- **Async contents prefetch** — optionally start background `web_contents`
+- **Async contents prefetch**: optionally start background `web_contents`
   extraction from `web_search` results and reuse the cached pages later
 
 ## 📦 Install
@@ -34,39 +34,45 @@ Run:
 ```
 
 This edits the global config file `~/.pi/agent/web-providers.json`. The
-settings UI mirrors the three sections below: tools, providers, and generic
-settings.
+settings UI mirrors the three sections below: tools, providers, and settings.
 
 Each tool can be routed to any compatible provider:
 
-| Provider       | search | contents | answer | research | Auth                   |
-| -------------- | :----: | :------: | :----: | :------: | ---------------------- |
-| **Claude**     |   ✔    |          |   ✔    |          | Local Claude Code auth |
-| **Codex**      |   ✔    |          |        |          | Local Codex CLI auth   |
-| **Exa**        |   ✔    |    ✔     |   ✔    |    ✔     | `EXA_API_KEY`          |
-| **Gemini**     |   ✔    |          |   ✔    |    ✔     | `GOOGLE_API_KEY`       |
-| **Perplexity** |   ✔    |          |   ✔    |    ✔     | `PERPLEXITY_API_KEY`   |
-| **Parallel**   |   ✔    |    ✔     |        |          | `PARALLEL_API_KEY`     |
-| **Valyu**      |   ✔    |    ✔     |   ✔    |    ✔     | `VALYU_API_KEY`        |
-
-Advanced option: `custom-cli` is a configurable adapter provider that can route
+| Provider       | search | contents | answer | research | Auth                                             |
+| -------------- | :----: | :------: | :----: | :------: | ------------------------------------------------ |
+| **Claude**     |   ✔    |          |   ✔    |          | Local Claude Code auth                           |
+| **Cloudflare** |        |    ✔     |        |          | `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID` |
+| **Codex**      |   ✔    |          |        |          | Local Codex CLI auth                             |
+| **Exa**        |   ✔    |    ✔     |   ✔    |    ✔     | `EXA_API_KEY`                                    |
+| **Firecrawl**  |   ✔    |    ✔     |        |          | `FIRECRAWL_API_KEY`                              |
+| **Gemini**     |   ✔    |          |   ✔    |    ✔     | `GOOGLE_API_KEY`                                 |
+| **Perplexity** |   ✔    |          |   ✔    |    ✔     | `PERPLEXITY_API_KEY`                             |
+| **Parallel**   |   ✔    |    ✔     |        |          | `PARALLEL_API_KEY`                               |
+| **Tavily**     |   ✔    |    ✔     |        |          | `TAVILY_API_KEY`                                 |
+| **Valyu**      |   ✔    |    ✔     |   ✔    |    ✔     | `VALYU_API_KEY`                                  |
+
+Advanced option: `custom` is a configurable adapter provider that can route
 any managed tool through a local wrapper command using a JSON stdin/stdout
 contract.
 
-See [`example-config.json`](example-config.json) for a full default
+See [`example-config.json`](example-config.json) for the minimal default
 configuration.
 
 ### Tools
 
-Each managed tool maps to one provider id or `null` for off under the top-level
-`tools` key. A tool is only exposed when it is mapped to a compatible provider
-and that provider is currently available. Tool-specific settings live under
-`toolSettings`; today this covers `toolSettings.search.prefetch`.
+Each managed tool maps to one provider id under the top-level `tools` key.
+Removing a tool mapping turns that tool off. A tool is only exposed when it is
+mapped to a compatible provider and that provider is currently available.
+Shared defaults and tool-specific settings live under `settings`; search-specific
+settings live under `settings.search`, and async research uses
+`settings.researchTimeoutMs`.
 
 #### `web_search`
 
 Search the public web for up to 10 queries in one call. It returns grouped
-titles, URLs, and snippets for each query.
+titles, URLs, and snippets for each query. Batch related queries when grouped
+comparison matters; use separate sibling `web_search` calls when independent
+results should arrive as soon as they are ready.
 
 <details>
 <summary><strong>Parameters and behavior</strong></summary>
@@ -81,14 +87,16 @@ titles, URLs, and snippets for each query.
 SDK. It accepts `provider`, `maxUrls`, `ttlMs`, and `contentsOptions`, and
 starts a background page-extraction workflow only when `prefetch.provider` is
 set. `/web-providers` can also persist default search prefetch settings under
-`toolSettings.search.prefetch`.
+`settings.search`.
 
 </details>
 
 #### `web_contents`
 
 Read the main text from one or more web pages. It reuses cached pages when they
-match and fetches only missing or stale URLs.
+match and fetches only missing or stale URLs. Batch related pages when they are
+meant to be read as one bundle; use separate sibling `web_contents` calls when
+each page can be acted on independently.
 
 <details>
 <summary><strong>Parameters and behavior</strong></summary>
@@ -99,7 +107,7 @@ match and fetches only missing or stale URLs.
 | `options` | object   | —        | Provider-specific extraction options |
 
 `web_contents` reuses any matching cached pages already present in the local
-content store—whether they came from prefetch or an earlier read—and only
+in-memory cache—whether they came from prefetch or an earlier read—and only
 fetches missing or stale URLs.
 
 </details>
@@ -107,7 +115,9 @@ fetches missing or stale URLs.
 #### `web_answer`
 
 Answer one or more questions using web-grounded evidence. When you ask more
-than one question, the response is grouped into per-question sections.
+than one question, the response is grouped into per-question sections. Batch
+related questions when the answers belong together; split them into sibling
+calls when earlier independent answers can unblock the next step.
 
 <details>
 <summary><strong>Parameters and behavior</strong></summary>
@@ -124,9 +134,10 @@ provided.
 
 #### `web_research`
 
-Investigate a topic across web sources and produce a longer report. The
-provider-specific `options` stay native to each SDK, and runtime options
-override provider configuration when both are set.
+Investigate a topic across web sources and produce a longer report.
+`web_research` is always asynchronous: it starts a background run, returns a
+short dispatch notice immediately, and later posts a completion message with a
+saved report path.
 
 <details>
 <summary><strong>Parameters and behavior</strong></summary>
@@ -136,34 +147,15 @@ override provider configuration when both are set.
 | `input`   | string | required | Research brief or question |
 | `options` | object | —        | Provider-specific options  |
 
-`options` are provider-native and provider-specific. Equivalent concepts can use
+`options` are provider-specific. Equivalent concepts can use
 different field names across SDKs—for example Perplexity uses `country`, Exa
 uses `userLocation`, and Valyu uses `countryCode`. Runtime `options` override
-provider-native config, but managed tool inputs and tool wiring stay fixed.
+provider config, but managed tool inputs and tool wiring stay fixed.
 
-</details>
-
-<details>
-<summary><strong>Timeout, retry, and delivery modes</strong></summary>
-
-The extension accepts local control fields for robustness: `requestTimeoutMs`,
-`retryCount`, and `retryDelayMs` on request/response tools, plus
-`pollIntervalMs`, `timeoutMs`, `maxConsecutivePollErrors`, and `resumeId` on
-`web_research` for lifecycle-based research providers. These fields are handled
-by the extension and are not forwarded into the provider SDK call.
-
-- Exa and Valyu research support polling, overall deadlines, and resume IDs
-  but reject `requestTimeoutMs` and do not retry non-idempotent job creation.
-- Perplexity research runs in streaming foreground mode and only supports
-  `requestTimeoutMs`, `retryCount`, and `retryDelayMs`.
-
-Providers deliver results in one of three modes:
-
-- **Silent foreground** — no intermediate output; result returned when done.
-- **Streaming foreground** — progress updates while running, but the result is
-  still only usable after the tool finishes.
-- **Background research** — the provider runs in the background; if
-  interrupted, the run can be resumed later via `resumeId`.
+Unlike the other managed tools, `web_research` does not accept local timeout,
+retry, polling, or resume controls. Research has one opinionated execution
+style: pi starts it asynchronously, tracks it locally, and saves the final
+report under `.pi/artifacts/research/`.
 
 </details>
 
@@ -177,19 +169,56 @@ The built-in providers below are thin adapters around official SDKs.
 - SDK: `@anthropic-ai/claude-agent-sdk`
 - Uses Claude Code's built-in `WebSearch` and `WebFetch` tools behind a
   structured JSON adapter
-- Runs in **silent foreground** mode
 - Supports request-shaping `options` such as `model`, `thinking`, `effort`, and
   `maxTurns`
 - Great for search plus grounded answers if you already use Claude Code locally
 
 </details>
 
+<details>
+<summary><strong>Cloudflare</strong></summary>
+
+- SDK: `cloudflare`
+- Supports `web_contents` via Cloudflare Browser Rendering's `/markdown`
+  endpoint
+- Good for JavaScript-heavy pages that need a real browser render before
+  extraction
+- Supports provider-specific markdown options such as `gotoOptions`,
+  `waitForSelector`, `waitForTimeout`, `cacheTTL`, and request filtering
+
+**Setup**
+
+1. In the Cloudflare dashboard, create an API token.
+2. Grant it this permission:
+   - `Account | Browser Rendering | Edit`
+3. Scope it to the account you want to use.
+4. Copy that account's **Account ID** from the Cloudflare dashboard.
+5. Configure pi with both values:
+
+```json
+{
+  "tools": {
+    "contents": "cloudflare"
+  },
+  "providers": {
+    "cloudflare": {
+      "apiToken": "CLOUDFLARE_API_TOKEN",
+      "accountId": "CLOUDFLARE_ACCOUNT_ID"
+    }
+  }
+}
+```
+
+If Cloudflare returns `401 Authentication error`, the token permission, token
+scope, or account ID is usually wrong.
+
+</details>
+
 <details>
 <summary><strong>Codex</strong></summary>
 
 - SDK: `@openai/codex-sdk`
 - Runs in read-only mode with web search enabled
-- Runs in **silent foreground** mode
 - Supports request-shaping `web_search.options` such as `model`,
   `modelReasoningEffort`, and `webSearchMode`
 - Best if you already use the local Codex CLI and auth flow
@@ -200,22 +229,37 @@ The built-in providers below are thin adapters around official SDKs.
 <summary><strong>Exa</strong></summary>
 
 - SDK: `exa-js`
-- Search, contents, and answer run in **silent foreground** mode
-- Research runs in **background research** mode and supports `resumeId`
+- Supports `web_search`, `web_contents`, `web_answer`, and `web_research`
+- `web_research` is exposed through pi's async research workflow
 - Neural, keyword, hybrid, and deep-research search modes
 - Inline text-content extraction on search results
 
 </details>
 
+<details>
+<summary><strong>Firecrawl</strong></summary>
+
+- SDK: `@mendable/firecrawl-js`
+- Supports `web_search` and `web_contents`
+- Search can optionally include Firecrawl scrape-backed result enrichment
+- Contents extraction uses Firecrawl scrape with markdown-first defaults
+- Supports provider-specific `options.search` such as `sources`, `categories`,
+  `location`, `timeout`, and `scrapeOptions`
+- Supports provider-specific `options.scrape` such as `formats`,
+  `onlyMainContent`, `waitFor`, `headers`, `location`, `mobile`, `proxy`, and
+  cache controls
+
+</details>
+
 <details>
 <summary><strong>Gemini</strong></summary>
 
 - SDK: `@google/genai`
-- Search and answer run in **silent foreground** mode
-- Research runs in **background research** mode and supports `resumeId`
+- Supports `web_search`, `web_answer`, and `web_research`
+- `web_research` is exposed through pi's async research workflow
 - Google Search grounding for answers
 - Deep-research agents via Google's Gemini API
-- Supports provider-native request options such as `model`, `config`,
+- Supports provider-specific request options such as `model`, `config`,
   `generation_config`, and `agent_config` depending on the tool
 
 </details>
@@ -224,8 +268,8 @@ The built-in providers below are thin adapters around official SDKs.
 <summary><strong>Perplexity</strong></summary>
 
 - SDK: `@perplexity-ai/perplexity_ai`
-- `web_search` and `web_answer` run in **silent foreground** mode
-- `web_research` runs in **streaming foreground** mode (no `resumeId` support)
+- Supports `web_search`, `web_answer`, and `web_research`
+- `web_research` is exposed through pi's async research workflow
 - Uses Perplexity Search for `web_search`
 - Uses Sonar for `web_answer` and `sonar-deep-research` for `web_research`
 - Supports provider-specific `web_search.options` such as `country`,
@@ -237,10 +281,25 @@ The built-in providers below are thin adapters around official SDKs.
 <summary><strong>Parallel</strong></summary>
 
 - SDK: `parallel-web`
-- Runs in **silent foreground** mode
 - Agentic and one-shot search modes
 - Page content extraction with excerpt and full-content toggles
-- Supports provider-native search and extraction options from the Parallel SDK
+- Supports provider-specific search and extraction options from the Parallel SDK
+
+</details>
+
+<details>
+<summary><strong>Tavily</strong></summary>
+
+- SDK: `@tavily/core`
+- Supports `web_search` via Tavily Search
+- Supports `web_contents` via Tavily Extract
+- Good for pairing LLM-oriented web search with lightweight page extraction
+- Supports provider-specific `options.search` such as `searchDepth`, `topic`,
+  `timeRange`, `includeRawContent`, `includeDomains`, `excludeDomains`,
+  `country`, `exactMatch`, and `includeFavicon`
+- Supports provider-specific `options.extract` such as `extractDepth`,
+  `format`, `includeImages`, `query`, `chunksPerSource`, and
+  `includeFavicon`
 
 </details>
 
@@ -248,23 +307,23 @@ The built-in providers below are thin adapters around official SDKs.
 <summary><strong>Valyu</strong></summary>
 
 - SDK: `valyu-js`
-- Search, contents, and answer run in **silent foreground** mode
-- Research runs in **background research** mode and supports `resumeId`
+- Supports `web_search`, `web_contents`, `web_answer`, and `web_research`
+- `web_research` is exposed through pi's async research workflow
 - Web, proprietary, and news search types
-- Supports provider-native options such as `countryCode`, `responseLength`, and
+- Supports provider-specific options such as `countryCode`, `responseLength`, and
   search/source filters
 - Configurable response length for answers and research
 
 </details>
 
-### Custom CLI provider
+### Custom provider
 
-The `custom-cli` provider lets you bring your own wrapper command for any
+The `custom` provider lets you bring your own wrapper command for any
 managed tool. Each capability can point at a different local command under
-`providers["custom-cli"].native`.
+`providers["custom"].options`.
 
 The repo includes actual wrapper examples under
-[`examples/custom-cli/wrappers/`](examples/custom-cli/wrappers/). They are
+[`examples/custom/wrappers/`](examples/custom/wrappers/). They are
 small bash scripts that use `jq` for JSON handling. Each one uses a different
 backend pattern:
 
@@ -281,15 +340,14 @@ Copy the example wrappers into a local `./wrappers/` directory, then configure:
 ```json
 {
   "tools": {
-    "search": "custom-cli",
-    "contents": "custom-cli",
-    "answer": "custom-cli",
-    "research": "custom-cli"
+    "search": "custom",
+    "contents": "custom",
+    "answer": "custom",
+    "research": "custom"
   },
   "providers": {
-    "custom-cli": {
-      "enabled": true,
-      "native": {
+    "custom": {
+      "options": {
         "search": {
           "argv": ["bash", "./wrappers/codex-search.sh"]
         },
@@ -316,9 +374,9 @@ one wrapper must run from a specific directory. Use `env` for per-command
 variables; each value can be a literal string, an environment variable name, or
 `!command`.
 
-`web_research` runs as a foreground wrapper command, so local polling controls
-(`pollIntervalMs`, `timeoutMs`, `maxConsecutivePollErrors`) and `resumeId` do
-not apply to `custom-cli`.
+`web_research` uses the same async workflow as every other research provider:
+pi starts the wrapper in the background, tracks the job locally, and writes the
+final report to a file when it finishes.
 
 Wrapper contract:
 
@@ -326,31 +384,51 @@ Wrapper contract:
   inputs (`query`, `urls`, `input`, `maxResults`, `options`, `cwd`)
 - `stdout`: one JSON response object
   - `search`: `{ "results": [{ "title", "url", "snippet" }] }`
-  - `contents` / `answer` / `research`: `{ "text": "...", "summary"?: "...", "itemCount"?: 1, "metadata"?: {} }`
+  - `contents`: `{ "answers": [{ "url", "content"?: "...", "summary"?: unknown, "metadata"?: {}, "error"?: "..." }] }`
+  - `answer` / `research`: `{ "text": "...", "summary"?: "...", "itemCount"?: 1, "metadata"?: {} }`
 - `stderr`: optional progress lines
 - exit code `0`: success
 - non-zero exit code: failure
 
 </details>
 
-See [`examples/custom-cli/README.md`](examples/custom-cli/README.md) for a
+See [`examples/custom/README.md`](examples/custom/README.md) for a
 copy-and-pasteable setup, and see
-[`examples/custom-cli/wrappers/`](examples/custom-cli/wrappers/) for the actual
+[`examples/custom/wrappers/`](examples/custom/wrappers/) for the actual
 wrapper files.
 
-### Generic settings
+### Settings
 
-The `genericSettings` block sets shared execution defaults that apply to all
-providers unless overridden in a provider's `policy` block:
+The `settings` block holds shared execution defaults that apply to all
+providers unless overridden in a provider's own `settings` block:
+
+| Field               | Default   | Description                                                 |
+| ------------------- | --------- | ----------------------------------------------------------- |
+| `requestTimeoutMs`  | `30000`   | Maximum time for a single provider request                  |
+| `retryCount`        | `3`       | Retries for transient failures                              |
+| `retryDelayMs`      | `2000`    | Initial delay before retrying                               |
+| `researchTimeoutMs` | `1800000` | Maximum total time for an async `web_research` job (30 min) |
+
+## 🔎 Live smoke tests
+
+Use the opt-in live smoke runner to validate the configured providers with the
+same config-resolution and execution path the extension uses at runtime:
+
+```bash
+npm run smoke:live
+```
+
+Optional filters:
+
+```bash
+npm run smoke:live -- --provider gemini
+npm run smoke:live -- --tool contents
+npm run smoke:live -- --include-research
+```
 
-| Field                              | Default    | Description                                    |
-| ---------------------------------- | ---------- | ---------------------------------------------- |
-| `requestTimeoutMs`                 | `30000`    | Maximum time for a single provider request     |
-| `retryCount`                       | `3`        | Retries for transient failures                 |
-| `retryDelayMs`                     | `2000`     | Initial delay before retrying                  |
-| `researchPollIntervalMs`           | `3000`     | How often to poll long-running research jobs   |
-| `researchTimeoutMs`                | `21600000` | Overall deadline for research before returning |
-| `researchMaxConsecutivePollErrors` | `3`        | Consecutive poll failures before stopping      |
+The default run exercises `search`, `contents`, and `answer`. Research probes
+are excluded unless you pass `--include-research`, because they are slower and
+may incur higher provider cost.
 
 ## 📄 License