pi-web-providers 0.2.0 → 1.0.0

package/README.md

@@ -1,42 +1,23 @@
 # 🌍 pi-web-providers
 
-A _meta_ web extension for [pi](https://pi.dev).
+A _meta_ web extension for [pi](https://pi.dev) that routes search, content
+extraction, answers, and research through configurable per-tool providers.
 
 ## Why?
 
-Most web extensions hard-wire a single search-and-fetch pipeline. That works
-until you want to swap providers, compare results, or use a capability—like deep
-research—that only one backend offers.
-
-**pi-web-providers** takes a different approach: it doesn't do web work itself.
-Instead it dispatches every request to a **configurable set of providers**,
-giving you maximum flexibility and choice when it comes to consuming web results.
-
-The tool surface is **capability-based, not static**. At startup the extension
-inspects which providers are available and what each one supports, then registers
-only the tools that make sense. If your active provider offers search and
-content extraction but not deep research, the agent never sees a research tool.
-Switch to a provider that supports it and the tool appears automatically.
-
-The extension also separates **available tools** from the **active tool set**.
-When a session starts, it can add every available managed tool. Before each
-agent run, it removes tools that are no longer available but keeps any managed
-tools that you explicitly removed from the active set disabled. That keeps the
-tool prompt aligned with the tools that the agent can actually call.
+Most web extensions hard-wire a single backend. **pi-web-providers** lets you
+mix and match providers per tool instead, so `web_search`, `web_contents`,
+`web_answer`, and `web_research` can each use a different backend or be turned
+off entirely.
 
 ## ✨ Features
 
-- **Provider-driven tool surface** — tools are injected based on what the active
-  provider actually supports, not a fixed list
-- **Six providers**: Claude, Codex, Exa, Gemini, Parallel, Valyu — each with
-  its own SDK, strengths, and capability set
-- **One config command** (`/web-providers`) with a TUI that adapts to the
-  selected provider
-- **Transparent fallback** — search falls back to Codex when no provider is
-  explicitly enabled and the local CLI is installed and authenticated
-- **Per-provider tool toggles** — disable individual capabilities you don't need
-  without switching providers
-- **Truncated output with temp-file spillover** for large results
+- **Multiple providers** — Claude, Codex, Exa, Gemini, Perplexity, Parallel,
+  Valyu
+- **Batched search and answers** — run several related queries in a single
+  `web_search` or `web_answer` call and get grouped results back in one response
+- **Async contents prefetch** — optionally start background `web_contents`
+  extraction from `web_search` results and reuse the cached pages later
 
 ## 📦 Install
 
@@ -52,218 +33,220 @@ Run:
 /web-providers
 ```
 
-This command edits a single global config file:
-`~/.pi/agent/web-providers.json`.
+This edits the global config file `~/.pi/agent/web-providers.json`. The
+settings UI mirrors the three sections below: tools, providers, and generic
+settings.
+
+Each tool can be routed to any compatible provider:
 
-The flow is provider-first: pick the active provider, then configure only that
-provider's tool toggles and settings. Each provider view surfaces the knobs that
-actually apply—Claude shows model/effort/turns settings; Codex shows
-reasoning-effort and web-search-mode toggles; Exa shows search type and
-text-content flags; and so on.
+| Provider       | search | contents | answer | research | Auth                   |
+| -------------- | :----: | :------: | :----: | :------: | ---------------------- |
+| **Claude**     |   ✔    |          |   ✔    |          | Local Claude Code auth |
+| **Codex**      |   ✔    |          |        |          | Local Codex CLI auth   |
+| **Exa**        |   ✔    |    ✔     |   ✔    |    ✔     | `EXA_API_KEY`          |
+| **Gemini**     |   ✔    |          |   ✔    |    ✔     | `GOOGLE_API_KEY`       |
+| **Perplexity** |   ✔    |          |   ✔    |    ✔     | `PERPLEXITY_API_KEY`   |
+| **Parallel**   |   ✔    |    ✔     |        |          | `PARALLEL_API_KEY`     |
+| **Valyu**      |   ✔    |    ✔     |   ✔    |    ✔     | `VALYU_API_KEY`        |
 
-## 🔧 Tools
+See [`example-config.json`](example-config.json) for a full default
+configuration.
 
-Which of the tools below are registered depends on the capabilities of the
-available providers. If no provider supports a given capability, the
-corresponding tool is never exposed to the agent.
+### Tools
 
-### `web_search`
+Each managed tool maps to one provider id or `null` for off under the top-level
+`tools` key. A tool is only exposed when it is mapped to a compatible provider
+and that provider is currently available. Tool-specific settings live under
+`toolSettings`; today this covers `toolSettings.search.prefetch`.
 
-Find likely sources on the public web and return titles, URLs, and snippets.
+#### `web_search`
 
-| Parameter    | Type    | Default  | Description                                                                   |
-| ------------ | ------- | -------- | ----------------------------------------------------------------------------- |
-| `query`      | string  | required | What to search for                                                            |
-| `maxResults` | integer | `5`      | Result count, clamped to `1–20`                                               |
-| `provider`   | string  | auto     | Optional override: `claude`, `codex`, `exa`, `gemini`, `parallel`, or `valyu` |
+Find likely sources on the public web for up to 10 queries in a single call
+and return titles, URLs, and snippets grouped by query.
 
-### `web_contents`
+| Parameter    | Type     | Default  | Description                                                          |
+| ------------ | -------- | -------- | -------------------------------------------------------------------- |
+| `queries`    | string[] | required | One or more search queries to run (max 10)                           |
+| `maxResults` | integer  | `5`      | Result count per query, clamped to `1–20`                            |
+| `options`    | object   | —        | Provider-specific search options plus local `prefetch` orchestration |
+
+`web_search.options.prefetch` is local-only and not forwarded into the provider
+SDK. It accepts `provider`, `maxUrls`, `ttlMs`, and `contentsOptions`, and
+starts a background page-extraction workflow only when `prefetch.provider` is
+set. `/web-providers` can also persist default search prefetch settings under
+`toolSettings.search.prefetch`.
+
+#### `web_contents`
 
 Read and extract the main contents of one or more web pages.
 
-| Parameter  | Type     | Default  | Description                                             |
-| ---------- | -------- | -------- | ------------------------------------------------------- |
-| `urls`     | string[] | required | One or more URLs to extract                             |
-| `options`  | object   | —        | Provider-specific extraction options                    |
-| `provider` | string   | auto     | Optional override among providers that support contents |
+| Parameter | Type     | Default  | Description                          |
+| --------- | -------- | -------- | ------------------------------------ |
+| `urls`    | string[] | required | One or more URLs to extract          |
+| `options` | object   | —        | Provider-specific extraction options |
+
+`web_contents` reuses any matching cached pages already present in the local
+content store—whether they came from prefetch or an earlier read—and only
+fetches missing or stale URLs.
 
-### `web_answer`
+#### `web_answer`
 
-Answer a question using web-grounded evidence.
+Answer one or more questions using web-grounded evidence.
 
-| Parameter  | Type   | Default  | Description                                            |
-| ---------- | ------ | -------- | ------------------------------------------------------ |
-| `query`    | string | required | Question to answer                                     |
-| `options`  | object | —        | Provider-specific answer options                       |
-| `provider` | string | auto     | Optional override among providers that support answers |
+| Parameter | Type     | Default  | Description                                          |
+| --------- | -------- | -------- | ---------------------------------------------------- |
+| `queries` | string[] | required | One or more questions to answer in one call (max 10) |
+| `options` | object   | —        | Provider-specific options                            |
 
-### `web_research`
+Responses are grouped into per-question sections when more than one question is provided.
+
+#### `web_research`
 
 Investigate a topic across web sources and produce a longer report.
 
-| Parameter  | Type   | Default  | Description                                             |
-| ---------- | ------ | -------- | ------------------------------------------------------- |
-| `input`    | string | required | Research brief or question                              |
-| `options`  | object | —        | Provider-specific research options                      |
-| `provider` | string | auto     | Optional override among providers that support research |
+| Parameter | Type   | Default  | Description                |
+| --------- | ------ | -------- | -------------------------- |
+| `input`   | string | required | Research brief or question |
+| `options` | object | —        | Provider-specific options  |
+
+`options` are provider-native and provider-specific. Equivalent concepts can use
+different field names across SDKs—for example Perplexity uses `country`, Exa
+uses `userLocation`, and Valyu uses `countryCode`. Runtime `options` override
+provider-native config, but managed tool inputs and tool wiring stay fixed.
+
+<details>
+<summary><strong>Timeout, retry, and delivery modes</strong></summary>
+
+The extension accepts local control fields for robustness: `requestTimeoutMs`,
+`retryCount`, and `retryDelayMs` on request/response tools, plus
+`pollIntervalMs`, `timeoutMs`, `maxConsecutivePollErrors`, and `resumeId` on
+`web_research` for lifecycle-based research providers. These fields are handled
+by the extension and are not forwarded into the provider SDK call.
+
+- Exa and Valyu research support polling, overall deadlines, and resume IDs
+  but reject `requestTimeoutMs` and do not retry non-idempotent job creation.
+- Perplexity research runs in streaming foreground mode and only supports
+  `requestTimeoutMs`, `retryCount`, and `retryDelayMs`.
+
+Providers deliver results in one of three modes:
 
-## 🔌 Providers
+- **Silent foreground** — no intermediate output; result returned when done.
+- **Streaming foreground** — progress updates while running, but the result is
+  still only usable after the tool finishes.
+- **Background research** — the provider runs in the background; if
+  interrupted, the run can be resumed later via `resumeId`.
 
-Every provider is a thin adapter around an official SDK. The table below
-summarises which capabilities each provider exposes:
+</details>
 
-| Provider     | search | contents | answer | research | Auth                   |
-| ------------ | :----: | :------: | :----: | :------: | ---------------------- |
-| **Claude**   |   ✓    |          |   ✓    |          | Local Claude Code auth |
-| **Codex**    |   ✓    |          |        |          | Local Codex CLI auth   |
-| **Exa**      |   ✓    |    ✓     |   ✓    |    ✓     | `EXA_API_KEY`          |
-| **Gemini**   |   ✓    |          |   ✓    |    ✓     | `GOOGLE_API_KEY`       |
-| **Parallel** |   ✓    |    ✓     |        |          | `PARALLEL_API_KEY`     |
-| **Valyu**    |   ✓    |    ✓     |   ✓    |    ✓     | `VALYU_API_KEY`        |
+### Providers
 
-### Claude
+Every provider is a thin adapter around an official SDK. Each provider has an
+`enabled` toggle that controls whether it is eligible for tool mappings.
+Provider config is split into `native` settings (forwarded to the SDK) and
+`policy` settings (local overrides that take precedence over generic settings);
+legacy `defaults` blocks are still accepted when reading. Secret-like values
+can be literal strings, environment variable names (e.g., `EXA_API_KEY`), or
+shell commands prefixed with `!`.
+
+<details>
+<summary><strong>Claude</strong></summary>
 
 - SDK: `@anthropic-ai/claude-agent-sdk`
 - Uses Claude Code's built-in `WebSearch` and `WebFetch` tools behind a
   structured JSON adapter
+- Runs in **silent foreground** mode
+- Supports request-shaping `options` such as `model`, `thinking`, `effort`, and
+  `maxTurns`
 - Great for search plus grounded answers if you already use Claude Code locally
 
-### Codex
+</details>
+
+<details>
+<summary><strong>Codex</strong></summary>
 
 - SDK: `@openai/codex-sdk`
 - Runs in read-only mode with web search enabled
+- Runs in **silent foreground** mode
+- Supports request-shaping `web_search.options` such as `model`,
+  `modelReasoningEffort`, and `webSearchMode`
 - Best if you already use the local Codex CLI and auth flow
 
-### Exa
+</details>
+
+<details>
+<summary><strong>Exa</strong></summary>
 
 - SDK: `exa-js`
+- Search, contents, and answer run in **silent foreground** mode
+- Research runs in **background research** mode and supports `resumeId`
 - Neural, keyword, hybrid, and deep-research search modes
 - Inline text-content extraction on search results
 
-### Gemini
+</details>
+
+<details>
+<summary><strong>Gemini</strong></summary>
 
 - SDK: `@google/genai`
-- Grounded answers and deep-research agents via Google's Gemini API
+- Search and answer run in **silent foreground** mode
+- Research runs in **background research** mode and supports `resumeId`
+- Google Search grounding for answers
+- Deep-research agents via Google's Gemini API
+- Supports provider-native request options such as `model`, `config`,
+  `generation_config`, and `agent_config` depending on the tool
+
+</details>
+
+<details>
+<summary><strong>Perplexity</strong></summary>
+
+- SDK: `@perplexity-ai/perplexity_ai`
+- `web_search` and `web_answer` run in **silent foreground** mode
+- `web_research` runs in **streaming foreground** mode (no `resumeId` support)
+- Uses Perplexity Search for `web_search`
+- Uses Sonar for `web_answer` and `sonar-deep-research` for `web_research`
+- Supports provider-specific `web_search.options` such as `country`,
+  `search_mode`, `search_domain_filter`, and `search_recency_filter`
 
-### Parallel
+</details>
+
+<details>
+<summary><strong>Parallel</strong></summary>
 
 - SDK: `parallel-web`
+- Runs in **silent foreground** mode
 - Agentic and one-shot search modes
 - Page content extraction with excerpt and full-content toggles
+- Supports provider-native search and extraction options from the Parallel SDK
+
+</details>
 
-### Valyu
+<details>
+<summary><strong>Valyu</strong></summary>
 
 - SDK: `valyu-js`
+- Search, contents, and answer run in **silent foreground** mode
+- Research runs in **background research** mode and supports `resumeId`
 - Web, proprietary, and news search types
+- Supports provider-native options such as `countryCode`, `responseLength`, and
+  search/source filters
 - Configurable response length for answers and research
 
-## 📝 Config Notes
-
-- `/web-providers` keeps exactly one provider active by writing `enabled: true`
-  for the selected provider and `enabled: false` for the others
-- Each provider can also enable or disable its individual tools through a `tools`
-  block
-- Managed tools are registered from available provider capabilities, but the
-  active tool set can still be narrower if you removed a tool from the session
-- If no provider is explicitly enabled for search, the extension falls back to
-  Codex when the local CLI is installed and authenticated, unless Codex was
-  explicitly configured as disabled
-- Tools stay inactive when no provider is available for their capability, so
-  they are not injected into the LLM prompt
-- Before each agent run, the extension removes newly unavailable managed tools
-  and keeps manually pruned managed tools inactive instead of re-adding them
-- Secret-like values can be:
-  - literal strings
-  - environment variable names such as `EXA_API_KEY`
-  - shell commands prefixed with `!`
-
-Example:
-
-```json
-{
-  "version": 1,
-  "providers": {
-    "claude": {
-      "enabled": false,
-      "tools": {
-        "search": true,
-        "answer": true
-      }
-    },
-    "codex": {
-      "enabled": true,
-      "tools": {
-        "search": true
-      },
-      "defaults": {
-        "webSearchMode": "live",
-        "networkAccessEnabled": true
-      }
-    },
-    "exa": {
-      "enabled": false,
-      "tools": {
-        "search": true,
-        "contents": true,
-        "answer": true,
-        "research": true
-      },
-      "apiKey": "EXA_API_KEY",
-      "defaults": {
-        "type": "auto",
-        "contents": {
-          "text": true
-        }
-      }
-    },
-    "gemini": {
-      "enabled": false,
-      "tools": {
-        "search": true,
-        "answer": true,
-        "research": true
-      },
-      "apiKey": "GOOGLE_API_KEY",
-      "defaults": {
-        "searchModel": "gemini-2.5-flash",
-        "answerModel": "gemini-2.5-flash",
-        "researchAgent": "deep-research-pro-preview-12-2025"
-      }
-    },
-    "parallel": {
-      "enabled": false,
-      "tools": {
-        "search": true,
-        "contents": true
-      },
-      "apiKey": "PARALLEL_API_KEY",
-      "defaults": {
-        "search": {
-          "mode": "agentic"
-        },
-        "extract": {
-          "excerpts": true,
-          "full_content": false
-        }
-      }
-    },
-    "valyu": {
-      "enabled": false,
-      "tools": {
-        "search": true,
-        "contents": true,
-        "answer": true,
-        "research": true
-      },
-      "apiKey": "VALYU_API_KEY",
-      "defaults": {
-        "searchType": "all",
-        "responseLength": "short"
-      }
-    }
-  }
-}
-```
+</details>
+
+### Generic settings
+
+The `genericSettings` block sets shared execution defaults that apply to all
+providers unless overridden in a provider's `policy` block:
+
+| Field                              | Default    | Description                                    |
+| ---------------------------------- | ---------- | ---------------------------------------------- |
+| `requestTimeoutMs`                 | `30000`    | Maximum time for a single provider request     |
+| `retryCount`                       | `3`        | Retries for transient failures                 |
+| `retryDelayMs`                     | `2000`     | Initial delay before retrying                  |
+| `researchPollIntervalMs`           | `3000`     | How often to poll long-running research jobs   |
+| `researchTimeoutMs`                | `21600000` | Overall deadline for research before returning |
+| `researchMaxConsecutivePollErrors` | `3`        | Consecutive poll failures before stopping      |
 
 ## 🛠️ Development