pi-web-providers 0.3.0 → 1.1.0

package/README.md

@@ -1,43 +1,23 @@
 # 🌍 pi-web-providers
 
-A _meta_ web extension for [pi](https://pi.dev).
+A _meta_ web extension for [pi](https://pi.dev) that routes search, content
+extraction, answers, and research through configurable per-tool providers.
 
 ## Why?
 
-Most web extensions hard-wire a single search-and-fetch pipeline. That works
-until you want to swap providers, compare results, or use a capability—like deep
-research—that only one backend offers.
-
-**pi-web-providers** takes a different approach: it doesn't do web work itself.
-Instead it dispatches every request to a **configurable set of providers**,
-giving you maximum flexibility and choice when it comes to consuming web results.
-
-The tool surface is **capability-based, not static**. At startup the extension
-inspects which providers are available and what each one supports, then registers
-only the tools that make sense. If your active provider offers search and
-content extraction but not deep research, the agent never sees a research tool.
-Switch to a provider that supports it and the tool appears automatically.
-
-The extension also separates **available tools** from the **active tool set**.
-When a session starts, it can add every available managed tool. Before each
-agent run, it removes tools that are no longer available but keeps any managed
-tools that you explicitly removed from the active set disabled. That keeps the
-tool prompt aligned with the tools that the agent can actually call.
+Most web extensions hard-wire a single backend. **pi-web-providers** lets you
+mix and match providers per tool instead, so `web_search`, `web_contents`,
+`web_answer`, and `web_research` can each use a different backend or be turned
+off entirely.
 
 ## ✨ Features
 
-- **Provider-driven tool surface** — tools are injected based on what the active
-  provider actually supports, not a fixed list
-- **Multiple providers**: Claude, Codex, Exa, Gemini, Perplexity, Parallel,
-  Valyu — each with
-  its own SDK, strengths, and capability set
-- **One config command** (`/web-providers`) with a TUI that adapts to the
-  selected provider
-- **Transparent fallback** — search falls back to Codex when no provider is
-  explicitly enabled and the local CLI is installed and authenticated
-- **Per-provider tool toggles** — disable individual capabilities you don't need
-  without switching providers
-- **Truncated output with temp-file spillover** for large results
+- **Multiple providers** — Claude, Codex, Custom CLI, Exa, Gemini,
+  Perplexity, Parallel, Valyu
+- **Batched search and answers** — run several related queries in a single
+  `web_search` or `web_answer` call and get grouped results back in one response
+- **Async contents prefetch** — optionally start background `web_contents`
+  extraction from `web_search` results and reuse the cached pages later
 
 ## 📦 Install
 
@@ -53,272 +33,324 @@ Run:
 /web-providers
 ```
 
-This command edits a single global config file:
-`~/.pi/agent/web-providers.json`.
+This edits the global config file `~/.pi/agent/web-providers.json`. The
+settings UI mirrors the three sections below: tools, providers, and generic
+settings.
+
+Each tool can be routed to any compatible provider:
+
+| Provider       | search | contents | answer | research | Auth                   |
+| -------------- | :----: | :------: | :----: | :------: | ---------------------- |
+| **Claude**     |   ✔    |          |   ✔    |          | Local Claude Code auth |
+| **Codex**      |   ✔    |          |        |          | Local Codex CLI auth   |
+| **Exa**        |   ✔    |    ✔     |   ✔    |    ✔     | `EXA_API_KEY`          |
+| **Gemini**     |   ✔    |          |   ✔    |    ✔     | `GOOGLE_API_KEY`       |
+| **Perplexity** |   ✔    |          |   ✔    |    ✔     | `PERPLEXITY_API_KEY`   |
+| **Parallel**   |   ✔    |    ✔     |        |          | `PARALLEL_API_KEY`     |
+| **Valyu**      |   ✔    |    ✔     |   ✔    |    ✔     | `VALYU_API_KEY`        |
+
+Advanced option: `custom-cli` is a configurable adapter provider that can route
+any managed tool through a local wrapper command using a JSON stdin/stdout
+contract.
+
+See [`example-config.json`](example-config.json) for a full default
+configuration.
+
+### Tools
+
+Each managed tool maps to one provider id or `null` for off under the top-level
+`tools` key. A tool is only exposed when it is mapped to a compatible provider
+and that provider is currently available. Tool-specific settings live under
+`toolSettings`; today this covers `toolSettings.search.prefetch`.
+
+#### `web_search`
+
+Search the public web for up to 10 queries in one call. It returns grouped
+titles, URLs, and snippets for each query.
+
+<details>
+<summary><strong>Parameters and behavior</strong></summary>
+
+| Parameter    | Type     | Default  | Description                                                    |
+| ------------ | -------- | -------- | -------------------------------------------------------------- |
+| `queries`    | string[] | required | One or more search queries to run (max 10)                     |
+| `maxResults` | integer  | `5`      | Result count per query, clamped to `1–20`                      |
+| `options`    | object   | —        | Provider-specific search options and local `prefetch` settings |
 
-The flow is provider-first: pick the active provider, then configure only that
-provider's tool toggles and settings. Each provider view surfaces the knobs that
-actually apply—Claude shows model/effort/turns settings; Codex shows
-reasoning-effort and web-search-mode toggles; Exa shows search type and
-text-content flags; and so on.
+`web_search.options.prefetch` is local-only and not forwarded into the provider
+SDK. It accepts `provider`, `maxUrls`, `ttlMs`, and `contentsOptions`, and
+starts a background page-extraction workflow only when `prefetch.provider` is
+set. `/web-providers` can also persist default search prefetch settings under
+`toolSettings.search.prefetch`.
 
-## 🔧 Tools
+</details>
 
-Which of the tools below are registered depends on the capabilities of the
-available providers. If no provider supports a given capability, the
-corresponding tool is never exposed to the agent.
+#### `web_contents`
 
-### `web_search`
+Read the main text from one or more web pages. It reuses cached pages when they
+match and fetches only missing or stale URLs.
 
-Find likely sources on the public web and return titles, URLs, and snippets.
+<details>
+<summary><strong>Parameters and behavior</strong></summary>
 
-| Parameter    | Type    | Default  | Description                                                                   |
-| ------------ | ------- | -------- | ----------------------------------------------------------------------------- |
-| `query`      | string  | required | What to search for                                                            |
-| `maxResults` | integer | `5`      | Result count, clamped to `1–20`                                               |
-| `options`    | object  | —        | Provider-specific search options                                              |
-| `provider`   | string  | auto     | Optional override: `claude`, `codex`, `exa`, `gemini`, `perplexity`, `parallel`, or `valyu` |
+| Parameter | Type     | Default  | Description                          |
+| --------- | -------- | -------- | ------------------------------------ |
+| `urls`    | string[] | required | One or more URLs to extract          |
+| `options` | object   | —        | Provider-specific extraction options |
 
-### `web_contents`
+`web_contents` reuses any matching cached pages already present in the local
+content store—whether they came from prefetch or an earlier read—and only
+fetches missing or stale URLs.
 
-Read and extract the main contents of one or more web pages.
+</details>
 
-| Parameter  | Type     | Default  | Description                                             |
-| ---------- | -------- | -------- | ------------------------------------------------------- |
-| `urls`     | string[] | required | One or more URLs to extract                             |
-| `options`  | object   | —        | Provider-specific extraction options                    |
-| `provider` | string   | auto     | Optional override among providers that support contents |
+#### `web_answer`
 
-### `web_answer`
+Answer one or more questions using web-grounded evidence. When you ask more
+than one question, the response is grouped into per-question sections.
 
-Answer a question using web-grounded evidence.
+<details>
+<summary><strong>Parameters and behavior</strong></summary>
 
-| Parameter  | Type   | Default  | Description                                            |
-| ---------- | ------ | -------- | ------------------------------------------------------ |
-| `query`    | string | required | Question to answer                                     |
-| `options`  | object | —        | Provider-specific answer options                       |
-| `provider` | string | auto     | Optional override among providers that support answers |
+| Parameter | Type     | Default  | Description                                          |
+| --------- | -------- | -------- | ---------------------------------------------------- |
+| `queries` | string[] | required | One or more questions to answer in one call (max 10) |
+| `options` | object   | —        | Provider-specific options                            |
 
-### `web_research`
+Responses are grouped into per-question sections when more than one question is
+provided.
 
-Investigate a topic across web sources and produce a longer report.
+</details>
 
-| Parameter  | Type   | Default  | Description                                             |
-| ---------- | ------ | -------- | ------------------------------------------------------- |
-| `input`    | string | required | Research brief or question                              |
-| `options`  | object | —        | Provider-specific research options                      |
-| `provider` | string | auto     | Optional override among providers that support research |
+#### `web_research`
 
-`options` are provider-native and provider-specific. Equivalent concepts can
-use different field names across SDKs, for example Perplexity uses `country`,
-Exa uses `userLocation`, and Valyu uses `countryCode`. Runtime `options`
-override provider defaults, but managed tool inputs and tool wiring stay fixed.
+Investigate a topic across web sources and produce a longer report. The
+provider-specific `options` stay native to each SDK, and runtime options
+override provider configuration when both are set.
 
-## 🔌 Providers
+<details>
+<summary><strong>Parameters and behavior</strong></summary>
 
-Every provider is a thin adapter around an official SDK. The table below
-summarises which capabilities each provider exposes:
+| Parameter | Type   | Default  | Description                |
+| --------- | ------ | -------- | -------------------------- |
+| `input`   | string | required | Research brief or question |
+| `options` | object | —        | Provider-specific options  |
 
-| Provider     | search | contents | answer | research | Auth                   |
-| ------------ | :----: | :------: | :----: | :------: | ---------------------- |
-| **Claude**   |   ✓    |          |   ✓    |          | Local Claude Code auth |
-| **Codex**    |   ✓    |          |        |          | Local Codex CLI auth   |
-| **Exa**      |   ✓    |    ✓     |   ✓    |    ✓     | `EXA_API_KEY`          |
-| **Gemini**   |   ✓    |    ✓     |   ✓    |    ✓     | `GOOGLE_API_KEY`       |
-| **Perplexity** | ✓    |          |   ✓    |    ✓     | `PERPLEXITY_API_KEY`   |
-| **Parallel** |   ✓    |    ✓     |        |          | `PARALLEL_API_KEY`     |
-| **Valyu**    |   ✓    |    ✓     |   ✓    |    ✓     | `VALYU_API_KEY`        |
+`options` are provider-native and provider-specific. Equivalent concepts can use
+different field names across SDKs—for example Perplexity uses `country`, Exa
+uses `userLocation`, and Valyu uses `countryCode`. Runtime `options` override
+provider-native config, but managed tool inputs and tool wiring stay fixed.
 
-### Claude
+</details>
+
+<details>
+<summary><strong>Timeout, retry, and delivery modes</strong></summary>
+
+The extension accepts local control fields for robustness: `requestTimeoutMs`,
+`retryCount`, and `retryDelayMs` on request/response tools, plus
+`pollIntervalMs`, `timeoutMs`, `maxConsecutivePollErrors`, and `resumeId` on
+`web_research` for lifecycle-based research providers. These fields are handled
+by the extension and are not forwarded into the provider SDK call.
+
+- Exa and Valyu research support polling, overall deadlines, and resume IDs
+  but reject `requestTimeoutMs` and do not retry non-idempotent job creation.
+- Perplexity research runs in streaming foreground mode and only supports
+  `requestTimeoutMs`, `retryCount`, and `retryDelayMs`.
+
+Providers deliver results in one of three modes:
+
+- **Silent foreground** — no intermediate output; result returned when done.
+- **Streaming foreground** — progress updates while running, but the result is
+  still only usable after the tool finishes.
+- **Background research** — the provider runs in the background; if
+  interrupted, the run can be resumed later via `resumeId`.
+
+</details>
+
+### Providers
+
+The built-in providers below are thin adapters around official SDKs.
+
+<details>
+<summary><strong>Claude</strong></summary>
 
 - SDK: `@anthropic-ai/claude-agent-sdk`
 - Uses Claude Code's built-in `WebSearch` and `WebFetch` tools behind a
   structured JSON adapter
+- Runs in **silent foreground** mode
 - Supports request-shaping `options` such as `model`, `thinking`, `effort`, and
   `maxTurns`
 - Great for search plus grounded answers if you already use Claude Code locally
 
-### Codex
+</details>
+
+<details>
+<summary><strong>Codex</strong></summary>
 
 - SDK: `@openai/codex-sdk`
 - Runs in read-only mode with web search enabled
+- Runs in **silent foreground** mode
 - Supports request-shaping `web_search.options` such as `model`,
   `modelReasoningEffort`, and `webSearchMode`
 - Best if you already use the local Codex CLI and auth flow
 
-### Exa
+</details>
+
+<details>
+<summary><strong>Exa</strong></summary>
 
 - SDK: `exa-js`
+- Search, contents, and answer run in **silent foreground** mode
+- Research runs in **background research** mode and supports `resumeId`
 - Neural, keyword, hybrid, and deep-research search modes
 - Inline text-content extraction on search results
 
-### Gemini
+</details>
+
+<details>
+<summary><strong>Gemini</strong></summary>
 
 - SDK: `@google/genai`
-- Google Search grounding for answers and URL Context extraction for page contents
+- Search and answer run in **silent foreground** mode
+- Research runs in **background research** mode and supports `resumeId`
+- Google Search grounding for answers
 - Deep-research agents via Google's Gemini API
 - Supports provider-native request options such as `model`, `config`,
   `generation_config`, and `agent_config` depending on the tool
 
-### Perplexity
+</details>
+
+<details>
+<summary><strong>Perplexity</strong></summary>
 
 - SDK: `@perplexity-ai/perplexity_ai`
+- `web_search` and `web_answer` run in **silent foreground** mode
+- `web_research` runs in **streaming foreground** mode (no `resumeId` support)
 - Uses Perplexity Search for `web_search`
 - Uses Sonar for `web_answer` and `sonar-deep-research` for `web_research`
 - Supports provider-specific `web_search.options` such as `country`,
   `search_mode`, `search_domain_filter`, and `search_recency_filter`
 
-### Parallel
+</details>
+
+<details>
+<summary><strong>Parallel</strong></summary>
 
 - SDK: `parallel-web`
+- Runs in **silent foreground** mode
 - Agentic and one-shot search modes
 - Page content extraction with excerpt and full-content toggles
 - Supports provider-native search and extraction options from the Parallel SDK
 
-### Valyu
+</details>
+
+<details>
+<summary><strong>Valyu</strong></summary>
 
 - SDK: `valyu-js`
+- Search, contents, and answer run in **silent foreground** mode
+- Research runs in **background research** mode and supports `resumeId`
 - Web, proprietary, and news search types
 - Supports provider-native options such as `countryCode`, `responseLength`, and
   search/source filters
 - Configurable response length for answers and research
 
-## 📝 Config Notes
-
-- `/web-providers` keeps exactly one provider active by writing `enabled: true`
-  for the selected provider and `enabled: false` for the others
-- Each provider can also enable or disable its individual tools through a `tools`
-  block
-- Managed tools are registered from available provider capabilities, but the
-  active tool set can still be narrower if you removed a tool from the session
-- If no provider is explicitly enabled for search, the extension falls back to
-  Codex when the local CLI is installed and authenticated, unless Codex was
-  explicitly configured as disabled
-- Tools stay inactive when no provider is available for their capability, so
-  they are not injected into the LLM prompt
-- Before each agent run, the extension removes newly unavailable managed tools
-  and keeps manually pruned managed tools inactive instead of re-adding them
-- Secret-like values can be:
-  - literal strings
-  - environment variable names such as `EXA_API_KEY`
-  - shell commands prefixed with `!`
-
-Example:
+</details>
+
+### Custom CLI provider
+
+The `custom-cli` provider lets you bring your own wrapper command for any
+managed tool. Each capability can point at a different local command under
+`providers["custom-cli"].native`.
+
+The repo includes actual wrapper examples under
+[`examples/custom-cli/wrappers/`](examples/custom-cli/wrappers/). They are
+small bash scripts that use `jq` for JSON handling. Each one uses a different
+backend pattern:
+
+- `codex --search exec` for `web_search`
+- Gemini API via `curl` for `web_contents`
+- `claude -p` for `web_answer`
+- Perplexity API via `curl` for `web_research`
+
+<details>
+<summary><strong>Configuration example</strong></summary>
+
+Copy the example wrappers into a local `./wrappers/` directory, then configure:
 
 ```json
 {
-  "version": 1,
+  "tools": {
+    "search": "custom-cli",
+    "contents": "custom-cli",
+    "answer": "custom-cli",
+    "research": "custom-cli"
+  },
   "providers": {
-    "claude": {
-      "enabled": false,
-      "tools": {
-        "search": true,
-        "answer": true
-      }
-    },
-    "codex": {
+    "custom-cli": {
       "enabled": true,
-      "tools": {
-        "search": true
-      },
-      "defaults": {
-        "webSearchMode": "live",
-        "networkAccessEnabled": true
-      }
-    },
-    "exa": {
-      "enabled": false,
-      "tools": {
-        "search": true,
-        "contents": true,
-        "answer": true,
-        "research": true
-      },
-      "apiKey": "EXA_API_KEY",
-      "defaults": {
-        "type": "auto",
-        "contents": {
-          "text": true
-        }
-      }
-    },
-    "gemini": {
-      "enabled": false,
-      "tools": {
-        "search": true,
-        "contents": true,
-        "answer": true,
-        "research": true
-      },
-      "apiKey": "GOOGLE_API_KEY",
-      "defaults": {
-        "searchModel": "gemini-2.5-flash",
-        "contentsModel": "gemini-2.5-flash",
-        "answerModel": "gemini-2.5-flash",
-        "researchAgent": "deep-research-pro-preview-12-2025"
-      }
-    },
-    "perplexity": {
-      "enabled": false,
-      "tools": {
-        "search": true,
-        "answer": true,
-        "research": true
-      },
-      "apiKey": "PERPLEXITY_API_KEY",
-      "defaults": {
+      "native": {
         "search": {
-          "country": "US"
+          "argv": ["bash", "./wrappers/codex-search.sh"]
+        },
+        "contents": {
+          "argv": ["bash", "./wrappers/gemini-contents.sh"]
         },
         "answer": {
-          "model": "sonar"
+          "argv": ["bash", "./wrappers/claude-answer.sh"]
         },
         "research": {
-          "model": "sonar-deep-research"
-        }
-      }
-    },
-    "parallel": {
-      "enabled": false,
-      "tools": {
-        "search": true,
-        "contents": true
-      },
-      "apiKey": "PARALLEL_API_KEY",
-      "defaults": {
-        "search": {
-          "mode": "agentic"
-        },
-        "extract": {
-          "excerpts": true,
-          "full_content": false
+          "argv": ["bash", "./wrappers/perplexity-research.sh"]
         }
       }
-    },
-    "valyu": {
-      "enabled": false,
-      "tools": {
-        "search": true,
-        "contents": true,
-        "answer": true,
-        "research": true
-      },
-      "apiKey": "VALYU_API_KEY",
-      "defaults": {
-        "searchType": "all",
-        "responseLength": "short"
-      }
     }
   }
 }
 ```
 
-## 🛠️ Development
-
-```bash
-npm run check
-npm test
-```
+Those example wrappers deliberately use different local CLIs and APIs so you
+can see several wrapper styles in one setup without extra glue code.
+
+Each capability can also set an optional `cwd` and `env` block. Use `cwd` when
+one wrapper must run from a specific directory. Use `env` for per-command
+variables; each value can be a literal string, an environment variable name, or
+`!command`.
+
+`web_research` runs as a foreground wrapper command, so local polling controls
+(`pollIntervalMs`, `timeoutMs`, `maxConsecutivePollErrors`) and `resumeId` do
+not apply to `custom-cli`.
+
+Wrapper contract:
+
+- `stdin`: one JSON request object with `capability` plus the per-call managed
+  inputs (`query`, `urls`, `input`, `maxResults`, `options`, `cwd`)
+- `stdout`: one JSON response object
+  - `search`: `{ "results": [{ "title", "url", "snippet" }] }`
+  - `contents` / `answer` / `research`: `{ "text": "...", "summary"?: "...", "itemCount"?: 1, "metadata"?: {} }`
+- `stderr`: optional progress lines
+- exit code `0`: success
+- non-zero exit code: failure
+
+</details>
+
+See [`examples/custom-cli/README.md`](examples/custom-cli/README.md) for a
+copy-and-pasteable setup, and see
+[`examples/custom-cli/wrappers/`](examples/custom-cli/wrappers/) for the actual
+wrapper files.
+
+### Generic settings
+
+The `genericSettings` block sets shared execution defaults that apply to all
+providers unless overridden in a provider's `policy` block:
+
+| Field                              | Default    | Description                                    |
+| ---------------------------------- | ---------- | ---------------------------------------------- |
+| `requestTimeoutMs`                 | `30000`    | Maximum time for a single provider request     |
+| `retryCount`                       | `3`        | Retries for transient failures                 |
+| `retryDelayMs`                     | `2000`     | Initial delay before retrying                  |
+| `researchPollIntervalMs`           | `3000`     | How often to poll long-running research jobs   |
+| `researchTimeoutMs`                | `21600000` | Overall deadline for research before returning |
+| `researchMaxConsecutivePollErrors` | `3`        | Consecutive poll failures before stopping      |
 
 ## 📄 License