pi-web-providers 2.1.0 → 2.3.0

package/README.md

@@ -1,7 +1,8 @@
 # 🌍 pi-web-providers
 
 A _meta_ web extension for [pi](https://pi.dev) that routes search, content
-extraction, answers, and research through configurable per-tool providers.
+extraction, answers, and research through configurable per-tool providers, with
+explicit provider-specific option schemas for each managed tool.
 
 ## Why?
 
@@ -12,12 +13,15 @@ off entirely.
 
 ## ✨ Features
 
-- **Multiple providers**: Claude, Cloudflare, Codex, Exa, Firecrawl,
-  Gemini, Linkup, Perplexity, Parallel, [Tavily](https://tavily.com), Valyu
-- **Batched search and answers**: run several related queries in a single
-  `web_search` or `web_answer` call and get grouped results back in one response
-- **Async contents prefetch**: optionally start background `web_contents`
-  extraction from `web_search` results and reuse the cached pages later
+- **Provider-aware tool options**: pi only exposes the provider settings that
+  actually apply to the backend you selected, so tool calls are easier to
+  discover and harder to get wrong
+- **Batched search and answers**: run several related queries or questions in a
+  single `web_search` or `web_answer` call and get grouped results back in one
+  response
+- **Background contents prefetch**: optionally start `web_contents`
+  extraction from `web_search` results in the background and reuse the cached
+  pages later for faster follow-up reads
 
 ## 📦 Install
 
@@ -38,17 +42,25 @@ settings UI mirrors the three sections below: tools, providers, and settings.
 
 Each tool can be routed to any compatible provider:
 
+**Built-in local providers**
+
+| Provider   | search | contents | answer | research | Auth                   |
+| ---------- | :----: | :------: | :----: | :------: | ---------------------- |
+| **Claude** |   ✔    |          |   ✔    |          | Local Claude Code auth |
+| **Codex**  |   ✔    |          |        |          | Local Codex CLI auth   |
+
+**API-backed providers**
+
 | Provider       | search | contents | answer | research | Auth                                             |
 | -------------- | :----: | :------: | :----: | :------: | ------------------------------------------------ |
-| **Claude**     |   ✔    |          |   ✔    |          | Local Claude Code auth                           |
 | **Cloudflare** |        |    ✔     |        |          | `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID` |
-| **Codex**      |   ✔    |          |        |          | Local Codex CLI auth                             |
 | **Exa**        |   ✔    |    ✔     |   ✔    |    ✔     | `EXA_API_KEY`                                    |
 | **Firecrawl**  |   ✔    |    ✔     |        |          | `FIRECRAWL_API_KEY`                              |
 | **Gemini**     |   ✔    |          |   ✔    |    ✔     | `GOOGLE_API_KEY`                                 |
 | **Linkup**     |   ✔    |    ✔     |        |          | `LINKUP_API_KEY`                                 |
-| **Perplexity** |   ✔    |          |   ✔    |    ✔     | `PERPLEXITY_API_KEY`                             |
+| **OpenAI**     |   ✔    |          |   ✔    |    ✔     | `OPENAI_API_KEY`                                 |
 | **Parallel**   |   ✔    |    ✔     |        |          | `PARALLEL_API_KEY`                               |
+| **Perplexity** |   ✔    |          |   ✔    |    ✔     | `PERPLEXITY_API_KEY`                             |
 | **Tavily**     |   ✔    |    ✔     |        |          | `TAVILY_API_KEY`                                 |
 | **Valyu**      |   ✔    |    ✔     |   ✔    |    ✔     | `VALYU_API_KEY`                                  |
 
@@ -78,17 +90,18 @@ results should arrive as soon as they are ready.
 <details>
 <summary><strong>Parameters and behavior</strong></summary>
 
-| Parameter    | Type     | Default  | Description                                                    |
-| ------------ | -------- | -------- | -------------------------------------------------------------- |
-| `queries`    | string[] | required | One or more search queries to run (max 10)                     |
-| `maxResults` | integer  | `5`      | Result count per query, clamped to `1–20`                      |
-| `options`    | object   | —        | Provider-specific search options and local `prefetch` settings |
+| Parameter    | Type     | Default  | Description                                                                                |
+| ------------ | -------- | -------- | ------------------------------------------------------------------------------------------ |
+| `queries`    | string[] | required | One or more search queries to run (max 10)                                                 |
+| `maxResults` | integer  | `5`      | Result count per query, clamped to `1–20`                                                  |
+| `options`    | object   | —        | `provider` settings exposed by the selected provider schema, plus local `runtime` settings |
 
-`web_search.options.prefetch` is local-only and not forwarded into the provider
-SDK. It accepts `provider`, `maxUrls`, `ttlMs`, and `contentsOptions`, and
-starts a background page-extraction workflow only when `prefetch.provider` is
-set. `/web-providers` can also persist default search prefetch settings under
-`settings.search`.
+`web_search.options.runtime.prefetch` is local-only and is not forwarded to the
+provider SDK. It accepts `provider`, `maxUrls`, and `ttlMs`, and starts a
+background page-extraction workflow only when `prefetch.provider` is set.
+`/web-providers` can also persist default search prefetch settings under
+`settings.search`. Per-call retry and timeout overrides also live under
+`web_search.options.runtime`.
 
 </details>
 
@@ -102,10 +115,10 @@ each page can be acted on independently.
 <details>
 <summary><strong>Parameters and behavior</strong></summary>
 
-| Parameter | Type     | Default  | Description                          |
-| --------- | -------- | -------- | ------------------------------------ |
-| `urls`    | string[] | required | One or more URLs to extract          |
-| `options` | object   | —        | Provider-specific extraction options |
+| Parameter | Type     | Default  | Description                                                                                                     |
+| --------- | -------- | -------- | --------------------------------------------------------------------------------------------------------------- |
+| `urls`    | string[] | required | One or more URLs to extract                                                                                     |
+| `options` | object   | —        | `provider` extraction settings exposed by the selected provider schema, plus optional local `runtime` overrides |
 
 `web_contents` reuses any matching cached pages already present in the local
 in-memory cache—whether they came from prefetch or an earlier read—and only
@@ -123,10 +136,10 @@ calls when earlier independent answers can unblock the next step.
 <details>
 <summary><strong>Parameters and behavior</strong></summary>
 
-| Parameter | Type     | Default  | Description                                          |
-| --------- | -------- | -------- | ---------------------------------------------------- |
-| `queries` | string[] | required | One or more questions to answer in one call (max 10) |
-| `options` | object   | —        | Provider-specific options                            |
+| Parameter | Type     | Default  | Description                                                                                          |
+| --------- | -------- | -------- | ---------------------------------------------------------------------------------------------------- |
+| `queries` | string[] | required | One or more questions to answer in one call (max 10)                                                 |
+| `options` | object   | —        | `provider` settings exposed by the selected provider schema, plus optional local `runtime` overrides |
 
 Responses are grouped into per-question sections when more than one question is
 provided.
@@ -143,15 +156,15 @@ saved report path.
 <details>
 <summary><strong>Parameters and behavior</strong></summary>
 
-| Parameter | Type   | Default  | Description                |
-| --------- | ------ | -------- | -------------------------- |
-| `input`   | string | required | Research brief or question |
-| `options` | object | —        | Provider-specific options  |
+| Parameter | Type   | Default  | Description                                                                   |
+| --------- | ------ | -------- | ----------------------------------------------------------------------------- |
+| `input`   | string | required | Research brief or question                                                    |
+| `options` | object | —        | Provider-specific `provider` settings exposed by the selected provider schema |
 
-`options` are provider-specific. Equivalent concepts can use
-different field names across SDKs—for example Perplexity uses `country`, Exa
-uses `userLocation`, and Valyu uses `countryCode`. Runtime `options` override
-provider config, but managed tool inputs and tool wiring stay fixed.
+`options.provider` is provider-specific. Equivalent concepts can use different
+field names across SDKs—for example Perplexity uses `country`, Exa uses
+`userLocation`, and Valyu uses `countryCode`. Unlike the other managed tools,
+`web_research` does not support per-call `options.runtime` overrides.
 
 Unlike the other managed tools, `web_research` does not accept local timeout,
 retry, polling, or resume controls. Research has one opinionated execution
@@ -170,8 +183,8 @@ The built-in providers below are thin adapters around official SDKs.
 - SDK: `@anthropic-ai/claude-agent-sdk`
 - Uses Claude Code's built-in `WebSearch` and `WebFetch` tools behind a
   structured JSON adapter
-- Supports request-shaping `options` such as `model`, `thinking`, `effort`, and
-  `maxTurns`
+- Exposes `model`, `thinking`, `effort`, `maxThinkingTokens`, `maxTurns`, and
+  `maxBudgetUsd` as provider options for search and answer calls
 - Great for search plus grounded answers if you already use Claude Code locally
 
 </details>
@@ -184,8 +197,7 @@ The built-in providers below are thin adapters around official SDKs.
   endpoint
 - Good for JavaScript-heavy pages that need a real browser render before
   extraction
-- Supports provider-specific markdown options such as `gotoOptions`,
-  `waitForSelector`, `waitForTimeout`, `cacheTTL`, and request filtering
+- Exposes `gotoOptions.waitUntil` as the provider-specific contents option
 
 **Setup**
 
@@ -220,8 +232,8 @@ scope, or account ID is usually wrong.
 
 - SDK: `@openai/codex-sdk`
 - Runs in read-only mode with web search enabled
-- Supports request-shaping `web_search.options` such as `model`,
-  `modelReasoningEffort`, and `webSearchMode`
+- Exposes `model`, `modelReasoningEffort`, and `webSearchMode` as provider
+  options for `web_search`
 - Best if you already use the local Codex CLI and auth flow
 
 </details>
@@ -234,6 +246,11 @@ scope, or account ID is usually wrong.
 - `web_research` is exposed through pi's async research workflow
 - Neural, keyword, hybrid, and deep-research search modes
 - Inline text-content extraction on search results
+- Exposes search options such as `category`, `type`, date filters,
+  `includeDomains`, `excludeDomains`, `userLocation`, and `contents`
+- Persisted Exa defaults are scoped under `providers.exa.options.search`
+- `web_contents`, `web_answer`, and `web_research` currently use fixed adapter
+  behavior with no extra per-call provider options
 
 </details>
 
@@ -244,11 +261,10 @@ scope, or account ID is usually wrong.
 - Supports `web_search` and `web_contents`
 - Search can optionally include Firecrawl scrape-backed result enrichment
 - Contents extraction uses Firecrawl scrape with markdown-first defaults
-- Supports provider-specific `options.search` such as `sources`, `categories`,
+- Exposes search options such as `lang`, `country`, `sources`, `categories`,
   `location`, `timeout`, and `scrapeOptions`
-- Supports provider-specific `options.scrape` such as `formats`,
-  `onlyMainContent`, `waitFor`, `headers`, `location`, `mobile`, `proxy`, and
-  cache controls
+- Exposes contents options such as `formats`, `onlyMainContent`, `includeTags`,
+  `excludeTags`, `waitFor`, `headers`, `location`, `mobile`, and `proxy`
 
 </details>
 
@@ -260,8 +276,9 @@ scope, or account ID is usually wrong.
 - `web_research` is exposed through pi's async research workflow
 - Google Search grounding for answers
 - Deep-research agents via Google's Gemini API
-- Supports provider-specific request options such as `model`, `config`,
-  `generation_config`, and `agent_config` depending on the tool
+- Exposes `model` and `generation_config` for search, `model` and `config`
+  for answers, and `agent_config`, `store`, `response_format`,
+  `response_modalities`, `system_instruction`, and `tools` for research
 
 </details>
 
@@ -271,14 +288,70 @@ scope, or account ID is usually wrong.
 - SDK: `linkup-sdk`
 - Supports `web_search` via Linkup Search with fixed `searchResults` output
 - Supports `web_contents` via Linkup Fetch and always returns markdown
-- Supports provider-specific `options.search` such as `depth`, domain filters,
-  image inclusion, and date filters
-- Supports provider-specific `options.fetch` such as `renderJs`,
-  `includeRawHtml`, and `extractImages`
+- Exposes search options `depth`, `includeImages`, `includeDomains`,
+  `excludeDomains`, `fromDate`, and `toDate`
+- Exposes contents options `renderJs`, `includeRawHtml`, and `extractImages`
 - Good fit for a simple search-plus-markdown setup without extra provider wiring
 
 </details>
 
+<details>
+<summary><strong>OpenAI</strong></summary>
+
+- SDK: `openai`
+- Supports `web_search`, `web_answer`, and `web_research`
+- Uses the Responses API for structured web search, grounded answers, and
+  deep-research runs
+- Always enables OpenAI's built-in `web_search_preview` tool for search,
+  answer, and research calls
+- Exposes `model` and `instructions` for `web_search` and `web_answer`
+- Exposes `model`, `instructions`, and `max_tool_calls` for `web_research`
+- Good fit when you want official OpenAI web-grounded search, answers, and deep
+  research behind pi's managed tool abstractions
+
+**Setup**
+
+1. Create or reuse an OpenAI API key.
+2. Configure pi to route `web_search`, `web_answer`, `web_research`, or any
+   subset of them to `openai`.
+3. Optionally set default models under `providers.openai.options.search.model`,
+   `providers.openai.options.answer.model`, and
+   `providers.openai.options.research.model`.
+
+```json
+{
+  "tools": {
+    "search": "openai",
+    "answer": "openai",
+    "research": "openai"
+  },
+  "providers": {
+    "openai": {
+      "apiKey": "OPENAI_API_KEY",
+      "options": {
+        "search": {
+          "model": "gpt-4.1"
+        },
+        "answer": {
+          "model": "gpt-4.1"
+        },
+        "research": {
+          "model": "o4-mini-deep-research"
+        }
+      }
+    }
+  }
+}
+```
+
+You can also set `instructions` as a provider default under
+`providers.openai.options.search`, `providers.openai.options.answer`, or
+`providers.openai.options.research`, and set `max_tool_calls` under
+`providers.openai.options.research`. All of them can also be overridden per
+call.
+
+</details>
+
 <details>
 <summary><strong>Perplexity</strong></summary>
 
@@ -287,8 +360,9 @@ scope, or account ID is usually wrong.
 - `web_research` is exposed through pi's async research workflow
 - Uses Perplexity Search for `web_search`
 - Uses Sonar for `web_answer` and `sonar-deep-research` for `web_research`
-- Supports provider-specific `web_search.options` such as `country`,
-  `search_mode`, `search_domain_filter`, and `search_recency_filter`
+- Exposes search options `country`, `search_mode`,
+  `search_domain_filter`, and `search_recency_filter`
+- Exposes `model` for answer and research calls
 
 </details>
 
@@ -298,7 +372,8 @@ scope, or account ID is usually wrong.
 - SDK: `parallel-web`
 - Agentic and one-shot search modes
 - Page content extraction with excerpt and full-content toggles
-- Supports provider-specific search and extraction options from the Parallel SDK
+- Exposes search option `mode`
+- Exposes contents options `excerpts` and `full_content`
 
 </details>
 
@@ -309,12 +384,11 @@ scope, or account ID is usually wrong.
 - Supports `web_search` via Tavily Search
 - Supports `web_contents` via Tavily Extract
 - Good for pairing LLM-oriented web search with lightweight page extraction
-- Supports provider-specific `options.search` such as `searchDepth`, `topic`,
-  `timeRange`, `includeRawContent`, `includeDomains`, `excludeDomains`,
-  `country`, `exactMatch`, and `includeFavicon`
-- Supports provider-specific `options.extract` such as `extractDepth`,
-  `format`, `includeImages`, `query`, `chunksPerSource`, and
-  `includeFavicon`
+- Exposes search options `topic`, `searchDepth`, `timeRange`, `country`,
+  `exactMatch`, `includeAnswer`, `includeRawContent`, `includeImages`,
+  `includeFavicon`, `includeDomains`, `excludeDomains`, and `days`
+- Exposes contents options `extractDepth`, `format`, `includeImages`, `query`,
+  `chunksPerSource`, and `includeFavicon`
 
 </details>
 
@@ -325,9 +399,12 @@ scope, or account ID is usually wrong.
 - Supports `web_search`, `web_contents`, `web_answer`, and `web_research`
 - `web_research` is exposed through pi's async research workflow
 - Web, proprietary, and news search types
-- Supports provider-specific options such as `countryCode`, `responseLength`, and
-  search/source filters
-- Configurable response length for answers and research
+- Exposes search options `searchType`, `responseLength`, and `countryCode`
+- Exposes answer and research options `responseLength` and `countryCode`
+- Persisted Valyu defaults are scoped under `providers.valyu.options.search`,
+  `providers.valyu.options.answer`, and `providers.valyu.options.research`
+- `web_contents` currently uses fixed adapter behavior with no extra per-call
+  provider options
 
 </details>
 
@@ -337,6 +414,10 @@ The `custom` provider lets you bring your own wrapper command for any
 managed tool. Each capability can point at a different local command under
 `providers["custom"].options`.
 
+`custom` does not expose standard per-call `options.provider` fields. Put
+provider-specific behavior in the wrapper configuration or in the wrapper
+implementation.
+
 The repo includes actual wrapper examples under
 [`examples/custom/wrappers/`](examples/custom/wrappers/). They are
 small bash scripts that use `jq` for JSON handling. Each one uses a different