pi-better-web-search 0.1.11 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (46) hide show
  1. package/CHANGELOG.md +154 -0
  2. package/README.md +269 -66
  3. package/SECURITY.md +31 -4
  4. package/assets/architecture.svg +43 -21
  5. package/assets/search-index.png +0 -0
  6. package/docs/README.zh-CN.md +224 -50
  7. package/docs/architecture.md +151 -43
  8. package/docs/providers.md +52 -9
  9. package/docs/publishing.md +9 -8
  10. package/docs/search-apis.md +250 -0
  11. package/docs/search-skills.md +265 -0
  12. package/docs/testing.md +86 -11
  13. package/extensions/index.ts +9 -0
  14. package/lib/api-command.ts +84 -0
  15. package/lib/atomic-file.ts +32 -0
  16. package/lib/attempt-timeout.ts +82 -0
  17. package/lib/auth.ts +9 -0
  18. package/lib/cache.ts +24 -16
  19. package/lib/candidates.ts +137 -112
  20. package/lib/credentials.ts +98 -0
  21. package/lib/dedicated-config.ts +61 -0
  22. package/lib/format.ts +14 -0
  23. package/lib/health.ts +24 -5
  24. package/lib/login-command.ts +87 -0
  25. package/lib/preferences.ts +62 -0
  26. package/lib/provider-command.ts +66 -0
  27. package/lib/providers/anthropic.ts +7 -4
  28. package/lib/providers/dedicated-http.ts +80 -0
  29. package/lib/providers/deepseek.ts +77 -10
  30. package/lib/providers/exa.ts +66 -0
  31. package/lib/providers/google.ts +4 -3
  32. package/lib/providers/http.ts +99 -47
  33. package/lib/providers/index.ts +9 -0
  34. package/lib/providers/keenable.ts +61 -0
  35. package/lib/providers/ollama.ts +13 -6
  36. package/lib/providers/openai.ts +6 -6
  37. package/lib/providers/perplexity.ts +20 -0
  38. package/lib/providers/sse.ts +93 -54
  39. package/lib/providers/types.ts +6 -2
  40. package/lib/redact.ts +15 -0
  41. package/lib/render.ts +41 -0
  42. package/lib/routes.ts +28 -0
  43. package/lib/schema.ts +26 -10
  44. package/lib/search.ts +124 -64
  45. package/lib/secret-input.ts +63 -0
  46. package/package.json +8 -2
package/CHANGELOG.md CHANGED
@@ -1,5 +1,159 @@
1
1
  # Changelog
2
2
 
3
+ ## 1.0.0 — Search is an agent superpower
4
+
5
+ ### The flagship search upgrade
6
+
7
+ - Bring **Keenable, Exa, and Perplexity**, the top three independent providers
8
+ in the Artificial Analysis Search Index snapshot, plus **DeepSeek native
9
+ Search/Fetch**, directly into pi without requiring a conversation-model change.
10
+ - Lead the English/Chinese READMEs with the search advantage: **33 model-only →
11
+ 83 with Keenable Pro (+50 points)**. Show why agents should find existing
12
+ libraries, official docs, and proven fixes before spending tokens reinventing
13
+ them. Add a sourced Search Index chart and redraw the architecture for four
14
+ standalone APIs, native fallback, terminal login, and the 96k output guard.
15
+
16
+ ### Regression fixes
17
+
18
+ - Stop OpenAI/xAI streams on completed/done/incomplete terminal events and
19
+ Anthropic streams on `message_stop`, cancelling connections that remain open
20
+ instead of timing out and discarding usable search results.
21
+ - Redact native-provider transport, SSE and JSON errors using a snapshot of sent
22
+ credential headers (including custom API tokens and Bearer/Basic auth). Keep
23
+ status/code metadata for fallback, scrub structured error bodies, and suppress
24
+ JSON parser snippets that could expose a partial credential.
25
+ - Require Anthropic server-tool evidence to name `web_search`; unrelated tools
26
+ no longer produce false search success. Skip non-object stream frames safely.
27
+ - Atomically replace settings and the success cache after writing/syncing a
28
+ same-directory temporary file. Partial-write and rename failures preserve the
29
+ previous file; configured file symlinks continue to update their targets.
30
+ - Recheck Keenable public-access consent before each adapter request, including
31
+ combined calls and URL batches. Key loss/logout no longer downgrades an
32
+ in-flight operation to public endpoints; preserve completed search/pages and
33
+ resolve auth-dependent endpoint paths from the same key snapshot as headers.
34
+ - Recognize aborts by signal, structured identity or canonical legacy messages,
35
+ not arbitrary text containing `abort`. Malformed page responses no longer
36
+ discard completed URLs, and dedicated transport AbortErrors remain redacted
37
+ even if credentials rotate while the request is in flight.
38
+ - Cancel stalled response bodies and responses arriving after an attempt timeout,
39
+ even when a custom transport ignores the signal. Ignore late adapter updates
40
+ after timeout/completion and prevent disposed deadlines from being restarted.
41
+ - Bound individual SSE frames to 4,194,304 UTF-16 code units, including unfinished
42
+ lines and multiline data, while accepting larger streams of bounded frames. Scan
43
+ fragmented lines incrementally instead of repeatedly rescanning their prefix.
44
+ - Share literal/JSON-escaped credential redaction across dedicated API errors,
45
+ native DeepSeek errors and Exa per-URL errors. Also redact HTTP status text and
46
+ structured error codes using the credentials actually sent.
47
+ - Keep routing priorities strict: **manual preference → Keenable → Exa →
48
+ Perplexity → standalone DeepSeek → native automatic routing**. Manual choices
49
+ get a first probe even after failure or Retry-After; cooldowns reorder only
50
+ remaining automatic model routes. Standalone DeepSeek failures do not suppress
51
+ native DeepSeek candidates using separate authentication.
52
+ - Route standalone DeepSeek through shared dedicated HTTP validation/backoff:
53
+ honor credential-specific `Retry-After` across Search/Fetch, reject malformed
54
+ header keys before HTTP, and leave native model/proxy auth unchanged.
55
+ - Redact Exa per-URL errors using the key actually sent even if login rotates it
56
+ in flight; honor canonical-URL error statuses that use `url` instead of `id`.
57
+ - Release bounded HTTP error streams without awaiting an underlying cancellation
58
+ promise that may never settle, so errors can reach fallback immediately.
59
+ - Isolate the live DeepSeek test's settings/cache and disable standalone API
60
+ discovery so other saved/environment search keys cannot trigger unrelated
61
+ provider calls or override the intended native DeepSeek fallback check.
62
+
63
+ ### Implementation details
64
+
65
+ - Explicitly provide no skills: remove vendor skill copies and their lockfile
66
+ from the checkout, retain only developer reference notes, and reject skill
67
+ directories/entrypoints in package verification. Keep normal tool descriptions,
68
+ schemas and API behavior unchanged; no automatic vendor skill context is added.
69
+ - Refine both description variants to address insufficient or conflicting
70
+ evidence and cite the source URLs actually used; make fetch conditional on
71
+ need in the URL-capable version, without adding fetch to search-only guidance.
72
+ - Improve capability-aware tool/field descriptions with natural-language
73
+ query guidance, source verification/citation, and optional evidence-driven
74
+ search refinement. Both variants start with the provider-neutral
75
+ `Search the live web and return the most relevant URLs and content.`; only
76
+ URL-capable variants instruct fetching pages. Keep `query` / `urls`, existing validation, adapter requests
77
+ and system prompts unchanged. Document skill wording, provider-specific length
78
+ limits and agent-driven chains without adding a fixed search loop.
79
+ - Add `/pi-better-web-search login`: select DeepSeek, Exa, Keenable or Perplexity
80
+ and enter an API key in a masked local terminal dialog. Persist keys separately
81
+ with restricted permissions, cross-process locking and atomic replacement;
82
+ refresh tool exposure immediately without environment setup or a chat model.
83
+ Saved keys override environment keys; preserve manual routing and fallback.
84
+ - Add safe presence-only `status` and confirmed `logout` (disables standalone
85
+ access without silently switching Keenable to public mode). Refuse RPC/print
86
+ secret entry and key arguments; preserve cancellation and existing credentials
87
+ on save failures. Credential files are local plaintext, not encrypted.
88
+ - Add standalone Exa Search/Fetch with `EXA_API_KEY` and `/web-search-api exa`.
89
+ Search sends the approved `numResults: 10` and `contents.highlights: true`;
90
+ known URLs use `POST /contents` with top-level `text: true`. Inspect per-URL
91
+ statuses even on HTTP 200, preserve successful raw documents, and retry only
92
+ unresolved URLs through existing fallback. Keep safe error redaction,
93
+ cancellation, Retry-After, API priority and the `query` / `urls` contract.
94
+ Document Exa/Keenable skill differences; objective and other advanced options
95
+ remain off; no Exa-specific parameter or system-prompt change is introduced.
96
+ - Add standalone Perplexity Search (`POST /search`, `search_type: web`, never
97
+ fast, no model/effort, no Fetch) and Keenable pro Search/live GET Fetch.
98
+ Preserve raw result objects and use existing timeout, cooldown, fallback and
99
+ partial-success handling. Configured APIs precede automatic model routing;
100
+ explicit per-phase provider preferences still win.
101
+ - Add `/web-search-api [deepseek|exa|keenable|perplexity]` to enable/disable APIs,
102
+ view safe key setup, and opt into Keenable public access with privacy consent.
103
+ This menu saves toggles only; masked login stores credentials separately. Honor Retry-After
104
+ across dedicated search/fetch calls, and add an opt-in status/shape-only smoke
105
+ test: `npm run test:live:search-apis -- all`. No new runtime dependency or MCP.
106
+ - Preserve successful search and URL-fetch data independently. Missing fetch
107
+ capability or fetch failures no longer replace successful search output with
108
+ a whole-call error; failed search (including query-length rejection) no longer
109
+ suppresses requested fetch work. Return partial-success JSON with `search.error`
110
+ and/or per-URL errors, and mark the whole call failed only if no operation
111
+ succeeded. Keep partial search failures visible in collapsed terminal results.
112
+ - Add `/web-search-provider [search|fetch]` terminal pickers with independent,
113
+ persistent search/fetch preferences and an Auto reset. Choices precede
114
+ automatic hints without disabling fallback, per-URL retries, or health
115
+ cooldowns; success-cache updates never overwrite explicit preferences.
116
+
117
+ - Clarify that providing `query` triggers a separate search even with `urls`;
118
+ show URL-only omission guidance only when URL support is exposed. Reject
119
+ additional input arguments in both schemas and at runtime, require non-empty
120
+ URL arrays when supplied, and constrain URL strings to 1–8192 Unicode code
121
+ points consistently with native DeepSeek validation.
122
+ - Strictly limit search and URL analysis candidates to OpenAI `luna`, Claude
123
+ `haiku`, and other brands' `flash` models. Select the newest numeric version
124
+ within the eligible brand/tier, including gateways; current-model and cache
125
+ hints now select a provider instead of pinning an older or larger model.
126
+ Never fall back to an excluded tier, and hide the tool without an eligible
127
+ model or configured standalone API.
128
+ - Compare dotted/dashed Claude minor versions numerically and ignore date
129
+ suffixes independently of the provider API family.
130
+ - Distinguish Search, Fetch, and combined operations in the terminal. Display
131
+ the actual DeepSeek/Ollama standalone API endpoint instead of an inference
132
+ model name; add `details.routes` and progress `details.activeRoute` metadata.
133
+ Raw provider output is unchanged and available in the expanded result.
134
+ - Preserve current-provider priority even when the conversation model's API is
135
+ unsupported, and derive discovery/cache fingerprints from one registry snapshot.
136
+ - Preserve completed DeepSeek/Ollama URL results after a later global HTTP error;
137
+ only unresolved URLs proceed to fallback. Show failed URL counts and truncation
138
+ warnings in collapsed results.
139
+ - Validate timeout overrides against Node's integer timer range and observe
140
+ adapter rejections even when the signal is already aborted.
141
+ - Handle SSE CR/LF/CRLF boundaries, data-less event resets, `[DONE]`, and pending
142
+ read cancellation without waiting for the provider connection to close.
143
+
144
+ ### Regression coverage and validation
145
+
146
+ - Add 62 regression tests across response/deadline lifecycle, late adapter
147
+ updates, bounded/fragmented SSE frames, credential redaction and rotation,
148
+ atomic settings/cache writes, runtime Keenable consent, abort classification,
149
+ native terminal events and search-evidence validation.
150
+ - Add four platform-independent simulated symlink tests for settings/cache target
151
+ updates and dangling-link refusal. Keep the real filesystem checks separately.
152
+ - Verify on Node 24: 386 tests total, 384 passed, no failures; two real file-symlink
153
+ tests skipped because Windows returns `EPERM` when creating file symlinks.
154
+ - Type checking and package verification pass. Provider requests are mocked;
155
+ these checks do not call billable live APIs.
156
+
3
157
  ## 0.1.11
4
158
 
5
159
  - Increase the default timeout to 120 seconds per search and URL fetch attempt
package/README.md CHANGED
@@ -3,9 +3,9 @@
3
3
  **One tool. Every provider. Raw results.**
4
4
 
5
5
  A [pi](https://pi.dev) extension that adds exactly one model-callable tool:
6
- `web_search`. It uses provider-native search APIs, analyzes URLs when a usable
7
- provider supports it, automatically falls back across authenticated providers,
8
- and returns provider data as-is.
6
+ `web_search`. Search with **Keenable, Exa, Perplexity, or DeepSeek**, fetch URLs
7
+ when supported, and automatically fall back across authenticated native
8
+ providers. Keep your conversation model; get provider results as-is.
9
9
 
10
10
  [![CI](https://github.com/zhexusun10/pi-better-web-search/actions/workflows/ci.yml/badge.svg)](https://github.com/zhexusun10/pi-better-web-search/actions/workflows/ci.yml)
11
11
  [![npm](https://img.shields.io/npm/v/pi-better-web-search)](https://www.npmjs.com/package/pi-better-web-search)
@@ -13,23 +13,80 @@ and returns provider data as-is.
13
13
  [![Pi 1.5.2](https://img.shields.io/badge/Pi-1.5.2-blue.svg)](https://pi.dev)
14
14
  [![Node](https://img.shields.io/badge/node-%3E%3D22.19-339933.svg)](https://nodejs.org)
15
15
 
16
- [简体中文](docs/README.zh-CN.md) · [Architecture](docs/architecture.md) · [Providers](docs/providers.md) · [Testing](docs/testing.md) · [Source](extensions/index.ts)
16
+ [简体中文](docs/README.zh-CN.md) · [Search APIs](docs/search-apis.md) · [Architecture](docs/architecture.md) · [Providers](docs/providers.md) · [Testing](docs/testing.md)
17
17
 
18
- ![pi-better-web-search architecture](https://raw.githubusercontent.com/zhexusun10/pi-better-web-search/main/assets/architecture.svg)
18
+ ## How it works
19
19
 
20
- > The npm registry may lag GitHub releases. If the version you want is not yet
21
- > published to npm, install the tagged GitHub release below instead.
20
+ ![pi-better-web-search: one tool, standalone search APIs, native provider fallback, and raw results](assets/architecture.svg)
21
+
22
+ Your agent calls `web_search` with a query, URLs, or both. The router honors
23
+ per-phase provider preferences, tries configured search APIs, and falls back to
24
+ eligible native providers. Search and fetch keep their successful results
25
+ independently; raw provider data goes straight back to the agent.
26
+
27
+ ## Search is an agent superpower
28
+
29
+ **33 without search. 83 with Keenable Pro. A 50-point leap.**
30
+
31
+ The Artificial Analysis Search Index makes the difference visible: the dashed
32
+ **model-only baseline is 33**, while **Keenable Pro reaches 83**, with Exa and
33
+ Perplexity at **80**. All three independent providers score ahead of the
34
+ integrated **OpenAI Web Search result at 74** in this snapshot.
35
+
36
+ ![Artificial Analysis Search Index: model-only 33, Keenable Pro 83, Exa auto 80, Perplexity medium 80, OpenAI Web Search 74, DeepSeek summary 69](assets/search-index.png)
37
+
38
+ *Source: [Artificial Analysis Search Index](https://artificialanalysis.ai/agents/search-api),
39
+ 15 Nov 2026 screenshot. Scores combine DeepSearchQA F1, BrowseComp accuracy and
40
+ AA-Omniscience accuracy. [Methodology](https://artificialanalysis.ai/methodology/search-api).*
41
+
42
+ For a coding agent, the web is working memory at internet scale: official docs,
43
+ existing libraries, battle-tested implementations, migration guides, and fixes
44
+ for the exact error in front of it. **An agent should not spend thousands of
45
+ tokens rediscovering a solution someone has already published.** One well-placed
46
+ `web_search` call can surface the right API, package, or fix—turning repeated
47
+ trial and error into a direct path to a successful task, with fewer wasted tokens.
48
+
49
+ ### 1.0.0: An epic release for the pi open-source community
50
+
51
+ Built to be the web-search tool you reach for first:
52
+
53
+ - **Search specialists, not just model-provider search.** Keenable Pro,
54
+ Exa highlights, Perplexity's standard web Search API, and native DeepSeek.
55
+ - **Your model, your search engine.** Standalone APIs work independently of the
56
+ conversation model. Log in once in a masked terminal dialog and start searching.
57
+ - **Search → read → build.** Search for solutions, fetch the primary sources,
58
+ and implement with evidence. One tool, just `query` and optional `urls`.
59
+ - **Resilient by design.** Independent search/fetch preferences, automatic
60
+ cross-provider fallback, cooldowns, and per-URL recovery that keeps successes.
61
+ - **A thin, open-source integration.** Raw provider results, no extra synthesis
62
+ pass, no MCP server, no new runtime dependency, no system-prompt modifications.
63
+
64
+ ## Install and search
65
+
66
+ **Version 1.0.0 is available.** Download the package from the [GitHub release](https://github.com/zhexusun10/pi-better-web-search/releases/tag/v1.0.0), or install from npm once the maintainer publishes it:
22
67
 
23
68
  ```bash
24
69
  pi install npm:pi-better-web-search
25
70
  ```
26
71
 
27
- Then start a new pi process, or run `/reload` in an existing session.
72
+ Then start a new pi process, or run `/reload` in an existing session. In pi's terminal:
28
73
 
29
- GitHub and local installations also work:
74
+ ```text
75
+ /pi-better-web-search login
76
+ ```
77
+
78
+ Choose **Keenable, Exa, Perplexity, or DeepSeek**, paste your key into the hidden
79
+ input, and press Enter. The API is available immediately—no chat-model setup
80
+ or environment-variable ceremony. Already have an eligible authenticated native
81
+ provider? It works as a fallback too.
82
+
83
+ Try: *“Find the official migration guide and an existing library for this before
84
+ implementing it from scratch.”*
85
+
86
+ Install the latest source from GitHub, or use a local checkout:
30
87
 
31
88
  ```bash
32
- pi install git:github.com/zhexusun10/pi-better-web-search@v0.1.11
89
+ pi install git:github.com/zhexusun10/pi-better-web-search
33
90
  pi install ./path/to/pi-better-web-search
34
91
  ```
35
92
 
@@ -38,16 +95,17 @@ pi install ./path/to/pi-better-web-search
38
95
 
39
96
  Requirements: **pi 1.3+** and **Node.js 22.19+**. Tested with **pi 1.5.2**.
40
97
 
41
- ## What it fixes
98
+ ## What makes it better
42
99
 
43
- `pi-better-web-search` started as an extraction of the `web_search` tool from
44
- `pi-web-search`, then changed the parts that made it unreliable:
100
+ A focused tool with the reliability features an agent actually needs:
45
101
 
46
102
  - **One tool, not two.** There is no separate `url_context` tool. URL analysis
47
- is the optional `urls` field on `web_search`.
103
+ is the optional `urls` field on `web_search`. No skills are installed,
104
+ registered, or shipped; model-facing guidance stays in the tool description
105
+ and parameter schema.
48
106
  - **Correct exposure.** The tool is hidden when the current model has no
49
107
  compatible search interface and no compatible provider is authenticated. A
50
- login or API key for any supported provider is enough to expose it.
108
+ configured dedicated search API or authenticated lightweight model exposes it.
51
109
  - **Automatic fallback.** If the current model cannot search, or a provider
52
110
  call fails, the extension tries other authenticated provider interfaces until
53
111
  one succeeds.
@@ -63,11 +121,12 @@ Requirements: **pi 1.3+** and **Node.js 22.19+**. Tested with **pi 1.5.2**.
63
121
  deduplication, URL normalization, junk filtering, re-ranking, summarization,
64
122
  citation-marker insertion, or appended Sources sections. A final character
65
123
  guard applies structured truncation only to oversized output.
66
- - **Correct semantics.** The tool description is exactly
67
- `Search the web or analyze specified URLs.` The schema makes clear that
68
- `urls` does not require `query`.
124
+ - **Capability-aware guidance.** The description covers current information,
125
+ fact-checking, source verification/citation, and refining a search when evidence
126
+ is insufficient. It mentions fetch only when usable. The schema prefers
127
+ natural-language queries and makes clear that `urls` does not require `query`.
69
128
  - **No system prompt changes.** The extension does not add prompt snippets,
70
- guidelines, loadout hooks, commands, providers, MCP servers, or other tools.
129
+ guidelines, loadout hooks, providers, MCP servers, or other tools.
71
130
  - **Native DeepSeek Search/Fetch.** DeepSeek uses the documented
72
131
  `/v1/tools/search` and `/v1/tools/fetch` endpoints with Bearer auth.
73
132
 
@@ -88,23 +147,88 @@ The runtime schema is logically:
88
147
  length and limit, asking the model to shorten the query and retry. It does not
89
148
  send the query or silently truncate it. Other providers may accept longer queries.
90
149
  - `urls` analyzes the listed pages. URLs can be used **alone** or together with
91
- `query`. A query plus URLs performs a search and then analyzes the URLs.
92
- - At most **20** URLs may be supplied.
150
+ `query`. A query plus URLs performs a separate web search and then analyzes
151
+ the URLs. Omit `query` when only fetching the supplied URLs.
152
+ - If supplied, `urls` must contain **1–20** non-empty strings, each at most
153
+ **8192 Unicode characters**.
154
+ - Both schema variants reject extra arguments (`additionalProperties: false`);
155
+ runtime validation also rejects unknown fields such as `model`, `provider`,
156
+ and `mode` before any provider request.
93
157
  - At least one non-empty `query` or one valid URL is required. This rule is
94
158
  enforced at runtime rather than with a top-level `anyOf`, for compatibility
95
159
  with all model providers.
96
160
  - The `urls` field is hidden when no usable interface supports URL analysis. In
97
- that configuration `query` is required.
161
+ that configuration `query` is required, and its description contains no
162
+ URL/fetch guidance.
98
163
  - Search and URL analysis may use different providers in the same call.
99
-
100
- Invalid input with no usable query or URL, a missing URL-capable provider for
101
- supplied URLs, or a call where every search/URL-only provider fails returns a
102
- tool error (`isError: true`) instead of silently dropping work. Individual
103
- invalid URLs alongside valid work are reported in the `fetched` entries.
164
+ - Research can use explicit follow-up calls: search, inspect the evidence, refine
165
+ the query if needed, then fetch relevant sources when available. There is no
166
+ automatic fixed search chain or global query-length cap from the skills; see
167
+ [query length and chained calls](docs/search-skills.md#query-length-and-chained-calls).
168
+
169
+ Search and URL fetching complete independently: a successful search (including
170
+ zero results) or any successful URL fetch is returned even if the other work
171
+ fails or lacks an interface. Search failures appear in `search.error`; URL
172
+ failures appear in `fetched[].error`. Successful provider data is not replaced
173
+ by a whole-call error. Only invalid request structure or no successful requested
174
+ operation produces a tool error (`isError: true`). Explicit cancellation still
175
+ stops immediately.
176
+
177
+ ### Agent-visible returns
178
+
179
+ All final content blocks use `type: "text"`; there are no separate success,
180
+ partial, or failure content types:
181
+
182
+ | Outcome | Agent-visible text | `isError` |
183
+ |---|---|---|
184
+ | Success | JSON containing the raw search and/or fetched data | omitted |
185
+ | Partial success | JSON retaining successful data plus `search.error` and/or `fetched[].error` | omitted |
186
+ | Complete failure | `Failed: ...` with actionable errors | `true` |
187
+
188
+ A search-only DeepSeek query-length rejection uses the actionable message
189
+ without the `Failed: ` prefix. For combined requests, the same message is in
190
+ `search.error` if fetching succeeds. Oversized JSON is guarded as documented
191
+ below and may end with `[truncated]`. Progress uses `onUpdate`, not an additional
192
+ final return type.
193
+
194
+ ## Dedicated search APIs
195
+
196
+ Run **`/pi-better-web-search login`** in pi's terminal, choose **DeepSeek, Exa,
197
+ Keenable, or Perplexity**, then paste/type the API key into the **hidden input**
198
+ and press Enter. The key is saved separately from settings/cache, and the API
199
+ becomes available immediately without restarting pi or configuring a chat model.
200
+ Saved keys override environment keys. Esc cancels without changing anything.
201
+
202
+ `/pi-better-web-search status` shows presence only; `/pi-better-web-search logout`
203
+ removes a saved key and disables its standalone API. Keys are stored in
204
+ `<pi agent dir>/better-web-search-auth.json` (restricted file permissions,
205
+ **not encrypted**). Never commit/share that file or paste keys into chat or
206
+ command arguments. Environment variables remain an optional fallback.
207
+ `/web-search-api` manages enable/disable flags; `/web-search-provider` selects
208
+ per-phase preferences. Keenable public access still requires explicit consent.
209
+ No new dependency or MCP server is installed.
210
+
211
+ Perplexity uses only `POST /search` with **`search_type: "web"` (not fast)**,
212
+ without any effort/preset/model; it does **not** support Fetch. Keenable uses
213
+ `mode: "pro"` search plus live GET Fetch. Exa returns search highlights and
214
+ fetches full page text via `/contents`; DeepSeek uses native Search/Fetch.
215
+
216
+ | Standalone API | Search configuration | URL fetch |
217
+ |---|---|---|
218
+ | **Keenable** | Pro, 10 results | Live GET Fetch |
219
+ | **Exa** | Default auto, 10 results + highlights | Full-text `/contents` |
220
+ | **Perplexity** | Standard web, 10 results; no inference model | Not supported |
221
+ | **DeepSeek** | 10 results + content | Native `/v1/tools/fetch` |
222
+
223
+ Priority is **manual preference → Keenable → Exa → Perplexity → standalone
224
+ DeepSeek → automatic native routing**. Fetch skips
225
+ Perplexity. See [API setup, protocol and privacy details](docs/search-apis.md).
104
226
 
105
227
  ## Provider support
106
228
 
107
- `pi-better-web-search` natively supports **10 distinct provider classes** (plus Anthropic Messages), automatically detecting authenticated credentials and routing each call through native provider protocols:
229
+ Beyond the **four standalone APIs** above, keep access to **10 native provider
230
+ classes plus Anthropic Messages**, with credential discovery and protocol-specific
231
+ routing:
108
232
 
109
233
  | # | Provider Class | Web Search | URL Analysis | Native Protocol & Endpoint |
110
234
  |:---:|---|:---:|:---:|---|
@@ -140,59 +264,117 @@ Provider detection follows the authenticated model's provider/API:
140
264
  5. OpenAI-compatible Responses APIs (OpenAI, Azure OpenAI, OpenAI Codex, GitHub Copilot, OpenCode)
141
265
  6. Anthropic Messages (Claude)
142
266
 
267
+ All four standalone APIs support masked terminal login or environment-key
268
+ fallbacks. `/web-search-api [deepseek|exa|keenable|perplexity]` manages their
269
+ enable/disable settings. The model-facing tool still has only `query` / `urls`.
270
+ See [dedicated API setup](docs/search-apis.md) and the
271
+ [Exa vs Keenable integration notes](docs/search-skills.md).
272
+
143
273
  See [Providers](docs/providers.md) for request shapes, auth quirks, and URL
144
274
  support per family.
145
275
 
146
- ## Automatic selection and fallback
276
+ ## Choose providers in the terminal
277
+
278
+ Run `/web-search-provider` to choose a preferred provider separately for
279
+ **Web Search** and **URL Fetch**. Or open either picker directly:
280
+
281
+ ```text
282
+ /web-search-provider search
283
+ /web-search-provider fetch
284
+ ```
285
+
286
+ The picker lists usable standalone APIs and authenticated native providers with
287
+ eligible lightweight models that support the selected operation. Choose **Auto** to restore automatic selection;
288
+ Esc leaves the setting unchanged. Selecting a provider does not change the
289
+ conversation model or the model-facing tool schema.
290
+
291
+ A manual choice is a **preference, not a lock**: the standalone API or newest
292
+ eligible native model is tried first, even after a previous failure or
293
+ Retry-After. Errors, timeouts, or an unavailable provider still fall back to
294
+ remaining candidates. URL fetching still retries only unresolved URLs. Input
295
+ validation and cancellation retain their existing behavior.
296
+
297
+ Choices persist across sessions in `~/.pi/agent/better-web-search-settings.json`
298
+ (respects `PI_CODING_AGENT_DIR`; override with `PI_BETTER_WEB_SEARCH_SETTINGS`):
147
299
 
148
- For a search:
300
+ ```json
301
+ { "version": 1, "search": "openai", "fetch": "deepseek" }
302
+ ```
303
+
304
+ Only provider ids and API enable/disable toggles are saved, never credentials.
305
+ These settings are separate
306
+ from the automatic success cache: fallback successes and fingerprint changes
307
+ never overwrite them. Unavailable choices are ignored until eligible again.
308
+
309
+ ## Automatic selection and fallback
149
310
 
150
- 1. The current conversation model is tried first when its provider family
151
- supports web search and neither the provider nor that model is cooling down
152
- after a recent failure.
311
+ Dedicated retrieval APIs have no chat-model tier and can expose the tool on
312
+ their own. For model-backed interfaces, only lightweight models are eligible:
313
+ **OpenAI = `luna`, Claude = `haiku`,
314
+ all other brands (including DeepSeek) = `flash`**. This is a strict allowlist,
315
+ not live price detection. It applies to search, URL analysis, current models,
316
+ cache hints, and all fallback attempts. Without an eligible model, the tool
317
+ stays hidden unless a standalone API is available; it never silently upgrades
318
+ to another model tier.
319
+
320
+ After any explicit provider preference, configured **Keenable → Exa →
321
+ Perplexity → standalone DeepSeek** APIs are tried first (Fetch skips Perplexity),
322
+ followed by this automatic ordering:
323
+
324
+ 1. The newest eligible model of the current conversation's provider is tried
325
+ first, subject to provider/model failure cooldowns. The conversation model
326
+ itself is not privileged over a newer lightweight sibling. An unsupported
327
+ conversation API does not discard this provider preference.
153
328
  2. A remembered successful provider from the preference cache is tried next
154
- when the cache fingerprint is current.
329
+ when the cache fingerprint is current, reselecting its newest eligible model.
155
330
  3. The remaining provider families are tried in this fixed order:
156
331
  **DeepSeek → Ollama → Google → OpenAI → xAI → Anthropic**.
157
- 4. Within a family, brand order comes first: DeepSeek models, then OpenAI,
158
- then Claude, then Gemini/Gemma, then other gateway brands. For brands
159
- without named tiers, variants order `flash → base → pro`; within a variant,
160
- newer versions come first. GPT and Claude keep version-first ordering, with
161
- their capability tier as a tiebreaker: OpenAI tiers are
162
- `luna → terra → sol → astra` and Claude tiers are
163
- `haiku → sonnet → opus → fable`. Ties are deterministic.
332
+ 4. Within a family, brand order comes first: DeepSeek, OpenAI, Claude,
333
+ Gemini/Gemma, then other gateway brands. Within a brand's lightweight tier,
334
+ numeric versions sort newest first: `6.1 > 6`, `5.10 > 5.5`. Claude supports
335
+ both dotted and dashed versions, ignoring date suffixes. Gateway models are
336
+ classified by brand, not API family. Ties are deterministic.
164
337
  5. Alternate providers are considered before additional models of an already
165
338
  represented provider; siblings are then tried round-robin across families.
166
339
  A provider-wide failure skips that provider, while a model-specific failure
167
340
  (including a model-specific 403) can fall back to a sibling.
168
341
 
169
- The search limit is 12 actual attempts; skipped candidates do not consume it.
170
- URL fetching follows the same idea with a separate limit of 6 actual attempts.
342
+ Search and URL fetching each allow one initial attempt plus at most 6 fallback
343
+ attempts (7 actual attempts per phase); skipped candidates do not consume the limit.
171
344
  Model ranking applies only to supported APIs: for example, the built-in
172
345
  `opencode-go` DeepSeek models use `openai-completions`, which this extension
173
346
  cannot search through and does not reinterpret as Responses.
174
347
 
175
- The current model is normally tried first when it supports search, unless its
176
- provider or model is in a failure cooldown. The cache does not replace a
177
- supported current model; it mainly lets an unsupported current model use a
178
- remembered compatible provider immediately on later calls.
348
+ Versions are discovered from pi's available authenticated models, not a
349
+ hardcoded list or an external catalog lookup. Older lightweight models can be
350
+ fallbacks, but `terra`, `sol`, `astra`, `sonnet`, `opus`, `pro`, base models,
351
+ and unknown tiers are not substitutes for the required lightweight tier.
179
352
 
180
353
  If a provider errors, the extension records the attempt and moves to the next
181
- interface. An overlong query rejected by DeepSeek instead returns immediately
182
- without trying another provider or updating its health state. Aborts stop
183
- immediately. Each search or URL fetch attempt times out after 120 seconds by
184
- default; explicit provider errors trigger fallback immediately without waiting
185
- for the timeout. Set `PI_BETTER_WEB_SEARCH_TIMEOUT_MS` to override the per-attempt
186
- timeout. If every attempt fails, the tool returns an error whose message contains
187
- the provider errors.
354
+ interface. An overlong query rejected by DeepSeek stops search fallback without
355
+ trying another search provider or updating its health state; independently
356
+ requested URL fetching still runs. Aborts stop immediately. Each search or URL
357
+ fetch attempt waits up to 60 seconds for HTTP response headers or provider output.
358
+ After headers, a separate 60-second inactivity deadline waits for output. Each
359
+ non-empty response body chunk or provider event resets it, so active streams are
360
+ not cut off due to their total duration. SSE keepalive comments do not count as
361
+ output. User cancellation and provider errors still stop the attempt. Set
362
+ `PI_BETTER_WEB_SEARCH_TIMEOUT_MS` to override both deadlines (integer milliseconds
363
+ from 1 to 2,147,483,647; invalid values use the default). If every attempt fails, the tool returns an error whose message
364
+ contains the provider errors. DeepSeek/Ollama preserve completed URLs if a later
365
+ request hits a global HTTP or transport error, retrying only unresolved URLs.
188
366
  Provider/model routing remains in `details.attempts`. For a URL-only call, the
189
367
  error also reports the per-URL failures. Provider error bodies and model-facing
190
368
  error text are bounded to prevent an oversized error from filling the context.
369
+ Individual SSE frames (including unfinished lines) are limited to 4,194,304
370
+ UTF-16 code units; an oversized frame triggers fallback rather than unbounded
371
+ buffering. This is not a total stream-size limit.
191
372
 
192
373
  Provider-wide failures (auth, balance, network, timeout, malformed request) and
193
374
  model-specific failures are remembered for the life of the extension process.
194
- On later calls the failed provider or model is moved behind healthy candidates
195
- for search and fetch independently. The cooldown starts at 60 seconds, doubles
375
+ On later calls, model-backed automatic routes are moved behind healthy model routes
376
+ for search and fetch independently. Explicit preferences and standalone API order
377
+ are not displaced by health cooldowns. The cooldown starts at 60 seconds, doubles
196
378
  when a subsequent attempt after expiry fails, and is capped at 10 minutes.
197
379
  Concurrent failures during the same cooldown do not extend it; a successful
198
380
  search or a fetch with at least one successful URL and no provider transport
@@ -208,13 +390,28 @@ or removing a provider, login, API key, or available model changes the
208
390
  fingerprint. A changed fingerprint means:
209
391
 
210
392
  - the old cached provider is **not** used as the first choice;
211
- - provider discovery starts again from the current model and family priority;
393
+ - automatic discovery starts again from the current provider and family priority,
394
+ after any usable explicit preference;
212
395
  - the cache is rewritten only after the next successful search/fetch;
213
396
  - stale, unverified preferences are not carried into the new fingerprint.
214
397
 
215
398
  A cache entry is only a hint. Credentials are resolved again on every call, and
216
399
  the referenced provider must still be available and support the operation.
217
400
 
401
+ ## Terminal display
402
+
403
+ The tool name remains `web_search`, but the terminal title distinguishes
404
+ **Web Search**, **Web Fetch** (URLs only), and **Web Search + Fetch**. The result
405
+ shows the actual successful operation and interface; expand it to see the raw
406
+ provider output. DeepSeek and Ollama use standalone APIs without a `model`
407
+ request parameter, so they display e.g. `Fetch: deepseek API /v1/tools/fetch`,
408
+ not a chat model name. `details.routes` records these interfaces; native API
409
+ routes omit `model`. Older `details.model`/cache fields identify the credential
410
+ selection candidate, not an inference model used by those native APIs.
411
+
412
+ If an older model/search label persists, reload or restart pi and check that
413
+ `pi-web-search` is not also loaded; both extensions register `web_search`.
414
+
218
415
  ## Cache
219
416
 
220
417
  Default path:
@@ -235,8 +432,8 @@ Example contents:
235
432
  {
236
433
  "version": 1,
237
434
  "fingerprint": "0123456789abcdef0123456789abcdef",
238
- "search": { "provider": "deepseek", "model": "deepseek-chat" },
239
- "fetch": { "provider": "ollama-cloud", "model": "gpt-oss:120b" }
435
+ "search": { "provider": "deepseek", "model": "deepseek-v4-flash" },
436
+ "fetch": { "provider": "ollama-cloud", "model": "qwen-4-flash" }
240
437
  }
241
438
  ```
242
439
 
@@ -323,10 +520,14 @@ needed.
323
520
 
324
521
  `web_search` is exposed when either:
325
522
 
326
- - the current conversation model supports a provider-native search interface;
327
- or
328
- - at least one authenticated available model maps to a supported provider
329
- family.
523
+ - the current conversation model is lightweight and supports a provider-native
524
+ search interface; or
525
+ - at least one authenticated available lightweight model maps to a supported
526
+ provider family; or
527
+ - a dedicated search API is configured and usable (no chat model needed).
528
+
529
+ For model-backed interfaces the same lightweight restriction applies to `urls`;
530
+ standalone Keenable, Exa, and DeepSeek expose it without a chat-model tier.
330
531
 
331
532
  For an unsupported current model with no compatible login or API key, the tool
332
533
  stays hidden (removed from the active tool set). The extension never calls an
@@ -371,8 +572,10 @@ error details.
371
572
 
372
573
  ## Security and privacy
373
574
 
374
- - Credentials are resolved through pi's model registry on each call. The
375
- extension does not store API keys or OAuth tokens.
575
+ - Native model credentials use pi's registry. Standalone API keys can be saved
576
+ through the masked terminal login in a separate restricted-permission,
577
+ unencrypted credential file; saved keys precede environment fallbacks.
578
+ Settings/cache never contain keys. Never commit/share the credential file.
376
579
  - The cache stores only provider/model identifiers and a fingerprint.
377
580
  - Search queries and requested URLs are sent to the selected provider's API.
378
581
  - The extension has no telemetry, analytics, or third-party service of its own.
@@ -427,7 +630,7 @@ npm run build
427
630
  ```
428
631
 
429
632
  `npm run build` runs the checks and tests, verifies the npm file list, and packs
430
- `dist/pi-better-web-search-0.1.9.tgz`. Nothing is published automatically.
633
+ `dist/pi-better-web-search-1.0.0.tgz`. Nothing is published automatically.
431
634
 
432
635
  The test suite uses plain `.mjs` files that import the `.ts` sources directly.
433
636
  It mocks `globalThis.fetch`, uses a temporary cache file, and exercises