pi-better-web-search 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +33 -0
- package/CONTRIBUTING.md +71 -0
- package/LICENSE +21 -0
- package/README.md +387 -2
- package/SECURITY.md +35 -0
- package/assets/architecture.svg +89 -0
- package/docs/README.zh-CN.md +322 -0
- package/docs/architecture.md +218 -0
- package/docs/providers.md +220 -0
- package/docs/publishing.md +95 -0
- package/docs/testing.md +115 -0
- package/package.json +74 -4
- package/src/auth.ts +193 -0
- package/src/cache.ts +114 -0
- package/src/candidates.ts +294 -0
- package/src/families.ts +57 -0
- package/src/format.ts +87 -0
- package/src/index.ts +92 -0
- package/src/providers/anthropic.ts +278 -0
- package/src/providers/deepseek.ts +236 -0
- package/src/providers/google.ts +393 -0
- package/src/providers/http.ts +76 -0
- package/src/providers/index.ts +33 -0
- package/src/providers/ollama.ts +193 -0
- package/src/providers/openai.ts +314 -0
- package/src/providers/sse.ts +100 -0
- package/src/providers/types.ts +46 -0
- package/src/schema.ts +47 -0
- package/src/search.ts +519 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.1.0
|
|
4
|
+
|
|
5
|
+
Initial release of `pi-better-web-search`.
|
|
6
|
+
|
|
7
|
+
- Add one model-callable tool, `web_search`. No separate `url_context` tool.
|
|
8
|
+
- Support provider-native search through DeepSeek, Ollama Cloud, Google Gemini
|
|
9
|
+
and Antigravity, OpenAI/Azure/Codex/Copilot/OpenCode Responses, xAI Grok, and
|
|
10
|
+
Anthropic Messages.
|
|
11
|
+
- Add native DeepSeek Search/Fetch support:
|
|
12
|
+
`/v1/tools/search` and `/v1/tools/fetch`, with Bearer authentication,
|
|
13
|
+
`count: 10`, and `need_content: true`.
|
|
14
|
+
- Hide the tool when the current model has no compatible search interface and
|
|
15
|
+
no compatible provider is authenticated.
|
|
16
|
+
- Expose the optional `urls` field only when a usable provider supports URL
|
|
17
|
+
analysis. `urls` can be used without `query`.
|
|
18
|
+
- Add automatic provider fallback. When the current model does not support
|
|
19
|
+
search, use a remembered compatible provider first, then try provider
|
|
20
|
+
families in a deterministic order.
|
|
21
|
+
- Add a preference cache under pi's agent directory with separate search and
|
|
22
|
+
fetch entries and an available-model fingerprint. Adding or removing a
|
|
23
|
+
provider/login/API key invalidates old preferences.
|
|
24
|
+
- Return provider data as-is inside a small JSON wrapper. Do not deduplicate,
|
|
25
|
+
normalize URLs, filter junk URLs, re-rank, summarize, or insert citation
|
|
26
|
+
markers.
|
|
27
|
+
- Guard model-facing output with a 64,000-character limit by default; override
|
|
28
|
+
with `PI_BETTER_WEB_SEARCH_MAX_CHARS`.
|
|
29
|
+
- Add CI, package verification, live DeepSeek test tooling, architecture and
|
|
30
|
+
provider documentation, and a Chinese README.
|
|
31
|
+
- Verification: 116 tests, a real DeepSeek Search/Fetch run, and a real pi
|
|
32
|
+
process end-to-end run with an unsupported current model and DeepSeek
|
|
33
|
+
credentials.
|
package/CONTRIBUTING.md
ADDED
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
Use English for issues, pull requests, commit messages, and primary
|
|
4
|
+
documentation. Chinese translations are welcome.
|
|
5
|
+
|
|
6
|
+
## Setup
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
npm ci --ignore-scripts
|
|
10
|
+
npm run check
|
|
11
|
+
npm test
|
|
12
|
+
npm run verify:package
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
The extension is TypeScript source loaded directly by pi. There is no compile
|
|
16
|
+
step and no runtime dependency to bundle.
|
|
17
|
+
|
|
18
|
+
## Rules
|
|
19
|
+
|
|
20
|
+
- Keep exactly one model-callable tool: `web_search`.
|
|
21
|
+
- Do not modify pi's system prompt, add prompt snippets/guidelines, or register
|
|
22
|
+
commands, providers, MCP servers, or additional tools.
|
|
23
|
+
- Do not post-process provider results. No deduplication, URL normalization,
|
|
24
|
+
junk-URL filtering, re-ranking, summarizing, citation-marker insertion, or
|
|
25
|
+
appended Sources sections. The only output transformation is the final
|
|
26
|
+
character guard.
|
|
27
|
+
- Do not call a provider that is not authenticated. Do not use a provider that
|
|
28
|
+
does not support the requested operation.
|
|
29
|
+
- Keep the `urls` field hidden when no usable provider supports URL analysis.
|
|
30
|
+
- Preserve provider request/SSE quirks: Codex account headers, Copilot base
|
|
31
|
+
URLs, OpenCode session headers, Grok dialect, Antigravity wrappers, and
|
|
32
|
+
Anthropic OAuth requirements.
|
|
33
|
+
- Add regression tests for request shape, raw result preservation, errors,
|
|
34
|
+
aborts, exposure, fallback, cache invalidation, and URL handling.
|
|
35
|
+
- Update `docs/providers.md`, `docs/architecture.md`, and `CHANGELOG.md` when
|
|
36
|
+
behavior changes.
|
|
37
|
+
|
|
38
|
+
## Provider work
|
|
39
|
+
|
|
40
|
+
A provider adapter lives in `src/providers/<family>.ts` and follows the
|
|
41
|
+
`ProviderAdapter` contract in `src/providers/types.ts`. Return provider data
|
|
42
|
+
unchanged. `search()` should throw when the provider did not perform a search;
|
|
43
|
+
`fetch()` should return one entry per URL and put per-URL failures in
|
|
44
|
+
`entry.error`.
|
|
45
|
+
|
|
46
|
+
## Live provider tests
|
|
47
|
+
|
|
48
|
+
The opt-in DeepSeek test requires a real key:
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
DEEPSEEK_API_KEY=... npm run test:live:deepseek
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Optional inputs:
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
PI_BWS_TEST_QUERY="DeepSeek V4 模型 发布" \
|
|
58
|
+
PI_BWS_TEST_URL="https://example.com" \
|
|
59
|
+
DEEPSEEK_API_KEY=... npm run test:live:deepseek
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Never commit API keys, OAuth tokens, private prompts, full session transcripts,
|
|
63
|
+
or generated provider responses containing private data. The preference cache
|
|
64
|
+
stores only provider/model identifiers and a fingerprint.
|
|
65
|
+
|
|
66
|
+
## Commits and pull requests
|
|
67
|
+
|
|
68
|
+
Use conventional commit prefixes such as `feat:`, `fix:`, `docs:`, `test:`, and
|
|
69
|
+
`chore:`. Keep pull requests focused. Complete the pull request checklist and
|
|
70
|
+
include the pi version, provider/model, and a minimal reproduction when fixing a
|
|
71
|
+
provider bug.
|
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Jerry (zhexusun10)
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,388 @@
|
|
|
1
|
-
#
|
|
1
|
+
# pi-better-web-search
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**One tool. Every provider. Raw results.**
|
|
4
|
+
|
|
5
|
+
A [pi](https://pi.dev) extension that adds exactly one model-callable tool:
|
|
6
|
+
`web_search`. It uses provider-native search APIs, analyzes URLs when a usable
|
|
7
|
+
provider supports it, automatically falls back across authenticated providers,
|
|
8
|
+
and returns provider data as-is.
|
|
9
|
+
|
|
10
|
+
[](https://github.com/zhexusun10/pi-better-web-search/actions/workflows/ci.yml)
|
|
11
|
+
[](https://www.npmjs.com/package/pi-better-web-search)
|
|
12
|
+
[](LICENSE)
|
|
13
|
+
[](https://pi.dev)
|
|
14
|
+
[](https://nodejs.org)
|
|
15
|
+
|
|
16
|
+
[简体中文](docs/README.zh-CN.md) · [Architecture](docs/architecture.md) · [Providers](docs/providers.md) · [Testing](docs/testing.md) · [Source](src/index.ts)
|
|
17
|
+
|
|
18
|
+

|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
pi install npm:pi-better-web-search
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
Then start a new pi process, or run `/reload` in an existing session.
|
|
25
|
+
|
|
26
|
+
GitHub and local installations also work:
|
|
27
|
+
|
|
28
|
+
```bash
|
|
29
|
+
pi install git:github.com/zhexusun10/pi-better-web-search
|
|
30
|
+
pi install ./path/to/pi-better-web-search
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
> Do not load this package together with `pi-web-search`. Both register a tool
|
|
34
|
+
> named `web_search`, and the second registration replaces the first.
|
|
35
|
+
|
|
36
|
+
Requirements: **pi 1.3+** and **Node.js 22.19+**.
|
|
37
|
+
|
|
38
|
+
## What it fixes
|
|
39
|
+
|
|
40
|
+
`pi-better-web-search` started as an extraction of the `web_search` tool from
|
|
41
|
+
`pi-web-search`, then changed the parts that made it unreliable:
|
|
42
|
+
|
|
43
|
+
- **One tool, not two.** There is no separate `url_context` tool. URL analysis
|
|
44
|
+
is the optional `urls` field on `web_search`.
|
|
45
|
+
- **Correct exposure.** The tool is hidden when the current model has no
|
|
46
|
+
compatible search interface and no compatible provider is authenticated. A
|
|
47
|
+
login or API key for any supported provider is enough to expose it.
|
|
48
|
+
- **Automatic fallback.** If the current model cannot search, or a provider
|
|
49
|
+
call fails, the extension tries other authenticated provider interfaces until
|
|
50
|
+
one succeeds.
|
|
51
|
+
- **Remembered interfaces.** The successful search and fetch providers are
|
|
52
|
+
cached under pi's agent directory. The next call uses the remembered
|
|
53
|
+
interface when the current model cannot search.
|
|
54
|
+
- **Fresh discovery.** Adding or removing a provider, login, API key, or
|
|
55
|
+
available model changes the cache fingerprint, so old preferences are not
|
|
56
|
+
blindly reused.
|
|
57
|
+
- **Dynamic URL field.** `urls` is exposed only when at least one usable
|
|
58
|
+
provider supports URL analysis. `urls` can be used alone or with `query`.
|
|
59
|
+
- **Raw results.** Provider data is returned in a JSON wrapper without
|
|
60
|
+
deduplication, URL normalization, junk filtering, re-ranking, summarization,
|
|
61
|
+
citation-marker insertion, or appended Sources sections. Only a final
|
|
62
|
+
character guard truncates oversized output.
|
|
63
|
+
- **Correct semantics.** The tool description is exactly
|
|
64
|
+
`Search the web or analyze specified URLs.` The schema makes clear that
|
|
65
|
+
`urls` does not require `query`.
|
|
66
|
+
- **No system prompt changes.** The extension does not add prompt snippets,
|
|
67
|
+
guidelines, loadout hooks, commands, providers, MCP servers, or other tools.
|
|
68
|
+
- **Native DeepSeek Search/Fetch.** DeepSeek uses the documented
|
|
69
|
+
`/v1/tools/search` and `/v1/tools/fetch` endpoints with Bearer auth.
|
|
70
|
+
|
|
71
|
+
## Tool semantics
|
|
72
|
+
|
|
73
|
+
The runtime schema is logically:
|
|
74
|
+
|
|
75
|
+
```ts
|
|
76
|
+
{
|
|
77
|
+
query?: string;
|
|
78
|
+
urls?: string[];
|
|
79
|
+
}
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
- `query` searches the web. Search operators such as `site:`, quotes, `AND`,
|
|
83
|
+
`OR`, and `-term` are passed through unchanged (only outer whitespace is
|
|
84
|
+
trimmed).
|
|
85
|
+
- `urls` analyzes the listed pages. URLs can be used **alone** or together with
|
|
86
|
+
`query`. A query plus URLs performs a search and then analyzes the URLs.
|
|
87
|
+
- At most **20** URLs may be supplied.
|
|
88
|
+
- At least one non-empty `query` or one valid URL is required. This rule is
|
|
89
|
+
enforced at runtime rather than with a top-level `anyOf`, for compatibility
|
|
90
|
+
with all model providers.
|
|
91
|
+
- The `urls` field is hidden when no usable interface supports URL analysis. In
|
|
92
|
+
that configuration `query` is required.
|
|
93
|
+
- Search and URL analysis may use different providers in the same call.
|
|
94
|
+
|
|
95
|
+
Invalid input with no usable query or URL, a missing URL-capable provider for
|
|
96
|
+
supplied URLs, or a call where every search/URL-only provider fails returns a
|
|
97
|
+
tool error (`isError: true`) instead of silently dropping work. Individual
|
|
98
|
+
invalid URLs alongside valid work are reported in the `fetched` entries.
|
|
99
|
+
|
|
100
|
+
## Provider support
|
|
101
|
+
|
|
102
|
+
| Provider / family | Web search | URL analysis | Native interface |
|
|
103
|
+
| --- | --- | --- | --- |
|
|
104
|
+
| DeepSeek | Yes | Yes | Native Search and Fetch: `/v1/tools/search`, `/v1/tools/fetch` |
|
|
105
|
+
| Ollama Cloud | Yes | Yes | `/api/web_search`, `/api/web_fetch` |
|
|
106
|
+
| Google Gemini / Antigravity | Yes | Yes | `google_search` grounding and `url_context` |
|
|
107
|
+
| OpenAI / Azure / Codex / Copilot / OpenCode | Yes | No | Responses API `web_search` tool |
|
|
108
|
+
| xAI Grok | Yes | No | Responses API `web_search` tool |
|
|
109
|
+
| Anthropic Messages | Yes | No | `web_search` server tool |
|
|
110
|
+
|
|
111
|
+
URL analysis is only exposed when at least one usable interface supports it.
|
|
112
|
+
For example, a current OpenAI model with only OpenAI credentials gets a
|
|
113
|
+
query-only schema, while the same session with DeepSeek, Ollama Cloud, or Google
|
|
114
|
+
credentials also gets `urls`.
|
|
115
|
+
|
|
116
|
+
Provider detection follows the authenticated model's provider/API:
|
|
117
|
+
|
|
118
|
+
1. `deepseek` provider
|
|
119
|
+
2. Antigravity or Google Generative AI
|
|
120
|
+
3. Ollama Cloud (`ollama-cloud` provider or exact `ollama.com` host)
|
|
121
|
+
4. xAI with `openai-responses`
|
|
122
|
+
5. OpenAI-compatible Responses APIs (`openai-responses`,
|
|
123
|
+
`azure-openai-responses`, `openai-codex-responses`)
|
|
124
|
+
6. Anthropic Messages
|
|
125
|
+
|
|
126
|
+
See [Providers](docs/providers.md) for request shapes, auth quirks, and URL
|
|
127
|
+
support per family.
|
|
128
|
+
|
|
129
|
+
## Automatic selection and fallback
|
|
130
|
+
|
|
131
|
+
For a search:
|
|
132
|
+
|
|
133
|
+
1. The current conversation model is tried first when its provider family
|
|
134
|
+
supports web search.
|
|
135
|
+
2. A remembered successful provider from the preference cache is tried next
|
|
136
|
+
when the cache fingerprint is current.
|
|
137
|
+
3. The remaining provider families are tried in this fixed order:
|
|
138
|
+
**DeepSeek → Ollama → Google → OpenAI → xAI → Anthropic**.
|
|
139
|
+
4. Within a family, a likely search-capable authenticated model is selected
|
|
140
|
+
first, with a deterministic tie-break. Remaining attempt slots are filled
|
|
141
|
+
with other authenticated provider interfaces, but a failed provider id is
|
|
142
|
+
never retried.
|
|
143
|
+
|
|
144
|
+
URL fetching uses the same idea, but considers only providers that implement
|
|
145
|
+
URL fetching.
|
|
146
|
+
|
|
147
|
+
The current model is always tried first when it supports search. The cache does
|
|
148
|
+
not replace a supported current model; it mainly lets an unsupported current
|
|
149
|
+
model use a remembered compatible provider immediately on later calls.
|
|
150
|
+
|
|
151
|
+
If a provider errors, the extension records the attempt and moves to the next
|
|
152
|
+
interface. Aborts stop immediately. Each attempt has a timeout (60 seconds by
|
|
153
|
+
default, configurable with `PI_BETTER_WEB_SEARCH_TIMEOUT_MS`). If every attempt
|
|
154
|
+
fails, the tool returns an error whose message lists the attempted
|
|
155
|
+
provider/model pairs and their errors. For a URL-only call, the error also
|
|
156
|
+
reports the per-URL failures.
|
|
157
|
+
|
|
158
|
+
### Cache fingerprint invalidation
|
|
159
|
+
|
|
160
|
+
The preference cache is stamped with a fingerprint of all authenticated chat
|
|
161
|
+
models known to pi, derived from sorted `provider/model@baseUrl` values. Adding
|
|
162
|
+
or removing a provider, login, API key, or available model changes the
|
|
163
|
+
fingerprint. A changed fingerprint means:
|
|
164
|
+
|
|
165
|
+
- the old cached provider is **not** used as the first choice;
|
|
166
|
+
- provider discovery starts again from the current model and family priority;
|
|
167
|
+
- the cache is rewritten only after the next successful search/fetch;
|
|
168
|
+
- stale, unverified preferences are not carried into the new fingerprint.
|
|
169
|
+
|
|
170
|
+
A cache entry is only a hint. Credentials are resolved again on every call, and
|
|
171
|
+
the referenced provider must still be available and support the operation.
|
|
172
|
+
|
|
173
|
+
## Cache
|
|
174
|
+
|
|
175
|
+
Default path:
|
|
176
|
+
|
|
177
|
+
```text
|
|
178
|
+
~/.pi/agent/better-web-search.json
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
The path respects pi's `PI_CODING_AGENT_DIR` setting. Override it directly with:
|
|
182
|
+
|
|
183
|
+
```bash
|
|
184
|
+
export PI_BETTER_WEB_SEARCH_CACHE=/path/to/better-web-search.json
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
Example contents:
|
|
188
|
+
|
|
189
|
+
```json
|
|
190
|
+
{
|
|
191
|
+
"version": 1,
|
|
192
|
+
"fingerprint": "0123456789abcdef0123456789abcdef",
|
|
193
|
+
"search": { "provider": "deepseek", "model": "deepseek-chat" },
|
|
194
|
+
"fetch": { "provider": "ollama-cloud", "model": "gpt-oss:120b" }
|
|
195
|
+
}
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
Only provider and model identifiers plus the fingerprint are stored. API keys,
|
|
199
|
+
OAuth tokens, and other credentials are never written to this file.
|
|
200
|
+
|
|
201
|
+
To clear the remembered preferences, delete the file:
|
|
202
|
+
|
|
203
|
+
```bash
|
|
204
|
+
rm ~/.pi/agent/better-web-search.json
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
A missing, unreadable, or corrupt cache is treated as empty and never breaks a
|
|
208
|
+
tool call.
|
|
209
|
+
|
|
210
|
+
## Raw provider results
|
|
211
|
+
|
|
212
|
+
Provider data is returned in a JSON wrapper without cleanup. A search looks like
|
|
213
|
+
this:
|
|
214
|
+
|
|
215
|
+
```json
|
|
216
|
+
{
|
|
217
|
+
"query": "the query",
|
|
218
|
+
"search": {
|
|
219
|
+
"provider": "deepseek",
|
|
220
|
+
"model": "deepseek-chat",
|
|
221
|
+
"answer": "provider text when available",
|
|
222
|
+
"results": "the provider's raw results",
|
|
223
|
+
"citations": "the provider's raw citations",
|
|
224
|
+
"searchQueries": ["provider query"],
|
|
225
|
+
"groundingMetadata": "raw provider metadata"
|
|
226
|
+
},
|
|
227
|
+
"fetched": [
|
|
228
|
+
{
|
|
229
|
+
"url": "https://example.com",
|
|
230
|
+
"provider": "deepseek",
|
|
231
|
+
"model": "deepseek-chat",
|
|
232
|
+
"data": "the provider's raw fetch payload"
|
|
233
|
+
}
|
|
234
|
+
]
|
|
235
|
+
}
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
- Duplicate URLs are not deduplicated.
|
|
239
|
+
- Tracking parameters and junk-looking URLs are not normalized or filtered.
|
|
240
|
+
- Results are not re-ranked or summarized.
|
|
241
|
+
- Citation markers are not inserted, and no "Sources" or "Additional results"
|
|
242
|
+
sections are appended.
|
|
243
|
+
- Redirects are not resolved and sources are not derived.
|
|
244
|
+
- Unknown provider fields are preserved.
|
|
245
|
+
|
|
246
|
+
The only transformation is a final output-size guard. The model-facing JSON is
|
|
247
|
+
truncated to **64,000 characters** by default and marked with `[Truncated]`;
|
|
248
|
+
`details.truncated` is set to `true`. Change the limit with:
|
|
249
|
+
|
|
250
|
+
```bash
|
|
251
|
+
export PI_BETTER_WEB_SEARCH_MAX_CHARS=100000
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
Truncation is Unicode code-point safe. The guard is intentionally applied after
|
|
255
|
+
the raw JSON is assembled so normal-sized provider payloads are not altered.
|
|
256
|
+
|
|
257
|
+
## Exposure rules
|
|
258
|
+
|
|
259
|
+
`web_search` is exposed when either:
|
|
260
|
+
|
|
261
|
+
- the current conversation model supports a provider-native search interface;
|
|
262
|
+
or
|
|
263
|
+
- at least one authenticated available model maps to a supported provider
|
|
264
|
+
family.
|
|
265
|
+
|
|
266
|
+
For an unsupported current model with no compatible login or API key, the tool
|
|
267
|
+
stays hidden (removed from the active tool set). The extension never calls an
|
|
268
|
+
unauthenticated provider. When the set of available providers or credentials
|
|
269
|
+
changes, the extension re-evaluates and re-registers the same `web_search` name
|
|
270
|
+
with the appropriate schema variant.
|
|
271
|
+
|
|
272
|
+
## DeepSeek native Search/Fetch API
|
|
273
|
+
|
|
274
|
+
This package supports DeepSeek's native search and fetch APIs. The older
|
|
275
|
+
project's Anthropic-compatible DeepSeek chat route is **replaced** by this
|
|
276
|
+
native API; DeepSeek models are routed to it even when their underlying chat API
|
|
277
|
+
is Anthropic Messages compatible.
|
|
278
|
+
|
|
279
|
+
```text
|
|
280
|
+
POST https://api.deepseek.com/v1/tools/search
|
|
281
|
+
POST https://api.deepseek.com/v1/tools/fetch
|
|
282
|
+
Authorization: Bearer <DeepSeek API Key>
|
|
283
|
+
Content-Type: application/json
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
Search uses `query`, `count: 10`, and `need_content: true`. The raw `results`
|
|
287
|
+
array is returned unchanged, including `rerank_score`, `title`, `url`,
|
|
288
|
+
`snippet`, `summary`, `content`, `site_name`, `site_icon`, `date`, `authority`,
|
|
289
|
+
and any unknown fields.
|
|
290
|
+
|
|
291
|
+
Fetch sends `{ "url": "..." }` and preserves the raw `{ title, content }`
|
|
292
|
+
response. URLs must be at most 8192 characters, use HTTP/HTTPS, include a host,
|
|
293
|
+
and omit credentials.
|
|
294
|
+
|
|
295
|
+
DeepSeek errors use the OpenAI-compatible error shape. Search errors and global
|
|
296
|
+
fetch errors switch providers; `403` and `404` fetch failures are reported
|
|
297
|
+
against the individual URL so the orchestrator can retry only that URL elsewhere.
|
|
298
|
+
|
|
299
|
+
See [Providers](docs/providers.md#deepseek) for the full request, response, and
|
|
300
|
+
error details.
|
|
301
|
+
|
|
302
|
+
## Security and privacy
|
|
303
|
+
|
|
304
|
+
- Credentials are resolved through pi's model registry on each call. The
|
|
305
|
+
extension does not store API keys or OAuth tokens.
|
|
306
|
+
- The cache stores only provider/model identifiers and a fingerprint.
|
|
307
|
+
- Search queries and requested URLs are sent to the selected provider's API.
|
|
308
|
+
- The extension has no telemetry, analytics, or third-party service of its own.
|
|
309
|
+
- A call may use an authenticated provider other than the current conversation
|
|
310
|
+
model, which can incur usage on that provider.
|
|
311
|
+
- Provider error text and tool results can contain request identifiers or page
|
|
312
|
+
content. Redact them before sharing logs or session exports.
|
|
313
|
+
|
|
314
|
+
See [SECURITY.md](SECURITY.md) for reporting vulnerabilities.
|
|
315
|
+
|
|
316
|
+
## Verified
|
|
317
|
+
|
|
318
|
+
- **116 unit and integration tests** covering exposure, schema variants,
|
|
319
|
+
provider request shapes, SSE parsing, raw-result preservation, fallback,
|
|
320
|
+
cache invalidation, URL handling, truncation, and abort behavior.
|
|
321
|
+
- **Real DeepSeek Search/Fetch run** with a real API key: 10 raw search results,
|
|
322
|
+
summaries and content, valid fetch payloads for normal pages, and documented
|
|
323
|
+
errors for 404 and invalid URLs.
|
|
324
|
+
- **Real pi process end-to-end run**: a mock unsupported current model plus
|
|
325
|
+
DeepSeek credentials exposed `web_search`, called it, auto-switched to the
|
|
326
|
+
native DeepSeek API, returned 10 raw results, and answered with the tool
|
|
327
|
+
result attached.
|
|
328
|
+
- Other provider families are covered by unit/integration tests with mocked
|
|
329
|
+
requests and streams; live credentials for every provider are not required.
|
|
330
|
+
|
|
331
|
+
The opt-in live DeepSeek test is reproducible from a repository checkout:
|
|
332
|
+
|
|
333
|
+
```bash
|
|
334
|
+
DEEPSEEK_API_KEY=... npm run test:live:deepseek
|
|
335
|
+
```
|
|
336
|
+
|
|
337
|
+
## Compatibility
|
|
338
|
+
|
|
339
|
+
- **pi 1.3+**. The extension uses the current extension tool and model registry
|
|
340
|
+
APIs.
|
|
341
|
+
- **Node.js 22.19+**. Pi loads the TypeScript extension directly; there is no
|
|
342
|
+
compile step.
|
|
343
|
+
- Pi provides `@earendil-works/pi-coding-agent`, `@earendil-works/pi-ai`, and
|
|
344
|
+
`typebox` as host packages. They are declared as peer dependencies and are
|
|
345
|
+
not bundled.
|
|
346
|
+
- Upstream provider API changes may require adapter updates. Unknown provider
|
|
347
|
+
fields are preserved rather than rewritten.
|
|
348
|
+
|
|
349
|
+
## Development
|
|
350
|
+
|
|
351
|
+
```bash
|
|
352
|
+
npm ci --ignore-scripts
|
|
353
|
+
npm run check
|
|
354
|
+
npm test
|
|
355
|
+
npm run verify:package
|
|
356
|
+
npm run build
|
|
357
|
+
```
|
|
358
|
+
|
|
359
|
+
`npm run build` runs the checks and tests, verifies the npm file list, and packs
|
|
360
|
+
`dist/pi-better-web-search-0.1.0.tgz`. Nothing is published automatically.
|
|
361
|
+
|
|
362
|
+
The test suite uses plain `.mjs` files that import the `.ts` sources directly.
|
|
363
|
+
It mocks `globalThis.fetch`, uses a temporary cache file, and exercises
|
|
364
|
+
exposure, fallback, cache invalidation, raw-result preservation, truncation,
|
|
365
|
+
URL handling, and provider switching.
|
|
366
|
+
|
|
367
|
+
See [Testing](docs/testing.md) for live-provider tests and the real pi
|
|
368
|
+
end-to-end procedure. See [Publishing](docs/publishing.md) for the release
|
|
369
|
+
checklist.
|
|
370
|
+
|
|
371
|
+
## Contributing
|
|
372
|
+
|
|
373
|
+
Use English for issues, pull requests, and primary documentation. Chinese
|
|
374
|
+
translations are welcome. Read [CONTRIBUTING.md](CONTRIBUTING.md) before
|
|
375
|
+
changing provider behavior, exposure rules, or output handling.
|
|
376
|
+
|
|
377
|
+
The extension keeps one invariant above all others: **trust the provider's
|
|
378
|
+
result**. Do not add cleanup that rewrites, filters, re-ranks, summarizes, or
|
|
379
|
+
reconstructs provider data.
|
|
380
|
+
|
|
381
|
+
## License
|
|
382
|
+
|
|
383
|
+
MIT. An independent community package, not an official Pi product.
|
|
384
|
+
|
|
385
|
+
The `web_search` extraction is derived from the MIT-licensed
|
|
386
|
+
[`pi-web-search`](https://github.com/ttttmr/pi-web-search) project. The
|
|
387
|
+
repository structure and documentation conventions are inspired by
|
|
388
|
+
[`pi-trim`](https://github.com/zhexusun10/pi-trim).
|
package/SECURITY.md
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# Security
|
|
2
|
+
|
|
3
|
+
## Supported versions
|
|
4
|
+
|
|
5
|
+
Security fixes target the latest `0.1.x` release.
|
|
6
|
+
|
|
7
|
+
## Reporting
|
|
8
|
+
|
|
9
|
+
Do not open a public issue for a vulnerability. Use GitHub's private
|
|
10
|
+
vulnerability reporting:
|
|
11
|
+
|
|
12
|
+
https://github.com/zhexusun10/pi-better-web-search/security/advisories/new
|
|
13
|
+
|
|
14
|
+
Include the affected version, pi version, provider, reproduction steps, and
|
|
15
|
+
whether credentials or private data may have been exposed. Redact API keys,
|
|
16
|
+
OAuth tokens, and private prompts.
|
|
17
|
+
|
|
18
|
+
## Credentials and data flow
|
|
19
|
+
|
|
20
|
+
`pi-better-web-search` resolves credentials through pi's model registry on each
|
|
21
|
+
call. It does not store API keys or OAuth tokens. The preference cache contains
|
|
22
|
+
only provider/model identifiers and a hash fingerprint of the available model
|
|
23
|
+
set.
|
|
24
|
+
|
|
25
|
+
Search queries and requested URLs are sent to the selected provider's API. The
|
|
26
|
+
extension has no telemetry, analytics, or third-party service of its own. It may
|
|
27
|
+
use any authenticated compatible provider, so a call can incur usage on a
|
|
28
|
+
provider other than the current conversation model. Review your provider
|
|
29
|
+
accounts and permissions before enabling the extension.
|
|
30
|
+
|
|
31
|
+
## Logs and issues
|
|
32
|
+
|
|
33
|
+
Provider error text can contain request identifiers or response snippets. Tool
|
|
34
|
+
results can contain page content. Remove secrets and private data before
|
|
35
|
+
sharing logs, tool results, session exports, or cache files in an issue.
|
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="640" viewBox="0 0 1200 640" role="img" aria-labelledby="title desc">
|
|
2
|
+
<title id="title">pi-better-web-search architecture</title>
|
|
3
|
+
<desc id="desc">One web_search tool routes to provider-native search APIs, falls back across authenticated providers, and returns raw provider JSON with a size guard.</desc>
|
|
4
|
+
<defs>
|
|
5
|
+
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
|
|
6
|
+
<path d="M 0 0 L 10 5 L 0 10 z" fill="#52525b"/>
|
|
7
|
+
</marker>
|
|
8
|
+
<style>
|
|
9
|
+
.bg { fill: #ffffff; }
|
|
10
|
+
.panel { fill: #fafafa; stroke: #e4e4e7; stroke-width: 1.5; }
|
|
11
|
+
.box { fill: #ffffff; stroke: #18181b; stroke-width: 1.6; }
|
|
12
|
+
.accent { fill: #18181b; stroke: #18181b; }
|
|
13
|
+
.dash { fill: none; stroke: #a1a1aa; stroke-width: 1.4; stroke-dasharray: 6 6; }
|
|
14
|
+
.title { font: 700 40px Arial, Helvetica, sans-serif; fill: #111111; }
|
|
15
|
+
.subtitle { font: 400 20px Arial, Helvetica, sans-serif; fill: #52525b; }
|
|
16
|
+
.h { font: 700 18px Arial, Helvetica, sans-serif; fill: #18181b; }
|
|
17
|
+
.t { font: 400 15px Arial, Helvetica, sans-serif; fill: #3f3f46; }
|
|
18
|
+
.muted { font: 400 14px Arial, Helvetica, sans-serif; fill: #71717a; }
|
|
19
|
+
.inv-h { font: 700 18px Arial, Helvetica, sans-serif; fill: #ffffff; }
|
|
20
|
+
.inv-t { font: 400 14px Arial, Helvetica, sans-serif; fill: #e4e4e7; }
|
|
21
|
+
.arrow { stroke: #52525b; stroke-width: 2; fill: none; marker-end: url(#arrow); }
|
|
22
|
+
.label { font: 600 13px Arial, Helvetica, sans-serif; fill: #52525b; }
|
|
23
|
+
.footer { font: 400 14px Arial, Helvetica, sans-serif; fill: #71717a; }
|
|
24
|
+
</style>
|
|
25
|
+
</defs>
|
|
26
|
+
|
|
27
|
+
<rect class="bg" x="0" y="0" width="1200" height="640"/>
|
|
28
|
+
<rect class="panel" x="20" y="20" width="1160" height="600" rx="22"/>
|
|
29
|
+
|
|
30
|
+
<text class="title" x="56" y="82">pi-better-web-search</text>
|
|
31
|
+
<text class="subtitle" x="56" y="116">one tool · provider-native search · raw results</text>
|
|
32
|
+
|
|
33
|
+
<rect class="box" x="56" y="166" width="226" height="136" rx="16"/>
|
|
34
|
+
<text class="h" x="78" y="198">Current pi model</text>
|
|
35
|
+
<text class="t" x="78" y="226">supports search?</text>
|
|
36
|
+
<text class="muted" x="78" y="252">yes → use it first</text>
|
|
37
|
+
<text class="muted" x="78" y="274">no → fallback</text>
|
|
38
|
+
|
|
39
|
+
<path class="arrow" d="M 286 234 L 334 234"/>
|
|
40
|
+
<text class="label" x="292" y="222">route</text>
|
|
41
|
+
|
|
42
|
+
<rect class="box" x="342" y="166" width="248" height="136" rx="16"/>
|
|
43
|
+
<text class="h" x="364" y="198">Candidate router</text>
|
|
44
|
+
<text class="t" x="364" y="226">current → cached →</text>
|
|
45
|
+
<text class="t" x="364" y="248">family priority</text>
|
|
46
|
+
<text class="muted" x="364" y="274">fingerprint-aware cache</text>
|
|
47
|
+
|
|
48
|
+
<path class="arrow" d="M 594 234 L 642 234"/>
|
|
49
|
+
<text class="label" x="600" y="222">try in order</text>
|
|
50
|
+
|
|
51
|
+
<text class="h" x="650" y="126">provider adapters</text>
|
|
52
|
+
<rect class="dash" x="642" y="140" width="258" height="318" rx="18"/>
|
|
53
|
+
|
|
54
|
+
<rect class="accent" x="654" y="152" width="234" height="52" rx="12"/>
|
|
55
|
+
<text class="inv-h" x="672" y="175">DeepSeek</text>
|
|
56
|
+
<text class="inv-t" x="672" y="194">native search + fetch</text>
|
|
57
|
+
|
|
58
|
+
<rect class="box" x="654" y="214" width="234" height="52" rx="12"/>
|
|
59
|
+
<text class="h" x="672" y="237">Ollama Cloud</text>
|
|
60
|
+
<text class="t" x="672" y="256">search + fetch</text>
|
|
61
|
+
|
|
62
|
+
<rect class="box" x="654" y="276" width="234" height="52" rx="12"/>
|
|
63
|
+
<text class="h" x="672" y="299">Google Gemini</text>
|
|
64
|
+
<text class="t" x="672" y="318">search + URL context</text>
|
|
65
|
+
|
|
66
|
+
<rect class="box" x="654" y="338" width="234" height="52" rx="12"/>
|
|
67
|
+
<text class="h" x="672" y="361">OpenAI / xAI</text>
|
|
68
|
+
<text class="t" x="672" y="380">Responses web search</text>
|
|
69
|
+
|
|
70
|
+
<rect class="box" x="654" y="400" width="234" height="52" rx="12"/>
|
|
71
|
+
<text class="h" x="672" y="423">Anthropic</text>
|
|
72
|
+
<text class="t" x="672" y="442">Messages web search</text>
|
|
73
|
+
|
|
74
|
+
<path class="arrow" d="M 906 234 L 954 234"/>
|
|
75
|
+
<text class="label" x="912" y="222">raw</text>
|
|
76
|
+
|
|
77
|
+
<rect class="box" x="962" y="166" width="178" height="136" rx="16"/>
|
|
78
|
+
<text class="h" x="982" y="198">Tool result</text>
|
|
79
|
+
<text class="t" x="982" y="226">raw JSON</text>
|
|
80
|
+
<text class="muted" x="982" y="252">no cleanup</text>
|
|
81
|
+
<text class="muted" x="982" y="274">64k char guard</text>
|
|
82
|
+
|
|
83
|
+
<rect class="panel" x="56" y="492" width="1084" height="94" rx="16"/>
|
|
84
|
+
<text class="h" x="78" y="522">Exposure & memory</text>
|
|
85
|
+
<text class="t" x="78" y="548">Hidden when the current model has no search interface and no compatible provider is authenticated.</text>
|
|
86
|
+
<text class="t" x="78" y="570">urls appears only when a usable provider supports URL analysis. Cache stores provider/model + fingerprint, never keys.</text>
|
|
87
|
+
|
|
88
|
+
<text class="footer" x="56" y="610">Pi 1.3+ · Node 22.19+ · MIT · github.com/zhexusun10/pi-better-web-search</text>
|
|
89
|
+
</svg>
|