pi-better-web-search 0.1.9 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (46) hide show
  1. package/CHANGELOG.md +168 -0
  2. package/README.md +277 -65
  3. package/SECURITY.md +31 -4
  4. package/assets/architecture.svg +43 -21
  5. package/assets/search-index.png +0 -0
  6. package/docs/README.zh-CN.md +230 -50
  7. package/docs/architecture.md +151 -43
  8. package/docs/providers.md +66 -11
  9. package/docs/publishing.md +9 -8
  10. package/docs/search-apis.md +250 -0
  11. package/docs/search-skills.md +265 -0
  12. package/docs/testing.md +86 -11
  13. package/extensions/index.ts +9 -0
  14. package/lib/api-command.ts +84 -0
  15. package/lib/atomic-file.ts +32 -0
  16. package/lib/attempt-timeout.ts +82 -0
  17. package/lib/auth.ts +9 -0
  18. package/lib/cache.ts +24 -16
  19. package/lib/candidates.ts +137 -112
  20. package/lib/credentials.ts +98 -0
  21. package/lib/dedicated-config.ts +61 -0
  22. package/lib/format.ts +14 -0
  23. package/lib/health.ts +24 -5
  24. package/lib/login-command.ts +87 -0
  25. package/lib/preferences.ts +62 -0
  26. package/lib/provider-command.ts +66 -0
  27. package/lib/providers/anthropic.ts +7 -4
  28. package/lib/providers/dedicated-http.ts +80 -0
  29. package/lib/providers/deepseek.ts +77 -10
  30. package/lib/providers/exa.ts +66 -0
  31. package/lib/providers/google.ts +4 -3
  32. package/lib/providers/http.ts +99 -47
  33. package/lib/providers/index.ts +9 -0
  34. package/lib/providers/keenable.ts +61 -0
  35. package/lib/providers/ollama.ts +13 -6
  36. package/lib/providers/openai.ts +15 -7
  37. package/lib/providers/perplexity.ts +20 -0
  38. package/lib/providers/sse.ts +93 -54
  39. package/lib/providers/types.ts +6 -2
  40. package/lib/redact.ts +15 -0
  41. package/lib/render.ts +41 -0
  42. package/lib/routes.ts +28 -0
  43. package/lib/schema.ts +26 -10
  44. package/lib/search.ts +123 -63
  45. package/lib/secret-input.ts +63 -0
  46. package/package.json +10 -4
package/CHANGELOG.md CHANGED
@@ -1,5 +1,173 @@
1
1
  # Changelog
2
2
 
3
+ ## 1.0.0 — Search is an agent superpower
4
+
5
+ ### The flagship search upgrade
6
+
7
+ - Bring **Keenable, Exa, and Perplexity**, the top three independent providers
8
+ in the Artificial Analysis Search Index snapshot, plus **DeepSeek native
9
+ Search/Fetch**, directly into pi without requiring a conversation-model change.
10
+ - Lead the English/Chinese READMEs with the search advantage: **33 model-only →
11
+ 83 with Keenable Pro (+50 points)**. Show why agents should find existing
12
+ libraries, official docs, and proven fixes before spending tokens reinventing
13
+ them. Add a sourced Search Index chart and redraw the architecture for four
14
+ standalone APIs, native fallback, terminal login, and the 96k output guard.
15
+
16
+ ### Regression fixes
17
+
18
+ - Stop OpenAI/xAI streams on completed/done/incomplete terminal events and
19
+ Anthropic streams on `message_stop`, cancelling connections that remain open
20
+ instead of timing out and discarding usable search results.
21
+ - Redact native-provider transport, SSE and JSON errors using a snapshot of sent
22
+ credential headers (including custom API tokens and Bearer/Basic auth). Keep
23
+ status/code metadata for fallback, scrub structured error bodies, and suppress
24
+ JSON parser snippets that could expose a partial credential.
25
+ - Require Anthropic server-tool evidence to name `web_search`; unrelated tools
26
+ no longer produce false search success. Skip non-object stream frames safely.
27
+ - Atomically replace settings and the success cache after writing/syncing a
28
+ same-directory temporary file. Partial-write and rename failures preserve the
29
+ previous file; configured file symlinks continue to update their targets.
30
+ - Recheck Keenable public-access consent before each adapter request, including
31
+ combined calls and URL batches. Key loss/logout no longer downgrades an
32
+ in-flight operation to public endpoints; preserve completed search/pages and
33
+ resolve auth-dependent endpoint paths from the same key snapshot as headers.
34
+ - Recognize aborts by signal, structured identity or canonical legacy messages,
35
+ not arbitrary text containing `abort`. Malformed page responses no longer
36
+ discard completed URLs, and dedicated transport AbortErrors remain redacted
37
+ even if credentials rotate while the request is in flight.
38
+ - Cancel stalled response bodies and responses arriving after an attempt timeout,
39
+ even when a custom transport ignores the signal. Ignore late adapter updates
40
+ after timeout/completion and prevent disposed deadlines from being restarted.
41
+ - Bound individual SSE frames to 4,194,304 UTF-16 code units, including unfinished
42
+ lines and multiline data, while accepting larger streams of bounded frames. Scan
43
+ fragmented lines incrementally instead of repeatedly rescanning their prefix.
44
+ - Share literal/JSON-escaped credential redaction across dedicated API errors,
45
+ native DeepSeek errors and Exa per-URL errors. Also redact HTTP status text and
46
+ structured error codes using the credentials actually sent.
47
+ - Keep routing priorities strict: **manual preference → Keenable → Exa →
48
+ Perplexity → standalone DeepSeek → native automatic routing**. Manual choices
49
+ get a first probe even after failure or Retry-After; cooldowns reorder only
50
+ remaining automatic model routes. Standalone DeepSeek failures do not suppress
51
+ native DeepSeek candidates using separate authentication.
52
+ - Route standalone DeepSeek through shared dedicated HTTP validation/backoff:
53
+ honor credential-specific `Retry-After` across Search/Fetch, reject malformed
54
+ header keys before HTTP, and leave native model/proxy auth unchanged.
55
+ - Redact Exa per-URL errors using the key actually sent even if login rotates it
56
+ in flight; honor canonical-URL error statuses that use `url` instead of `id`.
57
+ - Release bounded HTTP error streams without awaiting an underlying cancellation
58
+ promise that may never settle, so errors can reach fallback immediately.
59
+ - Isolate the live DeepSeek test's settings/cache and disable standalone API
60
+ discovery so other saved/environment search keys cannot trigger unrelated
61
+ provider calls or override the intended native DeepSeek fallback check.
62
+
63
+ ### Implementation details
64
+
65
+ - Explicitly provide no skills: remove vendor skill copies and their lockfile
66
+ from the checkout, retain only developer reference notes, and reject skill
67
+ directories/entrypoints in package verification. Keep normal tool descriptions,
68
+ schemas and API behavior unchanged; no automatic vendor skill context is added.
69
+ - Refine both description variants to address insufficient or conflicting
70
+ evidence and cite the source URLs actually used; make fetch conditional on
71
+ need in the URL-capable version, without adding fetch to search-only guidance.
72
+ - Improve capability-aware tool/field descriptions with natural-language
73
+ query guidance, source verification/citation, and optional evidence-driven
74
+ search refinement. Both variants start with the provider-neutral
75
+ `Search the live web and return the most relevant URLs and content.`; only
76
+ URL-capable variants instruct fetching pages. Keep `query` / `urls`, existing validation, adapter requests
77
+ and system prompts unchanged. Document skill wording, provider-specific length
78
+ limits and agent-driven chains without adding a fixed search loop.
79
+ - Add `/pi-better-web-search login`: select DeepSeek, Exa, Keenable or Perplexity
80
+ and enter an API key in a masked local terminal dialog. Persist keys separately
81
+ with restricted permissions, cross-process locking and atomic replacement;
82
+ refresh tool exposure immediately without environment setup or a chat model.
83
+ Saved keys override environment keys; preserve manual routing and fallback.
84
+ - Add safe presence-only `status` and confirmed `logout` (disables standalone
85
+ access without silently switching Keenable to public mode). Refuse RPC/print
86
+ secret entry and key arguments; preserve cancellation and existing credentials
87
+ on save failures. Credential files are local plaintext, not encrypted.
88
+ - Add standalone Exa Search/Fetch with `EXA_API_KEY` and `/web-search-api exa`.
89
+ Search sends the approved `numResults: 10` and `contents.highlights: true`;
90
+ known URLs use `POST /contents` with top-level `text: true`. Inspect per-URL
91
+ statuses even on HTTP 200, preserve successful raw documents, and retry only
92
+ unresolved URLs through existing fallback. Keep safe error redaction,
93
+ cancellation, Retry-After, API priority and the `query` / `urls` contract.
94
+ Document Exa/Keenable skill differences; objective and other advanced options
95
+ remain off; no Exa-specific parameter or system-prompt change is introduced.
96
+ - Add standalone Perplexity Search (`POST /search`, `search_type: web`, never
97
+ fast, no model/effort, no Fetch) and Keenable pro Search/live GET Fetch.
98
+ Preserve raw result objects and use existing timeout, cooldown, fallback and
99
+ partial-success handling. Configured APIs precede automatic model routing;
100
+ explicit per-phase provider preferences still win.
101
+ - Add `/web-search-api [deepseek|exa|keenable|perplexity]` to enable/disable APIs,
102
+ view safe key setup, and opt into Keenable public access with privacy consent.
103
+ This menu saves toggles only; masked login stores credentials separately. Honor Retry-After
104
+ across dedicated search/fetch calls, and add an opt-in status/shape-only smoke
105
+ test: `npm run test:live:search-apis -- all`. No new runtime dependency or MCP.
106
+ - Preserve successful search and URL-fetch data independently. Missing fetch
107
+ capability or fetch failures no longer replace successful search output with
108
+ a whole-call error; failed search (including query-length rejection) no longer
109
+ suppresses requested fetch work. Return partial-success JSON with `search.error`
110
+ and/or per-URL errors, and mark the whole call failed only if no operation
111
+ succeeded. Keep partial search failures visible in collapsed terminal results.
112
+ - Add `/web-search-provider [search|fetch]` terminal pickers with independent,
113
+ persistent search/fetch preferences and an Auto reset. Choices precede
114
+ automatic hints without disabling fallback, per-URL retries, or health
115
+ cooldowns; success-cache updates never overwrite explicit preferences.
116
+
117
+ - Clarify that providing `query` triggers a separate search even with `urls`;
118
+ show URL-only omission guidance only when URL support is exposed. Reject
119
+ additional input arguments in both schemas and at runtime, require non-empty
120
+ URL arrays when supplied, and constrain URL strings to 1–8192 Unicode code
121
+ points consistently with native DeepSeek validation.
122
+ - Strictly limit search and URL analysis candidates to OpenAI `luna`, Claude
123
+ `haiku`, and other brands' `flash` models. Select the newest numeric version
124
+ within the eligible brand/tier, including gateways; current-model and cache
125
+ hints now select a provider instead of pinning an older or larger model.
126
+ Never fall back to an excluded tier, and hide the tool without an eligible
127
+ model or configured standalone API.
128
+ - Compare dotted/dashed Claude minor versions numerically and ignore date
129
+ suffixes independently of the provider API family.
130
+ - Distinguish Search, Fetch, and combined operations in the terminal. Display
131
+ the actual DeepSeek/Ollama standalone API endpoint instead of an inference
132
+ model name; add `details.routes` and progress `details.activeRoute` metadata.
133
+ Raw provider output is unchanged and available in the expanded result.
134
+ - Preserve current-provider priority even when the conversation model's API is
135
+ unsupported, and derive discovery/cache fingerprints from one registry snapshot.
136
+ - Preserve completed DeepSeek/Ollama URL results after a later global HTTP error;
137
+ only unresolved URLs proceed to fallback. Show failed URL counts and truncation
138
+ warnings in collapsed results.
139
+ - Validate timeout overrides against Node's integer timer range and observe
140
+ adapter rejections even when the signal is already aborted.
141
+ - Handle SSE CR/LF/CRLF boundaries, data-less event resets, `[DONE]`, and pending
142
+ read cancellation without waiting for the provider connection to close.
143
+
144
+ ### Regression coverage and validation
145
+
146
+ - Add 62 regression tests across response/deadline lifecycle, late adapter
147
+ updates, bounded/fragmented SSE frames, credential redaction and rotation,
148
+ atomic settings/cache writes, runtime Keenable consent, abort classification,
149
+ native terminal events and search-evidence validation.
150
+ - Add four platform-independent simulated symlink tests for settings/cache target
151
+ updates and dangling-link refusal. Keep the real filesystem checks separately.
152
+ - Verify on Node 24: 386 tests total, 384 passed, no failures; two real file-symlink
153
+ tests skipped because Windows returns `EPERM` when creating file symlinks.
154
+ - Type checking and package verification pass. Provider requests are mocked;
155
+ these checks do not call billable live APIs.
156
+
157
+ ## 0.1.11
158
+
159
+ - Increase the default timeout to 120 seconds per search and URL fetch attempt
160
+ for all providers. Provider errors still trigger fallback immediately, and
161
+ `PI_BETTER_WEB_SEARCH_TIMEOUT_MS` can override the default.
162
+
163
+ ## 0.1.10
164
+
165
+ - Verify against Pi 1.5.2 and document support for the OpenAI ChatGPT OAuth
166
+ login as well as OpenAI API keys and legacy Codex OAuth.
167
+ - Use the lowest model-supported reasoning effort for OpenAI Responses search,
168
+ independently of the conversation's thinking level, as in pi-web-search.
169
+ - Document that Pi virtual models are not directly supported.
170
+
3
171
  ## 0.1.9
4
172
 
5
173
  - Preserve provider diversity and apply cooldowns before limiting actual search