pi-web-kit 0.2.4 → 0.4.0

package/CHANGELOG.md

@@ -6,6 +6,46 @@ This project follows the spirit of [Keep a Changelog](https://keepachangelog.com
 
 ## [Unreleased]
 
+## [0.4.0] - 2026-10-09
+
+### Added
+
+- Declare bounded structured results for all five research tools, plus `web` namespace discovery and behavior annotations. Codemode now receives typed objects without `JSON.parse`; direct JSON text remains available.
+
+### Fixed
+
+- Project provider responses to public fields and mask credentials consistently across text, structured data, details and progress. Omit arbitrary metadata, Context7 rules and raw backend error bodies.
+- Restrict Basic/Bearer masking to authorization-header values so ordinary authentication documentation remains readable, while short header credentials stay masked.
+- Honor explicit default-tool exclusions during late registration and retain manual deactivation across reload instead of force-activating optional research tools.
+- Activate newly available optional tools when credentials arrive on reload, including existing positive selections, without reactivating previously disabled tools. Remember only fixed tool names in non-model session metadata; older sessions without that history use a conservative first-reload fallback.
+- Reject cancelled calls even when a provider converts cancellation into per-page failure data.
+
+### Changed
+
+- Update the shared Pi development and contract-test baseline to 1.1.0; require Node.js >=22.19.0 to match the host runtime. Pi remains a host-supplied peer dependency.
+
+## [0.3.0] - 2026-08-31
+
+### Changed
+
+- Share install telemetry mechanics through `@mocito/install-telemetry` while preserving Pi-specific settings and state paths.
+- Treat `web_search.numResults` as a provider-agnostic desired limit, cap it to each provider's service constraints, and report requested, effective, returned, and omitted counts.
+- Add provider-agnostic `contextTokens` and `purpose` search controls, preserve provider-native grounding content, and allocate one bounded context budget across ranked results.
+- Use adaptive extraction: native Exa/Brave context is retained automatically, while TinyFish fetch and Firecrawl scrape-on-search run only when expanded context is requested.
+- Expose documented TinyFish search/fetch controls and Brave LLM Context controls.
+- Add provider-agnostic `maxAgeMs`, pass refresh/cache intent through to Exa, TinyFish, and Firecrawl, and request complete bounded pages from Exa fetch.
+
+### Fixed
+
+- Let `enableInstallTelemetry: false` override an enabled `PI_TELEMETRY` environment flag.
+- Paginate TinyFish searches instead of sending its ignored `limit` parameter.
+- Send markdown.new's documented `retain_images` field and report its response metadata.
+- Match TinyFish fetch responses and per-URL errors by canonical URL instead of response position.
+
+### Removed
+
+- Remove the Exa MCP search/fetch surface and make the direct Exa API the default. Existing `exa_mcp` configurations must select `exa` and provide `EXA_API_KEY`.
+
 ## [0.2.4] - 2026-07-28
 
 ### Changed

package/README.md

@@ -10,6 +10,7 @@ Give [Pi](https://pi.dev) current web knowledge, authoritative library docs, and
 - **Use docs that match the task** — resolve libraries and retrieve focused, version-aware documentation with code examples.
 - **Find proven implementation patterns** — search practical usage, setup, migrations, and error context across real code.
 - **Spend context wisely** — compact search results, chunked page reads, bounded output, and fetch caching keep research useful without overwhelming the model.
+- **Compose research in scripts** — typed results let codemode filter and join data without parsing model-facing text.
 - **Choose your providers** — mix Exa, TinyFish, Brave, Firecrawl, markdown.new, Context7, and Exa Code based on coverage, cost, and credentials.
 
 ## Installation
@@ -39,6 +40,7 @@ pi -e /path/to/pi-mono/packages/pi-web-kit --web-provider-fetch markdown_new --p
 ```
 
 This is an npm-compatible TypeScript Pi package. Bun is not required.
+Use Pi 1.1.0 or newer and Node.js >=22.19.0 for the structured tool contracts.
 
 ## Quick usage
 
@@ -70,11 +72,10 @@ Pi chooses `web_search` or `web_fetch` automatically when the request calls for
 
 ## Providers
 
-Defaults: `provider_search = "exa_mcp"`, `provider_fetch = "exa_mcp"`.
+Defaults: `provider_search = "exa"`, `provider_fetch = "exa"`.
 
 | Provider | Search | Fetch | Key |
 |---|---:|---:|---|
-| `exa_mcp` | yes | yes | optional `EXA_API_KEY` |
 | `exa` | yes | yes | `EXA_API_KEY` |
 | `tinyfish` | yes | yes | `TINYFISH_API_KEY` |
 | `brave` | yes | no | `BRAVE_SEARCH_API_KEY` |
@@ -83,6 +84,18 @@ Defaults: `provider_search = "exa_mcp"`, `provider_fetch = "exa_mcp"`.
 
 Tool schemas are tailored to the configured providers at startup/reload, so only supported provider-specific fields are exposed. Restart/reload Pi after changing provider config.
 
+Provider-native efficiencies are used as follows:
+
+| Service | Efficient path |
+|---|---|
+| Exa API | Keep included highlights; request bounded text in the same search only when requested; batch `/contents` fetches with freshness controls. |
+| TinyFish | Paginate only as needed; use dedicated filters; batch only enough top-result fetches to fill requested context; pass TTL and intent. |
+| Brave | Return pre-extracted LLM Context in one search call with native URL, token, snippet, threshold, and Goggles controls. |
+| Firecrawl | Return search metadata by default; use scrape-on-search only when requested; pass cache/main-content options. |
+| markdown.new | Keep `method: auto` so native Markdown falls back to AI and browser rendering only as needed; keep images opt-in. |
+| Context7 | Support `fast` mode to skip reranking and reduce latency. |
+| Exa Code | Send the requested/dynamic context token target directly to Exa Code Context. |
+
 ## Configuration
 
 Resolution order: defaults < environment variables < global config < trusted project config < CLI flags. Project config is ignored unless Pi trusts the current project, including in print, JSON, and RPC modes.
@@ -92,8 +105,8 @@ Resolution order: defaults < environment variables < global config < trusted pro
 ```bash
 PI_OFFLINE=1        # disables install/update telemetry
 PI_TELEMETRY=0      # disables install/update telemetry
-PI_WEB_KIT_PROVIDER_SEARCH=exa_mcp|exa|tinyfish|brave|firecrawl
-PI_WEB_KIT_PROVIDER_FETCH=exa_mcp|exa|tinyfish|markdown_new|firecrawl
+PI_WEB_KIT_PROVIDER_SEARCH=exa|tinyfish|brave|firecrawl
+PI_WEB_KIT_PROVIDER_FETCH=exa|tinyfish|markdown_new|firecrawl
 EXA_API_KEY=...          # enables Exa provider and code_search
 CONTEXT7_API_KEY=...     # enables library_search and library_docs
 TINYFISH_API_KEY=...
@@ -148,9 +161,15 @@ Searches with the active search provider and returns compact results grouped by
 |---|---|---|
 | `query` | string | Single search query. |
 | `queries` | string[] | Multiple related search queries. Max 5 after de-duplication. |
-| `numResults` | integer | Results per query. Range: 1-20. Default: 10. |
+| `numResults` | integer | Desired results per query. Default: 10. Values above the active provider's limit are capped rather than rejected. |
+| `contextTokens` | integer | Desired extracted context across the result set. Omitted native context uses an 8,192-token output budget; an explicit value can enable extraction. Any positive value; capped at 10,000. |
+| `purpose` | string | Optional task/use-case hint for providers that support separate intent. |
+
+`numResults` controls source breadth. `contextTokens` controls grounding depth. The tool automatically keeps native/included Exa highlights and Brave LLM Context. Explicit `contextTokens` enables extra extraction for Exa, TinyFish, and Firecrawl; this can add provider calls or provider cost. Search results keep a compact `snippet` plus ranked `content` and `contentFormat`, with one shared context budget and the existing 50KB tool-output limit.
+
+Provider caps are Exa 100, Brave 50, and Firecrawl 100. TinyFish is paginated internally through its service maximum of page 10 and may make up to 11 search requests for one query. Search output reports requested, effective, returned, and omitted result/context counts. Brave `maxUrls` remains as a deprecated alias for `numResults`.
 
-Provider-specific parameters are exposed only for the configured provider, such as Exa date/domain filters, TinyFish `page`, Brave locale/freshness options, or Firecrawl scrape/search options.
+Other provider-specific parameters are exposed only for the configured provider. These include Exa date/domain filters; TinyFish domain, date, geography, language, and publication filters; Brave locale, freshness, spellcheck, Goggles, and LLM Context controls; and Firecrawl scrape/search options.
 
 ### `web_fetch`
 
@@ -163,8 +182,9 @@ Fetches page content with the active fetch provider. Results are cached in memor
 | `offset` | integer | Character offset for cached/ranged reads. Single URL only. |
 | `limit` | integer | Maximum characters to return. Default: 30,000 for one URL, 8,000 for multiple URLs. |
 | `refresh` | boolean | Refetch even if cached. |
+| `maxAgeMs` | integer | Desired maximum local/provider-cached page age in milliseconds. `0` requests live content where supported. |
 
-Provider-specific parameters are exposed only for the configured provider, such as TinyFish `format`, markdown.new `method` / `retainImages`, or Firecrawl `format`, `waitFor`, `mobile`, `location`, and `maxAge`.
+Provider-specific parameters are exposed only for the configured provider. TinyFish supports `purpose`, `format`, links/images, selectors, per-URL timeout, and a seconds-based `ttl` alias. Exa supports the hours-based `maxAgeHours` alias. markdown.new supports `method` / `retainImages`. Firecrawl supports `format`, `waitFor`, `mobile`, structured `location`, and its existing `maxAge` alias. `refresh: true` also requests live content from Exa, TinyFish, and Firecrawl instead of only bypassing the local cache.
 
 ### `library_search`
 
@@ -210,16 +230,93 @@ Finds practical code examples, implementation context, setup snippets, migration
 | Max cached bytes | 20 MiB |
 | Max URLs per call | 10 |
 | Max queries per call | 5 |
-| Max `numResults` | 20 |
+| Provider `numResults` caps | Exa 100; Brave 50; Firecrawl 100 |
+| Search context budget | 10,000 tokens |
 | Max URL length | 2048 characters |
 
 Cache keys include the provider, canonical URL, fetch-affecting parameters, relevant provider defaults, and an opaque SHA-256 API-key/account scope. Internal cache keys are never returned in tool output. `refresh: true` bypasses and replaces the cached entry.
 
+## Structured results and discovery
+
+All five tools declare `outputSchema` and return objects through `structuredContent`.
+Codemode receives those objects directly; remove old `JSON.parse(await tools.…())`
+wrappers. Direct calls still return useful JSON text with exactly the same
+bounded, redacted value. Renderer `details` remain compact and are not the
+script API.
+
+| Tool | Script result |
+| --- | --- |
+| `web_search` | `{ provider, queries: [{ query, requestedResultLimit, effectiveResultLimit, requestedContextTokens?, effectiveContextTokens, contextCharacters, omittedContextCharacters?, resultCount, omittedResultCount?, results }] }`. Each result has `url` and optional `title`, `snippet`, `content`, `contentFormat`, `siteName`, `position`. |
+| `web_fetch` | `{ provider, results }`. Each item is either `{ url, error }` or `{ url, fetchedUrl, title?, content, format, cached, refreshed, range }`. `range` has `offset`, `limit`, `returned`, `total`, `truncated`, `hasPrevious`, `hasNext`, and optional `nextOffset`. |
+| `library_search` | `{ provider: "context7", libraryName, query, searchFilterApplied?, results }`. Each candidate has `id`; title, description, branch, dates, state, scores, counts, stars and versions are optional. |
+| `library_docs` | `{ provider: "context7", libraryId, query, codeSnippets, infoSnippets }`. Code snippets expose optional title/description/language/ID/page/source/token fields and `codeList: [{ language, code }]`; info snippets require `content` with optional page/breadcrumb/token fields. |
+| `code_search` | `{ provider: "exa", query, response, resultsCount?, searchTime?, outputTokens?, requestId? }`. Provider token counts are informational, not a second Pi usage report. |
+
+Every result also allows the top-level fallback `{ truncated: true, message }`
+when metadata alone cannot fit. Check this before reading ordinary fields.
+Both text and structured payloads stay within the same 50,000-byte JSON budget.
+Fetch slices retain range continuation; search results report omitted counts.
+Missing/invalid optional provider fields are omitted. Invalid required fields
+fail the call rather than yielding a guessed success object.
+
+```js
+const search = await tools.web_search({ query: "Pi extension API", numResults: 3 });
+if ("truncated" in search) throw new Error(search.message);
+const urls = search.queries.flatMap(q => q.results.map(r => r.url));
+if (urls.length) {
+  const pages = await tools.web_fetch({ urls: urls.slice(0, 10) });
+  if ("truncated" in pages) throw new Error(pages.message);
+  text(pages.results.filter(r => !("error" in r)).map(r => ({
+    url: r.url, excerpt: r.content.slice(0, 500), nextOffset: r.range.nextOffset
+  })));
+}
+```
+
+Validation, whole-request/provider, malformed-output and cancellation failures
+throw: direct calls become Pi tool errors and scripts must use `try`/`catch` or
+`Promise.allSettled`. Individual fetch failures are successful result data with
+an `error` field, not `isError: true`. These tools do not return structured success
+objects alongside `isError: true`. A trusted result hook can replace that contract;
+annotations do not bypass hooks or approvals.
+
+The namespace is `web`, not a name prefix: calls remain `tools.web_fetch(…)`.
+`await describeNamespace("web")`, `await describeTool("web_fetch")`, and
+`await searchTools("page content", { namespace: "web" })` work even with
+`codemode.inlineBudget: 0`. Independent reads can run in parallel. The advisory
+hints are read-only, non-destructive, idempotent, and open-world: repeated reads
+can return different content and incur provider charges.
+
+Direct exposure remains the default, and codemode is optional. No package-level
+deferred/codemode exposure switch is added: it would make inactive direct tools
+nested-callable and change the meaning of disabling them. Use Pi's `codemode.mode`
+(`on` or `only`) and `inlineBudget` to reduce declarations while retaining the
+active-tool boundary. Optional research tools still require their provider keys.
+Explicit `defaultTools` `-name` entries are honored at late registration; reload
+restores Pi's active selection rather than re-enabling manually disabled tools.
+New optional tools activate when credentials first become available on reload,
+including existing positive `defaultTools` selections. Previously registered
+tools are not force-reactivated, even if settings still positively select them.
+The extension records only previously registered tool names in branch-local
+`pi-web-kit:registered-tools` session metadata, outside model context; it does
+not store activation choices or credentials. The record is refreshed before
+reload so navigating to older branch history cannot revive a manually disabled
+tool. Older sessions without this metadata preserve inactivity on their first
+reload, so they may require explicit activation or a restart.
+Malformed/unknown-version metadata uses the same
+conservative fallback.
+CLI exclusions remain host-enforced.
+
 ## Privacy and security
 
 `pi-web-kit` sends search queries and fetched URLs to the configured provider. Developer-search tools send library/doc queries to Context7 and code-context queries to Exa when those tools are enabled. Fetch providers may also receive provider-specific options. API keys are read from environment variables or local config files and are used only for provider requests.
 
-The extension rejects non-HTTP(S) URLs and URLs with embedded username/password credentials. Provider responses are not sandboxed; they are returned to Pi as tool output.
+The extension rejects non-HTTP(S) URLs and URLs with embedded username/password credentials.
+Returned data is projected to declared public fields; raw provider metadata,
+Context7 `rules`, cache keys and HTTP error bodies are not returned. Known API
+keys, URL credential patterns and Basic/Bearer authorization-header values are masked in text, structured data,
+renderer details and progress. This is not a general sensitive-data scanner:
+page content remains untrusted, and Pi retains caller-supplied arguments in its
+own transcript. Do not put credentials in queries or URLs.
 
 Report security issues privately. See [SECURITY.md](SECURITY.md).
 
@@ -238,7 +335,7 @@ Report security issues privately. See [SECURITY.md](SECURITY.md).
 
 Requirements:
 
-- Node.js >= 20.6.0
+- Node.js >= 22.19.0
 - npm
 
 Common commands:

package/SECURITY.md

@@ -21,8 +21,30 @@ The maintainer will acknowledge reports as soon as practical and coordinate disc
 
 `pi-web-kit` is a Pi package. Pi extensions execute with the same permissions as the local user running Pi. Users should review installed Pi packages and only install packages from sources they trust.
 
-`pi-web-kit` sends search queries and fetched URLs to the configured third-party provider. Page content returned by providers is cached in memory for the lifetime of the Pi process, subject to TTL and size limits. API keys are read from environment variables or local config files and are used only for provider requests. Do not commit API keys, tokens, or config files containing secrets.
+`pi-web-kit` sends search queries and fetched URLs to the configured third-party provider. A TinyFish search may send the same query in up to 11 paginated requests to satisfy the requested result limit. When `contextTokens` is explicit, TinyFish also receives the selected result URLs for batched extraction, while Firecrawl may scrape search results. Page content returned by providers is cached in memory for the lifetime of the Pi process, subject to TTL and size limits. API keys are read from environment variables or local config files and are used only for provider requests. Do not commit API keys, tokens, or config files containing secrets.
 
-On startup, the extension sends a best-effort install/update telemetry ping to `mocito.dev` once per package version unless Pi telemetry is disabled or offline mode is enabled. The ping includes only the package name, version, and parsed platform/runtime/architecture from its User-Agent; it does not include prompts, queries, fetched URLs, file paths, config values, or API keys.
+On startup, `@mocito/install-telemetry` sends a best-effort install/update telemetry ping to the configured telemetry endpoint once per package version unless Pi telemetry is disabled or offline mode is enabled. The ping includes only the package name, version, and parsed platform/runtime/architecture from its User-Agent; it does not include prompts, queries, fetched URLs, file paths, config values, or API keys.
 
 The extension validates URLs before fetch calls and only accepts `http:` and `https:` URLs without embedded credentials. This validation reduces accidental misuse but does not sandbox provider responses or the local Pi process.
+
+Tool results use an allowlisted output schema before publication. Model text and
+`structuredContent` share the same 50,000-byte JSON budget and credential masking;
+renderer details and progress also mask configured API keys, URL credential
+patterns and Basic/Bearer authorization-header values (not ordinary auth prose).
+Arbitrary provider metadata, Context7 `rules`, cache keys,
+and HTTP failure bodies are excluded. Untrusted per-page error bodies become
+generic diagnostics; controlled HTTP status/timeout messages remain useful.
+This does not classify all private information in fetched content or sanitize
+Pi's stored caller arguments. Do not send secrets in queries or URLs.
+
+The `web` namespace and read-only/idempotent/open-world annotations are advisory.
+They do not approve provider spending, bypass tool hooks, or make an inactive
+direct tool reachable from nested dispatch. Project trust, URL validation,
+provider request limits and cancellation still apply.
+
+Branch-local `pi-web-kit:registered-tools` custom entries remember only this
+package's fixed tool names as they first become available. They contain no
+credentials, configuration values, activation decisions or usage data and do
+not enter model context. This distinguishes a newly available optional tool
+from re-registration of a manually disabled tool; Pi still owns activation
+and exclusions. Missing or invalid history does not force tools active on reload.