pi-web-kit 0.2.3 → 0.3.0

package/CHANGELOG.md

@@ -1,13 +1,41 @@
 # Changelog
 
-## 0.2.3 - 2026-07-17
-
 All notable changes to this project will be documented in this file.
 
 This project follows the spirit of [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and uses semantic versioning for releases.
 
 ## [Unreleased]
 
+## [0.3.0] - 2026-08-31
+
+### Changed
+
+- Share install telemetry mechanics through `@mocito/install-telemetry` while preserving Pi-specific settings and state paths.
+- Treat `web_search.numResults` as a provider-agnostic desired limit, cap it to each provider's service constraints, and report requested, effective, returned, and omitted counts.
+- Add provider-agnostic `contextTokens` and `purpose` search controls, preserve provider-native grounding content, and allocate one bounded context budget across ranked results.
+- Use adaptive extraction: native Exa/Brave context is retained automatically, while TinyFish fetch and Firecrawl scrape-on-search run only when expanded context is requested.
+- Expose documented TinyFish search/fetch controls and Brave LLM Context controls.
+- Add provider-agnostic `maxAgeMs`, pass refresh/cache intent through to Exa, TinyFish, and Firecrawl, and request complete bounded pages from Exa fetch.
+
+### Fixed
+
+- Let `enableInstallTelemetry: false` override an enabled `PI_TELEMETRY` environment flag.
+- Paginate TinyFish searches instead of sending its ignored `limit` parameter.
+- Send markdown.new's documented `retain_images` field and report its response metadata.
+- Match TinyFish fetch responses and per-URL errors by canonical URL instead of response position.
+
+### Removed
+
+- Remove the Exa MCP search/fetch surface and make the direct Exa API the default. Existing `exa_mcp` configurations must select `exa` and provide `EXA_API_KEY`.
+
+## [0.2.4] - 2026-07-28
+
+### Changed
+
+- Rewrite `library_search`, `library_docs`, `web_search`, and `code_search` prompts so `library_docs` is the default for versioned library/framework/SDK docs (even familiar libraries), `library_search` is reserved for inspecting candidate matches, and `web_search`/`code_search` defer to it. Removes the "ambiguous library" wording that suppressed calls to well-known libraries. The `web_search`/`code_search` deferrals are emitted only when Context7 is configured, so installations without `CONTEXT7_API_KEY` are unaffected. Provider behavior and tool schemas are unchanged.
+
+## [0.2.3] - 2026-07-17
+
 ### Fixed
 
 - Honor project-local config only after Pi confirms project trust.
@@ -24,6 +52,10 @@ This project follows the spirit of [Keep a Changelog](https://keepachangelog.com
 
 ## [0.2.1] - 2026-06-14
 
+### Changed
+
+- Simplify provider, configuration, cache, and URL handling while preserving the public tool surface.
+
 ## [0.2.0] - 2026-06-13
 
 ### Added

package/README.md

@@ -1,20 +1,16 @@
 # pi-web-kit
 
-Context-efficient web and developer search tools for [Pi](https://pi.dev): `web_search`, `web_fetch`, `library_search`, `library_docs`, and `code_search`.
+Give [Pi](https://pi.dev) current web knowledge, authoritative library docs, and real-world code examples without flooding model context.
 
-`pi-web-kit` provides provider-backed search, page fetching, library docs lookup, and code-context search with bounded output, chunked reads, URL validation, and an in-memory fetch cache designed for agent workflows.
+`pi-web-kit` combines search, page reading, version-aware documentation, and code research behind five agent-ready tools with bounded, cache-aware output.
 
 ## Features
 
-- `web_search` for current/external web information, including multi-query searches.
-- `web_fetch` for reading one or more URLs, with `offset` / `limit` chunk reads for long pages.
-- `library_search` and `library_docs` for library resolution and current, version-aware documentation/code examples.
-- `code_search` for practical examples and implementation context.
-- Multiple provider backends: Exa MCP, Exa API, TinyFish, Brave Search, Firecrawl, markdown.new, Context7, and Exa Code.
-- Provider-tailored tool schemas at Pi startup/reload.
-- URL validation: HTTP(S)-only, no embedded credentials, fragment stripping, duplicate removal, and length/count limits.
-- In-memory fetch cache with TTL, LRU eviction, max entry count, max byte count, and cache keys based on provider/config/fetch-affecting options.
-- Bounded provider concurrency and network timeouts.
+- **Research the live web** — search current information and read single or multiple pages without leaving Pi.
+- **Use docs that match the task** — resolve libraries and retrieve focused, version-aware documentation with code examples.
+- **Find proven implementation patterns** — search practical usage, setup, migrations, and error context across real code.
+- **Spend context wisely** — compact search results, chunked page reads, bounded output, and fetch caching keep research useful without overwhelming the model.
+- **Choose your providers** — mix Exa, TinyFish, Brave, Firecrawl, markdown.new, Context7, and Exa Code based on coverage, cost, and credentials.
 
 ## Installation
 
@@ -74,11 +70,10 @@ Pi chooses `web_search` or `web_fetch` automatically when the request calls for
 
 ## Providers
 
-Defaults: `provider_search = "exa_mcp"`, `provider_fetch = "exa_mcp"`.
+Defaults: `provider_search = "exa"`, `provider_fetch = "exa"`.
 
 | Provider | Search | Fetch | Key |
 |---|---:|---:|---|
-| `exa_mcp` | yes | yes | optional `EXA_API_KEY` |
 | `exa` | yes | yes | `EXA_API_KEY` |
 | `tinyfish` | yes | yes | `TINYFISH_API_KEY` |
 | `brave` | yes | no | `BRAVE_SEARCH_API_KEY` |
@@ -87,6 +82,18 @@ Defaults: `provider_search = "exa_mcp"`, `provider_fetch = "exa_mcp"`.
 
 Tool schemas are tailored to the configured providers at startup/reload, so only supported provider-specific fields are exposed. Restart/reload Pi after changing provider config.
 
+Provider-native efficiencies are used as follows:
+
+| Service | Efficient path |
+|---|---|
+| Exa API | Keep included highlights; request bounded text in the same search only when requested; batch `/contents` fetches with freshness controls. |
+| TinyFish | Paginate only as needed; use dedicated filters; batch only enough top-result fetches to fill requested context; pass TTL and intent. |
+| Brave | Return pre-extracted LLM Context in one search call with native URL, token, snippet, threshold, and Goggles controls. |
+| Firecrawl | Return search metadata by default; use scrape-on-search only when requested; pass cache/main-content options. |
+| markdown.new | Keep `method: auto` so native Markdown falls back to AI and browser rendering only as needed; keep images opt-in. |
+| Context7 | Support `fast` mode to skip reranking and reduce latency. |
+| Exa Code | Send the requested/dynamic context token target directly to Exa Code Context. |
+
 ## Configuration
 
 Resolution order: defaults < environment variables < global config < trusted project config < CLI flags. Project config is ignored unless Pi trusts the current project, including in print, JSON, and RPC modes.
@@ -96,8 +103,8 @@ Resolution order: defaults < environment variables < global config < trusted pro
 ```bash
 PI_OFFLINE=1        # disables install/update telemetry
 PI_TELEMETRY=0      # disables install/update telemetry
-PI_WEB_KIT_PROVIDER_SEARCH=exa_mcp|exa|tinyfish|brave|firecrawl
-PI_WEB_KIT_PROVIDER_FETCH=exa_mcp|exa|tinyfish|markdown_new|firecrawl
+PI_WEB_KIT_PROVIDER_SEARCH=exa|tinyfish|brave|firecrawl
+PI_WEB_KIT_PROVIDER_FETCH=exa|tinyfish|markdown_new|firecrawl
 EXA_API_KEY=...          # enables Exa provider and code_search
 CONTEXT7_API_KEY=...     # enables library_search and library_docs
 TINYFISH_API_KEY=...
@@ -152,9 +159,15 @@ Searches with the active search provider and returns compact results grouped by
 |---|---|---|
 | `query` | string | Single search query. |
 | `queries` | string[] | Multiple related search queries. Max 5 after de-duplication. |
-| `numResults` | integer | Results per query. Range: 1-20. Default: 10. |
+| `numResults` | integer | Desired results per query. Default: 10. Values above the active provider's limit are capped rather than rejected. |
+| `contextTokens` | integer | Desired extracted context across the result set. Omitted native context uses an 8,192-token output budget; an explicit value can enable extraction. Any positive value; capped at 10,000. |
+| `purpose` | string | Optional task/use-case hint for providers that support separate intent. |
+
+`numResults` controls source breadth. `contextTokens` controls grounding depth. The tool automatically keeps native/included Exa highlights and Brave LLM Context. Explicit `contextTokens` enables extra extraction for Exa, TinyFish, and Firecrawl; this can add provider calls or provider cost. Search results keep a compact `snippet` plus ranked `content` and `contentFormat`, with one shared context budget and the existing 50KB tool-output limit.
+
+Provider caps are Exa 100, Brave 50, and Firecrawl 100. TinyFish is paginated internally through its service maximum of page 10 and may make up to 11 search requests for one query. Search output reports requested, effective, returned, and omitted result/context counts. Brave `maxUrls` remains as a deprecated alias for `numResults`.
 
-Provider-specific parameters are exposed only for the configured provider, such as Exa date/domain filters, TinyFish `page`, Brave locale/freshness options, or Firecrawl scrape/search options.
+Other provider-specific parameters are exposed only for the configured provider. These include Exa date/domain filters; TinyFish domain, date, geography, language, and publication filters; Brave locale, freshness, spellcheck, Goggles, and LLM Context controls; and Firecrawl scrape/search options.
 
 ### `web_fetch`
 
@@ -167,8 +180,9 @@ Fetches page content with the active fetch provider. Results are cached in memor
 | `offset` | integer | Character offset for cached/ranged reads. Single URL only. |
 | `limit` | integer | Maximum characters to return. Default: 30,000 for one URL, 8,000 for multiple URLs. |
 | `refresh` | boolean | Refetch even if cached. |
+| `maxAgeMs` | integer | Desired maximum local/provider-cached page age in milliseconds. `0` requests live content where supported. |
 
-Provider-specific parameters are exposed only for the configured provider, such as TinyFish `format`, markdown.new `method` / `retainImages`, or Firecrawl `format`, `waitFor`, `mobile`, `location`, and `maxAge`.
+Provider-specific parameters are exposed only for the configured provider. TinyFish supports `purpose`, `format`, links/images, selectors, per-URL timeout, and a seconds-based `ttl` alias. Exa supports the hours-based `maxAgeHours` alias. markdown.new supports `method` / `retainImages`. Firecrawl supports `format`, `waitFor`, `mobile`, structured `location`, and its existing `maxAge` alias. `refresh: true` also requests live content from Exa, TinyFish, and Firecrawl instead of only bypassing the local cache.
 
 ### `library_search`
 
@@ -214,7 +228,8 @@ Finds practical code examples, implementation context, setup snippets, migration
 | Max cached bytes | 20 MiB |
 | Max URLs per call | 10 |
 | Max queries per call | 5 |
-| Max `numResults` | 20 |
+| Provider `numResults` caps | Exa 100; Brave 50; Firecrawl 100 |
+| Search context budget | 10,000 tokens |
 | Max URL length | 2048 characters |
 
 Cache keys include the provider, canonical URL, fetch-affecting parameters, relevant provider defaults, and an opaque SHA-256 API-key/account scope. Internal cache keys are never returned in tool output. `refresh: true` bypasses and replaces the cached entry.

package/SECURITY.md

@@ -21,8 +21,8 @@ The maintainer will acknowledge reports as soon as practical and coordinate disc
 
 `pi-web-kit` is a Pi package. Pi extensions execute with the same permissions as the local user running Pi. Users should review installed Pi packages and only install packages from sources they trust.
 
-`pi-web-kit` sends search queries and fetched URLs to the configured third-party provider. Page content returned by providers is cached in memory for the lifetime of the Pi process, subject to TTL and size limits. API keys are read from environment variables or local config files and are used only for provider requests. Do not commit API keys, tokens, or config files containing secrets.
+`pi-web-kit` sends search queries and fetched URLs to the configured third-party provider. A TinyFish search may send the same query in up to 11 paginated requests to satisfy the requested result limit. When `contextTokens` is explicit, TinyFish also receives the selected result URLs for batched extraction, while Firecrawl may scrape search results. Page content returned by providers is cached in memory for the lifetime of the Pi process, subject to TTL and size limits. API keys are read from environment variables or local config files and are used only for provider requests. Do not commit API keys, tokens, or config files containing secrets.
 
-On startup, the extension sends a best-effort install/update telemetry ping to `mocito.dev` once per package version unless Pi telemetry is disabled or offline mode is enabled. The ping includes only the package name, version, and parsed platform/runtime/architecture from its User-Agent; it does not include prompts, queries, fetched URLs, file paths, config values, or API keys.
+On startup, `@mocito/install-telemetry` sends a best-effort install/update telemetry ping to the configured telemetry endpoint once per package version unless Pi telemetry is disabled or offline mode is enabled. The ping includes only the package name, version, and parsed platform/runtime/architecture from its User-Agent; it does not include prompts, queries, fetched URLs, file paths, config values, or API keys.
 
 The extension validates URLs before fetch calls and only accepts `http:` and `https:` URLs without embedded credentials. This validation reduces accidental misuse but does not sandbox provider responses or the local Pi process.

package/extensions/index.ts

@@ -4,7 +4,7 @@ import { Text } from "@earendil-works/pi-tui";
 import { Type } from "typebox";
 import { fetchCache, type CachedPage } from "../src/cache.js";
 import { resolveConfig } from "../src/config.js";
-import { DEFAULT_FETCH_LIMIT, DEFAULT_NUM_RESULTS, MAX_LIMIT, MAX_NUM_RESULTS, MAX_OFFSET, MAX_QUERY_COUNT, MAX_URL_COUNT, MULTI_FETCH_LIMIT } from "../src/limits.js";
+import { applySearchContextBudget, capSearchResultLimit, DEFAULT_FETCH_LIMIT, DEFAULT_NUM_RESULTS, DEFAULT_SEARCH_CONTEXT_TOKENS, MAX_LIMIT, MAX_NUM_RESULTS, MAX_OFFSET, MAX_QUERY_COUNT, MAX_SEARCH_CONTEXT_TOKENS, MAX_URL_COUNT, MULTI_FETCH_LIMIT, safePrefix, TINYFISH_MAX_PAGE } from "../src/limits.js";
 import { createCodeSearchProvider, createContext7Provider, createFetchProvider, createSearchProvider } from "../src/providers/index.js";
 import { mapFetchResults } from "../src/providers/fallback.js";
 import type { FetchProviderName, SearchProviderName, WebFetchResult } from "../src/types.js";
@@ -14,11 +14,11 @@ import { reportInstallTelemetry } from "../src/install-telemetry.js";
 export default function (pi: ExtensionAPI) {
   void reportInstallTelemetry();
   pi.registerFlag("web-provider-search", {
-    description: "Temporary pi-web-kit search provider override (exa_mcp, exa, tinyfish, brave, firecrawl)",
+    description: "Temporary pi-web-kit search provider override (exa, tinyfish, brave, firecrawl)",
     type: "string",
   });
   pi.registerFlag("web-provider-fetch", {
-    description: "Temporary pi-web-kit fetch provider override (exa_mcp, exa, tinyfish, markdown_new, firecrawl)",
+    description: "Temporary pi-web-kit fetch provider override (exa, tinyfish, markdown_new, firecrawl)",
     type: "string",
   });
 
@@ -38,17 +38,34 @@ function registerTools(pi: ExtensionAPI, startupConfig: ReturnType<typeof resolv
     name: "web_search",
     label: "Web Search",
     description: buildSearchDescription(startupConfig.provider_search),
-    promptSnippet: "Find current or external web information.",
-    promptGuidelines: ["Use web_search to find current or external web information."],
+    promptSnippet: startupConfig.apiKeys.context7
+      ? "Search the live web for current events, news, and non-library topics."
+      : "Find current or external web information.",
+    promptGuidelines: [
+      startupConfig.apiKeys.context7
+        ? "Use web_search for current events, news, and non-library topics."
+        : "Use web_search to find current or external web information.",
+      ...(startupConfig.apiKeys.context7
+        ? ["Prefer library_docs over web_search for versioned library/framework/SDK documentation."]
+        : []),
+    ],
     parameters: buildSearchSchema(startupConfig.provider_search),
     async execute(_toolCallId, rawParams, signal, onUpdate, ctx) {
       const params = rawParams as Record<string, any>;
       const queries = normalizeQueries(params);
-      const numResults = parseInteger(params.numResults, DEFAULT_NUM_RESULTS, "numResults", 1, MAX_NUM_RESULTS);
       if (queries.length === 0) throw new Error("web_search requires query or queries.");
 
       const config = runtimeConfig(pi, ctx.cwd, projectIsTrusted(ctx));
       assertProviderUnchanged("web_search", startupConfig.provider_search, config.provider_search);
+      const requestedResultLimit = parseInteger(
+        params.numResults ?? (config.provider_search === "brave" ? params.maxUrls : undefined),
+        DEFAULT_NUM_RESULTS,
+        "numResults",
+        1,
+      );
+      const effectiveResultLimit = capSearchResultLimit(config.provider_search, requestedResultLimit);
+      const requestedContextTokens = params.contextTokens == null ? undefined : parseInteger(params.contextTokens, 0, "contextTokens", 1);
+      const effectiveContextTokens = Math.min(requestedContextTokens ?? DEFAULT_SEARCH_CONTEXT_TOKENS, MAX_SEARCH_CONTEXT_TOKENS);
       const provider = createSearchProvider(config);
       const grouped = [];
       const progress = createProgress("search", config.provider_search, queries);
@@ -56,9 +73,20 @@ function registerTools(pi: ExtensionAPI, startupConfig: ReturnType<typeof resolv
       for (const query of queries) {
         markProgressCurrent(progress, query);
         emitProgress(onUpdate, progress);
-        const result = await provider.search({ ...params, query, numResults }, signal);
-        grouped.push({ query, results: result.results });
-        markProgressDone(progress, query, `${result.results.length} results`);
+        const result = await provider.search({ ...params, query, numResults: effectiveResultLimit, contextTokens: requestedContextTokens == null ? undefined : effectiveContextTokens }, signal);
+        const bounded = applySearchContextBudget(result.results, effectiveContextTokens * 4);
+        grouped.push({
+          query,
+          requestedResultLimit,
+          effectiveResultLimit: result.effectiveResultLimit ?? effectiveResultLimit,
+          requestedContextTokens,
+          effectiveContextTokens,
+          contextCharacters: bounded.contextCharacters,
+          ...(bounded.omittedContextCharacters ? { omittedContextCharacters: bounded.omittedContextCharacters } : {}),
+          resultCount: bounded.results.length,
+          results: bounded.results,
+        });
+        markProgressDone(progress, query, `${bounded.results.length} results`);
         emitProgress(onUpdate, progress);
       }
       const result = { provider: config.provider_search, queries: grouped };
@@ -108,9 +136,9 @@ function registerTools(pi: ExtensionAPI, startupConfig: ReturnType<typeof resolv
     pi.registerTool({
       name: "library_search",
       label: "Library Search",
-      description: "Resolve library, package, framework, SDK, API, or CLI names to canonical library IDs.",
-      promptSnippet: "Resolve a library name to a canonical library ID before querying docs.",
-      promptGuidelines: ["Use library_search when a library/framework/package is ambiguous or you need a canonical library ID."],
+      description: "Resolve library, package, framework, SDK, API, or CLI names to canonical library IDs with version, trust, and snippet metadata. Use when you need to inspect candidate matches (official sources, versions, forks); library_docs resolves names automatically.",
+      promptSnippet: "List candidate library IDs; library_docs resolves names on its own.",
+      promptGuidelines: ["Use library_search only to inspect matching libraries (official source, versions, forks) before a precise library_docs call.", "Not needed for ordinary doc lookups — library_docs resolves a libraryName for you."],
       parameters: buildLibrarySearchSchema(),
       async execute(_toolCallId, rawParams, signal, _onUpdate, ctx) {
         const params = rawParams as Record<string, any>;
@@ -126,9 +154,9 @@ function registerTools(pi: ExtensionAPI, startupConfig: ReturnType<typeof resolv
     pi.registerTool({
       name: "library_docs",
       label: "Library Docs",
-      description: "Fetch current, version-aware documentation and code examples for a library.",
-      promptSnippet: "Get current library documentation and code examples.",
-      promptGuidelines: ["Use library_docs for current APIs, framework behavior, SDK examples, package docs, and version-specific library questions."],
+      description: "Fetch current, version-aware documentation and code examples for a library. Pass libraryName to resolve it automatically, or a known libraryId.",
+      promptSnippet: "Get current, version-aware library/framework/SDK docs and code examples.",
+      promptGuidelines: ["Use library_docs for any library, framework, SDK, API, or CLI — even familiar ones like React, Next.js, Prisma, or Express. Training data may be outdated; this returns version-specific docs.", "Prefer library_docs over web_search or your own knowledge for library/API questions. Pass libraryName and query; you rarely need library_search first."],
       parameters: buildLibraryDocsSchema(),
       async execute(_toolCallId, rawParams, signal, _onUpdate, ctx) {
         const params = rawParams as Record<string, any>;
@@ -154,7 +182,10 @@ function registerTools(pi: ExtensionAPI, startupConfig: ReturnType<typeof resolv
       label: "Code Search",
       description: "Find practical code examples, usage patterns, setup snippets, migrations, and error context.",
       promptSnippet: "Find real-world code examples, usage patterns, migrations, and error context.",
-      promptGuidelines: ["Use code_search for real-world code examples, GitHub/open-source usage patterns, API syntax examples, setup snippets, migrations, and error messages."],
+      promptGuidelines: [
+        "Use code_search for real-world code examples, usage patterns, setup snippets, migrations, and error context from open source.",
+        ...(startupConfig.apiKeys.context7 ? ["Prefer library_docs for official API/reference documentation."] : []),
+      ],
       parameters: buildCodeSearchSchema(),
       async execute(_toolCallId, rawParams, signal, _onUpdate, ctx) {
         const params = rawParams as Record<string, any>;
@@ -200,7 +231,9 @@ export function buildSearchSchema(provider: SearchProviderName) {
   const props: Record<string, any> = {
     query: Type.Optional(Type.String({ description: "Single search query" })),
     queries: Type.Optional(Type.Array(Type.String(), { description: `Multiple related search queries (max ${MAX_QUERY_COUNT})`, maxItems: MAX_QUERY_COUNT })),
-    numResults: Type.Optional(int("Results per query", 1, MAX_NUM_RESULTS)),
+    numResults: Type.Optional(int("Desired results per query; capped by the active provider", 1)),
+    contextTokens: Type.Optional(int("Desired extracted context tokens; capped to the tool output budget and may enable provider extraction", 1)),
+    purpose: Type.Optional(Type.String({ description: "Optional task/use-case hint when supported by the provider", minLength: 1, maxLength: 2_000 })),
   };
   if (provider === "exa") Object.assign(props, {
     includeDomains: Type.Optional(Type.Array(Type.String())),
@@ -211,13 +244,32 @@ export function buildSearchSchema(provider: SearchProviderName) {
     endCrawlDate: Type.Optional(Type.String()),
     type: Type.Optional(Type.String()),
     category: Type.Optional(Type.String()),
+    maxAgeHours: Type.Optional(int("Maximum Exa cached content age in hours; 0 forces live crawl, -1 disables live crawl", -1)),
+  });
+  if (provider === "tinyfish") Object.assign(props, {
+    page: Type.Optional(int("Deprecated starting result page", 0, TINYFISH_MAX_PAGE)),
+    location: Type.Optional(Type.String()), language: Type.Optional(Type.String()),
+    includeDomains: Type.Optional(Type.Array(Type.String())), excludeDomains: Type.Optional(Type.Array(Type.String())),
+    domainType: Type.Optional(Type.Union([Type.Literal("web"), Type.Literal("news"), Type.Literal("research_paper")])),
+    afterDate: Type.Optional(Type.String({ pattern: "^\\d{4}-\\d{2}-\\d{2}$" })), beforeDate: Type.Optional(Type.String({ pattern: "^\\d{4}-\\d{2}-\\d{2}$" })),
+    recencyMinutes: Type.Optional(int("Freshness window in minutes", 1, 5_256_000)),
+    pubYearMin: Type.Optional(int("Minimum publication year", 0, 9_999)), pubYearMax: Type.Optional(int("Maximum publication year", 0, 9_999)),
   });
-  if (provider === "tinyfish") Object.assign(props, { page: Type.Optional(int("Result page", 1, 10)) });
   if (provider === "brave") Object.assign(props, {
-    country: Type.Optional(Type.String()), searchLang: Type.Optional(Type.String()), uiLang: Type.Optional(Type.String()), safesearch: Type.Optional(Type.String()), freshness: Type.Optional(Type.String()), maxUrls: Type.Optional(int("Maximum URLs", 1, MAX_NUM_RESULTS)),
+    country: Type.Optional(Type.String({ minLength: 2, maxLength: 2 })), searchLang: Type.Optional(Type.String({ minLength: 2, maxLength: 10 })),
+    safesearch: Type.Optional(Type.Union([Type.Literal("off"), Type.Literal("moderate"), Type.Literal("strict")])),
+    freshness: Type.Optional(Type.String()), spellcheck: Type.Optional(Type.Boolean()),
+    contextThresholdMode: Type.Optional(Type.Union([Type.Literal("disabled"), Type.Literal("strict"), Type.Literal("balanced"), Type.Literal("lenient")])),
+    maxSnippets: Type.Optional(int("Maximum context snippets", 1, 256)),
+    maxTokensPerUrl: Type.Optional(int("Maximum context tokens per URL", 512, 8_192)),
+    maxSnippetsPerUrl: Type.Optional(int("Maximum context snippets per URL", 1, 100)),
+    goggles: Type.Optional(Type.Union([Type.String(), Type.Array(Type.String(), { maxItems: 3 })])),
+    maxUrls: Type.Optional(int("Deprecated alias for numResults", 1)),
   });
   if (provider === "firecrawl") Object.assign(props, {
-    location: Type.Optional(Type.String()), country: Type.Optional(Type.String()), includeDomains: Type.Optional(Type.Array(Type.String())), excludeDomains: Type.Optional(Type.Array(Type.String())), categories: Type.Optional(Type.Array(Type.String())), tbs: Type.Optional(Type.String()), scrape: Type.Optional(Type.Boolean({ description: "Enable default markdown scrape-on-search" })), scrapeOptions: Type.Optional(Type.Object({}, { additionalProperties: true, description: "Firecrawl scrapeOptions for search." })),
+    location: Type.Optional(Type.String()), country: Type.Optional(Type.String()), includeDomains: Type.Optional(Type.Array(Type.String())), excludeDomains: Type.Optional(Type.Array(Type.String())),
+    categories: Type.Optional(Type.Array(Type.Union([Type.Literal("research"), Type.Literal("pdf"), Type.Literal("developer")]))),
+    tbs: Type.Optional(Type.String()), scrape: Type.Optional(Type.Boolean({ description: "Enable default markdown scrape-on-search" })), scrapeOptions: Type.Optional(Type.Object({}, { additionalProperties: true, description: "Firecrawl scrapeOptions for search." })),
   });
   return Type.Object(props, { additionalProperties: false });
 }
@@ -229,12 +281,23 @@ export function buildFetchSchema(provider: FetchProviderName) {
     offset: Type.Optional(int("Character offset for cached/ranged reads", 0, MAX_OFFSET)),
     limit: Type.Optional(int("Maximum characters to return", 1, MAX_LIMIT)),
     refresh: Type.Optional(Type.Boolean({ description: "Refetch even if cached" })),
+    maxAgeMs: Type.Optional(int("Maximum local/provider-cached page age in milliseconds; 0 requests live content where supported", 0)),
   };
+  if (provider === "exa") Object.assign(props, { maxAgeHours: Type.Optional(int("Maximum Exa cached content age in hours; 0 forces live crawl, -1 disables live crawl", -1)) });
   if (provider === "tinyfish") Object.assign(props, { format: Type.Optional(Type.Union([Type.Literal("markdown"), Type.Literal("html"), Type.Literal("json")])), links: Type.Optional(Type.Boolean()), imageLinks: Type.Optional(Type.Boolean()) });
+  if (provider === "tinyfish") Object.assign(props, {
+    purpose: Type.Optional(Type.String({ minLength: 1, maxLength: 2_000 })),
+    ttl: Type.Optional(int("Provider cache freshness tolerance in seconds; 0 prefers live fetch", 0)),
+    perUrlTimeoutMs: Type.Optional(int("Per-URL timeout in milliseconds", 1, 110_000)),
+    includeSelectors: Type.Optional(Type.Array(Type.String({ minLength: 1, maxLength: 1_000 }), { minItems: 1, maxItems: 20 })),
+    excludeSelectors: Type.Optional(Type.Array(Type.String({ minLength: 1, maxLength: 1_000 }), { minItems: 1, maxItems: 20 })),
+  });
   if (provider === "markdown_new") Object.assign(props, { method: Type.Optional(Type.Union([Type.Literal("auto"), Type.Literal("ai"), Type.Literal("browser")])), retainImages: Type.Optional(Type.Boolean()) });
   if (provider === "firecrawl") Object.assign(props, {
     format: Type.Optional(Type.Union([Type.Literal("markdown"), Type.Literal("html"), Type.Literal("json")])),
-    onlyMainContent: Type.Optional(Type.Boolean()), waitFor: Type.Optional(int("Milliseconds to wait", 0, 60_000)), mobile: Type.Optional(Type.Boolean()), location: Type.Optional(Type.String()), maxAge: Type.Optional(int("Maximum cached page age", 0)),
+    onlyMainContent: Type.Optional(Type.Boolean()), waitFor: Type.Optional(int("Milliseconds to wait", 0, 60_000)), mobile: Type.Optional(Type.Boolean()),
+    location: Type.Optional(Type.Object({ country: Type.Optional(Type.String()), languages: Type.Optional(Type.Array(Type.String())) }, { additionalProperties: false })),
+    maxAge: Type.Optional(int("Maximum cached page age in milliseconds", 0)),
   });
   return Type.Object(props, { additionalProperties: false });
 }
@@ -292,6 +355,7 @@ export async function fetchWithCache(providerName: FetchProviderName, params: Re
   const defaultLimit = urls.length > 1 ? MULTI_FETCH_LIMIT : DEFAULT_FETCH_LIMIT;
   const limit = parseInteger(params.limit, defaultLimit, "limit", 1, MAX_LIMIT);
   const refresh = params.refresh === true;
+  const maxAgeMs = localCacheMaxAge(providerName, params);
 
   const pages = new Map<string, { page?: CachedPage; cached: boolean; refreshed: boolean; error?: string }>();
   const cacheKeys = new Map<string, string>();
@@ -300,7 +364,7 @@ export async function fetchWithCache(providerName: FetchProviderName, params: Re
     const cacheKey = buildCacheKey(providerName, url, params, config);
     cacheKeys.set(url, cacheKey);
     const cached = fetchCache.get(cacheKey);
-    if (cached && !refresh) {
+    if (cached && !refresh && (maxAgeMs == null || (maxAgeMs > 0 && Date.now() - cached.fetchedAt <= maxAgeMs))) {
       pages.set(url, { page: cached, cached: true, refreshed: false });
       onProgress?.({ status: "done", url, note: "cached" });
     } else {
@@ -344,6 +408,14 @@ export async function fetchWithCache(providerName: FetchProviderName, params: Re
   }) };
 }
 
+function localCacheMaxAge(provider: FetchProviderName, params: Record<string, any>): number | undefined {
+  if (params.maxAgeMs != null) return parseInteger(params.maxAgeMs, 0, "maxAgeMs", 0);
+  if (provider === "exa" && typeof params.maxAgeHours === "number" && params.maxAgeHours >= 0) return params.maxAgeHours * 3_600_000;
+  if (provider === "tinyfish" && typeof params.ttl === "number") return params.ttl * 1_000;
+  if (provider === "firecrawl" && typeof params.maxAge === "number") return params.maxAge;
+  return undefined;
+}
+
 
 export function pageSlice(page: CachedPage, offset: number, limit: number, cached: boolean, refreshed: boolean) {
   const total = page.content.length;
@@ -424,8 +496,20 @@ function isSearchResult(value: any): value is { provider?: string; queries: Arra
 }
 
 function boundSearchResult(value: { provider?: string; queries: Array<{ query?: string; results?: any[] }> }) {
-  const compact = limitStrings(value, 1_000) as typeof value;
-  const maxSnippet = Math.max(0, ...compact.queries.flatMap((group) => (group.results ?? []).map((item) => typeof item?.snippet === "string" ? item.snippet.length : 0)));
+  const compact = {
+    ...(limitStrings(value, 1_000) as Record<string, unknown>),
+    queries: value.queries.map((group) => ({
+      ...(limitStrings(group, 1_000) as Record<string, unknown>),
+      results: (group.results ?? []).map((item) => ({
+        ...(limitStrings(item, 1_000) as Record<string, unknown>),
+        ...(typeof item?.content === "string" ? { content: item.content } : {}),
+      })),
+    })),
+  } as typeof value;
+  const maxSnippet = Math.max(0, ...compact.queries.flatMap((group) => (group.results ?? []).flatMap((item) => [
+    typeof item?.snippet === "string" ? item.snippet.length : 0,
+    typeof item?.content === "string" ? item.content.length : 0,
+  ])));
   let low = 0;
   let high = maxSnippet;
   let best = searchResultWithLimits(compact, 0);
@@ -461,17 +545,29 @@ function boundSearchResult(value: { provider?: string; queries: Array<{ query?:
 function searchResultWithLimits(value: { provider?: string; queries: Array<{ query?: string; results?: any[] }> }, snippetChars: number, resultsPerQuery?: number) {
   return {
     ...value,
-    queries: value.queries.map((group) => ({
-      ...group,
-      results: (group.results ?? []).slice(0, resultsPerQuery).map((item) => {
-        if (typeof item?.snippet !== "string") return item;
-        if (snippetChars === 0) {
-          const { snippet: _snippet, ...metadata } = item;
-          return metadata;
+    queries: value.queries.map((group) => {
+      const originalContextCharacters = (group.results ?? []).reduce((total, item) => total + (typeof item?.content === "string" ? item.content.length : 0), 0);
+      const results = (group.results ?? []).slice(0, resultsPerQuery).map((item) => {
+        const bounded = { ...item };
+        for (const key of ["snippet", "content"] as const) {
+          if (typeof bounded[key] !== "string") continue;
+          if (snippetChars === 0) delete bounded[key];
+          else bounded[key] = safePrefix(bounded[key], snippetChars);
         }
-        return { ...item, snippet: safePrefix(item.snippet, snippetChars) };
-      }),
-    })),
+        return bounded;
+      });
+      const omittedResultCount = Math.max(0, (group.results?.length ?? 0) - results.length);
+      const contextCharacters = results.reduce((total, item) => total + (typeof item?.content === "string" ? item.content.length : 0), 0);
+      const omittedContextCharacters = (typeof (group as any).omittedContextCharacters === "number" ? (group as any).omittedContextCharacters : 0) + originalContextCharacters - contextCharacters;
+      return {
+        ...group,
+        contextCharacters,
+        ...(omittedContextCharacters ? { omittedContextCharacters } : {}),
+        resultCount: results.length,
+        ...(omittedResultCount ? { omittedResultCount } : {}),
+        results,
+      };
+    }),
   };
 }
 
@@ -509,12 +605,6 @@ function fetchResultWithContentLimit(value: { provider?: string; results: any[]
   };
 }
 
-function safePrefix(value: string, maxChars: number): string {
-  let end = Math.min(value.length, maxChars);
-  if (end > 0 && /[\uD800-\uDBFF]/.test(value[end - 1])) end--;
-  return value.slice(0, end);
-}
-
 function limitStrings(value: unknown, maxChars: number): unknown {
   if (typeof value === "string") return safePrefix(value, maxChars);
   if (Array.isArray(value)) return value.map((item) => limitStrings(item, maxChars));
@@ -522,9 +612,11 @@ function limitStrings(value: unknown, maxChars: number): unknown {
   return value;
 }
 
-function parseInteger(value: unknown, defaultValue: number, name: string, min: number, max: number): number {
+function parseInteger(value: unknown, defaultValue: number, name: string, min: number, max?: number): number {
   if (value == null) return defaultValue;
-  if (typeof value !== "number" || !Number.isInteger(value) || !Number.isFinite(value) || value < min || value > max) throw new Error(`${name} must be a finite integer between ${min} and ${max}.`);
+  if (typeof value !== "number" || !Number.isInteger(value) || !Number.isFinite(value) || value < min || (max != null && value > max)) {
+    throw new Error(`${name} must be a finite integer ${max == null ? `>= ${min}` : `between ${min} and ${max}`}.`);
+  }
   return value;
 }
 
@@ -553,7 +645,6 @@ function fetchConfigDefaults(provider: FetchProviderName, config?: any): Record<
 function providerScope(provider: FetchProviderName, config?: any): string {
   const keyMap: Partial<Record<FetchProviderName, string | undefined>> = {
     exa: config?.apiKeys?.exa,
-    exa_mcp: config?.apiKeys?.exa,
     tinyfish: config?.apiKeys?.tinyfish,
     firecrawl: config?.apiKeys?.firecrawl,
   };
@@ -579,7 +670,14 @@ function searchDetails(value: any) {
     provider: value.provider,
     queries: value.queries.map((q: any) => ({
       query: q.query,
+      requestedResultLimit: q.requestedResultLimit,
+      effectiveResultLimit: q.effectiveResultLimit,
+      requestedContextTokens: q.requestedContextTokens,
+      effectiveContextTokens: q.effectiveContextTokens,
+      contextCharacters: q.contextCharacters,
+      omittedContextCharacters: q.omittedContextCharacters,
       resultCount: (q.results ?? []).length,
+      omittedResultCount: q.omittedResultCount,
       results: (q.results ?? []).map((r: any) => ({ title: r.title, url: r.url, siteName: r.siteName, position: r.position })),
     })),
   };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-web-kit",
-  "version": "0.2.3",
+  "version": "0.3.0",
   "description": "Context-efficient web search and fetch tools for Pi.",
   "type": "module",
   "license": "MIT",
@@ -58,18 +58,21 @@
     "typebox": "*"
   },
   "devDependencies": {
-    "@earendil-works/pi-ai": "^0.80.0",
-    "@earendil-works/pi-coding-agent": "^0.80.0",
-    "@earendil-works/pi-tui": "^0.80.0",
-    "@types/node": "^26.1.0",
-    "tsx": "^4.23.0",
-    "typebox": "^1.3.4",
-    "typescript": "^6.0.3"
+    "@earendil-works/pi-ai": "^0.84.1",
+    "@earendil-works/pi-coding-agent": "^0.84.1",
+    "@earendil-works/pi-tui": "^0.84.1",
+    "@types/node": "^26.2.0",
+    "tsx": "^4.23.5",
+    "typebox": "^1.3.10",
+    "typescript": "^7.0.2"
   },
   "publishConfig": {
     "access": "public"
   },
   "engines": {
     "node": ">=20.6.0"
+  },
+  "dependencies": {
+    "@mocito/install-telemetry": "0.1.1"
   }
 }

package/src/config.ts

@@ -3,11 +3,11 @@ import { homedir } from "node:os";
 import { join } from "node:path";
 import type { FetchProviderName, SearchProviderName, WebKitConfig } from "./types.js";
 
-const SEARCH = ["exa_mcp", "exa", "tinyfish", "brave", "firecrawl"] as const;
-const FETCH = ["exa_mcp", "exa", "tinyfish", "markdown_new", "firecrawl"] as const;
+const SEARCH = ["exa", "tinyfish", "brave", "firecrawl"] as const;
+const FETCH = ["exa", "tinyfish", "markdown_new", "firecrawl"] as const;
 const DEFAULT_CONFIG: WebKitConfig = {
-  provider_search: "exa_mcp",
-  provider_fetch: "exa_mcp",
+  provider_search: "exa",
+  provider_fetch: "exa",
   apiKeys: {},
   markdownNew: { method: "auto", retainImages: false },
 };

package/src/http.ts

@@ -32,10 +32,19 @@ export async function requestJson<T>(url: string, init: RequestInit & { timeoutM
 }
 
 export function asSnippet(value: unknown): string | undefined {
+  return asText(value)?.slice(0, 1000) || undefined;
+}
+
+export function asText(value: unknown): string | undefined {
   if (value == null) return undefined;
-  if (Array.isArray(value)) return value.filter(Boolean).join("\n").slice(0, 1000) || undefined;
-  if (typeof value === "string") return value.slice(0, 1000) || undefined;
-  return JSON.stringify(value).slice(0, 1000);
+  if (Array.isArray(value)) return value.filter(Boolean).map((item) => typeof item === "string" ? item : JSON.stringify(item)).join("\n") || undefined;
+  if (typeof value === "string") return value || undefined;
+  return JSON.stringify(value);
+}
+
+export function withoutContent(value: any): Record<string, unknown> {
+  const { text: _text, content: _content, markdown: _markdown, html: _html, summary: _summary, highlights: _highlights, ...metadata } = value ?? {};
+  return metadata;
 }
 
 export function normalizeUrls(input: { url?: string; urls?: string[] }, maxCount?: number): string[] {