pi-web-kit 0.2.4 → 0.3.0

package/CHANGELOG.md

@@ -6,6 +6,28 @@ This project follows the spirit of [Keep a Changelog](https://keepachangelog.com
 
 ## [Unreleased]
 
+## [0.3.0] - 2026-08-31
+
+### Changed
+
+- Share install telemetry mechanics through `@mocito/install-telemetry` while preserving Pi-specific settings and state paths.
+- Treat `web_search.numResults` as a provider-agnostic desired limit, cap it to each provider's service constraints, and report requested, effective, returned, and omitted counts.
+- Add provider-agnostic `contextTokens` and `purpose` search controls, preserve provider-native grounding content, and allocate one bounded context budget across ranked results.
+- Use adaptive extraction: native Exa/Brave context is retained automatically, while TinyFish fetch and Firecrawl scrape-on-search run only when expanded context is requested.
+- Expose documented TinyFish search/fetch controls and Brave LLM Context controls.
+- Add provider-agnostic `maxAgeMs`, pass refresh/cache intent through to Exa, TinyFish, and Firecrawl, and request complete bounded pages from Exa fetch.
+
+### Fixed
+
+- Let `enableInstallTelemetry: false` override an enabled `PI_TELEMETRY` environment flag.
+- Paginate TinyFish searches instead of sending its ignored `limit` parameter.
+- Send markdown.new's documented `retain_images` field and report its response metadata.
+- Match TinyFish fetch responses and per-URL errors by canonical URL instead of response position.
+
+### Removed
+
+- Remove the Exa MCP search/fetch surface and make the direct Exa API the default. Existing `exa_mcp` configurations must select `exa` and provide `EXA_API_KEY`.
+
 ## [0.2.4] - 2026-07-28
 
 ### Changed

package/README.md

@@ -70,11 +70,10 @@ Pi chooses `web_search` or `web_fetch` automatically when the request calls for
 
 ## Providers
 
-Defaults: `provider_search = "exa_mcp"`, `provider_fetch = "exa_mcp"`.
+Defaults: `provider_search = "exa"`, `provider_fetch = "exa"`.
 
 | Provider | Search | Fetch | Key |
 |---|---:|---:|---|
-| `exa_mcp` | yes | yes | optional `EXA_API_KEY` |
 | `exa` | yes | yes | `EXA_API_KEY` |
 | `tinyfish` | yes | yes | `TINYFISH_API_KEY` |
 | `brave` | yes | no | `BRAVE_SEARCH_API_KEY` |
@@ -83,6 +82,18 @@ Defaults: `provider_search = "exa_mcp"`, `provider_fetch = "exa_mcp"`.
 
 Tool schemas are tailored to the configured providers at startup/reload, so only supported provider-specific fields are exposed. Restart/reload Pi after changing provider config.
 
+Provider-native efficiencies are used as follows:
+
+| Service | Efficient path |
+|---|---|
+| Exa API | Keep included highlights; request bounded text in the same search only when requested; batch `/contents` fetches with freshness controls. |
+| TinyFish | Paginate only as needed; use dedicated filters; batch only enough top-result fetches to fill requested context; pass TTL and intent. |
+| Brave | Return pre-extracted LLM Context in one search call with native URL, token, snippet, threshold, and Goggles controls. |
+| Firecrawl | Return search metadata by default; use scrape-on-search only when requested; pass cache/main-content options. |
+| markdown.new | Keep `method: auto` so native Markdown falls back to AI and browser rendering only as needed; keep images opt-in. |
+| Context7 | Support `fast` mode to skip reranking and reduce latency. |
+| Exa Code | Send the requested/dynamic context token target directly to Exa Code Context. |
+
 ## Configuration
 
 Resolution order: defaults < environment variables < global config < trusted project config < CLI flags. Project config is ignored unless Pi trusts the current project, including in print, JSON, and RPC modes.
@@ -92,8 +103,8 @@ Resolution order: defaults < environment variables < global config < trusted pro
 ```bash
 PI_OFFLINE=1        # disables install/update telemetry
 PI_TELEMETRY=0      # disables install/update telemetry
-PI_WEB_KIT_PROVIDER_SEARCH=exa_mcp|exa|tinyfish|brave|firecrawl
-PI_WEB_KIT_PROVIDER_FETCH=exa_mcp|exa|tinyfish|markdown_new|firecrawl
+PI_WEB_KIT_PROVIDER_SEARCH=exa|tinyfish|brave|firecrawl
+PI_WEB_KIT_PROVIDER_FETCH=exa|tinyfish|markdown_new|firecrawl
 EXA_API_KEY=...          # enables Exa provider and code_search
 CONTEXT7_API_KEY=...     # enables library_search and library_docs
 TINYFISH_API_KEY=...
@@ -148,9 +159,15 @@ Searches with the active search provider and returns compact results grouped by
 |---|---|---|
 | `query` | string | Single search query. |
 | `queries` | string[] | Multiple related search queries. Max 5 after de-duplication. |
-| `numResults` | integer | Results per query. Range: 1-20. Default: 10. |
+| `numResults` | integer | Desired results per query. Default: 10. Values above the active provider's limit are capped rather than rejected. |
+| `contextTokens` | integer | Desired extracted context across the result set. Omitted native context uses an 8,192-token output budget; an explicit value can enable extraction. Any positive value; capped at 10,000. |
+| `purpose` | string | Optional task/use-case hint for providers that support separate intent. |
+
+`numResults` controls source breadth. `contextTokens` controls grounding depth. The tool automatically keeps native/included Exa highlights and Brave LLM Context. Explicit `contextTokens` enables extra extraction for Exa, TinyFish, and Firecrawl; this can add provider calls or provider cost. Search results keep a compact `snippet` plus ranked `content` and `contentFormat`, with one shared context budget and the existing 50KB tool-output limit.
+
+Provider caps are Exa 100, Brave 50, and Firecrawl 100. TinyFish is paginated internally through its service maximum of page 10 and may make up to 11 search requests for one query. Search output reports requested, effective, returned, and omitted result/context counts. Brave `maxUrls` remains as a deprecated alias for `numResults`.
 
-Provider-specific parameters are exposed only for the configured provider, such as Exa date/domain filters, TinyFish `page`, Brave locale/freshness options, or Firecrawl scrape/search options.
+Other provider-specific parameters are exposed only for the configured provider. These include Exa date/domain filters; TinyFish domain, date, geography, language, and publication filters; Brave locale, freshness, spellcheck, Goggles, and LLM Context controls; and Firecrawl scrape/search options.
 
 ### `web_fetch`
 
@@ -163,8 +180,9 @@ Fetches page content with the active fetch provider. Results are cached in memor
 | `offset` | integer | Character offset for cached/ranged reads. Single URL only. |
 | `limit` | integer | Maximum characters to return. Default: 30,000 for one URL, 8,000 for multiple URLs. |
 | `refresh` | boolean | Refetch even if cached. |
+| `maxAgeMs` | integer | Desired maximum local/provider-cached page age in milliseconds. `0` requests live content where supported. |
 
-Provider-specific parameters are exposed only for the configured provider, such as TinyFish `format`, markdown.new `method` / `retainImages`, or Firecrawl `format`, `waitFor`, `mobile`, `location`, and `maxAge`.
+Provider-specific parameters are exposed only for the configured provider. TinyFish supports `purpose`, `format`, links/images, selectors, per-URL timeout, and a seconds-based `ttl` alias. Exa supports the hours-based `maxAgeHours` alias. markdown.new supports `method` / `retainImages`. Firecrawl supports `format`, `waitFor`, `mobile`, structured `location`, and its existing `maxAge` alias. `refresh: true` also requests live content from Exa, TinyFish, and Firecrawl instead of only bypassing the local cache.
 
 ### `library_search`
 
@@ -210,7 +228,8 @@ Finds practical code examples, implementation context, setup snippets, migration
 | Max cached bytes | 20 MiB |
 | Max URLs per call | 10 |
 | Max queries per call | 5 |
-| Max `numResults` | 20 |
+| Provider `numResults` caps | Exa 100; Brave 50; Firecrawl 100 |
+| Search context budget | 10,000 tokens |
 | Max URL length | 2048 characters |
 
 Cache keys include the provider, canonical URL, fetch-affecting parameters, relevant provider defaults, and an opaque SHA-256 API-key/account scope. Internal cache keys are never returned in tool output. `refresh: true` bypasses and replaces the cached entry.

package/SECURITY.md

@@ -21,8 +21,8 @@ The maintainer will acknowledge reports as soon as practical and coordinate disc
 
 `pi-web-kit` is a Pi package. Pi extensions execute with the same permissions as the local user running Pi. Users should review installed Pi packages and only install packages from sources they trust.
 
-`pi-web-kit` sends search queries and fetched URLs to the configured third-party provider. Page content returned by providers is cached in memory for the lifetime of the Pi process, subject to TTL and size limits. API keys are read from environment variables or local config files and are used only for provider requests. Do not commit API keys, tokens, or config files containing secrets.
+`pi-web-kit` sends search queries and fetched URLs to the configured third-party provider. A TinyFish search may send the same query in up to 11 paginated requests to satisfy the requested result limit. When `contextTokens` is explicit, TinyFish also receives the selected result URLs for batched extraction, while Firecrawl may scrape search results. Page content returned by providers is cached in memory for the lifetime of the Pi process, subject to TTL and size limits. API keys are read from environment variables or local config files and are used only for provider requests. Do not commit API keys, tokens, or config files containing secrets.
 
-On startup, the extension sends a best-effort install/update telemetry ping to `mocito.dev` once per package version unless Pi telemetry is disabled or offline mode is enabled. The ping includes only the package name, version, and parsed platform/runtime/architecture from its User-Agent; it does not include prompts, queries, fetched URLs, file paths, config values, or API keys.
+On startup, `@mocito/install-telemetry` sends a best-effort install/update telemetry ping to the configured telemetry endpoint once per package version unless Pi telemetry is disabled or offline mode is enabled. The ping includes only the package name, version, and parsed platform/runtime/architecture from its User-Agent; it does not include prompts, queries, fetched URLs, file paths, config values, or API keys.
 
 The extension validates URLs before fetch calls and only accepts `http:` and `https:` URLs without embedded credentials. This validation reduces accidental misuse but does not sandbox provider responses or the local Pi process.

package/extensions/index.ts

@@ -4,7 +4,7 @@ import { Text } from "@earendil-works/pi-tui";
 import { Type } from "typebox";
 import { fetchCache, type CachedPage } from "../src/cache.js";
 import { resolveConfig } from "../src/config.js";
-import { DEFAULT_FETCH_LIMIT, DEFAULT_NUM_RESULTS, MAX_LIMIT, MAX_NUM_RESULTS, MAX_OFFSET, MAX_QUERY_COUNT, MAX_URL_COUNT, MULTI_FETCH_LIMIT } from "../src/limits.js";
+import { applySearchContextBudget, capSearchResultLimit, DEFAULT_FETCH_LIMIT, DEFAULT_NUM_RESULTS, DEFAULT_SEARCH_CONTEXT_TOKENS, MAX_LIMIT, MAX_NUM_RESULTS, MAX_OFFSET, MAX_QUERY_COUNT, MAX_SEARCH_CONTEXT_TOKENS, MAX_URL_COUNT, MULTI_FETCH_LIMIT, safePrefix, TINYFISH_MAX_PAGE } from "../src/limits.js";
 import { createCodeSearchProvider, createContext7Provider, createFetchProvider, createSearchProvider } from "../src/providers/index.js";
 import { mapFetchResults } from "../src/providers/fallback.js";
 import type { FetchProviderName, SearchProviderName, WebFetchResult } from "../src/types.js";
@@ -14,11 +14,11 @@ import { reportInstallTelemetry } from "../src/install-telemetry.js";
 export default function (pi: ExtensionAPI) {
   void reportInstallTelemetry();
   pi.registerFlag("web-provider-search", {
-    description: "Temporary pi-web-kit search provider override (exa_mcp, exa, tinyfish, brave, firecrawl)",
+    description: "Temporary pi-web-kit search provider override (exa, tinyfish, brave, firecrawl)",
     type: "string",
   });
   pi.registerFlag("web-provider-fetch", {
-    description: "Temporary pi-web-kit fetch provider override (exa_mcp, exa, tinyfish, markdown_new, firecrawl)",
+    description: "Temporary pi-web-kit fetch provider override (exa, tinyfish, markdown_new, firecrawl)",
     type: "string",
   });
 
@@ -53,11 +53,19 @@ function registerTools(pi: ExtensionAPI, startupConfig: ReturnType<typeof resolv
     async execute(_toolCallId, rawParams, signal, onUpdate, ctx) {
       const params = rawParams as Record<string, any>;
       const queries = normalizeQueries(params);
-      const numResults = parseInteger(params.numResults, DEFAULT_NUM_RESULTS, "numResults", 1, MAX_NUM_RESULTS);
       if (queries.length === 0) throw new Error("web_search requires query or queries.");
 
       const config = runtimeConfig(pi, ctx.cwd, projectIsTrusted(ctx));
       assertProviderUnchanged("web_search", startupConfig.provider_search, config.provider_search);
+      const requestedResultLimit = parseInteger(
+        params.numResults ?? (config.provider_search === "brave" ? params.maxUrls : undefined),
+        DEFAULT_NUM_RESULTS,
+        "numResults",
+        1,
+      );
+      const effectiveResultLimit = capSearchResultLimit(config.provider_search, requestedResultLimit);
+      const requestedContextTokens = params.contextTokens == null ? undefined : parseInteger(params.contextTokens, 0, "contextTokens", 1);
+      const effectiveContextTokens = Math.min(requestedContextTokens ?? DEFAULT_SEARCH_CONTEXT_TOKENS, MAX_SEARCH_CONTEXT_TOKENS);
       const provider = createSearchProvider(config);
       const grouped = [];
       const progress = createProgress("search", config.provider_search, queries);
@@ -65,9 +73,20 @@ function registerTools(pi: ExtensionAPI, startupConfig: ReturnType<typeof resolv
       for (const query of queries) {
         markProgressCurrent(progress, query);
         emitProgress(onUpdate, progress);
-        const result = await provider.search({ ...params, query, numResults }, signal);
-        grouped.push({ query, results: result.results });
-        markProgressDone(progress, query, `${result.results.length} results`);
+        const result = await provider.search({ ...params, query, numResults: effectiveResultLimit, contextTokens: requestedContextTokens == null ? undefined : effectiveContextTokens }, signal);
+        const bounded = applySearchContextBudget(result.results, effectiveContextTokens * 4);
+        grouped.push({
+          query,
+          requestedResultLimit,
+          effectiveResultLimit: result.effectiveResultLimit ?? effectiveResultLimit,
+          requestedContextTokens,
+          effectiveContextTokens,
+          contextCharacters: bounded.contextCharacters,
+          ...(bounded.omittedContextCharacters ? { omittedContextCharacters: bounded.omittedContextCharacters } : {}),
+          resultCount: bounded.results.length,
+          results: bounded.results,
+        });
+        markProgressDone(progress, query, `${bounded.results.length} results`);
         emitProgress(onUpdate, progress);
       }
       const result = { provider: config.provider_search, queries: grouped };
@@ -212,7 +231,9 @@ export function buildSearchSchema(provider: SearchProviderName) {
   const props: Record<string, any> = {
     query: Type.Optional(Type.String({ description: "Single search query" })),
     queries: Type.Optional(Type.Array(Type.String(), { description: `Multiple related search queries (max ${MAX_QUERY_COUNT})`, maxItems: MAX_QUERY_COUNT })),
-    numResults: Type.Optional(int("Results per query", 1, MAX_NUM_RESULTS)),
+    numResults: Type.Optional(int("Desired results per query; capped by the active provider", 1)),
+    contextTokens: Type.Optional(int("Desired extracted context tokens; capped to the tool output budget and may enable provider extraction", 1)),
+    purpose: Type.Optional(Type.String({ description: "Optional task/use-case hint when supported by the provider", minLength: 1, maxLength: 2_000 })),
   };
   if (provider === "exa") Object.assign(props, {
     includeDomains: Type.Optional(Type.Array(Type.String())),
@@ -223,13 +244,32 @@ export function buildSearchSchema(provider: SearchProviderName) {
     endCrawlDate: Type.Optional(Type.String()),
     type: Type.Optional(Type.String()),
     category: Type.Optional(Type.String()),
+    maxAgeHours: Type.Optional(int("Maximum Exa cached content age in hours; 0 forces live crawl, -1 disables live crawl", -1)),
+  });
+  if (provider === "tinyfish") Object.assign(props, {
+    page: Type.Optional(int("Deprecated starting result page", 0, TINYFISH_MAX_PAGE)),
+    location: Type.Optional(Type.String()), language: Type.Optional(Type.String()),
+    includeDomains: Type.Optional(Type.Array(Type.String())), excludeDomains: Type.Optional(Type.Array(Type.String())),
+    domainType: Type.Optional(Type.Union([Type.Literal("web"), Type.Literal("news"), Type.Literal("research_paper")])),
+    afterDate: Type.Optional(Type.String({ pattern: "^\\d{4}-\\d{2}-\\d{2}$" })), beforeDate: Type.Optional(Type.String({ pattern: "^\\d{4}-\\d{2}-\\d{2}$" })),
+    recencyMinutes: Type.Optional(int("Freshness window in minutes", 1, 5_256_000)),
+    pubYearMin: Type.Optional(int("Minimum publication year", 0, 9_999)), pubYearMax: Type.Optional(int("Maximum publication year", 0, 9_999)),
   });
-  if (provider === "tinyfish") Object.assign(props, { page: Type.Optional(int("Result page", 1, 10)) });
   if (provider === "brave") Object.assign(props, {
-    country: Type.Optional(Type.String()), searchLang: Type.Optional(Type.String()), uiLang: Type.Optional(Type.String()), safesearch: Type.Optional(Type.String()), freshness: Type.Optional(Type.String()), maxUrls: Type.Optional(int("Maximum URLs", 1, MAX_NUM_RESULTS)),
+    country: Type.Optional(Type.String({ minLength: 2, maxLength: 2 })), searchLang: Type.Optional(Type.String({ minLength: 2, maxLength: 10 })),
+    safesearch: Type.Optional(Type.Union([Type.Literal("off"), Type.Literal("moderate"), Type.Literal("strict")])),
+    freshness: Type.Optional(Type.String()), spellcheck: Type.Optional(Type.Boolean()),
+    contextThresholdMode: Type.Optional(Type.Union([Type.Literal("disabled"), Type.Literal("strict"), Type.Literal("balanced"), Type.Literal("lenient")])),
+    maxSnippets: Type.Optional(int("Maximum context snippets", 1, 256)),
+    maxTokensPerUrl: Type.Optional(int("Maximum context tokens per URL", 512, 8_192)),
+    maxSnippetsPerUrl: Type.Optional(int("Maximum context snippets per URL", 1, 100)),
+    goggles: Type.Optional(Type.Union([Type.String(), Type.Array(Type.String(), { maxItems: 3 })])),
+    maxUrls: Type.Optional(int("Deprecated alias for numResults", 1)),
   });
   if (provider === "firecrawl") Object.assign(props, {
-    location: Type.Optional(Type.String()), country: Type.Optional(Type.String()), includeDomains: Type.Optional(Type.Array(Type.String())), excludeDomains: Type.Optional(Type.Array(Type.String())), categories: Type.Optional(Type.Array(Type.String())), tbs: Type.Optional(Type.String()), scrape: Type.Optional(Type.Boolean({ description: "Enable default markdown scrape-on-search" })), scrapeOptions: Type.Optional(Type.Object({}, { additionalProperties: true, description: "Firecrawl scrapeOptions for search." })),
+    location: Type.Optional(Type.String()), country: Type.Optional(Type.String()), includeDomains: Type.Optional(Type.Array(Type.String())), excludeDomains: Type.Optional(Type.Array(Type.String())),
+    categories: Type.Optional(Type.Array(Type.Union([Type.Literal("research"), Type.Literal("pdf"), Type.Literal("developer")]))),
+    tbs: Type.Optional(Type.String()), scrape: Type.Optional(Type.Boolean({ description: "Enable default markdown scrape-on-search" })), scrapeOptions: Type.Optional(Type.Object({}, { additionalProperties: true, description: "Firecrawl scrapeOptions for search." })),
   });
   return Type.Object(props, { additionalProperties: false });
 }
@@ -241,12 +281,23 @@ export function buildFetchSchema(provider: FetchProviderName) {
     offset: Type.Optional(int("Character offset for cached/ranged reads", 0, MAX_OFFSET)),
     limit: Type.Optional(int("Maximum characters to return", 1, MAX_LIMIT)),
     refresh: Type.Optional(Type.Boolean({ description: "Refetch even if cached" })),
+    maxAgeMs: Type.Optional(int("Maximum local/provider-cached page age in milliseconds; 0 requests live content where supported", 0)),
   };
+  if (provider === "exa") Object.assign(props, { maxAgeHours: Type.Optional(int("Maximum Exa cached content age in hours; 0 forces live crawl, -1 disables live crawl", -1)) });
   if (provider === "tinyfish") Object.assign(props, { format: Type.Optional(Type.Union([Type.Literal("markdown"), Type.Literal("html"), Type.Literal("json")])), links: Type.Optional(Type.Boolean()), imageLinks: Type.Optional(Type.Boolean()) });
+  if (provider === "tinyfish") Object.assign(props, {
+    purpose: Type.Optional(Type.String({ minLength: 1, maxLength: 2_000 })),
+    ttl: Type.Optional(int("Provider cache freshness tolerance in seconds; 0 prefers live fetch", 0)),
+    perUrlTimeoutMs: Type.Optional(int("Per-URL timeout in milliseconds", 1, 110_000)),
+    includeSelectors: Type.Optional(Type.Array(Type.String({ minLength: 1, maxLength: 1_000 }), { minItems: 1, maxItems: 20 })),
+    excludeSelectors: Type.Optional(Type.Array(Type.String({ minLength: 1, maxLength: 1_000 }), { minItems: 1, maxItems: 20 })),
+  });
   if (provider === "markdown_new") Object.assign(props, { method: Type.Optional(Type.Union([Type.Literal("auto"), Type.Literal("ai"), Type.Literal("browser")])), retainImages: Type.Optional(Type.Boolean()) });
   if (provider === "firecrawl") Object.assign(props, {
     format: Type.Optional(Type.Union([Type.Literal("markdown"), Type.Literal("html"), Type.Literal("json")])),
-    onlyMainContent: Type.Optional(Type.Boolean()), waitFor: Type.Optional(int("Milliseconds to wait", 0, 60_000)), mobile: Type.Optional(Type.Boolean()), location: Type.Optional(Type.String()), maxAge: Type.Optional(int("Maximum cached page age", 0)),
+    onlyMainContent: Type.Optional(Type.Boolean()), waitFor: Type.Optional(int("Milliseconds to wait", 0, 60_000)), mobile: Type.Optional(Type.Boolean()),
+    location: Type.Optional(Type.Object({ country: Type.Optional(Type.String()), languages: Type.Optional(Type.Array(Type.String())) }, { additionalProperties: false })),
+    maxAge: Type.Optional(int("Maximum cached page age in milliseconds", 0)),
   });
   return Type.Object(props, { additionalProperties: false });
 }
@@ -304,6 +355,7 @@ export async function fetchWithCache(providerName: FetchProviderName, params: Re
   const defaultLimit = urls.length > 1 ? MULTI_FETCH_LIMIT : DEFAULT_FETCH_LIMIT;
   const limit = parseInteger(params.limit, defaultLimit, "limit", 1, MAX_LIMIT);
   const refresh = params.refresh === true;
+  const maxAgeMs = localCacheMaxAge(providerName, params);
 
   const pages = new Map<string, { page?: CachedPage; cached: boolean; refreshed: boolean; error?: string }>();
   const cacheKeys = new Map<string, string>();
@@ -312,7 +364,7 @@ export async function fetchWithCache(providerName: FetchProviderName, params: Re
     const cacheKey = buildCacheKey(providerName, url, params, config);
     cacheKeys.set(url, cacheKey);
     const cached = fetchCache.get(cacheKey);
-    if (cached && !refresh) {
+    if (cached && !refresh && (maxAgeMs == null || (maxAgeMs > 0 && Date.now() - cached.fetchedAt <= maxAgeMs))) {
       pages.set(url, { page: cached, cached: true, refreshed: false });
       onProgress?.({ status: "done", url, note: "cached" });
     } else {
@@ -356,6 +408,14 @@ export async function fetchWithCache(providerName: FetchProviderName, params: Re
   }) };
 }
 
+function localCacheMaxAge(provider: FetchProviderName, params: Record<string, any>): number | undefined {
+  if (params.maxAgeMs != null) return parseInteger(params.maxAgeMs, 0, "maxAgeMs", 0);
+  if (provider === "exa" && typeof params.maxAgeHours === "number" && params.maxAgeHours >= 0) return params.maxAgeHours * 3_600_000;
+  if (provider === "tinyfish" && typeof params.ttl === "number") return params.ttl * 1_000;
+  if (provider === "firecrawl" && typeof params.maxAge === "number") return params.maxAge;
+  return undefined;
+}
+
 
 export function pageSlice(page: CachedPage, offset: number, limit: number, cached: boolean, refreshed: boolean) {
   const total = page.content.length;
@@ -436,8 +496,20 @@ function isSearchResult(value: any): value is { provider?: string; queries: Arra
 }
 
 function boundSearchResult(value: { provider?: string; queries: Array<{ query?: string; results?: any[] }> }) {
-  const compact = limitStrings(value, 1_000) as typeof value;
-  const maxSnippet = Math.max(0, ...compact.queries.flatMap((group) => (group.results ?? []).map((item) => typeof item?.snippet === "string" ? item.snippet.length : 0)));
+  const compact = {
+    ...(limitStrings(value, 1_000) as Record<string, unknown>),
+    queries: value.queries.map((group) => ({
+      ...(limitStrings(group, 1_000) as Record<string, unknown>),
+      results: (group.results ?? []).map((item) => ({
+        ...(limitStrings(item, 1_000) as Record<string, unknown>),
+        ...(typeof item?.content === "string" ? { content: item.content } : {}),
+      })),
+    })),
+  } as typeof value;
+  const maxSnippet = Math.max(0, ...compact.queries.flatMap((group) => (group.results ?? []).flatMap((item) => [
+    typeof item?.snippet === "string" ? item.snippet.length : 0,
+    typeof item?.content === "string" ? item.content.length : 0,
+  ])));
   let low = 0;
   let high = maxSnippet;
   let best = searchResultWithLimits(compact, 0);
@@ -473,17 +545,29 @@ function boundSearchResult(value: { provider?: string; queries: Array<{ query?:
 function searchResultWithLimits(value: { provider?: string; queries: Array<{ query?: string; results?: any[] }> }, snippetChars: number, resultsPerQuery?: number) {
   return {
     ...value,
-    queries: value.queries.map((group) => ({
-      ...group,
-      results: (group.results ?? []).slice(0, resultsPerQuery).map((item) => {
-        if (typeof item?.snippet !== "string") return item;
-        if (snippetChars === 0) {
-          const { snippet: _snippet, ...metadata } = item;
-          return metadata;
+    queries: value.queries.map((group) => {
+      const originalContextCharacters = (group.results ?? []).reduce((total, item) => total + (typeof item?.content === "string" ? item.content.length : 0), 0);
+      const results = (group.results ?? []).slice(0, resultsPerQuery).map((item) => {
+        const bounded = { ...item };
+        for (const key of ["snippet", "content"] as const) {
+          if (typeof bounded[key] !== "string") continue;
+          if (snippetChars === 0) delete bounded[key];
+          else bounded[key] = safePrefix(bounded[key], snippetChars);
         }
-        return { ...item, snippet: safePrefix(item.snippet, snippetChars) };
-      }),
-    })),
+        return bounded;
+      });
+      const omittedResultCount = Math.max(0, (group.results?.length ?? 0) - results.length);
+      const contextCharacters = results.reduce((total, item) => total + (typeof item?.content === "string" ? item.content.length : 0), 0);
+      const omittedContextCharacters = (typeof (group as any).omittedContextCharacters === "number" ? (group as any).omittedContextCharacters : 0) + originalContextCharacters - contextCharacters;
+      return {
+        ...group,
+        contextCharacters,
+        ...(omittedContextCharacters ? { omittedContextCharacters } : {}),
+        resultCount: results.length,
+        ...(omittedResultCount ? { omittedResultCount } : {}),
+        results,
+      };
+    }),
   };
 }
 
@@ -521,12 +605,6 @@ function fetchResultWithContentLimit(value: { provider?: string; results: any[]
   };
 }
 
-function safePrefix(value: string, maxChars: number): string {
-  let end = Math.min(value.length, maxChars);
-  if (end > 0 && /[\uD800-\uDBFF]/.test(value[end - 1])) end--;
-  return value.slice(0, end);
-}
-
 function limitStrings(value: unknown, maxChars: number): unknown {
   if (typeof value === "string") return safePrefix(value, maxChars);
   if (Array.isArray(value)) return value.map((item) => limitStrings(item, maxChars));
@@ -534,9 +612,11 @@ function limitStrings(value: unknown, maxChars: number): unknown {
   return value;
 }
 
-function parseInteger(value: unknown, defaultValue: number, name: string, min: number, max: number): number {
+function parseInteger(value: unknown, defaultValue: number, name: string, min: number, max?: number): number {
   if (value == null) return defaultValue;
-  if (typeof value !== "number" || !Number.isInteger(value) || !Number.isFinite(value) || value < min || value > max) throw new Error(`${name} must be a finite integer between ${min} and ${max}.`);
+  if (typeof value !== "number" || !Number.isInteger(value) || !Number.isFinite(value) || value < min || (max != null && value > max)) {
+    throw new Error(`${name} must be a finite integer ${max == null ? `>= ${min}` : `between ${min} and ${max}`}.`);
+  }
   return value;
 }
 
@@ -565,7 +645,6 @@ function fetchConfigDefaults(provider: FetchProviderName, config?: any): Record<
 function providerScope(provider: FetchProviderName, config?: any): string {
   const keyMap: Partial<Record<FetchProviderName, string | undefined>> = {
     exa: config?.apiKeys?.exa,
-    exa_mcp: config?.apiKeys?.exa,
     tinyfish: config?.apiKeys?.tinyfish,
     firecrawl: config?.apiKeys?.firecrawl,
   };
@@ -591,7 +670,14 @@ function searchDetails(value: any) {
     provider: value.provider,
     queries: value.queries.map((q: any) => ({
       query: q.query,
+      requestedResultLimit: q.requestedResultLimit,
+      effectiveResultLimit: q.effectiveResultLimit,
+      requestedContextTokens: q.requestedContextTokens,
+      effectiveContextTokens: q.effectiveContextTokens,
+      contextCharacters: q.contextCharacters,
+      omittedContextCharacters: q.omittedContextCharacters,
       resultCount: (q.results ?? []).length,
+      omittedResultCount: q.omittedResultCount,
       results: (q.results ?? []).map((r: any) => ({ title: r.title, url: r.url, siteName: r.siteName, position: r.position })),
     })),
   };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-web-kit",
-  "version": "0.2.4",
+  "version": "0.3.0",
   "description": "Context-efficient web search and fetch tools for Pi.",
   "type": "module",
   "license": "MIT",
@@ -58,12 +58,12 @@
     "typebox": "*"
   },
   "devDependencies": {
-    "@earendil-works/pi-ai": "^0.80.10",
-    "@earendil-works/pi-coding-agent": "^0.80.10",
-    "@earendil-works/pi-tui": "^0.80.10",
-    "@types/node": "^26.1.1",
-    "tsx": "^4.23.1",
-    "typebox": "^1.3.6",
+    "@earendil-works/pi-ai": "^0.84.1",
+    "@earendil-works/pi-coding-agent": "^0.84.1",
+    "@earendil-works/pi-tui": "^0.84.1",
+    "@types/node": "^26.2.0",
+    "tsx": "^4.23.5",
+    "typebox": "^1.3.10",
     "typescript": "^7.0.2"
   },
   "publishConfig": {
@@ -71,5 +71,8 @@
   },
   "engines": {
     "node": ">=20.6.0"
+  },
+  "dependencies": {
+    "@mocito/install-telemetry": "0.1.1"
   }
 }

package/src/config.ts

@@ -3,11 +3,11 @@ import { homedir } from "node:os";
 import { join } from "node:path";
 import type { FetchProviderName, SearchProviderName, WebKitConfig } from "./types.js";
 
-const SEARCH = ["exa_mcp", "exa", "tinyfish", "brave", "firecrawl"] as const;
-const FETCH = ["exa_mcp", "exa", "tinyfish", "markdown_new", "firecrawl"] as const;
+const SEARCH = ["exa", "tinyfish", "brave", "firecrawl"] as const;
+const FETCH = ["exa", "tinyfish", "markdown_new", "firecrawl"] as const;
 const DEFAULT_CONFIG: WebKitConfig = {
-  provider_search: "exa_mcp",
-  provider_fetch: "exa_mcp",
+  provider_search: "exa",
+  provider_fetch: "exa",
   apiKeys: {},
   markdownNew: { method: "auto", retainImages: false },
 };

package/src/http.ts

@@ -32,10 +32,19 @@ export async function requestJson<T>(url: string, init: RequestInit & { timeoutM
 }
 
 export function asSnippet(value: unknown): string | undefined {
+  return asText(value)?.slice(0, 1000) || undefined;
+}
+
+export function asText(value: unknown): string | undefined {
   if (value == null) return undefined;
-  if (Array.isArray(value)) return value.filter(Boolean).join("\n").slice(0, 1000) || undefined;
-  if (typeof value === "string") return value.slice(0, 1000) || undefined;
-  return JSON.stringify(value).slice(0, 1000);
+  if (Array.isArray(value)) return value.filter(Boolean).map((item) => typeof item === "string" ? item : JSON.stringify(item)).join("\n") || undefined;
+  if (typeof value === "string") return value || undefined;
+  return JSON.stringify(value);
+}
+
+export function withoutContent(value: any): Record<string, unknown> {
+  const { text: _text, content: _content, markdown: _markdown, html: _html, summary: _summary, highlights: _highlights, ...metadata } = value ?? {};
+  return metadata;
 }
 
 export function normalizeUrls(input: { url?: string; urls?: string[] }, maxCount?: number): string[] {

package/src/install-telemetry.ts

@@ -1,17 +1,35 @@
 import { readFileSync } from "node:fs";
-import { mkdir, writeFile } from "node:fs/promises";
 import { join } from "node:path";
+import { fileURLToPath } from "node:url";
+import { reportInstallTelemetry as report } from "@mocito/install-telemetry";
 import { getAgentDir } from "@earendil-works/pi-coding-agent";
 
 const PACKAGE_NAME = "pi-web-kit";
-const INSTALL_TELEMETRY_URL = "https://mocito.dev/api/report-install";
-const INSTALL_TELEMETRY_TIMEOUT_MS = 5000;
+const INSTALL_TELEMETRY_ENDPOINT = "https://mocito.dev/api/report-install";
+const CI_ENVIRONMENT_VARIABLES = [
+  "APPVEYOR",
+  "BITBUCKET_BUILD_NUMBER",
+  "BUILDKITE",
+  "CIRCLECI",
+  "CODESPACES",
+  "DRONE",
+  "GITHUB_ACTIONS",
+  "GITLAB_CI",
+  "JENKINS_URL",
+  "NETLIFY",
+  "TEAMCITY_VERSION",
+  "TF_BUILD",
+  "TRAVIS",
+  "VERCEL",
+];
 
-type InstallTelemetryState = { lastReportedVersion?: string };
+interface PiSettingsDocument {
+  enableInstallTelemetry?: unknown;
+}
 
 function readJsonFile(path: string): unknown {
   try {
-    return JSON.parse(readFileSync(path, "utf8"));
+    return JSON.parse(readFileSync(path, "utf8")) as unknown;
   } catch {
     return {};
   }
@@ -22,48 +40,38 @@ function isTruthyEnvFlag(value: string | undefined): boolean {
   return value === "1" || value.toLowerCase() === "true" || value.toLowerCase() === "yes";
 }
 
-function isInstallTelemetryEnabled(): boolean {
-  if (isTruthyEnvFlag(process.env.PI_OFFLINE)) return false;
-  if (isTruthyEnvFlag(process.env.CI) || isTruthyEnvFlag(process.env.GITHUB_ACTIONS)) return false;
-  if (process.env.PI_TELEMETRY !== undefined) return isTruthyEnvFlag(process.env.PI_TELEMETRY);
-  const settings = readJsonFile(join(getAgentDir(), "settings.json")) as { enableInstallTelemetry?: unknown };
-  return settings.enableInstallTelemetry !== false;
+function isPresentEnvFlag(value: string | undefined): boolean {
+  if (!value) return false;
+  const normalized = value.toLowerCase();
+  return normalized !== "0" && normalized !== "false" && normalized !== "no";
 }
 
-function getPackageVersion(): string {
-  try {
-    const packageJson = JSON.parse(readFileSync(new URL("../package.json", import.meta.url), "utf8")) as { version?: unknown };
-    return typeof packageJson.version === "string" && packageJson.version.length > 0 ? packageJson.version : "0.0.0";
-  } catch {
-    return "0.0.0";
-  }
+export function isInstallTelemetryEnabled(env: NodeJS.ProcessEnv = process.env, settingsPath = join(getAgentDir(), "settings.json")): boolean {
+  if (isTruthyEnvFlag(env.CI)) return false;
+  if (CI_ENVIRONMENT_VARIABLES.some((name) => isPresentEnvFlag(env[name]))) return false;
+  if (isTruthyEnvFlag(env.PI_OFFLINE)) return false;
+
+  const settings = readJsonFile(settingsPath) as PiSettingsDocument;
+  if (settings.enableInstallTelemetry === false) return false;
+  if (env.PI_TELEMETRY !== undefined) return isTruthyEnvFlag(env.PI_TELEMETRY);
+  return true;
 }
 
-function getInstallTelemetryUserAgent(version: string): string {
-  const runtimeVersions = process.versions as NodeJS.ProcessVersions & { bun?: string };
-  const runtime = runtimeVersions.bun ? `bun/${runtimeVersions.bun}` : `node/${process.version}`;
-  return `${PACKAGE_NAME}/${version} (${process.platform}; ${runtime}; ${process.arch})`;
+function getPackageVersion(): string {
+  const packageJson = readJsonFile(fileURLToPath(new URL("../package.json", import.meta.url))) as { version?: unknown };
+  return typeof packageJson.version === "string" && packageJson.version.length > 0 ? packageJson.version : "0.0.0";
 }
 
-export async function reportInstallTelemetry(): Promise<void> {
+export function reportInstallTelemetry(): void {
   try {
-    if (!isInstallTelemetryEnabled()) return;
-
-    const version = getPackageVersion();
-    const telemetryDir = join(getAgentDir(), "extensions");
-    const statePath = join(telemetryDir, "pi-web-kit-install.json");
-    const state = readJsonFile(statePath) as InstallTelemetryState;
-    if (state.lastReportedVersion === version) return;
-
-    await mkdir(telemetryDir, { recursive: true });
-    await writeFile(statePath, `${JSON.stringify({ lastReportedVersion: version }, null, 2)}\n`);
-
-    const params = new URLSearchParams({ tool: PACKAGE_NAME, version });
-    await fetch(`${INSTALL_TELEMETRY_URL}?${params.toString()}`, {
-      headers: { "User-Agent": getInstallTelemetryUserAgent(version) },
-      signal: AbortSignal.timeout(INSTALL_TELEMETRY_TIMEOUT_MS),
-    });
+    void report({
+      endpoint: INSTALL_TELEMETRY_ENDPOINT,
+      tool: PACKAGE_NAME,
+      version: getPackageVersion(),
+      statePath: join(getAgentDir(), "extensions", "pi-web-kit-install.json"),
+      enabled: isInstallTelemetryEnabled(),
+    }).catch(() => undefined);
   } catch {
-    // Best-effort install telemetry: ignore settings, filesystem, and network failures.
+    // Best-effort telemetry: ignore local policy and filesystem failures.
   }
 }

package/src/limits.ts

@@ -1,8 +1,19 @@
+import type { SearchProviderName } from "./types.js";
+
 export const MAX_QUERY_COUNT = 5;
 export const MAX_URL_COUNT = 10;
 export const MAX_URL_LENGTH = 2_048;
 export const MAX_NUM_RESULTS = 20;
 export const DEFAULT_NUM_RESULTS = 10;
+export const DEFAULT_SEARCH_CONTEXT_TOKENS = 8_192;
+export const MAX_SEARCH_CONTEXT_TOKENS = 10_000;
+export const SEARCH_RESULT_LIMITS: Readonly<Record<SearchProviderName, number | undefined>> = {
+  exa: 100,
+  brave: 50,
+  firecrawl: 100,
+  tinyfish: undefined,
+};
+export const TINYFISH_MAX_PAGE = 10;
 export const MAX_OFFSET = 10_000_000;
 export const MAX_LIMIT = 100_000;
 export const DEFAULT_FETCH_LIMIT = 30_000;
@@ -12,3 +23,31 @@ export const FETCH_CACHE_MAX_ENTRIES = 100;
 export const FETCH_CACHE_MAX_BYTES = 20 * 1024 * 1024;
 export const FETCH_CACHE_TTL_MS = 30 * 60 * 1000;
 export const FETCH_CONCURRENCY = 3;
+
+export function capSearchResultLimit(provider: SearchProviderName, requested: number): number {
+  return Math.min(requested, SEARCH_RESULT_LIMITS[provider] ?? requested);
+}
+
+export function applySearchContextBudget(results: Array<{ content?: string; [key: string]: unknown }>, maxCharacters: number) {
+  let remaining = maxCharacters;
+  let remainingContentResults = results.filter((result) => typeof result.content === "string").length;
+  let contextCharacters = 0;
+  let omittedContextCharacters = 0;
+  const bounded = results.map((result) => {
+    if (typeof result.content !== "string") return result;
+    const share = Math.max(0, Math.floor(remaining / remainingContentResults));
+    const content = safePrefix(result.content, share);
+    remaining -= content.length;
+    remainingContentResults--;
+    contextCharacters += content.length;
+    omittedContextCharacters += result.content.length - content.length;
+    return { ...result, content };
+  });
+  return { results: bounded, contextCharacters, omittedContextCharacters };
+}
+
+export function safePrefix(value: string, maxChars: number): string {
+  let end = Math.min(value.length, maxChars);
+  if (end > 0 && /[\uD800-\uDBFF]/.test(value[end - 1])) end--;
+  return value.slice(0, end);
+}

package/src/providers/brave.ts

@@ -1,4 +1,4 @@
-import { asSnippet, requestJson } from "../http.js";
+import { asSnippet, asText, requestJson } from "../http.js";
 import type { SearchInput, SearchProvider, WebKitConfig } from "../types.js";
 import { requireKey } from "../config.js";
 
@@ -7,24 +7,51 @@ export class BraveProvider implements SearchProvider {
   constructor(config: WebKitConfig) { this.key = requireKey(config, "brave"); }
 
   async search(input: SearchInput, signal?: AbortSignal) {
-    const url = new URL("https://api.search.brave.com/res/v1/llm/context");
-    url.searchParams.set("q", input.query);
+    const params: Record<string, string | number | boolean | string[]> = { q: boundedQuery(input.query) };
     if (input.numResults) {
-      const count = String(Math.max(1, Math.min(input.numResults, 20)));
-      url.searchParams.set("count", count);
-      url.searchParams.set("maximum_number_of_urls", String(input.maxUrls ?? count));
+      params.count = input.numResults;
+      params.maximum_number_of_urls = input.numResults;
+    }
+    if (input.contextTokens) params.maximum_number_of_tokens = Math.max(1_024, Math.min(input.contextTokens, 32_768));
+    if (typeof input.country === "string") params.country = input.country;
+    if (typeof input.searchLang === "string") params.search_lang = input.searchLang;
+    if (typeof input.safesearch === "string") params.safesearch = input.safesearch;
+    if (typeof input.freshness === "string") params.freshness = input.freshness;
+    if (typeof input.spellcheck === "boolean") params.spellcheck = input.spellcheck;
+    if (typeof input.contextThresholdMode === "string") params.context_threshold_mode = input.contextThresholdMode;
+    if (typeof input.maxSnippets === "number") params.maximum_number_of_snippets = input.maxSnippets;
+    if (typeof input.maxTokensPerUrl === "number") params.maximum_number_of_tokens_per_url = input.maxTokensPerUrl;
+    if (typeof input.maxSnippetsPerUrl === "number") params.maximum_number_of_snippets_per_url = input.maxSnippetsPerUrl;
+    if (typeof input.goggles === "string" || Array.isArray(input.goggles)) params.goggles = input.goggles;
+
+    const url = new URL("https://api.search.brave.com/res/v1/llm/context");
+    for (const [name, value] of Object.entries(params)) {
+      if (Array.isArray(value)) for (const item of value) url.searchParams.append(name, item);
+      else url.searchParams.set(name, String(value));
     }
-    if (typeof input.country === "string") url.searchParams.set("country", input.country);
-    if (typeof input.searchLang === "string") url.searchParams.set("search_lang", input.searchLang);
-    if (typeof input.uiLang === "string") url.searchParams.set("ui_lang", input.uiLang);
-    if (typeof input.safesearch === "string") url.searchParams.set("safesearch", input.safesearch);
-    if (typeof input.freshness === "string") url.searchParams.set("freshness", input.freshness);
-    const data = await requestJson<any>(url.toString(), { headers: { "X-Subscription-Token": this.key, accept: "application/json" }, signal, timeoutMs: 30000 });
+    const usePost = input.goggles != null || url.toString().length > 2_000;
+    const data = await requestJson<any>(usePost ? `${url.origin}${url.pathname}` : url.toString(), {
+      method: usePost ? "POST" : undefined,
+      headers: { "X-Subscription-Token": this.key, accept: "application/json", ...(usePost ? { "content-type": "application/json" } : {}) },
+      body: usePost ? JSON.stringify(params) : undefined,
+      signal,
+      timeoutMs: 30000,
+    });
     const sources = data.sources ?? data.web?.results ?? [];
     const snippets = data.grounding?.generic ?? [];
-    const rows = sources.length ? sources : snippets;
-    return { provider: "brave" as const, query: input.query, results: rows.map((r: any, i: number) => ({
-      title: r.title ?? r.name, url: r.url, snippet: asSnippet(r.snippets ?? r.description ?? snippets[i]?.snippets), siteName: r.site_name ?? r.source, position: i + 1,
-    })).filter((r: any) => r.url) };
+    const sourceRows = Array.isArray(sources)
+      ? sources
+      : Object.entries(sources).map(([url, source]: [string, any]) => ({ ...source, url }));
+    const rows = snippets.length ? snippets : sourceRows;
+    return { provider: "brave" as const, query: input.query, results: rows.map((r: any, i: number) => {
+      const content = asText(r.snippets ?? r.snippet ?? r.description ?? snippets[i]?.snippets);
+      return {
+        title: r.title ?? r.name, url: r.url, snippet: asSnippet(content), content, contentFormat: content ? "markdown" as const : undefined, siteName: r.site_name ?? r.source, position: i + 1,
+      };
+    }).filter((r: any) => r.url) };
   }
 }
+
+function boundedQuery(query: string): string {
+  return query.trim().split(/\s+/).slice(0, 50).join(" ").slice(0, 400);
+}

package/src/providers/exa.ts

@@ -1,4 +1,4 @@
-import { asSnippet, normalizeUrls, requestJson } from "../http.js";
+import { asSnippet, asText, normalizeUrls, requestJson, withoutContent } from "../http.js";
 import { DEFAULT_NUM_RESULTS } from "../limits.js";
 import { urlsMatch } from "../urls.js";
 import type { ExaCodeInput, ExaCodeResult, FetchInput, FetchProvider, SearchInput, SearchProvider, WebFetchResult, WebKitConfig } from "../types.js";
@@ -14,10 +14,15 @@ export class ExaProvider implements SearchProvider, FetchProvider {
   }
 
   async search(input: SearchInput, signal?: AbortSignal) {
+    const textCharacters = input.contextTokens ? Math.min(100_000, Math.max(1, Math.floor(input.contextTokens * 4 / (input.numResults ?? DEFAULT_NUM_RESULTS)))) : undefined;
     const body = {
       query: input.query,
       numResults: input.numResults ?? DEFAULT_NUM_RESULTS,
-      contents: input.contents ?? { highlights: true },
+      contents: input.contents ?? {
+        highlights: input.purpose ? { query: input.purpose } : true,
+        ...(textCharacters ? { text: { maxCharacters: textCharacters } } : {}),
+        ...(typeof input.maxAgeHours === "number" ? { maxAgeHours: input.maxAgeHours } : {}),
+      },
       includeDomains: input.includeDomains,
       excludeDomains: input.excludeDomains,
       startPublishedDate: input.startPublishedDate,
@@ -37,13 +42,18 @@ export class ExaProvider implements SearchProvider, FetchProvider {
     return {
       provider: "exa" as const,
       query: input.query,
-      results: (data.results ?? []).map((r: any, i: number) => ({
-        title: r.title,
-        url: r.url,
-        snippet: asSnippet(r.highlights ?? r.summary ?? r.text),
-        siteName: r.author ?? r.publishedDate,
-        position: i + 1,
-      })).filter((r: any) => r.url),
+      results: (data.results ?? []).map((r: any, i: number) => {
+        const content = asText(r.text ?? r.highlights ?? r.summary);
+        return {
+          title: r.title,
+          url: r.url,
+          snippet: asSnippet(content),
+          content,
+          contentFormat: content ? "markdown" as const : undefined,
+          siteName: r.author ?? r.publishedDate,
+          position: i + 1,
+        };
+      }).filter((r: any) => r.url),
     };
   }
 
@@ -53,7 +63,14 @@ export class ExaProvider implements SearchProvider, FetchProvider {
     const data = await requestJson<any>("https://api.exa.ai/contents", {
       method: "POST",
       headers: this.headers(),
-      body: JSON.stringify({ urls, text: true, highlights: false }),
+      body: JSON.stringify({
+        urls,
+        text: { maxCharacters: 100_000 },
+        highlights: false,
+        maxAgeHours: input.refresh === true
+          ? 0
+          : input.maxAgeHours ?? (typeof input.maxAgeMs === "number" ? Math.floor(input.maxAgeMs / 3_600_000) : undefined),
+      }),
       signal,
       timeoutMs: 45000,
     });
@@ -61,7 +78,7 @@ export class ExaProvider implements SearchProvider, FetchProvider {
     const primary: WebFetchResult = { provider: "exa", results: urls.map((url, i) => {
       const r: any = list.find((item: any) => urlsMatch(item.url, url)) ?? list[i];
       if (!r) return { url, error: "No content returned by Exa contents endpoint." };
-      return { url: r.url ?? url, title: r.title, content: r.text ?? r.summary ?? "", format: "markdown" as const, metadata: r };
+      return { url: r.url ?? url, title: r.title, content: r.text ?? r.summary ?? "", format: "markdown" as const, metadata: withoutContent(r) };
     }) };
     return applyExaFetchFallbacks(this.config, input, urls, primary, signal);
   }

package/src/providers/firecrawl.ts

@@ -1,4 +1,4 @@
-import { asSnippet, mapConcurrent, normalizeUrls, requestJson } from "../http.js";
+import { asSnippet, asText, mapConcurrent, normalizeUrls, requestJson } from "../http.js";
 import { DEFAULT_NUM_RESULTS, FETCH_CONCURRENCY } from "../limits.js";
 import type { FetchInput, FetchProvider, SearchInput, SearchProvider, WebKitConfig } from "../types.js";
 import { requireKey } from "../config.js";
@@ -20,13 +20,22 @@ export class FirecrawlProvider implements SearchProvider, FetchProvider {
         excludeDomains: input.excludeDomains,
         categories: input.categories,
         tbs: input.tbs,
-        scrapeOptions: input.scrapeOptions ?? (input.scrape === true ? { formats: ["markdown"] } : undefined),
+        scrapeOptions: input.scrapeOptions ?? (input.scrape === true || input.contextTokens ? {
+          formats: ["markdown"],
+          onlyMainContent: true,
+          maxAge: input.maxAge,
+        } : undefined),
       }),
     });
     const list = data.data?.web ?? data.web ?? data.data ?? [];
-    return { provider: "firecrawl" as const, query: input.query, results: list.map((r: any, i: number) => ({
-      title: r.title, url: r.url, snippet: asSnippet(r.description ?? r.markdown ?? r.content), siteName: r.siteName, position: i + 1,
-    })).filter((r: any) => r.url) };
+    return { provider: "firecrawl" as const, query: input.query, results: list.map((r: any, i: number) => {
+      const content = asText(r.markdown ?? r.content ?? r.highlights);
+      return {
+        title: r.title, url: r.url, snippet: asSnippet(r.highlights ?? r.description ?? content), content,
+        contentFormat: content ? (r.markdown ? "markdown" as const : "text" as const) : undefined,
+        siteName: r.siteName, position: i + 1,
+      };
+    }).filter((r: any) => r.url) };
   }
 
   async fetch(input: FetchInput, signal?: AbortSignal) {
@@ -44,7 +53,7 @@ export class FirecrawlProvider implements SearchProvider, FetchProvider {
             waitFor: input.waitFor,
             mobile: input.mobile,
             location: input.location,
-            maxAge: input.maxAge,
+            maxAge: input.refresh === true ? 0 : input.maxAge ?? input.maxAgeMs,
           }),
         });
         const d = data.data ?? data;

package/src/providers/index.ts

@@ -2,7 +2,6 @@ import { validateFetchProvider, validateSearchProvider } from "../config.js";
 import type { FetchProvider, SearchProvider, WebKitConfig } from "../types.js";
 import { BraveProvider } from "./brave.js";
 import { Context7Provider } from "./context7.js";
-import { ExaMcpProvider } from "./exa-mcp.js";
 import { ExaProvider } from "./exa.js";
 import { FirecrawlProvider } from "./firecrawl.js";
 import { MarkdownNewProvider } from "./markdown-new.js";
@@ -11,7 +10,6 @@ import { TinyFishProvider } from "./tinyfish.js";
 export function createSearchProvider(config: WebKitConfig): SearchProvider {
   validateSearchProvider(config.provider_search);
   switch (config.provider_search) {
-    case "exa_mcp": return new ExaMcpProvider(config);
     case "exa": return new ExaProvider(config);
     case "tinyfish": return new TinyFishProvider(config);
     case "brave": return new BraveProvider(config);
@@ -22,7 +20,6 @@ export function createSearchProvider(config: WebKitConfig): SearchProvider {
 export function createFetchProvider(config: WebKitConfig): FetchProvider {
   validateFetchProvider(config.provider_fetch);
   switch (config.provider_fetch) {
-    case "exa_mcp": return new ExaMcpProvider(config);
     case "exa": return new ExaProvider(config);
     case "tinyfish": return new TinyFishProvider(config);
     case "markdown_new": return new MarkdownNewProvider(config);

package/src/providers/markdown-new.ts

@@ -15,14 +15,22 @@ export class MarkdownNewProvider implements FetchProvider {
           body: JSON.stringify({
             url,
             method: input.method ?? this.config.markdownNew.method,
-            retainImages: input.retainImages ?? this.config.markdownNew.retainImages,
+            retain_images: input.retainImages ?? this.config.markdownNew.retainImages,
           }),
           signal,
           timeoutMs: 45_000,
         });
         const text = await res.text();
         if (!res.ok) throw new Error(`${res.status} ${res.statusText}: ${text.slice(0, 1000)}`);
-        return { url, content: text, format: "markdown" as const };
+        return {
+          url,
+          content: text,
+          format: "markdown" as const,
+          metadata: {
+            estimatedTokens: numberHeader(res.headers.get("x-markdown-tokens")),
+            conversionMethod: res.headers.get("x-markdown-method") ?? undefined,
+          },
+        };
       } catch (e) {
         return { url, error: e instanceof Error ? e.message : String(e) };
       }
@@ -30,3 +38,9 @@ export class MarkdownNewProvider implements FetchProvider {
     return { provider: "markdown_new" as const, results };
   }
 }
+
+function numberHeader(value: string | null): number | undefined {
+  if (value == null) return undefined;
+  const parsed = Number(value);
+  return Number.isFinite(parsed) ? parsed : undefined;
+}

package/src/providers/tinyfish.ts

@@ -1,6 +1,7 @@
-import { asSnippet, normalizeUrls, requestJson } from "../http.js";
-import { MAX_URL_COUNT } from "../limits.js";
-import type { FetchInput, FetchProvider, SearchInput, SearchProvider, WebKitConfig } from "../types.js";
+import { asSnippet, normalizeUrls, requestJson, withoutContent } from "../http.js";
+import { DEFAULT_NUM_RESULTS, MAX_URL_COUNT, TINYFISH_MAX_PAGE } from "../limits.js";
+import type { FetchInput, FetchProvider, SearchInput, SearchProvider, WebKitConfig, WebSearchResult } from "../types.js";
+import { canonicalWebUrl, urlsMatch } from "../urls.js";
 import { requireKey } from "../config.js";
 
 export class TinyFishProvider implements SearchProvider, FetchProvider {
@@ -8,15 +9,81 @@ export class TinyFishProvider implements SearchProvider, FetchProvider {
   constructor(config: WebKitConfig) { this.key = requireKey(config, "tinyfish"); }
 
   async search(input: SearchInput, signal?: AbortSignal) {
-    const url = new URL("https://api.search.tinyfish.ai/");
-    url.searchParams.set("query", input.query);
-    if (input.numResults) url.searchParams.set("limit", String(input.numResults));
-    if (typeof input.page === "number") url.searchParams.set("page", String(Math.min(input.page, 10)));
-    const data = await requestJson<any>(url.toString(), { headers: { "X-API-Key": this.key }, signal, timeoutMs: 10000 });
-    const list = data.results ?? data.data ?? data.web ?? [];
-    return { provider: "tinyfish" as const, query: input.query, results: list.map((r: any, i: number) => ({
-      title: r.title, url: r.url ?? r.link, snippet: asSnippet(r.snippet ?? r.description ?? r.text), siteName: r.siteName ?? r.source, position: r.position ?? i + 1,
-    })).filter((r: any) => r.url) };
+    const desired = input.numResults ?? DEFAULT_NUM_RESULTS;
+    const firstPage = typeof input.page === "number" ? Math.max(0, Math.min(input.page, TINYFISH_MAX_PAGE)) : 0;
+    const research = input.domainType === "research_paper";
+    const dates = orderedStrings(input.afterDate, input.beforeDate);
+    const years = orderedNumbers(input.pubYearMin, input.pubYearMax);
+    const results: WebSearchResult["results"] = [];
+    const seen = new Set<string>();
+    let lastPage = firstPage - 1;
+
+    for (let page = firstPage; page <= TINYFISH_MAX_PAGE && results.length < desired; page++) {
+      lastPage = page;
+      const url = new URL("https://api.search.tinyfish.ai/");
+      url.searchParams.set("query", input.query);
+      url.searchParams.set("page", String(page));
+      setString(url, "purpose", input.purpose);
+      setString(url, "location", input.location);
+      setString(url, "language", input.language);
+      setDomains(url, "include_domains", input.includeDomains);
+      setDomains(url, "exclude_domains", input.excludeDomains);
+      setString(url, "domain_type", input.domainType);
+      if (research) {
+        setNumber(url, "pub_year_min", years[0]);
+        setNumber(url, "pub_year_max", years[1]);
+      } else if (typeof input.recencyMinutes === "number") {
+        setNumber(url, "recency_minutes", input.recencyMinutes);
+      } else {
+        setString(url, "after_date", dates[0]);
+        setString(url, "before_date", dates[1]);
+      }
+      const data = await requestJson<any>(url.toString(), { headers: { "X-API-Key": this.key }, signal, timeoutMs: 10000 });
+      const list = data.results ?? data.data ?? data.web ?? [];
+      if (!Array.isArray(list) || list.length === 0) break;
+      for (const r of list) {
+        const resultUrl = r.url ?? r.link;
+        const key = typeof resultUrl === "string" ? canonicalResultUrl(resultUrl) : undefined;
+        if (!key || seen.has(key)) continue;
+        seen.add(key);
+        results.push({
+          title: r.title,
+          url: resultUrl,
+          snippet: asSnippet(r.snippet ?? r.description ?? r.text),
+          siteName: r.siteName ?? r.source,
+          position: results.length + 1,
+        });
+        if (results.length === desired) break;
+      }
+      if (typeof data.total_results === "number" && results.length >= data.total_results) break;
+    }
+
+    if (input.contextTokens && results.length) {
+      const contextCharacters = input.contextTokens * 4;
+      let extractedCharacters = 0;
+      for (let i = 0; i < results.length && extractedCharacters < contextCharacters;) {
+        // ponytail: budget for about 512 useful tokens per page; fetch another batch when pages are short.
+        const batchSize = Math.min(MAX_URL_COUNT, Math.max(1, Math.ceil((contextCharacters - extractedCharacters) / 2_048)));
+        const batch = results.slice(i, i + batchSize);
+        const fetched = await this.fetch({ urls: batch.map((item) => item.url), format: "markdown", purpose: input.purpose }, signal);
+        for (const item of batch) {
+          const page = fetched.results.find((candidate) => urlsMatch(candidate.url, item.url) || urlsMatch(candidate.metadata?.requestedUrl, item.url));
+          if (page?.content) {
+            item.content = page.content;
+            item.contentFormat = "markdown";
+            extractedCharacters += page.content.length;
+          }
+        }
+        i += batch.length;
+      }
+    }
+
+    return {
+      provider: "tinyfish" as const,
+      query: input.query,
+      effectiveResultLimit: lastPage === TINYFISH_MAX_PAGE && results.length < desired ? results.length : desired,
+      results,
+    };
   }
 
   async fetch(input: FetchInput, signal?: AbortSignal) {
@@ -25,20 +92,73 @@ export class TinyFishProvider implements SearchProvider, FetchProvider {
     const data = await requestJson<any>("https://api.fetch.tinyfish.ai", {
       method: "POST",
       headers: { "content-type": "application/json", "X-API-Key": this.key },
-      body: JSON.stringify({ urls, format: input.format ?? "markdown", links: input.links, image_links: input.imageLinks }),
+      body: JSON.stringify({
+        urls,
+        purpose: input.purpose,
+        format: input.format ?? "markdown",
+        links: input.links,
+        image_links: input.imageLinks,
+        ttl: input.refresh === true
+          ? 0
+          : input.ttl ?? (typeof input.maxAgeMs === "number" ? Math.floor(input.maxAgeMs / 1_000) : undefined),
+        per_url_timeout_ms: input.perUrlTimeoutMs,
+        include_selectors: input.includeSelectors,
+        exclude_selectors: input.excludeSelectors,
+      }),
       signal,
       timeoutMs: 150000,
     });
     const list = data.results ?? data.data ?? data.pages ?? [];
-    return { provider: "tinyfish" as const, results: urls.map((url, i) => {
-      const r = list.find((x: any) => (x.url ?? x.source_url) === url) ?? list[i];
-      if (!r) return { url, error: "No content returned by TinyFish." };
+    const errors = data.errors ?? [];
+    return { provider: "tinyfish" as const, results: urls.map((url) => {
+      const r = list.find((x: any) => urlsMatch(x.url ?? x.source_url, url));
+      const failure = errors.find((x: any) => urlsMatch(x.url, url));
+      if (!r) return { url, error: failure ? `${failure.error}${failure.status ? ` (${failure.status})` : ""}` : "No content returned by TinyFish." };
       const content = r.text ?? r.content ?? r.markdown ?? r.html;
-      return { url, content: stringifyContent(content), format: input.format ?? r.format ?? "markdown", title: r.title, metadata: r, error: r.error };
+      return {
+        url: r.final_url ?? r.url ?? url,
+        content: stringifyContent(content),
+        format: input.format ?? r.format ?? "markdown",
+        title: r.title,
+        metadata: { ...withoutContent(r), requestedUrl: url, finalUrl: r.final_url },
+        error: r.error,
+      };
     }) };
   }
 }
 
+function setString(url: URL, name: string, value: unknown) {
+  if (typeof value === "string" && value) url.searchParams.set(name, value);
+}
+
+function setNumber(url: URL, name: string, value: unknown) {
+  if (typeof value === "number") url.searchParams.set(name, String(value));
+}
+
+function setDomains(url: URL, name: string, value: unknown) {
+  if (Array.isArray(value) && value.length) url.searchParams.set(name, value.join(","));
+  else setString(url, name, value);
+}
+
+function orderedStrings(a: unknown, b: unknown): [string | undefined, string | undefined] {
+  const values = [a, b].filter((value): value is string => typeof value === "string").sort();
+  return [values[0], values[1]];
+}
+
+function orderedNumbers(a: unknown, b: unknown): [number | undefined, number | undefined] {
+  const values = [a, b].filter((value): value is number => typeof value === "number").sort((x, y) => x - y);
+  return [values[0], values[1]];
+}
+
 function stringifyContent(value: unknown): string | undefined {
   return typeof value === "string" ? value : JSON.stringify(value, null, 2);
 }
+
+function canonicalResultUrl(value: string): string {
+  try {
+    const canonical = canonicalWebUrl(value);
+    return canonical.endsWith("/") ? canonical.slice(0, -1) : canonical;
+  } catch {
+    return value;
+  }
+}

package/src/types.ts

@@ -1,5 +1,5 @@
-export type SearchProviderName = "exa_mcp" | "exa" | "tinyfish" | "brave" | "firecrawl";
-export type FetchProviderName = "exa_mcp" | "exa" | "tinyfish" | "markdown_new" | "firecrawl";
+export type SearchProviderName = "exa" | "tinyfish" | "brave" | "firecrawl";
+export type FetchProviderName = "exa" | "tinyfish" | "markdown_new" | "firecrawl";
 export type FetchFormat = "markdown" | "html" | "json";
 
 export interface WebKitConfig {
@@ -12,13 +12,16 @@ export interface WebKitConfig {
 export interface SearchInput {
   query: string;
   numResults?: number;
+  contextTokens?: number;
+  purpose?: string;
   [key: string]: unknown;
 }
 
 export interface WebSearchResult {
   provider: SearchProviderName;
   query: string;
-  results: Array<{ title?: string; url: string; snippet?: string; siteName?: string; position?: number }>;
+  effectiveResultLimit?: number;
+  results: Array<{ title?: string; url: string; snippet?: string; content?: string; contentFormat?: "markdown" | "text"; siteName?: string; position?: number }>;
 }
 
 export interface FetchInput {
@@ -27,9 +30,15 @@ export interface FetchInput {
   offset?: number;
   limit?: number;
   refresh?: boolean;
+  maxAgeMs?: number;
   format?: FetchFormat;
   links?: boolean;
   imageLinks?: boolean;
+  purpose?: string;
+  ttl?: number;
+  perUrlTimeoutMs?: number;
+  includeSelectors?: string[];
+  excludeSelectors?: string[];
   [key: string]: unknown;
 }
 

package/src/providers/exa-mcp.ts

@@ -1,102 +0,0 @@
-import { asSnippet, fetchWithTimeout, normalizeUrls } from "../http.js";
-import { DEFAULT_NUM_RESULTS } from "../limits.js";
-import { urlsMatch } from "../urls.js";
-import type { FetchInput, FetchProvider, SearchInput, SearchProvider, WebKitConfig } from "../types.js";
-import { applyExaFetchFallbacks } from "./fallback.js";
-
-export class ExaMcpProvider implements SearchProvider, FetchProvider {
-  private sessionId?: string;
-  private nextId = 1;
-  constructor(private config: WebKitConfig) {}
-
-  async search(input: SearchInput, signal?: AbortSignal) {
-    const result = await this.callTool("web_search_exa", { query: input.query, numResults: input.numResults ?? DEFAULT_NUM_RESULTS }, signal);
-    return { provider: "exa_mcp" as const, query: input.query, results: normalizeSearch(result) };
-  }
-
-  async fetch(input: FetchInput, signal?: AbortSignal) {
-    const urls = normalizeUrls(input);
-    const result = await this.callTool("web_fetch_exa", { urls }, signal);
-    const pages = normalizeFetch(result, urls);
-    return applyExaFetchFallbacks(this.config, input, urls, { provider: "exa_mcp" as const, results: pages }, signal);
-  }
-
-  private async callTool(name: string, args: Record<string, unknown>, signal?: AbortSignal): Promise<any> {
-    await this.initialize(signal);
-    return this.rpc("tools/call", { name, arguments: args }, signal);
-  }
-
-  private async initialize(signal?: AbortSignal) {
-    if (this.sessionId) return;
-    const data = await this.rpcRaw("initialize", { protocolVersion: "2025-06-18", capabilities: {}, clientInfo: { name: "pi-web-kit", version: "0.1.0" } }, signal);
-    this.sessionId = data.sessionId ?? data.payload?.sessionId;
-    if (!this.sessionId) throw new Error("Exa MCP initialize did not return an mcp-session-id response header.");
-    await this.rpcRaw("notifications/initialized", undefined, signal, true);
-  }
-
-  private async rpc(method: string, params: unknown, signal?: AbortSignal) {
-    const data = await this.rpcRaw(method, params, signal);
-    if (data.payload?.error) throw new Error(data.payload.error.message ?? JSON.stringify(data.payload.error));
-    return data.payload?.result ?? data.payload;
-  }
-
-  private async rpcRaw(method: string, params: unknown, signal?: AbortSignal, notification = false): Promise<{ payload?: any; sessionId?: string }> {
-    const headers: Record<string, string> = { "content-type": "application/json", accept: "application/json, text/event-stream" };
-    if (this.sessionId) headers["mcp-session-id"] = this.sessionId;
-    if (this.config.apiKeys.exa) headers["x-api-key"] = this.config.apiKeys.exa;
-    const body = notification ? { jsonrpc: "2.0", method, params } : { jsonrpc: "2.0", id: this.nextId++, method, params };
-    const res = await fetchWithTimeout("https://mcp.exa.ai/mcp", { method: "POST", headers, body: JSON.stringify(body), signal, timeoutMs: 45_000 });
-    const text = await res.text();
-    if (!res.ok) throw new Error(`Exa MCP ${method} failed: ${res.status} ${res.statusText}: ${text.slice(0, 1000)}`);
-    if (notification) return { sessionId: this.sessionId };
-    const sessionId = res.headers.get("mcp-session-id") ?? undefined;
-    const payload = parseMcpResponse(text, res.headers.get("content-type") ?? "");
-    return { payload, sessionId };
-  }
-}
-
-function parseMcpResponse(text: string, contentType: string): any {
-  if (!text.trim()) return undefined;
-  if (contentType.includes("text/event-stream") || text.startsWith("event:") || text.startsWith("data:")) {
-    const data = text.split(/\r?\n/).filter((line) => line.startsWith("data:")).map((line) => line.slice(5).trim()).join("\n");
-    return data ? JSON.parse(data) : undefined;
-  }
-  return JSON.parse(text);
-}
-
-function textFromContent(result: any): string {
-  const content = result?.content ?? result;
-  if (Array.isArray(content)) return content.map((c) => typeof c === "string" ? c : c.text ?? JSON.stringify(c)).join("\n");
-  return typeof content === "string" ? content : JSON.stringify(content);
-}
-
-function normalizeSearch(result: any) {
-  const structured = result?.structuredContent ?? result?.result ?? result;
-  const list = structured.results ?? structured.data ?? structured.items;
-  if (Array.isArray(list)) return list.map(toSearchResult).filter((r: any) => r.url);
-  const text = textFromContent(result);
-  const urls = [...text.matchAll(/https?:\/\/[^\s)\]}>"']+/g)].map((m) => m[0]);
-  return [...new Set(urls)].map((url, i) => ({ url, snippet: i === 0 ? text.slice(0, 1000) : undefined, position: i + 1 }));
-}
-
-function toSearchResult(r: any, index: number) {
-  return {
-    title: r.title,
-    url: r.url,
-    snippet: asSnippet(r.snippet ?? r.text ?? r.summary ?? r.highlights),
-    siteName: r.siteName,
-    position: index + 1,
-  };
-}
-
-function normalizeFetch(result: any, urls: string[]) {
-  const structured = result?.structuredContent ?? result?.result ?? result;
-  const list = structured.results ?? structured.data ?? structured.pages;
-  if (Array.isArray(list)) return urls.map((url, i) => {
-    const r = list.find((x: any) => urlsMatch(x.url, url)) ?? list[i];
-    if (!r) return { url, error: "No content returned by Exa MCP." };
-    return { url, title: r.title, content: r.markdown ?? r.text ?? r.content ?? r.html, format: "markdown" as const, metadata: r, error: r.error };
-  });
-  const text = textFromContent(result);
-  return urls.length <= 1 ? [{ url: urls[0] ?? "", content: text, format: "markdown" as const }] : urls.map((url) => ({ url, content: text, format: "markdown" as const }));
-}