pi-web-kit 0.2.2 → 0.2.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +20 -0
- package/README.md +10 -14
- package/extensions/index.ts +215 -26
- package/package.json +8 -8
- package/src/config.ts +11 -4
- package/src/providers/fallback.ts +6 -2
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,22 @@ This project follows the spirit of [Keep a Changelog](https://keepachangelog.com
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.2.4] - 2026-07-28
|
|
10
|
+
|
|
11
|
+
### Changed
|
|
12
|
+
|
|
13
|
+
- Rewrite `library_search`, `library_docs`, `web_search`, and `code_search` prompts so `library_docs` is the default for versioned library/framework/SDK docs (even familiar libraries), `library_search` is reserved for inspecting candidate matches, and `web_search`/`code_search` defer to it. Removes the "ambiguous library" wording that suppressed calls to well-known libraries. The `web_search`/`code_search` deferrals are emitted only when Context7 is configured, so installations without `CONTEXT7_API_KEY` are unaffected. Provider behavior and tool schemas are unchanged.
|
|
14
|
+
|
|
15
|
+
## [0.2.3] - 2026-07-17
|
|
16
|
+
|
|
17
|
+
### Fixed
|
|
18
|
+
|
|
19
|
+
- Honor project-local config only after Pi confirms project trust.
|
|
20
|
+
- Keep structured tool output valid JSON under the 50KB limit with continuation metadata.
|
|
21
|
+
- Derive opaque cache scopes from full API keys and hide internal cache keys from tool output.
|
|
22
|
+
- Prevent provider fetch results from being assigned to more than one requested URL.
|
|
23
|
+
- Remove the stale package-local lockfile in favor of the monorepo root lockfile.
|
|
24
|
+
|
|
9
25
|
## [0.2.2] - 2026-07-01
|
|
10
26
|
|
|
11
27
|
### Changed
|
|
@@ -14,6 +30,10 @@ This project follows the spirit of [Keep a Changelog](https://keepachangelog.com
|
|
|
14
30
|
|
|
15
31
|
## [0.2.1] - 2026-06-14
|
|
16
32
|
|
|
33
|
+
### Changed
|
|
34
|
+
|
|
35
|
+
- Simplify provider, configuration, cache, and URL handling while preserving the public tool surface.
|
|
36
|
+
|
|
17
37
|
## [0.2.0] - 2026-06-13
|
|
18
38
|
|
|
19
39
|
### Added
|
package/README.md
CHANGED
|
@@ -1,20 +1,16 @@
|
|
|
1
1
|
# pi-web-kit
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Give [Pi](https://pi.dev) current web knowledge, authoritative library docs, and real-world code examples without flooding model context.
|
|
4
4
|
|
|
5
|
-
`pi-web-kit`
|
|
5
|
+
`pi-web-kit` combines search, page reading, version-aware documentation, and code research behind five agent-ready tools with bounded, cache-aware output.
|
|
6
6
|
|
|
7
7
|
## Features
|
|
8
8
|
|
|
9
|
-
-
|
|
10
|
-
-
|
|
11
|
-
-
|
|
12
|
-
-
|
|
13
|
-
-
|
|
14
|
-
- Provider-tailored tool schemas at Pi startup/reload.
|
|
15
|
-
- URL validation: HTTP(S)-only, no embedded credentials, fragment stripping, duplicate removal, and length/count limits.
|
|
16
|
-
- In-memory fetch cache with TTL, LRU eviction, max entry count, max byte count, and cache keys based on provider/config/fetch-affecting options.
|
|
17
|
-
- Bounded provider concurrency and network timeouts.
|
|
9
|
+
- **Research the live web** — search current information and read single or multiple pages without leaving Pi.
|
|
10
|
+
- **Use docs that match the task** — resolve libraries and retrieve focused, version-aware documentation with code examples.
|
|
11
|
+
- **Find proven implementation patterns** — search practical usage, setup, migrations, and error context across real code.
|
|
12
|
+
- **Spend context wisely** — compact search results, chunked page reads, bounded output, and fetch caching keep research useful without overwhelming the model.
|
|
13
|
+
- **Choose your providers** — mix Exa, TinyFish, Brave, Firecrawl, markdown.new, Context7, and Exa Code based on coverage, cost, and credentials.
|
|
18
14
|
|
|
19
15
|
## Installation
|
|
20
16
|
|
|
@@ -89,7 +85,7 @@ Tool schemas are tailored to the configured providers at startup/reload, so only
|
|
|
89
85
|
|
|
90
86
|
## Configuration
|
|
91
87
|
|
|
92
|
-
Resolution order: defaults < environment variables < global config < project config < CLI flags.
|
|
88
|
+
Resolution order: defaults < environment variables < global config < trusted project config < CLI flags. Project config is ignored unless Pi trusts the current project, including in print, JSON, and RPC modes.
|
|
93
89
|
|
|
94
90
|
### Environment variables
|
|
95
91
|
|
|
@@ -217,7 +213,7 @@ Finds practical code examples, implementation context, setup snippets, migration
|
|
|
217
213
|
| Max `numResults` | 20 |
|
|
218
214
|
| Max URL length | 2048 characters |
|
|
219
215
|
|
|
220
|
-
Cache keys include the provider, canonical URL, fetch-affecting parameters, relevant provider defaults, and
|
|
216
|
+
Cache keys include the provider, canonical URL, fetch-affecting parameters, relevant provider defaults, and an opaque SHA-256 API-key/account scope. Internal cache keys are never returned in tool output. `refresh: true` bypasses and replaces the cached entry.
|
|
221
217
|
|
|
222
218
|
## Privacy and security
|
|
223
219
|
|
|
@@ -236,7 +232,7 @@ Report security issues privately. See [SECURITY.md](SECURITY.md).
|
|
|
236
232
|
| Invalid URL / scheme / credentials error | URL validation rejected the input. | Use an absolute `http:` or `https:` URL without username/password credentials. |
|
|
237
233
|
| Timeout error | Provider request exceeded its timeout. | Retry, reduce URL count, or switch provider. |
|
|
238
234
|
| No content returned | Provider returned no matching content or a redirected/canonicalized response could not be mapped. | Retry with `refresh: true`, fetch a single URL, or switch provider. |
|
|
239
|
-
| Large page is truncated | Tool output is bounded
|
|
235
|
+
| Large page is truncated | Tool output is bounded to valid JSON under 50KB. | Continue with the returned `range.nextOffset`. |
|
|
240
236
|
|
|
241
237
|
## Development
|
|
242
238
|
|
package/extensions/index.ts
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
|
+
import { createHash } from "node:crypto";
|
|
1
2
|
import { type ExtensionAPI, getAgentDir } from "@earendil-works/pi-coding-agent";
|
|
2
3
|
import { Text } from "@earendil-works/pi-tui";
|
|
3
4
|
import { Type } from "typebox";
|
|
4
5
|
import { fetchCache, type CachedPage } from "../src/cache.js";
|
|
5
6
|
import { resolveConfig } from "../src/config.js";
|
|
6
7
|
import { DEFAULT_FETCH_LIMIT, DEFAULT_NUM_RESULTS, MAX_LIMIT, MAX_NUM_RESULTS, MAX_OFFSET, MAX_QUERY_COUNT, MAX_URL_COUNT, MULTI_FETCH_LIMIT } from "../src/limits.js";
|
|
7
|
-
import { truncateText } from "../src/http.js";
|
|
8
8
|
import { createCodeSearchProvider, createContext7Provider, createFetchProvider, createSearchProvider } from "../src/providers/index.js";
|
|
9
9
|
import { mapFetchResults } from "../src/providers/fallback.js";
|
|
10
10
|
import type { FetchProviderName, SearchProviderName, WebFetchResult } from "../src/types.js";
|
|
@@ -22,17 +22,33 @@ export default function (pi: ExtensionAPI) {
|
|
|
22
22
|
type: "string",
|
|
23
23
|
});
|
|
24
24
|
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
25
|
+
let registeredConfig = "";
|
|
26
|
+
pi.on("session_start", (_event, ctx) => {
|
|
27
|
+
const startupConfig = runtimeConfig(pi, ctx.cwd, projectIsTrusted(ctx));
|
|
28
|
+
const signature = toolConfigSignature(startupConfig);
|
|
29
|
+
if (signature === registeredConfig) return;
|
|
30
|
+
registerTools(pi, startupConfig);
|
|
31
|
+
syncOptionalTools(pi, startupConfig);
|
|
32
|
+
registeredConfig = signature;
|
|
28
33
|
});
|
|
34
|
+
}
|
|
29
35
|
|
|
36
|
+
function registerTools(pi: ExtensionAPI, startupConfig: ReturnType<typeof resolveConfig>) {
|
|
30
37
|
pi.registerTool({
|
|
31
38
|
name: "web_search",
|
|
32
39
|
label: "Web Search",
|
|
33
40
|
description: buildSearchDescription(startupConfig.provider_search),
|
|
34
|
-
promptSnippet:
|
|
35
|
-
|
|
41
|
+
promptSnippet: startupConfig.apiKeys.context7
|
|
42
|
+
? "Search the live web for current events, news, and non-library topics."
|
|
43
|
+
: "Find current or external web information.",
|
|
44
|
+
promptGuidelines: [
|
|
45
|
+
startupConfig.apiKeys.context7
|
|
46
|
+
? "Use web_search for current events, news, and non-library topics."
|
|
47
|
+
: "Use web_search to find current or external web information.",
|
|
48
|
+
...(startupConfig.apiKeys.context7
|
|
49
|
+
? ["Prefer library_docs over web_search for versioned library/framework/SDK documentation."]
|
|
50
|
+
: []),
|
|
51
|
+
],
|
|
36
52
|
parameters: buildSearchSchema(startupConfig.provider_search),
|
|
37
53
|
async execute(_toolCallId, rawParams, signal, onUpdate, ctx) {
|
|
38
54
|
const params = rawParams as Record<string, any>;
|
|
@@ -40,7 +56,7 @@ export default function (pi: ExtensionAPI) {
|
|
|
40
56
|
const numResults = parseInteger(params.numResults, DEFAULT_NUM_RESULTS, "numResults", 1, MAX_NUM_RESULTS);
|
|
41
57
|
if (queries.length === 0) throw new Error("web_search requires query or queries.");
|
|
42
58
|
|
|
43
|
-
const config =
|
|
59
|
+
const config = runtimeConfig(pi, ctx.cwd, projectIsTrusted(ctx));
|
|
44
60
|
assertProviderUnchanged("web_search", startupConfig.provider_search, config.provider_search);
|
|
45
61
|
const provider = createSearchProvider(config);
|
|
46
62
|
const grouped = [];
|
|
@@ -80,7 +96,7 @@ export default function (pi: ExtensionAPI) {
|
|
|
80
96
|
throw new Error("web_fetch offset range reads require a single url, not urls.");
|
|
81
97
|
}
|
|
82
98
|
|
|
83
|
-
const config =
|
|
99
|
+
const config = runtimeConfig(pi, ctx.cwd, projectIsTrusted(ctx));
|
|
84
100
|
assertProviderUnchanged("web_fetch", startupConfig.provider_fetch, config.provider_fetch);
|
|
85
101
|
const progress = createProgress("fetch", config.provider_fetch, urls);
|
|
86
102
|
const result = await fetchWithCache(config.provider_fetch, params, urls, signal, config, (event) => {
|
|
@@ -101,16 +117,16 @@ export default function (pi: ExtensionAPI) {
|
|
|
101
117
|
pi.registerTool({
|
|
102
118
|
name: "library_search",
|
|
103
119
|
label: "Library Search",
|
|
104
|
-
description: "Resolve library, package, framework, SDK, API, or CLI names to canonical library IDs.",
|
|
105
|
-
promptSnippet: "
|
|
106
|
-
promptGuidelines: ["Use library_search
|
|
120
|
+
description: "Resolve library, package, framework, SDK, API, or CLI names to canonical library IDs with version, trust, and snippet metadata. Use when you need to inspect candidate matches (official sources, versions, forks); library_docs resolves names automatically.",
|
|
121
|
+
promptSnippet: "List candidate library IDs; library_docs resolves names on its own.",
|
|
122
|
+
promptGuidelines: ["Use library_search only to inspect matching libraries (official source, versions, forks) before a precise library_docs call.", "Not needed for ordinary doc lookups — library_docs resolves a libraryName for you."],
|
|
107
123
|
parameters: buildLibrarySearchSchema(),
|
|
108
124
|
async execute(_toolCallId, rawParams, signal, _onUpdate, ctx) {
|
|
109
125
|
const params = rawParams as Record<string, any>;
|
|
110
126
|
const libraryName = requiredString(params.libraryName, "libraryName");
|
|
111
127
|
const query = optionalString(params.query, "query") ?? libraryName;
|
|
112
128
|
const limit = parseInteger(params.limit, 10, "limit", 1, MAX_NUM_RESULTS);
|
|
113
|
-
const provider = createContext7Provider(runtimeConfig(pi, ctx.cwd));
|
|
129
|
+
const provider = createContext7Provider(runtimeConfig(pi, ctx.cwd, projectIsTrusted(ctx)));
|
|
114
130
|
const result = await provider.searchLibraries({ libraryName, query, fast: params.fast === true, limit }, signal);
|
|
115
131
|
return jsonToolResult(result);
|
|
116
132
|
},
|
|
@@ -119,15 +135,15 @@ export default function (pi: ExtensionAPI) {
|
|
|
119
135
|
pi.registerTool({
|
|
120
136
|
name: "library_docs",
|
|
121
137
|
label: "Library Docs",
|
|
122
|
-
description: "Fetch current, version-aware documentation and code examples for a library.",
|
|
123
|
-
promptSnippet: "Get current library
|
|
124
|
-
promptGuidelines: ["Use library_docs for
|
|
138
|
+
description: "Fetch current, version-aware documentation and code examples for a library. Pass libraryName to resolve it automatically, or a known libraryId.",
|
|
139
|
+
promptSnippet: "Get current, version-aware library/framework/SDK docs and code examples.",
|
|
140
|
+
promptGuidelines: ["Use library_docs for any library, framework, SDK, API, or CLI — even familiar ones like React, Next.js, Prisma, or Express. Training data may be outdated; this returns version-specific docs.", "Prefer library_docs over web_search or your own knowledge for library/API questions. Pass libraryName and query; you rarely need library_search first."],
|
|
125
141
|
parameters: buildLibraryDocsSchema(),
|
|
126
142
|
async execute(_toolCallId, rawParams, signal, _onUpdate, ctx) {
|
|
127
143
|
const params = rawParams as Record<string, any>;
|
|
128
144
|
const query = requiredString(params.query, "query");
|
|
129
145
|
const limit = parseInteger(params.limit, 10, "limit", 1, MAX_NUM_RESULTS);
|
|
130
|
-
const provider = createContext7Provider(runtimeConfig(pi, ctx.cwd));
|
|
146
|
+
const provider = createContext7Provider(runtimeConfig(pi, ctx.cwd, projectIsTrusted(ctx)));
|
|
131
147
|
let libraryId = optionalString(params.libraryId, "libraryId");
|
|
132
148
|
if (!libraryId) {
|
|
133
149
|
const libraryName = requiredString(params.libraryName, "libraryName");
|
|
@@ -147,13 +163,16 @@ export default function (pi: ExtensionAPI) {
|
|
|
147
163
|
label: "Code Search",
|
|
148
164
|
description: "Find practical code examples, usage patterns, setup snippets, migrations, and error context.",
|
|
149
165
|
promptSnippet: "Find real-world code examples, usage patterns, migrations, and error context.",
|
|
150
|
-
promptGuidelines: [
|
|
166
|
+
promptGuidelines: [
|
|
167
|
+
"Use code_search for real-world code examples, usage patterns, setup snippets, migrations, and error context from open source.",
|
|
168
|
+
...(startupConfig.apiKeys.context7 ? ["Prefer library_docs for official API/reference documentation."] : []),
|
|
169
|
+
],
|
|
151
170
|
parameters: buildCodeSearchSchema(),
|
|
152
171
|
async execute(_toolCallId, rawParams, signal, _onUpdate, ctx) {
|
|
153
172
|
const params = rawParams as Record<string, any>;
|
|
154
173
|
const query = requiredString(params.query, "query");
|
|
155
174
|
const tokensNum = parseTokensNum(params.tokensNum);
|
|
156
|
-
const provider = createCodeSearchProvider(runtimeConfig(pi, ctx.cwd));
|
|
175
|
+
const provider = createCodeSearchProvider(runtimeConfig(pi, ctx.cwd, projectIsTrusted(ctx)));
|
|
157
176
|
const result = await provider.searchCode({ query, tokensNum }, signal);
|
|
158
177
|
return jsonToolResult(result);
|
|
159
178
|
},
|
|
@@ -161,6 +180,27 @@ export default function (pi: ExtensionAPI) {
|
|
|
161
180
|
}
|
|
162
181
|
}
|
|
163
182
|
|
|
183
|
+
const OPTIONAL_TOOLS = ["library_search", "library_docs", "code_search"];
|
|
184
|
+
|
|
185
|
+
function toolConfigSignature(config: ReturnType<typeof resolveConfig>): string {
|
|
186
|
+
return JSON.stringify({
|
|
187
|
+
search: config.provider_search,
|
|
188
|
+
fetch: config.provider_fetch,
|
|
189
|
+
context7: !!config.apiKeys.context7,
|
|
190
|
+
exa: !!config.apiKeys.exa,
|
|
191
|
+
});
|
|
192
|
+
}
|
|
193
|
+
|
|
194
|
+
function syncOptionalTools(pi: ExtensionAPI, config: ReturnType<typeof resolveConfig>) {
|
|
195
|
+
if (typeof pi.getActiveTools !== "function" || typeof pi.setActiveTools !== "function") return;
|
|
196
|
+
const enabled = [
|
|
197
|
+
...(config.apiKeys.context7 ? ["library_search", "library_docs"] : []),
|
|
198
|
+
...(config.apiKeys.exa ? ["code_search"] : []),
|
|
199
|
+
];
|
|
200
|
+
const active = pi.getActiveTools().filter((name) => !OPTIONAL_TOOLS.includes(name));
|
|
201
|
+
pi.setActiveTools([...new Set([...active, ...enabled])]);
|
|
202
|
+
}
|
|
203
|
+
|
|
164
204
|
type ProgressKind = "search" | "fetch";
|
|
165
205
|
type ProgressItem = { label: string; status: "pending" | "current" | "done" | "error"; note?: string; error?: string };
|
|
166
206
|
type WebProgress = { kind: ProgressKind; provider: string; total: number; completed: number; items: ProgressItem[] };
|
|
@@ -320,7 +360,7 @@ export async function fetchWithCache(providerName: FetchProviderName, params: Re
|
|
|
320
360
|
export function pageSlice(page: CachedPage, offset: number, limit: number, cached: boolean, refreshed: boolean) {
|
|
321
361
|
const total = page.content.length;
|
|
322
362
|
const content = page.content.slice(offset, offset + limit);
|
|
323
|
-
return { url: page.requestedUrl ?? page.url, fetchedUrl: page.url, title: page.title, content, format: page.format, cached, refreshed,
|
|
363
|
+
return { url: page.requestedUrl ?? page.url, fetchedUrl: page.url, title: page.title, content, format: page.format, cached, refreshed, range: { offset, limit, returned: content.length, total, truncated: offset > 0 || offset + content.length < total, hasPrevious: offset > 0, hasNext: offset + content.length < total, nextOffset: offset + content.length < total ? offset + content.length : undefined } };
|
|
324
364
|
}
|
|
325
365
|
|
|
326
366
|
export function buildCacheKey(provider: FetchProviderName, url: string, params: Record<string, any>, config?: any): string {
|
|
@@ -335,13 +375,163 @@ export function buildCacheKey(provider: FetchProviderName, url: string, params:
|
|
|
335
375
|
return `${provider}\0${scope}\0${JSON.stringify(affecting)}\0${canonical}`;
|
|
336
376
|
}
|
|
337
377
|
|
|
338
|
-
function
|
|
339
|
-
return
|
|
378
|
+
function projectIsTrusted(ctx: { isProjectTrusted?: () => boolean }): boolean {
|
|
379
|
+
return typeof ctx.isProjectTrusted === "function" && ctx.isProjectTrusted() === true;
|
|
380
|
+
}
|
|
381
|
+
|
|
382
|
+
function runtimeConfig(pi: ExtensionAPI, cwd: string, projectTrusted: boolean) {
|
|
383
|
+
return resolveConfig(
|
|
384
|
+
{ providerSearch: pi.getFlag("web-provider-search"), providerFetch: pi.getFlag("web-provider-fetch") },
|
|
385
|
+
cwd,
|
|
386
|
+
process.env,
|
|
387
|
+
{ includeProject: projectTrusted },
|
|
388
|
+
);
|
|
389
|
+
}
|
|
390
|
+
|
|
391
|
+
const MAX_OUTPUT_BYTES = 50_000;
|
|
392
|
+
|
|
393
|
+
export function jsonToolResult(result: unknown) {
|
|
394
|
+
const bounded = boundStructuredResult(result);
|
|
395
|
+
return { content: [{ type: "text" as const, text: JSON.stringify(bounded) }], details: boundedDetails(bounded) };
|
|
396
|
+
}
|
|
397
|
+
|
|
398
|
+
function boundStructuredResult(result: unknown): unknown {
|
|
399
|
+
if (Buffer.byteLength(JSON.stringify(result)) <= MAX_OUTPUT_BYTES) return result;
|
|
400
|
+
|
|
401
|
+
if (isFetchResult(result)) {
|
|
402
|
+
const metadataBounded = limitFetchMetadata(result, 1_000);
|
|
403
|
+
let low = 0;
|
|
404
|
+
let high = Math.max(...metadataBounded.results.map((item: any) => typeof item.content === "string" ? item.content.length : 0));
|
|
405
|
+
const emptyContent = fetchResultWithContentLimit(metadataBounded, 0);
|
|
406
|
+
if (Buffer.byteLength(JSON.stringify(emptyContent)) > MAX_OUTPUT_BYTES) {
|
|
407
|
+
return { truncated: true, message: "Fetch metadata exceeded 50KB; retry with fewer URLs." };
|
|
408
|
+
}
|
|
409
|
+
let best: unknown = emptyContent;
|
|
410
|
+
while (low <= high) {
|
|
411
|
+
const middle = Math.floor((low + high) / 2);
|
|
412
|
+
const candidate = fetchResultWithContentLimit(metadataBounded, middle);
|
|
413
|
+
if (Buffer.byteLength(JSON.stringify(candidate)) <= MAX_OUTPUT_BYTES) {
|
|
414
|
+
best = candidate;
|
|
415
|
+
low = middle + 1;
|
|
416
|
+
} else {
|
|
417
|
+
high = middle - 1;
|
|
418
|
+
}
|
|
419
|
+
}
|
|
420
|
+
return best;
|
|
421
|
+
}
|
|
422
|
+
|
|
423
|
+
if (isSearchResult(result)) return boundSearchResult(result);
|
|
424
|
+
|
|
425
|
+
const compact = limitStrings(result, 1_000);
|
|
426
|
+
if (Buffer.byteLength(JSON.stringify(compact)) <= MAX_OUTPUT_BYTES) return compact;
|
|
427
|
+
return { truncated: true, message: "Structured result exceeded 50KB; refine the request or use smaller limits." };
|
|
428
|
+
}
|
|
429
|
+
|
|
430
|
+
function isFetchResult(value: any): value is { provider?: string; results: any[] } {
|
|
431
|
+
return !!value && Array.isArray(value.results) && value.results.some((item: any) => typeof item?.content === "string" && item?.range);
|
|
432
|
+
}
|
|
433
|
+
|
|
434
|
+
function isSearchResult(value: any): value is { provider?: string; queries: Array<{ query?: string; results?: any[] }> } {
|
|
435
|
+
return !!value && Array.isArray(value.queries) && value.queries.every((group: any) => Array.isArray(group?.results));
|
|
340
436
|
}
|
|
341
437
|
|
|
342
|
-
function
|
|
343
|
-
const
|
|
344
|
-
|
|
438
|
+
function boundSearchResult(value: { provider?: string; queries: Array<{ query?: string; results?: any[] }> }) {
|
|
439
|
+
const compact = limitStrings(value, 1_000) as typeof value;
|
|
440
|
+
const maxSnippet = Math.max(0, ...compact.queries.flatMap((group) => (group.results ?? []).map((item) => typeof item?.snippet === "string" ? item.snippet.length : 0)));
|
|
441
|
+
let low = 0;
|
|
442
|
+
let high = maxSnippet;
|
|
443
|
+
let best = searchResultWithLimits(compact, 0);
|
|
444
|
+
while (low <= high) {
|
|
445
|
+
const middle = Math.floor((low + high) / 2);
|
|
446
|
+
const candidate = searchResultWithLimits(compact, middle);
|
|
447
|
+
if (Buffer.byteLength(JSON.stringify(candidate)) <= MAX_OUTPUT_BYTES) {
|
|
448
|
+
best = candidate;
|
|
449
|
+
low = middle + 1;
|
|
450
|
+
} else {
|
|
451
|
+
high = middle - 1;
|
|
452
|
+
}
|
|
453
|
+
}
|
|
454
|
+
if (Buffer.byteLength(JSON.stringify(best)) <= MAX_OUTPUT_BYTES) return best;
|
|
455
|
+
|
|
456
|
+
low = 0;
|
|
457
|
+
high = Math.max(0, ...compact.queries.map((group) => group.results?.length ?? 0));
|
|
458
|
+
while (low <= high) {
|
|
459
|
+
const middle = Math.floor((low + high) / 2);
|
|
460
|
+
const candidate = searchResultWithLimits(compact, 0, middle);
|
|
461
|
+
if (Buffer.byteLength(JSON.stringify(candidate)) <= MAX_OUTPUT_BYTES) {
|
|
462
|
+
best = candidate;
|
|
463
|
+
low = middle + 1;
|
|
464
|
+
} else {
|
|
465
|
+
high = middle - 1;
|
|
466
|
+
}
|
|
467
|
+
}
|
|
468
|
+
return Buffer.byteLength(JSON.stringify(best)) <= MAX_OUTPUT_BYTES
|
|
469
|
+
? best
|
|
470
|
+
: { truncated: true, message: "Search metadata exceeded 50KB; retry with fewer queries or results." };
|
|
471
|
+
}
|
|
472
|
+
|
|
473
|
+
function searchResultWithLimits(value: { provider?: string; queries: Array<{ query?: string; results?: any[] }> }, snippetChars: number, resultsPerQuery?: number) {
|
|
474
|
+
return {
|
|
475
|
+
...value,
|
|
476
|
+
queries: value.queries.map((group) => ({
|
|
477
|
+
...group,
|
|
478
|
+
results: (group.results ?? []).slice(0, resultsPerQuery).map((item) => {
|
|
479
|
+
if (typeof item?.snippet !== "string") return item;
|
|
480
|
+
if (snippetChars === 0) {
|
|
481
|
+
const { snippet: _snippet, ...metadata } = item;
|
|
482
|
+
return metadata;
|
|
483
|
+
}
|
|
484
|
+
return { ...item, snippet: safePrefix(item.snippet, snippetChars) };
|
|
485
|
+
}),
|
|
486
|
+
})),
|
|
487
|
+
};
|
|
488
|
+
}
|
|
489
|
+
|
|
490
|
+
function limitFetchMetadata(value: { provider?: string; results: any[] }, maxChars: number) {
|
|
491
|
+
return {
|
|
492
|
+
...value,
|
|
493
|
+
results: value.results.map((item: any) => {
|
|
494
|
+
if (typeof item.content !== "string") return limitStrings(item, maxChars);
|
|
495
|
+
const content = item.content;
|
|
496
|
+
return { ...(limitStrings(item, maxChars) as Record<string, unknown>), content };
|
|
497
|
+
}),
|
|
498
|
+
};
|
|
499
|
+
}
|
|
500
|
+
|
|
501
|
+
function fetchResultWithContentLimit(value: { provider?: string; results: any[] }, maxChars: number) {
|
|
502
|
+
return {
|
|
503
|
+
...value,
|
|
504
|
+
results: value.results.map((item: any) => {
|
|
505
|
+
if (typeof item.content !== "string" || !item.range) return item;
|
|
506
|
+
const content = safePrefix(item.content, maxChars);
|
|
507
|
+
const returned = content.length;
|
|
508
|
+
const hasNext = item.range.offset + returned < item.range.total;
|
|
509
|
+
return {
|
|
510
|
+
...item,
|
|
511
|
+
content,
|
|
512
|
+
range: {
|
|
513
|
+
...item.range,
|
|
514
|
+
returned,
|
|
515
|
+
truncated: item.range.offset > 0 || hasNext,
|
|
516
|
+
hasNext,
|
|
517
|
+
nextOffset: hasNext ? item.range.offset + returned : undefined,
|
|
518
|
+
},
|
|
519
|
+
};
|
|
520
|
+
}),
|
|
521
|
+
};
|
|
522
|
+
}
|
|
523
|
+
|
|
524
|
+
function safePrefix(value: string, maxChars: number): string {
|
|
525
|
+
let end = Math.min(value.length, maxChars);
|
|
526
|
+
if (end > 0 && /[\uD800-\uDBFF]/.test(value[end - 1])) end--;
|
|
527
|
+
return value.slice(0, end);
|
|
528
|
+
}
|
|
529
|
+
|
|
530
|
+
function limitStrings(value: unknown, maxChars: number): unknown {
|
|
531
|
+
if (typeof value === "string") return safePrefix(value, maxChars);
|
|
532
|
+
if (Array.isArray(value)) return value.map((item) => limitStrings(item, maxChars));
|
|
533
|
+
if (value && typeof value === "object") return Object.fromEntries(Object.entries(value).map(([key, item]) => [key, limitStrings(item, maxChars)]));
|
|
534
|
+
return value;
|
|
345
535
|
}
|
|
346
536
|
|
|
347
537
|
function parseInteger(value: unknown, defaultValue: number, name: string, min: number, max: number): number {
|
|
@@ -380,7 +570,7 @@ function providerScope(provider: FetchProviderName, config?: any): string {
|
|
|
380
570
|
firecrawl: config?.apiKeys?.firecrawl,
|
|
381
571
|
};
|
|
382
572
|
const key = keyMap[provider];
|
|
383
|
-
return key ? `key:${String(key).
|
|
573
|
+
return key ? `key:${createHash("sha256").update(String(key)).digest("hex")}` : "default";
|
|
384
574
|
}
|
|
385
575
|
|
|
386
576
|
function assertProviderUnchanged(tool: string, startup: string, runtime: string) {
|
|
@@ -449,7 +639,6 @@ function fetchDetails(value: any) {
|
|
|
449
639
|
format: r.format,
|
|
450
640
|
cached: r.cached,
|
|
451
641
|
refreshed: r.refreshed,
|
|
452
|
-
cacheKey: r.cacheKey,
|
|
453
642
|
range: r.range,
|
|
454
643
|
error: r.error,
|
|
455
644
|
})),
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-web-kit",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.4",
|
|
4
4
|
"description": "Context-efficient web search and fetch tools for Pi.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -58,13 +58,13 @@
|
|
|
58
58
|
"typebox": "*"
|
|
59
59
|
},
|
|
60
60
|
"devDependencies": {
|
|
61
|
-
"@earendil-works/pi-ai": "^0.80.
|
|
62
|
-
"@earendil-works/pi-coding-agent": "^0.80.
|
|
63
|
-
"@earendil-works/pi-tui": "^0.80.
|
|
64
|
-
"@types/node": "^
|
|
65
|
-
"tsx": "^4.
|
|
66
|
-
"typebox": "^1.
|
|
67
|
-
"typescript": "^
|
|
61
|
+
"@earendil-works/pi-ai": "^0.80.10",
|
|
62
|
+
"@earendil-works/pi-coding-agent": "^0.80.10",
|
|
63
|
+
"@earendil-works/pi-tui": "^0.80.10",
|
|
64
|
+
"@types/node": "^26.1.1",
|
|
65
|
+
"tsx": "^4.23.1",
|
|
66
|
+
"typebox": "^1.3.6",
|
|
67
|
+
"typescript": "^7.0.2"
|
|
68
68
|
},
|
|
69
69
|
"publishConfig": {
|
|
70
70
|
"access": "public"
|
package/src/config.ts
CHANGED
|
@@ -24,7 +24,12 @@ type PartialConfig = Partial<Omit<WebKitConfig, "apiKeys" | "markdownNew">> & {
|
|
|
24
24
|
markdownNew?: Partial<WebKitConfig["markdownNew"]>;
|
|
25
25
|
};
|
|
26
26
|
|
|
27
|
-
export function resolveConfig(
|
|
27
|
+
export function resolveConfig(
|
|
28
|
+
flags: { providerSearch?: unknown; providerFetch?: unknown } = {},
|
|
29
|
+
cwd = process.cwd(),
|
|
30
|
+
env = process.env,
|
|
31
|
+
options: { includeProject?: boolean } = {},
|
|
32
|
+
): WebKitConfig {
|
|
28
33
|
let cfg = DEFAULT_CONFIG;
|
|
29
34
|
cfg = merge(cfg, {
|
|
30
35
|
provider_search: env.PI_WEB_KIT_PROVIDER_SEARCH as SearchProviderName | undefined,
|
|
@@ -38,9 +43,11 @@ export function resolveConfig(flags: { providerSearch?: unknown; providerFetch?:
|
|
|
38
43
|
},
|
|
39
44
|
});
|
|
40
45
|
const home = env.HOME ?? homedir();
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
46
|
+
const globalConfig = readConfigFile(join(home, ".pi/agent/pi-web-kit.json"));
|
|
47
|
+
if (globalConfig) cfg = merge(cfg, globalConfig);
|
|
48
|
+
if (options.includeProject !== false) {
|
|
49
|
+
const projectConfig = readConfigFile(join(cwd, ".pi-web-kit.json"));
|
|
50
|
+
if (projectConfig) cfg = merge(cfg, projectConfig);
|
|
44
51
|
}
|
|
45
52
|
cfg = merge(cfg, {
|
|
46
53
|
provider_search: flags.providerSearch as SearchProviderName | undefined,
|
|
@@ -69,8 +69,12 @@ export function mapFetchResults(requested: string[], fetched: WebFetchResult): M
|
|
|
69
69
|
const index = remaining.findIndex((item) => urlsMatch(item.url, url));
|
|
70
70
|
if (index >= 0) out.set(url, remaining.splice(index, 1)[0]);
|
|
71
71
|
}
|
|
72
|
-
//
|
|
73
|
-
|
|
72
|
+
// Providers may not echo the requested URL. Assign each unmatched result at
|
|
73
|
+
// most once, after exact canonical matches have consumed their results.
|
|
74
|
+
for (const url of requested) {
|
|
75
|
+
if (out.has(url) || remaining.length === 0) continue;
|
|
76
|
+
out.set(url, remaining.shift()!);
|
|
77
|
+
}
|
|
74
78
|
return out;
|
|
75
79
|
}
|
|
76
80
|
|