pi-better-web-search 0.1.5 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +8 -0
- package/README.md +28 -10
- package/docs/README.zh-CN.md +21 -6
- package/docs/architecture.md +11 -4
- package/docs/publishing.md +6 -6
- package/docs/testing.md +4 -3
- package/lib/format.ts +6 -9
- package/lib/truncate-json.ts +299 -0
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,13 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.1.6
|
|
4
|
+
|
|
5
|
+
- Keep oversized results as valid JSON, sharing the character budget across
|
|
6
|
+
search results and fetched pages rather than cutting off the serialized tail.
|
|
7
|
+
Report affected paths and mark shortened strings; retain unchanged provider
|
|
8
|
+
data when it fits, and compact whitespace before dropping data.
|
|
9
|
+
- Fix premature truncation immediately below the configured output limit.
|
|
10
|
+
|
|
3
11
|
## 0.1.5
|
|
4
12
|
|
|
5
13
|
- Rank models inside a provider family by capability tier first (cheapest tier
|
package/README.md
CHANGED
|
@@ -8,7 +8,7 @@ provider supports it, automatically falls back across authenticated providers,
|
|
|
8
8
|
and returns provider data as-is.
|
|
9
9
|
|
|
10
10
|
[](https://github.com/zhexusun10/pi-better-web-search/actions/workflows/ci.yml)
|
|
11
|
-
[](https://www.npmjs.com/package/pi-better-web-search)
|
|
12
12
|
[](LICENSE)
|
|
13
13
|
[](https://pi.dev)
|
|
14
14
|
[](https://nodejs.org)
|
|
@@ -58,8 +58,8 @@ Requirements: **pi 1.3+** and **Node.js 22.19+**.
|
|
|
58
58
|
provider supports URL analysis. `urls` can be used alone or with `query`.
|
|
59
59
|
- **Raw results.** Provider data is returned in a JSON wrapper without
|
|
60
60
|
deduplication, URL normalization, junk filtering, re-ranking, summarization,
|
|
61
|
-
citation-marker insertion, or appended Sources sections.
|
|
62
|
-
|
|
61
|
+
citation-marker insertion, or appended Sources sections. A final character
|
|
62
|
+
guard applies structured truncation only to oversized output.
|
|
63
63
|
- **Correct semantics.** The tool description is exactly
|
|
64
64
|
`Search the web or analyze specified URLs.` The schema makes clear that
|
|
65
65
|
`urls` does not require `query`.
|
|
@@ -250,18 +250,36 @@ this:
|
|
|
250
250
|
- Citation markers are not inserted, and no "Sources" or "Additional results"
|
|
251
251
|
sections are appended.
|
|
252
252
|
- Redirects are not resolved and sources are not derived.
|
|
253
|
-
- Unknown provider fields are preserved.
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
253
|
+
- Unknown provider fields are preserved when the result fits the size guard.
|
|
254
|
+
|
|
255
|
+
A final output-size guard limits successful model-facing results to **64,000
|
|
256
|
+
Unicode code points** by default. Normal-sized results retain their original
|
|
257
|
+
fields and formatting. If indentation alone exceeds the limit, compact JSON is
|
|
258
|
+
returned without dropping data or setting the truncation flag.
|
|
259
|
+
|
|
260
|
+
When even compact JSON is too large, the guard allocates space across sibling
|
|
261
|
+
fields and array entries, redistributing unused space. It tries to preserve
|
|
262
|
+
identifying fields such as URLs, titles and errors, while shortening long
|
|
263
|
+
strings or omitting fields and trailing array entries when needed. The result
|
|
264
|
+
remains valid JSON. Shortened strings end in `…[truncated]` when space permits,
|
|
265
|
+
and the model-facing `_truncation` object reports `truncated`, `originalChars`
|
|
266
|
+
(the original indented JSON size), and `affectedPaths`. `moreAffectedPaths`
|
|
267
|
+
indicates that the path list is incomplete. Array paths can indicate omitted
|
|
268
|
+
entries; with very small budgets, not every URL or field can be preserved.
|
|
269
|
+
`details.truncated` is also set to `true` when data is actually removed.
|
|
270
|
+
|
|
271
|
+
Change the limit **before starting pi**:
|
|
258
272
|
|
|
259
273
|
```bash
|
|
260
274
|
export PI_BETTER_WEB_SEARCH_MAX_CHARS=100000
|
|
261
275
|
```
|
|
262
276
|
|
|
263
|
-
|
|
264
|
-
|
|
277
|
+
In PowerShell, use `$env:PI_BETTER_WEB_SEARCH_MAX_CHARS = "100000"`. The limit
|
|
278
|
+
is in code points, **not tokens**, and does not adapt to the current model's
|
|
279
|
+
context window. Extremely small limits below 19 characters cannot fit a
|
|
280
|
+
model-visible truncation flag; the result is still valid JSON and
|
|
281
|
+
`details.truncated` remains true. There is no continuation/pagination cache:
|
|
282
|
+
retry individual URLs or raise the limit if more content is needed.
|
|
265
283
|
|
|
266
284
|
## Exposure rules
|
|
267
285
|
|
package/docs/README.zh-CN.md
CHANGED
|
@@ -46,7 +46,7 @@ pi install ./path/to/pi-better-web-search
|
|
|
46
46
|
`urls`。`urls` 可以单独使用,也可以和 `query` 一起使用。
|
|
47
47
|
- **原始结果。** provider 数据放在 JSON 外层中返回,不去重、不规范化 URL、
|
|
48
48
|
不过滤垃圾 URL、不重排、不总结、不插入 citation 标记、不追加 Sources。
|
|
49
|
-
|
|
49
|
+
只有超出最终字符数限制时才进行结构化截断。
|
|
50
50
|
- **正确语义。** 工具描述是 `Search the web or analyze specified URLs.`;
|
|
51
51
|
schema 明确说明 `urls` 不依赖 `query`。
|
|
52
52
|
- **不碰系统提示词。** 不添加 prompt snippet、guideline、loadout hook、
|
|
@@ -215,17 +215,32 @@ Provider 数据放在 JSON 外层中返回,不做清理。示例:
|
|
|
215
215
|
- 结果不重排、不总结。
|
|
216
216
|
- 不插入 citation 标记,不追加 "Sources" 或 "Additional results"。
|
|
217
217
|
- 不解析 redirect,不派生 sources。
|
|
218
|
-
- provider
|
|
218
|
+
- 未超出大小限制时,provider 的未知字段也会保留。
|
|
219
219
|
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
220
|
+
最终大小保护默认限制成功结果不超过 **64,000 个 Unicode code point**。
|
|
221
|
+
未超限时保留原始字段及格式;如果只是缩进导致超限,会先压缩 JSON,
|
|
222
|
+
不丢失数据,也不标记为截断。
|
|
223
|
+
|
|
224
|
+
压缩后仍超限时,按字段和数组条目分配预算,未用完的额度会重新分配;
|
|
225
|
+
尽量保留 URL、标题和错误等标识信息。必要时会缩短字符串或省略字段、
|
|
226
|
+
数组末尾的条目,但返回值始终是合法 JSON。空间允许时,缩短的字符串以
|
|
227
|
+
`…[truncated]` 结尾;模型可见的 `_truncation` 对象包含 `truncated`、
|
|
228
|
+
`originalChars`(原始缩进 JSON 的字符数)和 `affectedPaths`。
|
|
229
|
+
`moreAffectedPaths` 表示路径列表不完整,数组路径也可能表示部分条目被省略。
|
|
230
|
+
预算极小时无法保证保留所有 URL 或字段。确实丢失数据时,
|
|
231
|
+
`details.truncated` 也会设为 `true`。
|
|
232
|
+
|
|
233
|
+
可在**启动 pi 之前**通过环境变量调整:
|
|
223
234
|
|
|
224
235
|
```bash
|
|
225
236
|
export PI_BETTER_WEB_SEARCH_MAX_CHARS=100000
|
|
226
237
|
```
|
|
227
238
|
|
|
228
|
-
|
|
239
|
+
PowerShell 使用 `$env:PI_BETTER_WEB_SEARCH_MAX_CHARS = "100000"`。
|
|
240
|
+
上限是 code point,**不是 token**,不会根据当前模型的上下文窗口自动调整。
|
|
241
|
+
极小上限(少于 19 字符)无法在模型可见内容中容纳截断标记,此时仍返回
|
|
242
|
+
合法 JSON,并在 `details.truncated` 中报告。当前没有续读或分页缓存;
|
|
243
|
+
需要更多内容时可单独重新抓取 URL 或提高上限。
|
|
229
244
|
|
|
230
245
|
## 暴露规则
|
|
231
246
|
|
package/docs/architecture.md
CHANGED
|
@@ -38,7 +38,7 @@ tool execute
|
|
|
38
38
|
├─ run search candidates until one succeeds
|
|
39
39
|
├─ run fetch candidates per URL until each URL succeeds or candidates end
|
|
40
40
|
├─ write search/fetch preferences + fingerprint on success
|
|
41
|
-
└─ format raw JSON
|
|
41
|
+
└─ format raw JSON; compact or structurally truncate if the guard is exceeded
|
|
42
42
|
```
|
|
43
43
|
|
|
44
44
|
## Exposure
|
|
@@ -160,9 +160,16 @@ The result is a JSON wrapper:
|
|
|
160
160
|
|
|
161
161
|
Fields inside `search` and `fetched[].data` are provider data. The extension does
|
|
162
162
|
not deduplicate, normalize, filter, re-rank, summarize, insert citation markers,
|
|
163
|
-
or append source sections. The
|
|
164
|
-
|
|
165
|
-
|
|
163
|
+
or append source sections. The final guard keeps successful model-facing output
|
|
164
|
+
within 64,000 Unicode code points by default (`PI_BETTER_WEB_SEARCH_MAX_CHARS`).
|
|
165
|
+
It preserves ordinary formatted JSON, or uses compact JSON if that alone fits.
|
|
166
|
+
Otherwise it budgets sibling fields and array entries independently, preferring
|
|
167
|
+
identifiers when space is tight. Strings can be shortened, and fields or trailing
|
|
168
|
+
array entries omitted; the output remains valid JSON. `_truncation` records the
|
|
169
|
+
original indented JSON length and affected paths, while `details.truncated` marks
|
|
170
|
+
actual data loss. For very small limits below 19 characters, only the details
|
|
171
|
+
flag fits. This is a character guard, not a token or model-context limit; it does
|
|
172
|
+
not provide pagination or retain the complete provider payload for later use.
|
|
166
173
|
|
|
167
174
|
## Provider adapter contract
|
|
168
175
|
|
package/docs/publishing.md
CHANGED
|
@@ -30,7 +30,7 @@ local `pi-web-search` reference checkout, `.env` files, and `DESIGN.md`.
|
|
|
30
30
|
npm run build
|
|
31
31
|
```
|
|
32
32
|
|
|
33
|
-
This produces `dist/pi-better-web-search-0.1.
|
|
33
|
+
This produces `dist/pi-better-web-search-0.1.6.tgz`. The tarball is not
|
|
34
34
|
committed.
|
|
35
35
|
|
|
36
36
|
## Publish
|
|
@@ -42,7 +42,7 @@ npm publish --access public
|
|
|
42
42
|
Alternatively, publish the verified tarball directly:
|
|
43
43
|
|
|
44
44
|
```bash
|
|
45
|
-
npm publish dist/pi-better-web-search-0.1.
|
|
45
|
+
npm publish dist/pi-better-web-search-0.1.6.tgz --access public
|
|
46
46
|
```
|
|
47
47
|
|
|
48
48
|
`prepublishOnly` reruns the checks, tests, and package verification. npm may
|
|
@@ -51,10 +51,10 @@ require browser or OTP authentication.
|
|
|
51
51
|
## Tag and release
|
|
52
52
|
|
|
53
53
|
```bash
|
|
54
|
-
git tag -a v0.1.
|
|
55
|
-
git push origin v0.1.
|
|
56
|
-
gh release create v0.1.
|
|
57
|
-
--title "v0.1.
|
|
54
|
+
git tag -a v0.1.6 -m "v0.1.6"
|
|
55
|
+
git push origin v0.1.6
|
|
56
|
+
gh release create v0.1.6 dist/pi-better-web-search-0.1.6.tgz \
|
|
57
|
+
--title "v0.1.6" \
|
|
58
58
|
--notes-file /path/to/release-notes.md
|
|
59
59
|
```
|
|
60
60
|
|
package/docs/testing.md
CHANGED
|
@@ -12,12 +12,12 @@ npm run verify:package
|
|
|
12
12
|
```
|
|
13
13
|
|
|
14
14
|
- `npm run check` runs `tsc --noEmit`.
|
|
15
|
-
- `npm test` runs the
|
|
15
|
+
- `npm test` runs the unit and integration tests with Node's test runner.
|
|
16
16
|
- `npm run verify:package` runs `npm pack --dry-run --json` and checks that the
|
|
17
17
|
tarball contains runtime sources, docs, assets, and metadata, while excluding
|
|
18
18
|
tests, scripts, CI files, the local reference checkout, and `DESIGN.md`.
|
|
19
19
|
- `npm run build` runs the checks, tests, package verification, and packs
|
|
20
|
-
`dist/pi-better-web-search-0.1.
|
|
20
|
+
`dist/pi-better-web-search-0.1.6.tgz`.
|
|
21
21
|
|
|
22
22
|
Tests import the TypeScript sources directly. The suite mocks `globalThis.fetch`
|
|
23
23
|
and uses temporary cache directories. Coverage includes:
|
|
@@ -30,7 +30,8 @@ and uses temporary cache directories. Coverage includes:
|
|
|
30
30
|
- search and fetch fallback, including per-URL fallback;
|
|
31
31
|
- provider request shapes and SSE parsing;
|
|
32
32
|
- HTTP/stream errors and abort propagation;
|
|
33
|
-
- character-guard truncation
|
|
33
|
+
- structured character-guard truncation, valid JSON, Unicode and escaping,
|
|
34
|
+
boundary lengths, and fair allocation across search results and URLs;
|
|
34
35
|
- one-tool registration and no system-prompt hooks.
|
|
35
36
|
|
|
36
37
|
## Live DeepSeek test
|
package/lib/format.ts
CHANGED
|
@@ -1,9 +1,10 @@
|
|
|
1
1
|
import type { AgentToolResult } from "@earendil-works/pi-coding-agent";
|
|
2
|
+
import { formatJsonWithinLimit } from "./truncate-json.ts";
|
|
2
3
|
|
|
3
|
-
const configuredOutputLimit = Number(process.env.PI_BETTER_WEB_SEARCH_MAX_CHARS);
|
|
4
|
+
const configuredOutputLimit = Math.floor(Number(process.env.PI_BETTER_WEB_SEARCH_MAX_CHARS));
|
|
4
5
|
export const MAX_OUTPUT_CHARS =
|
|
5
|
-
Number.isFinite(configuredOutputLimit) && configuredOutputLimit
|
|
6
|
-
?
|
|
6
|
+
Number.isFinite(configuredOutputLimit) && configuredOutputLimit >= 1
|
|
7
|
+
? configuredOutputLimit
|
|
7
8
|
: 64_000;
|
|
8
9
|
|
|
9
10
|
export interface WebSearchOutput {
|
|
@@ -41,8 +42,6 @@ export interface WebSearchDetails {
|
|
|
41
42
|
error?: string;
|
|
42
43
|
}
|
|
43
44
|
|
|
44
|
-
const TRUNCATION_MARKER = "\n\n[Truncated]";
|
|
45
|
-
|
|
46
45
|
/**
|
|
47
46
|
* Truncate by Unicode code point so a surrogate pair is never split. The
|
|
48
47
|
* returned `text` is at most `maxChars` code points.
|
|
@@ -61,11 +60,9 @@ export function formatSuccess(
|
|
|
61
60
|
output: WebSearchOutput,
|
|
62
61
|
details: WebSearchDetails,
|
|
63
62
|
): AgentToolResult<WebSearchDetails> {
|
|
64
|
-
const
|
|
65
|
-
const marker = TRUNCATION_MARKER.slice(0, Math.max(0, MAX_OUTPUT_CHARS));
|
|
66
|
-
const { text, truncated } = truncateText(json, Math.max(0, MAX_OUTPUT_CHARS - marker.length));
|
|
63
|
+
const { text, truncated } = formatJsonWithinLimit(output, MAX_OUTPUT_CHARS);
|
|
67
64
|
return {
|
|
68
|
-
content: [{ type: "text", text
|
|
65
|
+
content: [{ type: "text", text }],
|
|
69
66
|
details: truncated ? { ...details, truncated: true } : details,
|
|
70
67
|
};
|
|
71
68
|
}
|
|
@@ -0,0 +1,299 @@
|
|
|
1
|
+
type JsonValue = null | boolean | number | string | JsonValue[] | { [key: string]: JsonValue };
|
|
2
|
+
|
|
3
|
+
interface ChangeTracker {
|
|
4
|
+
paths: string[];
|
|
5
|
+
seen: Set<string>;
|
|
6
|
+
more: boolean;
|
|
7
|
+
}
|
|
8
|
+
|
|
9
|
+
interface TruncationInfo {
|
|
10
|
+
truncated: true;
|
|
11
|
+
originalChars: number;
|
|
12
|
+
affectedPaths: string[];
|
|
13
|
+
moreAffectedPaths: boolean;
|
|
14
|
+
}
|
|
15
|
+
|
|
16
|
+
interface Slot {
|
|
17
|
+
index: number;
|
|
18
|
+
key?: string;
|
|
19
|
+
value: JsonValue;
|
|
20
|
+
minimum: number;
|
|
21
|
+
full: number;
|
|
22
|
+
allocation: number;
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
const MAX_RECORDED_PATHS = 64;
|
|
26
|
+
const MAX_PATH_LENGTH = 160;
|
|
27
|
+
const MAX_PATH_BUDGET = 2_048;
|
|
28
|
+
const STRING_MARKER = "…[truncated]";
|
|
29
|
+
const IDENTITY_KEYS = ["url", "uri", "title", "name", "id", "error", "status", "query", "answer", "content", "text"];
|
|
30
|
+
const IMPORTANT_KEYS = [...IDENTITY_KEYS, "search", "fetched", "results", "data"];
|
|
31
|
+
const KEY_PRIORITY = new Map(IMPORTANT_KEYS.map((key, index) => [key, index]));
|
|
32
|
+
|
|
33
|
+
/** Count Unicode code points, rather than UTF-16 code units. */
|
|
34
|
+
function charCount(text: string): number {
|
|
35
|
+
let count = 0;
|
|
36
|
+
for (const _character of text) count += 1;
|
|
37
|
+
return count;
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
function recordChange(tracker: ChangeTracker, path: string): void {
|
|
41
|
+
if (tracker.seen.has(path)) return;
|
|
42
|
+
if (tracker.paths.length >= MAX_RECORDED_PATHS) {
|
|
43
|
+
tracker.more = true;
|
|
44
|
+
return;
|
|
45
|
+
}
|
|
46
|
+
tracker.seen.add(path);
|
|
47
|
+
tracker.paths.push(path);
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
function propertyPath(parent: string, key: string): string {
|
|
51
|
+
// A very long or unusual provider field name is represented by its parent,
|
|
52
|
+
// rather than allowing an unbounded field name to consume the path budget.
|
|
53
|
+
if (parent.length + key.length > MAX_PATH_LENGTH) return parent;
|
|
54
|
+
if (/^[A-Za-z_$][\w$]*$/.test(key)) return `${parent}.${key}`;
|
|
55
|
+
return `${parent}[${JSON.stringify(key)}]`;
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
function indexPath(parent: string, index: number): string {
|
|
59
|
+
const path = `${parent}[${index}]`;
|
|
60
|
+
return path.length <= MAX_PATH_LENGTH ? path : parent;
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
/** Encoded length of one code point inside a JSON string (without quotes). */
|
|
64
|
+
function encodedCharacterLength(character: string): number {
|
|
65
|
+
if (character.length === 2) return 1; // a surrogate pair is one code point
|
|
66
|
+
const code = character.charCodeAt(0);
|
|
67
|
+
if (character === '"' || character === "\\") return 2;
|
|
68
|
+
if (code === 8 || code === 9 || code === 10 || code === 12 || code === 13) return 2;
|
|
69
|
+
if (code < 32 || (code >= 0xd800 && code <= 0xdfff)) return 6;
|
|
70
|
+
return 1;
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
function fitString(value: string, budget: number): string {
|
|
74
|
+
// A JSON string needs two quotes. Include a marker whenever it fits; paths in
|
|
75
|
+
// _truncation also identify shortened strings when the budget is very small.
|
|
76
|
+
const marker = budget >= charCount(STRING_MARKER) + 2 ? STRING_MARKER : budget >= 3 ? "…" : "";
|
|
77
|
+
let remaining = budget - 2 - charCount(marker);
|
|
78
|
+
const prefix: string[] = [];
|
|
79
|
+
for (const character of value) {
|
|
80
|
+
const size = encodedCharacterLength(character);
|
|
81
|
+
if (size > remaining) break;
|
|
82
|
+
prefix.push(character);
|
|
83
|
+
remaining -= size;
|
|
84
|
+
}
|
|
85
|
+
return prefix.join("") + marker;
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
/** Distribute the available space fairly, giving unused space to larger values. */
|
|
89
|
+
function allocate(slots: Slot[], available: number): void {
|
|
90
|
+
let remaining = available;
|
|
91
|
+
for (const slot of slots) {
|
|
92
|
+
slot.allocation = slot.minimum;
|
|
93
|
+
remaining -= slot.minimum;
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
const pending = slots
|
|
97
|
+
.filter((slot) => slot.full > slot.minimum)
|
|
98
|
+
.sort((a, b) => (a.full - a.minimum) - (b.full - b.minimum) || a.index - b.index);
|
|
99
|
+
|
|
100
|
+
for (let i = 0; i < pending.length && remaining > 0; i += 1) {
|
|
101
|
+
const slot = pending[i];
|
|
102
|
+
const count = pending.length - i;
|
|
103
|
+
const share = Math.floor(remaining / count);
|
|
104
|
+
const demand = slot.full - slot.minimum;
|
|
105
|
+
if (demand <= share) {
|
|
106
|
+
slot.allocation += demand;
|
|
107
|
+
remaining -= demand;
|
|
108
|
+
continue;
|
|
109
|
+
}
|
|
110
|
+
|
|
111
|
+
const remainder = remaining % count;
|
|
112
|
+
for (let j = i; j < pending.length; j += 1) {
|
|
113
|
+
pending[j].allocation += share + (j - i < remainder ? 1 : 0);
|
|
114
|
+
}
|
|
115
|
+
break;
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
function createFitter(tracker: ChangeTracker) {
|
|
120
|
+
const sizes = new WeakMap<object, number>();
|
|
121
|
+
|
|
122
|
+
const sizeOf = (value: JsonValue): number => {
|
|
123
|
+
if (value !== null && typeof value === "object") {
|
|
124
|
+
const cached = sizes.get(value);
|
|
125
|
+
if (cached !== undefined) return cached;
|
|
126
|
+
const size = charCount(JSON.stringify(value));
|
|
127
|
+
sizes.set(value, size);
|
|
128
|
+
return size;
|
|
129
|
+
}
|
|
130
|
+
return charCount(JSON.stringify(value));
|
|
131
|
+
};
|
|
132
|
+
|
|
133
|
+
const minimumSize = (value: JsonValue): number => {
|
|
134
|
+
if (typeof value === "string" || Array.isArray(value)) return 2; // "" or []
|
|
135
|
+
if (value !== null && typeof value === "object") {
|
|
136
|
+
// An array of hundreds of empty objects is not useful. Reserve enough
|
|
137
|
+
// for one identifying or text field when the item has one; if the
|
|
138
|
+
// budget cannot accommodate all entries, report an omitted suffix.
|
|
139
|
+
for (const key of IDENTITY_KEYS) {
|
|
140
|
+
if (!Object.hasOwn(value, key)) continue;
|
|
141
|
+
const child = value[key];
|
|
142
|
+
if (child !== null && typeof child === "object") continue;
|
|
143
|
+
const childMinimum = typeof child === "string" ? Math.min(sizeOf(child), 128) : sizeOf(child);
|
|
144
|
+
return 2 + charCount(JSON.stringify(key)) + 1 + childMinimum;
|
|
145
|
+
}
|
|
146
|
+
return 2;
|
|
147
|
+
}
|
|
148
|
+
return sizeOf(value);
|
|
149
|
+
};
|
|
150
|
+
|
|
151
|
+
const fit = (value: JsonValue, budget: number, path: string): JsonValue => {
|
|
152
|
+
if (sizeOf(value) <= budget) return value;
|
|
153
|
+
|
|
154
|
+
if (typeof value === "string") {
|
|
155
|
+
recordChange(tracker, path);
|
|
156
|
+
return fitString(value, budget);
|
|
157
|
+
}
|
|
158
|
+
|
|
159
|
+
if (Array.isArray(value)) {
|
|
160
|
+
const slots: Slot[] = [];
|
|
161
|
+
let fixed = 2; // brackets and commas
|
|
162
|
+
let minimums = 0;
|
|
163
|
+
for (let index = 0; index < value.length; index += 1) {
|
|
164
|
+
const child = value[index];
|
|
165
|
+
const minimum = minimumSize(child);
|
|
166
|
+
const punctuation = slots.length > 0 ? 1 : 0;
|
|
167
|
+
if (fixed + punctuation + minimums + minimum > budget) {
|
|
168
|
+
// Preserve array order and report that one or more entries were omitted.
|
|
169
|
+
recordChange(tracker, path);
|
|
170
|
+
break;
|
|
171
|
+
}
|
|
172
|
+
fixed += punctuation;
|
|
173
|
+
minimums += minimum;
|
|
174
|
+
slots.push({ index, value: child, minimum, full: sizeOf(child), allocation: minimum });
|
|
175
|
+
}
|
|
176
|
+
allocate(slots, budget - fixed);
|
|
177
|
+
return slots.map((slot) => fit(slot.value, slot.allocation, indexPath(path, slot.index)));
|
|
178
|
+
}
|
|
179
|
+
|
|
180
|
+
if (value !== null && typeof value === "object") {
|
|
181
|
+
const entries = Object.entries(value);
|
|
182
|
+
// Prefer identifiers and the search/fetch containers when even the
|
|
183
|
+
// structural overhead does not fit. Restore the original field order in
|
|
184
|
+
// the result; no provider data is re-ranked.
|
|
185
|
+
const candidates = entries.map(([key, child], index) => ({ key, child, index }));
|
|
186
|
+
candidates.sort((a, b) =>
|
|
187
|
+
(KEY_PRIORITY.get(a.key) ?? IMPORTANT_KEYS.length) - (KEY_PRIORITY.get(b.key) ?? IMPORTANT_KEYS.length)
|
|
188
|
+
|| a.index - b.index,
|
|
189
|
+
);
|
|
190
|
+
const slots: Slot[] = [];
|
|
191
|
+
let fixed = 2; // braces, keys, colons and commas
|
|
192
|
+
let minimums = 0;
|
|
193
|
+
for (const { key, child, index } of candidates) {
|
|
194
|
+
const minimum = minimumSize(child);
|
|
195
|
+
const punctuation = (slots.length > 0 ? 1 : 0) + charCount(JSON.stringify(key)) + 1;
|
|
196
|
+
if (fixed + punctuation + minimums + minimum > budget) {
|
|
197
|
+
recordChange(tracker, propertyPath(path, key));
|
|
198
|
+
continue;
|
|
199
|
+
}
|
|
200
|
+
fixed += punctuation;
|
|
201
|
+
minimums += minimum;
|
|
202
|
+
slots.push({ index, key, value: child, minimum, full: sizeOf(child), allocation: minimum });
|
|
203
|
+
}
|
|
204
|
+
slots.sort((a, b) => a.index - b.index);
|
|
205
|
+
allocate(slots, budget - fixed);
|
|
206
|
+
const result: { [key: string]: JsonValue } = Object.create(null);
|
|
207
|
+
for (const slot of slots) {
|
|
208
|
+
const key = slot.key!;
|
|
209
|
+
result[key] = fit(slot.value, slot.allocation, propertyPath(path, key));
|
|
210
|
+
}
|
|
211
|
+
return result;
|
|
212
|
+
}
|
|
213
|
+
|
|
214
|
+
// Parents reserve the complete serialized size for non-string scalars.
|
|
215
|
+
throw new Error("Insufficient JSON budget for scalar value");
|
|
216
|
+
};
|
|
217
|
+
|
|
218
|
+
return fit;
|
|
219
|
+
}
|
|
220
|
+
|
|
221
|
+
function minimalTruncatedJson(limit: number): string {
|
|
222
|
+
for (const text of ['{"_truncation":{"truncated":true}}', '{"_truncated":true}', "{}", "0"]) {
|
|
223
|
+
if (text.length <= limit) return text;
|
|
224
|
+
}
|
|
225
|
+
throw new RangeError("The JSON output limit must be at least one character");
|
|
226
|
+
}
|
|
227
|
+
|
|
228
|
+
/**
|
|
229
|
+
* Preserve ordinary output exactly. If compact JSON still exceeds the limit,
|
|
230
|
+
* fit its fields and array entries independently and report affected paths.
|
|
231
|
+
* The returned text is valid JSON and never exceeds `maxChars` code points.
|
|
232
|
+
*/
|
|
233
|
+
export function formatJsonWithinLimit(
|
|
234
|
+
output: object,
|
|
235
|
+
maxChars: number,
|
|
236
|
+
): { text: string; truncated: boolean } {
|
|
237
|
+
if (!Number.isFinite(maxChars) || maxChars < 1) {
|
|
238
|
+
throw new RangeError("The JSON output limit must be at least one character");
|
|
239
|
+
}
|
|
240
|
+
const limit = Math.floor(maxChars);
|
|
241
|
+
const pretty = JSON.stringify(output, null, 2);
|
|
242
|
+
const originalChars = charCount(pretty);
|
|
243
|
+
if (originalChars <= limit) return { text: pretty, truncated: false };
|
|
244
|
+
|
|
245
|
+
// Work from the already serialized value, so JSON's handling of undefined,
|
|
246
|
+
// dates and toJSON is identical even if an input object is not plain data.
|
|
247
|
+
const parsed = JSON.parse(pretty) as { [key: string]: JsonValue };
|
|
248
|
+
const compact = JSON.stringify(parsed);
|
|
249
|
+
if (charCount(compact) <= limit) return { text: compact, truncated: false };
|
|
250
|
+
|
|
251
|
+
const info: TruncationInfo = {
|
|
252
|
+
truncated: true,
|
|
253
|
+
originalChars,
|
|
254
|
+
affectedPaths: [],
|
|
255
|
+
moreAffectedPaths: false,
|
|
256
|
+
};
|
|
257
|
+
// Account for the metadata entry and a comma joining it to the root object.
|
|
258
|
+
const overhead = charCount(JSON.stringify({ _truncation: info })) - 2 + 1;
|
|
259
|
+
if (limit - overhead < 2) return { text: minimalTruncatedJson(limit), truncated: true };
|
|
260
|
+
|
|
261
|
+
const build = (reservedPaths: number) => {
|
|
262
|
+
const tracker: ChangeTracker = { paths: [], seen: new Set(), more: false };
|
|
263
|
+
const fit = createFitter(tracker);
|
|
264
|
+
const fitted = fit(parsed, limit - overhead - reservedPaths, "$") as { [key: string]: JsonValue };
|
|
265
|
+
return { fitted, tracker };
|
|
266
|
+
};
|
|
267
|
+
|
|
268
|
+
let result = build(0);
|
|
269
|
+
// Reserve just enough room for the first pass's affected paths, capped so
|
|
270
|
+
// reporting never consumes a disproportionate share of the result budget.
|
|
271
|
+
const desiredPathSpace = Math.min(
|
|
272
|
+
MAX_PATH_BUDGET,
|
|
273
|
+
Math.max(64, Math.floor(limit * 0.05)),
|
|
274
|
+
charCount(JSON.stringify(result.tracker.paths)) - 2,
|
|
275
|
+
);
|
|
276
|
+
if (desiredPathSpace > 0 && limit - overhead - desiredPathSpace >= 2) {
|
|
277
|
+
result = build(desiredPathSpace);
|
|
278
|
+
}
|
|
279
|
+
|
|
280
|
+
const combined: { [key: string]: unknown } = Object.create(null);
|
|
281
|
+
combined._truncation = info;
|
|
282
|
+
for (const [key, value] of Object.entries(result.fitted)) {
|
|
283
|
+
if (key !== "_truncation") combined[key] = value;
|
|
284
|
+
}
|
|
285
|
+
|
|
286
|
+
let remaining = limit - charCount(JSON.stringify(combined));
|
|
287
|
+
for (const path of result.tracker.paths) {
|
|
288
|
+
const cost = charCount(JSON.stringify(path)) + (info.affectedPaths.length > 0 ? 1 : 0);
|
|
289
|
+
if (cost > remaining) {
|
|
290
|
+
info.moreAffectedPaths = true;
|
|
291
|
+
break;
|
|
292
|
+
}
|
|
293
|
+
info.affectedPaths.push(path);
|
|
294
|
+
remaining -= cost;
|
|
295
|
+
}
|
|
296
|
+
if (result.tracker.more) info.moreAffectedPaths = true;
|
|
297
|
+
|
|
298
|
+
return { text: JSON.stringify(combined), truncated: true };
|
|
299
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-better-web-search",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.6",
|
|
4
4
|
"description": "A better provider-native web_search tool for pi: raw results, automatic provider fallback, URL analysis, and native DeepSeek Search/Fetch.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|