pi-better-web-search 0.1.5 → 0.1.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +19 -0
- package/README.md +35 -11
- package/docs/README.zh-CN.md +28 -7
- package/docs/architecture.md +17 -4
- package/docs/publishing.md +6 -6
- package/docs/testing.md +5 -3
- package/lib/format.ts +7 -10
- package/lib/truncate-json.ts +277 -0
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,24 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.1.7
|
|
4
|
+
|
|
5
|
+
- Keep recognized URL/URI/link and title/name/ID fields intact when truncating;
|
|
6
|
+
omit entries that cannot fit rather than returning partial links. Preserve
|
|
7
|
+
provider order and drop incomplete URL tokens at text boundaries.
|
|
8
|
+
- Raise the default model-facing output limit from 64,000 to 96,000 Unicode
|
|
9
|
+
code points; the environment override remains available.
|
|
10
|
+
- Replace model-facing truncation metadata and per-string markers with a single
|
|
11
|
+
`[truncated]` line at the end of oversized results. Keep the existing fair
|
|
12
|
+
allocation across search results and fetched pages.
|
|
13
|
+
|
|
14
|
+
## 0.1.6
|
|
15
|
+
|
|
16
|
+
- Keep oversized results as valid JSON, sharing the character budget across
|
|
17
|
+
search results and fetched pages rather than cutting off the serialized tail.
|
|
18
|
+
Report affected paths and mark shortened strings; retain unchanged provider
|
|
19
|
+
data when it fits, and compact whitespace before dropping data.
|
|
20
|
+
- Fix premature truncation immediately below the configured output limit.
|
|
21
|
+
|
|
3
22
|
## 0.1.5
|
|
4
23
|
|
|
5
24
|
- Rank models inside a provider family by capability tier first (cheapest tier
|
package/README.md
CHANGED
|
@@ -8,7 +8,7 @@ provider supports it, automatically falls back across authenticated providers,
|
|
|
8
8
|
and returns provider data as-is.
|
|
9
9
|
|
|
10
10
|
[](https://github.com/zhexusun10/pi-better-web-search/actions/workflows/ci.yml)
|
|
11
|
-
[](https://www.npmjs.com/package/pi-better-web-search)
|
|
12
12
|
[](LICENSE)
|
|
13
13
|
[](https://pi.dev)
|
|
14
14
|
[](https://nodejs.org)
|
|
@@ -58,8 +58,8 @@ Requirements: **pi 1.3+** and **Node.js 22.19+**.
|
|
|
58
58
|
provider supports URL analysis. `urls` can be used alone or with `query`.
|
|
59
59
|
- **Raw results.** Provider data is returned in a JSON wrapper without
|
|
60
60
|
deduplication, URL normalization, junk filtering, re-ranking, summarization,
|
|
61
|
-
citation-marker insertion, or appended Sources sections.
|
|
62
|
-
|
|
61
|
+
citation-marker insertion, or appended Sources sections. A final character
|
|
62
|
+
guard applies structured truncation only to oversized output.
|
|
63
63
|
- **Correct semantics.** The tool description is exactly
|
|
64
64
|
`Search the web or analyze specified URLs.` The schema makes clear that
|
|
65
65
|
`urls` does not require `query`.
|
|
@@ -250,18 +250,42 @@ this:
|
|
|
250
250
|
- Citation markers are not inserted, and no "Sources" or "Additional results"
|
|
251
251
|
sections are appended.
|
|
252
252
|
- Redirects are not resolved and sources are not derived.
|
|
253
|
-
- Unknown provider fields are preserved.
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
253
|
+
- Unknown provider fields are preserved when the result fits the size guard.
|
|
254
|
+
|
|
255
|
+
A final output-size guard limits successful model-facing results to **96,000
|
|
256
|
+
Unicode code points** by default. Normal-sized results retain their original
|
|
257
|
+
fields and formatting. If indentation alone exceeds the limit, compact JSON is
|
|
258
|
+
returned without dropping data or setting the truncation flag.
|
|
259
|
+
|
|
260
|
+
When even compact JSON is too large, the guard allocates space across sibling
|
|
261
|
+
fields and array entries, redistributing unused space. Recognized link and
|
|
262
|
+
identity fields (`url`, `uri`, `href`, `link`, `title`, `name`, `id`, and common
|
|
263
|
+
aliases) stay complete; if they cannot fit, the associated entry or branch is
|
|
264
|
+
omitted instead. Standalone URLs and entries in link/citation arrays also stay
|
|
265
|
+
whole. When a text cut would split a recognizable embedded URL token (such as
|
|
266
|
+
`https://`, `mailto:`, or `www.`), the incomplete token is omitted. Relative
|
|
267
|
+
links or bare domains in unrecognized fields cannot always be identified. Other long strings may be shortened, and fields or trailing array
|
|
268
|
+
entries omitted; results are not re-ranked or summarized.
|
|
269
|
+
|
|
270
|
+
The JSON portion remains valid, followed by a single `[truncated]` line. No
|
|
271
|
+
truncation metadata or per-string markers are added to the model-facing JSON.
|
|
272
|
+
Because of the trailing line, the complete text is not itself valid JSON.
|
|
273
|
+
`details.truncated` is also set to `true` when data is actually removed.
|
|
274
|
+
|
|
275
|
+
Change the limit **before starting pi**:
|
|
258
276
|
|
|
259
277
|
```bash
|
|
260
278
|
export PI_BETTER_WEB_SEARCH_MAX_CHARS=100000
|
|
261
279
|
```
|
|
262
280
|
|
|
263
|
-
|
|
264
|
-
|
|
281
|
+
In PowerShell, use `$env:PI_BETTER_WEB_SEARCH_MAX_CHARS = "100000"`. The limit
|
|
282
|
+
is in code points, **not tokens**, and does not adapt to the current model's
|
|
283
|
+
context window. The marker counts toward the limit. If truncation is needed and
|
|
284
|
+
the limit is 11–13 characters, only `[truncated]` is returned; below 11
|
|
285
|
+
characters the marker does not fit and only a minimal JSON value is returned.
|
|
286
|
+
`details.truncated` remains true in both cases. There is no continuation or
|
|
287
|
+
pagination cache: retry individual URLs or raise the limit if more content is
|
|
288
|
+
needed.
|
|
265
289
|
|
|
266
290
|
## Exposure rules
|
|
267
291
|
|
|
@@ -366,7 +390,7 @@ npm run build
|
|
|
366
390
|
```
|
|
367
391
|
|
|
368
392
|
`npm run build` runs the checks and tests, verifies the npm file list, and packs
|
|
369
|
-
`dist/pi-better-web-search-0.1.
|
|
393
|
+
`dist/pi-better-web-search-0.1.7.tgz`. Nothing is published automatically.
|
|
370
394
|
|
|
371
395
|
The test suite uses plain `.mjs` files that import the `.ts` sources directly.
|
|
372
396
|
It mocks `globalThis.fetch`, uses a temporary cache file, and exercises
|
package/docs/README.zh-CN.md
CHANGED
|
@@ -46,7 +46,7 @@ pi install ./path/to/pi-better-web-search
|
|
|
46
46
|
`urls`。`urls` 可以单独使用,也可以和 `query` 一起使用。
|
|
47
47
|
- **原始结果。** provider 数据放在 JSON 外层中返回,不去重、不规范化 URL、
|
|
48
48
|
不过滤垃圾 URL、不重排、不总结、不插入 citation 标记、不追加 Sources。
|
|
49
|
-
|
|
49
|
+
只有超出最终字符数限制时才进行结构化截断。
|
|
50
50
|
- **正确语义。** 工具描述是 `Search the web or analyze specified URLs.`;
|
|
51
51
|
schema 明确说明 `urls` 不依赖 `query`。
|
|
52
52
|
- **不碰系统提示词。** 不添加 prompt snippet、guideline、loadout hook、
|
|
@@ -215,17 +215,38 @@ Provider 数据放在 JSON 外层中返回,不做清理。示例:
|
|
|
215
215
|
- 结果不重排、不总结。
|
|
216
216
|
- 不插入 citation 标记,不追加 "Sources" 或 "Additional results"。
|
|
217
217
|
- 不解析 redirect,不派生 sources。
|
|
218
|
-
- provider
|
|
218
|
+
- 未超出大小限制时,provider 的未知字段也会保留。
|
|
219
219
|
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
220
|
+
最终大小保护默认限制成功结果不超过 **96,000 个 Unicode code point**。
|
|
221
|
+
未超限时保留原始字段及格式;如果只是缩进导致超限,会先压缩 JSON,
|
|
222
|
+
不丢失数据,也不标记为截断。
|
|
223
|
+
|
|
224
|
+
压缩后仍超限时,按字段和数组条目分配预算,未用完的额度会重新分配。
|
|
225
|
+
已识别的链接及标识字段(包括 `url`、`uri`、`href`、`link`、`title`、
|
|
226
|
+
`name`、`id` 和常见别名)保持完整;放不下时省略对应条目或分支,而不是
|
|
227
|
+
返回半截值。独立 URL 和链接、引用数组中的条目也保持完整;正文截断点
|
|
228
|
+
若落在可识别的 URL(如 `https://`、`mailto:` 或 `www.`)中间,会省略该
|
|
229
|
+
未完整显示的 URL。未知字段中的相对链接或裸域名无法保证都能识别。
|
|
230
|
+
其他长字符串可能被缩短,字段或数组末尾的条目可能被省略;
|
|
231
|
+
不对结果重排或总结。
|
|
232
|
+
|
|
233
|
+
JSON 部分仍然合法,末尾仅追加一行 `[truncated]`。不会向模型可见的 JSON
|
|
234
|
+
中添加截断元数据,也不会逐个标记缩短的字符串。由于末尾追加了这一行,
|
|
235
|
+
整个文本不再是合法 JSON。确实丢失数据时,`details.truncated` 也会设为
|
|
236
|
+
`true`。
|
|
237
|
+
|
|
238
|
+
可在**启动 pi 之前**通过环境变量调整:
|
|
223
239
|
|
|
224
240
|
```bash
|
|
225
241
|
export PI_BETTER_WEB_SEARCH_MAX_CHARS=100000
|
|
226
242
|
```
|
|
227
243
|
|
|
228
|
-
|
|
244
|
+
PowerShell 使用 `$env:PI_BETTER_WEB_SEARCH_MAX_CHARS = "100000"`。
|
|
245
|
+
上限是 code point,**不是 token**,不会根据当前模型的上下文窗口自动调整。
|
|
246
|
+
标记也计入上限。需要截断且上限为 11–13 字符时,只返回 `[truncated]`;
|
|
247
|
+
少于 11 字符时放不下标记,只返回一个最小 JSON 值。这两种情况仍会通过
|
|
248
|
+
`details.truncated` 报告截断。当前没有续读或分页缓存;需要更多内容时可
|
|
249
|
+
单独重新抓取 URL 或提高上限。
|
|
229
250
|
|
|
230
251
|
## 暴露规则
|
|
231
252
|
|
|
@@ -308,7 +329,7 @@ npm run build
|
|
|
308
329
|
```
|
|
309
330
|
|
|
310
331
|
`npm run build` 会运行检查、测试、包内容验证,并生成
|
|
311
|
-
`dist/pi-better-web-search-0.1.
|
|
332
|
+
`dist/pi-better-web-search-0.1.7.tgz`。不会自动发布。
|
|
312
333
|
|
|
313
334
|
测试是 `.mjs` 文件,直接导入 `.ts` 源码,mock `globalThis.fetch`,使用临时缓存
|
|
314
335
|
目录。详见 [测试](testing.md) 和 [发布](publishing.md)。
|
package/docs/architecture.md
CHANGED
|
@@ -38,7 +38,7 @@ tool execute
|
|
|
38
38
|
├─ run search candidates until one succeeds
|
|
39
39
|
├─ run fetch candidates per URL until each URL succeeds or candidates end
|
|
40
40
|
├─ write search/fetch preferences + fingerprint on success
|
|
41
|
-
└─ format raw JSON
|
|
41
|
+
└─ format raw JSON; compact or structurally truncate if the guard is exceeded
|
|
42
42
|
```
|
|
43
43
|
|
|
44
44
|
## Exposure
|
|
@@ -160,9 +160,22 @@ The result is a JSON wrapper:
|
|
|
160
160
|
|
|
161
161
|
Fields inside `search` and `fetched[].data` are provider data. The extension does
|
|
162
162
|
not deduplicate, normalize, filter, re-rank, summarize, insert citation markers,
|
|
163
|
-
or append source sections. The
|
|
164
|
-
|
|
165
|
-
|
|
163
|
+
or append source sections. The final guard keeps successful model-facing output
|
|
164
|
+
within 96,000 Unicode code points by default (`PI_BETTER_WEB_SEARCH_MAX_CHARS`).
|
|
165
|
+
It preserves ordinary formatted JSON, or uses compact JSON if that alone fits.
|
|
166
|
+
Otherwise it budgets sibling fields and array entries independently. Recognized
|
|
167
|
+
link and identity fields stay whole; entries or branches that cannot fit them
|
|
168
|
+
are omitted instead, without re-ranking. Standalone URLs and link/citation array
|
|
169
|
+
entries remain intact, and a recognizable URL crossing a text cut is omitted
|
|
170
|
+
rather than partially emitted. Unknown relative links or bare domains may not be
|
|
171
|
+
identifiable. Other strings can be shortened, and fields or trailing array entries
|
|
172
|
+
omitted. The JSON portion
|
|
173
|
+
stays valid, followed by one `[truncated]` line; the complete text is therefore
|
|
174
|
+
not valid JSON. No truncation metadata is injected into the model-facing JSON,
|
|
175
|
+
while `details.truncated` marks actual data loss. When truncation is needed with a limit of 11–13 characters,
|
|
176
|
+
only the marker is returned; below 11, a minimal JSON value is returned instead.
|
|
177
|
+
This is a character guard, not a token or model-context limit; it does not
|
|
178
|
+
provide pagination or retain the complete provider payload for later use.
|
|
166
179
|
|
|
167
180
|
## Provider adapter contract
|
|
168
181
|
|
package/docs/publishing.md
CHANGED
|
@@ -30,7 +30,7 @@ local `pi-web-search` reference checkout, `.env` files, and `DESIGN.md`.
|
|
|
30
30
|
npm run build
|
|
31
31
|
```
|
|
32
32
|
|
|
33
|
-
This produces `dist/pi-better-web-search-0.1.
|
|
33
|
+
This produces `dist/pi-better-web-search-0.1.7.tgz`. The tarball is not
|
|
34
34
|
committed.
|
|
35
35
|
|
|
36
36
|
## Publish
|
|
@@ -42,7 +42,7 @@ npm publish --access public
|
|
|
42
42
|
Alternatively, publish the verified tarball directly:
|
|
43
43
|
|
|
44
44
|
```bash
|
|
45
|
-
npm publish dist/pi-better-web-search-0.1.
|
|
45
|
+
npm publish dist/pi-better-web-search-0.1.7.tgz --access public
|
|
46
46
|
```
|
|
47
47
|
|
|
48
48
|
`prepublishOnly` reruns the checks, tests, and package verification. npm may
|
|
@@ -51,10 +51,10 @@ require browser or OTP authentication.
|
|
|
51
51
|
## Tag and release
|
|
52
52
|
|
|
53
53
|
```bash
|
|
54
|
-
git tag -a v0.1.
|
|
55
|
-
git push origin v0.1.
|
|
56
|
-
gh release create v0.1.
|
|
57
|
-
--title "v0.1.
|
|
54
|
+
git tag -a v0.1.7 -m "v0.1.7"
|
|
55
|
+
git push origin v0.1.7
|
|
56
|
+
gh release create v0.1.7 dist/pi-better-web-search-0.1.7.tgz \
|
|
57
|
+
--title "v0.1.7" \
|
|
58
58
|
--notes-file /path/to/release-notes.md
|
|
59
59
|
```
|
|
60
60
|
|
package/docs/testing.md
CHANGED
|
@@ -12,12 +12,12 @@ npm run verify:package
|
|
|
12
12
|
```
|
|
13
13
|
|
|
14
14
|
- `npm run check` runs `tsc --noEmit`.
|
|
15
|
-
- `npm test` runs the
|
|
15
|
+
- `npm test` runs the unit and integration tests with Node's test runner.
|
|
16
16
|
- `npm run verify:package` runs `npm pack --dry-run --json` and checks that the
|
|
17
17
|
tarball contains runtime sources, docs, assets, and metadata, while excluding
|
|
18
18
|
tests, scripts, CI files, the local reference checkout, and `DESIGN.md`.
|
|
19
19
|
- `npm run build` runs the checks, tests, package verification, and packs
|
|
20
|
-
`dist/pi-better-web-search-0.1.
|
|
20
|
+
`dist/pi-better-web-search-0.1.7.tgz`.
|
|
21
21
|
|
|
22
22
|
Tests import the TypeScript sources directly. The suite mocks `globalThis.fetch`
|
|
23
23
|
and uses temporary cache directories. Coverage includes:
|
|
@@ -30,7 +30,9 @@ and uses temporary cache directories. Coverage includes:
|
|
|
30
30
|
- search and fetch fallback, including per-URL fallback;
|
|
31
31
|
- provider request shapes and SSE parsing;
|
|
32
32
|
- HTTP/stream errors and abort propagation;
|
|
33
|
-
- character-guard truncation
|
|
33
|
+
- structured character-guard truncation, a single trailing marker, valid JSON
|
|
34
|
+
before the marker, Unicode and escaping, boundary lengths, fair allocation,
|
|
35
|
+
and intact links and identifiers across search results and fetched pages;
|
|
34
36
|
- one-tool registration and no system-prompt hooks.
|
|
35
37
|
|
|
36
38
|
## Live DeepSeek test
|
package/lib/format.ts
CHANGED
|
@@ -1,10 +1,11 @@
|
|
|
1
1
|
import type { AgentToolResult } from "@earendil-works/pi-coding-agent";
|
|
2
|
+
import { formatJsonWithinLimit } from "./truncate-json.ts";
|
|
2
3
|
|
|
3
|
-
const configuredOutputLimit = Number(process.env.PI_BETTER_WEB_SEARCH_MAX_CHARS);
|
|
4
|
+
const configuredOutputLimit = Math.floor(Number(process.env.PI_BETTER_WEB_SEARCH_MAX_CHARS));
|
|
4
5
|
export const MAX_OUTPUT_CHARS =
|
|
5
|
-
Number.isFinite(configuredOutputLimit) && configuredOutputLimit
|
|
6
|
-
?
|
|
7
|
-
:
|
|
6
|
+
Number.isFinite(configuredOutputLimit) && configuredOutputLimit >= 1
|
|
7
|
+
? configuredOutputLimit
|
|
8
|
+
: 96_000;
|
|
8
9
|
|
|
9
10
|
export interface WebSearchOutput {
|
|
10
11
|
query?: string;
|
|
@@ -41,8 +42,6 @@ export interface WebSearchDetails {
|
|
|
41
42
|
error?: string;
|
|
42
43
|
}
|
|
43
44
|
|
|
44
|
-
const TRUNCATION_MARKER = "\n\n[Truncated]";
|
|
45
|
-
|
|
46
45
|
/**
|
|
47
46
|
* Truncate by Unicode code point so a surrogate pair is never split. The
|
|
48
47
|
* returned `text` is at most `maxChars` code points.
|
|
@@ -61,11 +60,9 @@ export function formatSuccess(
|
|
|
61
60
|
output: WebSearchOutput,
|
|
62
61
|
details: WebSearchDetails,
|
|
63
62
|
): AgentToolResult<WebSearchDetails> {
|
|
64
|
-
const
|
|
65
|
-
const marker = TRUNCATION_MARKER.slice(0, Math.max(0, MAX_OUTPUT_CHARS));
|
|
66
|
-
const { text, truncated } = truncateText(json, Math.max(0, MAX_OUTPUT_CHARS - marker.length));
|
|
63
|
+
const { text, truncated } = formatJsonWithinLimit(output, MAX_OUTPUT_CHARS);
|
|
67
64
|
return {
|
|
68
|
-
content: [{ type: "text", text
|
|
65
|
+
content: [{ type: "text", text }],
|
|
69
66
|
details: truncated ? { ...details, truncated: true } : details,
|
|
70
67
|
};
|
|
71
68
|
}
|
|
@@ -0,0 +1,277 @@
|
|
|
1
|
+
type JsonValue = null | boolean | number | string | JsonValue[] | { [key: string]: JsonValue };
|
|
2
|
+
|
|
3
|
+
interface Slot {
|
|
4
|
+
index: number;
|
|
5
|
+
key?: string;
|
|
6
|
+
value: JsonValue;
|
|
7
|
+
minimum: number;
|
|
8
|
+
full: number;
|
|
9
|
+
allocation: number;
|
|
10
|
+
}
|
|
11
|
+
|
|
12
|
+
const TRUNCATION_MARKER = "[truncated]";
|
|
13
|
+
const IDENTITY_KEYS = ["url", "uri", "href", "link", "title", "name", "id", "error", "status", "query", "answer", "content", "text"];
|
|
14
|
+
const IMPORTANT_KEYS = [...IDENTITY_KEYS, "search", "fetched", "results", "data"];
|
|
15
|
+
const KEY_PRIORITY = new Map(IMPORTANT_KEYS.map((key, index) => [key, index]));
|
|
16
|
+
|
|
17
|
+
/** Link and identity fields are kept intact or their containing entry is omitted. */
|
|
18
|
+
function isProtectedKey(key: string): boolean {
|
|
19
|
+
return /^(?:url|uri|href|link|permalink|website|title|name|id)$/i.test(key)
|
|
20
|
+
|| /[_-](?:url|uri|href|link|permalink|website|title|name|id)$/i.test(key)
|
|
21
|
+
|| /(?:URL|URI|HREF|LINK|TITLE|NAME|Url|Uri|Href|Link|Title|Name|Id|ID)$/.test(key);
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
function isLinkCollectionKey(key: string): boolean {
|
|
25
|
+
return /^(?:urls?|uris?|hrefs?|links?|permalinks?|websites?|titles?|names?|ids?|sources?|citations?)$/i.test(key)
|
|
26
|
+
|| /[_-](?:urls|uris|hrefs|links|titles|names|ids)$/i.test(key)
|
|
27
|
+
|| /(?:URLs|URIs|Urls|Uris|Hrefs|Links|Titles|Names|Ids|IDs)$/.test(key);
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
function isStandaloneUrl(value: string): boolean {
|
|
31
|
+
return /^(?:[a-z][a-z0-9+.-]{0,31}:\/\/|mailto:|urn:|doi:|tel:|www\.)\S+$/i.test(value);
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
/** Count Unicode code points, rather than UTF-16 code units. */
|
|
35
|
+
function charCount(text: string): number {
|
|
36
|
+
let count = 0;
|
|
37
|
+
for (const _character of text) count += 1;
|
|
38
|
+
return count;
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
/** Encoded length of one code point inside a JSON string (without quotes). */
|
|
42
|
+
function encodedCharacterLength(character: string): number {
|
|
43
|
+
if (character.length === 2) return 1; // a surrogate pair is one code point
|
|
44
|
+
const code = character.charCodeAt(0);
|
|
45
|
+
if (character === '"' || character === "\\") return 2;
|
|
46
|
+
if (code === 8 || code === 9 || code === 10 || code === 12 || code === 13) return 2;
|
|
47
|
+
if (code < 32 || (code >= 0xd800 && code <= 0xdfff)) return 6;
|
|
48
|
+
return 1;
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
function fitString(value: string, budget: number): string {
|
|
52
|
+
// Reserve the quotes; the single truncation marker is added after the JSON.
|
|
53
|
+
let remaining = budget - 2;
|
|
54
|
+
const prefix: string[] = [];
|
|
55
|
+
for (const character of value) {
|
|
56
|
+
const size = encodedCharacterLength(character);
|
|
57
|
+
if (size > remaining) break;
|
|
58
|
+
prefix.push(character);
|
|
59
|
+
remaining -= size;
|
|
60
|
+
}
|
|
61
|
+
const visible = prefix.join("");
|
|
62
|
+
if (visible.length === value.length) return visible;
|
|
63
|
+
|
|
64
|
+
// A cut inside a URL embedded in prose must not leave a plausible but broken
|
|
65
|
+
// link. Drop the incomplete URL token instead of changing its address.
|
|
66
|
+
const urlStart = /\b(?:[a-z][a-z0-9+.-]{0,31}:\/\/|mailto:|urn:|doi:|tel:|www\.)/gi;
|
|
67
|
+
for (let match = urlStart.exec(value); match && match.index < visible.length; match = urlStart.exec(value)) {
|
|
68
|
+
let end = match.index + match[0].length;
|
|
69
|
+
while (end < value.length && !/\s/.test(value[end])) end += 1;
|
|
70
|
+
if (end > visible.length) return visible.slice(0, match.index);
|
|
71
|
+
urlStart.lastIndex = end;
|
|
72
|
+
}
|
|
73
|
+
return visible;
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
/** Distribute the available space fairly, giving unused space to larger values. */
|
|
77
|
+
function allocate(slots: Slot[], available: number): void {
|
|
78
|
+
let remaining = available;
|
|
79
|
+
for (const slot of slots) {
|
|
80
|
+
slot.allocation = slot.minimum;
|
|
81
|
+
remaining -= slot.minimum;
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
const pending = slots
|
|
85
|
+
.filter((slot) => slot.full > slot.minimum)
|
|
86
|
+
.sort((a, b) => (a.full - a.minimum) - (b.full - b.minimum) || a.index - b.index);
|
|
87
|
+
|
|
88
|
+
for (let i = 0; i < pending.length && remaining > 0; i += 1) {
|
|
89
|
+
const slot = pending[i];
|
|
90
|
+
const count = pending.length - i;
|
|
91
|
+
const share = Math.floor(remaining / count);
|
|
92
|
+
const demand = slot.full - slot.minimum;
|
|
93
|
+
if (demand <= share) {
|
|
94
|
+
slot.allocation += demand;
|
|
95
|
+
remaining -= demand;
|
|
96
|
+
continue;
|
|
97
|
+
}
|
|
98
|
+
|
|
99
|
+
const remainder = remaining % count;
|
|
100
|
+
for (let j = i; j < pending.length; j += 1) {
|
|
101
|
+
pending[j].allocation += share + (j - i < remainder ? 1 : 0);
|
|
102
|
+
}
|
|
103
|
+
break;
|
|
104
|
+
}
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
function createFitter() {
|
|
108
|
+
const sizes = new WeakMap<object, number>();
|
|
109
|
+
const requiredSizes = new WeakMap<object, number>();
|
|
110
|
+
|
|
111
|
+
const sizeOf = (value: JsonValue): number => {
|
|
112
|
+
if (value !== null && typeof value === "object") {
|
|
113
|
+
const cached = sizes.get(value);
|
|
114
|
+
if (cached !== undefined) return cached;
|
|
115
|
+
const size = charCount(JSON.stringify(value));
|
|
116
|
+
sizes.set(value, size);
|
|
117
|
+
return size;
|
|
118
|
+
}
|
|
119
|
+
return charCount(JSON.stringify(value));
|
|
120
|
+
};
|
|
121
|
+
|
|
122
|
+
function requiredPropertySize(key: string, value: JsonValue): number | undefined {
|
|
123
|
+
if (isProtectedKey(key) && !Array.isArray(value)) return sizeOf(value);
|
|
124
|
+
if (value !== null && typeof value === "object") {
|
|
125
|
+
if (Array.isArray(value)) return undefined;
|
|
126
|
+
const nested = requiredSize(value);
|
|
127
|
+
return nested > 0 ? nested : undefined;
|
|
128
|
+
}
|
|
129
|
+
if (typeof value === "string" && isStandaloneUrl(value)) return sizeOf(value);
|
|
130
|
+
return undefined;
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
// Reserve every direct or nested identity field before allocating space to
|
|
134
|
+
// optional content. Arrays can still omit a suffix of complete entries.
|
|
135
|
+
function requiredSize(value: { [key: string]: JsonValue }): number {
|
|
136
|
+
const cached = requiredSizes.get(value);
|
|
137
|
+
if (cached !== undefined) return cached;
|
|
138
|
+
let size = 2;
|
|
139
|
+
let count = 0;
|
|
140
|
+
for (const [key, child] of Object.entries(value)) {
|
|
141
|
+
const required = requiredPropertySize(key, child);
|
|
142
|
+
if (required === undefined) continue;
|
|
143
|
+
size += (count > 0 ? 1 : 0) + charCount(JSON.stringify(key)) + 1 + required;
|
|
144
|
+
count += 1;
|
|
145
|
+
}
|
|
146
|
+
const result = count > 0 ? size : 0;
|
|
147
|
+
requiredSizes.set(value, result);
|
|
148
|
+
return result;
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
const minimumSize = (value: JsonValue, key = "", linkCollection = false): number => {
|
|
152
|
+
if (Array.isArray(value)) return 2; // []
|
|
153
|
+
if (isProtectedKey(key)) return sizeOf(value);
|
|
154
|
+
if (typeof value === "string") {
|
|
155
|
+
return linkCollection || isStandaloneUrl(value) ? sizeOf(value) : 2;
|
|
156
|
+
}
|
|
157
|
+
if (value !== null && typeof value === "object") {
|
|
158
|
+
const required = requiredSize(value);
|
|
159
|
+
if (required > 0) return required;
|
|
160
|
+
// Avoid arrays of empty objects when a useful text field is available.
|
|
161
|
+
for (const identity of IDENTITY_KEYS) {
|
|
162
|
+
if (!Object.hasOwn(value, identity)) continue;
|
|
163
|
+
const child = value[identity];
|
|
164
|
+
if (child !== null && typeof child === "object") continue;
|
|
165
|
+
const childMinimum = typeof child === "string" ? Math.min(sizeOf(child), 128) : sizeOf(child);
|
|
166
|
+
return 2 + charCount(JSON.stringify(identity)) + 1 + childMinimum;
|
|
167
|
+
}
|
|
168
|
+
return 2;
|
|
169
|
+
}
|
|
170
|
+
return sizeOf(value);
|
|
171
|
+
};
|
|
172
|
+
|
|
173
|
+
const fit = (value: JsonValue, budget: number, linkCollection = false, root = false): JsonValue => {
|
|
174
|
+
if (sizeOf(value) <= budget) return value;
|
|
175
|
+
|
|
176
|
+
if (typeof value === "string") return fitString(value, budget);
|
|
177
|
+
|
|
178
|
+
if (Array.isArray(value)) {
|
|
179
|
+
const slots: Slot[] = [];
|
|
180
|
+
let fixed = 2; // brackets and commas
|
|
181
|
+
let minimums = 0;
|
|
182
|
+
for (let index = 0; index < value.length; index += 1) {
|
|
183
|
+
const child = value[index];
|
|
184
|
+
const minimum = minimumSize(child, "", linkCollection);
|
|
185
|
+
const punctuation = slots.length > 0 ? 1 : 0;
|
|
186
|
+
if (fixed + punctuation + minimums + minimum > budget) {
|
|
187
|
+
// Preserve array order when some entries must be omitted.
|
|
188
|
+
break;
|
|
189
|
+
}
|
|
190
|
+
fixed += punctuation;
|
|
191
|
+
minimums += minimum;
|
|
192
|
+
slots.push({ index, value: child, minimum, full: sizeOf(child), allocation: minimum });
|
|
193
|
+
}
|
|
194
|
+
allocate(slots, budget - fixed);
|
|
195
|
+
return slots.map((slot) => fit(slot.value, slot.allocation, linkCollection));
|
|
196
|
+
}
|
|
197
|
+
|
|
198
|
+
if (value !== null && typeof value === "object") {
|
|
199
|
+
// An object cannot be shown with only a fragment of a required identity.
|
|
200
|
+
if (!root && requiredSize(value) > budget) return Object.create(null) as JsonValue;
|
|
201
|
+
const entries = Object.entries(value);
|
|
202
|
+
// Reserve required identifiers first, but restore the original field
|
|
203
|
+
// order in the result; no provider data or array entry is re-ranked.
|
|
204
|
+
const candidates = entries.map(([key, child], index) => ({
|
|
205
|
+
key, child, index, required: requiredPropertySize(key, child) !== undefined,
|
|
206
|
+
}));
|
|
207
|
+
candidates.sort((a, b) =>
|
|
208
|
+
Number(b.required) - Number(a.required)
|
|
209
|
+
|| (KEY_PRIORITY.get(a.key) ?? IMPORTANT_KEYS.length) - (KEY_PRIORITY.get(b.key) ?? IMPORTANT_KEYS.length)
|
|
210
|
+
|| a.index - b.index,
|
|
211
|
+
);
|
|
212
|
+
const slots: Slot[] = [];
|
|
213
|
+
let fixed = 2; // braces, keys, colons and commas
|
|
214
|
+
let minimums = 0;
|
|
215
|
+
for (const { key, child, index } of candidates) {
|
|
216
|
+
const minimum = minimumSize(child, key);
|
|
217
|
+
const punctuation = (slots.length > 0 ? 1 : 0) + charCount(JSON.stringify(key)) + 1;
|
|
218
|
+
if (fixed + punctuation + minimums + minimum > budget) continue;
|
|
219
|
+
fixed += punctuation;
|
|
220
|
+
minimums += minimum;
|
|
221
|
+
slots.push({ index, key, value: child, minimum, full: sizeOf(child), allocation: minimum });
|
|
222
|
+
}
|
|
223
|
+
slots.sort((a, b) => a.index - b.index);
|
|
224
|
+
allocate(slots, budget - fixed);
|
|
225
|
+
const result: { [key: string]: JsonValue } = Object.create(null);
|
|
226
|
+
for (const slot of slots) {
|
|
227
|
+
const key = slot.key!;
|
|
228
|
+
result[key] = fit(slot.value, slot.allocation, isLinkCollectionKey(key));
|
|
229
|
+
}
|
|
230
|
+
return result;
|
|
231
|
+
}
|
|
232
|
+
|
|
233
|
+
// Parents reserve the complete serialized size for non-string scalars.
|
|
234
|
+
throw new Error("Insufficient JSON budget for scalar value");
|
|
235
|
+
};
|
|
236
|
+
|
|
237
|
+
return fit;
|
|
238
|
+
}
|
|
239
|
+
|
|
240
|
+
/**
|
|
241
|
+
* Preserve ordinary output exactly. If compact JSON still exceeds the limit,
|
|
242
|
+
* fit its fields and array entries independently and append one marker line.
|
|
243
|
+
* The JSON portion stays valid and the entire text fits within `maxChars` code
|
|
244
|
+
* points. Tiny limits may only fit the marker or a minimal JSON value.
|
|
245
|
+
*/
|
|
246
|
+
export function formatJsonWithinLimit(
|
|
247
|
+
output: object,
|
|
248
|
+
maxChars: number,
|
|
249
|
+
): { text: string; truncated: boolean } {
|
|
250
|
+
if (!Number.isFinite(maxChars) || maxChars < 1) {
|
|
251
|
+
throw new RangeError("The JSON output limit must be at least one character");
|
|
252
|
+
}
|
|
253
|
+
const limit = Math.floor(maxChars);
|
|
254
|
+
const pretty = JSON.stringify(output, null, 2);
|
|
255
|
+
if (charCount(pretty) <= limit) return { text: pretty, truncated: false };
|
|
256
|
+
|
|
257
|
+
// Work from the already serialized value, so JSON's handling of undefined,
|
|
258
|
+
// dates and toJSON is identical even if an input object is not plain data.
|
|
259
|
+
const parsed = JSON.parse(pretty) as JsonValue;
|
|
260
|
+
const compact = JSON.stringify(parsed);
|
|
261
|
+
if (charCount(compact) <= limit) return { text: compact, truncated: false };
|
|
262
|
+
|
|
263
|
+
// Below 14 characters there is not enough room for both an empty JSON object
|
|
264
|
+
// and the marker on a new line. Prefer a complete marker whenever it fits.
|
|
265
|
+
if (limit < TRUNCATION_MARKER.length + 3) {
|
|
266
|
+
const text = limit >= TRUNCATION_MARKER.length ? TRUNCATION_MARKER : limit >= 2 ? "{}" : "0";
|
|
267
|
+
return { text, truncated: true };
|
|
268
|
+
}
|
|
269
|
+
|
|
270
|
+
const budget = limit - TRUNCATION_MARKER.length - 1; // newline + marker
|
|
271
|
+
const fit = createFitter();
|
|
272
|
+
// A toJSON method can return a scalar that cannot be shortened to fit.
|
|
273
|
+
const fitted = parsed === null || typeof parsed === "boolean" || typeof parsed === "number"
|
|
274
|
+
? {}
|
|
275
|
+
: fit(parsed, budget, false, true);
|
|
276
|
+
return { text: `${JSON.stringify(fitted)}\n${TRUNCATION_MARKER}`, truncated: true };
|
|
277
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-better-web-search",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.7",
|
|
4
4
|
"description": "A better provider-native web_search tool for pi: raw results, automatic provider fallback, URL analysis, and native DeepSeek Search/Fetch.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|