partial-json-repair 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +181 -0
- package/README.md +1263 -2
- package/SECURITY.md +65 -0
- package/dist/AGENTS.md +85 -0
- package/dist/async-configuration.d.ts +14 -0
- package/dist/async-configuration.d.ts.map +1 -0
- package/dist/async-configuration.js +35 -0
- package/dist/async-configuration.js.map +1 -0
- package/dist/async-operation.d.ts +41 -0
- package/dist/async-operation.d.ts.map +1 -0
- package/dist/async-operation.js +171 -0
- package/dist/async-operation.js.map +1 -0
- package/dist/async-parser.d.ts +13 -0
- package/dist/async-parser.d.ts.map +1 -0
- package/dist/async-parser.js +131 -0
- package/dist/async-parser.js.map +1 -0
- package/dist/async-worker.js +615 -0
- package/dist/async-worker.js.map +1 -0
- package/dist/async.d.ts +36 -0
- package/dist/async.d.ts.map +1 -0
- package/dist/async.js +15 -0
- package/dist/async.js.map +1 -0
- package/dist/bare-values.js +61 -0
- package/dist/bare-values.js.map +1 -0
- package/dist/builder.js +646 -0
- package/dist/builder.js.map +1 -0
- package/dist/builtin-route.js +251 -0
- package/dist/builtin-route.js.map +1 -0
- package/dist/byte-positions.js +77 -0
- package/dist/byte-positions.js.map +1 -0
- package/dist/candidate-reader.js +700 -0
- package/dist/candidate-reader.js.map +1 -0
- package/dist/chars.js +158 -0
- package/dist/chars.js.map +1 -0
- package/dist/configuration-error.js +125 -0
- package/dist/configuration-error.js.map +1 -0
- package/dist/cooperative-executor.js +112 -0
- package/dist/cooperative-executor.js.map +1 -0
- package/dist/cut.js +49 -0
- package/dist/cut.js.map +1 -0
- package/dist/decoded-fragments.d.ts +7 -0
- package/dist/decoded-fragments.d.ts.map +1 -0
- package/dist/decoded-fragments.js +97 -0
- package/dist/decoded-fragments.js.map +1 -0
- package/dist/decoded-picks.js +107 -0
- package/dist/decoded-picks.js.map +1 -0
- package/dist/edit-log.js +76 -0
- package/dist/edit-log.js.map +1 -0
- package/dist/exact-number.d.ts +54 -0
- package/dist/exact-number.d.ts.map +1 -0
- package/dist/exact-number.js +177 -0
- package/dist/exact-number.js.map +1 -0
- package/dist/index.d.ts +46 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +43 -0
- package/dist/index.js.map +1 -0
- package/dist/joined-stop-signal.js +63 -0
- package/dist/joined-stop-signal.js.map +1 -0
- package/dist/lexical-window.js +617 -0
- package/dist/lexical-window.js.map +1 -0
- package/dist/measured-scan-cursor.js +35 -0
- package/dist/measured-scan-cursor.js.map +1 -0
- package/dist/metrics.d.ts +52 -0
- package/dist/metrics.d.ts.map +1 -0
- package/dist/metrics.js +218 -0
- package/dist/metrics.js.map +1 -0
- package/dist/native-slice.js +80 -0
- package/dist/native-slice.js.map +1 -0
- package/dist/number-fidelity.js +88 -0
- package/dist/number-fidelity.js.map +1 -0
- package/dist/options.d.ts +323 -0
- package/dist/options.d.ts.map +1 -0
- package/dist/options.js +309 -0
- package/dist/options.js.map +1 -0
- package/dist/owned-text.js +30 -0
- package/dist/owned-text.js.map +1 -0
- package/dist/parser-configuration.d.ts +19 -0
- package/dist/parser-configuration.d.ts.map +1 -0
- package/dist/parser-configuration.js +61 -0
- package/dist/parser-configuration.js.map +1 -0
- package/dist/parser-outcome.d.ts +20 -0
- package/dist/parser-outcome.d.ts.map +1 -0
- package/dist/parser-outcome.js +104 -0
- package/dist/parser-outcome.js.map +1 -0
- package/dist/parser-stream.d.ts +30 -0
- package/dist/parser-stream.d.ts.map +1 -0
- package/dist/parser-stream.js +735 -0
- package/dist/parser-stream.js.map +1 -0
- package/dist/parser.d.ts +105 -0
- package/dist/parser.d.ts.map +1 -0
- package/dist/path-filter.js +48 -0
- package/dist/path-filter.js.map +1 -0
- package/dist/picks.js +465 -0
- package/dist/picks.js.map +1 -0
- package/dist/positions.d.ts +12 -0
- package/dist/positions.d.ts.map +1 -0
- package/dist/positions.js +82 -0
- package/dist/positions.js.map +1 -0
- package/dist/pre-scan.js +586 -0
- package/dist/pre-scan.js.map +1 -0
- package/dist/presets.d.ts +51 -0
- package/dist/presets.d.ts.map +1 -0
- package/dist/presets.js +53 -0
- package/dist/presets.js.map +1 -0
- package/dist/quote-ambiguity.js +181 -0
- package/dist/quote-ambiguity.js.map +1 -0
- package/dist/repair-async.d.ts +13 -0
- package/dist/repair-async.d.ts.map +1 -0
- package/dist/repair-async.js +11 -0
- package/dist/repair-async.js.map +1 -0
- package/dist/repair-operation.js +83 -0
- package/dist/repair-operation.js.map +1 -0
- package/dist/repair-splice.js +217 -0
- package/dist/repair-splice.js.map +1 -0
- package/dist/report.d.ts +10 -0
- package/dist/report.d.ts.map +1 -0
- package/dist/report.js +289 -0
- package/dist/report.js.map +1 -0
- package/dist/root-filter.js +19 -0
- package/dist/root-filter.js.map +1 -0
- package/dist/root-log.js +68 -0
- package/dist/root-log.js.map +1 -0
- package/dist/same-thread-strategy.d.ts +16 -0
- package/dist/same-thread-strategy.d.ts.map +1 -0
- package/dist/same-thread-strategy.js +18 -0
- package/dist/same-thread-strategy.js.map +1 -0
- package/dist/scalar-reader.js +703 -0
- package/dist/scalar-reader.js.map +1 -0
- package/dist/scan-cursor.js +118 -0
- package/dist/scan-cursor.js.map +1 -0
- package/dist/scan-diagnostics.js +371 -0
- package/dist/scan-diagnostics.js.map +1 -0
- package/dist/scan-readers.js +44 -0
- package/dist/scan-readers.js.map +1 -0
- package/dist/scan.js +881 -0
- package/dist/scan.js.map +1 -0
- package/dist/selection-copy.js +70 -0
- package/dist/selection-copy.js.map +1 -0
- package/dist/selection-feature.d.ts +33 -0
- package/dist/selection-feature.d.ts.map +1 -0
- package/dist/selection-feature.js +84 -0
- package/dist/selection-feature.js.map +1 -0
- package/dist/selector.d.ts +27 -0
- package/dist/selector.d.ts.map +1 -0
- package/dist/selector.js +187 -0
- package/dist/selector.js.map +1 -0
- package/dist/snapshot-owned.d.ts +6 -0
- package/dist/snapshot-owned.d.ts.map +1 -0
- package/dist/snapshot-owned.js +27 -0
- package/dist/snapshot-owned.js.map +1 -0
- package/dist/snapshot-shared.d.ts +6 -0
- package/dist/snapshot-shared.d.ts.map +1 -0
- package/dist/snapshot-shared.js +42 -0
- package/dist/snapshot-shared.js.map +1 -0
- package/dist/snapshot.js +151 -0
- package/dist/snapshot.js.map +1 -0
- package/dist/source-fragments.d.ts +7 -0
- package/dist/source-fragments.d.ts.map +1 -0
- package/dist/source-fragments.js +104 -0
- package/dist/source-fragments.js.map +1 -0
- package/dist/source-picks.js +220 -0
- package/dist/source-picks.js.map +1 -0
- package/dist/span-async.d.ts +13 -0
- package/dist/span-async.d.ts.map +1 -0
- package/dist/span-async.js +11 -0
- package/dist/span-async.js.map +1 -0
- package/dist/span-operation.js +12 -0
- package/dist/span-operation.js.map +1 -0
- package/dist/stop-signal.d.ts +17 -0
- package/dist/stop-signal.d.ts.map +1 -0
- package/dist/stop-signal.js +22 -0
- package/dist/stop-signal.js.map +1 -0
- package/dist/stopper.js +96 -0
- package/dist/stopper.js.map +1 -0
- package/dist/stream-scalars.js +61 -0
- package/dist/stream-scalars.js.map +1 -0
- package/dist/stream.d.ts +54 -0
- package/dist/stream.d.ts.map +1 -0
- package/dist/stream.js +23 -0
- package/dist/stream.js.map +1 -0
- package/dist/string-escapes.js +18 -0
- package/dist/string-escapes.js.map +1 -0
- package/dist/stringify.d.ts +12 -0
- package/dist/stringify.d.ts.map +1 -0
- package/dist/stringify.js +210 -0
- package/dist/stringify.js.map +1 -0
- package/dist/transport.d.ts +50 -0
- package/dist/transport.d.ts.map +1 -0
- package/dist/transport.js +633 -0
- package/dist/transport.js.map +1 -0
- package/dist/tree.js +104 -0
- package/dist/tree.js.map +1 -0
- package/dist/trivia-run.js +111 -0
- package/dist/trivia-run.js.map +1 -0
- package/dist/try-parsers.d.ts +30 -0
- package/dist/try-parsers.d.ts.map +1 -0
- package/dist/try-parsers.js +83 -0
- package/dist/try-parsers.js.map +1 -0
- package/dist/types.d.ts +736 -0
- package/dist/types.d.ts.map +1 -0
- package/dist/types.js +63 -0
- package/dist/types.js.map +1 -0
- package/dist/utf8.js +290 -0
- package/dist/utf8.js.map +1 -0
- package/dist/whole-input.js +58 -0
- package/dist/whole-input.js.map +1 -0
- package/dist/whole-iteration.js +74 -0
- package/dist/whole-iteration.js.map +1 -0
- package/dist/whole-operation.d.ts +5 -0
- package/dist/whole-operation.d.ts.map +1 -0
- package/dist/whole-operation.js +270 -0
- package/dist/whole-operation.js.map +1 -0
- package/dist/whole-reader.js +399 -0
- package/dist/whole-reader.js.map +1 -0
- package/dist/whole-source.js +168 -0
- package/dist/whole-source.js.map +1 -0
- package/dist/worker-executor.d.ts +36 -0
- package/dist/worker-executor.d.ts.map +1 -0
- package/dist/worker-executor.js +749 -0
- package/dist/worker-executor.js.map +1 -0
- package/dist/worker-host.d.ts +5 -0
- package/dist/worker-host.d.ts.map +1 -0
- package/dist/worker-message.d.ts +155 -0
- package/dist/worker-message.d.ts.map +1 -0
- package/dist/worker-message.js +44 -0
- package/dist/worker-message.js.map +1 -0
- package/dist/worker-output.js +368 -0
- package/dist/worker-output.js.map +1 -0
- package/dist/worker-receive.js +41 -0
- package/dist/worker-receive.js.map +1 -0
- package/dist/worker-route.js +109 -0
- package/dist/worker-route.js.map +1 -0
- package/dist/worker-strategy.d.ts +10 -0
- package/dist/worker-strategy.d.ts.map +1 -0
- package/dist/worker-strategy.js +26 -0
- package/dist/worker-strategy.js.map +1 -0
- package/dist/worker-unavailable.js +12 -0
- package/dist/worker-unavailable.js.map +1 -0
- package/docs/PRIOR-ART.md +619 -0
- package/package.json +48 -4
- package/src/async-configuration.ts +64 -0
- package/src/async-operation.ts +371 -0
- package/src/async-parser.ts +175 -0
- package/src/async-worker.ts +446 -0
- package/src/async.ts +75 -0
- package/src/bare-values.ts +103 -0
- package/src/builder.ts +894 -0
- package/src/builtin-route.ts +321 -0
- package/src/byte-positions.ts +81 -0
- package/src/candidate-reader.ts +826 -0
- package/src/chars.ts +245 -0
- package/src/configuration-error.ts +163 -0
- package/src/cooperative-executor.ts +145 -0
- package/src/cut.ts +71 -0
- package/src/decoded-fragments.ts +116 -0
- package/src/decoded-picks.ts +191 -0
- package/src/edit-log.ts +87 -0
- package/src/exact-number.ts +197 -0
- package/src/index.ts +221 -0
- package/src/joined-stop-signal.ts +85 -0
- package/src/lexical-window.ts +991 -0
- package/src/measured-scan-cursor.ts +40 -0
- package/src/metrics.ts +344 -0
- package/src/native-slice.ts +108 -0
- package/src/number-fidelity.ts +131 -0
- package/src/options.ts +803 -0
- package/src/owned-text.ts +54 -0
- package/src/parser-configuration.ts +170 -0
- package/src/parser-outcome.ts +156 -0
- package/src/parser-stream.ts +972 -0
- package/src/parser.ts +173 -0
- package/src/path-filter.ts +80 -0
- package/src/picks.ts +610 -0
- package/src/positions.ts +119 -0
- package/src/pre-scan.ts +780 -0
- package/src/presets.ts +94 -0
- package/src/quote-ambiguity.ts +302 -0
- package/src/repair-async.ts +34 -0
- package/src/repair-operation.ts +93 -0
- package/src/repair-splice.ts +277 -0
- package/src/report.ts +386 -0
- package/src/root-filter.ts +32 -0
- package/src/root-log.ts +104 -0
- package/src/same-thread-strategy.ts +51 -0
- package/src/scalar-reader.ts +1093 -0
- package/src/scan-collect.ts +18 -0
- package/src/scan-contract.ts +192 -0
- package/src/scan-cursor.ts +142 -0
- package/src/scan-diagnostics.ts +436 -0
- package/src/scan-readers.ts +96 -0
- package/src/scan.ts +1238 -0
- package/src/selection-copy.ts +83 -0
- package/src/selection-feature.ts +199 -0
- package/src/selector.ts +219 -0
- package/src/snapshot-owned.ts +35 -0
- package/src/snapshot-shared.ts +60 -0
- package/src/snapshot.ts +215 -0
- package/src/source-fragments.ts +169 -0
- package/src/source-picks.ts +264 -0
- package/src/span-async.ts +36 -0
- package/src/span-operation.ts +14 -0
- package/src/stop-signal.ts +29 -0
- package/src/stopper.ts +104 -0
- package/src/stream-scalars.ts +96 -0
- package/src/stream.ts +115 -0
- package/src/string-escapes.ts +27 -0
- package/src/stringify.ts +225 -0
- package/src/transport.ts +816 -0
- package/src/tree.ts +109 -0
- package/src/trivia-run.ts +126 -0
- package/src/try-parsers.ts +104 -0
- package/src/types.ts +839 -0
- package/src/utf8.ts +354 -0
- package/src/whole-input.ts +87 -0
- package/src/whole-iteration.ts +85 -0
- package/src/whole-operation.ts +389 -0
- package/src/whole-reader.ts +573 -0
- package/src/whole-source.ts +203 -0
- package/src/worker-executor.ts +994 -0
- package/src/worker-host.ts +36 -0
- package/src/worker-message.ts +180 -0
- package/src/worker-output.ts +451 -0
- package/src/worker-receive.ts +55 -0
- package/src/worker-route.ts +177 -0
- package/src/worker-strategy.ts +33 -0
- package/src/worker-unavailable.ts +24 -0
package/README.md
CHANGED
|
@@ -1,3 +1,1264 @@
|
|
|
1
|
-
#
|
|
1
|
+
# partial-json-repair
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://www.npmjs.com/package/partial-json-repair) [](../../actions/workflows/check.yml) [](package.json) [](package.json) [](docs/PRIOR-ART.md) [](docs/PRIOR-ART.md) [](docs/PRIOR-ART.md) [](docs/PRIOR-ART.md) [](docs/PRIOR-ART.md) [](LICENSE)
|
|
4
|
+
|
|
5
|
+
**The fastest and most efficient JSON repairer, and the only one with full streaming support.**
|
|
6
|
+
|
|
7
|
+
Get usable values out of broken, cut-off or streaming JSON from LLMs, and know exactly what was
|
|
8
|
+
fixed and where the reply was cut.
|
|
9
|
+
|
|
10
|
+
- **Fastest, lightest repairer.** On clean input it runs at two thirds of `JSON.parse`, 1.6 to 6
|
|
11
|
+
times faster than the other repair libraries. On JSON with comments and trailing commas it beats
|
|
12
|
+
every parser measured, the JSONC specialists included. It uses no more memory than `JSON.parse`;
|
|
13
|
+
on Bun, `jsonrepair` and `repair-json-stream` use half as much again.
|
|
14
|
+
- **A cut reply never passes for a finished one.** Every value says whether it is `complete` and,
|
|
15
|
+
if not, where it was `cut`. The six repair libraries compared in
|
|
16
|
+
[docs/PRIOR-ART.md](docs/PRIOR-ART.md) return a half reply exactly as they return a whole one.
|
|
17
|
+
- **Reads what models write.** JSON inside prose or a code fence, cut off, with comments, single
|
|
18
|
+
quotes, unquoted keys, trailing commas or `NaN`, while a sentence like `see [1]` is never taken
|
|
19
|
+
for data. Of the 15 parsers in the capability table, it is the only one that handles every row.
|
|
20
|
+
When the model sends a corrected answer below the first, the result says `more: true`.
|
|
21
|
+
- **Every repair is a switch.** Turn any repair on or off per call with `recover` (dangerous keys
|
|
22
|
+
such as `__proto__` have `onDangerousKey`). The defaults are tuned for broken model output.
|
|
23
|
+
Readings that can drop source text stay off until you ask: `guessed-string-end` can cut a `}` off
|
|
24
|
+
a code value, and `leading-zero-number` turns `0123` into 123, losing the zero. See
|
|
25
|
+
[Options and presets](#options-and-presets).
|
|
26
|
+
- **Reports what it changed, when you ask.** Every repair and loss with its path, plus a confidence
|
|
27
|
+
score for deciding when to retry the model. A stray trailing comma costs no confidence; a
|
|
28
|
+
missing ending does.
|
|
29
|
+
- **Full streaming support.** Show the value as it arrives, act on each value the moment it
|
|
30
|
+
closes, or take a huge string in pieces without holding it. No other repair library hands you
|
|
31
|
+
values from a stream; with them you re-parse the whole buffer on every chunk. At 16-character
|
|
32
|
+
chunks this is 4 to 10 times faster than that re-parsing, which is what the OpenAI and Anthropic
|
|
33
|
+
SDKs' partial parsers do.
|
|
34
|
+
- **The same value however it arrives.** Text or bytes, one chunk or any split, synchronous or
|
|
35
|
+
async: you get what parsing the whole text at once gives.
|
|
36
|
+
- **Never throws for bad input.** Every failure is a typed value carrying whatever was recovered.
|
|
37
|
+
|
|
38
|
+
Zero runtime dependencies. One ESM build runs in browsers, Node, Bun, Deno and edge runtimes: the
|
|
39
|
+
root entry imports nothing platform-specific, and `/async` loads `node:worker_threads` only when it
|
|
40
|
+
starts a worker. You pay only for what you import: a bundle that uses just `parse` carries no
|
|
41
|
+
report, selector, streaming or live-view code. Types are checked against TypeScript 6 and 7 in CI.
|
|
42
|
+
|
|
43
|
+
## Install
|
|
44
|
+
|
|
45
|
+
```sh
|
|
46
|
+
bun add partial-json-repair # or: npm i / pnpm add partial-json-repair, deno add npm:partial-json-repair
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
The same library is also published as
|
|
50
|
+
[`@anizoptera/recover-json`](https://www.npmjs.com/package/@anizoptera/recover-json), at the same
|
|
51
|
+
version and with both entries (`@anizoptera/recover-json` and `@anizoptera/recover-json/async`).
|
|
52
|
+
Install whichever name you prefer; the examples below use `partial-json-repair`.
|
|
53
|
+
|
|
54
|
+
Coding agents: read the [usage guide for agents](CONSUMER-AGENTS.md), installed with the package
|
|
55
|
+
as `dist/AGENTS.md`.
|
|
56
|
+
|
|
57
|
+
The repository also holds a Rust port; it is not in this package. The full API contract is in
|
|
58
|
+
`docs/API.md`.
|
|
59
|
+
|
|
60
|
+
## The problem
|
|
61
|
+
|
|
62
|
+
A model returns this, and the connection drops:
|
|
63
|
+
|
|
64
|
+
```
|
|
65
|
+
Here you go: {"name": "Ada", "tags": ["x", "y
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
`JSON.parse` throws. So do four of the six repair libraries compared in
|
|
69
|
+
[docs/PRIOR-ART.md](docs/PRIOR-ART.md), on the prose in front; a fifth returns `"Here"`. Strip the
|
|
70
|
+
prose and all six return `{ name: "Ada", tags: ["x", "y"] }` and nothing else, so a reply cut in
|
|
71
|
+
half looks exactly like a finished one. This package returns the value **and** says it was cut, and
|
|
72
|
+
where:
|
|
73
|
+
|
|
74
|
+
```ts
|
|
75
|
+
import { parse } from "partial-json-repair";
|
|
76
|
+
|
|
77
|
+
const r = parse('Here you go: {"name": "Ada", "tags": ["x", "y');
|
|
78
|
+
if (!r.ok) throw new Error(r.failure.kind); // ok: true here: a cut-off reply is still a value
|
|
79
|
+
|
|
80
|
+
r.value; // { name: "Ada", tags: ["x", "y"] }
|
|
81
|
+
r.complete; // false: the source ended before the value did
|
|
82
|
+
r.cut; // { path: ["tags", 1], kind: "string" }: the string at tags[1] was still arriving
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Recovered values and repaired text carry `complete`, and `cut` whenever `complete` is false. The value
|
|
86
|
+
itself is never changed to show the cut: nothing is appended unless you ask for it with `cutSuffix`.
|
|
87
|
+
|
|
88
|
+
`parse`, `repair` and `findJsonSpan` return one value. When the text holds another after it, as when
|
|
89
|
+
a model sends a corrected version below its first answer, the result says so with `more: true`;
|
|
90
|
+
`parseMany` reads them all:
|
|
91
|
+
|
|
92
|
+
```ts
|
|
93
|
+
import { parse, parseMany } from "partial-json-repair";
|
|
94
|
+
|
|
95
|
+
const reply = '{"total": 12}\n\nSorry, that was wrong:\n\n{"total": 21}';
|
|
96
|
+
const first = parse(reply); // { ok: true, value: { total: 12 }, complete: true, more: true }
|
|
97
|
+
if (first.ok && first.more) for (const r of parseMany(reply)) if (r.ok) console.log(r.value);
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Ask for a report and you also get every repair, with the path to where it happened:
|
|
101
|
+
|
|
102
|
+
```ts
|
|
103
|
+
import { diagnostics, parse } from "partial-json-repair";
|
|
104
|
+
|
|
105
|
+
const r = parse('{"name": "Ada", "tags": ["x", "y', { diagnostics });
|
|
106
|
+
r.diagnostics.confidence; // 0.729
|
|
107
|
+
r.diagnostics.repairs.map((x) => [x.kind, x.at.offset, x.path]);
|
|
108
|
+
// [ ["unclosed-string", 30, ["tags", 1]],
|
|
109
|
+
// ["unclosed-array", 32, ["tags", 2]],
|
|
110
|
+
// ["unclosed-object", 32, []] ]
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
The report lists every change and loss in `repairs`, each with its path. If a report limit leaves
|
|
114
|
+
entries out, `entriesComplete` is false. Without `diagnostics` nothing is collected, and the result
|
|
115
|
+
type has no `diagnostics` field.
|
|
116
|
+
|
|
117
|
+
## Jobs
|
|
118
|
+
|
|
119
|
+
### Parse one model reply
|
|
120
|
+
|
|
121
|
+
`parse(text)` returns the first value in the text: inside prose, inside a code fence, cut off, with
|
|
122
|
+
comments, single quotes, unquoted keys or trailing commas.
|
|
123
|
+
|
|
124
|
+
```ts
|
|
125
|
+
import { parse } from "partial-json-repair";
|
|
126
|
+
|
|
127
|
+
const r = parse(modelOutput);
|
|
128
|
+
if (!r.ok) {
|
|
129
|
+
if (r.failure.recovered !== undefined) {
|
|
130
|
+
useValue(r.failure.recovered.value); // inspect the prefix; failure still requires a decision
|
|
131
|
+
}
|
|
132
|
+
handleFailure(r.failure); // "no-json-found", "policy-refused", "limit-reached", ...
|
|
133
|
+
} else if (!r.complete) {
|
|
134
|
+
// The reply was cut. r.value holds the available recovery;
|
|
135
|
+
// r.cut names the item that was cut and what kind of thing it was.
|
|
136
|
+
useValue(r.value);
|
|
137
|
+
} else {
|
|
138
|
+
useValue(r.value);
|
|
139
|
+
}
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
`ok` says recovery succeeded; `complete` says the source finished that value. Neither checks your
|
|
143
|
+
application schema: validate or narrow `value` before passing it to code that expects your types.
|
|
144
|
+
|
|
145
|
+
What `cut.kind` tells you:
|
|
146
|
+
|
|
147
|
+
| `kind` | Meaning | Example source | `cut` |
|
|
148
|
+
| --- | --- | --- | --- |
|
|
149
|
+
| `string` | A string, quoted or bare, was still arriving; the value holds every character written so far | `{"msg": "hello wor` | `{ path: ["msg"], kind: "string" }` |
|
|
150
|
+
| `number` | A number was cut where it is already valid; it may be short | `[1, 12` | `{ path: [1], kind: "number" }` |
|
|
151
|
+
| `missing` | The item at `path` was started but is not in the value | `[1, tru` | `{ path: [1], kind: "missing", text: "tru" }` |
|
|
152
|
+
| `between` | The source ended between items of the container at `path`; nothing written was dropped. A complete literal (`[1, true`) counts as finished | `{"a":1,"ke` | `{ path: [], kind: "between", text: '"ke' }` |
|
|
153
|
+
|
|
154
|
+
`text` holds the unfinished token the value leaves out (a partial word or number, a half escape, a
|
|
155
|
+
partial key), and is absent when there is none. A key the source had not finished is never
|
|
156
|
+
invented: `{"a":1,"ke` gives `between` at the object, not a member called `ke`. A cut-off number,
|
|
157
|
+
`true` or escape is never placed in the value as something it is not: `[1, 1e` recovers `[1]` with
|
|
158
|
+
`kind: "missing"`, and `"x\u00` recovers `"x"` with the half escape in `cut.text`.
|
|
159
|
+
|
|
160
|
+
To show the cut in the text itself, opt into a suffix on cut strings. It is appended only to a
|
|
161
|
+
string whose `kind` is `string`, never to a number, and never to an item that is missing:
|
|
162
|
+
|
|
163
|
+
```ts
|
|
164
|
+
import { parse } from "partial-json-repair";
|
|
165
|
+
|
|
166
|
+
const r = parse('{"msg": "hello wor', { cutSuffix: true });
|
|
167
|
+
if (r.ok) r.value; // { msg: "hello wor…" }
|
|
168
|
+
|
|
169
|
+
const custom = parse('{"msg": "hello wor', { cutSuffix: " [cut]" });
|
|
170
|
+
if (custom.ok) custom.value; // { msg: "hello wor [cut]" }
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
`true` appends `…` (U+2026). The same suffix appears in `parse`, in a stream's final value, in a
|
|
174
|
+
snapshot and in `repair()` text, so every route shows the same thing.
|
|
175
|
+
|
|
176
|
+
### Repair JSON text
|
|
177
|
+
|
|
178
|
+
`repair(text)` applies the same repairs to the _text_, for JSON that has to be handed to
|
|
179
|
+
something else: a log, another service, a schema validator.
|
|
180
|
+
|
|
181
|
+
```ts
|
|
182
|
+
import { repair } from "partial-json-repair";
|
|
183
|
+
|
|
184
|
+
const fixed = repair("{'a': 1, // note\n b: 2,}");
|
|
185
|
+
if (fixed.ok) fixed.value; // '{"a": 1, \n "b": 2}'
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
It splices the repairs into your source rather than re-serialising the value, so **a value that
|
|
189
|
+
needed no repair comes back byte-identical**: key order, number formatting and whitespace intact.
|
|
190
|
+
`JSON.stringify(parse(t).value)` would quietly turn `1.50` into `1.5`. What comes back is the
|
|
191
|
+
**value**, not the document: repairing ` ```json\n{"a":1}\n``` ` produces `{"a":1}`, without the
|
|
192
|
+
fences. On success the output parses under strict `JSON.parse`. `repair` fails exactly when `parse`
|
|
193
|
+
fails. A recovered value JSON cannot spell (`NaN`, an infinity, `undefined`) is written as `null`,
|
|
194
|
+
keeping its key or array position, and reported as `js-literal`; `parse` returns the real value.
|
|
195
|
+
`repair` results carry `complete` and `cut` like `parse` results.
|
|
196
|
+
|
|
197
|
+
### Locate JSON in a reply
|
|
198
|
+
|
|
199
|
+
`findJsonSpan(text)` gives its extent in the source, and the `locations`
|
|
200
|
+
feature puts the same `span` on every successful `parse` and `repair` result, so
|
|
201
|
+
`text.slice(0, span.start.offset)` and `text.slice(span.end.offset)` are the prose around it:
|
|
202
|
+
|
|
203
|
+
```ts
|
|
204
|
+
import { findJsonSpan, locations, parse } from "partial-json-repair";
|
|
205
|
+
|
|
206
|
+
const span = findJsonSpan('here you go: {"a":1} hope that helps');
|
|
207
|
+
if (span.ok) span.value; // { start: { offset: 13, unit: "utf16" }, end: { offset: 20, unit: "utf16" } }
|
|
208
|
+
|
|
209
|
+
const located = parse('here you go: {"a":1} hope that helps', { locations });
|
|
210
|
+
if (located.ok) located.span.start.offset; // 13
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
Span results report source boundaries. Use `parse(text, { locations })` when you also need the
|
|
214
|
+
recovered value's `complete` and `cut`.
|
|
215
|
+
|
|
216
|
+
### Render a reply while it streams
|
|
217
|
+
|
|
218
|
+
Rendering a model's JSON as it arrives needs the stream and a **view** of the value still being
|
|
219
|
+
written. Declare the view when the stream is built, then pull a snapshot once per frame you draw:
|
|
220
|
+
|
|
221
|
+
```ts
|
|
222
|
+
import { parseStream, sharedView } from "partial-json-repair";
|
|
223
|
+
|
|
224
|
+
const stop = new AbortController();
|
|
225
|
+
const stream = parseStream(chunks, { view: sharedView, signal: stop.signal });
|
|
226
|
+
let frameFailure: { cause: unknown } | undefined;
|
|
227
|
+
|
|
228
|
+
let frame = requestAnimationFrame(function draw() {
|
|
229
|
+
try {
|
|
230
|
+
const seen = stream.snapshot(); // the root still arriving, or the one just finished
|
|
231
|
+
if (seen.ok && seen.value !== undefined) render(seen.value.value);
|
|
232
|
+
frame = requestAnimationFrame(draw);
|
|
233
|
+
} catch (cause) {
|
|
234
|
+
// A frame runs outside the consuming loop; stop its pending read too.
|
|
235
|
+
frameFailure = { cause };
|
|
236
|
+
stop.abort(cause);
|
|
237
|
+
}
|
|
238
|
+
});
|
|
239
|
+
|
|
240
|
+
try {
|
|
241
|
+
for await (const result of stream) {
|
|
242
|
+
if (frameFailure !== undefined) throw frameFailure.cause;
|
|
243
|
+
if (result.ok) render(result.value); // a finished root
|
|
244
|
+
else {
|
|
245
|
+
if (result.failure.recovered !== undefined) {
|
|
246
|
+
render(result.failure.recovered.value);
|
|
247
|
+
}
|
|
248
|
+
handleFailure(result.failure);
|
|
249
|
+
}
|
|
250
|
+
}
|
|
251
|
+
} finally {
|
|
252
|
+
cancelAnimationFrame(frame);
|
|
253
|
+
}
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
A snapshot is `{ value, valueIndex, complete, cut? }`. `complete` is the root's finality, not the
|
|
257
|
+
source's end: a finished root stays in the snapshot, with `complete: true`, until the next root
|
|
258
|
+
begins. `cut` names the point still being written, which is where a renderer puts its cursor. A
|
|
259
|
+
refused or stopped root is a failure with the view under `failure.recovered`.
|
|
260
|
+
|
|
261
|
+
**Pull per frame, never per chunk.** Each pull copies the containers still open, so pulling after
|
|
262
|
+
every chunk of a growing array costs quadratic time, while pulling per frame costs one copy per
|
|
263
|
+
frame however fast chunks arrive. Under `shared`, two pulls with no input
|
|
264
|
+
between them return the same object.
|
|
265
|
+
|
|
266
|
+
`shared` copies only the containers still open and shares every finished branch: **O(depth + width
|
|
267
|
+
of the open containers), not O(size)**. Finished branches are frozen because they are shared, and so
|
|
268
|
+
is every value such a stream delivers. Treat a `shared` snapshot as read-only: the copy of the open
|
|
269
|
+
containers is left unfrozen only because freezing it costs too much per frame, and a change you make
|
|
270
|
+
to it shows in every later pull until new input arrives, since those pulls return the same object.
|
|
271
|
+
Changed paths get fresh identity and untouched branches keep theirs, so a framework re-renders
|
|
272
|
+
exactly what moved. `owned` is a deep copy you may keep and modify, costing the whole tree each time.
|
|
273
|
+
The live tree is never handed out.
|
|
274
|
+
|
|
275
|
+
Both views include the **string still arriving**: after `{"key": "val` you get `{ key: "val" }`.
|
|
276
|
+
Numbers and bare words are not shown while they arrive, because `1` of `123` is a different number
|
|
277
|
+
and `tr` of `true` is not a value; a half-read key is left out for the same reason.
|
|
278
|
+
|
|
279
|
+
**Owning the chunks yourself**, for a WebSocket, an SDK callback or a test, is `createStreamParser`:
|
|
280
|
+
|
|
281
|
+
```ts
|
|
282
|
+
import { createStreamParser, ownedView } from "partial-json-repair";
|
|
283
|
+
|
|
284
|
+
const sp = createStreamParser({ view: ownedView });
|
|
285
|
+
|
|
286
|
+
for (const result of sp.feed('{"title": "Hel')) handleResult(result); // nothing finished yet
|
|
287
|
+
let seen = sp.snapshot();
|
|
288
|
+
if (seen.ok) seen.value?.value; // { title: "Hel" }; seen.value?.cut is { path: ["title"], kind: "string" }
|
|
289
|
+
for (const result of sp.feed('lo", "items": [1,2')) handleResult(result);
|
|
290
|
+
seen = sp.snapshot();
|
|
291
|
+
if (seen.ok) seen.value?.value; // { title: "Hello", items: [1] }: the 2 may still grow
|
|
292
|
+
|
|
293
|
+
for (const done of sp.end()) {
|
|
294
|
+
if (done.ok) done.value; // { title: "Hello", items: [1, 2] }
|
|
295
|
+
if (done.ok) done.cut; // { path: ["items", 1], kind: "number" }: the input ended inside the array
|
|
296
|
+
}
|
|
297
|
+
```
|
|
298
|
+
|
|
299
|
+
`feed` and `end` return lazy iterators: the chunk is taken at the call and scanned as you pull. Read
|
|
300
|
+
what they return: every completed value is delivered exactly once, by the first iteration after it
|
|
301
|
+
completes. A root completes when the character after it arrives, because a comma touching it
|
|
302
|
+
would make it the first item of a list (`{"a":1},{"b":2}` reads as one array): a root followed by
|
|
303
|
+
anything inside its chunk comes from that `feed()`, and one that ends the chunk comes from the next
|
|
304
|
+
`feed()` or from `end()`. End each document with a line break (as JSONL does) to have it delivered
|
|
305
|
+
at once by its own `feed()`. `end()` delivers only what the end of input finished, such as the
|
|
306
|
+
cut-off root above. If you skip reading one `feed()`, the next iteration you read delivers the
|
|
307
|
+
skipped values first. Held values count toward `maxRetained`; past it the stream ends with
|
|
308
|
+
`limit-reached` naming `unread`, so a caller that reads only the view gets an error, not a memory
|
|
309
|
+
leak.
|
|
310
|
+
|
|
311
|
+
### Stream an agent's tool call
|
|
312
|
+
|
|
313
|
+
A tool call arrives as JSON arguments in pieces. Preview closed arguments, or process a huge
|
|
314
|
+
argument (a file, a document) without holding it.
|
|
315
|
+
|
|
316
|
+
**Which argument is still arriving.** `cut.path` names the unfinished source occurrence; a closed
|
|
317
|
+
occurrence can still be replaced by a later duplicate key or ancestor. Use previews for display;
|
|
318
|
+
wait for the completed root before acting on final arguments:
|
|
319
|
+
|
|
320
|
+
```ts
|
|
321
|
+
import { createStreamParser, sharedView } from "partial-json-repair";
|
|
322
|
+
|
|
323
|
+
const args = createStreamParser({ view: sharedView });
|
|
324
|
+
for (const result of args.feed('{"city": "Paris", "units": "metr')) handleResult(result);
|
|
325
|
+
|
|
326
|
+
const seen = args.snapshot();
|
|
327
|
+
if (seen.ok && seen.value !== undefined) {
|
|
328
|
+
const { value, cut } = seen.value;
|
|
329
|
+
const cityClosed = cut === undefined || cut.path[0] !== "city"; // true: "Paris" is closed
|
|
330
|
+
const unitsClosed = cut === undefined || cut.path[0] !== "units"; // false: still arriving
|
|
331
|
+
if (cityClosed) render(value); // preview; a later "city" may replace "Paris"
|
|
332
|
+
void unitsClosed;
|
|
333
|
+
}
|
|
334
|
+
```
|
|
335
|
+
|
|
336
|
+
The same test at any depth identifies the occurrence currently being extended, not final membership.
|
|
337
|
+
Paths hold array indexes as numbers, so `["items", 0]` matches and `["items", "0"]`
|
|
338
|
+
never does:
|
|
339
|
+
|
|
340
|
+
```ts
|
|
341
|
+
import type { Cut, SourcePath } from "partial-json-repair";
|
|
342
|
+
|
|
343
|
+
const isOpen = (path: SourcePath, cut: Cut | undefined): boolean =>
|
|
344
|
+
cut !== undefined && path.length <= cut.path.length && path.every((k, i) => k === cut.path[i]);
|
|
345
|
+
```
|
|
346
|
+
|
|
347
|
+
To receive each closed source occurrence rather than test paths, select the items with
|
|
348
|
+
`select([["items", ANY]], { occurrences: true })`: each one is delivered once, when it closes.
|
|
349
|
+
Use default selection when later duplicate keys must settle before acting on final values.
|
|
350
|
+
|
|
351
|
+
**A reply the model's token limit cut off.** Do not call `end()`. Request the continuation and feed
|
|
352
|
+
it to the same parser: the stream resumes inside the open string, key or container as if the reply
|
|
353
|
+
had never paused.
|
|
354
|
+
|
|
355
|
+
**A huge text argument, in pieces.** Select the field and ask for `fragments`: quoted strings arrive as
|
|
356
|
+
`start`, `data` and `end` items without retaining the whole string. Unquoted strings also stream once
|
|
357
|
+
their text cannot become a literal; trailing whitespace stays pending. Numeric and literal candidates
|
|
358
|
+
buffer until their recovered type and spelling settle.
|
|
359
|
+
|
|
360
|
+
Fragments describe source occurrences, including ones a later duplicate key or ancestor replaces.
|
|
361
|
+
Use them for previews or processing that accepts every occurrence. For final tool arguments, use
|
|
362
|
+
default selection and validate the completed root before acting; a fragment's `end` does not
|
|
363
|
+
establish final membership.
|
|
364
|
+
|
|
365
|
+
```ts
|
|
366
|
+
import { createStreamParser, decodedFragments, select } from "partial-json-repair";
|
|
367
|
+
|
|
368
|
+
const call = createStreamParser({ select: select([["content"]], { fragments: decodedFragments }) });
|
|
369
|
+
|
|
370
|
+
const write = (items: ReturnType<typeof call.end>): void => {
|
|
371
|
+
for (const item of items) {
|
|
372
|
+
if (item.ok && item.value.kind === "data") process.stdout.write(item.value.text);
|
|
373
|
+
else if (!item.ok) handleFailure(item.failure);
|
|
374
|
+
}
|
|
375
|
+
};
|
|
376
|
+
write(call.feed('{"path": "a.txt", "content": "hello wo'));
|
|
377
|
+
write(call.feed('rld"}'));
|
|
378
|
+
write(call.end()); // a string the input cut off ends here, so read `data` from `end()` too
|
|
379
|
+
// writes "hello world"; `start` named path ["content"] and scalar "string", `end` its source span
|
|
380
|
+
```
|
|
381
|
+
|
|
382
|
+
`start` opens an occurrence and names its path and scalar kind, `data` carries a piece of the decoded
|
|
383
|
+
content, and `end` closes it with the source span and `complete`, which is `false` when the input cut
|
|
384
|
+
the value off. Where the pieces split is internal batching, not part of the contract. String pieces
|
|
385
|
+
join to the decoded content, number and literal pieces to their spelling; a surrogate pair is never
|
|
386
|
+
split between pieces. With `cutSuffix`, the last `data` piece of a string the input cut off carries
|
|
387
|
+
the suffix. Fragments cannot be combined with a view, since a view would be the whole value you
|
|
388
|
+
asked not to keep. Selecting a container in this mode fails that root with `non-scalar-selection`:
|
|
389
|
+
select its scalar children instead.
|
|
390
|
+
|
|
391
|
+
### Read JSONL and multi-document output
|
|
392
|
+
|
|
393
|
+
`parseMany(text)` yields every JSON value in the text, lazily: JSONL, several documents in one
|
|
394
|
+
reply, or values scattered through prose. It suspends after each value, so consuming only the first
|
|
395
|
+
parses only the first. `parse` returns the first value and sets `more: true` when another follows;
|
|
396
|
+
with `diagnostics`, its report's `laterValues` also holds the span of every later value.
|
|
397
|
+
|
|
398
|
+
```ts
|
|
399
|
+
import { parseMany } from "partial-json-repair";
|
|
400
|
+
|
|
401
|
+
for (const r of parseMany('{"a":1}\n{"b":2}\n{"c":3}')) {
|
|
402
|
+
if (r.ok) useValue(r.value); // {a:1}, then {b:2}, then {c:3}
|
|
403
|
+
else handleFailure(r.failure);
|
|
404
|
+
}
|
|
405
|
+
```
|
|
406
|
+
|
|
407
|
+
A value whose source shows no colon, no quoted string inside brackets and no second item (`1`, `"a"`,
|
|
408
|
+
`[1]`) counts as JSON only when it stands alone, fills a code fence, or sits in a run of values
|
|
409
|
+
separated by line breaks. Among other text it is declined, because prose cites `[1] [2]` and quotes
|
|
410
|
+
`"a"`: `parseMany("[1]\n[2]")` reads both arrays, while `parseMany("see [1] [2]")` answers
|
|
411
|
+
`no-json-found`. For JSONL whose lines may be cut off, parse each line:
|
|
412
|
+
`text.split("\n").filter(Boolean).map((line) => parse(line))`. The newline is JSONL's own record
|
|
413
|
+
boundary, so a line cut inside a string stays one damaged record; `parseMany` cannot know that
|
|
414
|
+
boundary and reads the next line as the string's continuation. A line that holds no JSON at all
|
|
415
|
+
(`not json`) is prose between values, so both verbs pass over it without an item, and a damaged line
|
|
416
|
+
is repaired like any other value. Over the network, `parseStream` recognises the same roots, but
|
|
417
|
+
chunk boundaries are not line boundaries: for cut-off JSONL, assemble complete lines and `parse`
|
|
418
|
+
each. For marked roots such as objects, `parseStream` yields each value as soon as it closes, and a
|
|
419
|
+
root the policy refuses (a single quote under `recover: { "single-quote": false }`, say) fails alone
|
|
420
|
+
while the next one parses. Under `STRICT` the first refusal ends the scan instead:
|
|
421
|
+
|
|
422
|
+
```ts
|
|
423
|
+
import { parseStream } from "partial-json-repair";
|
|
424
|
+
|
|
425
|
+
for await (const r of parseStream(response.body!, { objectOnly: true })) {
|
|
426
|
+
if (r.ok) handle(r.value);
|
|
427
|
+
else handleFailure(r.failure); // a refused line; limits and source failures end the stream
|
|
428
|
+
}
|
|
429
|
+
```
|
|
430
|
+
|
|
431
|
+
`objectOnly` skips roots that are not objects; `requiredKeys: ["id"]` also requires own keys.
|
|
432
|
+
|
|
433
|
+
### Recipes: AI SDK, OpenAI, Anthropic
|
|
434
|
+
|
|
435
|
+
Each recipe hands the SDK's text deltas to `parseStream` or `feed`. Its function types the SDK
|
|
436
|
+
object only by the fields it reads, so it accepts the SDK's own types; the SDK call that produces
|
|
437
|
+
that object follows it.
|
|
438
|
+
|
|
439
|
+
**AI SDK (`streamText`).** `textStream` is an async iterable of text deltas:
|
|
440
|
+
|
|
441
|
+
```ts
|
|
442
|
+
import { parseStream, sharedView } from "partial-json-repair";
|
|
443
|
+
|
|
444
|
+
export async function showReply(result: { textStream: AsyncIterable<string> }): Promise<unknown> {
|
|
445
|
+
const stop = new AbortController();
|
|
446
|
+
const stream = parseStream(result.textStream, { view: sharedView, signal: stop.signal });
|
|
447
|
+
let frameFailure: { cause: unknown } | undefined;
|
|
448
|
+
let frame = requestAnimationFrame(function draw() {
|
|
449
|
+
try {
|
|
450
|
+
const seen = stream.snapshot();
|
|
451
|
+
if (seen.ok && seen.value !== undefined) render(seen.value.value);
|
|
452
|
+
frame = requestAnimationFrame(draw);
|
|
453
|
+
} catch (cause) {
|
|
454
|
+
// Frame callbacks run outside the consuming loop; stop its pending read too.
|
|
455
|
+
frameFailure = { cause };
|
|
456
|
+
stop.abort(cause);
|
|
457
|
+
}
|
|
458
|
+
});
|
|
459
|
+
let last: unknown;
|
|
460
|
+
try {
|
|
461
|
+
for await (const r of stream) {
|
|
462
|
+
if (frameFailure !== undefined) throw frameFailure.cause;
|
|
463
|
+
if (r.ok) {
|
|
464
|
+
last = r.value;
|
|
465
|
+
render(r.value); // completion may arrive before the next preview frame
|
|
466
|
+
}
|
|
467
|
+
else {
|
|
468
|
+
// Earlier completed roots stay rendered, but do not make a failed stream successful.
|
|
469
|
+
last = undefined;
|
|
470
|
+
// The last recoverable prefix may arrive after the final preview frame.
|
|
471
|
+
if (r.failure.recovered !== undefined) {
|
|
472
|
+
render(r.failure.recovered.value);
|
|
473
|
+
}
|
|
474
|
+
handleFailure(r.failure);
|
|
475
|
+
}
|
|
476
|
+
}
|
|
477
|
+
} finally {
|
|
478
|
+
// Stop scheduled previews even when stream consumption or failure handling throws.
|
|
479
|
+
cancelAnimationFrame(frame);
|
|
480
|
+
}
|
|
481
|
+
return last;
|
|
482
|
+
}
|
|
483
|
+
```
|
|
484
|
+
|
|
485
|
+
<!-- not-compiled -->
|
|
486
|
+
```ts
|
|
487
|
+
import { streamText } from "ai";
|
|
488
|
+
const result = streamText({ model, prompt: "Reply with a JSON object describing ..." });
|
|
489
|
+
await showReply(result);
|
|
490
|
+
```
|
|
491
|
+
|
|
492
|
+
Tool-call arguments in the AI SDK arrive on `fullStream` as `tool-input-delta` parts, one stream per
|
|
493
|
+
tool-call `id`. Pass that ID with every outcome so the handler can route interleaved calls:
|
|
494
|
+
|
|
495
|
+
```ts
|
|
496
|
+
import { createStreamParser } from "partial-json-repair";
|
|
497
|
+
|
|
498
|
+
/** What a recipe needs from a stream parser: `createStreamParser()` returns one. */
|
|
499
|
+
type Feedable = Pick<ReturnType<typeof createStreamParser>, "feed" | "end" | "close">;
|
|
500
|
+
|
|
501
|
+
export async function toolArguments(
|
|
502
|
+
fullStream: AsyncIterable<{ type: string; id?: string; delta?: string }>,
|
|
503
|
+
): Promise<void> {
|
|
504
|
+
const calls = new Map<string, Feedable>();
|
|
505
|
+
try {
|
|
506
|
+
for await (const part of fullStream) {
|
|
507
|
+
if (part.type !== "tool-input-delta" || part.id === undefined || part.delta === undefined) continue;
|
|
508
|
+
let call = calls.get(part.id);
|
|
509
|
+
if (call === undefined) calls.set(part.id, (call = createStreamParser()));
|
|
510
|
+
for (const r of call.feed(part.delta)) handleResult(r, part.id);
|
|
511
|
+
}
|
|
512
|
+
for (const [id, call] of calls) for (const r of call.end()) handleResult(r, id);
|
|
513
|
+
} finally {
|
|
514
|
+
// Handler and source failures must release unfinished parser buffers too.
|
|
515
|
+
for (const call of calls.values()) call.close();
|
|
516
|
+
}
|
|
517
|
+
}
|
|
518
|
+
```
|
|
519
|
+
|
|
520
|
+
**OpenAI (Chat Completions with `stream: true`).** Text is in `choices[0].delta.content`:
|
|
521
|
+
|
|
522
|
+
```ts
|
|
523
|
+
import { parseStream } from "partial-json-repair";
|
|
524
|
+
|
|
525
|
+
async function* deltas(
|
|
526
|
+
chunks: AsyncIterable<{ choices: { delta: { content?: string | null } }[] }>,
|
|
527
|
+
): AsyncGenerator<string> {
|
|
528
|
+
for await (const chunk of chunks) {
|
|
529
|
+
const text = chunk.choices[0]?.delta.content;
|
|
530
|
+
if (text) yield text;
|
|
531
|
+
}
|
|
532
|
+
}
|
|
533
|
+
|
|
534
|
+
export async function readReply(
|
|
535
|
+
completion: AsyncIterable<{ choices: { delta: { content?: string | null } }[] }>,
|
|
536
|
+
): Promise<void> {
|
|
537
|
+
for await (const r of parseStream(deltas(completion))) {
|
|
538
|
+
if (r.ok) useValue(r.value);
|
|
539
|
+
else handleFailure(r.failure);
|
|
540
|
+
}
|
|
541
|
+
}
|
|
542
|
+
```
|
|
543
|
+
|
|
544
|
+
<!-- not-compiled -->
|
|
545
|
+
```ts
|
|
546
|
+
import OpenAI from "openai";
|
|
547
|
+
const client = new OpenAI();
|
|
548
|
+
const completion = await client.chat.completions.create({ model, messages, stream: true });
|
|
549
|
+
await readReply(completion);
|
|
550
|
+
```
|
|
551
|
+
|
|
552
|
+
With the Responses API, the same text arrives as `response.output_text.delta` events (`event.delta`),
|
|
553
|
+
and tool-call arguments as `response.function_call_arguments.delta` events keyed by `item_id`.
|
|
554
|
+
|
|
555
|
+
**Anthropic (Messages streaming).** Text arrives as `content_block_delta` events with a `text_delta`;
|
|
556
|
+
a tool call's input arrives as `input_json_delta` pieces (`partial_json`), one content block per
|
|
557
|
+
`index`. Pass the block index with every outcome so the handler can identify its source:
|
|
558
|
+
|
|
559
|
+
```ts
|
|
560
|
+
import { createStreamParser } from "partial-json-repair";
|
|
561
|
+
|
|
562
|
+
/** What a recipe needs from a stream parser: `createStreamParser()` returns one. */
|
|
563
|
+
type Feedable = Pick<ReturnType<typeof createStreamParser>, "feed" | "end" | "close">;
|
|
564
|
+
|
|
565
|
+
export async function readMessage(
|
|
566
|
+
events: AsyncIterable<{ type: string; index?: number; delta?: object }>,
|
|
567
|
+
): Promise<void> {
|
|
568
|
+
const blocks = new Map<number, Feedable>();
|
|
569
|
+
try {
|
|
570
|
+
for await (const event of events) {
|
|
571
|
+
if (event.type !== "content_block_delta" || event.index === undefined || !event.delta) continue;
|
|
572
|
+
const piece =
|
|
573
|
+
"text" in event.delta ? event.delta.text
|
|
574
|
+
: "partial_json" in event.delta ? event.delta.partial_json
|
|
575
|
+
: undefined;
|
|
576
|
+
if (typeof piece !== "string") continue;
|
|
577
|
+
let block = blocks.get(event.index);
|
|
578
|
+
if (block === undefined) blocks.set(event.index, (block = createStreamParser()));
|
|
579
|
+
for (const r of block.feed(piece)) handleResult(r, event.index);
|
|
580
|
+
}
|
|
581
|
+
for (const [index, block] of blocks) for (const r of block.end()) handleResult(r, index);
|
|
582
|
+
} finally {
|
|
583
|
+
// Handler and source failures must release unfinished parser buffers too.
|
|
584
|
+
for (const block of blocks.values()) block.close();
|
|
585
|
+
}
|
|
586
|
+
}
|
|
587
|
+
```
|
|
588
|
+
|
|
589
|
+
<!-- not-compiled -->
|
|
590
|
+
```ts
|
|
591
|
+
import Anthropic from "@anthropic-ai/sdk";
|
|
592
|
+
const client = new Anthropic();
|
|
593
|
+
const stream = client.messages.stream({ model, max_tokens: maxTokens, messages, tools });
|
|
594
|
+
await readMessage(stream);
|
|
595
|
+
```
|
|
596
|
+
|
|
597
|
+
Without `eager_input_streaming: true` on the tool definition, the API buffers and validates each
|
|
598
|
+
parameter value before streaming it, so a large argument arrives only once it is finished. With it,
|
|
599
|
+
fragments arrive as Claude writes them, without server-side JSON validation (Anthropic, "Fine-grained
|
|
600
|
+
tool streaming", read 2026-10-04), which is why `complete` and `cut` matter here: run a tool only on
|
|
601
|
+
a completed root whose `complete` is true and whose arguments pass your application validation.
|
|
602
|
+
|
|
603
|
+
## Optional features are imports
|
|
604
|
+
|
|
605
|
+
Four features cost code that most calls do not need, so each is a value you import and pass under
|
|
606
|
+
its option key. What you do not import is not in your bundle.
|
|
607
|
+
|
|
608
|
+
| Feature | Pass | What it adds |
|
|
609
|
+
| --- | --- | --- |
|
|
610
|
+
| Report | `{ diagnostics }` or `{ diagnostics: diagnostics({ maxEntries, scope }) }` | `diagnostics` on every result: repairs, errors, counts, confidence |
|
|
611
|
+
| Source spans | `{ locations }` | `span` on every successful value |
|
|
612
|
+
| Selection | `{ select: select(paths, { occurrences, fragments }) }` | Matches instead of whole values; see Selection |
|
|
613
|
+
| Skipping | `{ skip: skip(paths) }` | Named object members are not built |
|
|
614
|
+
|
|
615
|
+
```ts
|
|
616
|
+
import { ANY, diagnostics, locations, parse, select, skip } from "partial-json-repair";
|
|
617
|
+
|
|
618
|
+
parse(text); // nothing optional in the bundle
|
|
619
|
+
parse(text, { diagnostics }); // report and confidence
|
|
620
|
+
parse(text, { diagnostics: diagnostics({ maxEntries: 100, scope: "all" }) });
|
|
621
|
+
parse(text, { locations }); // spans, no report
|
|
622
|
+
parse(text, { select: select([["items", ANY]], { occurrences: true }) });
|
|
623
|
+
parse(text, { skip: skip([["debug"]]) }); // works without select
|
|
624
|
+
```
|
|
625
|
+
|
|
626
|
+
Settings travel with their feature: `maxEntries` (default 1000) caps the entries one report retains,
|
|
627
|
+
and `scope: "all"` audits skipped and unselected content too. Passing `true` where a feature belongs
|
|
628
|
+
(`{ diagnostics: true }`) is a compile error, and at run time a `TypeError` whose message names the
|
|
629
|
+
import to use. Everything is exported from `partial-json-repair`, plus the async verbs from
|
|
630
|
+
`partial-json-repair/async`.
|
|
631
|
+
|
|
632
|
+
## What you get back
|
|
633
|
+
|
|
634
|
+
One shape, discriminated on `ok`. Status lives outside the data, so `{"ok":false}` parses to a
|
|
635
|
+
success holding that object, and a failure never has a `value` to mistake for one:
|
|
636
|
+
|
|
637
|
+
<!-- not-compiled -->
|
|
638
|
+
```ts
|
|
639
|
+
type Result<T, F> = { ok: true; value: T } | { ok: false; failure: F };
|
|
640
|
+
|
|
641
|
+
// Recovered values and repaired text also carry:
|
|
642
|
+
type Completion = {
|
|
643
|
+
complete: boolean; // false when the source ended before the value did
|
|
644
|
+
cut?: Cut; // present exactly when complete is false
|
|
645
|
+
};
|
|
646
|
+
type Cut = {
|
|
647
|
+
path: readonly (string | number)[];
|
|
648
|
+
kind: "string" | "number" | "missing" | "between";
|
|
649
|
+
text?: string; // the unfinished token the value leaves out
|
|
650
|
+
};
|
|
651
|
+
|
|
652
|
+
// With the `diagnostics` feature, on BOTH branches; a failure still reports what it read.
|
|
653
|
+
type Diagnostics = {
|
|
654
|
+
repairs: readonly Repair[]; // what was altered, and where
|
|
655
|
+
errors: readonly ParseError[]; // what was refused, and where
|
|
656
|
+
counts: { repairs: number; errors: number }; // totals, even past `maxEntries`
|
|
657
|
+
entriesComplete: boolean; // false when the cap left some entries out
|
|
658
|
+
scanComplete: boolean; // false when a limit stopped the scanner early
|
|
659
|
+
scope: "all" | "selected"; // under `select`, what the entries cover
|
|
660
|
+
confidence: number; // 0..1, from every repair, stored or not
|
|
661
|
+
laterValues: readonly { start: Position; end: Position }[]; // values after the one returned
|
|
662
|
+
};
|
|
663
|
+
|
|
664
|
+
// Text input counts UTF-16 code units; byte input counts bytes. In a stream, `chunk` is the index
|
|
665
|
+
// of the chunk fed and `offset` counts from its start, in that chunk's unit.
|
|
666
|
+
type Position = { offset: number; unit: "utf16" | "byte"; chunk?: number };
|
|
667
|
+
```
|
|
668
|
+
|
|
669
|
+
When a refusal, resource limit, source failure or cancellation retains partial data,
|
|
670
|
+
`failure.recovered` carries its `value`, `complete` and any `cut`. The call remains failed;
|
|
671
|
+
inspect that recovery before deciding whether to retry.
|
|
672
|
+
|
|
673
|
+
If a single-result async operation succeeds and then fails to clean up, `failure.completed` holds
|
|
674
|
+
its successful outcome, including requested reports and locations. That is the verb's own output:
|
|
675
|
+
repaired text from `repair`, a span from `findJsonSpan`, not a recovered value. The call itself
|
|
676
|
+
remains failed.
|
|
677
|
+
An independent cleanup failure beside an earlier failure is `failure.cleanupFailure`; inspect the
|
|
678
|
+
primary failure and its recovery first.
|
|
679
|
+
|
|
680
|
+
**Branch on typed tags, never error messages.** Failures and report entries carry literal `kind`
|
|
681
|
+
tags. An exhaustive switch can detect an unhandled case at compile time.
|
|
682
|
+
|
|
683
|
+
The exported `RepairKind` and `ErrorKind` types describe report entries; each verb's result type
|
|
684
|
+
defines its failures and their fields. Read the [canonical type definitions](src/types.ts), rather
|
|
685
|
+
than copying a tag list into your application. For runtime repair names, use the keys of the
|
|
686
|
+
exported `REPAIR_PENALTY` table. Narrow a result on `ok`, then its failure on `kind`.
|
|
687
|
+
|
|
688
|
+
There is no `message` field to read. English is a formatter's job, at the edge of your application.
|
|
689
|
+
|
|
690
|
+
**Nothing throws for anything the input did.** Finding no JSON is a value, not an exception:
|
|
691
|
+
|
|
692
|
+
```ts
|
|
693
|
+
import { parse } from "partial-json-repair";
|
|
694
|
+
|
|
695
|
+
parse("no json here at all");
|
|
696
|
+
// { ok: false, failure: { kind: "no-json-found" } }
|
|
697
|
+
```
|
|
698
|
+
|
|
699
|
+
**Invalid options throw before input is read**, because they are a bug in the calling code, not
|
|
700
|
+
an outcome to handle. `initParser`, `createStreamParser` and every verb given options throw a
|
|
701
|
+
`TypeError` before reading any input; its `cause` is the typed reason
|
|
702
|
+
(`{ kind: "invalid-options", field, reason }`) for callers that load options at run time.
|
|
703
|
+
Promise-returning async verbs reject with it; `parseManyAsync` throws when the iterator is created,
|
|
704
|
+
before its first pull. The option types are exact: a misspelt key is a compile error, inline or in a
|
|
705
|
+
variable.
|
|
706
|
+
|
|
707
|
+
Normal iteration reports cleanup failures as `cleanup-failed` outcomes. Explicit iterator disposal
|
|
708
|
+
can reject with a cleanup cause; `iterator.throw(cause)` also rejects with the consumer's cause,
|
|
709
|
+
combined with any cleanup failure. Await those promises. `parseStream` provides `close()` when you
|
|
710
|
+
need a typed cleanup result instead.
|
|
711
|
+
|
|
712
|
+
The value is `unknown`. There is no `parse<T>()`: a type parameter there would be an assertion
|
|
713
|
+
dressed as a check, and real validation belongs to a validator you already have.
|
|
714
|
+
|
|
715
|
+
## Confidence
|
|
716
|
+
|
|
717
|
+
`confidence` scores every repair in the requested scope, including entries omitted by `maxEntries`:
|
|
718
|
+
`∏ (1 - penalty(kind))`. Many small repairs decay rather than clipping at zero.
|
|
719
|
+
|
|
720
|
+
It is a heuristic about recovery changes, not the probability that the data is correct. Successful
|
|
721
|
+
recovery does not validate an application schema; narrow or validate `value` before using it.
|
|
722
|
+
|
|
723
|
+
- **1.0**: nothing was guessed.
|
|
724
|
+
- **< 0.7**: consider retrying the model.
|
|
725
|
+
- **< 0.5**: likely garbled.
|
|
726
|
+
|
|
727
|
+
**Formatting repairs cost nothing.** A trailing comma, a single quote or a comment tells you the
|
|
728
|
+
model was sloppy, not that the data is uncertain. Closing an unfinished string or container lowers
|
|
729
|
+
the score because its ending was missing; recovery preserves the available content rather than
|
|
730
|
+
guessing the missing text or members.
|
|
731
|
+
|
|
732
|
+
```ts
|
|
733
|
+
import { diagnostics, parse } from "partial-json-repair";
|
|
734
|
+
|
|
735
|
+
const r = parse("{'a': 1, // note\n b: 2,}", { diagnostics });
|
|
736
|
+
r.diagnostics.repairs.map((x) => x.kind);
|
|
737
|
+
// ["single-quote", "comment-removed", "unquoted-key", "trailing-comma"]
|
|
738
|
+
r.diagnostics.confidence; // 0.97
|
|
739
|
+
```
|
|
740
|
+
|
|
741
|
+
The penalty table is exported as `REPAIR_PENALTY`, and `confidenceOf(repairs)` scores the repairs
|
|
742
|
+
you pass with it, so you can score a subset or explain a number to a user. It matches `confidence`
|
|
743
|
+
only while the report stored every repair: past `maxEntries` the report still scores the repairs it
|
|
744
|
+
does not store.
|
|
745
|
+
|
|
746
|
+
## Selection
|
|
747
|
+
|
|
748
|
+
`select` names the parts you want, `skip` names the parts you do not. Both take a list of paths, and
|
|
749
|
+
a path is a list of keys and indexes: nothing to escape, no query language to learn, and the compiler
|
|
750
|
+
checks it.
|
|
751
|
+
|
|
752
|
+
```ts
|
|
753
|
+
import { ANY, ANY_DEPTH, parse, select } from "partial-json-repair";
|
|
754
|
+
|
|
755
|
+
const doc = '{"items":[{"id":1,"name":"a"},{"id":2,"name":"b"}],"meta":{"id":9}}';
|
|
756
|
+
|
|
757
|
+
const names = parse(doc, { select: select([["items", ANY, "name"]]) });
|
|
758
|
+
if (names.ok) names.value.map((m) => [m.value, m.path]);
|
|
759
|
+
// [["a", ["items", 0, "name"]], ["b", ["items", 1, "name"]]]
|
|
760
|
+
|
|
761
|
+
const ids = parse(doc, { select: select([[ANY_DEPTH, "id"]]) });
|
|
762
|
+
if (ids.ok) ids.value.map((m) => m.path);
|
|
763
|
+
// [["items", 0, "id"], ["items", 1, "id"], ["meta", "id"]]
|
|
764
|
+
```
|
|
765
|
+
|
|
766
|
+
`ANY` matches one key or index and `ANY_DEPTH` matches zero or more, so `[[ANY_DEPTH, "id"]]` finds
|
|
767
|
+
every `id` wherever it turns up. `[]` as a path selects the root; `select([])` selects nothing.
|
|
768
|
+
|
|
769
|
+
**You get matches, not a trimmed document.** Each carries its `value`, the `path` it came from, its
|
|
770
|
+
source `span` and the indexes of the `selectors` that hit it. `repair` gives each match its own
|
|
771
|
+
repaired text, `findJsonSpan` its span, and `parseMany` and the streams deliver matches one at a time
|
|
772
|
+
with the root's `valueIndex`. Matches arrive in source order with an ancestor before its descendants;
|
|
773
|
+
with `occurrences: true` each arrives when it closes, so a descendant comes before its ancestor.
|
|
774
|
+
|
|
775
|
+
**`skip` is the one that trims**, because removing an object member invents no holes:
|
|
776
|
+
|
|
777
|
+
```ts
|
|
778
|
+
import { parse, skip } from "partial-json-repair";
|
|
779
|
+
|
|
780
|
+
const trimmed = parse('{"text":"hi","embedding":[0.1,0.2,0.3]}', { skip: skip([["embedding"]]) });
|
|
781
|
+
if (trimmed.ok) trimmed.value; // { text: "hi" }
|
|
782
|
+
```
|
|
783
|
+
|
|
784
|
+
A skip path may not end in a wildcard or an index, so it can never target an array element; that is
|
|
785
|
+
checked when `skip(...)` is called. Where `select` and `skip` overlap, **skip wins**.
|
|
786
|
+
|
|
787
|
+
**Selection omits unrelated branches from the recovered tree.** The scanner still reads excluded
|
|
788
|
+
content to enforce policies and find selected values. Token decoding and parser state can allocate
|
|
789
|
+
temporary storage; selection does not promise zero allocations for excluded content.
|
|
790
|
+
|
|
791
|
+
By default a match is the **final** value at its path: a later duplicate member replaces everything
|
|
792
|
+
matched beneath the earlier one, so matches are held until their ancestors close.
|
|
793
|
+
`select(paths, { occurrences: true })` delivers each source occurrence as it closes instead: earlier,
|
|
794
|
+
and worth it in a stream, at the price that an occurrence need not survive into the final object.
|
|
795
|
+
`fragments: decodedFragments` (streams only) implies occurrences. Import the delivery capability so
|
|
796
|
+
ordinary selection does not retain fragment delivery code.
|
|
797
|
+
|
|
798
|
+
Diagnostics follow the same scope. With `select`, a report covers the matches and the recovery it
|
|
799
|
+
took to find them, and says so in `scope: "selected"`; `diagnostics({ scope: "all" })` audits the
|
|
800
|
+
skipped content too. Skipping itself is never reported: you wrote the selector, and one entry per
|
|
801
|
+
skipped member would grow without bound on exactly the input skipping exists to make cheap.
|
|
802
|
+
|
|
803
|
+
## Streaming details
|
|
804
|
+
|
|
805
|
+
`parseStream` takes an async iterable, a plain iterable, or a web `ReadableStream`, of `string` or
|
|
806
|
+
`Uint8Array` chunks, and yields per-value results as they complete. It takes an abort signal:
|
|
807
|
+
|
|
808
|
+
```ts
|
|
809
|
+
import { parseStream } from "partial-json-repair";
|
|
810
|
+
|
|
811
|
+
for await (const result of parseStream(source, { signal: controller.signal })) {
|
|
812
|
+
if (result.ok) render(result.value);
|
|
813
|
+
else {
|
|
814
|
+
if (result.failure.recovered !== undefined) {
|
|
815
|
+
render(result.failure.recovered.value);
|
|
816
|
+
}
|
|
817
|
+
handleFailure(result.failure);
|
|
818
|
+
}
|
|
819
|
+
}
|
|
820
|
+
```
|
|
821
|
+
|
|
822
|
+
Cancellation stops further reads and parsing of the chunk already in hand, and requests cancellation
|
|
823
|
+
of a pending web read. An `aborted` failure carries the signal's reason and available recovery;
|
|
824
|
+
an earlier source failure or terminal parser failure stays primary. With `summary: true`, the summary
|
|
825
|
+
follows the failure. A source that throws yields a `source-failed` item carrying the original cause
|
|
826
|
+
and any recoverable prefix in `recovered`, instead of throwing through the iterator.
|
|
827
|
+
|
|
828
|
+
**Node streams**: pass a `Readable` directly, or compose it through the platform's `Duplex.from`:
|
|
829
|
+
|
|
830
|
+
```ts
|
|
831
|
+
import { Duplex } from "node:stream";
|
|
832
|
+
import { parseStream } from "partial-json-repair";
|
|
833
|
+
|
|
834
|
+
res
|
|
835
|
+
.pipe(Duplex.from((src: AsyncIterable<Buffer>) => parseStream(src)))
|
|
836
|
+
.on("data", (result) => handleResult(result));
|
|
837
|
+
```
|
|
838
|
+
|
|
839
|
+
**Web streams**: where available, `ReadableStream.from(parseStream(source))` gives a stream of
|
|
840
|
+
items with the same values and cancellation. `parseStream` also reads native Web streams directly,
|
|
841
|
+
including `Response.body`, `Request.body`, `Bun.file(path).stream()` and `TransformStream.readable`;
|
|
842
|
+
no intermediate text decoder or Node/Web conversion is required.
|
|
843
|
+
|
|
844
|
+
**Older Bun compatibility:** on Bun 1.3.14, a native `Readable.toWeb()` → `Readable.fromWeb()`
|
|
845
|
+
round-trip reordered one-byte chunks with `highWaterMark: 1`, without this library involved.
|
|
846
|
+
Pass the Node `Readable` directly or use a native Web source on that release. It also lacks
|
|
847
|
+
`ReadableStream.from`; use the standard `ReadableStream` constructor when an output bridge is needed.
|
|
848
|
+
Its native full-duplex HTTP bridge also truncated an upload sent through repeated one-byte
|
|
849
|
+
`node:http` writes into a pull-based Web response, without this library involved. Sending the
|
|
850
|
+
remaining upload with `end(bytes)` preserved it; Bun 1.4.4 preserved both forms. Avoid that old
|
|
851
|
+
HTTP bridge pattern rather than adding a parser workaround.
|
|
852
|
+
|
|
853
|
+
**Leaving and closing.** A `break` out of the loop requests the source's cleanup and does not wait
|
|
854
|
+
for it, so a producer that never answers cannot hang you. When you need the acknowledgement,
|
|
855
|
+
`await stream.close()`: it stops delivery, shares one cleanup attempt across every call, and answers
|
|
856
|
+
`{ ok: true }` or a `cleanup-failed` failure carrying what the source threw. `close({ signal })`
|
|
857
|
+
bounds the wait, and an abandoned wait answers `aborted` rather than claiming the source released. A
|
|
858
|
+
source that finishes normally and then refuses to be released ends the stream with a
|
|
859
|
+
`cleanup-failed` item; one that fails mid-stream carries a later cleanup failure beside its own
|
|
860
|
+
cause as `cleanupFailure`, never instead of it. On a manual stream, leaving a `feed()` iteration
|
|
861
|
+
early with `break` closes the stream; `close()` is idempotent and always succeeds, since it holds no
|
|
862
|
+
source.
|
|
863
|
+
|
|
864
|
+
With `summary: true` the last item of the stream is `{ kind: "summary", complete, roots, matches,
|
|
865
|
+
unmatched }`: whether the whole source was examined, and how many roots were recognised, refused
|
|
866
|
+
ones included. A summary has `kind` and no `ok`, so `"kind" in item` tells it from a result.
|
|
867
|
+
|
|
868
|
+
**Feed bytes, not text.** `feed` and `parseStream` accept `Uint8Array` as well as `string`, and own
|
|
869
|
+
the incremental decode. Decoding each chunk independently can corrupt a multi-byte character split
|
|
870
|
+
across chunks. A stream may mix text and byte chunks, and every reported position names its `chunk`
|
|
871
|
+
and counts from that chunk's start in its own unit.
|
|
872
|
+
|
|
873
|
+
Every one-shot verb also takes a `Uint8Array`. Bytes are decoded once, inside the parser, and every
|
|
874
|
+
position then counts in bytes; a malformed sequence becomes U+FFFD and is reported as `invalid-utf8`
|
|
875
|
+
at its bytes. A UTF-8 byte-order mark is removed, with a `bom-removed` repair that costs no
|
|
876
|
+
confidence, and a UTF-16 or UTF-32 mark fails with `unsupported-encoding` naming the encoding.
|
|
877
|
+
|
|
878
|
+
## Reusing one configuration
|
|
879
|
+
|
|
880
|
+
Serving many calls or streams on one configuration? Validate it once:
|
|
881
|
+
|
|
882
|
+
```ts
|
|
883
|
+
import { createStreamParser, diagnostics, initParser, parseStream, sharedView } from "partial-json-repair";
|
|
884
|
+
|
|
885
|
+
const parser = initParser({ diagnostics }); // once, at startup; bad options throw here
|
|
886
|
+
|
|
887
|
+
for await (const result of parseStream(request.body!, parser)) {
|
|
888
|
+
if (result.ok) handle(result.value);
|
|
889
|
+
else handleFailure(result.failure);
|
|
890
|
+
}
|
|
891
|
+
|
|
892
|
+
const sp = createStreamParser(parser, { view: sharedView }); // per stream
|
|
893
|
+
for (const result of sp.feed('{"title": "Hel')) handleResult(result);
|
|
894
|
+
```
|
|
895
|
+
|
|
896
|
+
`parse(input, parser, options?)`, `parseMany`, `repair` and `findJsonSpan` accept per-call options.
|
|
897
|
+
Supplied fields replace the configured ones (a field set to `undefined` keeps the configured value),
|
|
898
|
+
and calls without overrides reuse the validated settings. `createStreamParser(parser, options?)` and
|
|
899
|
+
`parseStream(source, parser, options?)` take only the stream options (`view`, `summary`, and
|
|
900
|
+
`signal` on `parseStream`); a parse option there is refused, never ignored.
|
|
901
|
+
`close()` releases the parser; later calls answer `parser-closed`.
|
|
902
|
+
|
|
903
|
+
## Whole-input async work
|
|
904
|
+
|
|
905
|
+
Import whole-input async calls from `partial-json-repair/async` when the caller must stay responsive
|
|
906
|
+
or cancel work already in progress. The result shape matches the synchronous verb, including partial
|
|
907
|
+
recovery under an `aborted` failure. Features are passed the same way.
|
|
908
|
+
|
|
909
|
+
```ts
|
|
910
|
+
import { auto, parseAsync } from "partial-json-repair/async";
|
|
911
|
+
|
|
912
|
+
const result = await parseAsync(modelOutput, { execution: auto });
|
|
913
|
+
if (result.ok) useValue(result.value);
|
|
914
|
+
else {
|
|
915
|
+
if (result.failure.recovered !== undefined) {
|
|
916
|
+
useValue(result.failure.recovered.value);
|
|
917
|
+
}
|
|
918
|
+
handleFailure(result.failure);
|
|
919
|
+
}
|
|
920
|
+
```
|
|
921
|
+
|
|
922
|
+
Pass `{ signal: controller.signal }` when the caller can cancel the operation.
|
|
923
|
+
|
|
924
|
+
**Every async call names where it runs.** The strategy is a value imported from
|
|
925
|
+
`partial-json-repair/async`, and an app ships only the strategies it imports. A default would be
|
|
926
|
+
referenced by every call, so every app would carry its engine; that is why `execution` is required.
|
|
927
|
+
|
|
928
|
+
| Strategy | Runs | Use it when |
|
|
929
|
+
| --- | --- | --- |
|
|
930
|
+
| `auto` | On the calling thread. Small input finishes inline; larger input runs in bounded slices that yield to the host task queue | The usual choice: responsive, cancellable, works on every host |
|
|
931
|
+
| `cooperative` | On the calling thread, yielding between slices at every size | Even small calls must let timers and cancellation callbacks run |
|
|
932
|
+
| `worker` | In a lazily started background worker, reused on Node, Bun and Deno, ended when idle in browsers | The scan must leave the main thread entirely |
|
|
933
|
+
|
|
934
|
+
An app that imports only `worker` ships no same-thread engine, and one that never imports `worker`
|
|
935
|
+
contains no worker code. Pages without shared memory cancel through messages, so cross-origin
|
|
936
|
+
isolation is not required. Nothing moves work to a worker automatically: worker startup and transfer
|
|
937
|
+
costs per host and workload are not yet measured. The choice is fixed before any input is read.
|
|
938
|
+
|
|
939
|
+
```ts
|
|
940
|
+
import { parseAsync, worker } from "partial-json-repair/async";
|
|
941
|
+
|
|
942
|
+
const result = await parseAsync(modelOutput, { execution: worker });
|
|
943
|
+
if (!result.ok && result.failure.kind === "execution-unavailable") {
|
|
944
|
+
console.warn(result.failure.message); // what failed, why, and what to do
|
|
945
|
+
}
|
|
946
|
+
```
|
|
947
|
+
|
|
948
|
+
A wrong strategy is a bug at the call site, so it throws a `TypeError` before reading input. Its
|
|
949
|
+
message names the fix:
|
|
950
|
+
|
|
951
|
+
```text
|
|
952
|
+
partial-json-repair: option `execution` is required and takes an imported strategy, never a string or a copy: `import { auto } from "partial-json-repair/async"` and pass `{ execution: auto }` to run on this thread, `cooperative` to yield at every size, or `worker` for a background worker.
|
|
953
|
+
```
|
|
954
|
+
|
|
955
|
+
A worker that cannot run is an outcome, not an exception: `execution-unavailable` with a typed
|
|
956
|
+
`reason` to branch on and a `message` for logs. `reason: "unsupported"` means the host has no
|
|
957
|
+
`Worker` or `MessageChannel` (an edge runtime such as workerd), so pass `auto` there. `reason:
|
|
958
|
+
"failed"` means the worker did not start or stopped answering, usually because the bundle does not
|
|
959
|
+
deploy its file (see below). Both arrive before input is read unless the worker fails mid-call, and
|
|
960
|
+
then any recovered prefix comes with them.
|
|
961
|
+
|
|
962
|
+
The entry also exports `parseManyAsync`, `repairAsync`, `findJsonSpanAsync` and `initAsyncParser`,
|
|
963
|
+
which returns the parser directly.
|
|
964
|
+
|
|
965
|
+
If host-task scheduling fails, the call returns `scheduling-failed` with the original `cause` and
|
|
966
|
+
any available `recovered` value. Inspect that prefix and restore host scheduling before retrying;
|
|
967
|
+
stopped work does not silently restart on another route.
|
|
968
|
+
|
|
969
|
+
For a request or batch, put its cancellation signal on the reusable parser. A per-call signal
|
|
970
|
+
cancels only that sibling; it cannot override task cancellation. Both signals are observed, with the
|
|
971
|
+
first reason preserved. If both were already aborted, the task reason wins:
|
|
972
|
+
|
|
973
|
+
```ts
|
|
974
|
+
import { auto, initAsyncParser, parseAsync, repairAsync } from "partial-json-repair/async";
|
|
975
|
+
|
|
976
|
+
const task = new AbortController();
|
|
977
|
+
const sibling = new AbortController();
|
|
978
|
+
using parser = initAsyncParser({ execution: auto, signal: task.signal });
|
|
979
|
+
const first = parseAsync(modelOutput, parser, { signal: sibling.signal });
|
|
980
|
+
const second = repairAsync(modelOutput, parser);
|
|
981
|
+
sibling.abort("preview no longer needed"); // first only
|
|
982
|
+
const results = await Promise.all([first, second]);
|
|
983
|
+
for (const result of results) {
|
|
984
|
+
if (result.ok) useValue(result.value);
|
|
985
|
+
else handleFailure(result.failure);
|
|
986
|
+
}
|
|
987
|
+
task.abort("request finished"); // current and future operations on this parser
|
|
988
|
+
```
|
|
989
|
+
|
|
990
|
+
Closing the parser prevents new calls. Cancel its task to stop running calls; closing it does not
|
|
991
|
+
terminate a worker used by another parser.
|
|
992
|
+
|
|
993
|
+
**The worker in a bundled app.** Node, Bun and Deno load it from the package. In a browser build:
|
|
994
|
+
|
|
995
|
+
| Bundler | Worker |
|
|
996
|
+
| --- | --- |
|
|
997
|
+
| Vite, webpack, Rspack | Emitted and run with no setup |
|
|
998
|
+
| esbuild, Rolldown, `Bun.build` | Add the recipe below; without it a forced worker fails with `execution-unavailable` |
|
|
999
|
+
|
|
1000
|
+
The recipe emits the lazy receiver `worker-executor` with the app's chunks and builds `async-worker`
|
|
1001
|
+
on its own into the same directory. Keep the worker a separate build: as an entry of the app's build,
|
|
1002
|
+
its engine lands in a chunk the app loads at startup (esbuild: 105 KB of startup JavaScript instead
|
|
1003
|
+
of 20 KB), and the worker thread shares no module instances with the page anyway.
|
|
1004
|
+
|
|
1005
|
+
<!-- not-compiled -->
|
|
1006
|
+
```ts
|
|
1007
|
+
// esbuild (Rolldown takes the same maps as `input`, with entryFileNames: "[name].js")
|
|
1008
|
+
const shared = { bundle: true, splitting: true, format: "esm", outdir: "dist" } as const;
|
|
1009
|
+
await esbuild.build({
|
|
1010
|
+
...shared,
|
|
1011
|
+
entryPoints: {
|
|
1012
|
+
main: "src/main.ts",
|
|
1013
|
+
"worker-executor": "node_modules/partial-json-repair/dist/worker-executor.js",
|
|
1014
|
+
},
|
|
1015
|
+
});
|
|
1016
|
+
await esbuild.build({
|
|
1017
|
+
...shared,
|
|
1018
|
+
entryPoints: { "async-worker": "node_modules/partial-json-repair/dist/async-worker.js" },
|
|
1019
|
+
});
|
|
1020
|
+
// Bun.build keeps the package files' basenames as entry names.
|
|
1021
|
+
const options = { outdir: "dist", splitting: true, naming: { entry: "[name].[ext]" } } as const;
|
|
1022
|
+
await Bun.build({
|
|
1023
|
+
...options,
|
|
1024
|
+
entrypoints: ["src/main.ts", "node_modules/partial-json-repair/dist/worker-executor.js"],
|
|
1025
|
+
});
|
|
1026
|
+
await Bun.build({ ...options, entrypoints: ["node_modules/partial-json-repair/dist/async-worker.js"] });
|
|
1027
|
+
```
|
|
1028
|
+
|
|
1029
|
+
Cooperative execution needs no worker at all, so an app that skips the recipe loses only the
|
|
1030
|
+
option to move the scan off the main thread.
|
|
1031
|
+
|
|
1032
|
+
## Exact numbers
|
|
1033
|
+
|
|
1034
|
+
**A number a double cannot hold stays exact.** `12345678901234567890`, `1e400` and `1e-400` come back
|
|
1035
|
+
as an `ExactNumber` holding the source token, not as a rounded `number`, `Infinity` or `0`, and
|
|
1036
|
+
nothing is reported, because nothing was changed. Identity is `isExactNumber(v)`, never a field
|
|
1037
|
+
shape. Every conversion is checked:
|
|
1038
|
+
|
|
1039
|
+
```ts
|
|
1040
|
+
import { isExactNumber, parse, stringify } from "partial-json-repair";
|
|
1041
|
+
|
|
1042
|
+
const r = parse('{"id": 12345678901234567890, "tiny": 1e-400}');
|
|
1043
|
+
const id = r.ok ? (r.value as { id: unknown }).id : undefined;
|
|
1044
|
+
if (isExactNumber(id)) {
|
|
1045
|
+
id.text; // "12345678901234567890"
|
|
1046
|
+
id.toBigInt({ maxDigits: 30 }); // { ok: true, value: 12345678901234567890n }
|
|
1047
|
+
id.toNumber(); // { ok: false, failure: { kind: "inexact" } }
|
|
1048
|
+
id.toNumber({ allowRounding: true }); // { ok: true, value: 12345678901234567000 }
|
|
1049
|
+
`${id}`; // "12345678901234567890": string conversion gives the exact token
|
|
1050
|
+
// Number(id) and id * 2 throw a TypeError instead of rounding silently.
|
|
1051
|
+
}
|
|
1052
|
+
if (r.ok) stringify(r.value); // { ok: true, value: '{"id":12345678901234567890,"tiny":1e-400}' }
|
|
1053
|
+
```
|
|
1054
|
+
|
|
1055
|
+
`exactNumbers: true` makes EVERY number one, `0.1` included. `stringify(value, { maxOutput? })`
|
|
1056
|
+
writes them back unquoted and refuses what JSON cannot spell (`undefined`, `NaN`, a cycle, a getter)
|
|
1057
|
+
with a typed failure naming the path; it never calls `toJSON`.
|
|
1058
|
+
|
|
1059
|
+
## More helpers
|
|
1060
|
+
|
|
1061
|
+
`tryParsers` tries several data formats in order:
|
|
1062
|
+
|
|
1063
|
+
```ts
|
|
1064
|
+
import { parse, tryParsers } from "partial-json-repair";
|
|
1065
|
+
|
|
1066
|
+
const result = tryParsers(source, [
|
|
1067
|
+
parseYaml,
|
|
1068
|
+
(text) => {
|
|
1069
|
+
const parsed = parse(text);
|
|
1070
|
+
if (!parsed.ok) throw new Error(parsed.failure.kind);
|
|
1071
|
+
return parsed.value;
|
|
1072
|
+
},
|
|
1073
|
+
]);
|
|
1074
|
+
```
|
|
1075
|
+
|
|
1076
|
+
Parsers run synchronously in caller order. Thrown parser errors and rejected values advance to the
|
|
1077
|
+
next parser; Promise results, predicate errors and exhaustion stop with typed failures. The default
|
|
1078
|
+
predicate accepts non-null objects, including arrays; pass `isResult` to accept another domain.
|
|
1079
|
+
|
|
1080
|
+
`createMetrics` is a caller-owned counting scope, so ordinary parsing carries no counters:
|
|
1081
|
+
|
|
1082
|
+
```ts
|
|
1083
|
+
import { createMetrics, parse } from "partial-json-repair";
|
|
1084
|
+
|
|
1085
|
+
const measured = createMetrics({ onDangerousKey: "drop" });
|
|
1086
|
+
parse('{id: 1, "__proto__": 2}', measured);
|
|
1087
|
+
const snapshot = measured.read();
|
|
1088
|
+
snapshot.repairs["unquoted-key"]; // 1
|
|
1089
|
+
snapshot.security.dangerousKeysBlocked; // 1
|
|
1090
|
+
measured.reset();
|
|
1091
|
+
```
|
|
1092
|
+
|
|
1093
|
+
Every verb accepts the scope where it accepts an `initParser` handle, and counts what it returns:
|
|
1094
|
+
`parseMany` items as you read them, and `parseStream(source, measured)` or
|
|
1095
|
+
`createStreamParser(measured)` items plus retained-input peaks, compactions and buffer refusals. A
|
|
1096
|
+
handle rather than methods, so a bundle keeps only the verbs it imports. `read(true)` returns a
|
|
1097
|
+
detached snapshot and resets the interval in one call.
|
|
1098
|
+
|
|
1099
|
+
## Options and presets
|
|
1100
|
+
|
|
1101
|
+
Four presets, exported as constants rather than looked up by name, so the compiler checks them and a
|
|
1102
|
+
bundler can drop the ones you do not use.
|
|
1103
|
+
|
|
1104
|
+
| Preset | What it is for |
|
|
1105
|
+
| --- | --- |
|
|
1106
|
+
| `LLM_BALANCED` | **The default.** Comments, single quotes, trailing commas, unquoted keys, JS literals |
|
|
1107
|
+
| `LLM_FULL` | Maximum recovery: dangerous keys dropped as well |
|
|
1108
|
+
| `STRICT` | Every leniency off; the first refusal fails the value and stops the scan |
|
|
1109
|
+
| `COMPAT` | Legacy readings for migrating callers: `0123` as 123, a lone bare word as a string (see `docs/MIGRATION.md`) |
|
|
1110
|
+
|
|
1111
|
+
```ts
|
|
1112
|
+
import { parse, STRICT } from "partial-json-repair";
|
|
1113
|
+
|
|
1114
|
+
parse("{'a': 1}", STRICT);
|
|
1115
|
+
// { ok: false, failure: { kind: "policy-refused", issue: { kind: "unparsable-span", ... },
|
|
1116
|
+
// recovered: { value: {}, complete: false, cut: { path: ["a"], kind: "missing" } } } }
|
|
1117
|
+
// STRICT stops at the first refusal, so the recovered value says where reading stopped.
|
|
1118
|
+
```
|
|
1119
|
+
|
|
1120
|
+
A leniency flag being off does not mean the construct is unrecognised: it is still consumed, and
|
|
1121
|
+
reported as a refusal instead of a repair. That keeps the tree stable across presets, so tightening
|
|
1122
|
+
a flag changes the verdict and the report rather than the data, which is still there as `recovered`.
|
|
1123
|
+
|
|
1124
|
+
`recover` names one repair kind and wins over the flag that would decide it, in both directions:
|
|
1125
|
+
|
|
1126
|
+
```ts
|
|
1127
|
+
import { parse } from "partial-json-repair";
|
|
1128
|
+
|
|
1129
|
+
// This model never writes single quotes, so reading one is a guess about damage that is not there.
|
|
1130
|
+
parse(`{"a": {'b': 1}}`, { recover: { "single-quote": false } });
|
|
1131
|
+
// { ok: false, failure: { kind: "policy-refused", issue: { path: ["a"], ... },
|
|
1132
|
+
// recovered: { value: { a: { b: 1 } }, complete: true } } }
|
|
1133
|
+
|
|
1134
|
+
parse('{"a": NaN}', { recover: { "js-literal": true } });
|
|
1135
|
+
// { ok: true, value: { a: NaN }, complete: true }: the value the source spelled, which `repair()` writes as `null` and reports
|
|
1136
|
+
|
|
1137
|
+
// By default, a string missing its closing quote ends at a line break when a quoted key follows,
|
|
1138
|
+
// or when the rest of the input is only closers the string did not open:
|
|
1139
|
+
parse('{"a": "x\n "b": 1}');
|
|
1140
|
+
// { ok: true, value: { a: "x", b: 1 }, complete: true }
|
|
1141
|
+
parse('{"d": "unclosed\n}');
|
|
1142
|
+
// { ok: true, value: { d: "unclosed" }, complete: true }
|
|
1143
|
+
// A `}` that closes a `{` opened inside the string stays in it: models write code with raw line
|
|
1144
|
+
// breaks, so that `}` is code. This opt-in ends the string there anyway, and at a comma or a
|
|
1145
|
+
// bare-word key line:
|
|
1146
|
+
parse('{"code": "f() {\n}', { recover: { "guessed-string-end": true } });
|
|
1147
|
+
// { ok: true, value: { code: "f() {" }, complete: true }
|
|
1148
|
+
```
|
|
1149
|
+
|
|
1150
|
+
The individual options, all optional:
|
|
1151
|
+
|
|
1152
|
+
| Option | Default | What it does |
|
|
1153
|
+
| --- | --- | --- |
|
|
1154
|
+
| `allowComments`, `allowSingleQuotes`, `allowTrailingComma`, `allowUnquotedKeys` | `true` | The four leniencies models need most |
|
|
1155
|
+
| `allowJavaScript` | `true` | `undefined`, `NaN`, `Infinity`, hex/octal/binary numbers, `1_000` digit separators |
|
|
1156
|
+
| `repairBroken` | `true` | Recover broken syntax; off refuses it without changing data |
|
|
1157
|
+
| `recover` | from the flags | Override one repair kind, on or off, past the flag that decides it |
|
|
1158
|
+
| `cutSuffix` | absent | `true` appends `…` to a cut string; a string appends that text |
|
|
1159
|
+
| `diagnostics`, `locations`, `select`, `skip` | absent | Imported features; see above |
|
|
1160
|
+
| `exactNumbers` | `false` | Every number as an `ExactNumber`, fractions included; numbers a double cannot hold are exact even when off |
|
|
1161
|
+
| `builtinParsers` | `true` | `false` never hands input to `JSON.parse`, clean or rewritten from a cut-off reply — see Performance |
|
|
1162
|
+
| `maxErrors` | `1000` | Refusals a value may accumulate before the scan stops |
|
|
1163
|
+
| `maxBuffer`, `maxDepth`, `maxScalar` | `10 MB`, `512`, `1 KB` | Retained stream source, nesting and bare-root-scalar bounds |
|
|
1164
|
+
| `maxRetained` | `maxBuffer` | What a stream holds while waiting: pending matches, banked string parts, unread values |
|
|
1165
|
+
| `objectOnly`, `requiredKeys` | `false`, `[]` | Accept only object roots, with these own keys |
|
|
1166
|
+
| `onDangerousKey` | `"keep"` | `keep`, `drop` or `reject` for `__proto__` and friends |
|
|
1167
|
+
| `nullPrototype`, `customDangerousKeys` | `false`, `[]` | `Object.create(null)` objects: no pollution possible, but no `hasOwnProperty` or `toString` either; more keys to treat as dangerous |
|
|
1168
|
+
|
|
1169
|
+
Stream-only: `view` (imported `sharedView` or `ownedView`), `summary`, and `signal` on `parseStream`.
|
|
1170
|
+
|
|
1171
|
+
`maxBuffer` and `maxRetained` bound the **stream** parser only. You already hold the whole string
|
|
1172
|
+
when you call `parse(text)`, and refusing to read the second half of it would be truncation dressed
|
|
1173
|
+
as a resource bound. Exceeding any bound returns a `limit-reached` failure naming the resource, with
|
|
1174
|
+
whatever was read as `recovered`; a stream that hit one answers `stream-closed` from then on.
|
|
1175
|
+
|
|
1176
|
+
## Prototype pollution
|
|
1177
|
+
|
|
1178
|
+
The six dangerous keys (`__proto__`, `constructor`, `__defineGetter__`, `__defineSetter__`,
|
|
1179
|
+
`__lookupGetter__`, `__lookupSetter__`) are handled at the one point where a key enters the tree,
|
|
1180
|
+
with `Object.defineProperty` rather than assignment.
|
|
1181
|
+
|
|
1182
|
+
```ts
|
|
1183
|
+
import { diagnostics, LLM_FULL, parse } from "partial-json-repair";
|
|
1184
|
+
|
|
1185
|
+
const r = parse('{"__proto__": {"x":1}, "safe": 2}');
|
|
1186
|
+
if (r.ok) {
|
|
1187
|
+
Object.keys(r.value as object); // ["__proto__", "safe"]: an ORDINARY own property, defined not assigned
|
|
1188
|
+
Object.getPrototypeOf(r.value as object) === Object.prototype; // true: no prototype was touched
|
|
1189
|
+
}
|
|
1190
|
+
|
|
1191
|
+
parse('{"__proto__": {"x":1}, "safe": 2}', { ...LLM_FULL, diagnostics });
|
|
1192
|
+
// value: { safe: 2 } — dropped, and reported: diagnostics.repairs = ["dangerous-key"]
|
|
1193
|
+
```
|
|
1194
|
+
|
|
1195
|
+
The default is `keep`, which stores the key as **data**. `drop` removes it and reports a repair;
|
|
1196
|
+
`reject` refuses the whole object and reports an error. All three keep safe siblings.
|
|
1197
|
+
|
|
1198
|
+
See [SECURITY.md](SECURITY.md) for the full threat model and how to report a problem.
|
|
1199
|
+
|
|
1200
|
+
## Performance
|
|
1201
|
+
|
|
1202
|
+
Clean input uses `JSON.parse` when that route can preserve the requested outcome. So does a reply
|
|
1203
|
+
cut off mid-stream, or one holding comments or trailing commas: one pass appends the missing closers
|
|
1204
|
+
or deletes the comments and commas, and `JSON.parse` reads the result, several times faster than
|
|
1205
|
+
the recovery scanner and with the same value, `complete` and `cut`. A report, a selection, exact
|
|
1206
|
+
numbers, null-prototype objects and a refused `duplicate-key` repair select the scanner, because
|
|
1207
|
+
`JSON.parse` would erase information those calls require; `drop` or `reject` for dangerous keys
|
|
1208
|
+
costs one substring scan before the fast route.
|
|
1209
|
+
|
|
1210
|
+
[The measured comparison](docs/PRIOR-ART.md) covers every runtime the package ships to: Node, Bun,
|
|
1211
|
+
Deno, WebKit, Chromium and workerd. `bun run bench:baseline` reproduces the throughput matrix.
|
|
1212
|
+
|
|
1213
|
+
## Limitations
|
|
1214
|
+
|
|
1215
|
+
- **Recovery invents structure, and says so.** An unclosed string or object is closed, and a member
|
|
1216
|
+
whose value never arrived is dropped. On retained values, `complete` and `cut` identify source
|
|
1217
|
+
truncation; requested `diagnostics` explain recovery changes with paths. If you cannot tolerate
|
|
1218
|
+
invention, check `complete`, read the report and reject, or use `STRICT`.
|
|
1219
|
+
- **A number or word still being read is not in a view; a string is.** A snapshot after
|
|
1220
|
+
`{"n": 12` shows `{}`, because `1` of `123` is a different number rather than a partial one. At
|
|
1221
|
+
the end of input a valid number stays, marked `kind: "number"`, because it may be short.
|
|
1222
|
+
- **Fractions are doubles unless you ask.** `0.1` is the double `0.1`, as from `JSON.parse`; only an
|
|
1223
|
+
integer past 2^53, an overflow or an underflow is kept exact by default.
|
|
1224
|
+
- **Duplicate keys are reported only when you ask for a report.** Last wins either way, as
|
|
1225
|
+
`JSON.parse` does.
|
|
1226
|
+
- **Brackets inside a sentence are not JSON.** `see [1] for the citation` returns `no-json-found`,
|
|
1227
|
+
deliberately. A container standing alone or filling a fenced block is always a candidate; inside
|
|
1228
|
+
prose it must carry a colon, a quoted string, or more than one value.
|
|
1229
|
+
- **`{{name}}` at a candidate root is a template, not JSON.** An opener in the root object's first
|
|
1230
|
+
key position rejects that candidate; nested openers remain recoverable.
|
|
1231
|
+
- **A bare scalar counts when it stands alone, fills a code fence, or has a line of its own among
|
|
1232
|
+
other values.** `42` alone and each line of `1\n2` parse; the `42` in `answer is
|
|
1233
|
+
42, data: {"a":1}` does not, and you get the object. A lone unquoted word, or a lone token that is
|
|
1234
|
+
no number (`1..5`), is refused with the text in `recovered`, and so is text opened by an
|
|
1235
|
+
apostrophe that never closes (`'tis the season`); `COMPAT` reads either as a string.
|
|
1236
|
+
- **Nesting is bounded at 512 levels** by default. Past it the value fails with `limit-reached` for
|
|
1237
|
+
the `depth` resource and carries what was read as `recovered`, never a stack overflow.
|
|
1238
|
+
- **`parse()` is synchronous and does not yield.** For large input on a thread that must stay
|
|
1239
|
+
responsive, use `parseAsync` from `partial-json-repair/async`.
|
|
1240
|
+
- **Under Terser 5.51.2 (webpack's default minifier), never drain `feed()` with a discarded
|
|
1241
|
+
spread.** Terser rewrites `[...parser.feed(chunk)];` into a bare call that never iterates, for any
|
|
1242
|
+
generator, so the chunk is never parsed. Read the items, or drain with
|
|
1243
|
+
`for (const _ of parser.feed(chunk));`.
|
|
1244
|
+
|
|
1245
|
+
## How this differs from the neighbours
|
|
1246
|
+
|
|
1247
|
+
`partial-json` and `best-effort-json-parser` complete truncated JSON but do not tell you that they
|
|
1248
|
+
did, or where. `jsonrepair` repairs the text but hands you no diagnostics, and `json-repair-js` logs
|
|
1249
|
+
only a missing colon, without a path; neither gives streaming snapshots. This package marks every
|
|
1250
|
+
cut value and, on request, reports every change with its path. None of them lets you turn off
|
|
1251
|
+
one repair: `jsonrepair`, `untruncate-json` and `best-effort-json-parser` take no options,
|
|
1252
|
+
`json-repair-js`'s options set output format and logging, `repair-json-stream` only reports its
|
|
1253
|
+
repairs to a callback, and `partial-json`'s `Allow` flags choose which value types may be cut
|
|
1254
|
+
short. Here each repair kind is a switch.
|
|
1255
|
+
|
|
1256
|
+
**[docs/PRIOR-ART.md](docs/PRIOR-ART.md)** measures every one of them against this one: capability,
|
|
1257
|
+
agreement over the whole conformance corpus, JSONTestSuite strictness, speed, size and peak memory,
|
|
1258
|
+
including the runtimes' own parsers, each figure with its runtime and the command that reproduces
|
|
1259
|
+
it. Reach for `JSON.parse` when your input is supposed to be valid; reach for this when it is not,
|
|
1260
|
+
and when you need to know what changed.
|
|
1261
|
+
|
|
1262
|
+
## Licence
|
|
1263
|
+
|
|
1264
|
+
Apache-2.0.
|