@r3b1s/pi-repair-layer 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +159 -293
- package/dist/index.d.ts +2 -0
- package/dist/index.d.ts.map +1 -0
- package/{index.ts → dist/index.js} +2 -1
- package/dist/index.js.map +1 -0
- package/dist/src/core.d.ts +8 -0
- package/dist/src/core.d.ts.map +1 -0
- package/dist/src/core.js +7 -0
- package/dist/src/core.js.map +1 -0
- package/dist/src/envelope.d.ts +21 -0
- package/dist/src/envelope.d.ts.map +1 -0
- package/dist/src/envelope.js +158 -0
- package/dist/src/envelope.js.map +1 -0
- package/dist/src/grammar-recovery.d.ts +105 -0
- package/dist/src/grammar-recovery.d.ts.map +1 -0
- package/dist/src/grammar-recovery.js +1052 -0
- package/dist/src/grammar-recovery.js.map +1 -0
- package/dist/src/grammar.d.ts +2 -0
- package/dist/src/grammar.d.ts.map +1 -0
- package/dist/src/grammar.js +2 -0
- package/dist/src/grammar.js.map +1 -0
- package/dist/src/index.d.ts +36 -0
- package/dist/src/index.d.ts.map +1 -0
- package/dist/src/index.js +558 -0
- package/dist/src/index.js.map +1 -0
- package/dist/src/lifecycle.d.ts +34 -0
- package/dist/src/lifecycle.d.ts.map +1 -0
- package/dist/src/lifecycle.js +133 -0
- package/dist/src/lifecycle.js.map +1 -0
- package/dist/src/pi.d.ts +19 -0
- package/dist/src/pi.d.ts.map +1 -0
- package/dist/src/pi.js +38 -0
- package/dist/src/pi.js.map +1 -0
- package/dist/src/pipeline.d.ts +10 -0
- package/dist/src/pipeline.d.ts.map +1 -0
- package/dist/src/pipeline.js +166 -0
- package/dist/src/pipeline.js.map +1 -0
- package/dist/src/policy.d.ts +16 -0
- package/dist/src/policy.d.ts.map +1 -0
- package/dist/src/policy.js +31 -0
- package/dist/src/policy.js.map +1 -0
- package/dist/src/preprocess.d.ts +37 -0
- package/dist/src/preprocess.d.ts.map +1 -0
- package/dist/src/preprocess.js +216 -0
- package/dist/src/preprocess.js.map +1 -0
- package/dist/src/repair-engine.d.ts +91 -0
- package/dist/src/repair-engine.d.ts.map +1 -0
- package/dist/src/repair-engine.js +457 -0
- package/dist/src/repair-engine.js.map +1 -0
- package/dist/src/settings.d.ts +38 -0
- package/dist/src/settings.d.ts.map +1 -0
- package/dist/src/settings.js +87 -0
- package/dist/src/settings.js.map +1 -0
- package/dist/src/tables.d.ts +16 -0
- package/dist/src/tables.d.ts.map +1 -0
- package/dist/src/tables.js +302 -0
- package/dist/src/tables.js.map +1 -0
- package/dist/src/types.d.ts +52 -0
- package/dist/src/types.d.ts.map +1 -0
- package/dist/src/types.js +2 -0
- package/dist/src/types.js.map +1 -0
- package/dist/src/value-strips.d.ts +52 -0
- package/dist/src/value-strips.d.ts.map +1 -0
- package/dist/src/value-strips.js +191 -0
- package/dist/src/value-strips.js.map +1 -0
- package/docs/how-it-works.md +227 -0
- package/docs/operations.md +141 -0
- package/docs/research.md +247 -0
- package/docs/tool-owner-integration.md +327 -0
- package/package.json +31 -6
- package/src/grammar-recovery.ts +0 -1269
- package/src/index.ts +0 -621
- package/src/repair-engine.ts +0 -498
- package/src/settings.ts +0 -95
- package/src/tables.ts +0 -184
- package/src/value-strips.ts +0 -218
|
@@ -0,0 +1,227 @@
|
|
|
1
|
+
# How repair works
|
|
2
|
+
|
|
3
|
+
This document is the detailed behavior and safety reference for
|
|
4
|
+
pi-repair-layer. For a shorter overview and installation instructions, start
|
|
5
|
+
with the [README](../README.md). Tool authors should also read the
|
|
6
|
+
[integration guide](tool-owner-integration.md).
|
|
7
|
+
|
|
8
|
+
## Pipeline
|
|
9
|
+
|
|
10
|
+
Tool arguments move through these ordered stages:
|
|
11
|
+
|
|
12
|
+
1. bounded raw-envelope recovery;
|
|
13
|
+
2. the tool owner's compatibility `prepareArguments` hook;
|
|
14
|
+
3. selector-guided preprocessing;
|
|
15
|
+
4. policy/model-gated cleanup of values that already satisfy the schema;
|
|
16
|
+
5. strict validation without native conversion;
|
|
17
|
+
6. bounded iterative repair at reported schema-issue locations;
|
|
18
|
+
7. native conversion and final validation; and
|
|
19
|
+
8. structured outcome and result feedback.
|
|
20
|
+
|
|
21
|
+
Every mutation has a stable rule ID, pipeline stage, and explanatory note. A
|
|
22
|
+
`repaired` outcome is impossible unless the final value passes the supplied
|
|
23
|
+
schema. An `unrepairable` outcome preserves the original input and reports no
|
|
24
|
+
claimed changes.
|
|
25
|
+
|
|
26
|
+
### Profiles
|
|
27
|
+
|
|
28
|
+
| Profile | Envelope, exact preprocessing, schema repair | Truncated completion and model-gated value cleanup | Grammar text | Promotion |
|
|
29
|
+
|---|---|---|---|---|
|
|
30
|
+
| `conservative` | yes | observe only | observe only | never |
|
|
31
|
+
| `adaptive` | yes | schema-validated | strip known tools | never |
|
|
32
|
+
| `recover` | yes | schema-validated | strip known tools | gated known-tool calls |
|
|
33
|
+
|
|
34
|
+
`adaptive` is the default for new and migrated non-recover installs. Existing
|
|
35
|
+
users who explicitly enabled grammar recovery migrate to `recover` without
|
|
36
|
+
losing that choice.
|
|
37
|
+
|
|
38
|
+
Unknown or disallowed tool grammar is preserved under every profile. A separate
|
|
39
|
+
`unknownGrammarText: "strip"` setting can remove it, but can never make the
|
|
40
|
+
unknown call executable. The narrow grammar override supports `off`, `observe`,
|
|
41
|
+
`strip`, and `recover`.
|
|
42
|
+
|
|
43
|
+
## Repair catalog
|
|
44
|
+
|
|
45
|
+
### Envelope recovery
|
|
46
|
+
|
|
47
|
+
Before inspecting individual fields, the pipeline can:
|
|
48
|
+
|
|
49
|
+
- decode recursively JSON-stringified argument objects within configured limits;
|
|
50
|
+
- escape raw control characters inside JSON strings;
|
|
51
|
+
- unwrap a singleton array containing one plain object; and
|
|
52
|
+
- under adaptive/recover policy, try a fixed set of closing suffixes for a
|
|
53
|
+
truncated object.
|
|
54
|
+
|
|
55
|
+
A truncated candidate is accepted only if the complete downstream pipeline
|
|
56
|
+
validates it. Non-singleton arrays are never discarded into defaults, and
|
|
57
|
+
unrecoverable input never becomes `{}`.
|
|
58
|
+
|
|
59
|
+
### Schema-guided repairs
|
|
60
|
+
|
|
61
|
+
The iterative engine repairs only locations reported by strict validation. Its
|
|
62
|
+
fixed rules include:
|
|
63
|
+
|
|
64
|
+
| Rule | Example | Result |
|
|
65
|
+
|---|---|---|
|
|
66
|
+
| exact alias | `{file_path: "/x"}` | `{path: "/x"}` |
|
|
67
|
+
| drop invalid optional null | `{path: "/x", offset: null}` | `{path: "/x"}` |
|
|
68
|
+
| drop empty object placeholder | `{tags: {}}` where array expected | omit `tags` |
|
|
69
|
+
| parse stringified array | `{include: '["a","b"]'}` | `{include: ["a", "b"]}` |
|
|
70
|
+
| parse stringified object | object field contains JSON text | parsed object |
|
|
71
|
+
| wrap string as array | `{include: "foo"}` | `{include: ["foo"]}` |
|
|
72
|
+
| wrap root string | `"echo hi"` for bash | `{command: "echo hi"}` |
|
|
73
|
+
| structural edit fold | flat old/new fields | canonical `edits` array |
|
|
74
|
+
|
|
75
|
+
Aliases and structural changes must be configured by the tool owner. The
|
|
76
|
+
pipeline does not use edit distance, semantic guessing, or generic unknown-key
|
|
77
|
+
deletion. Built-in alias configuration covers common Claude Code shapes such as
|
|
78
|
+
`file_path`, `old_string`, and `new_string`; aider-style `search` and `replace`;
|
|
79
|
+
and generic alternatives including `cmd`, `query`, `text`, and `contents`,
|
|
80
|
+
including configured nested edit fields.
|
|
81
|
+
|
|
82
|
+
### Selector-guided preprocessing
|
|
83
|
+
|
|
84
|
+
Some bad values pass validation because they have the right JSON type.
|
|
85
|
+
Preprocessors address only configured semantic locations such as `/path`,
|
|
86
|
+
`/files/*`, or `/edits/*/oldText`.
|
|
87
|
+
|
|
88
|
+
Available selector preprocessors include:
|
|
89
|
+
|
|
90
|
+
- markdown auto-link cleanup on filesystem paths;
|
|
91
|
+
- string-or-array and scalar compatibility conversions;
|
|
92
|
+
- exact optional aliases that would otherwise pass as extra keys;
|
|
93
|
+
- model-gated anchor bleed cleanup; and
|
|
94
|
+
- model-gated `<arg_key>` / `<arg_value>` token cleanup.
|
|
95
|
+
|
|
96
|
+
The built-in tools use exact aliases, filesystem-path cleanup, model-gated
|
|
97
|
+
artifact cleanup, and the edit structural fold at their known-safe locations.
|
|
98
|
+
Custom tool owners can opt into the other selector kinds described in the
|
|
99
|
+
[integration guide](tool-owner-integration.md#configure-only-known-safe-transforms).
|
|
100
|
+
|
|
101
|
+
Anchor cleanup is never applied to `grep.pattern`, where `^` and `$` are valid
|
|
102
|
+
regex syntax. Content fields are not parsed or path-cleaned just because their
|
|
103
|
+
text resembles JSON or a filename. Markdown path cleanup unwraps only the
|
|
104
|
+
degenerate auto-link whose link text equals its URL without the protocol; real
|
|
105
|
+
markdown links remain unchanged.
|
|
106
|
+
|
|
107
|
+
The built-in model gates apply anchor cleanup to `kimi-k2`, `minimax`, and `glm`
|
|
108
|
+
families and grammar-token cleanup to `glm`. Custom tool owners choose their own
|
|
109
|
+
selectors and model-family expressions.
|
|
110
|
+
|
|
111
|
+
## Grammar-leak handling
|
|
112
|
+
|
|
113
|
+
A grammar leak occurs when a model prints its intended call as assistant text
|
|
114
|
+
instead of emitting a real tool call. The parser recognizes ten grammar
|
|
115
|
+
families adapted from pi-tool-repair and ignores examples inside fenced code.
|
|
116
|
+
|
|
117
|
+
- `off` leaves the message alone and records nothing.
|
|
118
|
+
- `observe` classifies recognized candidates without changing the message.
|
|
119
|
+
- `strip` removes recognized known-tool grammar without executing it.
|
|
120
|
+
- `recover` may promote a recognized call into a real same-turn tool call.
|
|
121
|
+
|
|
122
|
+
Promotion turns text into execution, so all of these gates must pass:
|
|
123
|
+
|
|
124
|
+
- the user explicitly selected `recover`;
|
|
125
|
+
- the message role is assistant;
|
|
126
|
+
- the original stop reason is `stop`, never `length`, `error`, or `aborted`;
|
|
127
|
+
- the message contains no real tool call already;
|
|
128
|
+
- the parsed arguments are non-empty; and
|
|
129
|
+
- the named tool is active or explicitly allowed.
|
|
130
|
+
|
|
131
|
+
The stop-reason gate preserves pi's protection that fails tool calls from
|
|
132
|
+
truncated output; see [research Claim 7](research.md#claim-7--stopreason-length-causes-pi-to-fail-all-tool-calls-in-the-message).
|
|
133
|
+
|
|
134
|
+
Unknown-tool text remains visible by default. Even the explicit unknown-text
|
|
135
|
+
strip setting removes text only; it cannot promote an unavailable tool.
|
|
136
|
+
|
|
137
|
+
When promotion succeeds, the generated call ID is associated directly with the
|
|
138
|
+
repair lifecycle so the eventual result receives one `<repair_note>`.
|
|
139
|
+
|
|
140
|
+
## Why repair runs before validation
|
|
141
|
+
|
|
142
|
+
pi's call sequence is:
|
|
143
|
+
|
|
144
|
+
```text
|
|
145
|
+
tool.prepareArguments(raw) → validateToolArguments() → tool_call → execute
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
The exact sequence, replacement propagation, and result lifecycle are verified
|
|
149
|
+
against pi source in [research Claims 1–5](research.md#claim-1--loop-ordering-preparearguments-runs-before-validation-which-runs-before-the-tool_call-event)
|
|
150
|
+
and exercised by upstream-drift tests.
|
|
151
|
+
|
|
152
|
+
A validation failure occurs before `tool_call`, so an event listener cannot
|
|
153
|
+
repair an invalid built-in call. pi-repair-layer instead replaces each built-in
|
|
154
|
+
definition with the original definition plus a chained `prepareArguments`.
|
|
155
|
+
Schemas, executors, prompt metadata, and renderers remain pi's own.
|
|
156
|
+
|
|
157
|
+
This ordering also prevents native conversion from silently changing classic
|
|
158
|
+
failure modes:
|
|
159
|
+
|
|
160
|
+
| Model sends | Native conversion can produce | Risk |
|
|
161
|
+
|---|---|---|
|
|
162
|
+
| `'["a","b"]'` for an array | `['["a","b"]']` | executes with one garbage item |
|
|
163
|
+
| `null` for an optional string | `"null"` | searches for or writes an unintended string |
|
|
164
|
+
| `null` for a required path | `"null"` | targets a file literally named `null` |
|
|
165
|
+
| `null` for an optional number | `0` | silently changes behavior |
|
|
166
|
+
|
|
167
|
+
The pipeline checks strictly first and repairs those issue locations before
|
|
168
|
+
native conversion. Benign conversion remains pi's responsibility when no
|
|
169
|
+
repair rule fires.
|
|
170
|
+
|
|
171
|
+
## Safety invariants
|
|
172
|
+
|
|
173
|
+
Property tests and deterministic fuzzing enforce that the core:
|
|
174
|
+
|
|
175
|
+
- does not throw or hang on bounded JSON-domain input;
|
|
176
|
+
- does not mutate caller-owned input or object prototypes;
|
|
177
|
+
- is deterministic and idempotent;
|
|
178
|
+
- preserves strictly valid input when no configured valid-value transform applies;
|
|
179
|
+
- records a note and stable rule ID for every mutation;
|
|
180
|
+
- reports repaired only after final schema validity; and
|
|
181
|
+
- terminates within configured byte, depth, attempt, candidate, and work limits.
|
|
182
|
+
|
|
183
|
+
Lifecycle queues are capped, expire stale entries, use FIFO collision handling,
|
|
184
|
+
and clear at session shutdown. Persisted indicators and telemetry contain only
|
|
185
|
+
value-free metadata. See the [operations guide](operations.md#telemetry-and-privacy)
|
|
186
|
+
for the exact record shape.
|
|
187
|
+
|
|
188
|
+
## Fail-closed behavior
|
|
189
|
+
|
|
190
|
+
If no bounded repair validates, the extension throws a model-readable retry
|
|
191
|
+
error before execution. This avoids returning raw input to native conversion,
|
|
192
|
+
which could turn `content: null` into the literal string `"null"` and write it.
|
|
193
|
+
|
|
194
|
+
`PI_TOOL_REPAIR_PASSTHROUGH=1` restores native behavior for compatibility, but
|
|
195
|
+
it weakens this guarantee and should be used deliberately.
|
|
196
|
+
|
|
197
|
+
## Prior art and hook placement
|
|
198
|
+
|
|
199
|
+
The value-strip rules and grammar-recovery approach come from Tom X Nguyen's
|
|
200
|
+
MIT-licensed [`monotykamary/pi-tool-repair`](https://github.com/monotykamary/pi-tool-repair).
|
|
201
|
+
This project adapted:
|
|
202
|
+
|
|
203
|
+
- anchor-bleed and grammar-token cleanup, moved to path-scoped preprocessing
|
|
204
|
+
before validation, exempting regex fields, and reporting through the shared
|
|
205
|
+
lifecycle; and
|
|
206
|
+
- grammar parsers, candidate selection, code-fence handling, and range removal,
|
|
207
|
+
with stricter role, tool, empty-argument, and stop-reason promotion gates.
|
|
208
|
+
|
|
209
|
+
Both projects draw on Command Code's validate-then-repair approach. The
|
|
210
|
+
different built-in hook follows directly from pi's sequence: a malformed call
|
|
211
|
+
fails validation before `tool_call`, and a successful `tool_call` propagates
|
|
212
|
+
in-place mutation but not input reassignment. Those facts are documented with
|
|
213
|
+
source citations in [research Claims 1–4](research.md#claim-1--loop-ordering-preparearguments-runs-before-validation-which-runs-before-the-tool_call-event).
|
|
214
|
+
|
|
215
|
+
## Limitations
|
|
216
|
+
|
|
217
|
+
- The installable extension wraps pi's seven built-ins. Other tools require
|
|
218
|
+
explicit owner integration through the compiled `/pi` adapter or `/core` API.
|
|
219
|
+
- Two extensions that replace the same built-in do not compose; load order
|
|
220
|
+
determines which definition wins.
|
|
221
|
+
- Unconfigured aliases, structural guesses, and unknown-field deletion are out
|
|
222
|
+
of scope.
|
|
223
|
+
- Phantom tool calls are not repaired. At the verified pi version, a tool-use
|
|
224
|
+
stop with no call is a clean terminal state rather than a stuck retry loop;
|
|
225
|
+
see [research Claim 6](research.md#claim-6--stopreason-error-and-aborted-is-terminal-not-a-retry).
|
|
226
|
+
- Grammar promotion cannot recover empty arguments, unavailable tools, existing
|
|
227
|
+
real calls, or truncated output.
|
|
@@ -0,0 +1,141 @@
|
|
|
1
|
+
# Operations and verification
|
|
2
|
+
|
|
3
|
+
This guide covers pi-repair-layer's user settings, local telemetry, diagnostics,
|
|
4
|
+
and test commands. Behavior and safety details live in
|
|
5
|
+
[How repair works](how-it-works.md).
|
|
6
|
+
|
|
7
|
+
## Runtime binding
|
|
8
|
+
|
|
9
|
+
pi's extension loader aliases `@earendil-works/pi-coding-agent` and `typebox` to
|
|
10
|
+
the running pi installation, so the extension uses the live runtime's own
|
|
11
|
+
modules and validator. This repository's `node_modules/` exists for development,
|
|
12
|
+
typechecking, and tests; it is not a second runtime loaded beside pi.
|
|
13
|
+
|
|
14
|
+
## Display and policy settings
|
|
15
|
+
|
|
16
|
+
Run this command in pi:
|
|
17
|
+
|
|
18
|
+
```text
|
|
19
|
+
/repair-settings
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
The menu controls:
|
|
23
|
+
|
|
24
|
+
- policy profile: `conservative`, `adaptive`, or `recover`;
|
|
25
|
+
- grammar override: `off`, `observe`, `strip`, or `recover`;
|
|
26
|
+
- unknown-tool grammar text: `preserve` or `strip`;
|
|
27
|
+
- the `🔨 ✓ input repaired (rules...)` TUI indicator; and
|
|
28
|
+
- visible repair-note text beneath the result.
|
|
29
|
+
|
|
30
|
+
Settings are stored at:
|
|
31
|
+
|
|
32
|
+
```text
|
|
33
|
+
~/.pi/agent/tool-repair/settings.json
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Override the location with `PI_TOOL_REPAIR_SETTINGS=<path>`. Existing settings
|
|
37
|
+
migrate in memory and are not rewritten until the user saves them again.
|
|
38
|
+
|
|
39
|
+
An optional `grammarAllowedTools` list restricts names eligible for promotion.
|
|
40
|
+
When it is empty, the active tool set is used. Unknown or disallowed calls are
|
|
41
|
+
never executable.
|
|
42
|
+
|
|
43
|
+
Repair indicators are persisted as value-free custom session entries, so they
|
|
44
|
+
survive `/reload` and session resume. The notes themselves are deliberately not
|
|
45
|
+
persisted.
|
|
46
|
+
|
|
47
|
+
## Telemetry and privacy
|
|
48
|
+
|
|
49
|
+
Repair outcomes append locally to:
|
|
50
|
+
|
|
51
|
+
```text
|
|
52
|
+
~/.pi/agent/tool-repair/telemetry.jsonl
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Each JSONL record may include timestamp, tool, model ID, profile, pipeline
|
|
56
|
+
stages, stable rule IDs, outcome, issue summary, grammar family, and an FNV-1a
|
|
57
|
+
fingerprint of the tool/failure shape.
|
|
58
|
+
|
|
59
|
+
Records never include raw or repaired arguments, filesystem paths, commands,
|
|
60
|
+
content, secrets, or value-bearing note text.
|
|
61
|
+
|
|
62
|
+
There are two channels:
|
|
63
|
+
|
|
64
|
+
- Tool records identify `repaired`, `unrepairable`, or `recovered` outcomes.
|
|
65
|
+
- Message records describe observe/strip grammar events that have no tool-call
|
|
66
|
+
ID to attach to.
|
|
67
|
+
|
|
68
|
+
Summarize the local records from pi with:
|
|
69
|
+
|
|
70
|
+
```text
|
|
71
|
+
/repair-stats
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
The report groups outcomes by tool and model and tallies rule and grammar
|
|
75
|
+
families. Existing telemetry records remain readable as fields are added.
|
|
76
|
+
|
|
77
|
+
Environment controls:
|
|
78
|
+
|
|
79
|
+
| Variable | Effect |
|
|
80
|
+
|---|---|
|
|
81
|
+
| `PI_TOOL_REPAIR_TELEMETRY=off` | Disable telemetry. |
|
|
82
|
+
| `PI_TOOL_REPAIR_TELEMETRY=<path>` | Write telemetry to another file. |
|
|
83
|
+
| `PI_TOOL_REPAIR_LOG=1` | Print repair decisions to stderr. |
|
|
84
|
+
| `PI_TOOL_REPAIR_PASSTHROUGH=1` | Return unrepairable input to pi's native conversion instead of failing closed. |
|
|
85
|
+
|
|
86
|
+
## Verification
|
|
87
|
+
|
|
88
|
+
The repository uses Node 22, pnpm, Vitest, TypeScript, Biome, and ESLint. Tool
|
|
89
|
+
versions and combined tasks are managed with mise.
|
|
90
|
+
|
|
91
|
+
```bash
|
|
92
|
+
pnpm install
|
|
93
|
+
mise run ci
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
`mise run ci` runs strict typechecking, both linters, and the complete Vitest
|
|
97
|
+
suite. Additional release checks are:
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
pnpm run test:package
|
|
101
|
+
pnpm run test:fuzz
|
|
102
|
+
test/run-chaos.sh
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
- `test:package` builds and packs the npm artifact, installs it into a clean
|
|
106
|
+
temporary consumer, imports every public subpath, and typechecks the fixture.
|
|
107
|
+
- `test:fuzz` runs the bounded deterministic raw-envelope campaign.
|
|
108
|
+
- `run-chaos.sh` registers a scripted provider and drives malformed calls
|
|
109
|
+
through the real pi binary without network requests or tokens.
|
|
110
|
+
|
|
111
|
+
The in-process upstream-drift suite also drives pi's real agent loop using the
|
|
112
|
+
faux provider. It asserts the ordering, event propagation, result replacement,
|
|
113
|
+
handler composition, schema shapes, truncation protection, and verified pi
|
|
114
|
+
minor version used by this extension.
|
|
115
|
+
|
|
116
|
+
### Larger and replayed fuzz runs
|
|
117
|
+
|
|
118
|
+
Run 10,000 local cases:
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
pnpm run test:fuzz:large
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Replay a reported seed and case budget:
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
PI_REPAIR_FUZZ_SEED=<seed> \
|
|
128
|
+
PI_REPAIR_FUZZ_CASES=<count> \
|
|
129
|
+
pnpm run test:fuzz
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
A discovered failure prints its seed and minimized input. Confirmed failures
|
|
133
|
+
should become named regression fixtures.
|
|
134
|
+
|
|
135
|
+
## Upgrading pi
|
|
136
|
+
|
|
137
|
+
`docs/research.md` records every pi behavior the implementation relies on, with
|
|
138
|
+
source locations and the verification date. Before changing the verified pi
|
|
139
|
+
minor version, follow the
|
|
140
|
+
[re-verification checklist](research.md#re-verification-checklist-run-on-any-pi-minor-bump)
|
|
141
|
+
and run all checks above.
|
package/docs/research.md
ADDED
|
@@ -0,0 +1,247 @@
|
|
|
1
|
+
# Research: pi agent-loop facts this extension relies on
|
|
2
|
+
|
|
3
|
+
This is the tracked, claim-by-claim ledger of the mechanical facts about pi that
|
|
4
|
+
`pi-repair-layer` is built on. Every design decision (which hook, why
|
|
5
|
+
`prepareArguments`, the grammar-recovery `stopReason` gate, the dropped
|
|
6
|
+
schema-anchor defense) traces back to one of the claims below. If a pi upgrade
|
|
7
|
+
changes one of these, the corresponding tripwire in `test/upstream-drift.test.ts`
|
|
8
|
+
is meant to fail loudly — this document tells the next reader what the tripwire
|
|
9
|
+
is protecting and where to re-verify it.
|
|
10
|
+
|
|
11
|
+
## Verification provenance
|
|
12
|
+
|
|
13
|
+
- **Verification date:** 2026-07-17
|
|
14
|
+
- **Source read:** pi monorepo TypeScript source, clone at `~/Local/docs/pi`,
|
|
15
|
+
git `0e6909f0` (`git describe`: `v0.80.6-24-g0e6909f0`). Package versions in
|
|
16
|
+
the clone: `pi-coding-agent`, `pi-agent-core` (`packages/agent`), and `pi-ai`
|
|
17
|
+
all `0.80.6`.
|
|
18
|
+
- **Installed (this repo's `node_modules`):** `@earendil-works/pi-ai`,
|
|
19
|
+
`@earendil-works/pi-coding-agent`, and `@earendil-works/pi-tui` `0.80.6`.
|
|
20
|
+
Clone citations below remain the source-of-truth citations; installed `dist/`
|
|
21
|
+
line numbers shift but the tripwires execute that published build directly.
|
|
22
|
+
- **Method:** direct source reading, plus the behavioral tripwires in
|
|
23
|
+
`test/upstream-drift.test.ts` that execute the claims against the installed
|
|
24
|
+
packages.
|
|
25
|
+
|
|
26
|
+
## Claims
|
|
27
|
+
|
|
28
|
+
### Claim 1 — Loop ordering: `prepareArguments` runs before validation, which runs before the `tool_call` event
|
|
29
|
+
|
|
30
|
+
Per tool call, pi runs, in order: `tool.prepareArguments(raw)` →
|
|
31
|
+
`validateToolArguments(...)` → (if configured) the `beforeToolCall` hook, which
|
|
32
|
+
is what surfaces the `tool_call` extension event → `tool.execute(validatedArgs)`.
|
|
33
|
+
|
|
34
|
+
- `packages/agent/src/agent-loop.ts:588-600` — `prepareToolCallArguments` calls
|
|
35
|
+
`tool.prepareArguments(toolCall.arguments)`.
|
|
36
|
+
- `packages/agent/src/agent-loop.ts:618-621` — inside `prepareToolCall`:
|
|
37
|
+
`prepareToolCallArguments(...)` then `validateToolArguments(tool, preparedToolCall)`
|
|
38
|
+
then `config.beforeToolCall(...)`.
|
|
39
|
+
- `packages/agent/src/agent-loop.ts:653-657` — only after the above does the
|
|
40
|
+
call resolve to `{ kind: "prepared", ... args: validatedArgs }` for execution.
|
|
41
|
+
|
|
42
|
+
**Why it matters:** `prepareArguments` is the *only* seam that sees the raw input
|
|
43
|
+
before validation can throw. This is the founding reason the extension overrides
|
|
44
|
+
built-in tools rather than handling the `tool_call` event.
|
|
45
|
+
|
|
46
|
+
### Claim 2 — `validateToolArguments` throws on failure (Convert-then-Check)
|
|
47
|
+
|
|
48
|
+
`validateToolArguments` clones the args, runs TypeBox `Value.Convert` (coercion),
|
|
49
|
+
then `validator.Check`; on failure it throws a formatted `Error`.
|
|
50
|
+
|
|
51
|
+
- `packages/ai/src/utils/validation.ts:278-309` — `structuredClone` →
|
|
52
|
+
`Value.Convert(tool.parameters, args)` → `validator.Check(args)` → on failure,
|
|
53
|
+
builds an error message and `throw new Error(errorMessage)` (line 309).
|
|
54
|
+
|
|
55
|
+
**Why it matters:** two things. (a) A validation failure throws before the
|
|
56
|
+
`tool_call` event fires, so that event can never repair a failing built-in call
|
|
57
|
+
(basis of Claim 1's consequence). (b) `Value.Convert` runs *first* and silently
|
|
58
|
+
coerces (`null` → `"null"`, `'["a"]'` → `['["a"]']`), which is why the repair
|
|
59
|
+
engine checks strictly **before** Convert — see `src/repair-engine.ts`.
|
|
60
|
+
|
|
61
|
+
### Claim 3 — `tool_call` event: in-place `input` mutation propagates, reassignment does not
|
|
62
|
+
|
|
63
|
+
The `tool_call` event's `input` is the very `validatedArgs` object the loop will
|
|
64
|
+
execute with, so mutating it in place reaches `execute`. But the event dispatcher
|
|
65
|
+
returns only the handler's `{ block, reason }` result — a handler that
|
|
66
|
+
*reassigns* `event.input = newObject` has that new object dropped.
|
|
67
|
+
|
|
68
|
+
- `packages/agent/src/agent-loop.ts:621-626` — `beforeToolCall` is passed
|
|
69
|
+
`args: validatedArgs`.
|
|
70
|
+
- `packages/coding-agent/src/core/agent-session.ts:424-436` — `beforeToolCall`
|
|
71
|
+
forwards to `runner.emitToolCall({ ..., input: args })`; `input` **is** the
|
|
72
|
+
`validatedArgs` reference.
|
|
73
|
+
- `packages/coding-agent/src/core/extensions/runner.ts:881-902` — `emitToolCall`
|
|
74
|
+
returns the handler result and the caller reads only `.block`
|
|
75
|
+
(`agent-loop.ts:638`). The executed object is the separate `validatedArgs`
|
|
76
|
+
reference (`agent-loop.ts:657`), so a reassignment of `event.input` is never
|
|
77
|
+
read back.
|
|
78
|
+
|
|
79
|
+
**Why it matters:** this is why an upstream extension that does
|
|
80
|
+
`event.input = repairedClone` on the `tool_call` event cannot repair a built-in
|
|
81
|
+
call even setting aside Claim 2 — its computed repair is discarded.
|
|
82
|
+
|
|
83
|
+
### Claim 4 — `message_end` replacement mutates the loop's message in place, listeners are awaited, and the replacement executes same-turn (with a role guard)
|
|
84
|
+
|
|
85
|
+
A `message_end` handler that returns `{ message }` has that message written back
|
|
86
|
+
onto the loop's own assistant-message object **in place**, before the loop
|
|
87
|
+
filters the message for tool calls — so a replacement whose content includes a
|
|
88
|
+
`toolCall` block is executed in the same turn. The replacement must keep the
|
|
89
|
+
original role.
|
|
90
|
+
|
|
91
|
+
- `packages/agent/src/agent-loop.ts:359,372` — `streamAssistantResponse` emits
|
|
92
|
+
`message_end` with `finalMessage`, then `return finalMessage` (line 373).
|
|
93
|
+
- `packages/agent/src/agent.ts:571-572` — the emit path awaits every listener
|
|
94
|
+
(`for (const listener of this.listeners) await listener(event, signal)`), so
|
|
95
|
+
the replacement is applied before `streamAssistantResponse` returns.
|
|
96
|
+
- `packages/coding-agent/src/core/agent-session.ts:709-726` — on `message_end`,
|
|
97
|
+
`emitMessageEnd` returns a replacement which is written via
|
|
98
|
+
`_replaceMessageInPlace(event.message, normalized)`.
|
|
99
|
+
- `packages/coding-agent/src/core/agent-session.ts:657-671` —
|
|
100
|
+
`_replaceMessageInPlace` deletes the target's keys and `Object.assign`s the
|
|
101
|
+
replacement onto the *same object reference* the loop holds.
|
|
102
|
+
- `packages/coding-agent/src/core/extensions/runner.ts:804-808` — role guard:
|
|
103
|
+
a handler result whose `message.role !== currentMessage.role` is rejected with
|
|
104
|
+
an error and skipped.
|
|
105
|
+
- `packages/agent/src/agent-loop.ts:203` — back in the loop, tool calls are read
|
|
106
|
+
from `message.content` *after* `streamAssistantResponse` returns; because the
|
|
107
|
+
replacement mutated that same object, injected `toolCall` blocks are seen and
|
|
108
|
+
executed this turn.
|
|
109
|
+
|
|
110
|
+
**Why it matters:** grammar-leak recovery hooks `message_end`, returns a
|
|
111
|
+
same-role replacement with the leaked text stripped, and (in `recover` mode)
|
|
112
|
+
appends `toolCall` blocks that execute same-turn and re-enter our
|
|
113
|
+
`prepareArguments` repair path. All of that depends on this claim.
|
|
114
|
+
|
|
115
|
+
### Claim 5 — `before_provider_request` propagates the returned payload
|
|
116
|
+
|
|
117
|
+
A `before_provider_request` handler's return value replaces the outgoing request
|
|
118
|
+
payload for later handlers and the provider.
|
|
119
|
+
|
|
120
|
+
- `packages/coding-agent/src/core/extensions/runner.ts:965-997` —
|
|
121
|
+
`emitBeforeProviderRequest` threads `currentPayload = handlerResult` whenever a
|
|
122
|
+
handler returns a non-`undefined` value, and returns the final payload.
|
|
123
|
+
|
|
124
|
+
**Why it matters:** this is the hook the *dropped* schema-anchor defense would
|
|
125
|
+
have used. It is documented here so that if scope ever broadens to non-built-in
|
|
126
|
+
tools with regex `pattern` schemas (see Claim 7), the mechanism is known-good.
|
|
127
|
+
|
|
128
|
+
### Claim 6 — `stopReason: "error"` (and `"aborted"`) is terminal, not a retry
|
|
129
|
+
|
|
130
|
+
When the assistant message's `stopReason` is `"error"` or `"aborted"`, the loop
|
|
131
|
+
emits `turn_end` + `agent_end` and returns — it does not auto-retry.
|
|
132
|
+
|
|
133
|
+
- `packages/agent/src/agent-loop.ts:196-200`.
|
|
134
|
+
|
|
135
|
+
**Why it matters:** it demotes the upstream "phantom toolUse → error triggers
|
|
136
|
+
auto-retry" claim (that normalization is out of scope here — a clean stop, not a
|
|
137
|
+
retry, on this pi). Also bounds what the grammar-recovery `stopReason` gate may
|
|
138
|
+
safely do.
|
|
139
|
+
|
|
140
|
+
### Claim 7 — `stopReason: "length"` causes pi to fail all tool calls in the message
|
|
141
|
+
|
|
142
|
+
If tool calls are present but `stopReason` is `"length"` (output truncated by the
|
|
143
|
+
token limit), pi fails every tool call rather than executing possibly-truncated
|
|
144
|
+
arguments.
|
|
145
|
+
|
|
146
|
+
- `packages/agent/src/agent-loop.ts:207-214` — when `toolCalls.length > 0`,
|
|
147
|
+
`message.stopReason === "length"` routes to
|
|
148
|
+
`failToolCallsFromTruncatedMessage(toolCalls, emit)` instead of
|
|
149
|
+
`executeToolCalls(...)`.
|
|
150
|
+
- `packages/agent/src/agent-loop.ts:383` — `failToolCallsFromTruncatedMessage`
|
|
151
|
+
definition.
|
|
152
|
+
|
|
153
|
+
**Why it matters:** this is exactly the protection the grammar-recovery
|
|
154
|
+
`stopReason` gate must not bypass. Upstream promotes leaked calls regardless of
|
|
155
|
+
`stopReason`, overwriting `"length"` with `"toolUse"` and defeating this
|
|
156
|
+
truncation guard. Our gate only *promotes* on `stopReason: "stop"` (stripping is
|
|
157
|
+
allowed on any `stopReason`), so a truncated message keeps failing its calls.
|
|
158
|
+
|
|
159
|
+
### Claim 8 — Built-in tool schemas carry no regex `pattern` keyword
|
|
160
|
+
|
|
161
|
+
Every field in the seven wrapped built-in tool schemas is a plain
|
|
162
|
+
`Type.String({ description })` / `Type.Number` / etc. No field declares a
|
|
163
|
+
JSON-schema `pattern` (regex) keyword. The literal string `pattern` appears only
|
|
164
|
+
as a *field name* (grep's and find's search-pattern parameter) and in
|
|
165
|
+
descriptions.
|
|
166
|
+
|
|
167
|
+
- `packages/coding-agent/src/core/tools/grep.ts:25` — `pattern: Type.String({
|
|
168
|
+
description: "Search pattern (regex or literal string)" })` — a plain string
|
|
169
|
+
field, no schema-level `pattern` keyword.
|
|
170
|
+
- `packages/coding-agent/src/core/tools/find.ts:21-23` — `pattern: Type.String({
|
|
171
|
+
description: ... })` — likewise a glob string, no regex keyword.
|
|
172
|
+
- The remaining tool files (`read.ts`, `write.ts`, `edit.ts`, `bash.ts`,
|
|
173
|
+
`ls.ts`) declare only plain-typed fields.
|
|
174
|
+
|
|
175
|
+
**Why it matters:** (a) it justifies dropping the schema-anchor poisoning defense
|
|
176
|
+
(there is no regex schema to sanitize among built-ins). (b) It justifies the
|
|
177
|
+
anchor-strip exemption for `grep.pattern`: because `grep`'s pattern *is* a real
|
|
178
|
+
regex field, a leading `^` / trailing `$` there may be intentional syntax and is
|
|
179
|
+
indistinguishable from bled anchors, so we never strip it.
|
|
180
|
+
|
|
181
|
+
### Claim 9 — `tool_result` is global, replaceable, and handler-composed
|
|
182
|
+
|
|
183
|
+
After any built-in or custom tool finishes, the coding-agent session emits a
|
|
184
|
+
global `tool_result` event containing `toolName`, `toolCallId`, validated
|
|
185
|
+
`input`, `content`, `details`, and `isError`. A handler may replace `content`,
|
|
186
|
+
`details`, or `isError`. The runner threads each replacement through the event
|
|
187
|
+
seen by later handlers and returns the final fields to the agent loop.
|
|
188
|
+
|
|
189
|
+
- Installed `dist/core/agent-session.js:211-238` — `afterToolCall` emits the
|
|
190
|
+
complete result event and returns the hook's replacement, preserving the
|
|
191
|
+
original error flag when no replacement is supplied.
|
|
192
|
+
- Installed `dist/core/extensions/runner.js:600-641` — `emitToolResult` copies
|
|
193
|
+
the event, visits handlers in registration order, writes each returned field
|
|
194
|
+
into `currentEvent`, and returns the composed final result.
|
|
195
|
+
- `test/upstream-drift.test.ts` — real-loop tripwires cover a successful custom
|
|
196
|
+
tool, a throwing custom tool, replacement content/details/error state, and
|
|
197
|
+
two-handler composition.
|
|
198
|
+
|
|
199
|
+
**Why it matters:** repair input is prepared before a call ID exists. The layer
|
|
200
|
+
queues value-free feedback, binds it at post-validation `tool_call`, then uses
|
|
201
|
+
`tool_result` to attach one note to success or error output. Grammar-promoted
|
|
202
|
+
custom calls already have IDs and can be associated directly, without wrapping
|
|
203
|
+
another extension's executor.
|
|
204
|
+
|
|
205
|
+
### Package/runtime assumptions
|
|
206
|
+
|
|
207
|
+
The published package targets Node 22+ and compiled ESM. `typebox` is a runtime
|
|
208
|
+
dependency; pi coding-agent and pi-tui are peers so the host supplies its live
|
|
209
|
+
integration/UI runtime. `pnpm run test:package` builds and packs the artifact,
|
|
210
|
+
installs it into a clean temporary project, imports `.`, `/core`, `/pi`, and
|
|
211
|
+
`/grammar`, runs the APIs, and typechecks the consumer fixture. This is the
|
|
212
|
+
tripwire for missing JavaScript, declarations, source maps, or stale exports.
|
|
213
|
+
|
|
214
|
+
## Re-verification checklist (run on any pi minor bump)
|
|
215
|
+
|
|
216
|
+
The version canary in `test/upstream-drift.test.ts` fails when installed
|
|
217
|
+
`pi-coding-agent` changes minor version (`VERIFIED_PI_VERSION`). When it does,
|
|
218
|
+
work through this list and update the citations/date above:
|
|
219
|
+
|
|
220
|
+
1. **Loop ordering (Claim 1):** confirm `prepareArguments` still runs before
|
|
221
|
+
`validateToolArguments`, which still runs before the `tool_call`/`beforeToolCall`
|
|
222
|
+
hook. Re-read `packages/agent/src/agent-loop.ts` `prepareToolCall`.
|
|
223
|
+
2. **Validation throws (Claim 2):** confirm `validateToolArguments` still does
|
|
224
|
+
`Value.Convert` then `Check` then `throw`. Re-read
|
|
225
|
+
`packages/ai/src/utils/validation.ts`.
|
|
226
|
+
3. **`tool_call` propagation (Claim 3):** confirm the event's `input` is still the
|
|
227
|
+
executed `validatedArgs` reference and that the dispatcher still reads only
|
|
228
|
+
`.block`. If pi adds an args-return channel to `tool_call`, revisit the
|
|
229
|
+
founding decision.
|
|
230
|
+
4. **`message_end` in-place replacement (Claim 4):** confirm replacement still
|
|
231
|
+
mutates the loop's message object in place, listeners are still awaited, the
|
|
232
|
+
role guard still holds, and toolCalls are still filtered after the stream
|
|
233
|
+
returns (so recovery still executes same-turn).
|
|
234
|
+
5. **`before_provider_request` (Claim 5):** confirm it still propagates the
|
|
235
|
+
returned payload.
|
|
236
|
+
6. **`stopReason` terminality (Claims 6, 7):** confirm `"error"`/`"aborted"` are
|
|
237
|
+
still terminal and `"length"` still fails all tool calls.
|
|
238
|
+
7. **Built-in schema shape (Claim 8):** re-run the schema-shape snapshot test and
|
|
239
|
+
confirm no wrapped built-in gained a regex `pattern` keyword. If any did,
|
|
240
|
+
revisit the `grep.pattern` exemption and the dropped schema-anchor defense.
|
|
241
|
+
8. Update `VERIFIED_PI_VERSION`, the verification date, and the commit hash above,
|
|
242
|
+
then commit the refreshed snapshot fixture deliberately.
|
|
243
|
+
9. **`tool_result` lifecycle (Claim 9):** confirm built-in and custom success/error
|
|
244
|
+
results still reach the global hook, replacements still reach conversation
|
|
245
|
+
messages, and handlers still compose in registration order.
|
|
246
|
+
10. **Package/runtime:** run `pnpm run test:package`; re-check Node engines,
|
|
247
|
+
peer/runtime dependencies, every `exports` target, and the pi extension path.
|