pi-condense 2.11.5 → 2.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +22 -0
- package/PRUNING.md +50 -32
- package/README.md +2 -0
- package/package.json +4 -4
- package/src/commands.test.ts +46 -0
- package/src/commands.ts +4 -4
- package/src/error-purge.test.ts +91 -1
- package/src/error-purge.transport.test.ts +4 -3
- package/src/error-purge.ts +48 -45
- package/src/frontier.test.ts +1 -0
- package/src/indexer.test.ts +28 -0
- package/src/indexer.ts +4 -0
- package/src/pruner.test.ts +112 -0
- package/src/pruner.ts +9 -8
- package/src/query-tool.test.ts +28 -0
- package/src/reload-rearm.integration.test.ts +2 -8
- package/src/summarizer-wiring.test.ts +138 -16
- package/src/summarizer.test.ts +29 -21
- package/src/summarizer.ts +41 -35
- package/src/supersede.test.ts +73 -3
- package/src/supersede.ts +37 -9
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,28 @@ Published to npm as [`pi-condense`](https://www.npmjs.com/package/pi-condense) (
|
|
|
7
7
|
Pushing a `vX.Y.Z` tag triggers `.github/workflows/release.yml`, which runs the tests and
|
|
8
8
|
publishes via OIDC trusted publishing. See `.agents/skills/release/SKILL.md`.
|
|
9
9
|
|
|
10
|
+
## [2.13.0] - 2026-10-20
|
|
11
|
+
|
|
12
|
+
### Changed
|
|
13
|
+
|
|
14
|
+
- Argument purge shrinks long string arguments of indexed, summarized calls in the same render that stubs their results; `context_tree_query` recovers the stored arguments and result. Dedup aliases keep their own arguments because only canonical arguments are indexed. The existing `purgeErrors` toggle and size gate cover both summarized and errored calls; cooldown remains error-only.
|
|
15
|
+
|
|
16
|
+
### Fixed
|
|
17
|
+
|
|
18
|
+
- Protected reads now supersede earlier reads of the same normalized path only when their delivered line ranges cover the earlier content. Disjoint chunks and tails outside a capped full read stay verbatim; identical capped reads still supersede the older copy. Supersession keeps its existing cache-rewrite cadence.
|
|
19
|
+
|
|
20
|
+
## [2.12.0] - 2026-10-17
|
|
21
|
+
|
|
22
|
+
### Fixed
|
|
23
|
+
|
|
24
|
+
- Virtual (router-backed) models work as the summarizer - via `summarizerModel: "default"`, an explicit virtual `provider/id`, or as the outage-fallback session model. Summarizer calls now go through `ctx.modelRegistry.streamSimple()` instead of pi-ai's provider table, which had no `pi-virtual` entry and threw `No API provider registered for api: pi-virtual`. (#20)
|
|
25
|
+
- A summarizer stream that ignores the abort signal (a virtual router whose `route()` never resolves) no longer wedges the flush: the idle and ceiling timers now end the wait themselves. (#20)
|
|
26
|
+
|
|
27
|
+
### Changed
|
|
28
|
+
|
|
29
|
+
- `summarizerThinking` applies on every provider; it was sent as an OpenAI-specific `reasoningEffort` option that Anthropic-style adapters ignored. `default` and `off` still send no reasoning option, which on Anthropic means `thinkingEnabled: false` as before. (#20)
|
|
30
|
+
- Minimum supported host is pi 1.3.0; dev dependencies on `@earendil-works/pi-ai`, `pi-coding-agent`, and `pi-tui` move from `^0.83.0` to `^1.3.0`. (#20)
|
|
31
|
+
|
|
10
32
|
## [2.11.5] - 2026-10-08
|
|
11
33
|
|
|
12
34
|
### Changed
|
package/PRUNING.md
CHANGED
|
@@ -30,7 +30,7 @@
|
|
|
30
30
|
10. [Chain Compression](#chain-compression)
|
|
31
31
|
- [Protected-output relocation](#protected-output-relocation)
|
|
32
32
|
11. [Occurrence Identity](#occurrence-identity)
|
|
33
|
-
12. [
|
|
33
|
+
12. [Argument Purge](#argument-purge)
|
|
34
34
|
13. [Orphan Sweep](#orphan-sweep)
|
|
35
35
|
14. [Diagnostics](#diagnostics)
|
|
36
36
|
15. [Why Summarization Works: Research Evidence](#why-summarization-works-research-evidence)
|
|
@@ -184,7 +184,7 @@ graph TB
|
|
|
184
184
|
|
|
185
185
|
**Key points:**
|
|
186
186
|
|
|
187
|
-
- The `AssistantMessage` tool-call blocks are **kept
|
|
187
|
+
- The `AssistantMessage` tool-call blocks are **kept**, preserving their ids and result pairing; eligible long argument strings shrink through [Argument Purge](#argument-purge).
|
|
188
188
|
- `ToolResultMessage` entries for summarized tool calls are **replaced with a small stub** (`[Summarized in pruner summary, ref \`tN\`. Use context_tree_query to retrieve full output.]`) carrying`role: "toolResult"`, the original`toolCallId`/`toolName`/`timestamp`, and`isError: false`. The stub preserves role alternation, so pi-ai's`transformMessages.insertSyntheticToolResults` no longer injects a synthetic `{ isError: true, "No result provided" }` for the (no-longer-)orphaned tool call. See [Stub-replace instead of delete](#stub-replace-instead-of-delete).
|
|
189
189
|
- Every pruned tool call is also copied into the pruner's runtime/session index with its `toolCallId`, tool name, args, status, turn index, timestamp, and full `resultText`.
|
|
190
190
|
- A summary message is injected as a `"steer"` (`pi.sendMessage` runtime path) or appended directly via `sessionManager.appendCustomMessageEntry` (session path, used when Pi may already be shutting down). Both deliver before the next LLM call.
|
|
@@ -283,7 +283,7 @@ So after pruning, the model is working with a **two-layer memory**:
|
|
|
283
283
|
|
|
284
284
|
| Part of old turn | After pruning | Why |
|
|
285
285
|
|---|---|---|
|
|
286
|
-
| Assistant tool-call block | **Kept in context** | Preserves
|
|
286
|
+
| Assistant tool-call block | **Kept in context; eligible long argument strings shrink** | Preserves `toolCallId` anchors and result pairing; indexed arguments remain recoverable through `context_tree_query` (dedup aliases keep their own arguments) |
|
|
287
287
|
| Tool result message | **Replaced by a short stub in active context** | Saves tokens (typical 50–100× reduction per call) while keeping the `toolCallId` anchor and giving the model an explicit `context_tree_query` breadcrumb. The stub keeps `role: "toolResult"` and `isError: false` so role alternation stays intact |
|
|
288
288
|
| Summary message | **Added to context** | Gives the model a compact description of what happened |
|
|
289
289
|
| Indexed tool-call record | **Stored in pruner index** (`context-prune-index` session entry) | Lets the model re-open the original raw output later via `context_tree_query` |
|
|
@@ -584,8 +584,8 @@ graph LR
|
|
|
584
584
|
captured batches (from turn_end or session scan)
|
|
585
585
|
│
|
|
586
586
|
├─ 1. Protected-tools/paths filter (capture-time, see below)
|
|
587
|
-
│
|
|
588
|
-
│
|
|
587
|
+
│ protected reads stay verbatim unless a later same-path read covers their
|
|
588
|
+
│ delivered lines; stubbing waits for a shared rewrite or cold cache (see below)
|
|
589
589
|
│ tool calls whose toolName is in protectedTools, OR whose args.path
|
|
590
590
|
│ matches any protectedPaths glob, never enter the batch
|
|
591
591
|
│
|
|
@@ -633,7 +633,7 @@ Implementation: `src/pruner.ts` `pruneMessages(messages, indexer)` returns `{ me
|
|
|
633
633
|
|
|
634
634
|
### Protected tools & paths
|
|
635
635
|
|
|
636
|
-
A tool call is protected if **either** its `toolName` is in `protectedTools` **or** its `args.path` (string) matches any glob in `protectedPaths`. Protected calls are filtered out **at capture time** - they never enter the `pendingBatches` queue, so their raw `ToolResultMessage` stays verbatim in future LLM context until a
|
|
636
|
+
A tool call is protected if **either** its `toolName` is in `protectedTools` **or** its `args.path` (string) matches any glob in `protectedPaths`. Protected calls are filtered out **at capture time** - they never enter the `pendingBatches` queue, so their raw `ToolResultMessage` stays verbatim in future LLM context until a later read of the same path covers its delivered lines and supersedes it (below).
|
|
637
637
|
|
|
638
638
|
**`protectedTools: string[]`** (default `[]`) - allowlist of tool names. Covers tools whose output is a small handle that must be reused byte-for-byte (e.g. a session-id) or planning tools like `todowrite` / `todoread`.
|
|
639
639
|
|
|
@@ -643,9 +643,30 @@ Glob contract: full-path match against the raw `args.path` string with `\` norma
|
|
|
643
643
|
|
|
644
644
|
**Render-time re-check:** stub replacement runs in-flight on every turn (`pruneMessages`). If a tool call's persisted `args` now satisfy `isProtected` (e.g. a pattern was added mid-session), the stub is skipped and the raw result is left verbatim - this repairs already-summarized records in existing sessions with no schema change. Declared limitation: records inside already-compressed chains (`context-prune-chain` entries) are NOT repaired - their `protectedToolCallIds` set is fixed at compression time (forward-only). Dedup-alias edge: an alias resolving to an unprotected original stays stubbed.
|
|
645
645
|
|
|
646
|
-
**Supersession (phase 1b, `src/supersede.ts`):** protected calls are never indexed, so content-hash dedup never sees them.
|
|
646
|
+
**Supersession (phase 1b, `src/supersede.ts`):** protected calls are never indexed, so content-hash dedup never sees them. A paired occurrence participates only when `isProtected(toolName, args)` is true and `typeof args.path === "string"`. Occurrences group by normalized `args.path` (backslash -> slash). An earlier occurrence is a supersession candidate only when **any later occurrence** of that path covers its delivered line range, not just when the last occurrence covers it. Different paths never interact; disjoint chunks stay verbatim.
|
|
647
647
|
|
|
648
|
-
**
|
|
648
|
+
**Requested and delivered ranges:** numeric `offset` normalizes to `max(1, offset)`; absent, `null`, or non-numeric offsets start at line 1. Numeric `limit > 0` requests `[start, start + limit - 1]`; absent, `null`, or non-numeric limits give `[start, inf)`. A `limit <= 0` delivers no lines: it supersedes nothing and is covered by any later nonempty read of the path. A read that returns pi's oversized-first-line warning instead of content delivers no lines, so it covers nothing. Otherwise, without a cap footer, the delivered range is the requested range. A pi footer such as `[Showing lines 1-1800 of 3000 (50.0KB limit). Use offset=1801 to continue.]` bounds the delivered end at 1800. The footer parser also accepts the line-cap form without the parenthetical and any `(<size> limit)` form. This bound applies to both earlier and later reads: a capped full read never covers a tail it did not deliver. Coverage requires the later delivered start to be at or below the earlier start and its end at or above the earlier end.
|
|
649
|
+
|
|
650
|
+
For a file of 3000 lines, "capped" means pi truncated the output:
|
|
651
|
+
|
|
652
|
+
| Sequence on one path | Superseded |
|
|
653
|
+
|---|---|
|
|
654
|
+
| full, full | first |
|
|
655
|
+
| chunk `1-2000`, chunk `2001-` | none |
|
|
656
|
+
| chunk `1-2000`, full capped at line 2000 | chunk (delivered ranges equal) |
|
|
657
|
+
| chunk `2001-3000`, full capped at line 2000 | none (the later read never delivered the tail) |
|
|
658
|
+
| full, chunk | none |
|
|
659
|
+
| chunk 1, chunk 2, chunk 1 again | first chunk 1 |
|
|
660
|
+
| full capped at line 2000, full capped again | first (same delivered lines) |
|
|
661
|
+
| `limit: 2500` byte-capped at line 1800, full capped at line 2000 | first (`[1, 1800]` is inside `[1, 2000]`) |
|
|
662
|
+
| full capped at line 2000, `limit: 2500` byte-capped at line 1800 | none |
|
|
663
|
+
| `offset: 0, limit: 100`, full | first (offset 0 is line 1) |
|
|
664
|
+
| `offset: null` no limit, chunk `1-50` | none (`[1, inf)` is not covered by `[1, 50]`) |
|
|
665
|
+
| chunk `1-50`, `offset: null, limit: 50` | first (`[1, 50]` covers `[1, 50]`) |
|
|
666
|
+
|
|
667
|
+
Once activated by the cadence gate below, a candidate becomes the one-line stub `[Superseded: <path> was read again later in this conversation - see the newer read. Re-read the file if this earlier content is needed.]`. The stub keeps `toolCallId`/`toolName`/`timestamp`, so pi-ai's orphan repair never fires; the assistant `toolCall` block is untouched. A call without a paired result never participates. Recovery is "re-read the file" - no `t<N>` ref. Results pair only with an open same-id call in the preceding assistant turn; each assistant turn resets the open set, and any non-assistant/non-toolResult message clears it. A paired result consumes its call, so reused provider ids never pair through a global per-id cursor.
|
|
668
|
+
|
|
669
|
+
**Cadence (prompt-cache economics):** a superseded read is stubbed only when the pruner is already rewriting at or before its position - `state.floor` is the earliest result timestamp phase 1 will stub on the next render (indexed batches, plus dedup aliases registered in the pre-flush pass - those are stubbed by phase 1 whatever their batch's outcome; a `skipped-trivial`/`skipped-oversized` batch's own calls set none) or the `startUserTimestamp` of a chain compressed this turn - or on a guaranteed-cold event: `session_start`, `session_tree`, `model_select`, `session_compact`, `thinking_level_select` (`state.floor = 0`, activate all). Activation is session-sticky (`state.activated`, an in-memory `occKey` set cleared on `session_start`/`session_tree`); between those moments a freshly superseded copy stays verbatim on purpose, since a mid-prefix rewrite re-bills the whole tail once. `isProtected` is evaluated live, so a path that stops matching `protectedPaths` drops out of supersession and rejoins the normal pipeline. No config key: supersession is on whenever protection is. Specs: [delivered-range coverage](doc/specs/2026-10-20-multi-stage-context-retention.md#change-2-range-aware-supersession) and [cadence and stub contract](doc/specs/2026-09-07-protected-path-supersede.md).
|
|
649
670
|
|
|
650
671
|
Names and patterns that don't match any captured tool call are silently ignored.
|
|
651
672
|
|
|
@@ -724,7 +745,7 @@ The frontier's `lastAttemptedTurnIndex` uses a session-wide numbering domain: it
|
|
|
724
745
|
- **Tree browser (`/pruner tree`):** interactive, foldable tree of pruned tool calls grouped under their summaries. `Ctrl-O` on a summary node opens the full markdown summary in a bordered overlay.
|
|
725
746
|
- **Configurable summarizer thinking (`summarizerThinking`):** trade summary cost / latency for quality (`off` / `minimal` / `low` / `medium` / `high` / `xhigh`). `default` omits the option entirely so the provider chooses.
|
|
726
747
|
- **Cumulative stats:** `context-prune-stats` entries track input/output tokens and cost of every summarizer call; full detail surfaces in `/pruner stats`. Cost is also emitted on the `cost:external` pi.events channel for external aggregators (cumulative per session, live only).
|
|
727
|
-
- **Live reclaim ratio:** measured once per `pruneMessages` call via `sizeMessages(messages)` = the sum of `charsOf(message)` (`src/context-metrics.ts`): whole-message JSON chars with each image block priced at a flat 1,600 tokens (6,400 chars) instead of its base64 length, the same estimate the frontier-gap trigger uses. This compares the input array before pruning to the result after. Estimated tokens = chars / 4. The measurement covers all five phases in a single point (stub-replace, supersede,
|
|
748
|
+
- **Live reclaim ratio:** measured once per `pruneMessages` call via `sizeMessages(messages)` = the sum of `charsOf(message)` (`src/context-metrics.ts`): whole-message JSON chars with each image block priced at a flat 1,600 tokens (6,400 chars) instead of its base64 length, the same estimate the frontier-gap trigger uses. This compares the input array before pruning to the result after. Estimated tokens = chars / 4. The measurement covers all five phases in a single point (stub-replace, supersede, argument-purge, chain-range-prune, orphan-sweep); appears on the status line as `│ prune: ON · 92.0k->14.0k (-85%)` once at least one prune has occurred (the leading `│` keeps the segment visually isolated in the shared footer, load-order independent - there is no trailing divider, since the footer's own space-join between segments already provides one).
|
|
728
749
|
- **Live progress for `/pruner now`:** an `aboveEditor` widget shows one row per pending batch with braille spinner, streamed summary-char count, and ✓ / ⚠ status.
|
|
729
750
|
|
|
730
751
|
### Summarizer outage fallback
|
|
@@ -918,7 +939,7 @@ raw messages from session
|
|
|
918
939
|
│
|
|
919
940
|
├─ [1] tool-result stub-replace (per-batch; existing)
|
|
920
941
|
├─ [1b] supersede (older protected reads of a re-read path -> stub; see § Supersession)
|
|
921
|
-
├─ [2]
|
|
942
|
+
├─ [2] argument-purge (phase 2)
|
|
922
943
|
├─ [3] chain-range-prune (runs AFTER stubs)
|
|
923
944
|
│ resolve each entry to a positional index range
|
|
924
945
|
│ drop assistant / toolResult / per-batch-summary inside the range
|
|
@@ -1099,45 +1120,42 @@ Short refs (`tN`) always resolve 1:1 - one ref, one record - and are the primary
|
|
|
1099
1120
|
|
|
1100
1121
|
---
|
|
1101
1122
|
|
|
1102
|
-
##
|
|
1123
|
+
## Argument Purge
|
|
1103
1124
|
|
|
1104
|
-
|
|
1125
|
+
An indexed call whose result has been summarized loses its long string arguments in the same render its result is stubbed; `context_tree_query` returns both the arguments and the result.
|
|
1105
1126
|
|
|
1106
|
-
|
|
1127
|
+
Two exceptions stay separate: an errored call waits `purgeErrors.cooldownTurns` assistant turns (default 2), whether or not it was indexed, so recovery through `context_tree_query` is available only when it was also indexed. A dedup alias is never shrunk: the index stores only the canonical call's arguments, and recovery on the shared ref returns those arguments, not the alias's own prompt.
|
|
1128
|
+
|
|
1129
|
+
**Shared gates and shape:** `purgeErrors.enabled` controls both sources. The JSON-stringified `block.arguments` body must be at least `purgeErrors.minArgChars` characters (default 500). Only strings longer than 200 characters (`PURGE_STRING_MIN_CHARS` in `src/error-purge.ts`, not configurable) shrink. Keys, JSON types, array lengths, non-string values, and strings of 200 characters or fewer survive. A qualifying body with no long string stays unchanged. Keeping the shape preserves provider schema validation while leaving short paths and commands readable.
|
|
1130
|
+
|
|
1131
|
+
Both markers carry the original string length; summarized calls also carry the short ref from the result stub:
|
|
1107
1132
|
|
|
1108
1133
|
```
|
|
1109
|
-
|
|
1134
|
+
<purged-args size="30000" ref="t6"/>
|
|
1135
|
+
<purged-errored-args size="30000"/>
|
|
1110
1136
|
```
|
|
1111
1137
|
|
|
1112
|
-
|
|
1113
|
-
|
|
1114
|
-
**What triggers a purge:**
|
|
1138
|
+
**Pairing and source selection:** phase 1 hands phase 2 a map from each stubbed result's resolved occurrence key to its short ref, excluding dedup aliases. Phase 2 pairs results positionally with open same-id tool-call blocks in the preceding assistant turn. Each assistant turn resets the open set, a result consumes its paired call, and any non-assistant/non-toolResult message clears the set. Rewrites target the exact assistant position and block id, so a reused provider id cannot authorize shrinking a sibling call in another turn. Timestamped results use `id@resultTimestamp`; timestamp-less results use the bare id for the handoff lookup, not for selecting a rewrite position.
|
|
1115
1139
|
|
|
1116
|
-
|
|
1117
|
-
- The error occurred at least `purgeErrors.cooldownTurns` assistant turns ago (default 2). The cooldown gives the model 1–2 turns to retry before context is mutated.
|
|
1118
|
-
- The JSON-stringified argument body is at least `purgeErrors.minArgChars` characters long (default 500). Small args are not worth the substitution. A body that qualifies but holds no string over 200 characters is returned unchanged.
|
|
1140
|
+
A result phase 1 stubs has `isError: false`, so its arguments get `<purged-args>` even if the original result errored. Before that stub render, an expired error cooldown can produce `<purged-errored-args>`. Argument purge does not rewrite result content; unsummarized error messages stay visible.
|
|
1119
1141
|
|
|
1120
|
-
**
|
|
1142
|
+
**Cadence and identity:** summarized calls have no separate cooldown. Shrinking coincides with the result-stub rewrite; delaying it would invalidate the prefix again on a later turn. Each context render starts from raw history and reapplies the same transformation: for fixed index and recovery-grace state, the emitted prefix bytes are deterministic across fresh-history renders. The input array is returned by identity only when nothing changes, including an already-shrunk input whose markers fall below the string floor.
|
|
1121
1143
|
|
|
1122
|
-
|
|
1123
|
-
- Non-errored `toolCall` argument bodies.
|
|
1124
|
-
- Argument bodies below `minArgChars`.
|
|
1125
|
-
- Keys, non-string values, array lengths, and strings of 200 characters or fewer inside a purged body.
|
|
1126
|
-
- Anything when `purgeErrors.enabled` is `false`.
|
|
1144
|
+
Protected calls and trivial batches have no phase 1 stub and do not qualify through the summarized source. Calls in the recovery grace window keep both their raw result and arguments until the window expires; stubbing and shrinking then happen together. Dedup alias results still get stubs, but their arguments stay intact. The separate errored source retains its cooldown behavior for calls with `isError: true`.
|
|
1127
1145
|
|
|
1128
|
-
**Transform position:**
|
|
1146
|
+
**Transform position:** argument purge runs in Phase 2, after stub-replace and supersession and before chain range prune.
|
|
1129
1147
|
|
|
1130
1148
|
```
|
|
1131
|
-
[stub-replace] → [supersede] → [
|
|
1149
|
+
[stub-replace] → [supersede] → [argument-purge] → [chain-range-prune] → [orphan-sweep]
|
|
1132
1150
|
```
|
|
1133
1151
|
|
|
1134
|
-
**Config keys:**
|
|
1152
|
+
**Config keys:** persisted names stay `purgeErrors.*`; the settings overlay labels the shared toggle and size gate as argument purge and the cooldown as error-only.
|
|
1135
1153
|
|
|
1136
1154
|
| Key | Default | Description |
|
|
1137
1155
|
|---|---|---|
|
|
1138
|
-
| `purgeErrors.enabled` | `true` |
|
|
1139
|
-
| `purgeErrors.cooldownTurns` | `2` |
|
|
1140
|
-
| `purgeErrors.minArgChars` | `500` | Minimum JSON body size for
|
|
1156
|
+
| `purgeErrors.enabled` | `true` | Toggle for summarized and errored argument purge |
|
|
1157
|
+
| `purgeErrors.cooldownTurns` | `2` | Assistant turns to wait after an error; applies only to the errored source |
|
|
1158
|
+
| `purgeErrors.minArgChars` | `500` | Minimum JSON body size for either source; only strings over 200 chars inside it are replaced |
|
|
1141
1159
|
|
|
1142
1160
|
---
|
|
1143
1161
|
|
package/README.md
CHANGED
|
@@ -126,6 +126,8 @@ These are most informative for long single-chain sessions where Phase 3 (chain c
|
|
|
126
126
|
|
|
127
127
|
Published to npm as [`pi-condense`](https://www.npmjs.com/package/pi-condense).
|
|
128
128
|
|
|
129
|
+
Requires pi >= 1.3.0: the summarizer calls `ctx.modelRegistry.streamSimple()`, which is what lets a virtual (router-backed) model serve as the summarizer. Older hosts are unsupported; before pi 0.86.0 the method does not exist and every summarizer call fails with a `TypeError` notice.
|
|
130
|
+
|
|
129
131
|
**User scope** (all repos under your pi profile):
|
|
130
132
|
|
|
131
133
|
```bash
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-condense",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.13.0",
|
|
4
4
|
"description": "Pi coding-agent extension that summarizes completed tool-call batches, replaces raw outputs with short stubs, compresses closed tool-call chains, and recovers any original on demand via context_tree_query.",
|
|
5
5
|
"author": "Jacek Juraszek",
|
|
6
6
|
"license": "MIT",
|
|
@@ -57,9 +57,9 @@
|
|
|
57
57
|
"@sinclair/typebox": "*"
|
|
58
58
|
},
|
|
59
59
|
"devDependencies": {
|
|
60
|
-
"@earendil-works/pi-ai": "^
|
|
61
|
-
"@earendil-works/pi-coding-agent": "^
|
|
62
|
-
"@earendil-works/pi-tui": "^
|
|
60
|
+
"@earendil-works/pi-ai": "^1.3.0",
|
|
61
|
+
"@earendil-works/pi-coding-agent": "^1.3.0",
|
|
62
|
+
"@earendil-works/pi-tui": "^1.3.0",
|
|
63
63
|
"@sinclair/typebox": "^0.34.52"
|
|
64
64
|
}
|
|
65
65
|
}
|
package/src/commands.test.ts
CHANGED
|
@@ -12,6 +12,8 @@ import { pruneStatusText, setPruneStatusWidget, registerCommands } from "./comma
|
|
|
12
12
|
import { settingsPath } from "./config.js";
|
|
13
13
|
import type { ContextPruneConfig, ContextMetricsSnapshot, SummarizerStats } from "./types.js";
|
|
14
14
|
import { DEFAULT_CONFIG } from "./types.js";
|
|
15
|
+
import { Container, SettingsList } from "@earendil-works/pi-tui";
|
|
16
|
+
import { initTheme } from "@earendil-works/pi-coding-agent";
|
|
15
17
|
|
|
16
18
|
const cfg = (enabled: boolean): ContextPruneConfig => ({ enabled } as ContextPruneConfig);
|
|
17
19
|
const cfgVisible = (enabled: boolean): ContextPruneConfig =>
|
|
@@ -41,6 +43,7 @@ function setupPrunerCommand(overrides: {
|
|
|
41
43
|
let handler: (args: string, ctx: any) => Promise<void>;
|
|
42
44
|
const notifications: { message: string; type?: string }[] = [];
|
|
43
45
|
const flushCalls: any[] = [];
|
|
46
|
+
let overlay: Container & { handleInput(data: string): void };
|
|
44
47
|
|
|
45
48
|
const flushPending =
|
|
46
49
|
overrides.flushPending ??
|
|
@@ -79,6 +82,9 @@ function setupPrunerCommand(overrides: {
|
|
|
79
82
|
notifications.push({ message, type });
|
|
80
83
|
},
|
|
81
84
|
setStatus() {},
|
|
85
|
+
async custom(factory: (...args: any[]) => typeof overlay) {
|
|
86
|
+
overlay = factory(undefined, undefined, undefined, () => {});
|
|
87
|
+
},
|
|
82
88
|
},
|
|
83
89
|
};
|
|
84
90
|
|
|
@@ -87,9 +93,49 @@ function setupPrunerCommand(overrides: {
|
|
|
87
93
|
notifications,
|
|
88
94
|
flushCalls,
|
|
89
95
|
currentConfig,
|
|
96
|
+
get overlay() { return overlay; },
|
|
90
97
|
};
|
|
91
98
|
}
|
|
92
99
|
|
|
100
|
+
describe("/pruner settings argument purge", () => {
|
|
101
|
+
initTheme("dark");
|
|
102
|
+
|
|
103
|
+
it.each([
|
|
104
|
+
["purgeErrorsEnabled", "Argument purge", "Shrink long strings in summarized calls when their results are stubbed in the same render, and in errored calls after the error cooldown, keeping argument shape."],
|
|
105
|
+
["purgeErrorsCooldown", "Error purge cooldown (turns)", "Wait this many turns after a tool error before purging its argument body."],
|
|
106
|
+
["purgeErrorsMinArgChars", "Argument purge min arg chars", "Only purge summarized or errored argument bodies at least this many chars."],
|
|
107
|
+
])("describes %s through the settings overlay", async (id, label, description) => {
|
|
108
|
+
const harness = setupPrunerCommand({ save: async () => {} });
|
|
109
|
+
await harness.run("settings");
|
|
110
|
+
const list = harness.overlay.children.find((child) => child instanceof SettingsList) as SettingsList;
|
|
111
|
+
list.selectItem(id);
|
|
112
|
+
|
|
113
|
+
const rendered = harness.overlay.render(400).join("\n");
|
|
114
|
+
expect(rendered).toContain(label);
|
|
115
|
+
expect(rendered).toContain(description);
|
|
116
|
+
if (id === "purgeErrorsCooldown") expect(rendered).not.toContain("same render");
|
|
117
|
+
});
|
|
118
|
+
|
|
119
|
+
it("persists the shared disabled toggle and minimum using only existing purgeErrors fields", async () => {
|
|
120
|
+
const saved: ContextPruneConfig[] = [];
|
|
121
|
+
const harness = setupPrunerCommand({ save: async (config) => { saved.push(structuredClone(config)); } });
|
|
122
|
+
const initial = structuredClone(harness.currentConfig.value);
|
|
123
|
+
await harness.run("settings");
|
|
124
|
+
const list = harness.overlay.children.find((child) => child instanceof SettingsList) as SettingsList;
|
|
125
|
+
|
|
126
|
+
list.selectItem("purgeErrorsEnabled");
|
|
127
|
+
harness.overlay.handleInput(" ");
|
|
128
|
+
list.selectItem("purgeErrorsMinArgChars");
|
|
129
|
+
harness.overlay.handleInput(" ");
|
|
130
|
+
|
|
131
|
+
expect(saved).toEqual([
|
|
132
|
+
{ ...initial, purgeErrors: { ...initial.purgeErrors, enabled: false } },
|
|
133
|
+
{ ...initial, purgeErrors: { ...initial.purgeErrors, enabled: false, minArgChars: 1000 } },
|
|
134
|
+
]);
|
|
135
|
+
expect(harness.currentConfig.value).toEqual(saved[1]);
|
|
136
|
+
});
|
|
137
|
+
});
|
|
138
|
+
|
|
93
139
|
describe("/pruner now (empty capture)", () => {
|
|
94
140
|
it("invokes flushPending even when nothing is pending, so a flush-metrics entry is recorded", async () => {
|
|
95
141
|
const harness = setupPrunerCommand({ capturePendingBatches: () => [] });
|
package/src/commands.ts
CHANGED
|
@@ -672,10 +672,10 @@ export function registerCommands(
|
|
|
672
672
|
},
|
|
673
673
|
{
|
|
674
674
|
id: "purgeErrorsEnabled",
|
|
675
|
-
label: "
|
|
675
|
+
label: "Argument purge",
|
|
676
676
|
values: ["true", "false"],
|
|
677
677
|
currentValue: String(config.purgeErrors.enabled),
|
|
678
|
-
description: `Shrink long strings
|
|
678
|
+
description: `Shrink long strings in summarized calls when their results are stubbed in the same render, and in errored calls after the error cooldown, keeping argument shape. Currently ${config.purgeErrors.enabled ? "ON" : "OFF"}.`,
|
|
679
679
|
},
|
|
680
680
|
{
|
|
681
681
|
id: "purgeErrorsCooldown",
|
|
@@ -688,12 +688,12 @@ export function registerCommands(
|
|
|
688
688
|
},
|
|
689
689
|
{
|
|
690
690
|
id: "purgeErrorsMinArgChars",
|
|
691
|
-
label: "
|
|
691
|
+
label: "Argument purge min arg chars",
|
|
692
692
|
values: PURGE_MIN_ARG_PRESETS.map((p) => p.value),
|
|
693
693
|
currentValue: PURGE_MIN_ARG_PRESETS.some((p) => p.value === String(config.purgeErrors.minArgChars))
|
|
694
694
|
? String(config.purgeErrors.minArgChars)
|
|
695
695
|
: PURGE_MIN_ARG_PRESETS[1].value,
|
|
696
|
-
description: `Only purge
|
|
696
|
+
description: `Only purge summarized or errored argument bodies at least this many chars. Currently ${config.purgeErrors.minArgChars}.`,
|
|
697
697
|
},
|
|
698
698
|
{
|
|
699
699
|
// Read-only display row. Editing goes through `/pruner protected-tools`
|
package/src/error-purge.test.ts
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
import { describe, expect, it } from "bun:test";
|
|
2
2
|
import { purgeErroredArgs } from "./error-purge.js";
|
|
3
|
+
import { occKey } from "./occurrence-key.js";
|
|
4
|
+
import { expectNoOrphanToolResults } from "./test-support.js";
|
|
3
5
|
import type { ErrorPurgeConfig } from "./types.js";
|
|
4
6
|
|
|
5
7
|
const defaultConfig: ErrorPurgeConfig = {
|
|
@@ -11,7 +13,7 @@ const defaultConfig: ErrorPurgeConfig = {
|
|
|
11
13
|
const LONG = "x".repeat(300);
|
|
12
14
|
const PLACEHOLDER_RE = /^<purged-errored-args size="\d+"\/>$/;
|
|
13
15
|
|
|
14
|
-
function makeAssistant(toolCallId: string, argsObj:
|
|
16
|
+
function makeAssistant(toolCallId: string, argsObj: any, turnN?: number) {
|
|
15
17
|
return {
|
|
16
18
|
role: "assistant",
|
|
17
19
|
content: [
|
|
@@ -271,3 +273,91 @@ describe("purgeErroredArgs", () => {
|
|
|
271
273
|
expect(original).toEqual(snapshot);
|
|
272
274
|
});
|
|
273
275
|
});
|
|
276
|
+
|
|
277
|
+
describe("positional argument purge", () => {
|
|
278
|
+
const marker = '<purged-args size="600" ref="t6"/>';
|
|
279
|
+
const body = "x".repeat(600);
|
|
280
|
+
|
|
281
|
+
it.each([
|
|
282
|
+
["below body threshold", { code: body }, 612, false],
|
|
283
|
+
["at body threshold", { code: body }, 611, true],
|
|
284
|
+
["200-char string", { code: "x".repeat(200) }, 10, false],
|
|
285
|
+
["201-char string", { code: "x".repeat(201) }, 10, true],
|
|
286
|
+
["only short strings", { code: Array(20).fill("x".repeat(100)) }, 500, false],
|
|
287
|
+
["top-level string", body, 10, true],
|
|
288
|
+
])("summarized gates: %s", (_, args, minArgChars, changed) => {
|
|
289
|
+
const messages = [makeAssistant("tc1", args), makeToolResult("tc1", false)];
|
|
290
|
+
expectNoOrphanToolResults(messages);
|
|
291
|
+
const refs = new Map([[occKey("tc1", 2), "t6"]]);
|
|
292
|
+
const result = purgeErroredArgs(messages, { ...defaultConfig, cooldownTurns: 99, minArgChars }, refs);
|
|
293
|
+
expectNoOrphanToolResults(result);
|
|
294
|
+
if (changed) {
|
|
295
|
+
expect(result).not.toBe(messages);
|
|
296
|
+
expect((result[0] as any).content[0].arguments).toEqual(
|
|
297
|
+
typeof args === "string" ? marker : { code: `<purged-args size="${args.code.length}" ref="t6"/>` },
|
|
298
|
+
);
|
|
299
|
+
} else expect(result).toBe(messages);
|
|
300
|
+
expect(purgeErroredArgs(result, { ...defaultConfig, minArgChars }, refs)).toBe(result);
|
|
301
|
+
});
|
|
302
|
+
|
|
303
|
+
it("keeps nested shapes, grammar keys, and own prototype keys without mutation", () => {
|
|
304
|
+
const args = { code: body, nested: [[body, true, null, 42, "short"]], ...JSON.parse(`{"__proto__":"${body}"}`) };
|
|
305
|
+
const snapshot = structuredClone(args);
|
|
306
|
+
const messages = [makeAssistant("tc1", args), makeToolResult("tc1", false)];
|
|
307
|
+
expectNoOrphanToolResults(messages);
|
|
308
|
+
const result = purgeErroredArgs(messages, defaultConfig, new Map([[occKey("tc1", 2), "t6"]]));
|
|
309
|
+
expectNoOrphanToolResults(result);
|
|
310
|
+
const shrunk = (result[0] as any).content[0].arguments;
|
|
311
|
+
expect(Object.keys(shrunk)).toEqual(Object.keys(args));
|
|
312
|
+
expect(shrunk).toEqual({ code: marker, nested: [[marker, true, null, 42, "short"]], ...JSON.parse(`{"__proto__":${JSON.stringify(marker)}}`) });
|
|
313
|
+
expect(args).toEqual(snapshot);
|
|
314
|
+
expect(result[1]).toBe(messages[1]);
|
|
315
|
+
});
|
|
316
|
+
|
|
317
|
+
it.each([false, true])("disabled leaves summarized and error sources intact (error=%s)", (isError) => {
|
|
318
|
+
const messages = [makeAssistant("tc1", { code: body }), makeToolResult("tc1", isError)];
|
|
319
|
+
expectNoOrphanToolResults(messages);
|
|
320
|
+
expect(purgeErroredArgs(messages, { ...defaultConfig, enabled: false, cooldownTurns: 0 }, new Map([[occKey("tc1", 2), "t6"]]))).toBe(messages);
|
|
321
|
+
});
|
|
322
|
+
|
|
323
|
+
it("prefers the summarized marker even when the paired result is errored", () => {
|
|
324
|
+
const messages = [makeAssistant("tc1", { code: body }), makeToolResult("tc1", true)];
|
|
325
|
+
expectNoOrphanToolResults(messages);
|
|
326
|
+
const result = purgeErroredArgs(messages, { ...defaultConfig, cooldownTurns: 0 }, new Map([[occKey("tc1", 2), "t6"]]));
|
|
327
|
+
expectNoOrphanToolResults(result);
|
|
328
|
+
expect((result[0] as any).content[0].arguments.code).toBe(marker);
|
|
329
|
+
});
|
|
330
|
+
|
|
331
|
+
it.each([
|
|
332
|
+
["timestamped summarized", "summarized", true, "paired"],
|
|
333
|
+
["timestamp-less summarized", "summarized", false, "paired"],
|
|
334
|
+
["error only", "error", true, "paired"],
|
|
335
|
+
["summarized after aborted call", "summarized", true, "aborted"],
|
|
336
|
+
["timestamp-less after aborted call", "summarized", false, "aborted"],
|
|
337
|
+
["error after aborted call", "error", true, "aborted"],
|
|
338
|
+
["summarized after user barrier", "summarized", true, "barrier"],
|
|
339
|
+
["timestamp-less after user barrier", "summarized", false, "barrier"],
|
|
340
|
+
["error after user barrier", "error", true, "barrier"],
|
|
341
|
+
])("isolates reused ids: %s", (_, source, timestamped, earlier) => {
|
|
342
|
+
const first = makeAssistant("reused", { code: body });
|
|
343
|
+
const lastResult: any = { ...makeToolResult("reused", source === "error"), timestamp: 4 };
|
|
344
|
+
if (!timestamped) delete lastResult.timestamp;
|
|
345
|
+
const messages: any[] = [first];
|
|
346
|
+
if (earlier === "paired") {
|
|
347
|
+
// A timestamp-less sibling must not resolve the later timestamped key (or vice versa).
|
|
348
|
+
const sibling: any = makeToolResult("reused", false);
|
|
349
|
+
if (timestamped) delete sibling.timestamp;
|
|
350
|
+
messages.push(sibling);
|
|
351
|
+
}
|
|
352
|
+
if (earlier === "barrier") messages.push({ role: "user", content: "continue" });
|
|
353
|
+
messages.push(makeAssistant("reused", { code: body }, 3), lastResult);
|
|
354
|
+
expectNoOrphanToolResults(messages);
|
|
355
|
+
const refs = source === "summarized" ? new Map([[occKey("reused", timestamped ? 4 : undefined), "t6"]]) : new Map<string, string>();
|
|
356
|
+
const result = purgeErroredArgs(messages, { ...defaultConfig, cooldownTurns: 0 }, refs);
|
|
357
|
+
expectNoOrphanToolResults(result);
|
|
358
|
+
expect(result[0]).toBe(first);
|
|
359
|
+
expect((result[result.length - 2] as any).content[0].arguments.code).toBe(
|
|
360
|
+
source === "summarized" ? marker : '<purged-errored-args size="600"/>',
|
|
361
|
+
);
|
|
362
|
+
});
|
|
363
|
+
});
|
|
@@ -35,6 +35,7 @@ const meta = { api: "openai-completions", provider: "openai", model: "grammar-te
|
|
|
35
35
|
|
|
36
36
|
function history(codeArgs: Record<string, unknown>) {
|
|
37
37
|
return [
|
|
38
|
+
{ role: "system", content: "", toolsAdded: [codemode], timestamp: 0 },
|
|
38
39
|
{ role: "user", content: "run it", timestamp: 1 },
|
|
39
40
|
{ role: "assistant", content: [{ type: "toolCall", id: "tc1", name: "codemode", arguments: codeArgs }], ...meta, stopReason: "toolUse", timestamp: 2 },
|
|
40
41
|
{ role: "toolResult", toolCallId: "tc1", toolName: "codemode", content: [{ type: "text", text: "Error: boom" }], isError: true, timestamp: 3 },
|
|
@@ -60,7 +61,7 @@ describe("error purge x pi-ai grammar tools (#19)", () => {
|
|
|
60
61
|
|
|
61
62
|
it("purged codemode args pass getGrammarToolInput (reporter repro)", () => {
|
|
62
63
|
const purged = purgeErroredArgs(history({ code: "x".repeat(1000) }), config);
|
|
63
|
-
const args = purged[
|
|
64
|
+
const args = purged[2].content[0].arguments;
|
|
64
65
|
expect(typeof args.code).toBe("string");
|
|
65
66
|
expect(Object.keys(args)).toEqual(["code"]);
|
|
66
67
|
expect(JSON.stringify(args).length).toBeLessThan(500);
|
|
@@ -69,7 +70,7 @@ describe("error purge x pi-ai grammar tools (#19)", () => {
|
|
|
69
70
|
|
|
70
71
|
it("request building succeeds on purged history; failure is the connection, not the grammar", async () => {
|
|
71
72
|
const purged = purgeErroredArgs(history({ code: "x".repeat(1000) }), config);
|
|
72
|
-
expect(purged[
|
|
73
|
+
expect(purged[2].content[0].arguments.code).toMatch(/^<purged-errored-args size=/);
|
|
73
74
|
const { result, calls } = await request(purged);
|
|
74
75
|
expect(result.stopReason).toBe("error");
|
|
75
76
|
// The OpenAI SDK wraps fetch rejections in APIConnectionError and drops the cause message.
|
|
@@ -80,7 +81,7 @@ describe("error purge x pi-ai grammar tools (#19)", () => {
|
|
|
80
81
|
|
|
81
82
|
it("negative control: the old { _purged } shape throws the grammar error before any request", async () => {
|
|
82
83
|
const legacy = history({ code: "x".repeat(1000) });
|
|
83
|
-
legacy[
|
|
84
|
+
legacy[2].content[0].arguments = { _purged: '<purged-errored-args size="1012"/>' };
|
|
84
85
|
const { result, calls } = await request(legacy);
|
|
85
86
|
expect(result.stopReason).toBe("error");
|
|
86
87
|
expect(result.errorMessage).toMatch(/Grammar tool call "codemode" requires argument "code" to be a string/);
|