pi-supernova 0.5.0 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. package/README.md +97 -11
  2. package/docs/CHANGELOG.md +150 -0
  3. package/docs/TOKEN_COSTS.md +71 -29
  4. package/index.js +126 -82
  5. package/package.json +2 -2
  6. package/src/adapters/bash.js +73 -0
  7. package/src/adapters/edit.js +249 -0
  8. package/src/adapters/errors.js +31 -0
  9. package/src/adapters/index.js +31 -0
  10. package/src/adapters/list.js +102 -0
  11. package/src/adapters/read.js +805 -0
  12. package/src/adapters/refs.js +41 -0
  13. package/src/adapters/write.js +96 -0
  14. package/src/bridge/catalog.js +30 -220
  15. package/src/bridge/host-bridge.js +142 -1032
  16. package/src/bridge/invoke.js +35 -0
  17. package/src/bridge/native-tools.js +1 -188
  18. package/src/context/evidence.js +142 -70
  19. package/src/context/fuzzy.js +61 -22
  20. package/src/context/ledger.js +43 -24
  21. package/src/context/outline.js +26 -12
  22. package/src/context/repo-index.js +242 -71
  23. package/src/context/search.js +189 -56
  24. package/src/context/snap.js +306 -150
  25. package/src/context/spans.js +2 -1
  26. package/src/context/surface.js +29 -14
  27. package/src/contract/bash.js +31 -0
  28. package/src/contract/edit.js +95 -0
  29. package/src/contract/read.js +220 -0
  30. package/src/fs/check.js +19 -7
  31. package/src/fs/diff.js +18 -7
  32. package/src/fs/json-read.js +66 -35
  33. package/src/fs/patch.js +97 -51
  34. package/src/fs/source-window.js +82 -0
  35. package/src/fs/text-ops.js +512 -0
  36. package/src/fs/vfs.js +289 -162
  37. package/src/fs/workspace.js +122 -105
  38. package/src/output/bottleneck.js +211 -107
  39. package/src/output/format.js +112 -63
  40. package/src/runtime/guest-deny-imports.js +34 -0
  41. package/src/runtime/guest-worker.js +306 -213
  42. package/src/runtime/parallel.js +99 -63
  43. package/src/runtime/program-batch.js +189 -69
  44. package/src/runtime/program-file.js +6 -3
  45. package/src/runtime/reference.js +13 -12
  46. package/src/runtime/runtime.js +327 -176
  47. package/src/shared/decode.js +61 -27
  48. package/src/ui/omp-frame.js +70 -46
  49. package/src/ui/render-measure.js +51 -29
  50. package/src/ui/render.js +242 -146
package/README.md CHANGED
@@ -12,6 +12,38 @@ Ordinary JavaScript control flow remains available; the guest command bindings
12
12
  are only `read`, `edit`, `write`, and `bash`. Supernova supplies retrieval,
13
13
  transactional file operations, batching, bounded results and the grouped nova UI.
14
14
 
15
+ ## Unreleased
16
+
17
+ - **`parallel: true` on `programs`:** independent entries run at once (up to 8),
18
+ keep result order, and do not stop siblings on failure. Sequential is still
19
+ the default.
20
+ - **JSON `.length`:** `read({path, json:".items.length"})` returns the array
21
+ length without dumping the array.
22
+ - Prefer `edit` for a file you already read; `write` still replaces the file.
23
+
24
+ ## What is new in 0.7.0
25
+
26
+ - **Faster batches:** guest workers pipeline their successor, so sequential
27
+ programs run about twice as fast with no token cost.
28
+ - **Hardened guest:** `process.kill` is sealed out; patch hunks report
29
+ relocation and ambiguous hunks fail instead of misapplying.
30
+ - **Leaner receipts:** multi-edit output caps at 32 matches with exact totals,
31
+ write receipts go workspace-relative, evidence/outline reads go compact.
32
+
33
+ ## What is new in 0.6.0
34
+
35
+ - **Shared batch input:** supply top-level `data` once; each program gets an
36
+ independent copy unless it supplies its own replacement data.
37
+ - **Conflict protection:** byte snapshots survive partial reads and body-cache
38
+ eviction; receipt generation cannot silently rebase a pending write.
39
+ - **Read fidelity:** staged declarations remain discoverable in large/new files,
40
+ line windows preserve source endings, and `complete` always means the whole file.
41
+ - **Explicit failures:** incompatible read modes, budget-limited matches, captured
42
+ overrides and conflicting new-file aliases no longer silently change outcomes.
43
+
44
+ See the [changelog](https://github.com/AdityaVG13/pi-stack/blob/main/packages/pi-supernova/docs/CHANGELOG.md)
45
+ and [token measurements](https://github.com/AdityaVG13/pi-stack/blob/main/packages/pi-supernova/docs/TOKEN_COSTS.md).
46
+
15
47
  ## Install and update
16
48
 
17
49
  Install the published package in your host:
@@ -29,9 +61,19 @@ pi install /path/to/pi-stack/packages/pi-supernova
29
61
 
30
62
  Git pushes do not update npm installations. Publish the new npm version first;
31
63
  then reinstall it in the host. Reinstall explicitly when an existing version
32
- range excludes the new minor version (for example, `^0.2.0` excludes `0.3.0`).
64
+ range excludes the new minor version (`^0.6.0` excludes `0.7.0`). After 0.7.0 is
65
+ published, pin that release with:
66
+
67
+ ```bash
68
+ pi install npm:pi-supernova@0.7.0
69
+ omp install npm:pi-supernova@0.7.0
70
+ ```
33
71
 
34
- Both package manifests use `index.js`. The old `src/bridge/pi-extension.ts` path
72
+ In Pi, `pi list` shows the configured package sources. A local path uses that
73
+ checkout directly; an npm source uses the installed npm copy. Do not assume that
74
+ pushing a checkout or running `/reload` updates the copy executing in your host.
75
+
76
+ Both host manifests use `index.js`. The old `src/bridge/pi-extension.ts` path
35
77
  remains a compatibility entrypoint but no longer imports Pi tool factories.
36
78
  After updating JavaScript sources, fully exit Pi and resume in a new process.
37
79
  Pi 0.85.1 can retain native ESM modules across `/reload`, even after its extension
@@ -54,7 +96,7 @@ settings. The runtime does not silently rewrite your tool policy.
54
96
  | `read` | `read({query,resolve:true})`; same view as `read("symbol")`: status, path, line, lines, text, complete |
55
97
  | `read` | `read({query,evidence:true})`; ranked evidence with provenance; optional `path` scopes discovery |
56
98
  | `read` | `read({path,outline:true})`; structural declarations |
57
- | `read` | `read([path1,path2])`; up to 64 paths, ordered values with labelled individual failures |
99
+ | `read` | `read([path1,path2])`; up to 64 paths, ordered values; rejects if any path fails |
58
100
  | `edit` | `edit(path,oldText,newText)`, `edit({path,edits:[{oldText,newText}]})`; unique in the file |
59
101
  | `edit` | `edit(view,text)` CAS-replaces that span; `edit(view,old,new)` is unique inside it |
60
102
  | `edit` | `edit({path,patch})`; unified patch application |
@@ -75,7 +117,8 @@ Source questions locate a declaration in one command. An exact
75
117
  declaration match uses one bounded direct ripgrep search, without a prerequisite
76
118
  file listing, persistent index, embeddings or summarization. A transient filename
77
119
  listing is a fallback for unmatched content or unresolved bare filenames. Natural-language
78
- questions reuse lexical stemming. Ripgrep must be available on PATH.
120
+ questions reuse lexical stemming. Source questions and focused `about` reads
121
+ accept at most 16 keywords. Ripgrep must be available on PATH.
79
122
 
80
123
  `read(path)` stays raw text. `read("symbol")` is the same view as
81
124
  `read({query, resolve:true})` — not the file, not a path/range header:
@@ -90,7 +133,13 @@ The view contains `status`, `path`, the matching `line`, span `lines`, unchanged
90
133
  `text`, `complete`, and `nextOffset` when a budget clip continues. A declaration
91
134
  snap is that span (`complete` is false unless the span is the whole file). Uncertain
92
135
  results report `ambiguous`, `not_found` or `incomplete` with no selected path.
93
- Use `{path: directory, about: question}` to narrow the scope.
136
+ Use `{path: directory, about: question}` to narrow the scope. Scoping a query
137
+ does not relabel a selected span as a complete file. Newly staged files and large
138
+ staged source participate in discovery before commit. Raw offset/limit windows
139
+ preserve LF/CRLF endings and the final newline; focused views add line labels.
140
+ A matching window too large for the output budget is reported as budget-limited,
141
+ not as an absent match. Do not combine incompatible modes such as `outline:true`
142
+ and `evidence:true`.
94
143
 
95
144
  Ordinary reads stay self-contained. Outlines and graph evidence remain explicit
96
145
  options, not mandatory stages of source resolution. Ordinary calls also get:
@@ -122,6 +171,14 @@ For intentionally writing literal marker documentation only, opt in with
122
171
  `write({path,content,allowReadArtifacts:true})`. This is a data-loss guard, not
123
172
  full dataflow tracking or a security sandbox.
124
173
 
174
+ Read/modify/write conflict checks retain a signature of the actual disk bytes,
175
+ including for partial and large-file reads. A fresh explicit text read refreshes
176
+ that observation; internal receipt reads and body-cache eviction do not. Commits
177
+ reject changed content and conflicting symlink aliases, including new file paths.
178
+ These checks do not provide a cross-process lock or make shell/import mutations
179
+ transactional. Extensionless filenames also support `complete:true`, for example
180
+ `read({path:"LICENSE",complete:true})`.
181
+
125
182
  Explicit read arrays reject missing/failed paths. For typed partial outcomes use
126
183
  `Promise.allSettled(paths.map(path => read(path)))`. Successful arrays remain arrays.
127
184
  For literal file content or scripts, prefer the optional tool-level `data` parameter:
@@ -136,8 +193,9 @@ For literal file content or scripts, prefer the optional tool-level `data` param
136
193
  }
137
194
  ```
138
195
 
139
- `data` crosses the worker boundary as JSON, never as JavaScript source. Its
140
- JSON-encoded length is capped separately at `maxCodeChars`; split larger inputs.
196
+ `data` crosses the worker boundary as JSON, never as JavaScript source. For a
197
+ single program its JSON-encoded length is capped separately at `maxCodeChars`;
198
+ batches use the combined admission budget described below. Split larger inputs.
141
199
  The binding exists only when supplied, so older programs declaring their own `data`
142
200
  remain valid. Syntax errors run no commands and give quoting guidance. For inline
143
201
  source, use `String.raw` (escaping backtick delimiters) or JSON-quoted strings.
@@ -179,7 +237,7 @@ no truncated prefix is executed. Review untrusted source before running it.
179
237
  Use ordinary `edit` to revise saved programs. This is explicit source reuse, not
180
238
  conversation compression: prior calls and read results remain intact. Creation
181
239
  costs an additional call unless combined with other work, so prefer inline code
182
- for short one-off operations. See [token measurements](docs/TOKEN_COSTS.md).
240
+ for short one-off operations. See [token measurements](https://github.com/AdityaVG13/pi-stack/blob/main/packages/pi-supernova/docs/TOKEN_COSTS.md).
183
241
 
184
242
  ### Batch already-known continuations
185
243
 
@@ -194,12 +252,32 @@ for short one-off operations. See [token measurements](docs/TOKEN_COSTS.md).
194
252
  }
195
253
  ~~~
196
254
 
197
- Use programs instead of top-level code/file/data. Supply 1--32 entries, each with
198
- code OR file and optional data; the JSON-encoded array must fit maxCodeChars.
255
+ Use programs instead of top-level code/file. Supply 1--32 entries, each with
256
+ code OR file and optional data. Top-level data supplies an optional default for
257
+ each entry; explicit entry data replaces it entirely, including null, false, 0
258
+ and empty strings. Every guest receives its own copy, not a shared mutable heap.
259
+ The JSON-encoded array (or `{programs,data}` when defaults are supplied) must fit
260
+ maxCodeChars. Common input counts once; result representations and output limits are unchanged.
199
261
  Entries run sequentially in fresh guests and commit separately. A successful
200
262
  entry can create the file executed by a later entry. No implicit retries,
201
263
  reordering, shared heap or nested batches are introduced.
202
264
 
265
+ For independent audits that use the same inputs, send them once:
266
+
267
+ ```json
268
+ {
269
+ "data": {"paths": ["src/a.js", "src/b.js"]},
270
+ "programs": [
271
+ {"code": "return await read(data.paths);"},
272
+ {"code": "return await Promise.all(data.paths.map(path => read({path, outline:true})));"}
273
+ ]
274
+ }
275
+ ```
276
+
277
+ This avoids repeating literal arguments, without a compression codec or result elision.
278
+ Mutating `data` in one guest cannot affect the next. An entry with `data:null`
279
+ receives null, not the shared object; there is no implicit object merge.
280
+
203
281
  The batch stops on the first failed entry, cancellation/deadline, or exhausted
204
282
  output/log/image budget. Earlier successful commits remain; only the active
205
283
  program's uncommitted writes roll back. Admission errors throw before any program.
@@ -208,6 +286,12 @@ results/images: isError and details.ok identify failure, details.programs contai
208
286
  every attempted result, and details.attempted/total identifies unstarted work.
209
287
  Single code/file invocations retain their existing throwing behavior.
210
288
 
289
+ Set `parallel: true` with `programs` to run independent entries concurrently
290
+ (up to 8 at once). Each still gets a fresh guest and its own commit; results stay
291
+ in submission order. A failed entry does not stop siblings. Two entries writing
292
+ the same file race: the losing commit reports a conflict. Sequential remains the
293
+ default. `parallel` is invalid on a lone `code` or `file` call.
294
+
211
295
  The outer deadline, host-call budget, log allowance, text budget and image limits
212
296
  are shared across the batch. Individual read budgets are not reduced. Every
213
297
  attempted program's original text is returned in length-delimited blocks; ordinary
@@ -266,6 +350,8 @@ plain .json reads also fail with a projection hint. Explicit offset/limit or
266
350
  resolve:true still allow raw inspection, but line windows are not JSON documents.
267
351
  Do not combine json with complete, line windows, or source views. External read
268
352
  overrides reject JSON projection rather than silently ignoring the option.
353
+ Other read options, even false-valued flags, do not bypass a captured external
354
+ read executor; its policy, transforms and failures remain authoritative.
269
355
 
270
356
  For large Markdown/log path audits, use read(path,{about:"document path"}) or
271
357
  explicit offset/limit, not complete:true. Larger JSON needs a streaming parser via
@@ -424,7 +510,7 @@ not hard real-time guarantees.
424
510
  It excludes model latency, provider tokens and prewarm time; it is not a universal
425
511
  comparison against every CodeMode implementation.
426
512
 
427
- See [the changelog](docs/CHANGELOG.md) for changes and compatibility notes.
513
+ See [the changelog](https://github.com/AdityaVG13/pi-stack/blob/main/packages/pi-supernova/docs/CHANGELOG.md) for changes and compatibility notes.
428
514
 
429
515
  ## Research and prior art
430
516
 
package/docs/CHANGELOG.md CHANGED
@@ -1,5 +1,155 @@
1
1
  # Changelog
2
2
 
3
+ ## Unreleased
4
+
5
+ ## [0.7.0] - 2026-09-17
6
+
7
+ ### Internals
8
+
9
+ - Shared `src/contract/` for read/edit/bash shapes. Guest and host classify once;
10
+ guest no longer reimplements exclusive-mode routing before RPC.
11
+ - Read dispatch is `classifyRead` → kind table. Disk vs staged `about` focus share
12
+ one `focusAbout` helper.
13
+ - Invoke permission/target resolution lives in `bridge/invoke.js`.
14
+ - Native adapters live in `src/adapters/{read,write,edit,bash,list}.js`;
15
+ `createNativeAdapters` is a 30-line assembler. The host kernel stays in
16
+ `host-bridge.js`.
17
+ - Line/edit helpers in `fs/text-ops.js`. Unused catalog search/describe APIs and
18
+ the unused native-tool registrar are gone.
19
+ - Per-function cyclomatic complexity is under 10: guest lifecycle is `GuestRun`,
20
+ program batches are `ProgramBatch`, snap ranking/read/edit/bash/VFS/evidence/
21
+ output/UI are extracted helpers and classify tables. Same public behavior;
22
+ BIND DAG unchanged.
23
+ - Guest isolate: no `fs` / `child_process` / `import` / `require`. After `read()`
24
+ of a path, `write()` of that path throws (use `edit`, or `replace:true`).
25
+ Raw reads of large source require `about` / offset or `complete:true`;
26
+ large JSON returns a shape routing value (top-level keys, or length for
27
+ arrays) instead of throwing.
28
+ - Standing reference stays one nova invocation: prefer `edit` after `read`,
29
+ `json`/about/`complete:true` for large files, no guest `fs`, and
30
+ `programs`/`parallel:true` for independent work *inside* one call. Overlapping
31
+ sibling `supernova` calls still hint that independent work belongs in one
32
+ program.
33
+
34
+ ### Added
35
+
36
+ - `parallel: true` on `programs`: independent entries run concurrently (up to 8
37
+ lanes) in fresh guests, results keep submission order, and a failed entry does
38
+ not stop siblings. Two entries writing the same file race; the losing commit
39
+ reports a conflict. Sequential batches still stop on the first failure.
40
+ `parallel` is rejected on a lone `code` or `file` call.
41
+ - JSON selectors accept array `.length` (for example `.items.length`) so a
42
+ catalog count does not require dumping the array. String `.length` is still
43
+ rejected; this is not jq.
44
+ - Raw reads of JSON above the bound return a shape routing value instead of
45
+ throwing: `{status:"too_large", path, chars, keys}` for objects (`length`
46
+ for top-level arrays), so the same program can project with `json:".field"`
47
+ and array reads survive one oversize member. Malformed JSON still throws;
48
+ non-JSON text still returns verbatim.
49
+
50
+ ### Changed
51
+
52
+ - Prefer `edit` for a file already read; `write` remains replace-the-file.
53
+ - Overlapping sibling `supernova` calls hint to batch independent work as
54
+ `programs` with `parallel:true`.
55
+ - Empty programs (no adapter calls) still draw a nova card with a result preview
56
+ instead of a one-line `complete` status.
57
+ - Shorter standing tool reference: 68 fewer tokens per request with the same
58
+ commands and surface needles. Wording compression plus dropped peripheral
59
+ clauses; the JSON/about signatures, prefer-edit rule, and batching nudge
60
+ stay. Ablation family verified live over 12 green gpt-6-astra runs.
61
+ - Standing tool reference trimmed by 120 more tokens per request (o200k_base),
62
+ below the previous release baseline: limits and failure patterns already
63
+ taught by engine errors are no longer repeated proactively, and the batching
64
+ nudge, guest-confinement rule, and checkpoint clause are compressed. All
65
+ surface needles, the prefer-edit rule, and the oversize-JSON routing line
66
+ stay. Live-verified over 2 green gpt-6-astra runs against 2 task-matched
67
+ controls with no strategy change.
68
+ - An acquired guest worker pipelines its successor while the run executes, so
69
+ back-to-back programs share construction cost: sequential batches run ~2x
70
+ faster (22.4ms to 10.5ms per realistic program). Isolated cold starts are
71
+ unchanged apart from the deferred successor spawn.
72
+ - Multi-edit receipts render the first 32 matches with exact totals and an
73
+ `…N more matches` note instead of unbounded lines.
74
+ - Write receipts report workspace-relative paths, matching `edited <rel>`.
75
+ - Evidence and outline reads return compact JSON instead of pretty-printed.
76
+ - Refused bridge calls (unknown or excluded tools) no longer consume the host
77
+ call budget; routing validation runs before charging.
78
+ - The VFS body cache is gone: every read hits disk or its overlay, and CAS
79
+ baselines are the only retained per-file state.
80
+
81
+ ### Fixed
82
+
83
+ - `bash({command, args}, opts)` no longer drops the second options argument:
84
+ `cwd` (and `timeoutMs`) merge in, with the params object's own keys winning
85
+ on conflict. Covered by argv-form tests.
86
+ - Over-budget JSON selections return an in-band routing value
87
+ (`{status:"too_large", path, selector, chars}` with `keys` or `length`)
88
+ instead of throwing, so one oversize field no longer kills the read and
89
+ small sibling selections flow through. The standing reference line covers
90
+ raw and selection routing at the same token cost; live-verified over 3
91
+ green runs (top-level keys now answer in 1 call instead of 2).
92
+ - Memory-limit failures now report the RSS growth, the in-flight operation,
93
+ host-call count, and tracked host bytes (index entries, overlays),
94
+ splitting tracked from untracked growth so host-side pressure
95
+ is distinguishable from tool-side growth.
96
+ - `process.kill` is sealed out of the guest realm: signals are process-wide
97
+ and could terminate the host. Stop processes with `bash`.
98
+ - Patch hunks that drift report their relocation per hunk, and a hunk whose
99
+ context matches more than one location fails with a disambiguation error
100
+ instead of applying at the first candidate.
101
+ - A failed commit keeps CAS baselines for files it never touched, so a later
102
+ write to a diverged path fails loudly instead of re-capturing unknown
103
+ bytes as the new truth.
104
+ - The workspace index reuses one scratch read buffer instead of allocating
105
+ 512 KiB per file, bounding transient RSS on large-tree scans.
106
+
107
+ ## [0.6.0] - 2026-09-15
108
+
109
+ ### Added
110
+
111
+ - Shared literal input for `programs`: top-level `data` defaults each entry, while
112
+ explicit entry data replaces it entirely, including falsy values. Every fresh
113
+ guest receives its own copy. The combined JSON admission budget counts common
114
+ input once; deadlines, host-call limits, separate commits and stop reports stay
115
+ unchanged. No implicit object merge, shared heap or inferred plan is introduced.
116
+
117
+ ### Hardened execution and reads
118
+
119
+ - Keep byte-accurate conflict snapshots independent of receipt/body caches.
120
+ Explicit rereads refresh observations; internal diff reads and cache eviction do
121
+ not rebase pending writes. Partial and focused reads retain full-file signatures,
122
+ including above 16 MiB, and invalid UTF-8 no longer causes a false conflict.
123
+ - Canonicalize new-file destinations before checking conflicting symlink aliases,
124
+ while preserving the existing logical paths in workspace-change notifications.
125
+ - Preserve read/mutation/checkpoint ordering across coalesced read waves. Tighten
126
+ input validation, cancellation handling and bounded output without reusing an
127
+ executed worker or reducing individual independent-read budgets.
128
+ - Keep captured read overrides authoritative when options are supplied. Align
129
+ native/guest evidence results and path-array aliases; reject incompatible modes
130
+ and enforce the same focused-query keyword cap for disk and staged content.
131
+ - Discover newly staged declarations in file-scoped queries and large overlays.
132
+ Preserve line endings, EOF characters and post-edit line coordinates. Distinguish
133
+ absent matches from matches that exceed the view budget. `complete` consistently
134
+ means the whole file; extensionless paths support `complete:true`.
135
+
136
+ ### Documentation and verification
137
+
138
+ - Document shared-input examples, commit/rollback and override boundaries, and the
139
+ need for a full host restart after JavaScript updates. `/reload` can retain old
140
+ native ESM modules; pushing GitHub does not update an npm installation.
141
+ - Restore the historical token fixture and hash-lock it. Version the single
142
+ terminating-newline expectation separately, without changing historical traffic,
143
+ programs, decision boundaries or acceptance thresholds.
144
+ - Add failure-first regressions for reviewed and newly found defects. Correct
145
+ oversized fixtures, non-finite timeout inputs and misleading test descriptions.
146
+ - Measure shared-input audits with identical complete outputs: 16,309 to 5,499
147
+ tokens (o200k_base) and 14,649 to 5,197 (cl100k_base), including replay, result
148
+ framing and added standing guidance. These are workload-specific non-compressive
149
+ savings, not provider-billing or live-model quality claims. See
150
+ [token measurements](https://github.com/AdityaVG13/pi-stack/blob/main/packages/pi-supernova/docs/TOKEN_COSTS.md)
151
+ for the unchanged historical gates, accounting and reproduction commands.
152
+
3
153
  ## [0.5.0] - 2026-09-12
4
154
 
5
155
  - Report overlapping `supernova` calls from every participant in the wave, including the first-started call that finishes last. Start-order or finish-order counters missed that side; a peak concurrent count resets when the wave drains. Sequential calls, `programs` batches, and failed programs still do not leak a split hint. The regression forces the slower first program so the last-finisher case cannot flake under load.
@@ -1,11 +1,12 @@
1
1
  # Token usage and benchmarks
2
2
 
3
- Supernova reduces repeated tool traffic through explicit program reuse and batching.
3
+ Supernova reduces repeated tool traffic through explicit program reuse, batching
4
+ and shared batch input defaults.
4
5
  It can also reduce escaping in nested multiline results with lossless text framing.
5
6
  These mechanisms do not summarize results, rewrite conversation history, or change
6
7
  reasoning settings. Savings depend on the workload.
7
8
 
8
- See the [API guide](../README.md) for program-file and batch usage.
9
+ See the [API guide](https://github.com/AdityaVG13/pi-stack/blob/main/packages/pi-supernova/README.md) for program-file and batch usage.
9
10
 
10
11
  ## Reproduce
11
12
 
@@ -69,12 +70,12 @@ text contributes to totals through later history, not as a second charge.
69
70
 
70
71
  ## Measured results
71
72
 
72
- Observed on macOS with Node v26.7.0.
73
+ Observed for 0.6.0 on 2026-09-15, on an Apple M5 Max running macOS and Node v26.7.0.
73
74
 
74
75
  | Tokenizer | Non-batched baseline | Batched baseline (d444eb7) | Current | Further reduction | Total reduction |
75
76
  | --- | ---: | ---: | ---: | ---: | ---: |
76
- | o200k_base | 28,130 | 18,535 | 9,537 | **48.55%** | **66.10%** |
77
- | cl100k_base | 27,841 | 18,310 | 9,420 | **48.55%** | **66.17%** |
77
+ | o200k_base | 28,130 | 18,535 | 9,843 | **46.90%** | **65.01%** |
78
+ | cl100k_base | 27,841 | 18,310 | 9,726 | **46.88%** | **65.07%** |
78
79
 
79
80
  The gate requires at least 40% reduction on **each tokenizer for the complete
80
81
  workload**, not for every scenario individually. Token counts and reductions are
@@ -83,15 +84,12 @@ returned by the runtime. A second gate requires another 19% against the measured
83
84
  batched baseline from commit d444eb7. Its six-call argument hash is pinned as well
84
85
  as the original workload: removing a decision boundary cannot satisfy this gate.
85
86
 
86
- ### What changed after the batched baseline
87
+ ### Definition and result accounting
87
88
 
88
- The serialized definition falls from 1,068 to 577 tokens with o200k_base and from
89
- 1,057 to 572 with cl100k_base. Duplicate object-form restatements, parameter
90
- prose already covered by the command list, and discoverable operational asides
91
- were removed; command signatures and safety rules (`complete:true`, JSON 16 MiB /
92
- no jq, array-read rejection, transactions, `programs` batch, edit oldText as an
93
- exact substring, `edit(view,text)`) remain in the standing reference. No source
94
- text or independent result is removed or compressed.
89
+ The current serialized definition is 602 tokens with o200k_base and 595 with
90
+ cl100k_base, versus 908 and 901 in the frozen non-batched baseline. It retains
91
+ command signatures, complete-read and JSON limits, array-read failure rules,
92
+ transaction boundaries, batch defaults and edit/view guidance on every request.
95
93
 
96
94
  For the fixed six-call schedule, the accounting can also be written as:
97
95
 
@@ -99,17 +97,9 @@ For the fixed six-call schedule, the accounting can also be written as:
99
97
  Total = (N+1)*D + sum((N-i+2)*A_i + (N-i+1)*R_i, i = 1..N)
100
98
  ~~~
101
99
 
102
- Seven definition appearances still save 7*491 = 3,437 tokens with o200k_base versus
103
- d444eb7's 1,068-token definition. Snap-to-span then changed the first repair
104
- observation from a whole-file view to the `MAX_JSON_BYTES` declaration
105
- (`lines:[3,3]`, `complete:false`). That smaller result is replayed through later
106
- requests; no source or independent result is compressed or dropped. Frozen
107
- programs, arguments and decision boundaries are unchanged. Combined with
108
- batching, current o200k traffic is 9,558 vs d444eb7's 18,535 (48.43%).
109
-
110
- The current definition is now *below* the non-batched baseline (577 vs 908
111
- o200k_base). Batching still adds result framing. One-off calls should not be
112
- assumed to benefit from the batch API.
100
+ The definition is counted seven times, including the final handoff. Batch result
101
+ framing and every attempted program's text are counted too. One-off calls should
102
+ not be assumed to benefit from batching.
113
103
 
114
104
  The report also includes separate source-framing and argument-reuse comparisons.
115
105
  Those component measurements are not total-session savings, and the reported
@@ -117,28 +107,78 @@ argument-only break-even excludes other request costs. Text framing is selected
117
107
  by character length, not a runtime tokenizer; it need not reduce tokens for every
118
108
  input or encoding.
119
109
 
110
+ ## Shared batch input: a separate 0.6.0 measurement
111
+
112
+ Eight independent audit programs use the same list of 48 source paths. The before
113
+ arm repeats the literal `data` in every entry; the after arm supplies it once at
114
+ the top level. Both execute the same programs and return the same complete source
115
+ strings and typed results. The before request fits the existing admission cap;
116
+ this is not a comparison against a hypothetical request that could never run.
117
+
118
+ | Tokenizer | Repeated-input traffic | Shared-input traffic | Reduction | Arguments before / after | Unchanged result tokens |
119
+ | --- | ---: | ---: | ---: | ---: | ---: |
120
+ | o200k_base | 16,309 | 5,441 | **66.64%** | 6,461 / 1,043 | 2,151 |
121
+ | cl100k_base | 14,649 | 5,135 | **64.95%** | 5,684 / 945 | 2,055 |
122
+
123
+ Each arm has one tool invocation followed by the final answer request:
124
+
125
+ ~~~text
126
+ Total = 2*D + 2*A + R
127
+ ~~~
128
+
129
+ Arguments are charged when generated and when replayed; the complete result is
130
+ charged on handoff. The standing guidance measures 602 definition tokens per
131
+ request with o200k_base (595 with cl100k_base), and that cost is included in
132
+ the after totals.
133
+ There is no source compression, result elision, hidden output or lost decision
134
+ boundary. Programs receive fresh data copies, not a shared mutable heap.
135
+
136
+ The gate requires at least 70% less argument traffic and 60% less replay-inclusive
137
+ traffic in each encoding, plus equality of the complete normalized output. This
138
+ workload deliberately exercises repeated input; it is not an average task-cost
139
+ estimate. The original 13-program/six-call benchmark remains separate and intact.
140
+
120
141
  ## Benchmark integrity
121
142
 
122
- The [baseline fixture](../tests/efficiency/token-baseline.json) contains the source
143
+ The [baseline fixture](https://github.com/AdityaVG13/pi-stack/blob/main/packages/pi-supernova/tests/efficiency/token-baseline.json) contains the source
123
144
  inputs, definition, workload hash, arguments and complete outputs. The
124
- [runner](../tests/efficiency/workflow.mjs) executes the registered tool with real
145
+ [runner](https://github.com/AdityaVG13/pi-stack/blob/main/packages/pi-supernova/tests/efficiency/workflow.mjs) executes the registered tool with real
125
146
  workers and filesystem operations. It checks that:
126
147
 
127
148
  - The workload hash matches the frozen baseline, and the six-call argument hash
128
149
  matches the prior batched execution.
129
- - Every original argument, complete logical result and expected failure matches.
150
+ - Every original argument and expected failure matches; complete logical results
151
+ match the explicit current newline contract described below.
130
152
  - Batched output contains every original result, with no unaccounted outer text.
131
153
  - No result is truncated, and final repaired source and JSON contents match.
132
154
 
133
155
  Only run IDs, elapsed times and temporary workspace prefixes in write receipts
134
156
  are normalized. Batch framing lengths are adjusted to match that normalized text.
135
157
  The workload, batch schedule, recorded comparison totals and acceptance thresholds
136
- are fixed test inputs, not production execution rules. The baseline is not regenerated by the benchmark.
158
+ are fixed test inputs, not production execution rules. The baseline is not regenerated
159
+ by the benchmark and is checked against its SHA-256 before execution.
160
+
161
+ Contract v2 preserves the terminating newline of one selected source line. The
162
+ runner derives that single expected-output correction from the frozen input, not
163
+ from candidate output. The historical fixture and its traffic counts stay
164
+ untouched; programs, arguments, failures and decision boundaries are unchanged.
165
+ The shared-input comparison separately requires equal complete result text in
166
+ both arms, after only run-metadata normalization.
167
+
168
+ Contract v3 reports write receipts relative to the workspace (`wrote rel/path`,
169
+ matching the long-standing `edited <rel>` form) instead of absolute paths. The
170
+ runner strips the frozen `/workspace/` prefix from baseline outputs; the
171
+ historical fixture and its traffic counts stay untouched. Temporary workspace
172
+ prefixes no longer appear in live receipts, so that normalization only applies
173
+ to the frozen baseline side.
174
+
175
+ The README and these docs ship in the npm tarball. Benchmarks and test fixtures
176
+ remain in the GitHub checkout, so their links above use GitHub URLs.
137
177
 
138
178
  Recorded baseline SHA-256:
139
179
 
140
180
  ~~~text
141
- 79a819eca82c8a5ff381e96b7669d8a5bf04fabbc8c10cf02f3a7c8817171b62
181
+ 96964f990f481ac05afaefdd02001bd15f61835a8381349a61e38c06209d7508
142
182
  ~~~
143
183
 
144
184
  ## Limitations
@@ -150,6 +190,8 @@ Recorded baseline SHA-256:
150
190
  reasoning settings does not establish unchanged end-to-end task quality.
151
191
  - Batching is appropriate only for already-chosen continuations. Actions requiring
152
192
  a new model decision must remain separate calls.
193
+ - The report also has an experimental citation-elision arm. It is disabled by
194
+ default and is not the source of the non-compressive savings reported here.
153
195
  - Existing read, output, log, image and execution limits still apply. The
154
196
  benchmark does not obtain savings by lowering them or hiding truncation.
155
197