pi-supernova 0.3.2 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # pi-supernova
2
2
 
3
- **One nova / `supernova({code})` invocation. Four commands inside CodeMode.**
3
+ **One `supernova` invocation. Inline code, a workspace program, or an explicit program batch. Four commands inside CodeMode.**
4
4
 
5
5
  ```javascript
6
6
  const source = await read("validateRefreshToken");
@@ -48,6 +48,8 @@ settings. The runtime does not silently rewrite your tool policy.
48
48
  | Function | Examples and behavior |
49
49
  | --- | --- |
50
50
  | `read` | `read(path, offset?, limit?)`, `read({path,offset,limit})`; one-based line windows |
51
+ | `read` | `return await read("plot.png")`; displays images directly, without a browser |
52
+ | `read` | `read({path,json:".verdict"})`; parse full JSON before bounded field selection |
51
53
  | `read` | `read(directory)`, `read("symbol or question")`, `read(path,{about:question})`; questions locate and open source directly |
52
54
  | `read` | `read({query,resolve:true})`; structured source and status for a resolve-to-edit handoff |
53
55
  | `read` | `read({query,evidence:true})`; ranked evidence with provenance; optional `path` scopes discovery |
@@ -122,16 +124,33 @@ full dataflow tracking or a security sandbox.
122
124
 
123
125
  Explicit read arrays reject missing/failed paths. For typed partial outcomes use
124
126
  `Promise.allSettled(paths.map(path => read(path)))`. Successful arrays remain arrays.
125
- For embedded source with backslash escapes, use `String.raw` template literals
126
- (and escape delimiter backticks), or JSON-quoted strings. Validate generated code.
127
+ For literal file content or scripts, prefer the optional tool-level `data` parameter:
128
+
129
+ ```json
130
+ {
131
+ "code": "await write(data.path,data.content); return await bash({command:\"python3\",args:[data.path]});",
132
+ "data": {
133
+ "path": "probe.py",
134
+ "content": "print(\"literal `backticks` and ${braces}\")\n"
135
+ }
136
+ }
137
+ ```
138
+
139
+ `data` crosses the worker boundary as JSON, never as JavaScript source. Its
140
+ JSON-encoded length is capped separately at `maxCodeChars`; split larger inputs.
141
+ The binding exists only when supplied, so older programs declaring their own `data`
142
+ remain valid. Syntax errors run no commands and give quoting guidance. For inline
143
+ source, use `String.raw` (escaping backtick delimiters) or JSON-quoted strings.
127
144
 
128
145
  On hosts exposing `sessionManager.getArtifactsDir()`, bare `agent://<id>` and
129
146
  `artifact://<number>` read files from the calling session's artifact directory.
130
147
  They preserve ID casing and support offset/limit and structured continuation.
131
148
  These resources are read-only; ambiguous artifact IDs and escaping symlinks fail.
132
- This is not the full OMP URI language: cross-session search, nested path/query
133
- selectors and other schemes are not implemented. Hosts without an artifact
134
- directory report that limitation rather than treating the URI as a local path.
149
+ A single `?q=.answer` (URL-encoded when needed) selects JSON from either resource
150
+ using the same bounded projection as `read({path,json})`. The resource must contain
151
+ valid JSON; ordinary Markdown is not parsed heuristically. This is not the full
152
+ OMP URI language: full jq, cross-session search and other schemes are not implemented.
153
+ Hosts without an artifact directory report that limitation rather than treating the URI as a local path.
135
154
 
136
155
  For unstructured logs/text, `read(path,{about:"STT database"})` returns bounded,
137
156
  line-numbered matching windows, or explicitly reports no matching text. It is not
@@ -139,6 +158,67 @@ a complete-file read. Write temporary investigation files under the workspace
139
158
  (e.g. `.work/probe.py`): ordinary `write`/`edit` paths cannot escape it, including
140
159
  absolute `/tmp` paths. Shell execution is a separate trusted boundary, not a sandbox.
141
160
 
161
+ ### Reuse a program without resending its source
162
+
163
+ Save a trusted JavaScript async body or arrow in the workspace, then invoke it:
164
+
165
+ ```json
166
+ {"file":".work/audit.js","data":{"paths":["src/a.js","src/b.js"],"term":"TODO"}}
167
+ ```
168
+
169
+ Supply exactly one of `code` or `file`. File programs get the same four commands,
170
+ optional `data`, limits, deadlines and transaction semantics. Each invocation
171
+ rereads the file and starts a fresh guest; there is no implicit last-program
172
+ state, auto-replay, or retained heap. Paths inside the program still resolve from
173
+ the calling workspace, not the script directory. This runs a Nova program, not
174
+ an arbitrary JavaScript module or a Python/shell script.
175
+
176
+ Program files must be regular UTF-8 files inside the workspace, including symlink
177
+ targets. Invalid encoding, oversized input and syntax errors fail before commands;
178
+ no truncated prefix is executed. Review untrusted source before running it.
179
+ Use ordinary `edit` to revise saved programs. This is explicit source reuse, not
180
+ conversation compression: prior calls and read results remain intact. Creation
181
+ costs an additional call unless combined with other work, so prefer inline code
182
+ for short one-off operations. See [token measurements](docs/TOKEN_COSTS.md).
183
+
184
+ ### Batch already-known continuations
185
+
186
+ ~~~json
187
+ {
188
+ "programs": [
189
+ {"code": "return await edit(data.path,data.oldText,data.newText);", "data": {"path":"src/config.js","oldText":"limit = 8","newText":"limit = 16"}},
190
+ {"code": "return await bash(\"npm test\");"},
191
+ {"file": ".work/audit.js", "data": {"paths":["src/config.js"],"term":"limit"}}
192
+ ],
193
+ "timeoutMs": 60000
194
+ }
195
+ ~~~
196
+
197
+ Use programs instead of top-level code/file/data. Supply 1--32 entries, each with
198
+ code OR file and optional data; the JSON-encoded array must fit maxCodeChars.
199
+ Entries run sequentially in fresh guests and commit separately. A successful
200
+ entry can create the file executed by a later entry. No implicit retries,
201
+ reordering, shared heap or nested batches are introduced.
202
+
203
+ The batch stops on the first failed entry, cancellation/deadline, or exhausted
204
+ output/log/image budget. Earlier successful commits remain; only the active
205
+ program's uncommitted writes roll back. Admission errors throw before any program.
206
+ Execution failures return a **typed stop report**, rather than throwing away prior
207
+ results/images: isError and details.ok identify failure, details.programs contains
208
+ every attempted result, and details.attempted/total identifies unstarted work.
209
+ Single code/file invocations retain their existing throwing behavior.
210
+
211
+ The outer deadline, host-call budget, log allowance, text budget and image limits
212
+ are shared across the batch. Individual read budgets are not reduced. Every
213
+ attempted program's original text is returned in length-delimited blocks; ordinary
214
+ limits still disclose clipping. Images retain program/image labels. Split a plan
215
+ that would exceed the aggregate output budget.
216
+
217
+ Batch only continuations already chosen by the agent, such as edit then known
218
+ verification, or create then run known audits. Keep a separate call whenever new
219
+ source/results are needed to decide the next action. This does not lower reasoning
220
+ settings, hide observations, or infer a plan on the agent's behalf.
221
+
142
222
  ### Large inputs and report outputs
143
223
 
144
224
  The default program limit is 48,000 UTF-16 code units (configurable via
@@ -159,17 +239,38 @@ output delivery; progress files survive shell execution but staged VFS writes ma
159
239
  roll back.
160
240
 
161
241
  Large returned objects are bounded previews, not retained artifacts. Select fields
162
- and array windows before returning, rather than parsing a truncated preview:
242
+ and array windows before returning, rather than parsing a truncated preview.
243
+
244
+ ## JSON reports and targeted text audits
245
+
246
+ For JSON, select fields inside the read adapter, **before** output budgeting:
163
247
 
164
248
  ```js
165
- const report = JSON.parse(await read({path:"report.json",complete:true}));
166
- return {verdict:report.verdict, values:report.values.slice(5000,5003)};
249
+ const [verdict, values] = await read({path:"report.json",json:[".verdict",".values[5000:5003]"]});
250
+ return {verdict, values};
167
251
  ```
168
252
 
169
- If the raw JSON exceeds the read budget, reconstruct exact source windows or run
170
- a bounded parser through `bash`. There is no implicit continuation handle for
171
- arbitrary guest objects. For embedded code, JSON-encode the source string once;
172
- do not nest shell, JavaScript, and Python quoting unless it is necessary.
253
+ Selectors support "." (root), .field, .nested[0], .items[0:10], and .["quoted.key"].
254
+ Use json:true for the complete parsed value. Selectors are not full jq: pipes,
255
+ filters, wildcards and negative indices fail explicitly. Missing keys and indices
256
+ fail; false, zero and null remain values. Slices use an exclusive end and clamp to
257
+ array length. Only own JSON properties are traversed; nothing is evaluated.
258
+
259
+ Inputs are capped at 16 MiB, including staged files. JSON reads require regular
260
+ files and reject named pipes without waiting for a writer. The entire input must
261
+ be valid JSON before any selection. Each selector is budgeted before allocating
262
+ the next slice; sparse selector/path/edit arrays are rejected. Selected JSON must
263
+ fit the ordinary read budget or the
264
+ read throws; it is never returned as malformed/truncated JSON. Oversized unwindowed
265
+ plain .json reads also fail with a projection hint. Explicit offset/limit or
266
+ resolve:true still allow raw inspection, but line windows are not JSON documents.
267
+ Do not combine json with complete, line windows, or source views. External read
268
+ overrides reject JSON projection rather than silently ignoring the option.
269
+
270
+ For large Markdown/log path audits, use read(path,{about:"document path"}) or
271
+ explicit offset/limit, not complete:true. Larger JSON needs a streaming parser via
272
+ bash. Arbitrary returned objects still have bounded previews, not implicit
273
+ continuation handles.
173
274
 
174
275
  ## Execution and automatic batching
175
276
 
@@ -191,7 +292,12 @@ File changes are staged until program success. A throw before an external-mutati
191
292
  barrier rolls them back. Shell execution flushes preceding changes; external shell
192
293
  side effects cannot be rolled back. Stale commits fail explicitly rather than
193
294
  silently overwriting successful concurrent changes. This is not a cross-process
194
- filesystem lock.
295
+ filesystem lock. Outcomes explicitly report committed/rolledBack **file versions**
296
+ (counted per flush/checkpoint, not unique paths) and external-call attempts. A
297
+ successful inner checkpoint merges into the program, not necessarily onto disk.
298
+ Pending commits or failed recovery are reported as uncertain: inspect disk and
299
+ recovery backups before retrying. Import-based mutations and shell side effects
300
+ are outside the VFS counters; this is not a filesystem audit.
195
301
 
196
302
  `edit(async () => {...})` creates a nested filesystem checkpoint. It returns
197
303
  `{ok:true,committed:true,value}` on success or `{ok:false,committed:false,error}` on
@@ -212,6 +318,10 @@ outside the active callback are rejected. Await the checkpoint before proceeding
212
318
  strings and result types are unchanged. This is output framing, not source
213
319
  compression. It is chosen only when shorter in characters than escaped output;
214
320
  it does not guarantee lower billed tokens for every tokenizer or input.
321
+ Nested multiline strings use the same approach: the complete value structure
322
+ references `raw[n]`, followed by length-delimited verbatim string blocks. Literal
323
+ `"raw[0]"` values stay quoted; duplicate source is never deduplicated. Small values
324
+ keep their existing format. Machine-facing result values are unchanged.
215
325
  - Intermediate values stay inside CodeMode unless returned or logged. Final text,
216
326
  errors and logs are bounded with explicit truncation. Details support rendering;
217
327
  they are not a second model-facing transcript.
@@ -220,6 +330,12 @@ outside the active callback are rejected. Await the checkpoint before proceeding
220
330
 
221
331
  Default limits are in `src/config/config.default.json`. Configuration loads from
222
332
  `~/.pi/agent/supernova.json`, the configured host directory, or `PI_SUPERNOVA_CONFIG`.
333
+
334
+ Citation elision is **disabled by default** (`seenWindow: 0`): each result remains
335
+ self-contained within the normal output budgets. A positive `seenWindow` explicitly
336
+ opts into an experimental ledger with known retention gaps: hidden message details,
337
+ later context transforms and missing citation targets can invalidate its assumptions.
338
+ It is not a proven lossless optimization and is not recommended for production.
223
339
  Text limits are character budgets, not tokenizer counts. `/supernova` reports
224
340
  programs and output characters without labelling characters as tokens.
225
341
 
@@ -259,7 +375,9 @@ privileges. Do not run untrusted programs as though these adapters isolate them.
259
375
  Pi preflights the outer `supernova` call. Internal primitives do not emit ordinary
260
376
  native `tool_call` events, so third-party guards that only recognize top-level
261
377
  `edit` or `bash` need CodeMode-aware handling. Configured exclusions and supported
262
- host-session execution safeguards remain enforced. Actual-host smoke checks are
378
+ host-session execution safeguards remain enforced. Guards inspecting code/file
379
+ inputs must also understand the programs array; its entries do not emit separate
380
+ top-level tool_call events. Actual-host smoke checks are
263
381
  not a claim that every third-party permission extension has been validated.
264
382
 
265
383
  ## Development and evidence
@@ -284,6 +402,7 @@ arbitrary wall-clock assertions.
284
402
  npm test --prefix packages/pi-supernova
285
403
  npm run lint:supernova
286
404
  npm run measure --prefix packages/pi-supernova
405
+ npm run test:tokens --prefix packages/pi-supernova
287
406
 
288
407
  PI_SUPERNOVA_PI_ROOT=/path/to/pi-coding-agent \
289
408
  PI_SUPERNOVA_OMP=/path/to/omp \
package/docs/CHANGELOG.md CHANGED
@@ -1,5 +1,33 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.4.0] - 2026-09-10
4
+
5
+ - Disable citation elision by default (`seenWindow: 0`), including direct bridge/ledger defaults. A failure-first integration regression preserves complete results across hidden, reordered and removed context. Positive windows remain an explicit research opt-in, not a validated retention guarantee. The historical ledger measurements below describe that experimental mode only.
6
+
7
+ - Make the traffic gate able to see the retention ledger, and prove it can. The gate previously ran only a host that emits no lifecycle events, so it measured the conservative floor and could not observe the largest token lever in the package. It now also runs the frozen schedule against a production-shaped host that fires `context` before each request, comparing like for like against the unbatched baseline, and gates three things: an unobserved ceiling recorded after this pass, a floor requiring the observing host to beat the same schedule, and an anti-vacuity check requiring the fidelity assertion to have had a subject. Fidelity in that arm is asserted as elision-only rather than byte-equality: every surviving line must match in order and every citation must cover a run of exactly the lines it replaced.
8
+ - Add self-tests for the elision checker itself. A checker that cannot reject anything makes every gate that relies on it vacuous; seven failure-first cases (a vanished line, an overstated or understated citation, a sub-run citation, an invented line, a reordered elision) fail if the checker is weakened or reports that it saw nothing.
9
+ - Fix an information-loss bug the retention proof exposes: a program inside a `programs` batch could cite a peer in the same batch, whose result the model had not received yet. Nothing in a batch is collapsible against its own batch, and a regression now holds that line. The pre-existing frozen output assertion already caught this; it now has a named test.
10
+ - Record the observed-arm ceiling in `token-baseline.json` with provenance. The frozen workload is effectively read-once, so the ledger's saving on it is 0.77%/0.85%; repeat-heavy sessions measure 43.6% and are guarded by `tests/fidelity/ledger-retention.test.mjs`.
11
+ - Experimental only: retain the seen-ledger observation mechanism for research. Its earlier "retention proof" claim was invalid: metadata is not necessarily model-visible, context handlers can transform later, and a missing citation target is not rehydrated. The extension now subscribes to Pi's `context` hook and collapses a run only when those exact lines were observed in the messages about to be sent. The blanket `seenWindow: 0` existed because local cache residency cannot establish model-context residency; the pre-conversion host observation does not establish final-payload residency. Future collapse stops when lines disappear from the observation, but existing citations can still be dangling; a citation is never evidence of its own contents, and citations never nest. Observation costs ~1.7 ms per request on a 0.87 MB payload. On a 12-call repeated-read session: 42.8% fewer published result characters and 31.0% less cumulative replay. Changed lines are still never collapsed, explicit offset/limit reads stay pinned, and a host that never fires `context` publishes byte-identical results, so the frozen `events` and the batched arm of the gate are untouched; only a new observed-arm ceiling was added to the baseline.
12
+ - Consolidate overlapping tests into multi-operation failure-and-repair contracts covering JSON, saved programs, checkpoints, literal argv, images and concurrent workspaces. Retain distinct admission, isolation and resource-limit regressions.
13
+ - Fix truncation reporting for complete tool responses, including logs and wrapper metadata; make log-limit omissions visible in model-facing output.
14
+ - Remove duplicate startup guidance and batch wrapper metadata while retaining every original per-program result. The unchanged six-call benchmark uses another 5.15%/5.17% fewer tokens than d444eb7; add a request-hash guard against moving model decision boundaries.
15
+
16
+ - Add explicit sequential `programs` batches with fresh guests and separate commits. Preserve complete per-program text and earlier images in typed stop reports; share deadlines, host calls, logs and outer output/image limits. Reject malformed/nested/oversized plans before execution.
17
+ - Make the additional 40% token gate part of `npm test`, with a frozen post-previous-pass baseline, exact logical-result checks, both tokenizers, full tool-history replay and the final answer handoff. Measured 56.19%/56.26% fewer modeled tool tokens for the fixed workload; retain the full startup reference. No provider billing or model-quality claim.
18
+
19
+ - Add explicit workspace program-file input as an alternative to inline code, with fresh loading/guests, strict bounded UTF-8 admission and unchanged deadlines/transactions. No implicit replay or persistent heap.
20
+ - Extend verbatim string framing to nested source results, preserving all fields, types, duplicate strings and lengths; retain compact scalar output. Remove redundant guidance already present in parameter/command descriptions.
21
+ - Add reproducible two-tokenizer measurements and package comparisons in `TOKEN_COSTS.md`, including creation/definition overhead and explicit no-claim boundaries for billing and end-to-end quality.
22
+
23
+ - Stress follow-up: budget each JSON selector before allocating the next slice, reject non-regular JSON inputs without blocking on FIFO open, and reject sparse selector/path/edit arrays. Add a heap-limited host-survival regression, exact input boundaries, concurrent queried-resource isolation, and failed-recovery backup verification. The mixed stress lane now exercises literal input data and JSON projection.
24
+
25
+ - Add literal tool-level `data` input for Markdown/scripts/argv without nested JavaScript quoting; parse failures explicitly state no commands ran.
26
+ - Parse full JSON before bounded field/index/slice projection with `read({path,json})`, including session `?q=.answer` resources. Reject invalid selectors, missing fields, oversized inputs/selections and incompatible read options. Oversized unwindowed plain JSON reads fail with actionable guidance.
27
+ - Validate edit overloads before dispatch and show supported signatures, without touching a file on invalid input.
28
+ - Report committed/rolledBack file versions and external-call attempts on failure, and distinguish uncertain commit recovery from complete rollback.
29
+ - Advertise direct image viewing and targeted large-text reads in model-facing guidance. Add Spark papercut regressions through the registered tool and real worker.
30
+
3
31
  ## [0.3.2] - 2026-09-07
4
32
 
5
33
  - Focus `read(path,{about})` on matching line windows in unstructured logs/text instead of returning a truncated unrelated prefix; report no matches explicitly.
@@ -0,0 +1,171 @@
1
+ # Token usage and benchmarks
2
+
3
+ Supernova reduces repeated tool traffic through explicit program reuse and batching.
4
+ It can also reduce escaping in nested multiline results with lossless text framing.
5
+ These mechanisms do not summarize results, rewrite conversation history, or change
6
+ reasoning settings. Savings depend on the workload.
7
+
8
+ See the [API guide](../README.md) for program-file and batch usage.
9
+
10
+ ## Reproduce
11
+
12
+ From a repository checkout with development dependencies installed:
13
+
14
+ ~~~sh
15
+ npm run test:tokens --prefix packages/pi-supernova
16
+ npm test --prefix packages/pi-supernova
17
+ ~~~
18
+
19
+ The benchmark uses js-tiktoken 1.0.21, pinned as a development dependency. It runs
20
+ real Supernova programs in temporary workspaces without provider calls or downloads.
21
+ The token regression gate also runs in the normal test suite.
22
+
23
+ ## Workload
24
+
25
+ The baseline is a frozen, non-batched implementation snapshot with program-file
26
+ reuse and nested source framing already enabled. It is not a comparison against
27
+ a separately published release.
28
+
29
+ | Scenario | Baseline calls | Batched calls |
30
+ | --- | ---: | ---: |
31
+ | Inspect source, reproduce a failure, repair and verify | 4 | 3 |
32
+ | Inspect, update and verify a JSON report | 3 | 2 |
33
+ | Create an audit program and run it for five search terms | 6 | 1 |
34
+ | **Total** | **13** | **6** |
35
+
36
+ Both executions perform the same 13 logical programs. The batch schedule is
37
+ hand-authored: it groups known edit-then-verify and create-then-audit continuations,
38
+ while keeping decision points after inspection and the failing check separate.
39
+ No model chooses the schedule during this test. The runtime does not infer plans
40
+ or contain benchmark-specific paths, search terms, or expected results.
41
+
42
+ This workload deliberately includes repeated audits that benefit from batching.
43
+ It is a reproducible regression fixture, not a representative sample of all agent
44
+ tasks or a guarantee of savings when a model decides how to use the tool.
45
+
46
+ ## Accounting
47
+
48
+ For each model request, the benchmark counts:
49
+
50
+ - The serialized Supernova definition: description, parameters and prompt guidance.
51
+ - All preceding tool arguments and complete result text in the request history.
52
+ - Newly generated tool arguments, serialized as JSON.
53
+
54
+ It also counts the definition and full tool history sent for the final answer
55
+ request. A newly produced tool result is counted when the next model request
56
+ consumes it, not twice. Program creation is included. The complete startup
57
+ reference is retained, so this workload needs no separate discovery call.
58
+
59
+ Formally, with definition cost D, generated argument cost A_i, result cost R_i,
60
+ and prior tool-history cost H_i = sum(A_j + R_j) for j < i:
61
+
62
+ ~~~text
63
+ Total = sum(D + A_i + H_i, i = 1..N) + D + H_(N+1)
64
+ ~~~
65
+
66
+ This assumes full tool-history replay. The JSON report includes per-scenario
67
+ breakdowns. Its results field records newly produced text for inspection; that
68
+ text contributes to totals through later history, not as a second charge.
69
+
70
+ ## Measured results
71
+
72
+ Observed on macOS with Node v26.7.0 and Linux aarch64 with Node v24.16.0; both
73
+ produced the same token counts.
74
+
75
+ | Tokenizer | Non-batched baseline | Batched baseline (d444eb7) | Current | Further reduction | Total reduction |
76
+ | --- | ---: | ---: | ---: | ---: | ---: |
77
+ | o200k_base | 40,129 | 18,535 | 17,581 | **5.15%** | **56.19%** |
78
+ | cl100k_base | 39,697 | 18,310 | 17,363 | **5.17%** | **56.26%** |
79
+
80
+ The gate requires at least 40% reduction on **each tokenizer for the complete
81
+ workload**, not for every scenario individually. Token counts and reductions are
82
+ computed from the recorded baseline and fresh execution results, not constants
83
+ returned by the runtime. A second gate requires another 5% against the measured
84
+ batched baseline from commit d444eb7. Its six-call argument hash is pinned as well
85
+ as the original workload: removing a decision boundary cannot satisfy this gate.
86
+
87
+ ### What changed after the batched baseline
88
+
89
+ The serialized definition falls from 1,068 to 947 tokens with o200k_base and from
90
+ 1,057 to 937 with cl100k_base. Repeated guidance now has one model-visible home;
91
+ the command reference, safety rules and schema constraints remain available.
92
+ Batch framing declares UTF-16 length units once instead of on every entry and
93
+ omits redundant aggregate mutation totals from the wrapper. Every original
94
+ per-program result, including its mutation report, remains intact. Structured
95
+ aggregate counters are unchanged. No source text or independent result is removed.
96
+
97
+ For the fixed six-call schedule, the accounting can also be written as:
98
+
99
+ ~~~text
100
+ Total = (N+1)*D + sum((N-i+2)*A_i + (N-i+1)*R_i, i = 1..N)
101
+ ~~~
102
+
103
+ Seven definition appearances save 7*121 = 847 tokens with o200k_base. The smaller
104
+ batch wrappers save another 107 after history replay, for 954/18,535 = 5.15%.
105
+ For cl100k_base the corresponding saving is 7*120 + 107 = 947 tokens. The logical
106
+ programs, their arguments, complete results and decision boundaries are unchanged.
107
+
108
+ There are still costs: the current definition exceeds the non-batched baseline
109
+ by 39/36 tokens per request, and batching adds result framing. One-off calls should
110
+ not be assumed to benefit from the batch API.
111
+
112
+ The report also includes separate source-framing and argument-reuse comparisons.
113
+ Those component measurements are not total-session savings, and the reported
114
+ argument-only break-even excludes other request costs. Text framing is selected
115
+ by character length, not a runtime tokenizer; it need not reduce tokens for every
116
+ input or encoding.
117
+
118
+ ## Benchmark integrity
119
+
120
+ The [baseline fixture](../tests/efficiency/token-baseline.json) contains the source
121
+ inputs, definition, workload hash, arguments and complete outputs. The
122
+ [runner](../tests/efficiency/workflow.mjs) executes the registered tool with real
123
+ workers and filesystem operations. It checks that:
124
+
125
+ - The workload hash matches the frozen baseline, and the six-call argument hash
126
+ matches the prior batched execution.
127
+ - Every original argument, complete logical result and expected failure matches.
128
+ - Batched output contains every original result, with no unaccounted outer text.
129
+ - No result is truncated, and final repaired source and JSON contents match.
130
+
131
+ Only run IDs, elapsed times and temporary workspace prefixes in write receipts
132
+ are normalized. Batch framing lengths are adjusted to match that normalized text.
133
+ The workload, batch schedule, recorded comparison totals and acceptance thresholds
134
+ are fixed test inputs, not production execution rules. The baseline is not regenerated by the benchmark.
135
+
136
+ Recorded baseline SHA-256:
137
+
138
+ ~~~text
139
+ 79a819eca82c8a5ff381e96b7669d8a5bf04fabbc8c10cf02f3a7c8817171b62
140
+ ~~~
141
+
142
+ ## Limitations
143
+
144
+ - These are local estimates of model tool-token traffic, not provider billing.
145
+ Provider-specific envelopes, unrelated user/system messages, reasoning tokens,
146
+ image token costs and cache discounts are excluded.
147
+ - There is no live-model A/B quality evaluation. Preserving observations and
148
+ reasoning settings does not establish unchanged end-to-end task quality.
149
+ - Batching is appropriate only for already-chosen continuations. Actions requiring
150
+ a new model decision must remain separate calls.
151
+ - Existing read, output, log, image and execution limits still apply. The
152
+ benchmark does not obtain savings by lowering them or hiding truncation.
153
+
154
+ ## Design references
155
+
156
+ These sources informed the implementation choices; the measurements above compare
157
+ Supernova implementations only, not the performance of these packages.
158
+
159
+ | Reference | Mechanisms studied |
160
+ | --- | --- |
161
+ | [pi-codex-conversion 3.0.31](https://github.com/IgorWarzocha/howaboua-pi-stuff/tree/94eb6c0745e2f516bf19603f912f7b6478b43355) | Code transport, concise signatures and deferred discovery |
162
+ | [pi-codemcp](https://github.com/yolonir/pi-codemcp) | Saved call chains and intermediate results |
163
+ | [Ian Pascoe's pi-codemode](https://github.com/ian-pascoe/pi-extensions/tree/main/packages/pi-codemode) | Notebook state and tool declarations |
164
+ | [Nick Nisi's codemode](https://github.com/nicknisi/pi-extensions/tree/main/packages/codemode) | Named snippets and explicit composition |
165
+ | [Boozedog's pi-codemode 0.3.0](https://github.com/boozedog/pi-codemode/tree/1390c938ef4f23a0d50b814e7e3a14c03a08d39d) | Literal inputs and typed CLI capabilities |
166
+ | [pi-cache-optimizer 2.8.2](https://github.com/jiangge/pi-cache-optimizer) | Cache-prefix and prompt handling |
167
+
168
+ Unpinned repository links refer to inspected source, not independently verified
169
+ package versions. Supernova retains fresh guests, explicit transaction boundaries
170
+ and its general command surface rather than adopting persistent notebook state,
171
+ restricted CLI catalogs or provider/history rewriting.