pi-supernova 0.10.1 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +376 -221
- package/docs/CHANGELOG.md +107 -5
- package/docs/TOKEN_COSTS.md +149 -324
- package/index.js +40 -11
- package/package.json +4 -3
- package/src/adapters/bash.js +40 -10
- package/src/adapters/edit.js +77 -23
- package/src/adapters/errors.js +4 -0
- package/src/adapters/index.js +1 -0
- package/src/adapters/list.js +6 -3
- package/src/adapters/read-focus.js +6 -0
- package/src/adapters/read-image.js +17 -9
- package/src/adapters/read-indexed.js +72 -0
- package/src/adapters/read-text.js +19 -2
- package/src/adapters/read.js +57 -13
- package/src/adapters/refs.js +17 -7
- package/src/adapters/write.js +14 -21
- package/src/bridge/host-bridge.js +152 -39
- package/src/bridge/invoke.js +3 -1
- package/src/bridge/tool-registry.js +132 -16
- package/src/context/evidence-graph.js +1 -0
- package/src/context/evidence-rank.js +1 -0
- package/src/context/evidence.js +4 -2
- package/src/context/fuzzy.js +31 -8
- package/src/context/outline.js +33 -9
- package/src/context/query.js +7 -2
- package/src/context/repo-index.js +8 -3
- package/src/context/search-files.js +20 -0
- package/src/context/search.js +25 -10
- package/src/context/snap-search.js +28 -6
- package/src/context/snap.js +8 -3
- package/src/context/source-entry.js +64 -28
- package/src/context/spans.js +1 -0
- package/src/contract/bash.js +16 -7
- package/src/contract/edit.js +26 -7
- package/src/contract/program.js +2 -1
- package/src/contract/read.js +46 -46
- package/src/fs/background.js +10 -2
- package/src/fs/check.js +844 -46
- package/src/fs/commit.js +89 -15
- package/src/fs/diff.js +12 -9
- package/src/fs/directory.js +14 -4
- package/src/fs/file-io.js +28 -7
- package/src/fs/glob.js +93 -11
- package/src/fs/json-read.js +3 -2
- package/src/fs/json-size.js +2 -1
- package/src/fs/lines.js +49 -2
- package/src/fs/patch.js +26 -10
- package/src/fs/process-tree.js +19 -1
- package/src/fs/read-window.js +36 -20
- package/src/fs/source-window.js +5 -10
- package/src/fs/text-ops.js +39 -19
- package/src/fs/vfs.js +296 -41
- package/src/fs/workspace.js +24 -6
- package/src/output/bottleneck.js +50 -25
- package/src/output/final.js +20 -0
- package/src/output/format.js +97 -36
- package/src/output/outcome.js +25 -9
- package/src/runtime/batch-input.js +17 -3
- package/src/runtime/guest-api.js +32 -7
- package/src/runtime/guest-worker.js +22 -4
- package/src/runtime/parallel.js +9 -49
- package/src/runtime/program-batch.js +53 -28
- package/src/runtime/program-file.js +2 -1
- package/src/runtime/program.js +94 -41
- package/src/runtime/reference.js +10 -9
- package/src/runtime/runtime.js +86 -17
- package/src/runtime/worker-pool.js +10 -1
- package/src/shared/decode.js +9 -8
- package/src/shared/image-worker.js +7 -0
- package/src/shared/image.js +58 -15
- package/src/shared/png.js +18 -0
- package/src/shared/result.js +30 -9
- package/src/shared/syntax-context.js +7 -1
- package/src/shared/utf8.js +11 -13
- package/src/ui/host-render.js +4 -0
- package/src/ui/omp-frame.js +2 -0
- package/src/ui/progress.js +2 -1
- package/src/ui/render-measure.js +11 -0
- package/src/ui/render.js +92 -31
- package/src/ui/trace.js +8 -5
package/docs/TOKEN_COSTS.md
CHANGED
|
@@ -1,390 +1,215 @@
|
|
|
1
|
-
# Token usage and
|
|
1
|
+
# Token usage and local speed (0.11.0)
|
|
2
2
|
|
|
3
|
-
Supernova
|
|
4
|
-
and
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
3
|
+
Supernova 0.11.0 cuts repeated tool traffic by batching already-known
|
|
4
|
+
continuations, sharing program source and batch input, and keeping a shorter
|
|
5
|
+
standing definition. Source text and program results stay complete. Native
|
|
6
|
+
`edit` / `write` steps stay committed; independent `programs` continue after
|
|
7
|
+
ordinary errors; `indexed:true` opens owned source. These mechanisms do not
|
|
8
|
+
summarize results, rewrite conversation history, or change reasoning settings.
|
|
9
|
+
Savings depend on the workload.
|
|
8
10
|
|
|
9
|
-
See the [API guide](https://github.com/AdityaVG13/pi-stack/blob/main/packages/pi-supernova/README.md)
|
|
11
|
+
See the [API guide](https://github.com/AdityaVG13/pi-stack/blob/main/packages/pi-supernova/README.md)
|
|
12
|
+
for program-file and batch usage.
|
|
10
13
|
|
|
11
14
|
## Reproduce
|
|
12
15
|
|
|
13
16
|
From a repository checkout with development dependencies installed:
|
|
14
17
|
|
|
15
|
-
|
|
18
|
+
```sh
|
|
16
19
|
npm run test:tokens --prefix packages/pi-supernova
|
|
17
|
-
npm
|
|
18
|
-
|
|
20
|
+
npm run measure --prefix packages/pi-supernova
|
|
21
|
+
```
|
|
19
22
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
The token regression gate also runs in the normal test suite.
|
|
23
|
+
Recorded 2026-10-04 on Apple M5 Max, darwin, Node v22.23.3, js-tiktoken 1.0.21
|
|
24
|
+
(dev dependency). Real programs in temporary workspaces; no provider calls or
|
|
25
|
+
downloads. The token regression gate also runs in the normal test suite.
|
|
23
26
|
|
|
24
|
-
##
|
|
27
|
+
## Model tool traffic
|
|
25
28
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
29
|
+
Traffic counts model requests: serialized definition + generated arguments +
|
|
30
|
+
all prior tool arguments/results replayed, including the final answer handoff.
|
|
31
|
+
New result text is counted when consumed, not charged twice. Includes setup;
|
|
32
|
+
the full startup reference is retained, with no separate discovery call.
|
|
33
|
+
Hand-authored text-only workload and batch schedule; only counters, timing, and
|
|
34
|
+
temporary write receipts are normalized. Excludes provider envelopes, unrelated
|
|
35
|
+
conversation, reasoning tokens, and cache/billing. No model-quality A/B claim.
|
|
30
36
|
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
coalescing, with regressions reported rather than hidden.
|
|
49
|
-
|
|
50
|
-
The frozen six-call traffic is now 9,771 / 9,647 tokens (o200k_base / cl100k_base),
|
|
51
|
-
versus 9,813 / 9,689 at the pre-tuning checkpoint. The difference is shorter
|
|
52
|
-
standing guidance, not missing observations. Shared-source/object-input traffic
|
|
53
|
-
is 3,567 / 3,511; every output and all 32 files still match the frozen snapshot.
|
|
54
|
-
|
|
55
|
-
Focused regressions verify the formerly failing full-text/JSON/directory cases,
|
|
56
|
-
64 two-MiB reads in a Node host capped at 128 MiB, blocked-delivery cancellation
|
|
57
|
-
and follow-up use, bounded billion-character string expansions, sparse returns,
|
|
58
|
-
and complete/partial source-view edit rules. Bun runs the same tests, but its
|
|
59
|
-
heap flags and worker sampling do not provide Node's hard heap enforcement.
|
|
60
|
-
Installed copies were not modified; publication and restart remain separate.
|
|
61
|
-
|
|
62
|
-
## Unreleased explicit batch reuse (2026-09-19)
|
|
63
|
-
|
|
64
|
-
**79.82% / 79.92% less traffic on the shared-source/object-input workload, not a
|
|
65
|
-
claim of 80% savings on every task.** Source and object defaults remove repeated
|
|
66
|
-
arguments without compression, citations, result elision, context rewriting,
|
|
67
|
-
reduced limits or changes to reasoning settings. Every program still executes in
|
|
68
|
-
a fresh guest with its own commit. No model decision point is removed.
|
|
69
|
-
|
|
70
|
-
The missing capability was combining common executable/input values with distinct
|
|
71
|
-
per-entry inputs without repeating them or first saving helper files. Batches now
|
|
72
|
-
accept top-level `code` OR `file` as a source default. `mergeData:true` explicitly
|
|
73
|
-
opts into a shallow object overlay; the legacy whole-input replacement remains
|
|
74
|
-
the default. Entries can override the source, and nested data objects are replaced,
|
|
75
|
-
not recursively merged. Nothing is inferred or automatically deduplicated.
|
|
76
|
-
|
|
77
|
-
### Frozen eligible workload
|
|
78
|
-
|
|
79
|
-
The pre-feature working tree, including the earlier unreleased papercut fixes,
|
|
80
|
-
ran 16 independent package-doc scaffolds. Each program wrote a LICENSE and README,
|
|
81
|
-
reread both exact staged contents, and returned its complete receipts. The control
|
|
82
|
-
repeated the same executable and literal license/introduction with a distinct
|
|
83
|
-
directory per entry. Its encoded program array was **32,353 characters**, within
|
|
84
|
-
the existing 48,000-character admission cap. This is not an inadmissible baseline.
|
|
85
|
-
|
|
86
|
-
The candidate supplies the same complete code and common data once, plus those
|
|
87
|
-
same per-entry directories. Both arms use **one tool invocation plus the final
|
|
88
|
-
model handoff**, the same 16-program schedule, and all 32 final files. The baseline
|
|
89
|
-
was captured before the implementation; neither program source nor data was padded
|
|
90
|
-
or changed to meet the gate.
|
|
91
|
-
|
|
92
|
-
| Tokenizer | Before | After | Reduction | Argument tokens before / after | Unchanged result tokens |
|
|
93
|
-
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
94
|
-
| o200k_base | 17,771 | 3,587 | **79.82%** | 7,685 / 643 | 1,039 |
|
|
95
|
-
| cl100k_base | 17,595 | 3,533 | **79.92%** | 7,619 / 637 | 1,007 |
|
|
96
|
-
|
|
97
|
-
Accounting is `2*definition + 2*arguments + complete result`. It includes both
|
|
98
|
-
requests' standing definitions, generated arguments and their replay. The complete
|
|
99
|
-
normalized output equals the pre-feature snapshot byte-for-byte, and the contents
|
|
100
|
-
and complete inventory of all 32 files are checked. Only run counters and elapsed
|
|
101
|
-
times are normalized, as in the existing benchmark. No hidden helper files appear.
|
|
102
|
-
|
|
103
|
-
The new gate requires **at least 78% on both tokenizers**, not an argument-only
|
|
104
|
-
percentage. Its fixture is
|
|
105
|
-
[`batch-reuse-baseline.json`](https://github.com/AdityaVG13/pi-stack/blob/main/packages/pi-supernova/tests/efficiency/batch-reuse-baseline.json),
|
|
106
|
-
with SHA-256 `589960c417006630872c16fae9ae5be4f5cdbdddbf86dda9fd9377651352055a`.
|
|
107
|
-
|
|
108
|
-
This workload benefits from substantial repeated source and input. Unique-input,
|
|
109
|
-
large-result, or already-file-backed jobs need not see that gain. Saving a reusable
|
|
110
|
-
program/input file was already an alternative when extra workspace artifacts and
|
|
111
|
-
setup were acceptable; this is not a claim of 80% over that different workflow.
|
|
112
|
-
|
|
113
|
-
### Existing workload stays separate
|
|
114
|
-
|
|
115
|
-
The original six-call schedule, its argument hash, all 13 logical operations,
|
|
116
|
-
failures, decision boundaries and complete outputs remain unchanged. It already
|
|
117
|
-
uses saved programs and does not exercise the new source defaults. Clearer shorter
|
|
118
|
-
standing guidance offsets the new option's schema cost:
|
|
119
|
-
|
|
120
|
-
| Tokenizer | At start of this pass | Now | Additional reduction | Versus original non-batched baseline |
|
|
121
|
-
| --- | ---: | ---: | ---: | ---: |
|
|
122
|
-
| o200k_base | 10,191 | 9,841 | **3.43%** | 65.02% |
|
|
123
|
-
| cl100k_base | 10,067 | 9,724 | **3.41%** | 65.07% |
|
|
124
|
-
|
|
125
|
-
The definition measures 631/626 tokens, versus 681/675 at the start. The original
|
|
126
|
-
traffic gate was strengthened from 40% to 65%, without changing its baseline. An
|
|
127
|
-
initial shorter guidance draft failed the existing recovery/surface checks; the
|
|
128
|
-
copyable offset recovery and explicit selector/view guidance were restored rather
|
|
129
|
-
than removing those requirements. There is no claim of 70--80% on this workload.
|
|
130
|
-
|
|
131
|
-
### Verification and boundaries
|
|
132
|
-
|
|
133
|
-
New source/overlay regressions and the reuse gate failed before implementation.
|
|
134
|
-
The strengthened existing traffic gate also failed first. Current verification:
|
|
135
|
-
239 package tests, both tokenizers' gates, lint, actual Pi argument validation and
|
|
136
|
-
TUI smoke, actual network-denied OMP execution, and 71 targeted tests on an isolated
|
|
137
|
-
Spark copy under Node v24.16.0. The actual-host probes exercise shared source,
|
|
138
|
-
object overlays and independent guest copies.
|
|
139
|
-
|
|
140
|
-
The existing stress runner passed 33,024 independent reads, 20 commits/140 conflicts,
|
|
141
|
-
cancellation, complete-source fidelity and program batches. On Apple M5 Max,
|
|
142
|
-
Node v26.7.0, the 300-sample, eight-file check measured pristine-warm median/p95
|
|
143
|
-
1.645/2.390 ms versus unbatched-cold p95 13.848 ms; its existing acceptance check
|
|
144
|
-
passed. This is a runtime smoke measurement, **not a new end-to-end speedup claim**.
|
|
145
|
-
|
|
146
|
-
Complete-output and file equality prove the tested contracts, not unchanged
|
|
147
|
-
live-model quality. There was no provider/model A/B evaluation, no reasoning-budget
|
|
148
|
-
change, and no compression feature enabled. Deployment, session restart and
|
|
149
|
-
publication are separate; installed OMP packages were not changed by this pass.
|
|
150
|
-
|
|
151
|
-
## Unreleased optimization pass (2026-09-19)
|
|
152
|
-
|
|
153
|
-
Measured against the working tree immediately before this pass, **including the
|
|
154
|
-
unreleased papercut fixes**, not against a published release or the older baselines
|
|
155
|
-
below. Machine: Apple M5 Max, 18 CPUs, 48 GiB RAM, macOS arm64, Node v26.7.0.
|
|
156
|
-
|
|
157
|
-
| Measurement | Before | After | Reduction |
|
|
158
|
-
| --- | ---: | ---: | ---: |
|
|
159
|
-
| Definition tokens, o200k_base | 699 | 658 | 5.87% |
|
|
160
|
-
| Definition tokens, cl100k_base | 691 | 651 | 5.79% |
|
|
161
|
-
| Fixed six-call traffic, o200k_base | 10,317 | 10,030 | 2.78% |
|
|
162
|
-
| Fixed six-call traffic, cl100k_base | 10,179 | 9,899 | 2.75% |
|
|
163
|
-
| 200-row report packaging, median | 0.1665 ms | 0.1314 ms | 21.09% |
|
|
164
|
-
| Nested source packaging, median | 0.0366 ms | 0.0240 ms | 34.53% |
|
|
165
|
-
|
|
166
|
-
A subsequent Cortex papercut follow-up adds explicit data-limit, workspace-write,
|
|
167
|
-
and inherited-timeout guidance. That later definition measures 674/667 tokens
|
|
168
|
-
(o200k_base/cl100k_base), with fixed-workload totals of 10,142/10,011. The table
|
|
169
|
-
above records the optimization checkpoint, before that additional guidance.
|
|
170
|
-
|
|
171
|
-
The Mac/Spark recheck adds optional-read recovery and removes the unsupported BMP
|
|
172
|
-
attachment advertisement. The resulting definition is 681/675 tokens, with fixed
|
|
173
|
-
traffic totals of 10,191/10,067. Both frozen-workload gates still pass. Verification
|
|
174
|
-
now includes 235 Mac package tests, 51 focused tests on an isolated Spark source
|
|
175
|
-
copy (Node v24.16.0), and actual Pi/OMP smoke on Mac. Installed OMP packages were
|
|
176
|
-
not updated by either follow-up.
|
|
177
|
-
|
|
178
|
-
Token savings come only from shorter standing guidance. Arguments, decision
|
|
179
|
-
boundaries and complete logical result text remain unchanged. No compression,
|
|
180
|
-
elision, lower output limits, or history rewriting was introduced or enabled.
|
|
181
|
-
|
|
182
|
-
Packaging now copies only changed branches and skips an escaped-source rendering
|
|
183
|
-
when the existing complete raw rendering is provably shorter. The benchmark runs
|
|
184
|
-
10,000 measured iterations per payload after warmup. Complete packaged-output
|
|
185
|
-
SHA-256 hashes match before/after, including typed values, emitted text, images,
|
|
186
|
-
logs and truncation flags. Local acceptance: identical hashes and at least 10%
|
|
187
|
-
lower median packaging time for both fixtures; this is not a timing-sensitive CI
|
|
188
|
-
gate or an end-to-end agent speed claim.
|
|
189
|
-
|
|
190
|
-
Successful runs no longer leave a 250 ms drain timer alive; early completion of
|
|
191
|
-
pending calls clears its fallback timer. A cold child-process probe exited at
|
|
192
|
-
267 ms before versus 17 ms after, while result delivery itself remained about
|
|
193
|
-
17 ms in both. Cancellation and the 250 ms bound on stuck calls remain tested.
|
|
194
|
-
The 300-sample, eight-file benchmark measured prewarmed median/p95 of
|
|
195
|
-
1.904/2.743 ms before and 1.765/2.530 ms after. These filesystem timings are noisy;
|
|
196
|
-
no generalized I/O speedup is claimed, and prewarming/model latency is excluded.
|
|
197
|
-
|
|
198
|
-
Reproduce the local latency and stress checks:
|
|
199
|
-
|
|
200
|
-
~~~sh
|
|
201
|
-
SUPERNOVA_MEASURE_SAMPLES=300 npm run measure --prefix packages/pi-supernova
|
|
202
|
-
node packages/pi-supernova/tests/efficiency/stress.mjs
|
|
203
|
-
~~~
|
|
204
|
-
|
|
205
|
-
Verification: 229 package tests, both tokenizers' existing frozen-workload gates,
|
|
206
|
-
lint, actual Pi loader/TUI and OMP execution smoke tests, plus the stress runner
|
|
207
|
-
(33,024 independent reads, write contention, cancellation and source fidelity).
|
|
208
|
-
No version bump, installation update, commit or publication is part of this pass.
|
|
209
|
-
|
|
210
|
-
## Workload
|
|
211
|
-
|
|
212
|
-
The baseline is a frozen, non-batched implementation snapshot with program-file
|
|
213
|
-
reuse and nested source framing already enabled. It is not a comparison against
|
|
214
|
-
a separately published release.
|
|
215
|
-
|
|
216
|
-
| Scenario | Baseline calls | Batched calls |
|
|
37
|
+
The mixed-workload baseline is a frozen non-batched snapshot of the same
|
|
38
|
+
programs, not a separate product. All rows below deliver identical complete
|
|
39
|
+
outputs and files; only call and argument placement change.
|
|
40
|
+
|
|
41
|
+
Formally, with definition cost D, generated argument cost A_i, result cost R_i,
|
|
42
|
+
and prior tool-history cost H_i = sum(A_j + R_j) for j < i:
|
|
43
|
+
|
|
44
|
+
```text
|
|
45
|
+
Total = sum(D + A_i + H_i, i = 1..N) + D + H_(N+1)
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
### Mixed 13-program replay
|
|
49
|
+
|
|
50
|
+
Fixed inspect / reproduce / repair / verify, JSON report update, and five-term
|
|
51
|
+
audit. Full tool-history replay.
|
|
52
|
+
|
|
53
|
+
| Scenario | Unbatched calls | 0.11.0 calls |
|
|
217
54
|
| --- | ---: | ---: |
|
|
218
55
|
| Inspect source, reproduce a failure, repair and verify | 4 | 3 |
|
|
219
56
|
| Inspect, update and verify a JSON report | 3 | 2 |
|
|
220
57
|
| Create an audit program and run it for five search terms | 6 | 1 |
|
|
221
58
|
| **Total** | **13** | **6** |
|
|
222
59
|
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
or contain benchmark-specific paths, search terms, or expected results.
|
|
228
|
-
|
|
229
|
-
This workload deliberately includes repeated audits that benefit from batching.
|
|
230
|
-
It is a reproducible regression fixture, not a representative sample of all agent
|
|
231
|
-
tasks or a guarantee of savings when a model decides how to use the tool.
|
|
60
|
+
| Tokenizer | Unbatched | Supernova 0.11.0 | Reduction |
|
|
61
|
+
| --- | ---: | ---: | ---: |
|
|
62
|
+
| o200k_base | 28,130 | 9,819 | 65.09% |
|
|
63
|
+
| cl100k_base | 27,841 | 9,657 | 65.31% |
|
|
232
64
|
|
|
233
|
-
|
|
65
|
+
Same six-call schedule versus frozen revision `d444eb7`: 18,535 -> 9,819
|
|
66
|
+
(47.02%) on o200k_base and 18,310 -> 9,657 (47.26%) on cl100k_base. The gate
|
|
67
|
+
requires at least another 19% on unchanged programs and results, and at most
|
|
68
|
+
35% of unbatched traffic.
|
|
234
69
|
|
|
235
|
-
|
|
70
|
+
By group, o200k_base:
|
|
236
71
|
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
72
|
+
| Group | Unbatched calls | Unbatched tokens | 0.11.0 calls | 0.11.0 tokens |
|
|
73
|
+
| --- | ---: | ---: | ---: | ---: |
|
|
74
|
+
| repair | 4 | 4,242 | 3 | 2,245 |
|
|
75
|
+
| report | 3 | 4,231 | 2 | 2,253 |
|
|
76
|
+
| reuse | 6 | 15,807 | 1 | 1,643 |
|
|
77
|
+
| final handoff | 0 | 3,850 | 0 | 3,678 |
|
|
240
78
|
|
|
241
|
-
|
|
242
|
-
request. A newly produced tool result is counted when the next model request
|
|
243
|
-
consumes it, not twice. Program creation is included. The complete startup
|
|
244
|
-
reference is retained, so this workload needs no separate discovery call.
|
|
79
|
+
By group, cl100k_base:
|
|
245
80
|
|
|
246
|
-
|
|
247
|
-
|
|
81
|
+
| Group | Unbatched calls | Unbatched tokens | 0.11.0 calls | 0.11.0 tokens |
|
|
82
|
+
| --- | ---: | ---: | ---: | ---: |
|
|
83
|
+
| repair | 4 | 4,211 | 3 | 2,206 |
|
|
84
|
+
| report | 3 | 4,197 | 2 | 2,217 |
|
|
85
|
+
| reuse | 6 | 15,632 | 1 | 1,615 |
|
|
86
|
+
| final handoff | 0 | 3,801 | 0 | 3,619 |
|
|
248
87
|
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
88
|
+
Standing tool definition per request: 908 -> 645 tokens (o200k_base) and
|
|
89
|
+
901 -> 633 (cl100k_base). That serialized definition is resent on every model
|
|
90
|
+
request. Source and result text are not compressed.
|
|
252
91
|
|
|
253
|
-
|
|
254
|
-
breakdowns. Its results field records newly produced text for inspection; that
|
|
255
|
-
text contributes to totals through later history, not as a second charge.
|
|
92
|
+
### Shared source and data
|
|
256
93
|
|
|
257
|
-
|
|
94
|
+
16 independent programs, 32 final files, one model call. Complete outputs match.
|
|
95
|
+
Accounting is `2*definition + 2*arguments + complete result`.
|
|
258
96
|
|
|
259
|
-
|
|
97
|
+
| Tokenizer | Repeated arguments | Shared defaults | Reduction |
|
|
98
|
+
| --- | ---: | ---: | ---: |
|
|
99
|
+
| o200k_base | 17,771 | 3,487 | 80.38% |
|
|
100
|
+
| cl100k_base | 17,595 | 3,435 | 80.48% |
|
|
260
101
|
|
|
261
|
-
|
|
262
|
-
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
263
|
-
| o200k_base | 28,130 | 18,535 | 9,843 | **46.90%** | **65.01%** |
|
|
264
|
-
| cl100k_base | 27,841 | 18,310 | 9,726 | **46.88%** | **65.07%** |
|
|
102
|
+
Eight programs sharing a 48-path input, complete outputs equal:
|
|
265
103
|
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
batched baseline from commit d444eb7. Its six-call argument hash is pinned as well
|
|
271
|
-
as the original workload: removing a decision boundary cannot satisfy this gate.
|
|
104
|
+
| Tokenizer | Repeated arguments | Shared defaults | Reduction |
|
|
105
|
+
| --- | ---: | ---: | ---: |
|
|
106
|
+
| o200k_base | 16,309 | 5,527 | 66.11% |
|
|
107
|
+
| cl100k_base | 14,649 | 5,211 | 64.43% |
|
|
272
108
|
|
|
273
|
-
|
|
109
|
+
These are not a claim of 80% savings on every task. Shared defaults require
|
|
110
|
+
equal complete result text in both arms after only run-metadata normalization.
|
|
274
111
|
|
|
275
|
-
|
|
276
|
-
cl100k_base, versus 908 and 901 in the frozen non-batched baseline. It retains
|
|
277
|
-
command signatures, complete-read and JSON limits, array-read failure rules,
|
|
278
|
-
transaction boundaries, batch defaults and edit/view guidance on every request.
|
|
112
|
+
### File-program reuse (arguments only)
|
|
279
113
|
|
|
280
|
-
|
|
114
|
+
Five later invocations of a saved program versus five inline copies, plus one
|
|
115
|
+
setup write. Break-even is two executions on both encodings.
|
|
281
116
|
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
117
|
+
| Tokenizer | Five inline | Five file + setup | Saved argument tokens |
|
|
118
|
+
| --- | ---: | ---: | ---: |
|
|
119
|
+
| o200k_base | 580 | 249 | 331 |
|
|
120
|
+
| cl100k_base | 570 | 242 | 328 |
|
|
285
121
|
|
|
286
|
-
|
|
287
|
-
framing and every attempted program's text are counted too. One-off calls should
|
|
288
|
-
not be assumed to benefit from batching.
|
|
122
|
+
### Component formatting
|
|
289
123
|
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
argument-only break-even excludes other request costs. Text framing is selected
|
|
293
|
-
by character length, not a runtime tokenizer; it need not reduce tokens for every
|
|
294
|
-
input or encoding.
|
|
124
|
+
Separate from the traffic gate. Lossless text framing on nested multiline
|
|
125
|
+
source; compact scalars and short line lists are unchanged.
|
|
295
126
|
|
|
296
|
-
|
|
127
|
+
| Fixture | o200k before | o200k after | o200k saved | cl100k before | cl100k after | cl100k saved |
|
|
128
|
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
129
|
+
| jsonSource | 1,697 | 1,488 | 12.32% | 1,680 | 1,468 | 12.62% |
|
|
130
|
+
| runtimeSource | 4,406 | 3,862 | 12.35% | 4,379 | 3,834 | 12.45% |
|
|
131
|
+
| nestedSources | 6,115 | 5,348 | 12.54% | 6,071 | 5,300 | 12.70% |
|
|
132
|
+
| compactScalars | 17 | 17 | 0% | 16 | 16 | 0% |
|
|
133
|
+
| shortLines | 17 | 17 | 0% | 17 | 17 | 0% |
|
|
297
134
|
|
|
298
|
-
|
|
299
|
-
arm repeats the literal `data` in every entry; the after arm supplies it once at
|
|
300
|
-
the top level. Both execute the same programs and return the same complete source
|
|
301
|
-
strings and typed results. The before request fits the existing admission cap;
|
|
302
|
-
this is not a comparison against a hypothetical request that could never run.
|
|
135
|
+
## Local worker speed
|
|
303
136
|
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
137
|
+
`npm run measure --prefix packages/pi-supernova`. 200 measured waves, 8 files
|
|
138
|
+
per wave. Identical full per-file results; coalesced arms use 1 bridge call
|
|
139
|
+
instead of 8. Pristine-warm p95 is below unbatched cold. Local filesystem and
|
|
140
|
+
worker only: excludes prewarm time, model latency, and provider tokens.
|
|
141
|
+
Character counts are not token counts. Max is worst observed, not a real-time
|
|
142
|
+
guarantee.
|
|
308
143
|
|
|
309
|
-
|
|
144
|
+
| Arm | p50 ms | p95 ms | Bridge calls |
|
|
145
|
+
| --- | ---: | ---: | ---: |
|
|
146
|
+
| unbatched CodeMode cold | 17.69 | 19.75 | 8 |
|
|
147
|
+
| coalesced CodeMode cold | 20.09 | 26.56 | 1 |
|
|
148
|
+
| coalesced CodeMode pristine-warm | 2.31 | 3.14 | 1 |
|
|
310
149
|
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
~~~
|
|
150
|
+
Coalesced cold is not faster than unbatched cold on this machine. The cold cost
|
|
151
|
+
is worker spawn; warm reuse is the speed win.
|
|
314
152
|
|
|
315
|
-
|
|
316
|
-
charged on handoff. The standing guidance measures 602 definition tokens per
|
|
317
|
-
request with o200k_base (595 with cl100k_base), and that cost is included in
|
|
318
|
-
the after totals.
|
|
319
|
-
There is no source compression, result elision, hidden output or lost decision
|
|
320
|
-
boundary. Programs receive fresh data copies, not a shared mutable heap.
|
|
153
|
+
Packaging, 10,000 iterations:
|
|
321
154
|
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
155
|
+
| Fixture | p50 ms | p95 ms | chars |
|
|
156
|
+
| --- | ---: | ---: | ---: |
|
|
157
|
+
| 200-row report | 0.163 | 0.171 | 10,474 |
|
|
158
|
+
| nested source | 0.037 | 0.040 | 8,046 |
|
|
326
159
|
|
|
327
160
|
## Benchmark integrity
|
|
328
161
|
|
|
329
|
-
The [baseline fixture](https://github.com/AdityaVG13/pi-stack/blob/main/packages/pi-supernova/tests/efficiency/token-baseline.json)
|
|
330
|
-
inputs, definition, workload hash, arguments and complete
|
|
331
|
-
|
|
332
|
-
|
|
162
|
+
The [baseline fixture](https://github.com/AdityaVG13/pi-stack/blob/main/packages/pi-supernova/tests/efficiency/token-baseline.json)
|
|
163
|
+
contains the source inputs, definition, workload hash, arguments and complete
|
|
164
|
+
outputs. The
|
|
165
|
+
[runner](https://github.com/AdityaVG13/pi-stack/blob/main/packages/pi-supernova/tests/efficiency/workflow.mjs)
|
|
166
|
+
executes the registered tool with real workers and filesystem operations.
|
|
333
167
|
|
|
334
168
|
- The workload hash matches the frozen baseline, and the six-call argument hash
|
|
335
169
|
matches the prior batched execution.
|
|
336
|
-
- Every original argument and expected failure matches
|
|
337
|
-
match the explicit current newline contract described below.
|
|
170
|
+
- Every original argument and expected failure matches.
|
|
338
171
|
- Batched output contains every original result, with no unaccounted outer text.
|
|
339
172
|
- No result is truncated, and final repaired source and JSON contents match.
|
|
340
173
|
|
|
341
|
-
Only run IDs, elapsed times and temporary workspace prefixes in write receipts
|
|
342
|
-
are normalized.
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
by the benchmark and is checked against its SHA-256 before execution.
|
|
346
|
-
|
|
347
|
-
Contract v2 preserves the terminating newline of one selected source line. The
|
|
348
|
-
runner derives that single expected-output correction from the frozen input, not
|
|
349
|
-
from candidate output. The historical fixture and its traffic counts stay
|
|
350
|
-
untouched; programs, arguments, failures and decision boundaries are unchanged.
|
|
351
|
-
The shared-input comparison separately requires equal complete result text in
|
|
352
|
-
both arms, after only run-metadata normalization.
|
|
353
|
-
|
|
354
|
-
Contract v3 reports write receipts relative to the workspace (`wrote rel/path`,
|
|
355
|
-
matching the long-standing `edited <rel>` form) instead of absolute paths. The
|
|
356
|
-
runner strips the frozen `/workspace/` prefix from baseline outputs; the
|
|
357
|
-
historical fixture and its traffic counts stay untouched. Temporary workspace
|
|
358
|
-
prefixes no longer appear in live receipts, so that normalization only applies
|
|
359
|
-
to the frozen baseline side.
|
|
360
|
-
|
|
361
|
-
The README and these docs ship in the npm tarball. Benchmarks and test fixtures
|
|
362
|
-
remain in the GitHub checkout, so their links above use GitHub URLs.
|
|
174
|
+
Only run IDs, elapsed times, and temporary workspace prefixes in write receipts
|
|
175
|
+
are normalized. The workload, batch schedule, recorded comparison totals, and
|
|
176
|
+
acceptance thresholds are fixed test inputs, not production execution rules.
|
|
177
|
+
The baseline is not regenerated by the benchmark.
|
|
363
178
|
|
|
364
179
|
Recorded baseline SHA-256:
|
|
365
180
|
|
|
366
|
-
|
|
181
|
+
```text
|
|
367
182
|
96964f990f481ac05afaefdd02001bd15f61835a8381349a61e38c06209d7508
|
|
368
|
-
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
Shared-source reuse fixture SHA-256:
|
|
186
|
+
|
|
187
|
+
```text
|
|
188
|
+
589960c417006630872c16fae9ae5be4f5cdbdddbf86dda9fd9377651352055a
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
Citation elision (`seenWindow`) is experimental, disabled by default, and is
|
|
192
|
+
not the source of the savings above.
|
|
193
|
+
|
|
194
|
+
The README and these docs ship in the npm tarball. Benchmarks and test fixtures
|
|
195
|
+
remain in the GitHub checkout, so their links above use GitHub URLs.
|
|
369
196
|
|
|
370
197
|
## Limitations
|
|
371
198
|
|
|
372
199
|
- These are local estimates of model tool-token traffic, not provider billing.
|
|
373
200
|
Provider-specific envelopes, unrelated user/system messages, reasoning tokens,
|
|
374
|
-
image token costs and cache discounts are excluded.
|
|
201
|
+
image token costs, and cache discounts are excluded.
|
|
375
202
|
- There is no live-model A/B quality evaluation. Preserving observations and
|
|
376
203
|
reasoning settings does not establish unchanged end-to-end task quality.
|
|
377
|
-
- Batching is appropriate only for already-chosen continuations. Actions
|
|
378
|
-
a new model decision must remain separate calls.
|
|
379
|
-
-
|
|
380
|
-
default and is not the source of the non-compressive savings reported here.
|
|
381
|
-
- Existing read, output, log, image and execution limits still apply. The
|
|
204
|
+
- Batching is appropriate only for already-chosen continuations. Actions that
|
|
205
|
+
need a new model decision must remain separate calls.
|
|
206
|
+
- Existing read, output, log, image, and execution limits still apply. The
|
|
382
207
|
benchmark does not obtain savings by lowering them or hiding truncation.
|
|
383
208
|
|
|
384
209
|
## Design references
|
|
385
210
|
|
|
386
|
-
These sources informed the implementation choices; the measurements above
|
|
387
|
-
Supernova implementations only, not the performance of these packages.
|
|
211
|
+
These sources informed the implementation choices; the measurements above
|
|
212
|
+
compare Supernova implementations only, not the performance of these packages.
|
|
388
213
|
|
|
389
214
|
| Reference | Mechanisms studied |
|
|
390
215
|
| --- | --- |
|
|
@@ -396,6 +221,6 @@ Supernova implementations only, not the performance of these packages.
|
|
|
396
221
|
| [pi-cache-optimizer 2.8.2](https://github.com/jiangge/pi-cache-optimizer) | Cache-prefix and prompt handling |
|
|
397
222
|
|
|
398
223
|
Unpinned repository links refer to inspected source, not independently verified
|
|
399
|
-
package versions. Supernova retains fresh guests, explicit transaction
|
|
400
|
-
and its general command surface rather than adopting persistent
|
|
401
|
-
restricted CLI catalogs or provider/history rewriting.
|
|
224
|
+
package versions. Supernova retains fresh guests, explicit transaction
|
|
225
|
+
boundaries, and its general command surface rather than adopting persistent
|
|
226
|
+
notebook state, restricted CLI catalogs, or provider/history rewriting.
|