@cosyte/bench 0.1.2 → 0.1.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +47 -0
- package/README.md +57 -14
- package/dist/child-entry.mjs +8 -0
- package/dist/child-entry.mjs.map +1 -1
- package/dist/cli.mjs +168 -47
- package/dist/cli.mjs.map +1 -1
- package/dist/index.d.ts +111 -2
- package/dist/index.mjs +50 -18
- package/dist/index.mjs.map +1 -1
- package/package.json +2 -2
- package/schema/results-v1.schema.json +100 -45
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,52 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.1.4
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- c854d59: `@cosyte/script-utils`: the attw gate (`cosyte-attw`, `runAttwGate` and `runAttwCli`) now starts on
|
|
8
|
+
Windows. It used to check `node_modules/.bin/attw` and start it without a shell; on Windows that file
|
|
9
|
+
is a POSIX shell script, so the start failed with ENOENT, and npm and pnpm, which the gate also
|
|
10
|
+
starts, are `.cmd` files there that Node will not start without a shell. On Windows the gate now
|
|
11
|
+
checks and starts `node_modules/.bin/attw.cmd`, and starts all three tools through `cmd.exe` with
|
|
12
|
+
every argument and every path on the command line in double quotes, a trailing run of backslashes
|
|
13
|
+
doubled so each argument arrives whole. Elsewhere nothing changes. Two new exports,
|
|
14
|
+
`attwBinaryPath(callerUrl, platform?)` and `toolCommand(file, args, platform?)`, say which binary and
|
|
15
|
+
which command line. The gate now refuses an argument carrying a double quote, a percent sign or a
|
|
16
|
+
line break, on every system: no argument it accepts needs one. A repository that skipped the attw
|
|
17
|
+
step on Windows can drop the skip once it takes this version. Callers change nothing.
|
|
18
|
+
|
|
19
|
+
`@cosyte/bench`: the README now says what the small-input memory budget gives up: a cost in fewer
|
|
20
|
+
than half of a small case's 20 calls is not bounded, the rule switches at a hard 64 KiB edge, and
|
|
21
|
+
the empty-call floor's highest call is not a fixed figure (usually about 29 KiB, once 369,888
|
|
22
|
+
bytes) while its median is stable at about 3.4 KiB. No behavior changes.
|
|
23
|
+
|
|
24
|
+
## 0.1.3
|
|
25
|
+
|
|
26
|
+
### Patch Changes
|
|
27
|
+
|
|
28
|
+
- 027dbf5: A reading measures the empty-call floor once per run: the peak of a call that does nothing, read in
|
|
29
|
+
its own process through the same probe as the cases, which is what the probe itself costs. It is in
|
|
30
|
+
the reading as `memoryFloor`, the Markdown prints it, and every case's peak also gives each figure
|
|
31
|
+
above the floor's median, never below zero: `peakTotalAboveFloorBytes`,
|
|
32
|
+
`medianPeakTotalAboveFloorBytes`, and the same for the heap and external memory.
|
|
33
|
+
|
|
34
|
+
`budgetPeakBytes(caseReading)` is the peak a memory budget reads. For a case whose inputs average
|
|
35
|
+
under `BENCH_CONTRACT.FLOOR_INPUT_BYTES` (64 KiB) it is the median call's total above the floor's
|
|
36
|
+
median; for any other, or for a suite that gave no byte count, the highest raw `peakTotalBytes`, as
|
|
37
|
+
before. On a GitHub-hosted runner the floor's median call is about 3.3 KiB and its highest about
|
|
38
|
+
28 KiB, and that highest lands in a small case's highest peak, so a small input reads its median
|
|
39
|
+
call. A budget script that read `peakTotalBytes` should call `budgetPeakBytes` instead, and a suite
|
|
40
|
+
that added its own empty "floor" case can drop it.
|
|
41
|
+
|
|
42
|
+
One case that cannot be measured no longer stops the run. `cosyte-bench run` writes every other case,
|
|
43
|
+
lists what failed in the reading's `failures` (the phase, `timing`, `memory` or `floor`, the case,
|
|
44
|
+
and the first line of the error) and in the Markdown, prints the whole error to stderr, and exits 2.
|
|
45
|
+
A case whose timing failed is left out of `cases`; one whose memory failed keeps its timing. A script
|
|
46
|
+
that ran each case alone to get past one failure can run them together and read `failures`.
|
|
47
|
+
|
|
48
|
+
Every new field is added within results version 1, and documents written before still validate.
|
|
49
|
+
|
|
3
50
|
## 0.1.2
|
|
4
51
|
|
|
5
52
|
### Patch Changes
|
package/README.md
CHANGED
|
@@ -98,14 +98,21 @@ cosyte-bench check bench.json
|
|
|
98
98
|
Exit codes: 0 clean, 1 a finding (a regression, or a results file that does not match the schema),
|
|
99
99
|
2 could not measure (a skip, a process that died, an input it could not read). A skip is not a pass.
|
|
100
100
|
|
|
101
|
+
A `run` whose measuring process fails for one case still measures and writes every other case. The
|
|
102
|
+
reading lists what failed in `failures` (the phase, `timing`, `memory` or `floor`, the case, and the
|
|
103
|
+
first line of the error; the whole error goes to stderr) and the command exits 2. A case whose timing
|
|
104
|
+
failed is left out of `cases`; one whose memory failed keeps its timing with its memory `null`. A
|
|
105
|
+
case whose warmup never settled is not a failure: it is in `cases`, its throughput `skipped` with
|
|
106
|
+
the reason `warmup-unstable`.
|
|
107
|
+
|
|
101
108
|
## Entry points
|
|
102
109
|
|
|
103
|
-
| entry point | what it is
|
|
104
|
-
| --------------------------------------------- |
|
|
105
|
-
| `cosyte-bench` | the bin: `run`, `compare`, `calibrate` and `check`
|
|
106
|
-
| `@cosyte/bench` | writing a suite (`defineSuite`, `defineBenchmark`), judging ratios (`judge`), reading results (`readResults`, `RESULTS_SCHEMA`), and `BENCH_CONTRACT` |
|
|
107
|
-
| `@cosyte/bench/schema/results-v1.schema.json` | the results schema as JSON Schema, for a tool in another language
|
|
108
|
-
| `@cosyte/bench/package.json` | the manifest
|
|
110
|
+
| entry point | what it is |
|
|
111
|
+
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
112
|
+
| `cosyte-bench` | the bin: `run`, `compare`, `calibrate` and `check` |
|
|
113
|
+
| `@cosyte/bench` | writing a suite (`defineSuite`, `defineBenchmark`), judging ratios (`judge`), reading results (`readResults`, `RESULTS_SCHEMA`, `budgetPeakBytes`), and `BENCH_CONTRACT` |
|
|
114
|
+
| `@cosyte/bench/schema/results-v1.schema.json` | the results schema as JSON Schema, for a tool in another language |
|
|
115
|
+
| `@cosyte/bench/package.json` | the manifest |
|
|
109
116
|
|
|
110
117
|
The module entry point has no side effects: importing it starts no process. The schema is plain JSON:
|
|
111
118
|
|
|
@@ -137,19 +144,53 @@ the command line:
|
|
|
137
144
|
|
|
138
145
|
## What it measures
|
|
139
146
|
|
|
140
|
-
| Measure | How
|
|
141
|
-
| -------------------------------------- |
|
|
142
|
-
| Throughput, small to very large inputs | Per size, fresh processes; a warmup by the scaling gate's rule, or by the large-case plan below when every warmup batch is one long pass; 5 samples of whole passes of at least 50 ms; the median of the processes' medians
|
|
143
|
-
| Latency p50, p95, p99 | One call at a time on a nanosecond clock, up to 2 s or 5,000 calls; a percentile is reported only with at least 10 calls above it
|
|
144
|
-
| Allocations per operation | Heap growth over windows of `n` and `2n` operations in which no collection ran, in a process with a 64 MiB young generation; the difference cancels the cost of reading the heap. When every window collects, the largest pair is counted with what its collections reclaimed (`method: "gc-accounted"`)
|
|
145
|
-
| Peak memory | Per call, the higher of what was in use just before any collection during the call and at its end, above what was in use before it: the V8 heap (`peakHeapBytes`), external memory such as Buffers and ArrayBuffers (`peakExternalBytes`), and the two together (`peakTotalBytes`)
|
|
146
|
-
| Heap flat over a long run | 30 rounds, the heap settled after each; a Theil-Sen slope over the last 24; growing when the fitted growth reaches 64 KiB. On the long-run size and on the largest size whose 30 rounds fit in 60 s
|
|
147
|
-
| Cold import and loaded bytes | 20 fresh processes import the target, 20 import an empty module, interleaved; one more records every file the import loads, with its gzip size
|
|
147
|
+
| Measure | How | What it cannot see |
|
|
148
|
+
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
|
|
149
|
+
| Throughput, small to very large inputs | Per size, fresh processes; a warmup by the scaling gate's rule, or by the large-case plan below when every warmup batch is one long pass; 5 samples of whole passes of at least 50 ms; the median of the processes' medians | Anything the suite's sizes do not cover; numbers from another machine |
|
|
150
|
+
| Latency p50, p95, p99 | One call at a time on a nanosecond clock, up to 2 s or 5,000 calls; a percentile is reported only with at least 10 calls above it | Calls shorter than 20 clock reads (skipped, typed); the tail of a size with few calls |
|
|
151
|
+
| Allocations per operation | Heap growth over windows of `n` and `2n` operations in which no collection ran, in a process with a 64 MiB young generation; the difference cancels the cost of reading the heap. When every window collects, the largest pair is counted with what its collections reclaimed (`method: "gc-accounted"`) | Native memory a library allocates outside V8's accounting |
|
|
152
|
+
| Peak memory | Per call, the higher of what was in use just before any collection during the call and at its end, above what was in use before it: the V8 heap (`peakHeapBytes`), external memory such as Buffers and ArrayBuffers (`peakExternalBytes`), and the two together (`peakTotalBytes`); each also above the run's empty-call floor (`peakTotalAboveFloorBytes` and the like), below | Native memory a library allocates outside V8's accounting |
|
|
153
|
+
| Heap flat over a long run | 30 rounds, the heap settled after each; a Theil-Sen slope over the last 24; growing when the fitted growth reaches 64 KiB. On the long-run size and on the largest size whose 30 rounds fit in 60 s | A leak too small to reach 64 KiB over the run (each result says the smallest it could see, `detectableBytesPerOp`); external memory |
|
|
154
|
+
| Cold import and loaded bytes | 20 fresh processes import the target, 20 import an empty module, interleaved; one more records every file the import loads, with its gzip size | What a bundler keeps after tree shaking |
|
|
148
155
|
|
|
149
156
|
Memory measurements need `gc()`; the runner starts those processes with `--expose-gc` itself, and
|
|
150
157
|
with `--no-concurrent-array-buffer-sweeping`, so a collection frees the ArrayBuffers it found dead
|
|
151
158
|
before it ends and the external memory it reports before and after is exact.
|
|
152
159
|
|
|
160
|
+
### The empty-call floor and small inputs
|
|
161
|
+
|
|
162
|
+
Reading the peak costs memory of its own: settling the heap, the profiler that records each
|
|
163
|
+
collection, the record itself. A reading therefore measures an empty call once, in a fresh process
|
|
164
|
+
through the same probe (`memoryFloor`), and gives each peak figure both raw and above the floor's
|
|
165
|
+
median, never below zero (`peakTotalAboveFloorBytes`, `medianPeakTotalAboveFloorBytes`, and the same
|
|
166
|
+
for the heap and external memory).
|
|
167
|
+
|
|
168
|
+
On a GitHub-hosted runner the floor's median call is stable, about 3.4 KiB. Its highest call is not:
|
|
169
|
+
usually about 29 KiB (one call, the first), and once 369,888 bytes. That highest call is the
|
|
170
|
+
probe's, and it lands in a case's highest peak too, which is why a few hundred bytes of input could
|
|
171
|
+
not read under 20 times its size whatever the code did. Read the floor's median as the probe's cost;
|
|
172
|
+
its highest is not a fixed figure.
|
|
173
|
+
|
|
174
|
+
A memory budget reads `budgetPeakBytes(caseReading)`: for a case whose inputs average under
|
|
175
|
+
`BENCH_CONTRACT.FLOOR_INPUT_BYTES` (64 KiB) it is the median call's total above the floor's median,
|
|
176
|
+
the code's typical call without the probe; for any other case, or one whose suite gave no byte
|
|
177
|
+
count, it is the highest raw `peakTotalBytes`. At 64 KiB the probe's usual highest call is about
|
|
178
|
+
2% of a 20 times budget, so a large input keeps the raw highest figure, which needs nothing from
|
|
179
|
+
another process. A suite that added its own empty "floor" case can drop it.
|
|
180
|
+
|
|
181
|
+
What the rule gives up:
|
|
182
|
+
|
|
183
|
+
- **A cost in fewer than half the calls is not bounded.** Under 64 KiB the budget reads the median
|
|
184
|
+
of the case's 20 calls, so a cost that shows in fewer than half of them does not move it. In one
|
|
185
|
+
reading a small case's highest call was 292,784 bytes against a median of 38,672: the budget read
|
|
186
|
+
the median. The highest raw peak is still in the reading (`peakTotalBytes`) for anyone who looks.
|
|
187
|
+
- **The switch is a hard edge at 64 KiB.** A case whose inputs average 64 KiB or more reads its
|
|
188
|
+
highest raw total, and one a byte under reads its median call above the floor. Two sizes either
|
|
189
|
+
side of the edge can therefore read very different budgets for the same code.
|
|
190
|
+
- **The 2% is the usual highest, not every highest.** The one 369,888-byte floor call is about 28%
|
|
191
|
+
of a 20 times budget at 64 KiB, so a large case whose highest call lands on such a probe outlier
|
|
192
|
+
reads it as the code's.
|
|
193
|
+
|
|
153
194
|
### Large cases
|
|
154
195
|
|
|
155
196
|
A case whose single call takes longer than a warmup batch (50 ms) cannot settle by the scaling
|
|
@@ -203,6 +244,8 @@ CI runner image, never by host name, user or absolute path.
|
|
|
203
244
|
- `judge(ratios, threshold?, alpha?)`: the comparison's verdict for one case from its per-round ratios.
|
|
204
245
|
- `readResults(text)`: parse and check a results document; throws a `TypeError` listing every
|
|
205
246
|
problem. `RESULTS_SCHEMA` is the schema, also shipped as `schema/results-v1.schema.json`.
|
|
247
|
+
- `budgetPeakBytes(caseReading)`: the peak a memory budget reads for a case: the median call above
|
|
248
|
+
the empty-call floor for an input under 64 KiB, the highest raw total for any other.
|
|
206
249
|
- `BENCH_CONTRACT`: the frozen constants.
|
|
207
250
|
- `BenchError`: what a run that could not measure throws.
|
|
208
251
|
|
package/dist/child-entry.mjs
CHANGED
|
@@ -21,6 +21,7 @@ var BENCH_CONTRACT = Object.freeze({
|
|
|
21
21
|
ALLOC_WINDOW_MS: 100,
|
|
22
22
|
ALLOC_SEMI_SPACE_MIB: 64,
|
|
23
23
|
PEAK_CALLS: 20,
|
|
24
|
+
FLOOR_INPUT_BYTES: 64 * 1024,
|
|
24
25
|
LONG_RUN_ROUNDS: 30,
|
|
25
26
|
LONG_RUN_SETTLE_ROUNDS: 5,
|
|
26
27
|
LONG_RUN_ROUND_MS: 100,
|
|
@@ -487,10 +488,17 @@ function measureJob(job, shape, warmup, pass, call, deps) {
|
|
|
487
488
|
};
|
|
488
489
|
}
|
|
489
490
|
function measuredSomething(result) {
|
|
491
|
+
if (result.mode === "floor") return result.peak.status === "measured";
|
|
490
492
|
if (result.mode === "timing") return result.status === "measured";
|
|
491
493
|
return [result.allocations, result.peak, result.longRun].some((r) => r?.status === "measured");
|
|
492
494
|
}
|
|
493
495
|
async function runJob(job, deps = realDeps()) {
|
|
496
|
+
if (job.mode === "floor") {
|
|
497
|
+
return {
|
|
498
|
+
mode: "floor",
|
|
499
|
+
peak: deps.probe === null ? SKIPPED_NO_GC : measurePeak(() => void 0, deps.probe)
|
|
500
|
+
};
|
|
501
|
+
}
|
|
494
502
|
const suiteModule = await import(pathToFileURL(job.suite).href);
|
|
495
503
|
const suite = checkSuite(suiteModule.default);
|
|
496
504
|
const target = await import(pathToFileURL(job.target).href);
|