ostia 0.2.5 → 0.2.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +213 -1160
- package/chunk-5qtsmr3p.js +21 -0
- package/chunk-n02qf2mf.js +16 -0
- package/cli.js +107 -259
- package/index.d.ts +37 -62
- package/index.js +1 -1
- package/package.json +1 -1
- package/runner.ts +4 -5
- package/chunk-h5md141w.js +0 -22
- package/chunk-v0efhnv2.js +0 -12
package/README.md
CHANGED
|
@@ -1,21 +1,14 @@
|
|
|
1
1
|
# ostia
|
|
2
2
|
|
|
3
|
-
ostia is a profiling and benchmarking toolkit for Bun.
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
regression. `ostia ci` gates a whole `ostia.config.json`/`ostia.config.ts` of
|
|
13
|
-
workloads - subprocess commands and in-process `group()`/`task()` suites alike -
|
|
14
|
-
against a saved baseline, skipping anything whose input fingerprint hasn't changed.
|
|
15
|
-
Any task can run in its own subprocess for clean JIT/heap isolation from its
|
|
16
|
-
suite-mates, and every command has a `--format minimal` mode - a versioned JSON
|
|
17
|
-
protocol built for piping straight into an LLM agent's context (see
|
|
18
|
-
[Using ostia from an AI agent](#using-ostia-from-an-ai-agent)).
|
|
3
|
+
ostia is a profiling and benchmarking toolkit for Bun. It times subprocess commands
|
|
4
|
+
(like hyperfine) and in-process functions (like mitata), optionally captures CPU
|
|
5
|
+
profiles, heap snapshots, JIT tiers and allocation counts, and writes everything to one
|
|
6
|
+
schema-versioned JSON document (`ProfileDocument`). Two documents compare with a
|
|
7
|
+
bootstrap confidence interval and a Mann-Whitney test, with the regression threshold
|
|
8
|
+
widened to the machine's measured noise floor. `ostia ci` gates a config file of
|
|
9
|
+
workloads against a saved baseline, and `--format minimal` gives scripts and LLM agents
|
|
10
|
+
a compact JSON line protocol. The CLI is a thin wrapper over the library, so anything
|
|
11
|
+
`ostia time`/`ostia bench` do, `time()`/`bench()` do too.
|
|
19
12
|
|
|
20
13
|
Zero runtime dependencies. Requires Bun ≥ 1.4.
|
|
21
14
|
|
|
@@ -27,86 +20,94 @@ bun add ostia
|
|
|
27
20
|
|
|
28
21
|
## Quick start
|
|
29
22
|
|
|
30
|
-
|
|
23
|
+
### Time two commands
|
|
31
24
|
|
|
32
25
|
```sh
|
|
33
|
-
ostia time --samples 10
|
|
26
|
+
ostia time --samples 10 "bun fixtures/fast.ts" "bun fixtures/slow.ts"
|
|
34
27
|
```
|
|
35
28
|
|
|
36
29
|
```
|
|
37
|
-
Apple M2 · 8 cores · load
|
|
30
|
+
Apple M2 · 8 cores · load 3.6 · noise floor 0.5%
|
|
38
31
|
|
|
39
|
-
Task Median Spread Range Relative
|
|
40
|
-
|
|
41
|
-
bun fixtures/fast.ts
|
|
42
|
-
|
|
43
|
-
|
|
32
|
+
Task Median Spread Range User/Sys Relative
|
|
33
|
+
---------------------------------------------------------------------------------------------------
|
|
34
|
+
bun fixtures/fast.ts 9.38 ms 9.52 ms…9.74 ms 9.10 ms…9.76 ms 6.98 ms/3.06 ms 1.00×
|
|
35
|
+
bun fixtures/slow.ts 23.2 ms 23.3 ms…23.8 ms 23.0 ms…23.9 ms 20.9 ms/2.89 ms 2.47× slower
|
|
36
|
+
! outliers-detected
|
|
44
37
|
|
|
45
38
|
Warnings:
|
|
46
|
-
bun fixtures/
|
|
39
|
+
bun fixtures/slow.ts: 1 outlier(s) detected (1 severe, 0 mild).
|
|
47
40
|
```
|
|
48
41
|
|
|
49
|
-
The header line
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
threshold to at least the noise floor, so a change smaller than the machine's
|
|
53
|
-
own jitter is never called a regression - see
|
|
54
|
-
[Statistics](#statistics-a-real-significance-test-not-a-percentage-threshold)
|
|
55
|
-
below.
|
|
42
|
+
The header line shows the machine, its load average, and the noise floor from a ~200ms
|
|
43
|
+
reference measurement taken once per run (`--no-noise-check` skips it). Spread is
|
|
44
|
+
p75…p99; User/Sys is the median user/system CPU time per trial.
|
|
56
45
|
|
|
57
|
-
|
|
58
|
-
timing numbers above):
|
|
46
|
+
### Benchmark a function
|
|
59
47
|
|
|
60
|
-
```
|
|
61
|
-
|
|
62
|
-
|
|
48
|
+
```ts
|
|
49
|
+
// suite.ts
|
|
50
|
+
import { group, task } from "ostia"
|
|
63
51
|
|
|
52
|
+
const input = Array.from({ length: 2_000 }, (_, i) => i % 500)
|
|
53
|
+
|
|
54
|
+
group("dedupe", () => {
|
|
55
|
+
task("naive (indexOf scan, O(n²))", () => dedupeNaive(input))
|
|
56
|
+
task("Set-based (O(n))", () => [...new Set(input)])
|
|
57
|
+
})
|
|
64
58
|
```
|
|
65
|
-
Apple M2 · 8 cores · load 8.5 · noise floor 0.9%
|
|
66
59
|
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
! noisy-machine
|
|
60
|
+
```sh
|
|
61
|
+
ostia bench suite.ts
|
|
62
|
+
```
|
|
71
63
|
|
|
72
|
-
|
|
73
|
-
|
|
64
|
+
```
|
|
65
|
+
Apple M2 · 8 cores · load 3.5 · noise floor 0.4%
|
|
74
66
|
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
67
|
+
Task Median Spread Range Relative
|
|
68
|
+
------------------------------------------------------------------------------------------------
|
|
69
|
+
dedupe:
|
|
70
|
+
dedupe/naive (indexOf scan, O(n²)) 173.9 µs 180.4 µs…205.8 µs 170.1 µs…635.5 µs 7.40× slower
|
|
71
|
+
dedupe/Set-based (O(n)) 23.5 µs 24.9 µs…72.5 µs 18.7 µs…208.3 µs 1.00×
|
|
80
72
|
```
|
|
81
73
|
|
|
82
|
-
|
|
83
|
-
everywhere, no setup needed - same convention as Babel/ESLint/Jest caches). Baselines are
|
|
84
|
-
the one exception: they default to `.ostia/baselines/` at the repo root instead, since
|
|
85
|
-
they need to survive `node_modules` reinstalls between branches and CI jobs, so gitignore
|
|
86
|
-
`.ostia/` if you use `ostia ci`.
|
|
74
|
+
### Gate CI on a baseline
|
|
87
75
|
|
|
88
|
-
|
|
76
|
+
```json
|
|
77
|
+
// ostia.config.json
|
|
78
|
+
{
|
|
79
|
+
"samples": 10,
|
|
80
|
+
"workloads": [
|
|
81
|
+
{ "label": "work", "command": ["bun", "fixtures/work.ts"], "inputs": ["fixtures/**"] }
|
|
82
|
+
]
|
|
83
|
+
}
|
|
84
|
+
```
|
|
89
85
|
|
|
90
86
|
```sh
|
|
91
|
-
ostia baseline save # on known-good:
|
|
92
|
-
ostia ci # on your
|
|
87
|
+
ostia baseline save # on known-good code: writes .ostia/baselines/main.json
|
|
88
|
+
ostia ci # on your change: exit 1 on a regression
|
|
93
89
|
```
|
|
94
90
|
|
|
95
91
|
```
|
|
96
|
-
|
|
97
|
-
0
|
|
98
|
-
|
|
99
|
-
0
|
|
100
|
-
2 passed 0 regressed
|
|
92
|
+
1 workloads
|
|
93
|
+
0 cached
|
|
94
|
+
1 executed
|
|
95
|
+
0 passed 1 regressed (+44.1% median on work)
|
|
101
96
|
|
|
102
|
-
Profile CI:
|
|
97
|
+
Profile CI: ✗
|
|
98
|
+
...
|
|
99
|
+
✗ work
|
|
100
|
+
timing: +44.1% median, 95% CI [+41.4%, +45.6%], p<0.001 (regressed)
|
|
103
101
|
```
|
|
104
102
|
|
|
103
|
+
Scratch output (cache, artifacts) goes to `node_modules/.cache/ostia`. Baselines go to
|
|
104
|
+
`.ostia/baselines/` so they survive reinstalls; add `.ostia/` to `.gitignore`.
|
|
105
|
+
|
|
105
106
|
## Using ostia from an AI agent
|
|
106
107
|
|
|
107
|
-
`--format minimal`
|
|
108
|
-
|
|
109
|
-
|
|
108
|
+
`--format minimal` (on `time`, `bench`, `compare`, `report`, `ci`) prints one JSON
|
|
109
|
+
object per line on stdout and nothing else. Every line has `event` and
|
|
110
|
+
`protocolVersion: 1`. Timing values are in nanoseconds.
|
|
110
111
|
|
|
111
112
|
```sh
|
|
112
113
|
ostia time --samples 10 "bun a.ts" --format minimal
|
|
@@ -114,1194 +115,246 @@ ostia compare before.json after.json --format minimal
|
|
|
114
115
|
ostia ci --format minimal; echo $?
|
|
115
116
|
```
|
|
116
117
|
|
|
117
|
-
Every line is a JSON object with `event` and `protocolVersion: 1`:
|
|
118
|
-
|
|
119
118
|
| `event` | When | Key fields |
|
|
120
119
|
|---|---|---|
|
|
121
|
-
| `run` |
|
|
122
|
-
| `unmatched` |
|
|
123
|
-
| `summary` |
|
|
124
|
-
|
|
125
|
-
Stability: keys are never renamed or removed within `protocolVersion: 1` - only ever
|
|
126
|
-
added, so an agent that reads a field it knows keeps working as the protocol grows.
|
|
127
|
-
|
|
128
|
-
Exit codes are the same across every command that produces a verdict: `0` pass, `1`
|
|
129
|
-
at least one workload regressed (`compare`/`ci` only - `time`/`bench` never return
|
|
130
|
-
`1`), `2` a harness error (a command failed to run cleanly, nothing was compared, a
|
|
131
|
-
bad flag, a missing config/baseline). On any exit `2`, stderr's last line is one more
|
|
132
|
-
JSON object - `{ event: "error", protocolVersion: 1, code, message, data? }`, `code`
|
|
133
|
-
one of `invalid-flag` / `config-missing` / `baseline-missing` / `no-matches` /
|
|
134
|
-
`spawn-failed` / `command-failed` / `timeout` / `time-source-no-match` /
|
|
135
|
-
`document-load-failed` / `no-cpu-evidence` / `internal` - so a script doesn't have to
|
|
136
|
-
pattern-match prose to tell one failure from another. This error line (and only this
|
|
137
|
-
line) is on stderr; every `minimal`/`jsonl`/`json` line above is pure JSON on stdout,
|
|
138
|
-
nothing else mixed in.
|
|
139
|
-
|
|
140
|
-
`--format jsonl` is the same idea for the full document instead of the condensed
|
|
141
|
-
protocol above: one line per `Measurement`, plus a `document` header line, each
|
|
142
|
-
tagged `kind: "document" | "measurement"` so a consumer doesn't have to guess a
|
|
143
|
-
line's shape.
|
|
144
|
-
|
|
145
|
-
## What ostia is for
|
|
146
|
-
|
|
147
|
-
- Time subprocesses or in-process functions without a profiler attached to the timing runs.
|
|
148
|
-
- Capture CPU (`--cpu`), heap (`--heap`), or JSC JIT tiers (`profile(..., { origin: "jsc" })`)
|
|
149
|
-
as separate evidence on the same document.
|
|
150
|
-
- Diff two documents (`ostia compare`) or fail CI (`ostia ci`) with exit codes `0` / `1` / `2`.
|
|
151
|
-
- Emit files other tools already understand: collapsed stacks, Mermaid, speedscope JSON,
|
|
152
|
-
raw `.cpuprofile`.
|
|
153
|
-
|
|
154
|
-
## CLI reference
|
|
120
|
+
| `run` | One per timing measurement, every command | `workloadId`, `task`, `group?`, `params?`, `skipped?`, `unit`, `samples`, `batch`, `mean`/`median`/`stddev`/`stddevPct`/`min`/`max`/`p75`/`p99`/`mad`, `userNs`/`systemNs` (subprocess only), `relative?`, `noiseFloorPct?`, `warnings[]`, and on `compare`/`ci`: `delta: { medianPct, meanPct, verdict, pass, ci95?, pValue?, effectiveTimingPct, matched }` |
|
|
121
|
+
| `unmatched` | One per workload on only one side of `compare`/`ci` | `workloadId`, `task`, `side: "base" \| "cand"` |
|
|
122
|
+
| `summary` | Last line of `compare`/`ci` only | `command`, `matched`/`regressed`/`improved`/`unchanged`/`unmatched`, `cached`/`executed`/`failed`/`missingBaseline` (`ci`), `geomeanPct`, `effectiveTimingPct`, `noiseFloorPct?`, `baseline?` (`ci`), `git?`, `exportedTo?`, `verdict`, `exitCode` |
|
|
155
123
|
|
|
156
124
|
```
|
|
157
|
-
|
|
158
|
-
ostia
|
|
159
|
-
ostia compare <a> <b> diff two ProfileDocuments
|
|
160
|
-
ostia report <document.json> render a saved document (table/json/markdown/collapsed/...)
|
|
161
|
-
ostia ci run configured workloads vs a baseline, gate regressions
|
|
162
|
-
ostia baseline save|list|show manage baseline ProfileDocuments
|
|
125
|
+
{"event":"run","protocolVersion":1,"schemaVersion":2,"workloadId":"wl_11e8562f3622d528","task":"work","unit":"ns","samples":10,"batch":1,"mean":21012800,"median":20999900,"stddev":231456,"stddevPct":1.1015,"min":20664000,"max":21552300,"warnings":[{"code":"outliers-detected","data":{"mild":1,"severe":0}}],"p75":21086100,"p99":21517600,"mad":126625,"userNs":15519000,"systemNs":6015500,"noiseFloorPct":2.09286,"delta":{"medianPct":44.0989,"meanPct":43.9626,"verdict":"regressed","pass":false,"effectiveTimingPct":10,"matched":true,"ci95":[41.4394,45.5841],"pValue":0.000157103}}
|
|
126
|
+
{"event":"summary","protocolVersion":1,"command":"ci","matched":1,"regressed":1,"improved":0,"unchanged":0,"unmatched":0,"geomeanPct":44.098920968212305,"effectiveTimingPct":10,"verdict":"fail","exitCode":1,"cached":1,"executed":0,"failed":0,"missingBaseline":0,"baseline":{"name":"main","path":".ostia/baselines/main.json"},"noiseFloorPct":2.09286}
|
|
163
127
|
```
|
|
164
128
|
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
### `ostia time`
|
|
129
|
+
Within `protocolVersion: 1`, keys are only ever added, never renamed or removed.
|
|
168
130
|
|
|
169
|
-
|
|
170
|
-
trial each, labeled separately in the document.
|
|
131
|
+
Exit codes, the same for every command:
|
|
171
132
|
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
133
|
+
| Code | Meaning |
|
|
134
|
+
|---|---|
|
|
135
|
+
| `0` | Pass |
|
|
136
|
+
| `1` | At least one workload regressed (`compare`/`ci` only; `time`/`bench` never return 1) |
|
|
137
|
+
| `2` | Harness error: a command exited non-zero or produced no samples, a suite failed, nothing matched, a bad flag, a missing/invalid config or baseline |
|
|
138
|
+
| `130` | Cancelled with Ctrl-C (`time`/`bench`; partial results are still exported) |
|
|
177
139
|
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
140
|
+
On exit 2, stderr's last line is `{"event":"error","protocolVersion":1,"code":...,"message":...,"data"?:...}`
|
|
141
|
+
when stderr is not a TTY or a machine format (`minimal`/`json`/`jsonl`) was requested.
|
|
142
|
+
A person at a terminal sees only the prose message. `code` is one of `invalid-flag`,
|
|
143
|
+
`config-missing`, `config-invalid`, `baseline-missing`, `no-matches`, `spawn-failed`,
|
|
144
|
+
`command-failed`, `timeout`, `time-source-no-match`, `document-load-failed`,
|
|
145
|
+
`no-cpu-evidence`, `internal`. Full reference: [docs/agent-protocol.md](docs/agent-protocol.md).
|
|
183
146
|
|
|
184
|
-
|
|
185
|
-
ostia time -- bun -e "console.log('a b')"
|
|
186
|
-
```
|
|
147
|
+
## Commands
|
|
187
148
|
|
|
188
|
-
|
|
189
|
-
with a space needs `ostia.config.ts`'s array form (`prepare: ["cp", "fixture a", "fixture b"]`)
|
|
190
|
-
instead, since the CLI flag is always one whitespace-split string.
|
|
191
|
-
|
|
192
|
-
`--samples N` is an exact trial count *per command* (with 2+ commands, each gets its
|
|
193
|
-
own N trials, not a total split across them); `--budget MS`
|
|
194
|
-
is a wall-clock time budget instead (default: a hyperfine-style ~3s min-total-time
|
|
195
|
-
loop when neither is given); `--min-samples N` is a hard floor when `--samples` isn't
|
|
196
|
-
given. The same three names work on `ostia bench` (`--budget`/`--samples`/
|
|
197
|
-
`--min-samples`), where `--samples`/`--budget` are per-task the same way -
|
|
198
|
-
`warmup` differs by surface, though: a trial count here, a
|
|
199
|
-
*fraction* of the budget for `ostia bench`, since in-process warmup has no natural
|
|
200
|
-
"N calls" unit before the JIT has even seen the function once.
|
|
201
|
-
|
|
202
|
-
With 2+ commands, trials round-robin across them by default (one trial per command,
|
|
203
|
-
repeated) rather than running one command's whole loop to completion before the next
|
|
204
|
-
starts - drift over the run's wall-clock span (thermal throttling, a noisy neighbor
|
|
205
|
-
process) then lands on every command equally instead of favoring whichever ran first
|
|
206
|
-
or last. `--no-interleave` (`interleave: false`) goes back to running each command's
|
|
207
|
-
loop to completion in turn. Interleaved measurements carry `Measurement.interleaved: true`.
|
|
208
|
-
Meaningless (and ignored) with a single command.
|
|
209
|
-
|
|
210
|
-
`--prepare CMD` runs `CMD` before *every* trial (warmup and `--cpu`/`--heap` trials
|
|
211
|
-
included), unmeasured, in the same cwd - hyperfine's `--prepare`. It's whitespace-split
|
|
212
|
-
like the commands themselves (no shell) and must exit 0. Given once it applies to every
|
|
213
|
-
command; given once per command it pairs up in order, which is how the same command gets
|
|
214
|
-
timed warm and cold side by side:
|
|
149
|
+
Every command takes `--help`. Per-flag detail is in [docs/cli.md](docs/cli.md).
|
|
215
150
|
|
|
216
|
-
|
|
217
|
-
ostia time --prepare "true" --prepare "rm -rf dist" "bun build.ts" "bun build.ts"
|
|
218
|
-
```
|
|
151
|
+
### `ostia time`
|
|
219
152
|
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
caches them separately. The library API also takes a function
|
|
223
|
-
(`prepare: ({ phase, index }) => ...`, see [`time(opts)`](#timeopts--profiledocument)).
|
|
224
|
-
|
|
225
|
-
A hook's stderr is captured rather than streamed live to the terminal (it would otherwise
|
|
226
|
-
flood it, re-running before every one of possibly hundreds of trials) - bounded to 1 MiB
|
|
227
|
-
(head 512 KiB + tail 512 KiB, joined by a `bytes elided` marker if it goes over) and
|
|
228
|
-
folded into the thrown error on the trial where the hook actually times out or exits
|
|
229
|
-
non-zero, so a broken setup script is still easy to debug without an unbounded capture
|
|
230
|
-
risking the run's memory. Contrast `--time-source`'s own output capture (above), which is
|
|
231
|
-
deliberately *not* bounded: the summary line the regex needs could be anywhere in a large
|
|
232
|
-
output, so truncating it there would trade a memory bound for silently-wrong matches.
|
|
233
|
-
|
|
234
|
-
`--time-source REGEX` takes each trial's time from the command's *own output* instead of
|
|
235
|
-
its wall clock: the first `REGEX` match in stdout (then stderr), capture group 1, in
|
|
236
|
-
`--time-unit` units (`ns` | `us` | `ms` | `s`, default `ms`). Meant for tools that report a
|
|
237
|
-
more precise cost than wall time - a build tool whose `built in 342ms` line excludes the
|
|
238
|
-
runtime's startup - so a Bun startup regression isn't misattributed to the tool, and vice
|
|
239
|
-
versa:
|
|
153
|
+
Times commands as subprocesses. `--cpu`/`--heap` add one separate instrumented trial each;
|
|
154
|
+
the profiler never runs during timing trials.
|
|
240
155
|
|
|
241
156
|
```sh
|
|
157
|
+
ostia time "bun a.ts" "bun b.ts"
|
|
158
|
+
ostia time --samples 25 --cpu --heap "bun src/server.ts"
|
|
159
|
+
ostia time --prepare "rm -rf dist" "bun build.ts"
|
|
242
160
|
ostia time --time-source "built in (\d+)ms" "bun build.ts"
|
|
243
161
|
```
|
|
244
162
|
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
the reported time independently, declare the command twice - once plain, once with
|
|
255
|
-
`--time-source` - and they're two workloads with two verdicts. Note the reported number
|
|
256
|
-
has whatever resolution the tool printed (usually whole ms), so its confidence interval
|
|
257
|
-
is coarser than a nanosecond wall clock's.
|
|
258
|
-
|
|
259
|
-
A `RegExp` pattern must not carry the `g`, `y`, or `d` flag - the same compiled pattern is
|
|
260
|
-
`exec`'d once per trial for the run's whole life, and `g`/`y` would make it alternate
|
|
261
|
-
match/no-match across trials via `lastIndex` instead of testing the same thing every time.
|
|
262
|
-
Constructing a workload with one throws `RangeError: timeSource pattern must not use the
|
|
263
|
-
g or y flag` immediately, before any trial runs. A plain (flagless) `RegExp` or a string
|
|
264
|
-
pattern is always safe to reuse. (`--time-source` on the CLI is always a plain string, so
|
|
265
|
-
this only comes up with a `RegExp` literal in the library API.)
|
|
266
|
-
|
|
267
|
-
`--timeout MS` kills a trial (or `--prepare` hook) with SIGKILL if it hasn't finished after
|
|
268
|
-
`MS` ms, so a hung command can't stall the whole run. No default for `time`/`bench` (unset
|
|
269
|
-
never times out); `ostia ci` defaults every workload to 10 minutes unless its config sets
|
|
270
|
-
`timeoutMs`. A timed-out trial resolves (never throws) with `Trial.timedOut: true` and
|
|
271
|
-
contributes no sample; if every trial of a command times out, that command has no timing
|
|
272
|
-
stats and prints like a skipped workload instead of an empty row.
|
|
273
|
-
|
|
274
|
-
`time(opts)` / `bench(opts)` also take a `signal?: AbortSignal`: aborting kills every
|
|
275
|
-
in-flight child process with SIGKILL, stops scheduling new trials, and resolves (never
|
|
276
|
-
rejects) with the document built from whatever measurements had already completed, plus
|
|
277
|
-
an `aborted` warning on the document's last measurement. `Ctrl-C` on the CLI wires this up
|
|
278
|
-
for you - `ostia time`/`ostia bench` cancel cleanly, still write `--export-json` of
|
|
279
|
-
whatever finished, and exit `130`, instead of the process just dying mid-spawn.
|
|
280
|
-
|
|
281
|
-
A non-zero exit doesn't stop a command's trial loop by default: every trial still runs,
|
|
282
|
-
each trial's own exit code lands on `Trial.exitCode`, and the measurement carries a
|
|
283
|
-
`nonzero-exit` warning (`data.exitCodes`) same as always. `--ignore-failure[=CODE,...]`
|
|
284
|
-
(hyperfine's flag; given bare, ignores every exit code) treats the listed codes as
|
|
285
|
-
success - the trial still contributes its sample, just with no warning and no effect on
|
|
286
|
-
the exit code below. `--fail-on-nonzero` stops a command's loop after its *first*
|
|
287
|
-
non-ignored non-zero exit instead of always running its full sample count (that trial's
|
|
288
|
-
sample is still recorded):
|
|
289
|
-
|
|
290
|
-
```sh
|
|
291
|
-
ostia time --ignore-failure=1 "may-exit-1-harmlessly.sh"
|
|
292
|
-
ostia time --fail-on-nonzero "bun build.ts"
|
|
293
|
-
```
|
|
294
|
-
|
|
295
|
-
Exit codes: `0` pass, `2` harness error (a command had a non-ignored non-zero exit, or a
|
|
296
|
-
workload has no timing stats at all - see `--timeout`/`--time-source` above - or a bad
|
|
297
|
-
flag/missing command), `130` cancelled with `Ctrl-C`. `1` is never returned by `time` or
|
|
298
|
-
`bench` - it's reserved for `compare`/`ci` regressions, so a script can tell "the
|
|
299
|
-
benchmark itself couldn't run cleanly" apart from "it ran, and got slower."
|
|
300
|
-
|
|
301
|
-
Timing table (two commands get a Relative column automatically):
|
|
302
|
-
|
|
303
|
-
```
|
|
304
|
-
Task Median Spread Range Relative
|
|
305
|
-
--------------------------------------------------------------------------------
|
|
306
|
-
bun fixtures/fast.ts 7.94 ms 8.31 ms…8.56 ms 7.68 ms…8.57 ms 1.00×
|
|
307
|
-
bun fixtures/slow.ts 21.3 ms 21.5 ms…22.1 ms 21.1 ms…22.1 ms 2.69× slower
|
|
308
|
-
! outliers-detected
|
|
309
|
-
|
|
310
|
-
Warnings:
|
|
311
|
-
bun fixtures/slow.ts: 1 outlier(s) detected (1 severe, 0 mild).
|
|
312
|
-
```
|
|
313
|
-
|
|
314
|
-
Heap summary (type counts from the snapshot trial):
|
|
315
|
-
|
|
316
|
-
```
|
|
317
|
-
Task Median Spread Range
|
|
318
|
-
---------------------------------------------------------------------------
|
|
319
|
-
bun fixtures/allocate.ts 23.8 ms 24.9 ms…29.4 ms 22.6 ms…30.0 ms
|
|
320
|
-
! outliers-detected
|
|
321
|
-
|
|
322
|
-
Warnings:
|
|
323
|
-
bun fixtures/allocate.ts: 8 outlier(s) detected (1 severe, 7 mild).
|
|
324
|
-
|
|
325
|
-
Heap snapshot - bun fixtures/allocate.ts (instrumented, 2516 objects, 0.12MB)
|
|
326
|
-
1369 string
|
|
327
|
-
423 code
|
|
328
|
-
319 closure
|
|
329
|
-
216 object shape
|
|
330
|
-
105 hidden
|
|
331
|
-
artifact: node_modules/.cache/ostia/artifacts/<run-id>-heap.heapsnapshot
|
|
332
|
-
```
|
|
163
|
+
- Each command string is whitespace-split into argv, with no shell. Everything after
|
|
164
|
+
`--` is one more command's argv, verbatim: `ostia time -- bun -e "console.log('a b')"`.
|
|
165
|
+
- Default sampling: 3 warmup trials, then trials until ~3s have elapsed and at least 10
|
|
166
|
+
ran. `--samples N` gives an exact count per command; `--budget MS`/`--min-samples N`
|
|
167
|
+
tune the loop.
|
|
168
|
+
- With 2+ commands, trials round-robin across commands (`--no-interleave` to run them
|
|
169
|
+
one after another).
|
|
170
|
+
- A command stops at its first non-ignored non-zero exit, and `ostia time` exits 2.
|
|
171
|
+
`--ignore-failure[=CODE,...]` treats the listed codes (bare: all) as success.
|
|
333
172
|
|
|
334
173
|
### `ostia bench`
|
|
335
174
|
|
|
336
|
-
|
|
337
|
-
for `--budget` (default 500ms). `--min-samples` is a hard floor kept even when it
|
|
338
|
-
overruns the budget. Left unset, the floor is cost-aware in both directions: as many
|
|
339
|
-
trials as fit in the budget (capped at 20) so one slow task can't blow the suite's total,
|
|
340
|
-
but never below the floor a task's per-trial cost earns it - 3 at ≤1ms, two more per
|
|
341
|
-
decade of cost, 10 from about 3s up. Cheap tasks are time-bound and collect thousands of
|
|
342
|
-
trials either way; only the few expensive tasks in a suite pay for the extra rigor, and
|
|
343
|
-
those are exactly where a 3-sample mean is shakiest. Fast calls are batched so a trial
|
|
344
|
-
spans at least 1µs and a full budget yields about 10k trials at most.
|
|
345
|
-
|
|
346
|
-
Exit codes: `0` pass, `2` harness error (a suite file failed to import/run, a suite or
|
|
347
|
-
isolated task's subprocess timed out - see `--timeout` below - or a bad flag), `130`
|
|
348
|
-
cancelled with `Ctrl-C`. Tasks are in-process function calls, not subprocesses, so there's
|
|
349
|
-
no per-task exit code / `--ignore-failure` the way `ostia time` has; a task that throws
|
|
350
|
-
fails its suite's subprocess the same way it always has. `1` is never returned - it's
|
|
351
|
-
reserved for `compare`/`ci` regressions.
|
|
352
|
-
|
|
353
|
-
| per-trial cost | fits in 500ms | default floor |
|
|
354
|
-
|---|---|---|
|
|
355
|
-
| 30ns | thousands | 20 (time-bound; ends in the tens of thousands) |
|
|
356
|
-
| 30ms | 16 | 16 |
|
|
357
|
-
| 140ms | 3 | 7 |
|
|
358
|
-
| 2.4s | 0 | 10 |
|
|
359
|
-
|
|
360
|
-
A run that ends below its cost-class floor (only possible with an explicit
|
|
361
|
-
`--min-samples` or per-task `minSamples`) carries a `low-sample-count` warning with
|
|
362
|
-
`{ samples, target, trialCostNs }`, so a renderer or an agent can flag a thin number
|
|
363
|
-
without re-deriving the policy from the raw sample array.
|
|
175
|
+
Runs in-process `group()`/`task()` suites. Each suite file runs in its own child process.
|
|
364
176
|
|
|
365
177
|
```sh
|
|
366
178
|
ostia bench bench/*.ts
|
|
367
|
-
ostia bench --
|
|
368
|
-
ostia bench bench/*.ts --
|
|
369
|
-
ostia bench bench
|
|
370
|
-
```
|
|
371
|
-
|
|
372
|
-
`--jobs N|auto` runs that many suite files at once, each still in its own child process.
|
|
373
|
-
Files are independent by design, so for a multi-file suite this is close to a linear
|
|
374
|
-
wall-clock win - but concurrent CPU-bound processes contend for cores, caches and turbo
|
|
375
|
-
headroom, so numbers taken at `--jobs > 1` are noisier and not like-for-like with a
|
|
376
|
-
baseline measured at 1. It defaults to 1 for that reason; opt in for exploratory runs,
|
|
377
|
-
keep 1 for anything you `compare` or `ci` against.
|
|
378
|
-
|
|
379
|
-
`--gc`/`--cpu`/`--alloc`/`--isolate` each take a `--no-` counterpart
|
|
380
|
-
(`--no-gc`/`--no-cpu`/`--no-alloc`/`--no-isolate`) that resolves to an explicit `false`,
|
|
381
|
-
overriding a `true` from `ostia.config.json`'s `bench` section the same way the plain
|
|
382
|
-
flag overrides a config `false` - each flag is `cli ?? config ?? builtin default`, so
|
|
383
|
-
`ostia bench --no-gc` always wins over a config-wide `{ "gc": true }` for that one run.
|
|
384
|
-
|
|
385
|
-
`--isolate` gives every task its own child process instead of sharing its suite file's,
|
|
386
|
-
isolating each task's JIT tier state, inline caches and heap shape from every other task
|
|
387
|
-
in the run - the same guarantee suite files already get from each other, at task
|
|
388
|
-
granularity. `task(name, fn, { isolate })` / `group(name, fn, { isolate })` override the
|
|
389
|
-
suite-wide default for mixed suites (e.g. a couple of outlier-prone tasks isolated, the
|
|
390
|
-
rest sharing a process). `--jobs` then pools across those per-task processes the same way
|
|
391
|
-
it pools across per-file ones, so pair a higher `--jobs` with `--isolate` deliberately -
|
|
392
|
-
overhead now scales with task count, not file count.
|
|
393
|
-
|
|
394
|
-
`--gc` calls `Bun.gc(true)` between trials (default: off, which hides allocation cost as
|
|
395
|
-
Bun/V8 batch calls together and amortize it away). `task(name, fn, { gc })` /
|
|
396
|
-
`group(name, fn, { gc })` override the suite-wide default per task or group, the same
|
|
397
|
-
override pattern as `isolate` - useful when a few allocation-heavy tasks need GC settled
|
|
398
|
-
between trials but the rest of the suite doesn't.
|
|
399
|
-
|
|
400
|
-
`--cpu` captures one extra `phase: "cpu"` measurement per task on top of its timing
|
|
401
|
-
numbers: the task looped for a fixed 200ms window under the JSC sampling profiler
|
|
402
|
-
(JIT tiers included), never mixed into the timing numbers themselves. `--alloc` captures
|
|
403
|
-
an extra `phase: "memstats"` measurement: bytes allocated per call, from a
|
|
404
|
-
`Bun.gc(true)`-bracketed batch of 100 calls (`MemoryEvidence.bytesPerOp`). Both follow the
|
|
405
|
-
same per-task/per-group override pattern as `isolate`/`gc`: `task(name, fn, { cpu, alloc })`
|
|
406
|
-
/ `group(name, fn, { cpu, alloc })`. The terminal table prints an `Alloc/op` column when a
|
|
407
|
-
`memstats` measurement is present. With `--cpu` on, `ostia compare` reports per-frame CPU
|
|
408
|
-
deltas for bench tasks the same way it already does for `ostia time --cpu`.
|
|
409
|
-
|
|
410
|
-
When a `--cpu` capture spends more than 20% of its samples in the llint/baseline tiers,
|
|
411
|
-
the JIT never warmed the task up in that 200ms window, so its CPU numbers (and by
|
|
412
|
-
extension its timing) may not reflect steady state - the cpu measurement carries a
|
|
413
|
-
`jit-cold` warning (`{ llintPct, baselinePct, dfgPct, ftlPct }`), printed alongside the
|
|
414
|
-
CPU capture in the terminal table and folded into the task's line in `--format minimal`.
|
|
415
|
-
|
|
416
|
-
`--timeout MS` kills a suite file's subprocess (or, under `--isolate`, one task's dedicated
|
|
417
|
-
subprocess) with SIGKILL if it hasn't finished after `MS` ms - the same option `ostia time`
|
|
418
|
-
has, applied at the subprocess granularity `--isolate` already runs at rather than per task.
|
|
419
|
-
No default (unset never times out); `ostia ci` defaults every `suites` entry to 10 minutes
|
|
420
|
-
unless its config sets `bench.timeoutMs`.
|
|
421
|
-
|
|
422
|
-
`--preload PATH` (repeatable) imports a script before each suite file loads, in the same
|
|
423
|
-
subprocess - the same shape as Bun's own `--preload` / `bunfig.toml`'s `preload` array. Use
|
|
424
|
-
it to install globals a suite needs at import time (jsdom's `document`/`window`) or register
|
|
425
|
-
a `Bun.plugin()` file-loader (e.g. compiling `.svelte`/`.vue` SFCs) before the suite's own
|
|
426
|
-
top-level code runs. Multiple `--preload` scripts run in the order given, so state one
|
|
427
|
-
installs (a plugin registration, a global) is visible to the next and to the suite itself.
|
|
428
|
-
ostia ships none of this itself - just the hook point (for a full jsdom/happy-dom global
|
|
429
|
-
setup or a `Bun.plugin()` component-compile hook, see
|
|
430
|
-
[docs/preload-recipes.md](docs/preload-recipes.md)):
|
|
431
|
-
|
|
432
|
-
```ts
|
|
433
|
-
// bench/jsdom-setup.ts
|
|
434
|
-
import { JSDOM } from "jsdom"
|
|
435
|
-
const dom = new JSDOM("<!doctype html>")
|
|
436
|
-
Object.assign(globalThis, { document: dom.window.document, window: dom.window })
|
|
437
|
-
```
|
|
438
|
-
|
|
439
|
-
```sh
|
|
440
|
-
ostia bench --preload ./bench/jsdom-setup.ts bench/*.dom.bench.ts
|
|
441
|
-
```
|
|
442
|
-
|
|
443
|
-
`--bun-flags FLAGS` (repeatable, space-separated flags within one value are all appended)
|
|
444
|
-
passes extra flags through to the `bun` invocation that spawns each suite file - the fix for
|
|
445
|
-
packages whose `package.json` `exports` map branches on a resolution condition Bun doesn't
|
|
446
|
-
set by default. Svelte 5's `exports` map, for example, is `{ "browser": "./src/index-client.js",
|
|
447
|
-
"default": "./src/index-server.js" }`: without `--conditions browser`, Bun resolves `default`
|
|
448
|
-
(the server-rendering build), and mounting a component via `@testing-library/svelte` throws
|
|
449
|
-
`lifecycle_function_unavailable` since `mount()` isn't available server-side. The same applies
|
|
450
|
-
to Vue and other dual-target frameworks:
|
|
451
|
-
|
|
452
|
-
```sh
|
|
453
|
-
ostia bench --bun-flags="--conditions=browser" bench/*.dom.bench.ts
|
|
454
|
-
```
|
|
455
|
-
|
|
456
|
-
Unlike `BUN_OPTIONS` (an env var Bun's CLI reads to prepend flags, which only reaches the
|
|
457
|
-
spawned suite process today because ostia's `Bun.spawn()` happens to inherit `process.env`),
|
|
458
|
-
`--bun-flags` is a declared, documented integration point that doesn't depend on the parent
|
|
459
|
-
shell's environment.
|
|
460
|
-
|
|
461
|
-
|
|
179
|
+
ostia bench bench/*.ts --filter parse --cpu --alloc
|
|
180
|
+
ostia bench bench/*.ts --isolate
|
|
181
|
+
ostia bench --preload ./bench/dom-setup.ts --bun-flags="--conditions=browser" bench/*.ts
|
|
462
182
|
```
|
|
463
|
-
Task Median Spread Range Relative
|
|
464
|
-
----------------------------------------------------------------------------------------------------
|
|
465
|
-
stats:
|
|
466
|
-
stats/computeTimingStats (1e3 samples) 24.7 µs 33.8 µs…151.6 µs 22.4 µs…1073.2 µs 1.00×
|
|
467
|
-
! outliers-detected
|
|
468
|
-
stats/computeTimingStats (1e4 samples) 255.2 µs 331.7 µs…1067.6 µs 223.2 µs…20789.4 µs 10.33× slower
|
|
469
|
-
! outliers-detected
|
|
470
|
-
stats/timingWarnings (1e3 samples) 47.3 µs 88.5 µs…510.4 µs 43.0 µs…14383.1 µs 1.91× slower
|
|
471
|
-
! outliers-detected
|
|
472
|
-
|
|
473
|
-
Warnings:
|
|
474
|
-
stats/computeTimingStats (1e3 samples): 2455 outlier(s) detected (2108 severe, 347 mild).
|
|
475
|
-
stats/computeTimingStats (1e4 samples): 215 outlier(s) detected (70 severe, 145 mild).
|
|
476
|
-
stats/timingWarnings (1e3 samples): 541 outlier(s) detected (191 severe, 350 mild).
|
|
477
|
-
```
|
|
478
|
-
|
|
479
|
-
Tasks with a `group()` print the group name once, indented; ungrouped tasks and
|
|
480
|
-
subprocess commands print flat.
|
|
481
183
|
|
|
482
|
-
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
184
|
+
- Each task samples for `--budget` ms (default 500). Fast calls are batched so one trial
|
|
185
|
+
spans at least 1µs; the budget-driven loop stops at 20,000 trials.
|
|
186
|
+
- `--isolate` runs every task in its own process, isolating JIT state, builtin call-site
|
|
187
|
+
feedback (e.g. `Array.prototype.map`) and GC heap from other tasks. Use it when you need
|
|
188
|
+
the most comparable numbers.
|
|
189
|
+
- `--jobs N|auto` runs suite files in parallel. Faster, but noisier; keep the default of 1
|
|
190
|
+
for anything you `compare` or gate in `ci`.
|
|
191
|
+
- With no files, `ostia bench` uses the config's `bench` section. Each flag overrides its
|
|
192
|
+
config field; `--no-gc`/`--no-cpu`/`--no-alloc`/`--no-isolate` override a config `true`.
|
|
488
193
|
|
|
489
194
|
### `ostia compare`
|
|
490
195
|
|
|
491
|
-
|
|
196
|
+
Matches two documents' workloads by id and reports a verdict per workload.
|
|
492
197
|
|
|
493
198
|
```sh
|
|
494
199
|
ostia compare before.json after.json
|
|
495
200
|
ostia compare after.json --baseline .ostia/baselines/main.json
|
|
201
|
+
ostia compare before.json after.json --format markdown
|
|
496
202
|
```
|
|
497
203
|
|
|
498
204
|
```
|
|
499
205
|
✗ bun fixtures/work.ts
|
|
500
|
-
timing: +
|
|
501
|
-
```
|
|
502
|
-
|
|
503
|
-
Exit codes: `0` pass, `1` at least one workload regressed, `2` nothing was compared (zero
|
|
504
|
-
matched workloads - a stale baseline, a totally rewritten config) or a harness error
|
|
505
|
-
(documents failed to load, or a bad flag).
|
|
506
|
-
|
|
507
|
-
A workload id present on only one document prints in an `Unmatched` section (table and
|
|
508
|
-
markdown formats) instead of silently vanishing:
|
|
509
|
-
|
|
510
|
-
```
|
|
511
|
-
Unmatched:
|
|
512
|
-
baseline only: old-task
|
|
513
|
-
candidate only: new-task
|
|
514
|
-
```
|
|
515
|
-
|
|
516
|
-
`--format table` also prints a `threshold` header line when the machine's noise floor
|
|
517
|
-
widened the effective threshold past `thresholds.timingPct`:
|
|
518
|
-
|
|
519
|
-
```
|
|
520
|
-
threshold 5% (widened to 6.2% by noise floor)
|
|
521
|
-
```
|
|
522
|
-
|
|
523
|
-
`ostia compare` reads `ostia.config.ts`/`ostia.config.json`'s `thresholds` when present
|
|
524
|
-
(same discovery as `ostia ci`), so a project-tuned threshold applies here too instead of
|
|
525
|
-
only gating `ostia ci`; `--no-config` ignores it and uses `DEFAULT_THRESHOLDS`.
|
|
526
|
-
`--timing-pct N` / `--alpha N` override individual fields on top of whichever base was
|
|
527
|
-
picked. The source is printed above the report:
|
|
528
|
-
|
|
529
|
-
```
|
|
530
|
-
thresholds: ostia.config.ts
|
|
206
|
+
timing: +23.8% median, 95% CI [+18.3%, +30.1%], p<0.001 (regressed)
|
|
531
207
|
```
|
|
532
208
|
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
```
|
|
539
|
-
|
|
540
|
-
The verdict needs both a confidence interval clear of `timingPct` and a
|
|
541
|
-
significant Mann-Whitney p-value (`thresholds.alpha`, default `0.01`), not
|
|
542
|
-
just a point estimate past the threshold - see
|
|
543
|
-
[Statistics](#statistics-a-real-significance-test-not-a-percentage-threshold)
|
|
544
|
-
below. Comparisons with fewer than 5 samples on either side fall back to the
|
|
545
|
-
old point-estimate rule and carry a `thin-comparison` warning instead.
|
|
546
|
-
|
|
547
|
-
When `base`/`cand` differ in `platform.os`, `platform.arch`, `bunVersion`, or (when both
|
|
548
|
-
carry `environment`) `cpuModel`/`cores`, every comparison in the document carries an
|
|
549
|
-
`environment-mismatch` warning (`data.fields`: `{ field, base, cand }[]`) - a timing delta
|
|
550
|
-
between two different machines or Bun versions may reflect that, not the code change under
|
|
551
|
-
test. `table` and `markdown` print it once, in the header, instead of once per workload;
|
|
552
|
-
`minimal` folds it into each task line's `warnings[]` alongside its measurement warnings,
|
|
553
|
-
so `line.warnings.some(w => w.code === "environment-mismatch")` keeps working the same way
|
|
554
|
-
it does for a measurement warning.
|
|
555
|
-
|
|
556
|
-
#### Statistics: a real significance test, not a percentage threshold
|
|
557
|
-
|
|
558
|
-
A point estimate past `timingPct` is not enough to call something a
|
|
559
|
-
regression - both documents already carry full sample arrays, so `compare`
|
|
560
|
-
runs two tests instead:
|
|
561
|
-
|
|
562
|
-
- A **bootstrap confidence interval** on the difference of medians:
|
|
563
|
-
resample both sides with replacement `thresholds.bootstrapIterations`
|
|
564
|
-
times (default 2000; each side is randomly subsampled to at most 2000
|
|
565
|
-
samples first, so a many-thousand-sample task doesn't turn a compare into
|
|
566
|
-
a multi-second operation), and report the 2.5th/97.5th percentiles as
|
|
567
|
-
`ci95` (percent of the baseline median). Reproducible: the PRNG seed is
|
|
568
|
-
stored in `Comparison.timing.seed`.
|
|
569
|
-
- A **Mann-Whitney U test** (tie-corrected, normal approximation), reported
|
|
570
|
-
as `pValue` - whether the two sample distributions differ at all, without
|
|
571
|
-
assuming normality the way a t-test would.
|
|
572
|
-
|
|
573
|
-
`regressed` requires `ci95[0] > thresholds.timingPct` (the *whole interval*
|
|
574
|
-
clears the threshold) **and** `pValue < thresholds.alpha`; `improved` is the
|
|
575
|
-
mirror. Otherwise `unchanged`. This is why the earlier example (`+11.2%
|
|
576
|
-
median, 95% CI [+10.0%, +16.4%]`) is a clean regression: even the low end of
|
|
577
|
-
the interval is well past `timingPct`.
|
|
578
|
-
|
|
579
|
-
Both `time()` and `bench()` also stamp `environment` on every document (a
|
|
580
|
-
fixed-cost, deterministic, allocation-free hash loop sampled for ~200ms,
|
|
581
|
-
`noise.floorPct = mad / median`) unless `noiseCheck: false` / `--no-noise-check`
|
|
582
|
-
skips it. `compare` widens the effective threshold to
|
|
583
|
-
`max(thresholds.timingPct, base.environment.noise.floorPct,
|
|
584
|
-
cand.environment.noise.floorPct)` (`Comparison.thresholds.effectiveTimingPct`),
|
|
585
|
-
so a delta smaller than the machine's own jitter right now is never called a
|
|
586
|
-
regression. A `noisy-machine` warning fires when the 1-minute load average is
|
|
587
|
-
already past 75% of available cores at measurement time.
|
|
209
|
+
A regression needs the whole 95% CI above the threshold and a Mann-Whitney p-value below
|
|
210
|
+
`alpha` (default 0.01). Thresholds come from the config file when one exists, otherwise
|
|
211
|
+
the defaults (`timingPct: 5`). The bootstrap is seeded from the samples, so the same two
|
|
212
|
+
documents always give the same verdict. See [docs/statistics.md](docs/statistics.md).
|
|
213
|
+
Exit: `0` pass, `1` regression, `2` nothing matched or a load error.
|
|
588
214
|
|
|
589
215
|
### `ostia report`
|
|
590
216
|
|
|
591
|
-
|
|
592
|
-
both the data formats (`table`/`json`/`jsonl`/`markdown`/`minimal`) and the CPU
|
|
593
|
-
visualization formats (`collapsed`/`mermaid`/`speedscope`/`cpuprofile`) - one
|
|
594
|
-
command instead of two. `time`/`bench`/`compare --format` only accept the data
|
|
595
|
-
formats; export the document and run `ostia report --format <viz>` on it for
|
|
596
|
-
a visualization.
|
|
597
|
-
|
|
598
|
-
```sh
|
|
599
|
-
ostia report out.json # table (default)
|
|
600
|
-
ostia report out.json --format markdown
|
|
601
|
-
ostia report out.json --format json
|
|
602
|
-
ostia report out.json --format jsonl
|
|
603
|
-
ostia report out.json --format minimal
|
|
604
|
-
```
|
|
605
|
-
|
|
606
|
-
Minimal format - one JSON object per timing run, no header, no raw sample array, no prose.
|
|
607
|
-
See [Using ostia from an AI agent](#using-ostia-from-an-ai-agent) above for the full
|
|
608
|
-
protocol (event types, the exit-code contract, the stderr error line). A `run` event:
|
|
609
|
-
|
|
610
|
-
```
|
|
611
|
-
{"event":"run","protocolVersion":1,"schemaVersion":2,"workloadId":"wl_1a2b3c4d5e6f7890","task":"diffText()/append at end","group":"diffText()","unit":"ns","samples":9282,"batch":1,"mean":50213.4,"median":49871,"stddev":2104.7,"stddevPct":4.19,"min":48120,"max":81002,"p75":50920,"p99":58011,"mad":1780,"relative":1,"warnings":[]}
|
|
612
|
-
```
|
|
613
|
-
|
|
614
|
-
`ostia compare`/`ostia ci --format minimal` add `delta: { medianPct, meanPct, verdict, pass,
|
|
615
|
-
ci95?, pValue?, effectiveTimingPct, matched }` to each `run` line, so "did this PR regress" is
|
|
616
|
-
`lines.some(l => l.delta?.verdict === "regressed")` - and a trailing `summary` line carries
|
|
617
|
-
the same verdict for the whole run.
|
|
618
|
-
|
|
619
|
-
Markdown:
|
|
620
|
-
|
|
621
|
-
```
|
|
622
|
-
# Profile Report
|
|
623
|
-
|
|
624
|
-
Bun 1.4.1 · ostia 0.1.0 · darwin/arm64 · 2026-09-05T13:14:50.085Z · a1b2c3d (main)
|
|
625
|
-
|
|
626
|
-
## Timing
|
|
627
|
-
|
|
628
|
-
| Task | Median | Spread (p75…p99) | Mean ± SD | Range | MAD |
|
|
629
|
-
|---|---|---|---|---|---|
|
|
630
|
-
| bun -e 1 | 5.03 ms | 5.42 ms…9.70 ms | 5.29 ms ± 0.84 ms | 4.77 ms…13.4 ms | 0.16 ms |
|
|
631
|
-
```
|
|
632
|
-
|
|
633
|
-
#### CPU visualization formats
|
|
634
|
-
|
|
635
|
-
Turn CPU evidence into files for other tools. Formats: `collapsed`, `mermaid`,
|
|
636
|
-
`speedscope`, `cpuprofile` (pass-through of a real CDP artifact when present).
|
|
637
|
-
`--measurement <id>` renders only that measurement (default: every CPU
|
|
638
|
-
measurement in the document); `--out-dir PATH` writes files there instead of
|
|
639
|
-
stdout.
|
|
217
|
+
Renders a saved document without re-running anything.
|
|
640
218
|
|
|
641
219
|
```sh
|
|
642
|
-
ostia report doc.json --format
|
|
643
|
-
ostia report doc.json --format
|
|
644
|
-
ostia report doc.json --format speedscope
|
|
645
|
-
|
|
646
|
-
|
|
647
|
-
Collapsed stacks (one line per stack; feeds `flamegraph.pl` and friends):
|
|
648
|
-
|
|
649
|
-
```
|
|
650
|
-
(root);(module);hashLoop 209
|
|
220
|
+
ostia report doc.json --format markdown
|
|
221
|
+
ostia report doc.json --format minimal
|
|
222
|
+
ostia report doc.json --format speedscope --out-dir viz/
|
|
223
|
+
ostia report doc.json --format collapsed | flamegraph.pl > flame.svg
|
|
651
224
|
```
|
|
652
225
|
|
|
653
|
-
|
|
654
|
-
|
|
655
|
-
|
|
656
|
-
|
|
657
|
-
n1["(root) (self 0.00ms, total 267.91ms)"]
|
|
658
|
-
n2["(module) (self 0.00ms, total 267.91ms)"]
|
|
659
|
-
n3["hashLoop (self 267.91ms, total 267.91ms)"]
|
|
660
|
-
n1 --> n2
|
|
661
|
-
n2 --> n3
|
|
662
|
-
```
|
|
226
|
+
Formats: `table` (default), `json`, `jsonl`, `markdown`, `minimal`, and, for documents
|
|
227
|
+
with CPU evidence, `collapsed`, `mermaid`, `speedscope`, `cpuprofile`. `time`, `bench`,
|
|
228
|
+
`compare` and `ci` accept only the first five; export a document and use `report` for the
|
|
229
|
+
visualization formats.
|
|
663
230
|
|
|
664
231
|
### `ostia ci`
|
|
665
232
|
|
|
666
|
-
|
|
667
|
-
|
|
233
|
+
Runs the config's workloads, compares them against a named baseline, and exits 1 on a
|
|
234
|
+
regression.
|
|
668
235
|
|
|
669
236
|
```sh
|
|
670
237
|
ostia ci
|
|
671
|
-
ostia ci --full # ignore cache
|
|
672
|
-
ostia ci --baseline
|
|
673
|
-
ostia ci --
|
|
674
|
-
ostia ci --save-baseline # after a pass, promote today's numbers to the baseline
|
|
675
|
-
ostia ci --on-missing-baseline fail
|
|
676
|
-
ostia ci --no-noise-check
|
|
677
|
-
```
|
|
678
|
-
|
|
679
|
-
Pass:
|
|
680
|
-
|
|
681
|
-
```
|
|
682
|
-
1 workloads
|
|
683
|
-
0 cached
|
|
684
|
-
1 executed
|
|
685
|
-
1 passed 0 regressed
|
|
686
|
-
|
|
687
|
-
Profile CI: ✓
|
|
688
|
-
```
|
|
689
|
-
|
|
690
|
-
Fail:
|
|
691
|
-
|
|
692
|
-
```
|
|
693
|
-
1 workloads
|
|
694
|
-
0 cached
|
|
695
|
-
1 executed
|
|
696
|
-
0 passed 1 regressed (+1278.7% median on work)
|
|
697
|
-
|
|
698
|
-
Profile CI: ✗
|
|
238
|
+
ostia ci --full # ignore the cache
|
|
239
|
+
ostia ci --baseline release
|
|
240
|
+
ostia ci --save-baseline # after a pass, make this run the new baseline
|
|
699
241
|
```
|
|
700
242
|
|
|
701
|
-
|
|
702
|
-
|
|
703
|
-
|
|
704
|
-
|
|
705
|
-
|
|
706
|
-
|
|
707
|
-
|
|
708
|
-
|
|
709
|
-
`
|
|
710
|
-
stale/wrong baseline - and otherwise lists what's missing in the report without affecting
|
|
711
|
-
the exit code, since one new workload next to an otherwise-matching baseline isn't a hard
|
|
712
|
-
error. `"fail"`/`"warn"` explicitly always fail/never fail on any mismatch, regardless of
|
|
713
|
-
how many workloads are missing.
|
|
714
|
-
|
|
715
|
-
`ci` also measures this machine's noise floor once per invocation (the same ~200ms
|
|
716
|
-
reference measurement `time()`/`bench()` take) and stamps it on both the candidate
|
|
717
|
-
document and `ostia baseline save`'s output, so `compare`'s noise-floor threshold widening
|
|
718
|
-
(see [Statistics](#statistics-a-real-significance-test-not-a-percentage-threshold)) applies
|
|
719
|
-
to `ci`-gated regressions too, not only ad hoc `time`/`bench` runs. `noiseCheck: false` in
|
|
720
|
-
config, or `--no-noise-check`, skips it.
|
|
721
|
-
|
|
722
|
-
#### `ostia.config.ts` / `ostia.config.json`
|
|
723
|
-
|
|
724
|
-
`loadConfig` looks for `ostia.config.ts` first (Bun imports TypeScript natively), falling
|
|
725
|
-
back to `ostia.config.json`. Both forms are fully supported; pick `.ts` for autocomplete
|
|
726
|
-
and type-checking on every field, via `defineConfig` (an identity function purely for
|
|
727
|
-
typing, the same pattern as Vite/Vitest/ESLint):
|
|
243
|
+
- Command workloads are cached by their declared `inputs`: no `inputs` field always
|
|
244
|
+
reruns; `inputs: []` means "depends on nothing" and caches; otherwise the run is reused
|
|
245
|
+
while the matched files' contents are unchanged. `suites` workloads always run.
|
|
246
|
+
- `suites` workloads run with the config's `bench` section, the same way `ostia bench`
|
|
247
|
+
reads it.
|
|
248
|
+
- Exit 2 if any command workload exits non-zero (not ignored) or produces no samples, or
|
|
249
|
+
if the baseline file is missing. A baseline that matches none of the configured
|
|
250
|
+
workloads is also an error; one missing only some lists them and carries on
|
|
251
|
+
(`onMissingBaseline` in the config changes this).
|
|
728
252
|
|
|
729
|
-
|
|
730
|
-
// ostia.config.ts
|
|
731
|
-
import { defineConfig } from "ostia"
|
|
732
|
-
|
|
733
|
-
export default defineConfig({
|
|
734
|
-
baseline: "main",
|
|
735
|
-
thresholds: { timingPct: 5 },
|
|
736
|
-
workloads: [
|
|
737
|
-
{ label: "parse", command: ["bun", "bench/parse.ts"], inputs: ["src/**/*.ts"] },
|
|
738
|
-
{ label: "dogfood-suites", suites: ["bench/*.ts"] },
|
|
739
|
-
// Same command, three states: warm no-op, one edited input, cold. `prepare`
|
|
740
|
-
// runs before every trial; `timeSource` reads the tool's own "in Nms" line.
|
|
741
|
-
{ label: "build:warm", command: ["bun", "cli.ts", "build", "fixture"], timeSource: { pattern: "in (\\d+)ms" } },
|
|
742
|
-
{ label: "build:incremental", command: ["bun", "cli.ts", "build", "fixture"], timeSource: { pattern: "in (\\d+)ms" },
|
|
743
|
-
prepare: () => touchPost("fixture/posts/hello.md") },
|
|
744
|
-
{ label: "build:cold", command: ["bun", "cli.ts", "build", "fixture"], timeSource: { pattern: "in (\\d+)ms" },
|
|
745
|
-
prepare: "rm -rf fixture/dist" },
|
|
746
|
-
],
|
|
747
|
-
})
|
|
748
|
-
```
|
|
749
|
-
|
|
750
|
-
```json
|
|
751
|
-
// ostia.config.json - equivalent, no defineConfig wrapper needed
|
|
752
|
-
{
|
|
753
|
-
"baseline": "main",
|
|
754
|
-
"thresholds": { "timingPct": 5 },
|
|
755
|
-
"workloads": [
|
|
756
|
-
{
|
|
757
|
-
"label": "parse",
|
|
758
|
-
"command": ["bun", "bench/parse.ts"],
|
|
759
|
-
"inputs": ["src/**/*.ts"]
|
|
760
|
-
},
|
|
761
|
-
{
|
|
762
|
-
"label": "dogfood-suites",
|
|
763
|
-
"suites": ["bench/*.ts"]
|
|
764
|
-
}
|
|
765
|
-
]
|
|
766
|
-
}
|
|
767
|
-
```
|
|
768
|
-
|
|
769
|
-
Each workload is exactly one of `command` (a subprocess timed with `runs`/`warmup`) or
|
|
770
|
-
`suites` (glob patterns, same resolution as `bench`'s own `suites` config, run via
|
|
771
|
-
`bench()`). A `suites` workload gates every task in those files individually - one
|
|
772
|
-
candidate-vs-baseline comparison per task, matched by workload id the same way a `command`
|
|
773
|
-
workload already is, so `ostia ci`'s regression detection covers in-process microbenchmarks,
|
|
774
|
-
not only subprocess commands. Unlike `command` workloads, a `suites` workload always
|
|
775
|
-
executes (there's no cheap way to know a suite file's task ids, and so its per-task cache
|
|
776
|
-
keys, without importing it first) - `inputs`-based cache skipping is `command`-only for now.
|
|
777
|
-
|
|
778
|
-
`inputs` is optional (and, for now, only consulted for `command` workloads). Workloads
|
|
779
|
-
with no `inputs` always rerun (cache fails conservative).
|
|
780
|
-
|
|
781
|
-
`prepare` and `timeSource` are `command`-only and mean the same as `ostia time`'s
|
|
782
|
-
`--prepare` / `--time-source`: `prepare` is a command string or argv array in both config
|
|
783
|
-
forms, or a function in `ostia.config.ts`; `timeSource` is `{ pattern, group?, unit? }`
|
|
784
|
-
with `pattern` a regex source string (or a `RegExp` in `.ts`). Both are part of the workload
|
|
785
|
-
id. A function-form `prepare` can't be fingerprinted, so that workload never comes from
|
|
786
|
-
cache - it always executes, like a workload with no `inputs`.
|
|
787
|
-
|
|
788
|
-
`timeoutMs`, `ignoreExitCodes`, and `failOnNonzero` are also `command`-only and mean the
|
|
789
|
-
same as `ostia time`'s `--timeout` / `--ignore-failure` / `--fail-on-nonzero`. `ostia ci`
|
|
790
|
-
defaults every workload's `timeoutMs` to 10 minutes when the workload doesn't set one
|
|
791
|
-
(`bench.timeoutMs` does the same for `suites` workloads); none of the three are part of
|
|
792
|
-
the workload id, so tuning them doesn't orphan a cached run or a saved baseline.
|
|
793
|
-
|
|
794
|
-
Two directory options, both optional: `outDir` (default `node_modules/.cache/ostia`) for
|
|
795
|
-
scratch/cache/artifacts, and `baselineDir` (default `.ostia/baselines`) for baselines. They're
|
|
796
|
-
independent - `baselineDir` doesn't move just because you override `outDir`.
|
|
797
|
-
|
|
798
|
-
`onMissingBaseline` (`"warn"` | `"fail"`, default unset - see [`ostia
|
|
799
|
-
ci`](#ostia-ci) above) and `noiseCheck` (default `true`) are top-level config fields, not
|
|
800
|
-
per-workload: `--on-missing-baseline` / `--no-noise-check` override them per invocation.
|
|
801
|
-
|
|
802
|
-
#### Baselines (local and CI)
|
|
803
|
-
|
|
804
|
-
Baselines are JSON under `.ostia/baselines/` (gitignored). `ostia ci` only needs the file
|
|
805
|
-
on disk; it does not need to be committed.
|
|
806
|
-
|
|
807
|
-
A workload's id identifies *what* is measured, not where the measuring process ran: for a
|
|
808
|
-
`command` workload it hashes the command argv, `prepare`, and `timeSource`, and
|
|
809
|
-
deliberately excludes `process.cwd()`. That means the same command measured from a CI
|
|
810
|
-
runner, a developer's checkout, or a different git worktree of the same repo produces the
|
|
811
|
-
same id and matches the same baseline row - `label` changes and switching directories
|
|
812
|
-
never orphan a baseline.
|
|
813
|
-
|
|
814
|
-
`ostia baseline save [name]` measures every configured workload (the same code path
|
|
815
|
-
`ostia ci` gates against, no comparison) and writes it to `<baselineDir>/<name>.json`
|
|
816
|
-
(default name: config's `"baseline"` field, or `"main"`). `ostia baseline list` shows every
|
|
817
|
-
saved baseline (name, created date, workload count, and git sha/branch when available);
|
|
818
|
-
`ostia baseline show <name> [--format]` renders one (delegates to `ostia report`). A
|
|
819
|
-
baseline name must match `/^[A-Za-z0-9._-]+$/` and can't start with `-` - a typo'd flag
|
|
820
|
-
(`ostia baseline save --verbose`) is a usage error instead of a literal filename.
|
|
821
|
-
|
|
822
|
-
Every document stamps `git: { sha, branch, dirty }` (from `git rev-parse` / `git status
|
|
823
|
-
--porcelain` in the process's cwd, 200ms timeout, silently absent outside a repo or
|
|
824
|
-
without `git` installed) - metadata only, never part of any fingerprint or id, so a
|
|
825
|
-
commit or a dirty working tree never orphans a cached run or baseline. Printed in the
|
|
826
|
-
markdown report's header line and `ostia baseline list`.
|
|
827
|
-
|
|
828
|
-
Local branch workflow:
|
|
829
|
-
|
|
830
|
-
```sh
|
|
831
|
-
git checkout master # known-good tip
|
|
832
|
-
ostia baseline save # -> .ostia/baselines/main.json
|
|
833
|
-
|
|
834
|
-
git checkout -b my-opt
|
|
835
|
-
# ... change code ...
|
|
836
|
-
ostia ci # or: bun run dogfood
|
|
837
|
-
```
|
|
838
|
-
|
|
839
|
-
The baseline survives branch switches because it is not tracked. Re-seed only when you
|
|
840
|
-
intentionally accept a new floor. Seeding on the branch you are guarding compares that
|
|
841
|
-
branch to itself.
|
|
842
|
-
|
|
843
|
-
`ostia ci --save-baseline` folds that re-seed into the gate itself: after a pass (no
|
|
844
|
-
regressions), it writes the just-measured document as the new baseline at the same path
|
|
845
|
-
it just compared against - useful in a CI job that gates every merge to a trunk branch,
|
|
846
|
-
so each green run becomes the next run's floor with no separate step.
|
|
847
|
-
|
|
848
|
-
One-off outside this repo's config:
|
|
253
|
+
### `ostia baseline`
|
|
849
254
|
|
|
850
255
|
```sh
|
|
851
|
-
ostia
|
|
852
|
-
ostia
|
|
256
|
+
ostia baseline save # measure the configured workloads -> .ostia/baselines/main.json
|
|
257
|
+
ostia baseline save my-feature
|
|
258
|
+
ostia baseline list
|
|
259
|
+
ostia baseline show main --format markdown
|
|
853
260
|
```
|
|
854
261
|
|
|
855
|
-
|
|
856
|
-
and runs `ostia ci` against it. If the base has no `package.json` / `ostia.config.json`
|
|
857
|
-
yet (empty starter commit), CI seeds from the PR tip instead so dogfood still runs.
|
|
262
|
+
`save` uses the same measurement code path as `ci`. `show` accepts `report`'s flags.
|
|
858
263
|
|
|
859
264
|
## Library API
|
|
860
265
|
|
|
861
|
-
The CLI is a thin wrapper around the library. Same `ProfileDocument` either way.
|
|
862
|
-
|
|
863
266
|
```ts
|
|
864
267
|
import {
|
|
865
|
-
time,
|
|
866
|
-
|
|
867
|
-
bench,
|
|
868
|
-
group,
|
|
869
|
-
task,
|
|
870
|
-
range,
|
|
871
|
-
sweep,
|
|
872
|
-
keep,
|
|
873
|
-
compareDocuments,
|
|
874
|
-
createDocument,
|
|
875
|
-
defineConfig,
|
|
876
|
-
renderers,
|
|
877
|
-
saveDocument,
|
|
878
|
-
loadDocument,
|
|
268
|
+
time, bench, group, task, sweep, range, run, profile, keep,
|
|
269
|
+
compareDocuments, defineConfig, createDocument, loadDocument, saveDocument, renderers,
|
|
879
270
|
} from "ostia"
|
|
880
|
-
import type { ProfileDocument } from "ostia"
|
|
271
|
+
import type { ProfileDocument, MinimalEvent } from "ostia"
|
|
881
272
|
```
|
|
882
273
|
|
|
883
|
-
|
|
884
|
-
|
|
885
|
-
Subprocess timing,
|
|
274
|
+
| Export | Does |
|
|
275
|
+
|---|---|
|
|
276
|
+
| `time(opts)` | Subprocess timing, same as `ostia time`. Returns a `ProfileDocument`. |
|
|
277
|
+
| `bench(opts)` | Runs suite files, same as `ostia bench`. |
|
|
278
|
+
| `group(name, fn, opts?)` / `task(name, fn, opts?)` | Register in-process tasks; `.skip`/`.only` variants. |
|
|
279
|
+
| `sweep(dims, fn)` / `range(start, end, mult?)` | Parameter sweeps; tasks inherit the point as `params`. |
|
|
280
|
+
| `run(opts?)` | Runs the tasks registered in the current file, in this process (`bun suite.ts`). |
|
|
281
|
+
| `profile(fn, opts?)` | In-process CPU capture; `origin: "jsc"` adds JIT tier data. |
|
|
282
|
+
| `keep(value)` | Pins an intermediate value against dead-code elimination. |
|
|
283
|
+
| `compareDocuments(base, cand, thresholds?)` | Same comparison as `ostia compare`. |
|
|
284
|
+
| `defineConfig(config)` | Typing helper for `ostia.config.ts`. |
|
|
285
|
+
| `createDocument` / `loadDocument` / `saveDocument` | Build, read (schema v2 only), and write documents. |
|
|
286
|
+
| `renderers` | `table`, `markdown`, `json`, `jsonl`, `minimal`, `collapsed`, `mermaid`, `speedscope`, `cpuprofile`. |
|
|
886
287
|
|
|
887
288
|
```ts
|
|
888
289
|
const doc = await time({
|
|
889
|
-
commands: ["bun a.ts", "bun b.ts"],
|
|
890
|
-
|
|
891
|
-
// function: ({ phase, index }) => ..., phase is "warmup" | "timing" | "cpu" | "heap"
|
|
892
|
-
timeSource: { pattern: /in (\d+)ms/, unit: "ms" }, // samples from the command's own output
|
|
893
|
-
samples: 10, // exact trial count
|
|
894
|
-
// budgetMs: 3000, // wall-clock budget instead of an exact count
|
|
895
|
-
// minSamples: 10, // hard floor when samples isn't given
|
|
896
|
-
warmup: 2,
|
|
897
|
-
interleave: true, // default when 2+ commands: round-robins trials across them
|
|
290
|
+
commands: ["bun a.ts", { command: "bun b.ts", label: "b", prepare: "rm -rf dist" }],
|
|
291
|
+
samples: 20,
|
|
898
292
|
cpu: true,
|
|
899
|
-
heap: false,
|
|
900
|
-
cpuIntervalUs: 200,
|
|
901
|
-
outDir: "node_modules/.cache/ostia", // default; artifacts land under here
|
|
902
|
-
noiseCheck: true, // default; set false to skip the ~200ms noise floor measurement
|
|
903
|
-
timeoutMs: 30_000, // kill a hung trial/prepare hook with SIGKILL; no default
|
|
904
|
-
signal: controller.signal, // abort to cancel: kills in-flight children, keeps partial results
|
|
905
|
-
ignoreExitCodes: [1], // treat exit code 1 as success; still samples, no nonzero-exit warning
|
|
906
|
-
failOnNonzero: false, // default; true stops a command's loop after its first bad exit
|
|
907
|
-
})
|
|
908
|
-
```
|
|
909
|
-
|
|
910
|
-
A command can also be an object - `{ command, label?, prepare?, timeSource? }` - whose
|
|
911
|
-
`prepare`/`timeSource` override the top-level ones for that command. That's how one
|
|
912
|
-
command becomes several labeled workloads in the same document:
|
|
913
|
-
|
|
914
|
-
```ts
|
|
915
|
-
const build = ["bun", "cli.ts", "build", "fixture"]
|
|
916
|
-
const inMs = { pattern: /in (\d+)ms/ }
|
|
917
|
-
const doc = await time({
|
|
918
|
-
commands: [
|
|
919
|
-
{ command: build, label: "warm", timeSource: inMs },
|
|
920
|
-
{ command: build, label: "incremental", timeSource: inMs, prepare: () => touchPost() },
|
|
921
|
-
{ command: build, label: "cold", timeSource: inMs, prepare: "rm -rf fixture/dist" },
|
|
922
|
-
{ command: build, label: "wall clock" }, // same command, no timeSource: wall time
|
|
923
|
-
],
|
|
924
|
-
samples: 5,
|
|
925
|
-
})
|
|
926
|
-
```
|
|
927
|
-
|
|
928
|
-
### `profile(fn, opts)` → `{ result, measurement, document }`
|
|
929
|
-
|
|
930
|
-
In-process capture. `origin: "jsc"` is the only path that reports JIT tiers
|
|
931
|
-
(LLInt / Baseline / DFG / FTL). Default `origin: "inspector"` writes portable CDP-shaped
|
|
932
|
-
evidence instead. `document` is a full `ProfileDocument` (the one workload and
|
|
933
|
-
measurement), so it composes with `renderers.*` or `saveDocument` directly. `profile()`
|
|
934
|
-
also takes a `signal?: AbortSignal`, but `fn` runs in this process - there's no child to
|
|
935
|
-
kill, so an already-aborted signal only skips the profiler instrumentation (still running
|
|
936
|
-
`fn` plain and returning its `result`, with an `aborted` warning in place of CPU evidence);
|
|
937
|
-
it can't interrupt `fn` once it's running.
|
|
938
|
-
|
|
939
|
-
```ts
|
|
940
|
-
const { result, measurement, document } = await profile(
|
|
941
|
-
() => hashLoop(8_000_000),
|
|
942
|
-
{ origin: "jsc", intervalUs: 100 },
|
|
943
|
-
)
|
|
944
|
-
|
|
945
|
-
console.log(measurement.jit?.tiers)
|
|
946
|
-
// {
|
|
947
|
-
// llint: 0,
|
|
948
|
-
// baseline: 9,
|
|
949
|
-
// dfg: 37,
|
|
950
|
-
// ftl: 2825,
|
|
951
|
-
// }
|
|
952
|
-
|
|
953
|
-
const { files } = await renderers.collapsed.render(document)
|
|
954
|
-
```
|
|
955
|
-
|
|
956
|
-
### `createDocument(workloads, measurements)` → `ProfileDocument`
|
|
957
|
-
|
|
958
|
-
For composing a document from several `profile()` calls (each of which returns
|
|
959
|
-
just one workload and measurement):
|
|
960
|
-
|
|
961
|
-
```ts
|
|
962
|
-
const a = await profile(() => taskA())
|
|
963
|
-
const b = await profile(() => taskB())
|
|
964
|
-
const document = createDocument(
|
|
965
|
-
[a.document.workloads[0]!, b.document.workloads[0]!],
|
|
966
|
-
[a.measurement, b.measurement],
|
|
967
|
-
)
|
|
968
|
-
```
|
|
969
|
-
|
|
970
|
-
### `group` / `task` / `bench`
|
|
971
|
-
|
|
972
|
-
Register in-process suites, then run them (same as `ostia bench`):
|
|
973
|
-
|
|
974
|
-
```ts
|
|
975
|
-
// suite.ts
|
|
976
|
-
import { group, task } from "ostia"
|
|
977
|
-
|
|
978
|
-
group("parse", () => {
|
|
979
|
-
task("small input", () => parse(smallBuf))
|
|
980
|
-
task("large input", () => parse(largeBuf))
|
|
981
|
-
// Per-task options override the suite-wide time budget / min samples.
|
|
982
|
-
task("full pipeline", () => build(), { budgetMs: 2000, minSamples: 10 })
|
|
983
|
-
})
|
|
984
|
-
```
|
|
985
|
-
|
|
986
|
-
That is the whole registration surface: `group()` and `task()`. Presentation lives in
|
|
987
|
-
the renderers (`--format`), not in the suite file.
|
|
988
|
-
|
|
989
|
-
All module-scope code in a suite file runs up front, before any task is sampled.
|
|
990
|
-
`{ before, after }` is the hook that runs a task's own setup immediately before its
|
|
991
|
-
sampling and its teardown immediately after - once each, unmeasured, in the task's
|
|
992
|
-
own process (so both work with `isolate`):
|
|
993
|
-
|
|
994
|
-
```ts
|
|
995
|
-
group(
|
|
996
|
-
"parse",
|
|
997
|
-
() => {
|
|
998
|
-
let doc: Document
|
|
999
|
-
task("append", () => doc.append(node), {
|
|
1000
|
-
before: () => {
|
|
1001
|
-
doc = mountDocument()
|
|
1002
|
-
},
|
|
1003
|
-
after: () => doc.destroy(),
|
|
1004
|
-
})
|
|
1005
|
-
},
|
|
1006
|
-
{
|
|
1007
|
-
// Runs once around the whole group, outside every task's own before/after.
|
|
1008
|
-
before: () => setupSharedFixture(),
|
|
1009
|
-
after: () => teardownSharedFixture(),
|
|
1010
|
-
},
|
|
1011
|
-
)
|
|
1012
|
-
```
|
|
1013
|
-
|
|
1014
|
-
There is no per-trial hook (no setup/teardown between individual samples) - that
|
|
1015
|
-
would defeat batching, which is how ostia keeps a sub-microsecond task's timer
|
|
1016
|
-
overhead down. Reach for `{ gc }` (`Bun.gc(true)` between trials) or `{ isolate }`
|
|
1017
|
-
(a fresh process per task) for per-trial concerns instead. Because `before`/`after`
|
|
1018
|
-
run once per task, not once per instance, a suite that builds more than one instance
|
|
1019
|
-
of something stateful (a mounted UI component, an open connection, a server) still
|
|
1020
|
-
has to scope a query to the instance it belongs to, not write it against a
|
|
1021
|
-
global/ambient lookup that assumes it's the only one alive - a suite that opens a
|
|
1022
|
-
component's menu and queries `getByRole(...)` unscoped, for example, breaks once a
|
|
1023
|
-
second instance of that component exists in the document; scope the query with
|
|
1024
|
-
something like `within(instance.container)` instead.
|
|
1025
|
-
|
|
1026
|
-
Task functions may be async (`() => unknown | Promise<unknown>`, same for
|
|
1027
|
-
`before`/`after`); `await` on a plain synchronous value still costs a microtask
|
|
1028
|
-
turn, so an `async` task function measures a few nanoseconds slower per call than
|
|
1029
|
-
the same body written synchronously - immaterial above microsecond cost, worth
|
|
1030
|
-
knowing for a task near the timer's resolution floor.
|
|
1031
|
-
|
|
1032
|
-
Call `keep(value)` on an intermediate value inside a task body - a subcomputation
|
|
1033
|
-
whose result the task doesn't return - to pin it against dead-code elimination the
|
|
1034
|
-
same way ostia already protects a task's own return value:
|
|
1035
|
-
|
|
1036
|
-
```ts
|
|
1037
|
-
import { keep } from "ostia"
|
|
1038
|
-
|
|
1039
|
-
task("parse then validate", () => {
|
|
1040
|
-
const ast = parse(input)
|
|
1041
|
-
keep(ast) // the task returns validate's result; without this, a smart-enough
|
|
1042
|
-
// JIT could in principle prove `ast` is otherwise unused and skip building it
|
|
1043
|
-
return validate(ast)
|
|
1044
293
|
})
|
|
1045
|
-
```
|
|
1046
294
|
|
|
1047
|
-
Both take an optional `description` that flows into the document
|
|
1048
|
-
(`Workload.description` / `Workload.groupDescription`) and into `--format minimal`, so
|
|
1049
|
-
what a number measures and why travels with the data instead of living only in a
|
|
1050
|
-
source comment a reader has to go find:
|
|
1051
|
-
|
|
1052
|
-
```ts
|
|
1053
|
-
group(
|
|
1054
|
-
"repaint",
|
|
1055
|
-
() => {
|
|
1056
|
-
task("1,000 chars", () => repaint(doc1k))
|
|
1057
|
-
task("4,000 chars", () => repaint(doc4k), {
|
|
1058
|
-
description: "worst case: full repaint every keystroke at the max document size",
|
|
1059
|
-
})
|
|
1060
|
-
},
|
|
1061
|
-
{ description: "editor repaint cost as document size grows" },
|
|
1062
|
-
)
|
|
1063
|
-
```
|
|
1064
|
-
|
|
1065
|
-
Mark one task per group as the `Relative` reference with `{ baseline: true }`;
|
|
1066
|
-
otherwise `Relative` defaults to the fastest task in the group:
|
|
1067
|
-
|
|
1068
|
-
```ts
|
|
1069
295
|
group("parse", () => {
|
|
1070
|
-
|
|
1071
|
-
|
|
1072
|
-
})
|
|
1073
|
-
```
|
|
1074
|
-
|
|
1075
|
-
`task.skip(...)` / `group.skip(...)` register without measuring: the runner never
|
|
1076
|
-
samples them, but the document still carries the workload (marked
|
|
1077
|
-
`Workload.skipped`), so a renderer prints `- skipped` instead of the task just
|
|
1078
|
-
being absent, and `compare` reports it as `unchanged` (with a `skipped` warning)
|
|
1079
|
-
rather than silently passing or failing to match a baseline. `task.only(...)` /
|
|
1080
|
-
`group.only(...)` restrict a suite file to only the `.only`-marked tasks - `--filter`
|
|
1081
|
-
still applies on top - and print a one-line notice (`bench: 2 task(s) selected by
|
|
1082
|
-
.only`) to stderr, so a forgotten `.only` doesn't quietly narrow a run in CI:
|
|
1083
|
-
|
|
1084
|
-
```ts
|
|
1085
|
-
group("parse", () => {
|
|
1086
|
-
task.only("fast path", () => parse(buf)) // only this task runs this time
|
|
1087
|
-
task("slow path", () => parseSlow(buf))
|
|
1088
|
-
task.skip("flaky on CI", () => parseFlaky(buf))
|
|
1089
|
-
})
|
|
1090
|
-
```
|
|
1091
|
-
|
|
1092
|
-
`{ cpu }` / `{ alloc }` on `task()` or `group()` override the suite-wide `bench({ cpu, alloc })`
|
|
1093
|
-
/ `--cpu` / `--alloc` default for that task or group, the same pattern as `isolate`/`gc`:
|
|
1094
|
-
one extra `phase: "cpu"` measurement (JIT tiers included, from a fixed 200ms window under
|
|
1095
|
-
the JSC sampling profiler) and/or one extra `phase: "memstats"` measurement
|
|
1096
|
-
(`MemoryEvidence.bytesPerOp`, from a `Bun.gc(true)`-bracketed batch of 100 calls) alongside
|
|
1097
|
-
the task's timing numbers, never mixed into them:
|
|
1098
|
-
|
|
1099
|
-
```ts
|
|
1100
|
-
group("parse", () => {
|
|
1101
|
-
task("current impl", () => parse(buf))
|
|
1102
|
-
task("candidate impl", () => parseFast(buf), { cpu: true, alloc: true })
|
|
1103
|
-
})
|
|
1104
|
-
```
|
|
1105
|
-
|
|
1106
|
-
### `sweep(dims, fn)` → `void`
|
|
1107
|
-
|
|
1108
|
-
Cartesian product over one or more named dimensions, calling `fn` once per point.
|
|
1109
|
-
`task()` calls inside `fn` automatically inherit that point as `Workload.params` -
|
|
1110
|
-
a structured alternative to baking the point into the task name, so renderers can
|
|
1111
|
-
pivot on it and `compare` matches on the same point across runs instead of just a name:
|
|
1112
|
-
|
|
1113
|
-
```ts
|
|
1114
|
-
import { group, task, range, sweep } from "ostia"
|
|
1115
|
-
|
|
1116
|
-
group("parse", () => {
|
|
1117
|
-
sweep({ size: range(100, 10_000), impl: ["current", "fast"] }, ({ size, impl }) => {
|
|
1118
|
-
const input = buildInput(size) // setup, runs once per point, unmeasured
|
|
1119
|
-
task(`${impl}`, () => impls[impl](input))
|
|
296
|
+
sweep({ size: range(100, 10_000) }, ({ size }) => {
|
|
297
|
+
const input = buildInput(size) // unmeasured setup, once per point
|
|
298
|
+
task("parse", () => parse(input), { isolate: true })
|
|
1120
299
|
})
|
|
1121
300
|
})
|
|
1122
|
-
```
|
|
1123
301
|
|
|
1124
|
-
|
|
1125
|
-
|
|
302
|
+
const result = compareDocuments(await loadDocument("before.json"), doc)
|
|
303
|
+
if (result.summary.verdict === "fail") process.exitCode = 1
|
|
1126
304
|
|
|
1127
|
-
|
|
1128
|
-
task(`${impl}`, () => impls[impl](input), { params: { size, impl, variant: "warm" } })
|
|
305
|
+
const { text } = await renderers.markdown.render(doc, {})
|
|
1129
306
|
```
|
|
1130
307
|
|
|
1131
|
-
|
|
1132
|
-
|
|
1133
|
-
it shares the same two param keys - exactly what the example above produces - and
|
|
1134
|
-
otherwise renders params as a `key=value` suffix on the task name.
|
|
1135
|
-
|
|
1136
|
-
### `range(start, end, multiplier?)` → `number[]`
|
|
1137
|
-
|
|
1138
|
-
Geometric point generator that feeds `sweep()` (and works standalone): default
|
|
1139
|
-
multiplier `8`, always ending on `end` even if the last step overshot it.
|
|
1140
|
-
|
|
1141
|
-
```ts
|
|
1142
|
-
range(100, 10_000) // -> [100, 800, 6400, 10000]
|
|
1143
|
-
range(100, 100_000) // -> [100, 800, 6400, 51200, 100000]
|
|
1144
|
-
```
|
|
1145
|
-
|
|
1146
|
-
### `bench(opts)` → `ProfileDocument`
|
|
1147
|
-
|
|
1148
|
-
In-process suite runner, same behavior as `ostia bench`.
|
|
1149
|
-
|
|
1150
|
-
```ts
|
|
1151
|
-
// demo.ts
|
|
1152
|
-
import { bench } from "ostia"
|
|
1153
|
-
|
|
1154
|
-
const doc = await bench({
|
|
1155
|
-
suites: ["suite.ts"],
|
|
1156
|
-
budgetMs: 500,
|
|
1157
|
-
// samples: 50, // exact per-task trial count instead of a budget
|
|
1158
|
-
minSamples: 50,
|
|
1159
|
-
jobs: 1, // suite files at once; > 1 trades fidelity for wall time
|
|
1160
|
-
noiseCheck: true, // default; set false to skip the ~200ms noise floor measurement
|
|
1161
|
-
})
|
|
1162
|
-
```
|
|
308
|
+
Full reference, including task options, hooks, and a mitata/hyperfine migration table:
|
|
309
|
+
[docs/library.md](docs/library.md).
|
|
1163
310
|
|
|
1164
|
-
|
|
311
|
+
## Configuration
|
|
1165
312
|
|
|
1166
|
-
|
|
1167
|
-
`bun suite.ts` (no `ostia bench` CLI, no `bench({ suites })` call) to execute every
|
|
1168
|
-
`group()`/`task()` registered so far, print a report, and return the document.
|
|
313
|
+
`ostia.config.ts` (checked first) or `ostia.config.json`, in the current directory.
|
|
1169
314
|
|
|
1170
315
|
```ts
|
|
1171
|
-
//
|
|
1172
|
-
import {
|
|
1173
|
-
import { group, run, task } from "ostia"
|
|
1174
|
-
|
|
1175
|
-
group("parse", () => {
|
|
1176
|
-
task("small input", () => parse(smallBuf))
|
|
1177
|
-
task("large input", () => parse(largeBuf))
|
|
1178
|
-
})
|
|
1179
|
-
|
|
1180
|
-
try {
|
|
1181
|
-
await run()
|
|
1182
|
-
} finally {
|
|
1183
|
-
await rm(fixtureDir, { recursive: true })
|
|
1184
|
-
}
|
|
1185
|
-
```
|
|
1186
|
-
|
|
1187
|
-
```sh
|
|
1188
|
-
bun suite.ts
|
|
1189
|
-
```
|
|
1190
|
-
|
|
1191
|
-
`run({ filter: "parse" })` narrows to matching `group/name` ids, same regex as `ostia bench
|
|
1192
|
-
--filter`/`bench({ filter })` - there's no CLI here to read a `--filter` flag from, so pass it
|
|
1193
|
-
as an option, e.g. from `process.argv` or an env var your `run()` call reads itself.
|
|
1194
|
-
|
|
1195
|
-
This trades away the isolation `ostia bench`/`bench()` give each suite file (and each
|
|
1196
|
-
isolated task under `--isolate`) its own fresh subprocess: everything under `run()` runs in
|
|
1197
|
-
the process that already imported the suite, so `TaskOptions.isolate` has nothing to isolate
|
|
1198
|
-
into and is ignored. Prefer `ostia bench`/`bench()` for numbers you'll `compare`/`ci`
|
|
1199
|
-
against; reach for `run()` for a single suite file's inline edit/run loop, or when a `finally`
|
|
1200
|
-
around the run needs to clean up fixtures the suite set up (`ostia bench`'s subprocess model
|
|
1201
|
-
has no call in the file that returns after every task finishes, so that cleanup would
|
|
1202
|
-
otherwise need a `process.on("exit", ...)` hook instead).
|
|
1203
|
-
|
|
1204
|
-
`run(opts)` accepts the same suite-wide `filter`/`budgetMs`/`samples`/`minSamples`/`warmup`/
|
|
1205
|
-
`gc`/`cpu`/`alloc`/`noiseCheck` fields as `bench(opts)`, plus `quiet` (skip the printed
|
|
1206
|
-
report, still return the document) and `format` (renderer for that report, default `"table"`).
|
|
1207
|
-
|
|
1208
|
-
### `compareDocuments(base, cand, thresholds?)` → `CompareResult`
|
|
1209
|
-
|
|
1210
|
-
Same matching and thresholds as `ostia compare` / `ostia ci`.
|
|
1211
|
-
|
|
1212
|
-
```ts
|
|
1213
|
-
const result = compareDocuments(baselineDoc, candidateDoc, {
|
|
1214
|
-
timingPct: 5,
|
|
1215
|
-
frameSelfPct: 10,
|
|
1216
|
-
heapTypePct: 10,
|
|
1217
|
-
minFrameSelfUs: 1000,
|
|
1218
|
-
alpha: 0.01, // Mann-Whitney significance level
|
|
1219
|
-
bootstrapIterations: 2000,
|
|
1220
|
-
})
|
|
1221
|
-
|
|
1222
|
-
result.comparisons // Comparison[], one per workload id present on both sides
|
|
1223
|
-
result.unmatched // { baseOnly: Workload[]; candOnly: Workload[] } - present on only one side
|
|
1224
|
-
result.summary // { matched, regressed, improved, unchanged, geomeanPct, effectiveTimingPct, verdict }
|
|
1225
|
-
```
|
|
1226
|
-
|
|
1227
|
-
`summary.geomeanPct` is the geometric mean of `cand/base` median ratios over matched timing
|
|
1228
|
-
comparisons, as a signed percent (negative: candidate faster on average); `null` when no
|
|
1229
|
-
comparison had a finite timing ratio. `summary.verdict` is `"fail"` when any comparison
|
|
1230
|
-
failed. `ostia compare` persists `result.comparisons` as `comparisons`, `result.summary` as
|
|
1231
|
-
`comparisonSummary`, and `result.unmatched`'s workload ids (not full `Workload`s, to keep the
|
|
1232
|
-
document small) as `unmatched: { baseOnly: string[]; candOnly: string[] }` on the candidate
|
|
1233
|
-
document it writes/renders.
|
|
1234
|
-
|
|
1235
|
-
### `saveDocument` / `loadDocument`
|
|
1236
|
-
|
|
1237
|
-
```ts
|
|
1238
|
-
await saveDocument(doc, "doc.json")
|
|
1239
|
-
const loaded: ProfileDocument = await loadDocument("doc.json")
|
|
1240
|
-
```
|
|
1241
|
-
|
|
1242
|
-
### `defineConfig(config)` → `Partial<OstiaConfig>`
|
|
1243
|
-
|
|
1244
|
-
Identity function purely for typing `ostia.config.ts` - see
|
|
1245
|
-
[`ostia.config.ts` / `ostia.config.json`](#ostiaconfigts--ostiaconfigjson) above.
|
|
1246
|
-
|
|
1247
|
-
### `renderers`
|
|
1248
|
-
|
|
1249
|
-
Pure functions of a `ProfileDocument`. Each returns `{ text? }` and/or `{ files? }`.
|
|
1250
|
-
|
|
1251
|
-
| Name | Output |
|
|
1252
|
-
|---|---|
|
|
1253
|
-
| `table` | terminal timing / CPU / heap / comparison text |
|
|
1254
|
-
| `markdown` | agent- and human-readable report |
|
|
1255
|
-
| `json` | pretty JSON document |
|
|
1256
|
-
| `jsonl` | one `kind: "document"` header line, then one `kind: "measurement"` line per run |
|
|
1257
|
-
| `minimal` | protocol v1: one `run`/`unmatched`/`summary` event per line, no sample array; for LLM/CI consumption (see [Using ostia from an AI agent](#using-ostia-from-an-ai-agent)) |
|
|
1258
|
-
| `collapsed` | folded stacks (`name;name;name count`) |
|
|
1259
|
-
| `mermaid` | top-N call tree |
|
|
1260
|
-
| `speedscope` | speedscope.app JSON |
|
|
1261
|
-
| `cpuprofile` | verbatim `.cpuprofile` when a CDP artifact exists |
|
|
316
|
+
// ostia.config.ts
|
|
317
|
+
import { defineConfig } from "ostia"
|
|
1262
318
|
|
|
1263
|
-
|
|
1264
|
-
|
|
1265
|
-
|
|
1266
|
-
|
|
319
|
+
export default defineConfig({
|
|
320
|
+
baseline: "main",
|
|
321
|
+
samples: 15, // command workloads; or budgetMs/minSamples
|
|
322
|
+
warmup: 3,
|
|
323
|
+
thresholds: { timingPct: 5 },
|
|
324
|
+
workloads: [
|
|
325
|
+
{ label: "cold-start", command: ["bun", "src/cli.ts", "--help"], inputs: ["src/**/*.ts"] },
|
|
326
|
+
{ label: "spawn", command: ["bun", "-e", "1"], inputs: [] },
|
|
327
|
+
{ label: "build:cold", command: ["bun", "build.ts"], prepare: "rm -rf dist" },
|
|
328
|
+
{ label: "suites", suites: ["bench/*.ts"] },
|
|
329
|
+
],
|
|
330
|
+
bench: { budgetMs: 500, isolate: true, preload: ["bench/setup.ts"] },
|
|
1267
331
|
})
|
|
1268
332
|
```
|
|
1269
333
|
|
|
1270
|
-
|
|
334
|
+
A config that still uses the old `runs` field fails to load with a message naming
|
|
335
|
+
`samples` (error code `config-invalid`). All fields: [docs/config.md](docs/config.md).
|
|
1271
336
|
|
|
1272
|
-
##
|
|
337
|
+
## Documentation
|
|
1273
338
|
|
|
1274
|
-
|
|
1275
|
-
|
|
1276
|
-
|
|
1277
|
-
|
|
1278
|
-
|
|
1279
|
-
|
|
1280
|
-
|
|
1281
|
-
| `do_not_optimize(value)` | `keep(value)` |
|
|
1282
|
-
| `hyperfine -L var a,b,c cmd-{var}` | `sweep({ var: ["a", "b", "c"] }, ...)` or per-command `params` |
|
|
1283
|
-
| `hyperfine --runs N --warmup N` | `ostia time --samples N --warmup N` |
|
|
1284
|
-
| `hyperfine --prepare CMD` | `ostia time --prepare CMD` (also `time({ prepare })` / config `prepare`) |
|
|
1285
|
-
| `hyperfine --export-json` / `--export-markdown` | `ostia time --export-json PATH` / `--format markdown` |
|
|
1286
|
-
|
|
1287
|
-
`sweep()`/`range()`/`params` are in-process (`ostia bench`); `hyperfine -L` substitutes into
|
|
1288
|
-
a shell command template for a subprocess instead, so a direct port is one literal `ostia time`
|
|
1289
|
-
command per substitution value rather than a template - see [`sweep(dims, fn)`](#sweepdims-fn--void)
|
|
1290
|
-
and [`ostia.config.ts` / `ostia.config.json`](#ostiaconfigts--ostiaconfigjson) above.
|
|
339
|
+
- [docs/cli.md](docs/cli.md): every command and flag
|
|
340
|
+
- [docs/config.md](docs/config.md): config file reference
|
|
341
|
+
- [docs/library.md](docs/library.md): library API reference
|
|
342
|
+
- [docs/agent-protocol.md](docs/agent-protocol.md): `--format minimal`, exit codes, error codes
|
|
343
|
+
- [docs/statistics.md](docs/statistics.md): sampling, the comparison test, noise floor
|
|
344
|
+
- [docs/document-schema.md](docs/document-schema.md): `ProfileDocument`, workload ids, warnings
|
|
345
|
+
- [docs/preload-recipes.md](docs/preload-recipes.md): jsdom, happy-dom and `Bun.plugin()` preloads
|
|
1291
346
|
|
|
1292
347
|
## Examples
|
|
1293
348
|
|
|
1294
|
-
[`examples/`](examples/) has
|
|
1295
|
-
|
|
1296
|
-
|
|
1297
|
-
|
|
1298
|
-
|
|
1299
|
-
|
|
1300
|
-
|
|
1301
|
-
- [`profile-in-process`](examples/profile-in-process/). `profile(fn, { origin: "jsc" })`.
|
|
1302
|
-
- [`benchmark-a-function`](examples/benchmark-a-function/). `bench()` / `group()` / `task()`.
|
|
349
|
+
[`examples/`](examples/) has runnable recipes (they use `../../src` directly, no install):
|
|
350
|
+
[`compare-two-commands`](examples/compare-two-commands/),
|
|
351
|
+
[`find-a-hotspot`](examples/find-a-hotspot/),
|
|
352
|
+
[`heap-usage`](examples/heap-usage/),
|
|
353
|
+
[`gate-a-regression`](examples/gate-a-regression/),
|
|
354
|
+
[`profile-in-process`](examples/profile-in-process/),
|
|
355
|
+
[`benchmark-a-function`](examples/benchmark-a-function/).
|
|
1303
356
|
|
|
1304
357
|
```sh
|
|
1305
358
|
cd examples/find-a-hotspot && bun run demo
|
|
1306
|
-
bun run examples # all of them from the repo root
|
|
359
|
+
bun run examples # all of them, from the repo root
|
|
1307
360
|
```
|