ostia 0.1.6 → 0.2.0

package/README.md

@@ -1,11 +1,20 @@
 # ostia
 
-ostia is a profiling and benchmarking toolkit for Bun.
-
-Use it when you want wall-clock timings, CPU hotspots, heap summaries, or JIT tier
-data in one place, then diff those results or fail CI when something gets slower.
-Everything lands in one schema-versioned JSON document (`ProfileDocument`), so the
-CLI and the library speak the same language.
+ostia is a profiling and benchmarking toolkit for Bun. One schema-versioned JSON
+document (`ProfileDocument`) holds wall-clock timing, CPU hotspots, heap summaries,
+JIT tier data, and allocation counts for a subprocess command or an in-process
+function - the CLI and the library speak the same language, so anything you can do
+with `ostia time`/`ostia bench` you can do with `time()`/`bench()` too.
+
+Comparing two documents reports a bootstrap confidence interval and a Mann-Whitney
+p-value, not a bare percentage past a threshold, and widens the regression bar to
+the machine's own noise floor so ambient jitter is never mistaken for a real
+regression. `ostia ci` gates a whole `ostia.config.json`/`ostia.config.ts` of
+workloads - subprocess commands and in-process `group()`/`task()` suites alike -
+against a saved baseline, skipping anything whose input fingerprint hasn't changed.
+Any task can run in its own subprocess for clean JIT/heap isolation from its
+suite-mates, and every renderer has a `minimal` JSON-lines mode built for piping
+straight into an LLM agent's context.
 
 Zero runtime dependencies. Requires Bun ≥ 1.4.
 
@@ -20,31 +29,52 @@ bun add ostia
 Compare two commands:
 
 ```sh
-ostia run --runs 10 --warmup 2 "bun fixtures/fast.ts" "bun fixtures/slow.ts"
+ostia time --samples 10 --warmup 2 "bun fixtures/fast.ts" "bun fixtures/slow.ts"
 ```
 
 ```
-Command                Mean [ms]        Min…Max [ms]        Relative
---------------------------------------------------------------------
-bun fixtures/fast.ts   8.413 ± 1.340    7.623…12.365        1.00×
-bun fixtures/slow.ts   21.712 ± 1.157   20.831…24.220       2.62× slower
+Apple M2 · 8 cores · load 10.1 · noise floor 6.2%
+
+Task                   Median     Spread             Range              Relative
+--------------------------------------------------------------------------------
+bun fixtures/fast.ts   16.6 ms    25.3 ms…32.8 ms    10.4 ms…33.2 ms    1.00×
+  ! noisy-machine
+bun fixtures/slow.ts   40.2 ms    48.2 ms…63.9 ms    28.6 ms…64.7 ms    2.43× slower
+
+Warnings:
+  bun fixtures/fast.ts: Load average 10.10 exceeds 75% of 8 available core(s); timing noise may be elevated.
 ```
 
+The header line (machine, cores, load, noise floor) prints whenever a document
+carries an `environment` - on by default; skip the ~200ms reference
+measurement with `--no-noise-check`. `compare`/`ci` widen the regression
+threshold to at least the noise floor, so a change smaller than the machine's
+own jitter is never called a regression - see
+[Statistics](#statistics-a-real-significance-test-not-a-percentage-threshold)
+below.
+
 Find a CPU hotspot (profiler runs as a separate labeled trial, never mixed into the
 timing numbers above):
 
 ```sh
-ostia run --runs 5 --cpu --cpu-interval 200 --export-json node_modules/.cache/ostia/doc.json fixtures/work.ts
+ostia time --samples 5 --cpu --cpu-interval 200 --export-json node_modules/.cache/ostia/doc.json "bun fixtures/work.ts"
 ```
 
 ```
-Command                Mean [ms]        Min…Max [ms]
-----------------------------------------------------
-bun fixtures/work.ts   275.815 ± 2.853  273.334…281.384
+Apple M2 · 8 cores · load 8.5 · noise floor 0.9%
+
+Task                   Median     Spread             Range
+-----------------------------------------------------------------------
+bun fixtures/work.ts   470.6 ms   478.7 ms…499.3 ms  400.8 ms…500.1 ms
+  ! noisy-machine
+
+Warnings:
+  bun fixtures/work.ts: Load average 8.48 exceeds 75% of 8 available core(s); timing noise may be elevated.
 
-CPU capture - bun fixtures/work.ts (instrumented, 200µs interval, diagnostic wall 297.578ms)
-  100.0%    284.19ms self  hashLoop
+CPU capture - bun fixtures/work.ts (instrumented, 200µs interval, diagnostic wall 531.143ms)
+  100.0%    502.67ms self  hashLoop
     0.0%      0.00ms self  (root)
+    0.0%      0.00ms self  (module)
   artifact: node_modules/.cache/ostia/artifacts/<run-id>-cpu.cpuprofile
 ```
 
@@ -57,8 +87,8 @@ they need to survive `node_modules` reinstalls between branches and CI jobs, so
 Gate a change against a local baseline:
 
 ```sh
-bun run baseline   # on known-good: measure ostia.config.json -> .ostia/baselines/main.json
-ostia ci           # on your branch: rerun changed workloads, exit 1 on regression
+ostia baseline save   # on known-good: measure ostia.config.json -> .ostia/baselines/main.json
+ostia ci              # on your branch: rerun changed workloads, exit 1 on regression
 ```
 
 ```
@@ -83,56 +113,81 @@ Profile CI: ✓
 ## CLI reference
 
 ```
-ostia run <command...>       time commands; optional --cpu / --heap capture
+ostia time <command...>      time commands; optional --cpu / --heap capture
 ostia bench <suite.ts...>    in-process group()/task() suites (time-budgeted)
 ostia compare <a> <b>        diff two ProfileDocuments
-ostia report <document.json> render a saved document
-ostia viz <document.json>    render CPU evidence to a file format
+ostia report <document.json> render a saved document (table/json/markdown/collapsed/...)
 ostia ci                     run configured workloads vs a baseline, gate regressions
+ostia baseline save|list|show  manage baseline ProfileDocuments
 ```
 
 Every subcommand takes `--help` for its full flag list.
 
-### `ostia run`
+### `ostia time`
 
 Clean wall-clock timing by default. `--cpu` / `--heap` schedule one extra instrumented
 trial each, labeled separately in the document.
 
 ```sh
-ostia run "bun a.ts" "bun b.ts"
-ostia run --runs 25 --warmup 3 --cpu --heap "bun src/server.ts"
-ostia run --format json --export-json out.json "bun a.ts"
-```
+ostia time "bun a.ts" "bun b.ts"
+ostia time --samples 25 --warmup 3 --cpu --heap "bun src/server.ts"
+ostia time --format json --export-json out.json "bun a.ts"
+```
+
+`--samples N` is an exact trial count; `--budget MS`
+is a wall-clock time budget instead (default: a hyperfine-style ~3s min-total-time
+loop when neither is given); `--min-samples N` is a hard floor when `--samples` isn't
+given. The same three names work on `ostia bench` (`--budget`/`--samples`/
+`--min-samples`), where `--budget` is a per-task sampling window -
+`warmup` differs by surface, though: a trial count here, a
+*fraction* of the budget for `ostia bench`, since in-process warmup has no natural
+"N calls" unit before the JIT has even seen the function once.
+
+With 2+ commands, trials round-robin across them by default (one trial per command,
+repeated) rather than running one command's whole loop to completion before the next
+starts - drift over the run's wall-clock span (thermal throttling, a noisy neighbor
+process) then lands on every command equally instead of favoring whichever ran first
+or last. `--no-interleave` (`interleave: false`) goes back to running each command's
+loop to completion in turn. Interleaved measurements carry `Measurement.interleaved: true`.
+Meaningless (and ignored) with a single command.
 
 Timing table (two commands get a Relative column automatically):
 
 ```
-Command                Mean [ms]        Min…Max [ms]        Relative
---------------------------------------------------------------------
-bun fixtures/fast.ts   8.413 ± 1.340    7.623…12.365        1.00×
-bun fixtures/slow.ts   21.712 ± 1.157   20.831…24.220       2.62× slower
+Task                   Median     Spread             Range              Relative
+--------------------------------------------------------------------------------
+bun fixtures/fast.ts   7.94 ms    8.31 ms…8.56 ms    7.68 ms…8.57 ms    1.00×
+bun fixtures/slow.ts   21.3 ms    21.5 ms…22.1 ms    21.1 ms…22.1 ms    2.69× slower
+  ! outliers-detected
+
+Warnings:
+  bun fixtures/slow.ts: 1 outlier(s) detected (1 severe, 0 mild).
 ```
 
 Heap summary (type counts from the snapshot trial):
 
 ```
-Command                    Mean [ms]        Min…Max [ms]
---------------------------------------------------------
-bun fixtures/allocate.ts   27.079 ± 6.844   23.888…91.799
+Task                       Median     Spread             Range
+---------------------------------------------------------------------------
+bun fixtures/allocate.ts   23.8 ms    24.9 ms…29.4 ms    22.6 ms…30.0 ms
+  ! outliers-detected
+
+Warnings:
+  bun fixtures/allocate.ts: 8 outlier(s) detected (1 severe, 7 mild).
 
-Heap snapshot - bun fixtures/allocate.ts (instrumented, 2518 objects, 0.12MB)
+Heap snapshot - bun fixtures/allocate.ts (instrumented, 2516 objects, 0.12MB)
     1369  string
-     426  code
-     321  closure
+     423  code
+     319  closure
      216  object shape
-     104  hidden
+     105  hidden
   artifact: node_modules/.cache/ostia/artifacts/<run-id>-heap.heapsnapshot
 ```
 
 ### `ostia bench`
 
 In-process microbenchmarks registered with `group()` / `task()`. Each task samples
-for `--time-budget` (default 500ms). `--min-samples` is a hard floor kept even when it
+for `--budget` (default 500ms). `--min-samples` is a hard floor kept even when it
 overruns the budget. Left unset, the floor is cost-aware in both directions: as many
 trials as fit in the budget (capped at 20) so one slow task can't blow the suite's total,
 but never below the floor a task's per-trial cost earns it - 3 at ≤1ms, two more per
@@ -155,7 +210,7 @@ without re-deriving the policy from the raw sample array.
 
 ```sh
 ostia bench bench/*.ts
-ostia bench --time-budget 500 --min-samples 50 bench/stats.ts
+ostia bench --budget 500 --min-samples 50 bench/stats.ts
 ostia bench bench/*.ts --jobs auto          # suite files in parallel, see below
 ostia bench bench/*.ts --format minimal       # one compact JSON object per task
 ```
@@ -182,13 +237,31 @@ Bun/V8 batch calls together and amortize it away). `task(name, fn, { gc })` /
 override pattern as `isolate` - useful when a few allocation-heavy tasks need GC settled
 between trials but the rest of the suite doesn't.
 
+`--cpu` captures one extra `phase: "cpu"` measurement per task on top of its timing
+numbers: the task looped for a fixed 200ms window under the JSC sampling profiler
+(JIT tiers included), never mixed into the timing numbers themselves. `--alloc` captures
+an extra `phase: "memstats"` measurement: bytes allocated per call, from a
+`Bun.gc(true)`-bracketed batch of 100 calls (`MemoryEvidence.bytesPerOp`). Both follow the
+same per-task/per-group override pattern as `isolate`/`gc`: `task(name, fn, { cpu, alloc })`
+/ `group(name, fn, { cpu, alloc })`. The terminal table prints an `Alloc/op` column when a
+`memstats` measurement is present. With `--cpu` on, `ostia compare` reports per-frame CPU
+deltas for bench tasks the same way it already does for `ostia time --cpu`.
+
+When a `--cpu` capture spends more than 20% of its samples in the llint/baseline tiers,
+the JIT never warmed the task up in that 200ms window, so its CPU numbers (and by
+extension its timing) may not reflect steady state - the cpu measurement carries a
+`jit-cold` warning (`{ llintPct, baselinePct, dfgPct, ftlPct }`), printed alongside the
+CPU capture in the terminal table and folded into the task's line in `--format minimal`.
+
 `--preload PATH` (repeatable) imports a script before each suite file loads, in the same
 subprocess - the same shape as Bun's own `--preload` / `bunfig.toml`'s `preload` array. Use
 it to install globals a suite needs at import time (jsdom's `document`/`window`) or register
 a `Bun.plugin()` file-loader (e.g. compiling `.svelte`/`.vue` SFCs) before the suite's own
 top-level code runs. Multiple `--preload` scripts run in the order given, so state one
 installs (a plugin registration, a global) is visible to the next and to the suite itself.
-ostia ships none of this itself - just the hook point:
+ostia ships none of this itself - just the hook point (for a full jsdom/happy-dom global
+setup or a `Bun.plugin()` component-compile hook, see
+[docs/preload-recipes.md](docs/preload-recipes.md)):
 
 ```ts
 // bench/jsdom-setup.ts
@@ -201,15 +274,45 @@ Object.assign(globalThis, { document: dom.window.document, window: dom.window })
 ostia bench --preload ./bench/jsdom-setup.ts bench/*.dom.bench.ts
 ```
 
+`--bun-flags FLAGS` (repeatable, space-separated flags within one value are all appended)
+passes extra flags through to the `bun` invocation that spawns each suite file - the fix for
+packages whose `package.json` `exports` map branches on a resolution condition Bun doesn't
+set by default. Svelte 5's `exports` map, for example, is `{ "browser": "./src/index-client.js",
+"default": "./src/index-server.js" }`: without `--conditions browser`, Bun resolves `default`
+(the server-rendering build), and mounting a component via `@testing-library/svelte` throws
+`lifecycle_function_unavailable` since `mount()` isn't available server-side. The same applies
+to Vue and other dual-target frameworks:
+
+```sh
+ostia bench --bun-flags="--conditions=browser" bench/*.dom.bench.ts
+```
+
+Unlike `BUN_OPTIONS` (an env var Bun's CLI reads to prepend flags, which only reaches the
+spawned suite process today because ostia's `Bun.spawn()` happens to inherit `process.env`),
+`--bun-flags` is a declared, documented integration point that doesn't depend on the parent
+shell's environment.
+
 
 ```
-Command                                  Mean [ms]        Min…Max [ms]        Relative
---------------------------------------------------------------------------------------
-stats/computeTimingStats (1e3 samples)   0.014 ± 0.016    0.012…0.598         1.00×
-stats/computeTimingStats (1e4 samples)   0.484 ± 0.052    0.434…0.680         38.22× slower
-stats/timingWarnings (1e3 samples)       0.036 ± 0.020    0.032…0.541         2.82× slower
+Task                                       Median     Spread             Range              Relative
+----------------------------------------------------------------------------------------------------
+stats:
+  stats/computeTimingStats (1e3 samples)   24.7 µs    33.8 µs…151.6 µs   22.4 µs…1073.2 µs  1.00×
+    ! outliers-detected
+  stats/computeTimingStats (1e4 samples)   255.2 µs   331.7 µs…1067.6 µs 223.2 µs…20789.4 µs 10.33× slower
+    ! outliers-detected
+  stats/timingWarnings (1e3 samples)       47.3 µs    88.5 µs…510.4 µs   43.0 µs…14383.1 µs 1.91× slower
+    ! outliers-detected
+
+Warnings:
+  stats/computeTimingStats (1e3 samples): 2455 outlier(s) detected (2108 severe, 347 mild).
+  stats/computeTimingStats (1e4 samples): 215 outlier(s) detected (70 severe, 145 mild).
+  stats/timingWarnings (1e3 samples): 541 outlier(s) detected (191 severe, 350 mild).
 ```
 
+Tasks with a `group()` print the group name once, indented; ungrouped tasks and
+subprocess commands print flat.
+
 ### `ostia compare`
 
 Match workloads by id, rank timing / frame / heap deltas, print a verdict per workload.
@@ -220,16 +323,63 @@ ostia compare after.json --baseline .ostia/baselines/main.json
 ```
 
 ```
-✓ bun fixtures/fast.ts
-  timing: -2.9% median (unchanged)
+✗ bun fixtures/work.ts
+  timing: +11.2% median, 95% CI [+10.0%, +16.4%], p<0.001 (regressed)
+```
+
+When both documents carry `git` metadata (see below), `ostia compare` prints a summary
+line above the verdicts:
 
-✗ work
-  timing: +1249.2% median (regressed)
 ```
+base a1b2c3d (main) → cand d4e5f6a (my-opt, dirty)
+```
+
+The verdict needs both a confidence interval clear of `timingPct` and a
+significant Mann-Whitney p-value (`thresholds.alpha`, default `0.01`), not
+just a point estimate past the threshold - see
+[Statistics](#statistics-a-real-significance-test-not-a-percentage-threshold)
+below. Comparisons with fewer than 5 samples on either side fall back to the
+old point-estimate rule and carry a `thin-comparison` warning instead.
+
+#### Statistics: a real significance test, not a percentage threshold
+
+A point estimate past `timingPct` is not enough to call something a
+regression - both documents already carry full sample arrays, so `compare`
+runs two tests instead:
+
+- A **bootstrap confidence interval** on the difference of medians:
+  resample both sides with replacement `thresholds.bootstrapIterations`
+  times (default 2000; each side is randomly subsampled to at most 2000
+  samples first, so a many-thousand-sample task doesn't turn a compare into
+  a multi-second operation), and report the 2.5th/97.5th percentiles as
+  `ci95` (percent of the baseline median). Reproducible: the PRNG seed is
+  stored in `Comparison.timing.seed`.
+- A **Mann-Whitney U test** (tie-corrected, normal approximation), reported
+  as `pValue` - whether the two sample distributions differ at all, without
+  assuming normality the way a t-test would.
+
+`regressed` requires `ci95[0] > thresholds.timingPct` (the *whole interval*
+clears the threshold) **and** `pValue < thresholds.alpha`; `improved` is the
+mirror. Otherwise `unchanged`. This is why the earlier example (`+11.2%
+median, 95% CI [+10.0%, +16.4%]`) is a clean regression: even the low end of
+the interval is well past `timingPct`.
+
+Both `time()` and `bench()` also stamp `environment` on every document (a
+fixed-cost, deterministic, allocation-free hash loop sampled for ~200ms,
+`noise.floorPct = mad / median`) unless `noiseCheck: false` / `--no-noise-check`
+skips it. `compare` widens the effective threshold to
+`max(thresholds.timingPct, base.environment.noise.floorPct,
+cand.environment.noise.floorPct)` (`Comparison.thresholds.effectiveTimingPct`),
+so a delta smaller than the machine's own jitter right now is never called a
+regression. A `noisy-machine` warning fires when the 1-minute load average is
+already past 75% of available cores at measurement time.
 
 ### `ostia report`
 
-Render a saved `ProfileDocument` without re-running anything.
+Render a saved `ProfileDocument` without re-running anything. `--format` covers
+both the data formats (`table`/`json`/`jsonl`/`markdown`/`minimal`) and the CPU
+visualization formats (`collapsed`/`mermaid`/`speedscope`/`cpuprofile`) - one
+command instead of two.
 
 ```sh
 ostia report out.json                 # table (default)
@@ -245,7 +395,7 @@ sample (tens of thousands for a fast task), which is tokens a reviewer never rea
 Numbers stay in ns so they line up with `compare` deltas and the JSON document.
 
 ```
-{"task":"diffText()/append at end","group":"diffText()","samples":9282,"mean":50213.4,"median":49871,"stddev":2104.7,"stddevPct":4.19,"min":48120,"max":81002,"relative":1,"warnings":[],"unit":"ns"}
+{"task":"diffText()/append at end","group":"diffText()","samples":9282,"mean":50213.4,"median":49871,"stddev":2104.7,"stddevPct":4.19,"min":48120,"max":81002,"p75":50920,"p99":58011,"mad":1780,"relative":1,"warnings":[],"unit":"ns"}
 {"task":"repaint/4000 chars","group":"repaint","description":"full repaint every keystroke","samples":3,"mean":2.61e9,"median":2.4e9,"stddevPct":15.3,"relative":47800,"warnings":[{"code":"low-sample-count","data":{"samples":3,"target":10}}],"unit":"ns"}
 ```
 
@@ -257,39 +407,42 @@ Markdown:
 ```
 # Profile Report
 
-Bun 1.4.0 · ostia 0.1.0 · darwin/arm64 · 2026-09-04T03:46:02.961Z
+Bun 1.4.1 · ostia 0.1.0 · darwin/arm64 · 2026-09-05T13:14:50.085Z · a1b2c3d (main)
 
 ## Timing
 
-| Command | Mean ± SD (ms) | Min…Max (ms) | Median (ms) |
-|---|---|---|---|
-| bun -e 1 | 5.477 ± 0.303 | 5.153…5.882 | 5.396 |
+| Task | Median | Spread (p75…p99) | Mean ± SD | Range | MAD |
+|---|---|---|---|---|---|
+| bun -e 1 | 5.03 ms | 5.42 ms…9.70 ms | 5.29 ms ± 0.84 ms | 4.77 ms…13.4 ms | 0.16 ms |
 ```
 
-### `ostia viz`
+#### CPU visualization formats
 
 Turn CPU evidence into files for other tools. Formats: `collapsed`, `mermaid`,
 `speedscope`, `cpuprofile` (pass-through of a real CDP artifact when present).
+`--measurement <id>` renders only that measurement (default: every CPU
+measurement in the document); `--out-dir PATH` writes files there instead of
+stdout.
 
 ```sh
-ostia viz doc.json --format collapsed
-ostia viz doc.json --format mermaid
-ostia viz doc.json --format speedscope > flame.json
+ostia report doc.json --format collapsed
+ostia report doc.json --format mermaid
+ostia report doc.json --format speedscope > flame.json
 ```
 
 Collapsed stacks (one line per stack; feeds `flamegraph.pl` and friends):
 
 ```
-(root);(module);hashLoop 1055
+(root);(module);hashLoop 209
 ```
 
 Mermaid call tree (top N nodes by self time, never the whole profile):
 
 ```
 graph TD
-  n1["(root) (self 0.00ms, total 284.19ms)"]
-  n2["(module) (self 0.00ms, total 284.19ms)"]
-  n3["hashLoop (self 284.19ms, total 284.19ms)"]
+  n1["(root) (self 0.00ms, total 267.91ms)"]
+  n2["(module) (self 0.00ms, total 267.91ms)"]
+  n3["hashLoop (self 267.91ms, total 267.91ms)"]
   n1 --> n2
   n2 --> n3
 ```
@@ -304,6 +457,7 @@ ostia ci
 ostia ci --full                  # ignore cache
 ostia ci --baseline main
 ostia ci --export-json out.json
+ostia ci --save-baseline         # after a pass, promote today's numbers to the baseline
 ```
 
 Pass:
@@ -325,16 +479,36 @@ Fail:
 1 affected by this change
 0 cached
 1 executed
-0 passed  1 regressed (+1249.2% median on work)
+0 passed  1 regressed (+1278.7% median on work)
 
 Profile CI: ✗
 ```
 
 Exit codes: `0` pass, `1` regression, `2` harness error (missing config/baseline, spawn failure).
 
-#### `ostia.config.json`
+#### `ostia.config.ts` / `ostia.config.json`
+
+`loadConfig` looks for `ostia.config.ts` first (Bun imports TypeScript natively), falling
+back to `ostia.config.json`. Both forms are fully supported; pick `.ts` for autocomplete
+and type-checking on every field, via `defineConfig` (an identity function purely for
+typing, the same pattern as Vite/Vitest/ESLint):
+
+```ts
+// ostia.config.ts
+import { defineConfig } from "ostia"
+
+export default defineConfig({
+  baseline: "main",
+  thresholds: { timingPct: 5 },
+  workloads: [
+    { label: "parse", command: ["bun", "bench/parse.ts"], inputs: ["src/**/*.ts"] },
+    { label: "dogfood-suites", suites: ["bench/*.ts"] },
+  ],
+})
+```
 
 ```json
+// ostia.config.json - equivalent, no defineConfig wrapper needed
 {
   "baseline": "main",
   "thresholds": { "timingPct": 5 },
@@ -343,12 +517,26 @@ Exit codes: `0` pass, `1` regression, `2` harness error (missing config/baseline
       "label": "parse",
       "command": ["bun", "bench/parse.ts"],
       "inputs": ["src/**/*.ts"]
+    },
+    {
+      "label": "dogfood-suites",
+      "suites": ["bench/*.ts"]
     }
   ]
 }
 ```
 
-`inputs` is optional. Workloads with no `inputs` always rerun (cache fails conservative).
+Each workload is exactly one of `command` (a subprocess timed with `runs`/`warmup`) or
+`suites` (glob patterns, same resolution as `bench`'s own `suites` config, run via
+`bench()`). A `suites` workload gates every task in those files individually - one
+candidate-vs-baseline comparison per task, matched by workload id the same way a `command`
+workload already is, so `ostia ci`'s regression detection covers in-process microbenchmarks,
+not only subprocess commands. Unlike `command` workloads, a `suites` workload always
+executes (there's no cheap way to know a suite file's task ids, and so its per-task cache
+keys, without importing it first) - `inputs`-based cache skipping is `command`-only for now.
+
+`inputs` is optional (and, for now, only consulted for `command` workloads). Workloads
+with no `inputs` always rerun (cache fails conservative).
 
 Two directory options, both optional: `outDir` (default `node_modules/.cache/ostia`) for
 scratch/cache/artifacts, and `baselineDir` (default `.ostia/baselines`) for baselines. They're
@@ -359,11 +547,23 @@ independent - `baselineDir` doesn't move just because you override `outDir`.
 Baselines are JSON under `.ostia/baselines/` (gitignored). `ostia ci` only needs the file
 on disk; it does not need to be committed.
 
+`ostia baseline save [name]` measures every configured workload (the same code path
+`ostia ci` gates against, no comparison) and writes it to `<baselineDir>/<name>.json`
+(default name: config's `"baseline"` field, or `"main"`). `ostia baseline list` shows every
+saved baseline (name, created date, workload count, and git sha/branch when available);
+`ostia baseline show <name> [--format]` renders one (delegates to `ostia report`).
+
+Every document stamps `git: { sha, branch, dirty }` (from `git rev-parse` / `git status
+--porcelain` in the process's cwd, 200ms timeout, silently absent outside a repo or
+without `git` installed) - metadata only, never part of any fingerprint or id, so a
+commit or a dirty working tree never orphans a cached run or baseline. Printed in the
+markdown report's header line and `ostia baseline list`.
+
 Local branch workflow:
 
 ```sh
 git checkout master          # known-good tip
-bun run baseline             # -> .ostia/baselines/main.json
+ostia baseline save          # -> .ostia/baselines/main.json
 
 git checkout -b my-opt
 # ... change code ...
@@ -374,10 +574,15 @@ The baseline survives branch switches because it is not tracked. Re-seed only wh
 intentionally accept a new floor. Seeding on the branch you are guarding compares that
 branch to itself.
 
+`ostia ci --save-baseline` folds that re-seed into the gate itself: after a pass (no
+regressions), it writes the just-measured document as the new baseline at the same path
+it just compared against - useful in a CI job that gates every merge to a trunk branch,
+so each green run becomes the next run's floor with no separate step.
+
 One-off outside this repo's config:
 
 ```sh
-ostia run --export-json .ostia/baselines/main.json "bun bench.ts"
+ostia time --export-json .ostia/baselines/main.json "bun bench.ts"
 ostia ci
 ```
 
@@ -391,13 +596,17 @@ The CLI is a thin wrapper around the library. Same `ProfileDocument` either way.
 
 ```ts
 import {
-  run,
+  time,
   profile,
   bench,
   group,
   task,
   range,
+  sweep,
+  keep,
   compareDocuments,
+  createDocument,
+  defineConfig,
   renderers,
   saveDocument,
   loadDocument,
@@ -405,41 +614,62 @@ import {
 import type { ProfileDocument } from "ostia"
 ```
 
-### `run(opts)` → `ProfileDocument`
+### `time(opts)` → `ProfileDocument`
 
-Subprocess timing, optional CPU/heap capture. Same behavior as `ostia run`.
+Subprocess timing, optional CPU/heap capture. Same behavior as `ostia time`.
 
 ```ts
-const doc = await run({
+const doc = await time({
   commands: ["bun a.ts", "bun b.ts"],
-  runs: 10,
+  samples: 10, // exact trial count
+  // budgetMs: 3000,   // wall-clock budget instead of an exact count
+  // minSamples: 10,   // hard floor when samples isn't given
   warmup: 2,
+  interleave: true, // default when 2+ commands: round-robins trials across them
   cpu: true,
   heap: false,
   cpuIntervalUs: 200,
   outDir: "node_modules/.cache/ostia", // default; artifacts land under here
+  noiseCheck: true, // default; set false to skip the ~200ms noise floor measurement
 })
 ```
 
-### `profile(fn, opts)` → `{ result, run }`
+### `profile(fn, opts)` → `{ result, measurement, document }`
 
 In-process capture. `origin: "jsc"` is the only path that reports JIT tiers
 (LLInt / Baseline / DFG / FTL). Default `origin: "inspector"` writes portable CDP-shaped
-evidence instead.
+evidence instead. `document` is a full `ProfileDocument` (the one workload and
+measurement), so it composes with `renderers.*` or `saveDocument` directly.
 
 ```ts
-const { result, run } = await profile(() => hashLoop(8_000_000), {
-  origin: "jsc",
-  intervalUs: 100,
-})
+const { result, measurement, document } = await profile(
+  () => hashLoop(8_000_000),
+  { origin: "jsc", intervalUs: 100 },
+)
 
-console.log(run.jit?.tiers)
+console.log(measurement.jit?.tiers)
 // {
 //   llint: 0,
 //   baseline: 9,
 //   dfg: 37,
 //   ftl: 2825,
 // }
+
+const { files } = await renderers.collapsed.render(document)
+```
+
+### `createDocument(workloads, measurements)` → `ProfileDocument`
+
+For composing a document from several `profile()` calls (each of which returns
+just one workload and measurement):
+
+```ts
+const a = await profile(() => taskA())
+const b = await profile(() => taskB())
+const document = createDocument(
+  [a.document.workloads[0]!, b.document.workloads[0]!],
+  [a.measurement, b.measurement],
+)
 ```
 
 ### `group` / `task` / `bench`
@@ -454,13 +684,71 @@ group("parse", () => {
   task("small input", () => parse(smallBuf))
   task("large input", () => parse(largeBuf))
   // Per-task options override the suite-wide time budget / min samples.
-  task("full pipeline", () => build(), { timeBudgetMs: 2000, minSamples: 10 })
+  task("full pipeline", () => build(), { budgetMs: 2000, minSamples: 10 })
 })
 ```
 
 That is the whole registration surface: `group()` and `task()`. Presentation lives in
 the renderers (`--format`), not in the suite file.
 
+All module-scope code in a suite file runs up front, before any task is sampled.
+`{ before, after }` is the hook that runs a task's own setup immediately before its
+sampling and its teardown immediately after - once each, unmeasured, in the task's
+own process (so both work with `isolate`):
+
+```ts
+group(
+  "parse",
+  () => {
+    let doc: Document
+    task("append", () => doc.append(node), {
+      before: () => {
+        doc = mountDocument()
+      },
+      after: () => doc.destroy(),
+    })
+  },
+  {
+    // Runs once around the whole group, outside every task's own before/after.
+    before: () => setupSharedFixture(),
+    after: () => teardownSharedFixture(),
+  },
+)
+```
+
+There is no per-trial hook (no setup/teardown between individual samples) - that
+would defeat batching, which is how ostia keeps a sub-microsecond task's timer
+overhead down. Reach for `{ gc }` (`Bun.gc(true)` between trials) or `{ isolate }`
+(a fresh process per task) for per-trial concerns instead. Because `before`/`after`
+run once per task, not once per instance, a suite that builds more than one instance
+of something stateful (a mounted UI component, an open connection, a server) still
+has to scope a query to the instance it belongs to, not write it against a
+global/ambient lookup that assumes it's the only one alive - a suite that opens a
+component's menu and queries `getByRole(...)` unscoped, for example, breaks once a
+second instance of that component exists in the document; scope the query with
+something like `within(instance.container)` instead.
+
+Task functions may be async (`() => unknown | Promise<unknown>`, same for
+`before`/`after`); `await` on a plain synchronous value still costs a microtask
+turn, so an `async` task function measures a few nanoseconds slower per call than
+the same body written synchronously - immaterial above microsecond cost, worth
+knowing for a task near the timer's resolution floor.
+
+Call `keep(value)` on an intermediate value inside a task body - a subcomputation
+whose result the task doesn't return - to pin it against dead-code elimination the
+same way ostia already protects a task's own return value:
+
+```ts
+import { keep } from "ostia"
+
+task("parse then validate", () => {
+  const ast = parse(input)
+  keep(ast) // the task returns validate's result; without this, a smart-enough
+  // JIT could in principle prove `ast` is otherwise unused and skip building it
+  return validate(ast)
+})
+```
+
 Both take an optional `description` that flows into the document
 (`Workload.description` / `Workload.groupDescription`) and into `--format minimal`, so
 what a number measures and why travels with the data instead of living only in a
@@ -479,9 +767,8 @@ group(
 )
 ```
 
-Mark one task per group as the `Relative` reference with `{ baseline: true }`
-(mirrors mitata's `baseline()`); otherwise `Relative` defaults to the fastest
-task in the group:
+Mark one task per group as the `Relative` reference with `{ baseline: true }`;
+otherwise `Relative` defaults to the fastest task in the group:
 
 ```ts
 group("parse", () => {
@@ -490,24 +777,72 @@ group("parse", () => {
 })
 ```
 
-### `range(start, end, multiplier?)` → `number[]`
+`task.skip(...)` / `group.skip(...)` register without measuring: the runner never
+samples them, but the document still carries the workload (marked
+`Workload.skipped`), so a renderer prints `- skipped` instead of the task just
+being absent, and `compare` reports it as `unchanged` (with a `skipped` warning)
+rather than silently passing or failing to match a baseline. `task.only(...)` /
+`group.only(...)` restrict a suite file to only the `.only`-marked tasks - `--filter`
+still applies on top - and print a one-line notice (`bench: 2 task(s) selected by
+.only`) to stderr, so a forgotten `.only` doesn't quietly narrow a run in CI:
+
+```ts
+group("parse", () => {
+  task.only("fast path", () => parse(buf)) // only this task runs this time
+  task("slow path", () => parseSlow(buf))
+  task.skip("flaky on CI", () => parseFlaky(buf))
+})
+```
 
-Geometric sweep points for parameterizing `task()` over a size dimension - mitata's
-`.range(name, start, end, multiplier)` point generation (default multiplier `8`, always
-ending on `end` even if the last step overshot it), without the name templating: build
-the task name yourself in the loop.
+`{ cpu }` / `{ alloc }` on `task()` or `group()` override the suite-wide `bench({ cpu, alloc })`
+/ `--cpu` / `--alloc` default for that task or group, the same pattern as `isolate`/`gc`:
+one extra `phase: "cpu"` measurement (JIT tiers included, from a fixed 200ms window under
+the JSC sampling profiler) and/or one extra `phase: "memstats"` measurement
+(`MemoryEvidence.bytesPerOp`, from a `Bun.gc(true)`-bracketed batch of 100 calls) alongside
+the task's timing numbers, never mixed into them:
 
 ```ts
-import { group, task, range } from "ostia"
+group("parse", () => {
+  task("current impl", () => parse(buf))
+  task("candidate impl", () => parseFast(buf), { cpu: true, alloc: true })
+})
+```
+
+### `sweep(dims, fn)` → `void`
+
+Cartesian product over one or more named dimensions, calling `fn` once per point.
+`task()` calls inside `fn` automatically inherit that point as `Workload.params` -
+a structured alternative to baking the point into the task name, so renderers can
+pivot on it and `compare` matches on the same point across runs instead of just a name:
+
+```ts
+import { group, task, range, sweep } from "ostia"
 
 group("parse", () => {
-  for (const size of range(100, 10_000)) {
+  sweep({ size: range(100, 10_000), impl: ["current", "fast"] }, ({ size, impl }) => {
     const input = buildInput(size) // setup, runs once per point, unmeasured
-    task(`${size} items`, () => parse(input))
-  }
+    task(`${impl}`, () => impls[impl](input))
+  })
 })
 ```
 
+An explicit `{ params }` on a particular `task()` call merges over (and wins
+against) the current sweep point:
+
+```ts
+task(`${impl}`, () => impls[impl](input), { params: { size, impl, variant: "warm" } })
+```
+
+`--format minimal` includes `params` on every line. The markdown renderer pivots a
+group into a table (rows = first dimension, columns = second) when every task in
+it shares the same two param keys - exactly what the example above produces - and
+otherwise renders params as a `key=value` suffix on the task name.
+
+### `range(start, end, multiplier?)` → `number[]`
+
+Geometric point generator that feeds `sweep()` (and works standalone): default
+multiplier `8`, always ending on `end` even if the last step overshot it.
+
 ```ts
 range(100, 10_000)   // -> [100, 800, 6400, 10000]
 range(100, 100_000)  // -> [100, 800, 6400, 51200, 100000]
@@ -519,9 +854,11 @@ import { bench } from "ostia"
 
 const doc = await bench({
   suites: ["suite.ts"],
-  timeBudgetMs: 500,
+  budgetMs: 500,
+  // samples: 50,   // exact per-task trial count instead of a budget
   minSamples: 50,
   jobs: 1, // suite files at once; > 1 trades fidelity for wall time
+  noiseCheck: true, // default; set false to skip the ~200ms noise floor measurement
 })
 ```
 
@@ -535,6 +872,8 @@ const diffs = compareDocuments(baselineDoc, candidateDoc, {
   frameSelfPct: 10,
   heapTypePct: 10,
   minFrameSelfUs: 1000,
+  alpha: 0.01, // Mann-Whitney significance level
+  bootstrapIterations: 2000,
 })
 ```
 
@@ -545,6 +884,11 @@ await saveDocument(doc, "doc.json")
 const loaded: ProfileDocument = await loadDocument("doc.json")
 ```
 
+### `defineConfig(config)` → `Partial<OstiaConfig>`
+
+Identity function purely for typing `ostia.config.ts` - see
+[`ostia.config.ts` / `ostia.config.json`](#ostiaconfigts--ostiaconfigjson) above.
+
 ### `renderers`
 
 Pure functions of a `ProfileDocument`. Each returns `{ text? }` and/or `{ files? }`.
@@ -563,11 +907,31 @@ Pure functions of a `ProfileDocument`. Each returns `{ text? }` and/or `{ files?
 
 ```ts
 const { text } = await renderers.markdown.render(doc)
-const { files } = await renderers.collapsed.render(doc, { runId: someCpuRunId })
+const { files } = await renderers.collapsed.render(doc, {
+  measurementId: someCpuMeasurementId,
+})
 ```
 
 Units in the IR are fixed: ns (time), bytes (memory), µs (sampling interval).
 
+## Migrating from mitata or hyperfine
+
+| mitata / hyperfine | ostia |
+|---|---|
+| `bench("name", fn)` | `task("name", fn)` |
+| `baseline()` | `{ baseline: true }` on a `task()` |
+| `.range(name, start, end, mult)` | `sweep({ dim: range(start, end, mult) }, ...)` |
+| generator setup (`function* () { ...; yield () => fn() }`) | `task(name, fn, { before, after })` |
+| `do_not_optimize(value)` | `keep(value)` |
+| `hyperfine -L var a,b,c cmd-{var}` | `sweep({ var: ["a", "b", "c"] }, ...)` or per-command `params` |
+| `hyperfine --runs N --warmup N` | `ostia time --samples N --warmup N` |
+| `hyperfine --export-json` / `--export-markdown` | `ostia time --export-json PATH` / `--format markdown` |
+
+`sweep()`/`range()`/`params` are in-process (`ostia bench`); `hyperfine -L` substitutes into
+a shell command template for a subprocess instead, so a direct port is one literal `ostia time`
+command per substitution value rather than a template - see [`sweep(dims, fn)`](#sweepdims-fn--void)
+and [`ostia.config.ts` / `ostia.config.json`](#ostiaconfigts--ostiaconfigjson) above.
+
 ## Examples
 
 [`examples/`](examples/) has six runnable recipes (they spawn `../../src/cli/main.ts`