@cosyte/bench 0.1.1 → 0.1.3

package/CHANGELOG.md

@@ -1,5 +1,57 @@
 # Changelog
 
+## 0.1.3
+
+### Patch Changes
+
+- 027dbf5: A reading measures the empty-call floor once per run: the peak of a call that does nothing, read in
+  its own process through the same probe as the cases, which is what the probe itself costs. It is in
+  the reading as `memoryFloor`, the Markdown prints it, and every case's peak also gives each figure
+  above the floor's median, never below zero: `peakTotalAboveFloorBytes`,
+  `medianPeakTotalAboveFloorBytes`, and the same for the heap and external memory.
+
+  `budgetPeakBytes(caseReading)` is the peak a memory budget reads. For a case whose inputs average
+  under `BENCH_CONTRACT.FLOOR_INPUT_BYTES` (64 KiB) it is the median call's total above the floor's
+  median; for any other, or for a suite that gave no byte count, the highest raw `peakTotalBytes`, as
+  before. On a GitHub-hosted runner the floor's median call is about 3.3 KiB and its highest about
+  28 KiB, and that highest lands in a small case's highest peak, so a small input reads its median
+  call. A budget script that read `peakTotalBytes` should call `budgetPeakBytes` instead, and a suite
+  that added its own empty "floor" case can drop it.
+
+  One case that cannot be measured no longer stops the run. `cosyte-bench run` writes every other case,
+  lists what failed in the reading's `failures` (the phase, `timing`, `memory` or `floor`, the case,
+  and the first line of the error) and in the Markdown, prints the whole error to stderr, and exits 2.
+  A case whose timing failed is left out of `cases`; one whose memory failed keeps its timing. A script
+  that ran each case alone to get past one failure can run them together and read `failures`.
+
+  Every new field is added within results version 1, and documents written before still validate.
+
+## 0.1.2
+
+### Patch Changes
+
+- c0c095a: Large cases are measured instead of skipped. A case whose single call takes longer than a warmup
+  batch (50 ms) and has not settled by the scaling gate's rule after 500 ms moves to the large-case
+  plan: one call at a time until the median of the last 5 calls is within 5% of the median of the 5
+  before them, at most 30 s. Each timing process records the plan it ended on (`plan` in a reading's
+  `throughput.processes`, `basePlans` and `headPlans` in a comparison), and a reading marks such a
+  case. A case that settles by the standard rule warms up exactly as before.
+
+  Allocations per operation are counted on a case whose every window collects: the largest pair of
+  windows is counted with what its collections reclaimed (`method: "gc-accounted"`, with the number of
+  `collections`), where before the case was skipped as `allocation-window-gc`.
+
+  The peak reports external memory (Buffers, ArrayBuffers) beside the V8 heap, and the two together:
+  `peakExternalBytes`, `peakTotalBytes` and their medians. A memory budget reads `peakTotalBytes`.
+  Memory processes run with `--no-concurrent-array-buffer-sweeping`, so a collection frees dead
+  ArrayBuffers before it ends and the external memory it reports is exact.
+
+  The long run also runs on a benchmark's largest size whose 30 rounds fit in 60 s, besides its
+  long-run size. A reading now times every size of a benchmark before measuring its memory.
+
+  Every new field is added within results version 1 and none is renamed; documents written before
+  still validate.
+
 ## 0.1.1
 
 ### Patch Changes

package/README.md

@@ -98,14 +98,21 @@ cosyte-bench check bench.json
 Exit codes: 0 clean, 1 a finding (a regression, or a results file that does not match the schema),
 2 could not measure (a skip, a process that died, an input it could not read). A skip is not a pass.
 
+A `run` whose measuring process fails for one case still measures and writes every other case. The
+reading lists what failed in `failures` (the phase, `timing`, `memory` or `floor`, the case, and the
+first line of the error; the whole error goes to stderr) and the command exits 2. A case whose timing
+failed is left out of `cases`; one whose memory failed keeps its timing with its memory `null`. A
+case whose warmup never settled is not a failure: it is in `cases`, its throughput `skipped` with
+the reason `warmup-unstable`.
+
 ## Entry points
 
-| entry point                                   | what it is                                                                                                                                            |
-| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `cosyte-bench`                                | the bin: `run`, `compare`, `calibrate` and `check`                                                                                                    |
-| `@cosyte/bench`                               | writing a suite (`defineSuite`, `defineBenchmark`), judging ratios (`judge`), reading results (`readResults`, `RESULTS_SCHEMA`), and `BENCH_CONTRACT` |
-| `@cosyte/bench/schema/results-v1.schema.json` | the results schema as JSON Schema, for a tool in another language                                                                                     |
-| `@cosyte/bench/package.json`                  | the manifest                                                                                                                                          |
+| entry point                                   | what it is                                                                                                                                                               |
+| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `cosyte-bench`                                | the bin: `run`, `compare`, `calibrate` and `check`                                                                                                                       |
+| `@cosyte/bench`                               | writing a suite (`defineSuite`, `defineBenchmark`), judging ratios (`judge`), reading results (`readResults`, `RESULTS_SCHEMA`, `budgetPeakBytes`), and `BENCH_CONTRACT` |
+| `@cosyte/bench/schema/results-v1.schema.json` | the results schema as JSON Schema, for a tool in another language                                                                                                        |
+| `@cosyte/bench/package.json`                  | the manifest                                                                                                                                                             |
 
 The module entry point has no side effects: importing it starts no process. The schema is plain JSON:
 
@@ -131,21 +138,54 @@ the command line:
 
 - the sizes, and which of them a pull request's comparison runs (`compare: false` leaves a slow size
   to the readings);
-- `weigh`, the number a result is reduced to, and `longRun`, the size the long run uses;
+- `weigh`, the number a result is reduced to, and `longRun`, the size the long run uses besides the
+  largest size whose long run fits in 60 s;
 - `--only <benchmark or case>`, `--processes <n>`, `--no-memory` and `--no-import` on `run`.
 
 ## What it measures
 
-| Measure                                | How                                                                                                                                                                              | What it cannot see                                                                                                 |
-| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
-| Throughput, small to very large inputs | Per size, fresh processes; a warmup by the scaling gate's rule; 5 samples of whole passes of at least 50 ms; the median of the processes' medians                                | Anything the suite's sizes do not cover; numbers from another machine                                              |
-| Latency p50, p95, p99                  | One call at a time on a nanosecond clock, up to 2 s or 5,000 calls; a percentile is reported only with at least 10 calls above it                                                | Calls shorter than 20 clock reads (skipped, typed); the tail of a size with few calls                              |
-| Allocations per operation              | Heap growth over windows of `n` and `2n` operations in which no collection ran, in a process with a 64 MiB young generation; the difference cancels the cost of reading the heap | A window that always collects (skipped, typed)                                                                     |
-| Peak heap                              | Per call, the higher of the heap just before any collection during the call and at its end, above the settled heap before it                                                     | Memory outside the V8 heap                                                                                         |
-| Heap flat over a long run              | 30 rounds, the heap settled after each; a Theil-Sen slope over the last 24; growing when the fitted growth reaches 64 KiB                                                        | A leak too small to reach 64 KiB over the run (each result says the smallest it could see, `detectableBytesPerOp`) |
-| Cold import and loaded bytes           | 20 fresh processes import the target, 20 import an empty module, interleaved; one more records every file the import loads, with its gzip size                                   | What a bundler keeps after tree shaking                                                                            |
-
-Memory measurements need `gc()`; the runner starts those processes with `--expose-gc` itself.
+| Measure                                | How                                                                                                                                                                                                                                                                                                                                                                             | What it cannot see                                                                                                                  |
+| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| Throughput, small to very large inputs | Per size, fresh processes; a warmup by the scaling gate's rule, or by the large-case plan below when every warmup batch is one long pass; 5 samples of whole passes of at least 50 ms; the median of the processes' medians                                                                                                                                                     | Anything the suite's sizes do not cover; numbers from another machine                                                               |
+| Latency p50, p95, p99                  | One call at a time on a nanosecond clock, up to 2 s or 5,000 calls; a percentile is reported only with at least 10 calls above it                                                                                                                                                                                                                                               | Calls shorter than 20 clock reads (skipped, typed); the tail of a size with few calls                                               |
+| Allocations per operation              | Heap growth over windows of `n` and `2n` operations in which no collection ran, in a process with a 64 MiB young generation; the difference cancels the cost of reading the heap. When every window collects, the largest pair is counted with what its collections reclaimed (`method: "gc-accounted"`)                                                                        | Native memory a library allocates outside V8's accounting                                                                           |
+| Peak memory                            | Per call, the higher of what was in use just before any collection during the call and at its end, above what was in use before it: the V8 heap (`peakHeapBytes`), external memory such as Buffers and ArrayBuffers (`peakExternalBytes`), and the two together (`peakTotalBytes`); each also above the run's empty-call floor (`peakTotalAboveFloorBytes` and the like), below | Native memory a library allocates outside V8's accounting                                                                           |
+| Heap flat over a long run              | 30 rounds, the heap settled after each; a Theil-Sen slope over the last 24; growing when the fitted growth reaches 64 KiB. On the long-run size and on the largest size whose 30 rounds fit in 60 s                                                                                                                                                                             | A leak too small to reach 64 KiB over the run (each result says the smallest it could see, `detectableBytesPerOp`); external memory |
+| Cold import and loaded bytes           | 20 fresh processes import the target, 20 import an empty module, interleaved; one more records every file the import loads, with its gzip size                                                                                                                                                                                                                                  | What a bundler keeps after tree shaking                                                                                             |
+
+Memory measurements need `gc()`; the runner starts those processes with `--expose-gc` itself, and
+with `--no-concurrent-array-buffer-sweeping`, so a collection frees the ArrayBuffers it found dead
+before it ends and the external memory it reports before and after is exact.
+
+### The empty-call floor and small inputs
+
+Reading the peak costs memory of its own: settling the heap, the profiler that records each
+collection, the record itself. A reading therefore measures an empty call once, in a fresh process
+through the same probe (`memoryFloor`), and gives each peak figure both raw and above the floor's
+median, never below zero (`peakTotalAboveFloorBytes`, `medianPeakTotalAboveFloorBytes`, and the same
+for the heap and external memory).
+
+On a GitHub-hosted runner the floor's median call is about 3.3 KiB and its highest about 28 KiB (one
+call, the first). That highest call is the probe's, and it lands in a case's highest peak too, which
+is why a few hundred bytes of input could not read under 20 times its size whatever the code did.
+
+A memory budget reads `budgetPeakBytes(caseReading)`: for a case whose inputs average under
+`BENCH_CONTRACT.FLOOR_INPUT_BYTES` (64 KiB) it is the median call's total above the floor's median,
+the code's typical call without the probe; for any other case, or one whose suite gave no byte
+count, it is the highest raw `peakTotalBytes`. At 64 KiB even the probe's highest call is about 2%
+of a 20 times budget, so a large input keeps the raw highest figure, which needs nothing from
+another process. A suite that added its own empty "floor" case can drop it.
+
+### Large cases
+
+A case whose single call takes longer than a warmup batch (50 ms) cannot settle by the scaling
+gate's warmup rule on a shared runner: every batch is one call, and one call's jitter is what the
+batch was there to average away. When 500 ms of warmup have passed, the rule has not settled, and
+the last three batches were each one call of at least 50 ms, the warmup moves to the large-case
+plan: one call at a time until the median of the last 5 calls is within 5% of the median of the 5
+before them, at most 30 s from the start. Its samples are the same 5, each one call at that size.
+Each process records the plan it ended on (`plan`), and a reading marks such a case. A case that
+settles by the standard rule never moves, so small cases are measured exactly as before.
 
 ## The comparison
 
@@ -189,6 +229,8 @@ CI runner image, never by host name, user or absolute path.
 - `judge(ratios, threshold?, alpha?)`: the comparison's verdict for one case from its per-round ratios.
 - `readResults(text)`: parse and check a results document; throws a `TypeError` listing every
   problem. `RESULTS_SCHEMA` is the schema, also shipped as `schema/results-v1.schema.json`.
+- `budgetPeakBytes(caseReading)`: the peak a memory budget reads for a case: the median call above
+  the empty-call floor for an input under 64 KiB, the highest raw total for any other.
 - `BENCH_CONTRACT`: the frozen constants.
 - `BenchError`: what a run that could not measure throws.
 

package/dist/child-entry.mjs

@@ -21,6 +21,7 @@ var BENCH_CONTRACT = Object.freeze({
   ALLOC_WINDOW_MS: 100,
   ALLOC_SEMI_SPACE_MIB: 64,
   PEAK_CALLS: 20,
+  FLOOR_INPUT_BYTES: 64 * 1024,
   LONG_RUN_ROUNDS: 30,
   LONG_RUN_SETTLE_ROUNDS: 5,
   LONG_RUN_ROUND_MS: 100,
@@ -29,6 +30,10 @@ var BENCH_CONTRACT = Object.freeze({
   GC_SETTLE_ROUNDS: 2,
   GC_MAX_ROUNDS: 7,
   IMPORT_RUNS: 20,
+  LARGE_PASS_MS: 50,
+  LARGE_WARMUP_WINDOW: 5,
+  LARGE_WARMUP_MAX_MS: 3e4,
+  LONG_RUN_MAX_MS: 6e4,
   REQUIRED_FALSE_ALARM_RATE: 0.01,
   REQUIRED_CONFIDENCE: 0.95
 });
@@ -78,9 +83,27 @@ function realProbe() {
       profiler.start();
       fn();
       const stats = profiler.stop().statistics;
-      let peak = 0;
-      for (const s of stats) peak = Math.max(peak, s.beforeGC.heapStatistics.usedHeapSize);
-      return { gcs: stats.length, peakBeforeGc: peak };
+      const report = {
+        gcs: stats.length,
+        peakBeforeGc: 0,
+        peakExternalBeforeGc: 0,
+        peakTotalBeforeGc: 0,
+        reclaimedHeap: 0,
+        reclaimedExternal: 0
+      };
+      for (const s of stats) {
+        const before = s.beforeGC.heapStatistics;
+        const after = s.afterGC.heapStatistics;
+        report.peakBeforeGc = Math.max(report.peakBeforeGc, before.usedHeapSize);
+        report.peakExternalBeforeGc = Math.max(report.peakExternalBeforeGc, before.externalMemory);
+        report.peakTotalBeforeGc = Math.max(
+          report.peakTotalBeforeGc,
+          before.usedHeapSize + before.externalMemory
+        );
+        report.reclaimedHeap += before.usedHeapSize - after.usedHeapSize;
+        report.reclaimedExternal += before.externalMemory - after.externalMemory;
+      }
+      return report;
     }
   };
 }
@@ -102,43 +125,63 @@ function measureAllocations(call, probe, msPerOp2) {
     const s = settle(probe);
     const h0 = probe.usedHeap();
     const e0 = probe.external();
-    const { gcs } = probe.profile(() => {
+    const p = probe.profile(() => {
       for (let i = 0; i < ops; i++) call(i);
     });
-    return { settled: s.settled, gcs, heap: probe.usedHeap() - h0, ext: probe.external() - e0 };
+    return {
+      settled: s.settled,
+      gcs: p.gcs,
+      heap: probe.usedHeap() - h0 + p.reclaimedHeap,
+      ext: probe.external() - e0 + p.reclaimedExternal
+    };
   };
+  const perOp = (method, n, a2, b2) => ({
+    status: "measured",
+    method,
+    collections: a2.gcs + b2.gcs,
+    windowOps: n,
+    heapBytesPerOp: round((b2.heap - a2.heap) / n, 1),
+    externalBytesPerOp: round((b2.ext - a2.ext) / n, 1)
+  });
   const paced = Math.floor(BENCH_CONTRACT.ALLOC_WINDOW_MS / msPerOp2);
   const first = Math.max(1, Math.min(BENCH_CONTRACT.ALLOC_WINDOW_OPS, paced));
+  let largest = null;
   for (let n = first; n >= 1; n = Math.floor(n / 2)) {
-    const a = window(n);
-    const b = window(2 * n);
-    if (!a.settled || !b.settled) return { status: "skipped", reason: "gc-unsettled" };
-    if (a.gcs === 0 && b.gcs === 0) {
-      return {
-        status: "measured",
-        windowOps: n,
-        heapBytesPerOp: round((b.heap - a.heap) / n, 1),
-        externalBytesPerOp: round((b.ext - a.ext) / n, 1)
-      };
-    }
+    const a2 = window(n);
+    const b2 = window(2 * n);
+    if (!a2.settled || !b2.settled) return { status: "skipped", reason: "gc-unsettled" };
+    if (a2.gcs === 0 && b2.gcs === 0) return perOp("no-gc-window", n, a2, b2);
+    largest ??= [a2, b2];
   }
-  return { status: "skipped", reason: "allocation-window-gc" };
+  const [a, b] = largest;
+  return perOp("gc-accounted", first, a, b);
 }
 function measurePeak(call, probe) {
-  const peaks = [];
+  const heap = [];
+  const external = [];
+  const total = [];
   for (let i = 0; i < BENCH_CONTRACT.PEAK_CALLS; i++) {
     const s = settle(probe);
     if (!s.settled) return { status: "skipped", reason: "gc-unsettled" };
-    const { peakBeforeGc } = probe.profile(() => {
+    const e0 = probe.external();
+    const p = probe.profile(() => {
       call(i);
     });
-    peaks.push(Math.max(peakBeforeGc, probe.usedHeap()) - s.heap);
+    const h1 = probe.usedHeap();
+    const e1 = probe.external();
+    heap.push(Math.max(p.peakBeforeGc, h1) - s.heap);
+    external.push(Math.max(p.peakExternalBeforeGc, e1) - e0);
+    total.push(Math.max(p.peakTotalBeforeGc, h1 + e1) - (s.heap + e0));
   }
   return {
     status: "measured",
-    calls: peaks.length,
-    peakHeapBytes: Math.max(...peaks),
-    medianPeakHeapBytes: median(peaks)
+    calls: heap.length,
+    peakHeapBytes: Math.max(...heap),
+    medianPeakHeapBytes: median(heap),
+    peakExternalBytes: Math.max(...external),
+    medianPeakExternalBytes: median(external),
+    peakTotalBytes: Math.max(...total),
+    medianPeakTotalBytes: median(total)
   };
 }
 function measureLongRun(pass, opsPerPass, probe, clock) {
@@ -208,6 +251,17 @@ function batchesAreStable(batches) {
   const mid = median(window);
   return window.every((b) => Math.abs(b - mid) / mid <= WARMUP.STABLE_TOL);
 }
+function isLargeCase(batches) {
+  if (batches.length < WARMUP.STABLE_BATCHES) return false;
+  return batches.slice(-WARMUP.STABLE_BATCHES).every((b) => b >= BENCH_CONTRACT.LARGE_PASS_MS);
+}
+function windowsAreStable(passes) {
+  const w = BENCH_CONTRACT.LARGE_WARMUP_WINDOW;
+  if (passes.length < 2 * w) return false;
+  const last = median(passes.slice(-w));
+  const before = median(passes.slice(-2 * w, -w));
+  return Math.abs(last - before) / before <= WARMUP.STABLE_TOL;
+}
 function warmUp(pass, clock) {
   const start = clock();
   const batches = [];
@@ -224,10 +278,31 @@ function warmUp(pass, clock) {
     passes += inBatch;
     batches.push(batchMs / inBatch);
     const elapsedMs = clock() - start;
-    if (elapsedMs >= WARMUP.MIN_MS && batchesAreStable(batches)) {
-      return { stable: true, elapsedMs, passes, batches };
+    if (elapsedMs >= WARMUP.MIN_MS) {
+      if (batchesAreStable(batches)) {
+        return { stable: true, elapsedMs, passes, batches, plan: "standard" };
+      }
+      if (isLargeCase(batches)) return warmUpLarge(pass, clock, start, batches, passes);
+    }
+    if (elapsedMs >= WARMUP.MAX_MS) {
+      return { stable: false, elapsedMs, passes, batches, plan: "standard" };
+    }
+  }
+}
+function warmUpLarge(pass, clock, start, batches, passesSoFar) {
+  let passes = passesSoFar;
+  for (; ; ) {
+    const p0 = clock();
+    pass();
+    const t = clock();
+    passes++;
+    batches.push(t - p0);
+    const elapsedMs = t - start;
+    if (windowsAreStable(batches))
+      return { stable: true, elapsedMs, passes, batches, plan: "large" };
+    if (elapsedMs >= BENCH_CONTRACT.LARGE_WARMUP_MAX_MS) {
+      return { stable: false, elapsedMs, passes, batches, plan: "large" };
     }
-    if (elapsedMs >= WARMUP.MAX_MS) return { stable: false, elapsedMs, passes, batches };
   }
 }
 function sampleThroughput(pass, opsPerPass, clock, inject = 0) {
@@ -413,10 +488,17 @@ function measureJob(job, shape, warmup, pass, call, deps) {
   };
 }
 function measuredSomething(result) {
+  if (result.mode === "floor") return result.peak.status === "measured";
   if (result.mode === "timing") return result.status === "measured";
   return [result.allocations, result.peak, result.longRun].some((r) => r?.status === "measured");
 }
 async function runJob(job, deps = realDeps()) {
+  if (job.mode === "floor") {
+    return {
+      mode: "floor",
+      peak: deps.probe === null ? SKIPPED_NO_GC : measurePeak(() => void 0, deps.probe)
+    };
+  }
   const suiteModule = await import(pathToFileURL(job.suite).href);
   const suite = checkSuite(suiteModule.default);
   const target = await import(pathToFileURL(job.target).href);