@cosyte/bench 0.1.1 → 0.1.2

package/CHANGELOG.md

@@ -1,5 +1,31 @@
 # Changelog
 
+## 0.1.2
+
+### Patch Changes
+
+- c0c095a: Large cases are measured instead of skipped. A case whose single call takes longer than a warmup
+  batch (50 ms) and has not settled by the scaling gate's rule after 500 ms moves to the large-case
+  plan: one call at a time until the median of the last 5 calls is within 5% of the median of the 5
+  before them, at most 30 s. Each timing process records the plan it ended on (`plan` in a reading's
+  `throughput.processes`, `basePlans` and `headPlans` in a comparison), and a reading marks such a
+  case. A case that settles by the standard rule warms up exactly as before.
+
+  Allocations per operation are counted on a case whose every window collects: the largest pair of
+  windows is counted with what its collections reclaimed (`method: "gc-accounted"`, with the number of
+  `collections`), where before the case was skipped as `allocation-window-gc`.
+
+  The peak reports external memory (Buffers, ArrayBuffers) beside the V8 heap, and the two together:
+  `peakExternalBytes`, `peakTotalBytes` and their medians. A memory budget reads `peakTotalBytes`.
+  Memory processes run with `--no-concurrent-array-buffer-sweeping`, so a collection frees dead
+  ArrayBuffers before it ends and the external memory it reports is exact.
+
+  The long run also runs on a benchmark's largest size whose 30 rounds fit in 60 s, besides its
+  long-run size. A reading now times every size of a benchmark before measuring its memory.
+
+  Every new field is added within results version 1 and none is renamed; documents written before
+  still validate.
+
 ## 0.1.1
 
 ### Patch Changes

package/README.md

@@ -131,21 +131,35 @@ the command line:
 
 - the sizes, and which of them a pull request's comparison runs (`compare: false` leaves a slow size
   to the readings);
-- `weigh`, the number a result is reduced to, and `longRun`, the size the long run uses;
+- `weigh`, the number a result is reduced to, and `longRun`, the size the long run uses besides the
+  largest size whose long run fits in 60 s;
 - `--only <benchmark or case>`, `--processes <n>`, `--no-memory` and `--no-import` on `run`.
 
 ## What it measures
 
-| Measure                                | How                                                                                                                                                                              | What it cannot see                                                                                                 |
-| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
-| Throughput, small to very large inputs | Per size, fresh processes; a warmup by the scaling gate's rule; 5 samples of whole passes of at least 50 ms; the median of the processes' medians                                | Anything the suite's sizes do not cover; numbers from another machine                                              |
-| Latency p50, p95, p99                  | One call at a time on a nanosecond clock, up to 2 s or 5,000 calls; a percentile is reported only with at least 10 calls above it                                                | Calls shorter than 20 clock reads (skipped, typed); the tail of a size with few calls                              |
-| Allocations per operation              | Heap growth over windows of `n` and `2n` operations in which no collection ran, in a process with a 64 MiB young generation; the difference cancels the cost of reading the heap | A window that always collects (skipped, typed)                                                                     |
-| Peak heap                              | Per call, the higher of the heap just before any collection during the call and at its end, above the settled heap before it                                                     | Memory outside the V8 heap                                                                                         |
-| Heap flat over a long run              | 30 rounds, the heap settled after each; a Theil-Sen slope over the last 24; growing when the fitted growth reaches 64 KiB                                                        | A leak too small to reach 64 KiB over the run (each result says the smallest it could see, `detectableBytesPerOp`) |
-| Cold import and loaded bytes           | 20 fresh processes import the target, 20 import an empty module, interleaved; one more records every file the import loads, with its gzip size                                   | What a bundler keeps after tree shaking                                                                            |
-
-Memory measurements need `gc()`; the runner starts those processes with `--expose-gc` itself.
+| Measure                                | How                                                                                                                                                                                                                                                                                                                     | What it cannot see                                                                                                                  |
+| -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| Throughput, small to very large inputs | Per size, fresh processes; a warmup by the scaling gate's rule, or by the large-case plan below when every warmup batch is one long pass; 5 samples of whole passes of at least 50 ms; the median of the processes' medians                                                                                             | Anything the suite's sizes do not cover; numbers from another machine                                                               |
+| Latency p50, p95, p99                  | One call at a time on a nanosecond clock, up to 2 s or 5,000 calls; a percentile is reported only with at least 10 calls above it                                                                                                                                                                                       | Calls shorter than 20 clock reads (skipped, typed); the tail of a size with few calls                                               |
+| Allocations per operation              | Heap growth over windows of `n` and `2n` operations in which no collection ran, in a process with a 64 MiB young generation; the difference cancels the cost of reading the heap. When every window collects, the largest pair is counted with what its collections reclaimed (`method: "gc-accounted"`)                | Native memory a library allocates outside V8's accounting                                                                           |
+| Peak memory                            | Per call, the higher of what was in use just before any collection during the call and at its end, above what was in use before it: the V8 heap (`peakHeapBytes`), external memory such as Buffers and ArrayBuffers (`peakExternalBytes`), and the two together (`peakTotalBytes`), which is what a memory budget reads | Native memory a library allocates outside V8's accounting                                                                           |
+| Heap flat over a long run              | 30 rounds, the heap settled after each; a Theil-Sen slope over the last 24; growing when the fitted growth reaches 64 KiB. On the long-run size and on the largest size whose 30 rounds fit in 60 s                                                                                                                     | A leak too small to reach 64 KiB over the run (each result says the smallest it could see, `detectableBytesPerOp`); external memory |
+| Cold import and loaded bytes           | 20 fresh processes import the target, 20 import an empty module, interleaved; one more records every file the import loads, with its gzip size                                                                                                                                                                          | What a bundler keeps after tree shaking                                                                                             |
+
+Memory measurements need `gc()`; the runner starts those processes with `--expose-gc` itself, and
+with `--no-concurrent-array-buffer-sweeping`, so a collection frees the ArrayBuffers it found dead
+before it ends and the external memory it reports before and after is exact.
+
+### Large cases
+
+A case whose single call takes longer than a warmup batch (50 ms) cannot settle by the scaling
+gate's warmup rule on a shared runner: every batch is one call, and one call's jitter is what the
+batch was there to average away. When 500 ms of warmup have passed, the rule has not settled, and
+the last three batches were each one call of at least 50 ms, the warmup moves to the large-case
+plan: one call at a time until the median of the last 5 calls is within 5% of the median of the 5
+before them, at most 30 s from the start. Its samples are the same 5, each one call at that size.
+Each process records the plan it ended on (`plan`), and a reading marks such a case. A case that
+settles by the standard rule never moves, so small cases are measured exactly as before.
 
 ## The comparison
 

package/dist/child-entry.mjs

@@ -29,6 +29,10 @@ var BENCH_CONTRACT = Object.freeze({
   GC_SETTLE_ROUNDS: 2,
   GC_MAX_ROUNDS: 7,
   IMPORT_RUNS: 20,
+  LARGE_PASS_MS: 50,
+  LARGE_WARMUP_WINDOW: 5,
+  LARGE_WARMUP_MAX_MS: 3e4,
+  LONG_RUN_MAX_MS: 6e4,
   REQUIRED_FALSE_ALARM_RATE: 0.01,
   REQUIRED_CONFIDENCE: 0.95
 });
@@ -78,9 +82,27 @@ function realProbe() {
       profiler.start();
       fn();
       const stats = profiler.stop().statistics;
-      let peak = 0;
-      for (const s of stats) peak = Math.max(peak, s.beforeGC.heapStatistics.usedHeapSize);
-      return { gcs: stats.length, peakBeforeGc: peak };
+      const report = {
+        gcs: stats.length,
+        peakBeforeGc: 0,
+        peakExternalBeforeGc: 0,
+        peakTotalBeforeGc: 0,
+        reclaimedHeap: 0,
+        reclaimedExternal: 0
+      };
+      for (const s of stats) {
+        const before = s.beforeGC.heapStatistics;
+        const after = s.afterGC.heapStatistics;
+        report.peakBeforeGc = Math.max(report.peakBeforeGc, before.usedHeapSize);
+        report.peakExternalBeforeGc = Math.max(report.peakExternalBeforeGc, before.externalMemory);
+        report.peakTotalBeforeGc = Math.max(
+          report.peakTotalBeforeGc,
+          before.usedHeapSize + before.externalMemory
+        );
+        report.reclaimedHeap += before.usedHeapSize - after.usedHeapSize;
+        report.reclaimedExternal += before.externalMemory - after.externalMemory;
+      }
+      return report;
     }
   };
 }
@@ -102,43 +124,63 @@ function measureAllocations(call, probe, msPerOp2) {
     const s = settle(probe);
     const h0 = probe.usedHeap();
     const e0 = probe.external();
-    const { gcs } = probe.profile(() => {
+    const p = probe.profile(() => {
       for (let i = 0; i < ops; i++) call(i);
     });
-    return { settled: s.settled, gcs, heap: probe.usedHeap() - h0, ext: probe.external() - e0 };
+    return {
+      settled: s.settled,
+      gcs: p.gcs,
+      heap: probe.usedHeap() - h0 + p.reclaimedHeap,
+      ext: probe.external() - e0 + p.reclaimedExternal
+    };
   };
+  const perOp = (method, n, a2, b2) => ({
+    status: "measured",
+    method,
+    collections: a2.gcs + b2.gcs,
+    windowOps: n,
+    heapBytesPerOp: round((b2.heap - a2.heap) / n, 1),
+    externalBytesPerOp: round((b2.ext - a2.ext) / n, 1)
+  });
   const paced = Math.floor(BENCH_CONTRACT.ALLOC_WINDOW_MS / msPerOp2);
   const first = Math.max(1, Math.min(BENCH_CONTRACT.ALLOC_WINDOW_OPS, paced));
+  let largest = null;
   for (let n = first; n >= 1; n = Math.floor(n / 2)) {
-    const a = window(n);
-    const b = window(2 * n);
-    if (!a.settled || !b.settled) return { status: "skipped", reason: "gc-unsettled" };
-    if (a.gcs === 0 && b.gcs === 0) {
-      return {
-        status: "measured",
-        windowOps: n,
-        heapBytesPerOp: round((b.heap - a.heap) / n, 1),
-        externalBytesPerOp: round((b.ext - a.ext) / n, 1)
-      };
-    }
+    const a2 = window(n);
+    const b2 = window(2 * n);
+    if (!a2.settled || !b2.settled) return { status: "skipped", reason: "gc-unsettled" };
+    if (a2.gcs === 0 && b2.gcs === 0) return perOp("no-gc-window", n, a2, b2);
+    largest ??= [a2, b2];
   }
-  return { status: "skipped", reason: "allocation-window-gc" };
+  const [a, b] = largest;
+  return perOp("gc-accounted", first, a, b);
 }
 function measurePeak(call, probe) {
-  const peaks = [];
+  const heap = [];
+  const external = [];
+  const total = [];
   for (let i = 0; i < BENCH_CONTRACT.PEAK_CALLS; i++) {
     const s = settle(probe);
     if (!s.settled) return { status: "skipped", reason: "gc-unsettled" };
-    const { peakBeforeGc } = probe.profile(() => {
+    const e0 = probe.external();
+    const p = probe.profile(() => {
       call(i);
     });
-    peaks.push(Math.max(peakBeforeGc, probe.usedHeap()) - s.heap);
+    const h1 = probe.usedHeap();
+    const e1 = probe.external();
+    heap.push(Math.max(p.peakBeforeGc, h1) - s.heap);
+    external.push(Math.max(p.peakExternalBeforeGc, e1) - e0);
+    total.push(Math.max(p.peakTotalBeforeGc, h1 + e1) - (s.heap + e0));
   }
   return {
     status: "measured",
-    calls: peaks.length,
-    peakHeapBytes: Math.max(...peaks),
-    medianPeakHeapBytes: median(peaks)
+    calls: heap.length,
+    peakHeapBytes: Math.max(...heap),
+    medianPeakHeapBytes: median(heap),
+    peakExternalBytes: Math.max(...external),
+    medianPeakExternalBytes: median(external),
+    peakTotalBytes: Math.max(...total),
+    medianPeakTotalBytes: median(total)
   };
 }
 function measureLongRun(pass, opsPerPass, probe, clock) {
@@ -208,6 +250,17 @@ function batchesAreStable(batches) {
   const mid = median(window);
   return window.every((b) => Math.abs(b - mid) / mid <= WARMUP.STABLE_TOL);
 }
+function isLargeCase(batches) {
+  if (batches.length < WARMUP.STABLE_BATCHES) return false;
+  return batches.slice(-WARMUP.STABLE_BATCHES).every((b) => b >= BENCH_CONTRACT.LARGE_PASS_MS);
+}
+function windowsAreStable(passes) {
+  const w = BENCH_CONTRACT.LARGE_WARMUP_WINDOW;
+  if (passes.length < 2 * w) return false;
+  const last = median(passes.slice(-w));
+  const before = median(passes.slice(-2 * w, -w));
+  return Math.abs(last - before) / before <= WARMUP.STABLE_TOL;
+}
 function warmUp(pass, clock) {
   const start = clock();
   const batches = [];
@@ -224,10 +277,31 @@ function warmUp(pass, clock) {
     passes += inBatch;
     batches.push(batchMs / inBatch);
     const elapsedMs = clock() - start;
-    if (elapsedMs >= WARMUP.MIN_MS && batchesAreStable(batches)) {
-      return { stable: true, elapsedMs, passes, batches };
+    if (elapsedMs >= WARMUP.MIN_MS) {
+      if (batchesAreStable(batches)) {
+        return { stable: true, elapsedMs, passes, batches, plan: "standard" };
+      }
+      if (isLargeCase(batches)) return warmUpLarge(pass, clock, start, batches, passes);
+    }
+    if (elapsedMs >= WARMUP.MAX_MS) {
+      return { stable: false, elapsedMs, passes, batches, plan: "standard" };
+    }
+  }
+}
+function warmUpLarge(pass, clock, start, batches, passesSoFar) {
+  let passes = passesSoFar;
+  for (; ; ) {
+    const p0 = clock();
+    pass();
+    const t = clock();
+    passes++;
+    batches.push(t - p0);
+    const elapsedMs = t - start;
+    if (windowsAreStable(batches))
+      return { stable: true, elapsedMs, passes, batches, plan: "large" };
+    if (elapsedMs >= BENCH_CONTRACT.LARGE_WARMUP_MAX_MS) {
+      return { stable: false, elapsedMs, passes, batches, plan: "large" };
     }
-    if (elapsedMs >= WARMUP.MAX_MS) return { stable: false, elapsedMs, passes, batches };
   }
 }
 function sampleThroughput(pass, opsPerPass, clock, inject = 0) {