benchforge 0.1.1 → 0.1.3

package/README.md

@@ -151,18 +151,17 @@ The `--profile` flag executes exactly one iteration with no warmup, making it id
 Results are displayed in a formatted table:
 
 ```
-╔═════════════════╤═══════════════════════════════════════════╤═══════╤═════════╗
-║                 │                   time                    │       │         ║
-║ name            │ mean  Δ% CI                    p50   p99  │ conv% │ runs    ║
-╟─────────────────┼───────────────────────────────────────────┼───────┼─────────╢
-║ quicksort       │ 0.17  +5.5% [+4.7%, +6.2%]     0.15  0.63 │ 100%  │ 1,134   ║
-║ insertion sort  │ 0.24  +25.9% [+25.3%, +27.4%]  0.18  0.36 │ 100%  │ 807     ║
-║ --> native sort │ 0.16                           0.15  0.41 │ 100%  │ 1,210   ║
-╚═════════════════╧═══════════════════════════════════════════╧═══════╧═════════╝
+╔═════════════════╤═══════════════════════════════════════════╤═════════╗
+║                 │                   time                    │         ║
+║ name            │ mean  Δ% CI                    p50   p99  │ runs    ║
+╟─────────────────┼───────────────────────────────────────────┼─────────╢
+║ quicksort       │ 0.17  +5.5% [+4.7%, +6.2%]     0.15  0.63 │ 1,134   ║
+║ insertion sort  │ 0.24  +25.9% [+25.3%, +27.4%]  0.18  0.36 │ 807     ║
+║ --> native sort │ 0.16                           0.15  0.41 │ 1,210   ║
+╚═════════════════╧═══════════════════════════════════════════╧═════════╝
 ```
 
 - **Δ% CI**: Percentage difference from baseline with bootstrap confidence interval
-- **conv%**: Convergence percentage (100% = stable measurements)
 
 ### HTML
 
@@ -284,136 +283,23 @@ V8's sampling profiler uses Poisson-distributed sampling. When an allocation occ
 - Node.js 22.6+ (for native TypeScript support)
 - Use `--expose-gc --allow-natives-syntax` flags for garbage collection monitoring and V8 native functions
 
-## Adaptive Mode
+## Adaptive Mode (Experimental)
 
-Adaptive mode automatically adjusts the number of benchmark iterations until measurements stabilize, providing statistically significant results without excessive runtime.
+Adaptive mode (`--adaptive`) automatically adjusts iteration count until measurements stabilize. The algorithm is still being tuned — use `--help` for available options.
 
-### Using Adaptive Mode
+## Interpreting Results
 
-```bash
-# Enable adaptive benchmarking with default settings
-simple-cli.ts --adaptive
-
-# Customize time limits
-simple-cli.ts --adaptive --time 60 --min-time 5
-
-# Combine with other options
-simple-cli.ts --adaptive --filter "quicksort"
-```
-
-### CLI Options for Adaptive Mode
-
-- `--adaptive` - Enable adaptive sampling mode
-- `--min-time <seconds>` - Minimum time before convergence can stop (default: 1s)
-- `--convergence <percent>` - Confidence threshold 0-100 (default: 95)
-- `--time <seconds>` - Maximum time limit (default: 20s in adaptive mode)
-
-### How It Works
-
-1. **Initial Sampling**: Collects initial batch of ~100 samples (includes warmup)
-2. **Window Comparison**: Compares recent samples against previous window
-3. **Stability Detection**: Checks median drift and outlier impact between windows
-4. **Convergence**: Stops when both metrics are stable (<5% drift) or reaches threshold
-
-Progress is shown during execution:
-```
-◊ quicksort: 75% confident (2.1s)
-```
-
-### Output with Adaptive Mode
-
-```
-╔═════════════════╤═════════════════════════════════════════════╤═══════╤═════════╤══════╗
-║                 │                    time                     │       │         │      ║
-║ name            │ median  Δ% CI                    mean  p99  │ conv% │ runs    │ time ║
-╟─────────────────┼─────────────────────────────────────────────┼───────┼─────────┼──────╢
-║ quicksort       │ 0.17    +17.3% [+15.4%, +20.0%]  0.20  0.65 │ 100%  │ 526     │ 0.0s ║
-║ insertion sort  │ 0.18    +24.2% [+23.9%, +24.6%]  0.19  0.36 │ 100%  │ 529     │ 0.0s ║
-║ --> native sort │ 0.15                             0.15  0.25 │ 100%  │ 647     │ 0.0s ║
-╚═════════════════╧═════════════════════════════════════════════╧═══════╧═════════╧══════╝
-```
-
-- **conv%**: Convergence percentage (100% = stable measurements)
-- **time**: Total sampling duration for that benchmark
-
-## Statistical Considerations: Mean vs Median
-
-### When to Use Mean with Confidence Intervals
-
-**Best for:**
-- **Normally distributed data** - When benchmark times follow a bell curve
-- **Statistical comparison** - Comparing performance between implementations
-- **Throughput analysis** - Understanding average system performance
-- **Resource planning** - Estimating typical resource usage
-
-**Advantages:**
-- Provides confidence intervals for statistical significance
-- Captures the full distribution including outliers
-- Better for detecting small but consistent performance differences
-- Standard in academic performance research
-
-**Example use cases:**
-- Comparing algorithm implementations
-- Measuring API response times under normal load
-- Evaluating compiler optimizations
-- Benchmarking pure computational functions
-
-### When to Use Median (p50)
-
-**Best for:**
-- **Skewed distributions** - When outliers are common
-- **Latency-sensitive applications** - Where typical user experience matters
-- **Noisy environments** - Systems with unpredictable interference
-- **Service Level Agreements** - "50% of requests complete within X ms"
-
-**Advantages:**
-- Robust to outliers and system noise
-- Better represents "typical" performance
-- More stable in virtualized/cloud environments
-- Less affected by GC pauses and OS scheduling
-
-**Example use cases:**
-- Web server response times
-- Database query performance
-- UI responsiveness metrics
-- Real-time system benchmarks
-
-### Interpreting Results
-
-#### Baseline Comparison (Δ% CI)
+### Baseline Comparison (Δ% CI)
 ```
 0.17  +5.5% [+4.7%, +6.2%]
 ```
-This shows the benchmark is 5.5% slower than baseline, with a bootstrap confidence interval of [+4.7%, +6.2%]. Use this for comparing implementations.
+The benchmark is 5.5% slower than baseline, with a bootstrap confidence interval of [+4.7%, +6.2%].
 
-#### Percentiles
+### Percentiles
 ```
 p50: 0.15ms, p99: 0.27ms
 ```
-This shows that 50% of runs completed in ≤0.15ms and 99% in ≤0.27ms. Use this when you care about consistency and tail latencies.
-
-### Practical Guidelines
-
-1. **Use adaptive mode when:**
-   - You want automatic convergence detection
-   - Benchmarks have varying execution times
-   - You need stable measurements without guessing iteration counts
-
-2. **Use fixed iterations when:**
-   - Comparing across runs/machines (reproducibility)
-   - You know roughly how many samples you need
-   - Running in CI pipelines with time constraints
-
-3. **Interpreting conv%:**
-   - 100% = measurements are stable
-   - <100% = still converging or high variance
-   - Red color indicates low confidence
-
-### Statistical Notes
-
-- **Bootstrap CI**: Baseline comparison uses permutation testing with bootstrap confidence intervals
-- **Window Stability**: Adaptive mode compares sliding windows for median drift and outlier impact
-- **Independence**: Assumes benchmark iterations are independent (use `--worker` flag for better isolation)
+50% of runs completed in ≤0.15ms and 99% in ≤0.27ms. Use percentiles when you care about consistency and tail latencies.
 
 ## Understanding GC Time Measurements
 

package/dist/BenchRunner-BLfGX2wQ.d.mts

@@ -0,0 +1,225 @@
+//#region src/heap-sample/HeapSampler.d.ts
+interface ProfileNode {
+  callFrame: {
+    functionName: string;
+    url: string;
+    lineNumber: number;
+    columnNumber?: number;
+  };
+  selfSize: number;
+  children?: ProfileNode[];
+}
+interface HeapProfile {
+  head: ProfileNode;
+  samples?: number[];
+}
+//#endregion
+//#region src/NodeGC.d.ts
+/** Individual GC event for visualization */
+interface GcEvent {
+  /** Offset from collection start (ms) - can be negative for warmup GCs */
+  offset: number;
+  /** Duration of GC pause (ms) */
+  duration: number;
+}
+/** GC time measured by Node's performance hooks */
+interface NodeGCTime {
+  inRun: number;
+  before: number;
+  after: number;
+  total: number;
+  collects: number;
+  /** Individual GC events during sample collection (for visualization) */
+  events: GcEvent[];
+}
+//#endregion
+//#region src/runners/GcStats.d.ts
+/** GC statistics aggregated from V8 trace events.
+*  Node (--trace-gc-nvp) provides all fields.
+*  Browser (CDP Tracing) provides counts, collected, and pause only. */
+interface GcStats {
+  scavenges: number;
+  markCompacts: number;
+  totalCollected: number;
+  gcPauseTime: number;
+  totalAllocated?: number;
+  totalPromoted?: number;
+  totalSurvived?: number;
+}
+//#endregion
+//#region src/MeasuredResults.d.ts
+/** CPU performance counter stats */
+interface CpuCounts {
+  instructions?: number;
+  cycles?: number;
+  branchMisses?: number;
+}
+/** Benchmark results: times in milliseconds, sizes in kilobytes */
+interface MeasuredResults {
+  name: string;
+  /** Raw execution time samples for custom statistics */
+  samples: number[];
+  /** Warmup iteration timings (ms) - captured before gc/settle */
+  warmupSamples?: number[];
+  /** Raw allocation samples per iteration (KB) */
+  allocationSamples?: number[];
+  /** Heap size per sample (bytes) - used for charts */
+  heapSamples?: number[];
+  /** Wall-clock timestamps per sample (μs since process start) - for Perfetto export */
+  timestamps?: number[];
+  /** Execution time in milliseconds (measurement overhead excluded by mitata) */
+  time: {
+    min: number;
+    max: number;
+    avg: number;
+    p25?: number;
+    p50: number;
+    p75: number;
+    p95?: number;
+    p99: number;
+    p999: number;
+    cv?: number;
+    mad?: number;
+    outlierRate?: number;
+  };
+  /** Heap size increase during test run (kilobytes) */
+  heapSize?: {
+    avg: number;
+    min: number;
+    max: number;
+  };
+  /**
+  * Time for explicit gc() call after test execution (milliseconds).
+  * Does not include GC time during test execution.
+  * Only reported by mitata runner.
+  */
+  gcTime?: {
+    avg: number;
+    min: number;
+    max: number;
+  };
+  /** CPU counter stats from @mitata/counters (requires root access) */
+  cpu?: CpuCounts;
+  /** L1 cache miss rate */
+  cpuCacheMiss?: number;
+  /** CPU stall rate (macOS only) */
+  cpuStall?: number;
+  /**
+  * Stop-the-world GC time blocking main thread (milliseconds).
+  * Measured via Node's performance hooks when nodeObserveGC is true.
+  * Excludes parallel thread collection time and indirect slowdowns.
+  */
+  nodeGcTime?: NodeGCTime;
+  /** Total time spent collecting samples (seconds) */
+  totalTime?: number;
+  /** Convergence information for adaptive mode */
+  convergence?: {
+    converged: boolean;
+    confidence: number;
+    reason: string;
+  };
+  /** V8 optimization tier tracking (requires --allow-natives-syntax) */
+  optStatus?: OptStatusInfo;
+  /** Per-sample V8 optimization status codes (for chart visualization) */
+  optSamples?: number[];
+  /** Points where pauses occurred for V8 optimization */
+  pausePoints?: PausePoint[];
+  /** GC stats from V8's --trace-gc-nvp (requires --gc-stats and worker mode) */
+  gcStats?: GcStats;
+  /** Heap sampling allocation profile (requires --heap-sample and worker mode) */
+  heapProfile?: HeapProfile;
+}
+/** A pause point during sample collection for V8 optimization */
+interface PausePoint {
+  /** Sample index where pause occurred (after this iteration) */
+  sampleIndex: number;
+  /** Pause duration in milliseconds */
+  durationMs: number;
+}
+/** V8 optimization tier distribution */
+interface OptTierInfo {
+  count: number;
+  medianMs: number;
+}
+/** V8 optimization status summary */
+interface OptStatusInfo {
+  /** Samples by tier name (e.g., "turbofan", "sparkplug") */
+  byTier: Record<string, OptTierInfo>;
+  /** Number of samples with deopt flag set */
+  deoptCount: number;
+}
+//#endregion
+//#region src/Benchmark.d.ts
+/** Single benchmark function specification */
+interface BenchmarkSpec<T = unknown> {
+  name: string;
+  fn: BenchmarkFunction<T>;
+  /** Path to module exporting the benchmark function (for worker mode) */
+  modulePath?: string;
+  /** Name of the exported function in the module (defaults to default export) */
+  exportName?: string;
+  /** Setup function export name - called once in worker, result passed to fn */
+  setupExportName?: string;
+}
+type BenchmarkFunction<T = unknown> = ((params: T) => void) | (() => void);
+/** Group of benchmarks with shared setup */
+interface BenchGroup<T = unknown> {
+  name: string;
+  /** Prepare parameters for all benchmarks in this group */
+  setup?: () => T | Promise<T>;
+  benchmarks: BenchmarkSpec<T>[];
+  /** Baseline benchmark for comparison */
+  baseline?: BenchmarkSpec<T>;
+  /** Metadata for reporting (e.g., lines of code) */
+  metadata?: Record<string, any>;
+}
+/** Collection of benchmark groups */
+interface BenchSuite {
+  name: string;
+  groups: BenchGroup<any>[];
+}
+//#endregion
+//#region src/runners/BenchRunner.d.ts
+interface RunnerOptions {
+  /** Minimum time to run each benchmark (milliseconds) */
+  minTime?: number;
+  /** Maximum time to run each benchmark - ignored by mitata (milliseconds) */
+  maxTime?: number;
+  /** Maximum iterations per benchmark - ignored by TinyBench */
+  maxIterations?: number;
+  /** Warmup iterations before measurement (default: 0) */
+  warmup?: number;
+  /** Warmup time before measurement (milliseconds) */
+  warmupTime?: number;
+  /** Warmup samples - mitata only, for reducing test time */
+  warmupSamples?: number;
+  /** Warmup threshold - mitata only (nanoseconds) */
+  warmupThreshold?: number;
+  /** Minimum samples required - mitata only */
+  minSamples?: number;
+  /** Force GC after each iteration (requires --expose-gc) */
+  collect?: boolean;
+  /** Enable CPU performance counters (requires root access) */
+  cpuCounters?: boolean;
+  /** Trace V8 optimization tiers (requires --allow-natives-syntax) */
+  traceOpt?: boolean;
+  /** Skip post-warmup settle time (default: false) */
+  noSettle?: boolean;
+  /** Iterations before first pause (then pauseInterval applies) */
+  pauseFirst?: number;
+  /** Iterations between pauses for V8 optimization (0 to disable) */
+  pauseInterval?: number;
+  /** Pause duration in ms for V8 optimization */
+  pauseDuration?: number;
+  /** Collect GC stats via --trace-gc-nvp (requires worker mode) */
+  gcStats?: boolean;
+  /** Heap sampling allocation attribution */
+  heapSample?: boolean;
+  /** Heap sampling interval in bytes */
+  heapInterval?: number;
+  /** Heap sampling stack depth */
+  heapDepth?: number;
+}
+//#endregion
+export { MeasuredResults as a, BenchmarkSpec as i, BenchGroup as n, HeapProfile as o, BenchSuite as r, RunnerOptions as t };
+//# sourceMappingURL=BenchRunner-BLfGX2wQ.d.mts.map