flexi-bench 0.1.0 → 0.2.0

package/CHANGELOG.md

@@ -1,3 +1,22 @@
+## 0.2.0 (2026-02-02)
+
+
+### 🚀 Features
+
+- add result metadata support (iterations, totalDuration, benchmarkName, variationName) ([22dfc33](https://github.com/agentender/flexi-bench/commit/22dfc33))
+- improve console reporters with metadata display ([bfb4482](https://github.com/agentender/flexi-bench/commit/bfb4482))
+- add composite and JSON suite reporters with examples ([c29f381](https://github.com/agentender/flexi-bench/commit/c29f381))
+- include git info on results when in CI ([3b2a63d](https://github.com/agentender/flexi-bench/commit/3b2a63d))
+- **docs-site:** upload algolia settings ([aea3eff](https://github.com/agentender/flexi-bench/commit/aea3eff))
+
+### 🩹 Fixes
+
+- **e2e:** ensure output directory exists and improve error handling ([c0e79ea](https://github.com/agentender/flexi-bench/commit/c0e79ea))
+
+### ❤️  Thank You
+
+- Craigory Coppola @AgentEnder
+
 ## 0.1.0 (2024-07-28)
 
 

package/README.md

@@ -332,6 +332,50 @@ const benchmark = new Benchmark('My Benchmark', {
 );
 ```
 
+#### Variation Context
+
+For more complex scenarios where you need to pass objects or data directly to the benchmark action, use the context API instead of environment variables:
+
+```javascript
+const { Benchmark, Variation } = require('flexi-bench');
+
+// Define different implementations
+const loopProcessor = {
+  process: (data) => {
+    /* loop */
+  },
+};
+const reduceProcessor = {
+  process: (data) => {
+    /* reduce */
+  },
+};
+
+// Use FromContexts for clean, declarative variation setup
+const benchmark = new Benchmark('Process Data')
+  .withIterations(100)
+  .withVariations(
+    Variation.FromContexts('processor', [
+      ['loop', loopProcessor],
+      ['reduce', reduceProcessor],
+    ]),
+  )
+  .withAction((variation) => {
+    // Use get() to retrieve context data
+    const processor = variation.get('processor');
+    processor.process(data);
+  });
+```
+
+The context API provides:
+
+- `Variation.FromContexts(key, [[name, value], ...])` - Create variations with context (cleanest API)
+- `withContext(key, value)` - Attach custom data to a single variation
+- `get(key)` - Retrieve context data (returns `T | undefined`)
+- `getOrDefault(key, defaultValue)` - Retrieve with fallback value
+
+Use `FromContexts` when creating multiple variations with the same context key - it's cleaner and more concise than individual `withVariation` calls with `withContext`.
+
 Variations can also be added to suites. Variations added to a suite will be applied to all benchmarks in the suite.
 
 For example, the below suite would run each benchmark with 'NO_DAEMON' set to true, and then with 'OTHER_VAR' set to 'value1' for a total of 4 benchmark runs in the suite:
@@ -364,10 +408,239 @@ const suite = new Suite('My Suite')
   );
 ```
 
+## Reporters
+
+Reporters control how benchmark results are output. FlexiBench provides several built-in reporters:
+
+### Console Reporters
+
+```javascript
+const {
+  Benchmark,
+  BenchmarkConsoleReporter,
+  SuiteConsoleReporter,
+} = require('flexi-bench');
+
+// For single benchmarks
+const benchmark = new Benchmark('My Benchmark', {
+  iterations: 10,
+  action: () => {
+    /* ... */
+  },
+  reporter: new BenchmarkConsoleReporter(),
+});
+
+// For suites
+const suite = new Suite('My Suite')
+  .withReporter(new SuiteConsoleReporter())
+  .addBenchmark(benchmark);
+```
+
+Both console reporters support the `NO_COLOR` environment variable for disabling colors:
+
+```bash
+NO_COLOR=1 node my-benchmark.js
+```
+
+Or explicitly via options:
+
+```javascript
+new SuiteConsoleReporter({ noColor: true });
+new BenchmarkConsoleReporter({ noColor: true });
+```
+
+### Markdown Reporters
+
+```javascript
+const {
+  MarkdownBenchmarkReporter,
+  MarkdownSuiteReporter,
+} = require('flexi-bench');
+
+// For single benchmark output
+const benchmarkReporter = new MarkdownBenchmarkReporter({
+  outputFile: 'results.md',
+  fields: ['min', 'average', 'p95', 'max'],
+  append: true, // Set to true to avoid overwriting when running multiple benchmarks
+});
+
+// For suite-level output (recommended)
+const suiteReporter = new MarkdownSuiteReporter({
+  outputFile: 'results.md',
+  title: 'Benchmark Results',
+  fields: ['min', 'average', 'p95', 'max', 'iterations'],
+});
+```
+
+#### Automatic Comparison Tables
+
+When a benchmark has multiple variations, both `MarkdownBenchmarkReporter` and `MarkdownSuiteReporter` automatically generate a comparison table showing:
+
+- Average time for each variation
+- Percentage difference vs the fastest variation
+- Multiplier (e.g., "2.5x" slower)
+- Trophy emoji (🏆) marking the fastest variation
+
+This makes it easy to see which implementation performs best at a glance.
+
+### JSON Reporter
+
+For CI/CD integration:
+
+```javascript
+const { JsonSuiteReporter } = require('flexi-bench');
+
+const reporter = new JsonSuiteReporter({
+  outputFile: 'results.json',
+  pretty: true,
+  includeMetadata: true, // Includes timestamp, platform, node version
+});
+```
+
+### Composite Reporter
+
+Use multiple reporters simultaneously:
+
+```javascript
+const {
+  CompositeReporter,
+  SuiteConsoleReporter,
+  MarkdownSuiteReporter,
+  JsonSuiteReporter,
+} = require('flexi-bench');
+
+const suite = new Suite('My Suite').withReporter(
+  new CompositeReporter([
+    new SuiteConsoleReporter(),
+    new MarkdownSuiteReporter({ outputFile: 'results.md' }),
+    new JsonSuiteReporter({ outputFile: 'results.json' }),
+  ]),
+);
+```
+
+### Custom Reporters
+
+Create custom reporters by implementing the `SuiteReporter` interface:
+
+```typescript
+import { SuiteReporter, Result } from 'flexi-bench';
+
+class MyCustomReporter implements SuiteReporter {
+  // Optional lifecycle hooks
+  onSuiteStart?(suiteName: string): void {
+    console.log(`Starting suite: ${suiteName}`);
+  }
+
+  onBenchmarkStart?(benchmarkName: string): void {
+    console.log(`Running: ${benchmarkName}`);
+  }
+
+  onBenchmarkEnd?(benchmarkName: string, results: Result[]): void {
+    console.log(`Completed: ${benchmarkName}`);
+  }
+
+  // Required: called after all benchmarks complete
+  report(results: Record<string, Result[]>): void {
+    // Process results here
+    for (const [name, result] of Object.entries(results)) {
+      console.log(`${name}: ${result[0].average}ms average`);
+    }
+  }
+}
+```
+
+## Result Type
+
+The `Result` type contains comprehensive information about benchmark runs:
+
+```typescript
+interface Result {
+  // Basic metrics
+  label: string; // Name of benchmark or variation
+  min: number; // Minimum duration (ms)
+  max: number; // Maximum duration (ms)
+  average: number; // Average duration (ms)
+  p95: number; // 95th percentile duration (ms)
+  raw: (number | Error)[]; // Raw durations/errors
+
+  // Failure information
+  failed?: boolean; // Whether any iteration failed
+  failureRate?: number; // Rate of failures (0-1)
+
+  // Metadata (useful for custom reporters)
+  iterations?: number; // Number of iterations run
+  totalDuration?: number; // Total wall-clock time (ms)
+  benchmarkName?: string; // Name of parent benchmark
+  variationName?: string; // Name of variation
+
+  // Subresults from performance observer
+  subresults?: Result[];
+}
+```
+
+The Result type is exported from the main package:
+
+```typescript
+import { Result } from 'flexi-bench';
+```
+
+## Cookbook
+
+### Benchmarking Multiple Implementations
+
+Compare different implementations of the same interface:
+
+```javascript
+const {
+  Suite,
+  Benchmark,
+  Variation,
+  MarkdownSuiteReporter,
+} = require('flexi-bench');
+
+// Define your implementations
+const implementations = {
+  loop: (data) => {
+    /* loop implementation */
+  },
+  reduce: (data) => {
+    /* reduce implementation */
+  },
+};
+
+const suite = new Suite('Implementation Comparison')
+  .withReporter(new MarkdownSuiteReporter({ outputFile: 'results.md' }))
+  .addBenchmark(
+    new Benchmark('Process Data')
+      .withIterations(100)
+      // Create variations with context - no environment variables needed!
+      .withVariations(
+        Variation.FromContexts('impl', [
+          ['loop', implementations.loop],
+          ['reduce', implementations.reduce],
+        ]),
+      )
+      .withAction((variation) => {
+        // Retrieve the implementation directly from context
+        const impl = variation.get('impl');
+        const data = [
+          /* test data */
+        ];
+        impl(data);
+      }),
+  );
+```
+
 ## Examples
 
 See examples folder.
 
-- ./examples/benchmark.ts is the motivation for this project. It benchmarks the performance of Nx commands with and without a daemon.
-- ./examples/performance-observer.ts is a simple example of how to use the PerformanceObserver API to measure the performance of a function.
-- ./examples/simple-command.ts demonstrates how to benchmark a simple command.
+- `./examples/benchmark.ts` - Full benchmark suite with environment variable variations
+- `./examples/performance-observer.ts` - Using PerformanceObserver API
+- `./examples/simple-command.ts` - Benchmarking CLI commands
+- `./examples/multiple-reporters.ts` - Using CompositeReporter for multiple outputs
+- `./examples/custom-reporter.ts` - Creating custom reporters with Result type
+- `./examples/cookbook-different-implementations.ts` - Comparing implementations
+- `./examples/no-color-support.ts` - Disabling colors in CI environments
+- `./examples/markdown-reporter-append.ts` - Using append mode for MarkdownReporter
+- `./examples/markdown-comparison.ts` - Demonstrates automatic comparison tables with variations

package/dist/api-types.d.ts

@@ -19,6 +19,26 @@ export interface BenchmarkReporter {
     report: (benchmark: Benchmark, results: Result[]) => void;
 }
 export interface SuiteReporter {
+    /**
+     * Called before the suite starts running.
+     * @param suiteName - The name of the suite
+     */
+    onSuiteStart?: (suiteName: string) => void;
+    /**
+     * Called before each benchmark starts.
+     * @param benchmarkName - The name of the benchmark
+     */
+    onBenchmarkStart?: (benchmarkName: string) => void;
+    /**
+     * Called after each benchmark completes.
+     * @param benchmarkName - The name of the benchmark
+     * @param results - The results for the benchmark
+     */
+    onBenchmarkEnd?: (benchmarkName: string, results: Result[]) => void;
+    /**
+     * Called after all benchmarks complete.
+     * @param results - All benchmark results keyed by benchmark name
+     */
     report: (results: Record<string, Result[]>) => void;
 }
 /**

package/dist/benchmark.js

@@ -94,6 +94,7 @@ class Benchmark extends shared_api_1.BenchmarkBase {
         let totalCompletedIterations = 0;
         for (let variationIndex = 0; variationIndex < this.variations.length; variationIndex++) {
             const variation = this.variations[variationIndex];
+            const variationStartTime = performance.now();
             const iterationResults = [];
             // SETUP
             const oldEnv = { ...process.env };
@@ -175,13 +176,22 @@ class Benchmark extends shared_api_1.BenchmarkBase {
             }
             process.env = oldEnv;
             // REPORT
-            const result = (0, results_1.calculateResultsFromDurations)(variation.name, iterationResults);
+            const variationEndTime = performance.now();
+            const result = (0, results_1.calculateResultsFromDurations)(variation.name, iterationResults, {
+                iterations: iterationResults.length,
+                totalDuration: variationEndTime - variationStartTime,
+                benchmarkName: this.name,
+                variationName: variation.name,
+            });
             // PerformanceObserver needs a chance to flush
             if (this.watcher) {
                 const measures = await this.watcher.getMeasures();
                 for (const key in measures) {
                     result.subresults ??= [];
-                    result.subresults.push((0, results_1.calculateResultsFromDurations)(key, measures[key]));
+                    result.subresults.push((0, results_1.calculateResultsFromDurations)(key, measures[key], {
+                        benchmarkName: this.name,
+                        variationName: variation.name,
+                    }));
                 }
                 this.watcher.clearMeasures();
             }

package/dist/index.d.ts

@@ -1,10 +1,14 @@
 export * from './reporters/benchmark-console-reporter';
 export * from './reporters/markdown-benchmark-reporter';
+export * from './reporters/markdown-suite-reporter';
 export * from './reporters/suite-console-reporter';
 export * from './reporters/noop-reporter';
+export * from './reporters/composite-reporter';
+export * from './reporters/json-suite-reporter';
 export * from './api-types';
 export * from './benchmark';
 export * from './variation';
 export * from './suite';
 export * from './performance-observer';
 export * from './benchmark-runner';
+export * from './results';

package/dist/index.js

@@ -16,11 +16,15 @@ var __exportStar = (this && this.__exportStar) || function(m, exports) {
 Object.defineProperty(exports, "__esModule", { value: true });
 __exportStar(require("./reporters/benchmark-console-reporter"), exports);
 __exportStar(require("./reporters/markdown-benchmark-reporter"), exports);
+__exportStar(require("./reporters/markdown-suite-reporter"), exports);
 __exportStar(require("./reporters/suite-console-reporter"), exports);
 __exportStar(require("./reporters/noop-reporter"), exports);
+__exportStar(require("./reporters/composite-reporter"), exports);
+__exportStar(require("./reporters/json-suite-reporter"), exports);
 __exportStar(require("./api-types"), exports);
 __exportStar(require("./benchmark"), exports);
 __exportStar(require("./variation"), exports);
 __exportStar(require("./suite"), exports);
 __exportStar(require("./performance-observer"), exports);
 __exportStar(require("./benchmark-runner"), exports);
+__exportStar(require("./results"), exports);

package/dist/reporters/benchmark-console-reporter.d.ts

@@ -3,8 +3,11 @@ import { Benchmark } from '../benchmark';
 import { SingleBar } from 'cli-progress';
 import { Result } from '../results';
 export declare class BenchmarkConsoleReporter implements BenchmarkReporter {
+    private noColor;
     bar: SingleBar;
-    constructor();
+    constructor(opts?: {
+        noColor?: boolean;
+    });
     progress(name: string, percent: number, context: ProgressContext): void;
     report(benchmark: Benchmark, results: Result[]): void;
 }

package/dist/reporters/benchmark-console-reporter.js

@@ -2,16 +2,28 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.BenchmarkConsoleReporter = void 0;
 const cli_progress_1 = require("cli-progress");
+function getNoColorOption(explicitNoColor) {
+    if (explicitNoColor !== undefined) {
+        return explicitNoColor;
+    }
+    return process.env.NO_COLOR !== undefined && process.env.NO_COLOR !== '';
+}
 class BenchmarkConsoleReporter {
-    bar = new cli_progress_1.SingleBar({
-        format: 'Running variation {label}: {bar} {percentage}% | {value}/{total} - ETA: {eta}s',
-        barCompleteChar: '\u2588',
-        barIncompleteChar: '\u2591',
-        hideCursor: true,
-        stopOnComplete: true,
-        clearOnComplete: true,
-    });
-    constructor() { }
+    noColor;
+    bar;
+    constructor(opts) {
+        this.noColor = getNoColorOption(opts?.noColor);
+        this.bar = new cli_progress_1.SingleBar({
+            format: this.noColor
+                ? 'Running variation {label}: [{bar}] {percentage}% | {value}/{total} - ETA: {eta}s'
+                : 'Running variation {label}: {bar} {percentage}% | {value}/{total} - ETA: {eta}s',
+            barCompleteChar: this.noColor ? '#' : '\u2588',
+            barIncompleteChar: this.noColor ? '-' : '\u2591',
+            hideCursor: true,
+            stopOnComplete: true,
+            clearOnComplete: true,
+        });
+    }
     progress(name, percent, context) {
         if (!this.bar.isActive) {
             this.bar.start(context.totalIterations ?? 100, 0);

package/dist/reporters/composite-reporter.d.ts

@@ -0,0 +1,21 @@
+import { SuiteReporter } from '../api-types';
+import { Result } from '../results';
+/**
+ * A reporter that chains multiple suite reporters together.
+ * Useful when you want to output results to multiple destinations
+ * (e.g., console and file) simultaneously.
+ *
+ * @example
+ * ```typescript
+ * suite.withReporter(new CompositeReporter([
+ *   new SuiteConsoleReporter(),
+ *   new MarkdownSuiteReporter({ outputFile: 'results.md' }),
+ *   new JsonSuiteReporter({ outputFile: 'results.json' }),
+ * ]))
+ * ```
+ */
+export declare class CompositeReporter implements SuiteReporter {
+    private reporters;
+    constructor(reporters: SuiteReporter[]);
+    report: (results: Record<string, Result[]>) => void;
+}

package/dist/reporters/composite-reporter.js

@@ -0,0 +1,29 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.CompositeReporter = void 0;
+/**
+ * A reporter that chains multiple suite reporters together.
+ * Useful when you want to output results to multiple destinations
+ * (e.g., console and file) simultaneously.
+ *
+ * @example
+ * ```typescript
+ * suite.withReporter(new CompositeReporter([
+ *   new SuiteConsoleReporter(),
+ *   new MarkdownSuiteReporter({ outputFile: 'results.md' }),
+ *   new JsonSuiteReporter({ outputFile: 'results.json' }),
+ * ]))
+ * ```
+ */
+class CompositeReporter {
+    reporters;
+    constructor(reporters) {
+        this.reporters = reporters;
+    }
+    report = (results) => {
+        for (const reporter of this.reporters) {
+            reporter.report(results);
+        }
+    };
+}
+exports.CompositeReporter = CompositeReporter;

package/dist/reporters/json-suite-reporter.d.ts

@@ -0,0 +1,26 @@
+import { SuiteReporter } from '../api-types';
+import { Result } from '../results';
+export interface JsonSuiteReporterOptions {
+    outputFile: string;
+    pretty?: boolean;
+    includeMetadata?: boolean;
+}
+/**
+ * A reporter that outputs benchmark results as JSON.
+ * Useful for CI/CD integration and programmatic analysis.
+ *
+ * @example
+ * ```typescript
+ * suite.withReporter(new JsonSuiteReporter({
+ *   outputFile: 'results.json',
+ *   pretty: true,
+ * }))
+ * ```
+ */
+export declare class JsonSuiteReporter implements SuiteReporter {
+    private outputFile;
+    private pretty;
+    private includeMetadata;
+    constructor(opts: JsonSuiteReporterOptions);
+    report: (results: Record<string, Result[]>) => void;
+}

package/dist/reporters/json-suite-reporter.js

@@ -0,0 +1,38 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.JsonSuiteReporter = void 0;
+const fs_1 = require("fs");
+/**
+ * A reporter that outputs benchmark results as JSON.
+ * Useful for CI/CD integration and programmatic analysis.
+ *
+ * @example
+ * ```typescript
+ * suite.withReporter(new JsonSuiteReporter({
+ *   outputFile: 'results.json',
+ *   pretty: true,
+ * }))
+ * ```
+ */
+class JsonSuiteReporter {
+    outputFile;
+    pretty;
+    includeMetadata;
+    constructor(opts) {
+        this.outputFile = opts.outputFile;
+        this.pretty = opts.pretty ?? false;
+        this.includeMetadata = opts.includeMetadata ?? true;
+    }
+    report = (results) => {
+        const output = this.includeMetadata
+            ? {
+                timestamp: new Date().toISOString(),
+                platform: process.platform,
+                nodeVersion: process.version,
+                results,
+            }
+            : results;
+        (0, fs_1.writeFileSync)(this.outputFile, JSON.stringify(output, null, this.pretty ? 2 : undefined));
+    };
+}
+exports.JsonSuiteReporter = JsonSuiteReporter;

package/dist/reporters/markdown-benchmark-reporter.d.ts

@@ -3,10 +3,22 @@ import { Benchmark } from '../benchmark';
 import { Result } from '../results';
 export declare class MarkdownBenchmarkReporter implements BenchmarkReporter {
     outputFile: string;
-    fields: Array<keyof Omit<Result, 'subresults' | 'label'>>;
+    fields: Array<keyof Omit<Result, 'subresults' | 'label' | 'raw'>>;
+    append: boolean;
+    private isFirstReport;
+    private accumulatedResults;
     constructor(opts: {
         outputFile: string;
         fields?: MarkdownBenchmarkReporter['fields'];
+        append?: boolean;
     });
+    private formatDuration;
     report: (benchmark: Benchmark, results: Result[]) => void;
+    private generateBenchmarkContent;
+    private generateResultsTable;
+    private generateComparison;
+    /**
+     * Clears the output file. Useful when re-running benchmarks.
+     */
+    clear(): void;
 }

package/dist/reporters/markdown-benchmark-reporter.js

@@ -6,24 +6,160 @@ const markdown_factory_1 = require("markdown-factory");
 class MarkdownBenchmarkReporter {
     outputFile;
     fields;
+    append;
+    isFirstReport = true;
+    accumulatedResults = new Map();
     constructor(opts) {
         this.outputFile = opts.outputFile;
         this.fields = opts.fields ?? ['min', 'average', 'p95', 'max'];
+        this.append = opts.append ?? false;
+    }
+    formatDuration(value) {
+        if (value === undefined)
+            return '';
+        const totalMs = value;
+        const hours = Math.floor(totalMs / (1000 * 60 * 60));
+        const remainingAfterHours = totalMs % (1000 * 60 * 60);
+        const minutes = Math.floor(remainingAfterHours / (1000 * 60));
+        const remainingAfterMinutes = remainingAfterHours % (1000 * 60);
+        const seconds = Math.floor(remainingAfterMinutes / 1000);
+        const milliseconds = remainingAfterMinutes % 1000;
+        const parts = [];
+        if (hours > 0) {
+            parts.push(`${hours}h`);
+            if (minutes > 0)
+                parts.push(`${minutes}m`);
+            if (seconds > 0 || milliseconds > 0) {
+                const totalSeconds = seconds + milliseconds / 1000;
+                parts.push(`${totalSeconds.toFixed(1)}s`);
+            }
+        }
+        else if (minutes > 0) {
+            parts.push(`${minutes}m`);
+            if (seconds > 0 || milliseconds > 0) {
+                const totalSeconds = seconds + milliseconds / 1000;
+                parts.push(`${totalSeconds.toFixed(1)}s`);
+            }
+        }
+        else if (seconds > 0) {
+            const totalSeconds = seconds + milliseconds / 1000;
+            parts.push(`${totalSeconds.toFixed(1)}s`);
+        }
+        else {
+            parts.push(`${milliseconds.toFixed(1)}ms`);
+        }
+        return parts.join(' ');
     }
     report = (benchmark, results) => {
-        (0, fs_1.writeFileSync)(this.outputFile, (0, markdown_factory_1.h1)(benchmark.name, results.some((r) => !!r.subresults)
-            ? (0, markdown_factory_1.lines)(results.map((r) => {
+        if (this.append) {
+            // In append mode, accumulate results and write incrementally
+            this.accumulatedResults.set(benchmark.name, results);
+            const content = this.generateBenchmarkContent(benchmark, results);
+            if (this.isFirstReport) {
+                // First report: clear the file and write header
+                (0, fs_1.writeFileSync)(this.outputFile, content);
+                this.isFirstReport = false;
+            }
+            else {
+                // Subsequent reports: append with a separator
+                (0, fs_1.appendFileSync)(this.outputFile, '\n\n---\n\n' + content);
+            }
+        }
+        else {
+            // Legacy mode: overwrite file (useful for single benchmark reporting)
+            (0, fs_1.writeFileSync)(this.outputFile, this.generateBenchmarkContent(benchmark, results));
+        }
+    };
+    generateBenchmarkContent(benchmark, results) {
+        const sections = [];
+        // Main results section
+        sections.push(this.generateResultsTable(results));
+        // Comparison section (if multiple variations)
+        if (results.length > 1) {
+            sections.push(this.generateComparison(results));
+        }
+        return (0, markdown_factory_1.h1)(benchmark.name, (0, markdown_factory_1.lines)(sections));
+    }
+    generateResultsTable(results) {
+        const fieldConfigs = [
+            { field: 'label', label: '' },
+            ...this.fields.map((field) => ({
+                field: field,
+                label: field,
+                mapFn: (item) => {
+                    const value = item[field];
+                    if (field === 'iterations') {
+                        return String(value ?? '');
+                    }
+                    if (typeof value === 'number') {
+                        return this.formatDuration(value);
+                    }
+                    return String(value ?? '');
+                },
+            })),
+        ];
+        if (results.some((r) => !!r.subresults)) {
+            return (0, markdown_factory_1.lines)(results.map((r) => {
                 const entries = [{ ...r, label: 'total' }];
                 delete entries[0].subresults;
                 for (const subresult of r.subresults ?? []) {
                     entries.push(subresult);
                 }
-                return (0, markdown_factory_1.h2)(r.label, (0, markdown_factory_1.table)(entries, [
-                    { field: 'label', label: '' },
-                    ...this.fields,
-                ]));
-            }))
-            : (0, markdown_factory_1.table)(results, [{ field: 'label', label: '' }, ...this.fields])));
-    };
+                return (0, markdown_factory_1.h2)(r.label, (0, markdown_factory_1.table)(entries, fieldConfigs));
+            }));
+        }
+        return (0, markdown_factory_1.table)(results, fieldConfigs);
+    }
+    generateComparison(results) {
+        const sorted = [...results].sort((a, b) => a.average - b.average);
+        const fastest = sorted[0];
+        const hasVaryingIterations = results.some((r) => r.iterations !== results[0].iterations);
+        const entries = sorted.map((result, index) => {
+            const entry = {
+                label: result.label,
+                indicator: index === 0 ? '🏆' : '',
+            };
+            for (const field of this.fields) {
+                if (field === 'iterations') {
+                    if (hasVaryingIterations) {
+                        entry.iterations = String(result.iterations);
+                    }
+                }
+                else {
+                    const value = result[field];
+                    const fastestValue = fastest[field];
+                    if (index === 0) {
+                        entry[field] = 'baseline';
+                    }
+                    else {
+                        const factor = value / fastestValue;
+                        entry[field] = `${factor.toFixed(2)}x slower`;
+                    }
+                }
+            }
+            return entry;
+        });
+        const columns = [
+            { field: 'label', label: 'Variation' },
+            ...this.fields
+                .filter((field) => field !== 'iterations' || hasVaryingIterations)
+                .map((field) => ({ field, label: field })),
+            { field: 'indicator', label: '' },
+        ];
+        const header = hasVaryingIterations
+            ? 'Comparison'
+            : `Comparison (${results[0].iterations} iterations)`;
+        return (0, markdown_factory_1.h3)(header, (0, markdown_factory_1.table)(entries.map((e) => ({ ...e })), columns));
+    }
+    /**
+     * Clears the output file. Useful when re-running benchmarks.
+     */
+    clear() {
+        if ((0, fs_1.existsSync)(this.outputFile)) {
+            (0, fs_1.unlinkSync)(this.outputFile);
+        }
+        this.isFirstReport = true;
+        this.accumulatedResults.clear();
+    }
 }
 exports.MarkdownBenchmarkReporter = MarkdownBenchmarkReporter;

package/dist/reporters/markdown-suite-reporter.d.ts

@@ -0,0 +1,16 @@
+import { SuiteReporter } from '../api-types';
+import { Result } from '../results';
+export declare class MarkdownSuiteReporter implements SuiteReporter {
+    outputFile: string;
+    title: string;
+    fields: Array<keyof Omit<Result, 'subresults' | 'label' | 'raw'>>;
+    constructor(opts: {
+        outputFile: string;
+        title?: string;
+        fields?: MarkdownSuiteReporter['fields'];
+    });
+    private formatDuration;
+    report: (results: Record<string, Result[]>) => void;
+    private renderBenchmarkSection;
+    private generateComparison;
+}

package/dist/reporters/markdown-suite-reporter.js

@@ -0,0 +1,140 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.MarkdownSuiteReporter = void 0;
+const fs_1 = require("fs");
+const markdown_factory_1 = require("markdown-factory");
+class MarkdownSuiteReporter {
+    outputFile;
+    title;
+    fields;
+    constructor(opts) {
+        this.outputFile = opts.outputFile;
+        this.title = opts.title ?? 'Benchmark Results';
+        this.fields = opts.fields ?? ['min', 'average', 'p95', 'max'];
+    }
+    formatDuration(value) {
+        if (value === undefined)
+            return '';
+        const totalMs = value;
+        const hours = Math.floor(totalMs / (1000 * 60 * 60));
+        const remainingAfterHours = totalMs % (1000 * 60 * 60);
+        const minutes = Math.floor(remainingAfterHours / (1000 * 60));
+        const remainingAfterMinutes = remainingAfterHours % (1000 * 60);
+        const seconds = Math.floor(remainingAfterMinutes / 1000);
+        const milliseconds = remainingAfterMinutes % 1000;
+        const parts = [];
+        if (hours > 0) {
+            parts.push(`${hours}h`);
+            if (minutes > 0)
+                parts.push(`${minutes}m`);
+            if (seconds > 0 || milliseconds > 0) {
+                const totalSeconds = seconds + milliseconds / 1000;
+                parts.push(`${totalSeconds.toFixed(1)}s`);
+            }
+        }
+        else if (minutes > 0) {
+            parts.push(`${minutes}m`);
+            if (seconds > 0 || milliseconds > 0) {
+                const totalSeconds = seconds + milliseconds / 1000;
+                parts.push(`${totalSeconds.toFixed(1)}s`);
+            }
+        }
+        else if (seconds > 0) {
+            const totalSeconds = seconds + milliseconds / 1000;
+            parts.push(`${totalSeconds.toFixed(1)}s`);
+        }
+        else {
+            parts.push(`${milliseconds.toFixed(1)}ms`);
+        }
+        return parts.join(' ');
+    }
+    report = (results) => {
+        const benchmarkSections = Object.entries(results).map(([benchmarkName, benchmarkResults]) => {
+            return this.renderBenchmarkSection(benchmarkName, benchmarkResults);
+        });
+        (0, fs_1.writeFileSync)(this.outputFile, (0, markdown_factory_1.h1)(this.title, (0, markdown_factory_1.lines)(benchmarkSections)));
+    };
+    renderBenchmarkSection(benchmarkName, results) {
+        const hasSubresults = results.some((r) => !!r.subresults);
+        const hasMultipleVariations = results.length > 1;
+        const sections = [];
+        const fieldConfigs = [
+            { field: 'label', label: '' },
+            ...this.fields.map((field) => ({
+                field: field,
+                label: field,
+                mapFn: (item) => {
+                    const value = item[field];
+                    if (field === 'iterations') {
+                        return String(value ?? '');
+                    }
+                    if (typeof value === 'number') {
+                        return this.formatDuration(value);
+                    }
+                    return String(value ?? '');
+                },
+            })),
+        ];
+        // Main results table
+        if (hasSubresults) {
+            sections.push((0, markdown_factory_1.lines)(results.map((r) => {
+                const entries = [{ ...r, label: 'total' }];
+                delete entries[0].subresults;
+                for (const subresult of r.subresults ?? []) {
+                    entries.push(subresult);
+                }
+                return (0, markdown_factory_1.h2)(r.label, (0, markdown_factory_1.table)(entries, fieldConfigs));
+            })));
+        }
+        else {
+            sections.push((0, markdown_factory_1.table)(results, fieldConfigs));
+        }
+        // Comparison section (if multiple variations)
+        if (hasMultipleVariations) {
+            sections.push(this.generateComparison(results));
+        }
+        return (0, markdown_factory_1.h2)(benchmarkName, (0, markdown_factory_1.lines)(sections));
+    }
+    generateComparison(results) {
+        const sorted = [...results].sort((a, b) => a.average - b.average);
+        const fastest = sorted[0];
+        const hasVaryingIterations = results.some((r) => r.iterations !== results[0].iterations);
+        const entries = sorted.map((result, index) => {
+            const entry = {
+                label: result.label,
+                indicator: index === 0 ? '🏆' : '',
+            };
+            for (const field of this.fields) {
+                if (field === 'iterations') {
+                    if (hasVaryingIterations) {
+                        entry.iterations = String(result.iterations);
+                    }
+                }
+                else {
+                    const value = result[field];
+                    const fastestValue = fastest[field];
+                    if (index === 0) {
+                        entry[field] = 'baseline';
+                    }
+                    else {
+                        const factor = value / fastestValue;
+                        entry[field] = `${factor.toFixed(2)}x slower`;
+                    }
+                }
+            }
+            return entry;
+        });
+        const columns = [
+            { field: 'label', label: 'Variation' },
+            ...this.fields
+                .filter((field) => field !== 'iterations' || hasVaryingIterations)
+                .map((field) => ({ field, label: field })),
+            { field: 'indicator', label: '' },
+        ];
+        const header = hasVaryingIterations
+            ? 'Comparison'
+            : `Comparison (${results[0].iterations} iterations)`;
+        return (0, markdown_factory_1.h3)(header, (0, markdown_factory_1.table)(entries.map((e) => ({ ...e })), columns));
+    }
+}
+exports.MarkdownSuiteReporter = MarkdownSuiteReporter;

package/dist/reporters/suite-console-reporter.d.ts

@@ -1,5 +1,9 @@
 import { SuiteReporter } from '../api-types';
 import { Result } from '../results';
 export declare class SuiteConsoleReporter implements SuiteReporter {
+    private noColor;
+    constructor(opts?: {
+        noColor?: boolean;
+    });
     report: (results: Record<string, Result[]>) => void;
 }

package/dist/reporters/suite-console-reporter.js

@@ -1,9 +1,24 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.SuiteConsoleReporter = void 0;
+function getNoColorOption(explicitNoColor) {
+    if (explicitNoColor !== undefined) {
+        return explicitNoColor;
+    }
+    return process.env.NO_COLOR !== undefined && process.env.NO_COLOR !== '';
+}
 class SuiteConsoleReporter {
+    noColor;
+    constructor(opts) {
+        this.noColor = getNoColorOption(opts?.noColor);
+    }
     report = (results) => {
-        console.log('Suite Results:');
+        if (this.noColor) {
+            console.log('Suite Results:');
+        }
+        else {
+            console.log('\x1b[1mSuite Results:\x1b[0m');
+        }
         for (const [name, result] of Object.entries(results)) {
             const tableEntries = result.map(({ raw, ...rest }) => ({
                 ...rest,

package/dist/results.d.ts

@@ -1,3 +1,8 @@
+export type GitInfo = {
+    head: string;
+    sha: string;
+    dirty: boolean;
+};
 /**
  * A measurement result.
  */
@@ -38,5 +43,31 @@ export type Result = {
      * Subresults, if any. Typically sourced from performance observer.
      */
     subresults?: Result[];
+    /**
+     * The number of iterations that were run.
+     */
+    iterations?: number;
+    /**
+     * The total wall-clock duration for all iterations in milliseconds.
+     */
+    totalDuration?: number;
+    /**
+     * The name of the benchmark that produced this result.
+     */
+    benchmarkName?: string;
+    /**
+     * The name of the variation that produced this result.
+     */
+    variationName?: string;
+    /**
+     * Git information about the repository state when this result was produced.
+     */
+    git?: GitInfo;
 };
-export declare function calculateResultsFromDurations(label: string, durations: (number | Error)[]): Result;
+export declare function getGitInfo(): GitInfo | undefined;
+export declare function calculateResultsFromDurations(label: string, durations: (number | Error)[], metadata?: {
+    iterations?: number;
+    totalDuration?: number;
+    benchmarkName?: string;
+    variationName?: string;
+}): Result;

package/dist/results.js

@@ -1,7 +1,50 @@
 "use strict";
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
 Object.defineProperty(exports, "__esModule", { value: true });
+exports.getGitInfo = getGitInfo;
 exports.calculateResultsFromDurations = calculateResultsFromDurations;
-function calculateResultsFromDurations(label, durations) {
+const child_process_1 = require("child_process");
+const is_ci_1 = __importDefault(require("is-ci"));
+let cachedGitInfo = null;
+function getGitInfo() {
+    if (cachedGitInfo !== null) {
+        return cachedGitInfo;
+    }
+    if (!is_ci_1.default) {
+        cachedGitInfo = undefined;
+        return undefined;
+    }
+    const getGitOutput = (args) => {
+        try {
+            const result = (0, child_process_1.spawn)('git', args, {
+                stdio: ['ignore', 'pipe', 'ignore'],
+            });
+            let output = '';
+            result.stdout?.on('data', (data) => {
+                output += data.toString();
+            });
+            result.stdout?.on('end', () => {
+                return output.trim();
+            });
+            return output.trim();
+        }
+        catch {
+            return '';
+        }
+    };
+    const head = getGitOutput(['rev-parse', '--abbrev-ref', 'HEAD']);
+    const sha = getGitOutput(['rev-parse', 'HEAD']);
+    const dirty = getGitOutput(['status', '--porcelain']).length > 0;
+    cachedGitInfo = {
+        head,
+        sha,
+        dirty,
+    };
+    return cachedGitInfo;
+}
+function calculateResultsFromDurations(label, durations, metadata) {
     const errors = [];
     const results = [];
     for (const duration of durations) {
@@ -22,5 +65,7 @@ function calculateResultsFromDurations(label, durations) {
         raw: durations,
         failed: errors.length > 0,
         failureRate: errors.length / durations.length,
+        ...metadata,
+        ...(getGitInfo() ? { git: getGitInfo() } : {}),
     };
 }

package/dist/suite.js

@@ -50,8 +50,10 @@ class Suite {
         return this;
     }
     async run() {
+        this.reporter.onSuiteStart?.(this.name);
         const results = {};
         for (const benchmark of this.benchmarks) {
+            this.reporter.onBenchmarkStart?.(benchmark.name);
             benchmark.withVariations(this.variations);
             if (this.benchmarkReporter) {
                 benchmark.withReporter(this.benchmarkReporter);
@@ -59,7 +61,9 @@ class Suite {
             if (this.shouldSetErrorStrategyOnBenchmarks) {
                 benchmark.withErrorStrategy(this.errorStrategy);
             }
-            results[benchmark.name] = await benchmark.run();
+            const benchmarkResults = await benchmark.run();
+            results[benchmark.name] = benchmarkResults;
+            this.reporter.onBenchmarkEnd?.(benchmark.name, benchmarkResults);
         }
         this.reporter.report(results);
         if (this.errorStrategy === api_types_1.ErrorStrategy.DelayedThrow) {

package/dist/variation.d.ts

@@ -4,7 +4,46 @@ export declare class Variation extends BenchmarkBase {
     name: string;
     environment: Partial<NodeJS.ProcessEnv>;
     cliArgs: string[];
+    private context;
     constructor(name: string);
+    /**
+     * Sets a value in the variation's context.
+     * This allows passing custom data to the action without using environment variables.
+     *
+     * @example
+     * ```typescript
+     * new Variation('fast-impl')
+     *   .withContext('implementation', fastProcessor)
+     *   .withAction((variation) => {
+     *     const impl = variation.get<DataProcessor>('implementation');
+     *     impl.process(data);
+     *   })
+     * ```
+     */
+    withContext<T>(key: string, value: T): this;
+    /**
+     * Gets a value from the variation's context.
+     * Returns undefined if the key is not found.
+     *
+     * @example
+     * ```typescript
+     * b.withAction((variation) => {
+     *   const driver = variation.get<Driver>('driver');
+     *   driver.connect();
+     * });
+     * ```
+     */
+    get<T>(key: string): T | undefined;
+    /**
+     * Gets a value from the variation's context with a default value.
+     * Returns the default if the key is not found.
+     *
+     * @example
+     * ```typescript
+     * const iterations = variation.getOrDefault<number>('iterations', 10);
+     * ```
+     */
+    getOrDefault<T>(key: string, defaultValue: T): T;
     static FromEnvironmentVariables(variables: EnvironmentVariableOptions): Variation[];
     /**
      *
@@ -23,6 +62,29 @@ export declare class Variation extends BenchmarkBase {
      * ]);
      */
     static FromCliArgs(args: Array<string | string[]>): Variation[];
+    /**
+     * Creates variations with context values for a single key.
+     * Useful for swapping implementations or configurations.
+     *
+     * @param key The context key to set
+     * @param values Array of [name, value] tuples
+     * @returns An array of variations with context set
+     *
+     * @example
+     * ```typescript
+     * const variations = Variation.FromContext('processor', [
+     *   ['loop', loopProcessor],
+     *   ['reduce', reduceProcessor],
+     * ]);
+     *
+     * benchmark.withVariations(variations);
+     * benchmark.withAction((variation) => {
+     *   const processor = variation.get<DataProcessor>('processor');
+     *   processor.process(data);
+     * });
+     * ```
+     */
+    static FromContexts<T>(key: string, values: readonly (readonly [name: string, value: T])[]): Variation[];
     withEnvironmentVariables(env: Partial<NodeJS.ProcessEnv>): this;
     withEnvironmentVariable(name: string, value: string): this;
     withCliArgs(...args: string[]): this;

package/dist/variation.js

@@ -7,10 +7,56 @@ class Variation extends shared_api_1.BenchmarkBase {
     name;
     environment = {};
     cliArgs = [];
+    context = new Map();
     constructor(name) {
         super();
         this.name = name;
     }
+    /**
+     * Sets a value in the variation's context.
+     * This allows passing custom data to the action without using environment variables.
+     *
+     * @example
+     * ```typescript
+     * new Variation('fast-impl')
+     *   .withContext('implementation', fastProcessor)
+     *   .withAction((variation) => {
+     *     const impl = variation.get<DataProcessor>('implementation');
+     *     impl.process(data);
+     *   })
+     * ```
+     */
+    withContext(key, value) {
+        this.context.set(key, value);
+        return this;
+    }
+    /**
+     * Gets a value from the variation's context.
+     * Returns undefined if the key is not found.
+     *
+     * @example
+     * ```typescript
+     * b.withAction((variation) => {
+     *   const driver = variation.get<Driver>('driver');
+     *   driver.connect();
+     * });
+     * ```
+     */
+    get(key) {
+        return this.context.get(key);
+    }
+    /**
+     * Gets a value from the variation's context with a default value.
+     * Returns the default if the key is not found.
+     *
+     * @example
+     * ```typescript
+     * const iterations = variation.getOrDefault<number>('iterations', 10);
+     * ```
+     */
+    getOrDefault(key, defaultValue) {
+        return this.context.get(key) ?? defaultValue;
+    }
     static FromEnvironmentVariables(variables) {
         const combinations = (0, utils_1.findCombinations)(variables.map(([name, values]) => values.map((value) => [name, value])));
         variables.reduce((acc, [name, values]) => {
@@ -50,6 +96,33 @@ class Variation extends shared_api_1.BenchmarkBase {
             return new Variation(label).withCliArgs(...cliArgs);
         });
     }
+    /**
+     * Creates variations with context values for a single key.
+     * Useful for swapping implementations or configurations.
+     *
+     * @param key The context key to set
+     * @param values Array of [name, value] tuples
+     * @returns An array of variations with context set
+     *
+     * @example
+     * ```typescript
+     * const variations = Variation.FromContext('processor', [
+     *   ['loop', loopProcessor],
+     *   ['reduce', reduceProcessor],
+     * ]);
+     *
+     * benchmark.withVariations(variations);
+     * benchmark.withAction((variation) => {
+     *   const processor = variation.get<DataProcessor>('processor');
+     *   processor.process(data);
+     * });
+     * ```
+     */
+    static FromContexts(key, values) {
+        return values.map(([name, value]) => {
+            return new Variation(name).withContext(key, value);
+        });
+    }
     withEnvironmentVariables(env) {
         this.environment = {
             ...this.environment,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "flexi-bench",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "",
   "main": "dist/index.js",
   "repository": {
@@ -23,6 +23,7 @@
     "@swc/core": "~1.5.7",
     "@swc/helpers": "~0.5.11",
     "@types/cli-progress": "^3.11.6",
+    "@types/is-ci": "^3.0.4",
     "@types/node": "^20.14.10",
     "nx": "19.4.2",
     "patch-package": "^8.0.0",
@@ -34,6 +35,7 @@
   },
   "dependencies": {
     "cli-progress": "^3.12.0",
+    "is-ci": "^4.1.0",
     "markdown-factory": "^0.0.6"
   },
   "nx": {},