flexi-bench 0.0.0-alpha.3 → 0.1.0

package/CHANGELOG.md

@@ -1,3 +1,42 @@
+## 0.1.0 (2024-07-28)
+
+
+### 🚀 Features
+
+- add different error strategies for handling failing actions ([#11](https://github.com/AgentEnder/flexi-bench/pull/11))
+- **repo:** add ability to publish only docs ([4cde432](https://github.com/AgentEnder/flexi-bench/commit/4cde432))
+
+### ❤️  Thank You
+
+- Craigory Coppola @AgentEnder
+
+# 0.0.0 (2024-07-28)
+
+
+### 🚀 Features
+
+- add different error strategies for handling failing actions ([#11](https://github.com/AgentEnder/flexi-bench/pull/11))
+- **repo:** add ability to publish only docs ([4cde432](https://github.com/AgentEnder/flexi-bench/commit/4cde432))
+
+### ❤️  Thank You
+
+- Craigory Coppola @AgentEnder
+
+## 0.0.0-alpha.4 (2024-07-17)
+
+
+### 🚀 Features
+
+- add suites ([caaa5ed](https://github.com/AgentEnder/flexi-bench/commit/caaa5ed))
+- add syntax to run commands ([#6](https://github.com/AgentEnder/flexi-bench/pull/6))
+- add performance-observer API draft ([#7](https://github.com/AgentEnder/flexi-bench/pull/7))
+- initial draft of runner API ([#8](https://github.com/AgentEnder/flexi-bench/pull/8))
+- docs-site ([#9](https://github.com/AgentEnder/flexi-bench/pull/9))
+
+### ❤️  Thank You
+
+- Craigory Coppola @AgentEnder
+
 ## 0.0.0-alpha.3 (2024-07-11)
 
 This was a version bump only, there were no code changes.

package/README.md

@@ -0,0 +1,373 @@
+# FlexiBench
+
+FlexiBench is a flexible benchmarking library for JavaScript and TypeScript. It is designed to be simple to use, but also powerful and flexible. It is inspired by common testing framework APIs, and aims to provide a similar experience for benchmarking.
+
+## Installation
+
+```bash
+npm install --save-dev flexi-bench
+```
+
+```bash
+yarn add --dev flexi-bench
+```
+
+```bash
+pnpm add --save-dev flexi-bench
+```
+
+## Features
+
+- [Variations](#variations): Run the same benchmark with different configurations.
+- [Setup and Teardown](#setup-and-teardown): Run setup and teardown code before and after the benchmark.
+- [Commands](#commands): Run simple commands as benchmarks.
+
+## Usage
+
+### Runner API
+
+The runner API is the simplest way to run benchmarks. It allows running a single benchmark or a suite of benchmarks.
+
+```javascript
+const { suite, benchmark, setup, teardown } = require('flexi-bench');
+
+// A `suite` call at the top level will be evaluated as a whole suite.
+suite('My Suite', () => {
+  // Nested `benchmark` calls will not be evaluated until the parent `suite` is evaluated.
+  benchmark('My Benchmark', (b) => {
+    setup(() => {
+      // some setup to run before the entire benchmark
+    });
+
+    teardown(() => {
+      // some teardown to run after the entire benchmark
+    });
+
+    b.withAction(() => {
+      // some action to benchmark
+    });
+  });
+});
+
+// Top-level `benchmark` calls will be evaluated immediately.
+benchmark('My Benchmark', (b) => {
+  b.withIterations(10).withAction(() => {
+    // some action to benchmark
+  });
+});
+```
+
+Within the callbacks for each `suite` or `benchmark` you can utilize the builder API for full customization of the benchmark. Variations can either be added directly via the builder API, or by nesting a `variation` call within the `benchmark` call, or at the `suite` level to apply the variation to all benchmarks in the suite.
+
+```javascript
+const { suite, benchmark, variation } = require('flexi-bench');
+
+suite('My Suite', () => {
+  benchmark('My Benchmark', (b) => {
+    b.withIterations(10).withAction(() => {
+      // some action to benchmark
+    });
+
+    variation('with NO_DAEMON', (v) =>
+      v.withEnvironmentVariable('NO_DAEMON', 'true'),
+    );
+  });
+});
+```
+
+### Basic Benchmarks
+
+More detailed documentation will come soon. For now, here is a simple example:
+
+```javascript
+const { Benchmark } = require('flexi-bench');
+
+const benchmark = new Benchmark('My Benchmark', {
+  iterations: 10,
+  action: () => {
+    // some action to benchmark
+  },
+});
+
+await benchmark.run();
+```
+
+Most options for the benchmark can also be provided by a builder API:
+
+```javascript
+const { Benchmark } = require('flexi-bench');
+
+const benchmark = new Benchmark('My Benchmark')
+  .withIterations(10)
+  .withSetup(() => {
+    // some setup to run before the entire benchmark
+  })
+  .withAction(() => {
+    // some action to benchmark
+  });
+```
+
+#### Setup and Teardown
+
+Some benchmarks will require some work before the benchmark to setup the environment, and some work after the benchmark to clean up. This can be done with the `setup`/`setupEach` and `teardown`/`teardownEach` methods:
+
+```javascript
+benchmark('My Benchmark', (b) => {
+  setup(() => {
+    // some setup to run before the entire benchmark
+  });
+
+  setupEach(() => {
+    // some setup that is ran before each iteration of the benchmark
+  });
+
+  teardown(() => {
+    // some teardown to run after the entire benchmark
+  });
+
+  teardownEach(() => {
+    // some teardown that is ran after each iteration of the benchmark
+  });
+});
+```
+
+These can also be set using the builder API:
+
+```javascript
+const { Benchmark } = require('flexi-bench');
+
+const benchmark = new Benchmark('My Benchmark')
+  .withIterations(10)
+  .withSetup(() => {
+    // some setup to run before the entire benchmark
+  })
+  .withSetupEach(() => {
+    // some setup that is ran before each iteration of the benchmark
+  })
+  .withTeardown(() => {
+    // some teardown to run after the entire benchmark
+  })
+  .withTeardownEach(() => {
+    // some teardown that is ran after each iteration of the benchmark
+  });
+```
+
+#### Understanding Actions
+
+The `action` for a benchmark provides the actual event that is being benchmarked. FlexiBench can currently benchmark 2 types of actions:
+
+- Callbacks
+- Commands
+
+An `action` can be specified via the `action` property of the `benchmark` or `variation` constructor, or via `.withAction` on the builder API. Actions specified on the running `variation` will override the action specified on the parent `benchmark`.
+
+For example, the following code will run the `action` specified on the `variation`, and not the `action` specified on the `benchmark`. This would result in the output `bar` being printed 10 times, instead of `foo`:
+
+```javascript
+const { Benchmark } = require('flexi-bench');
+
+const benchmark = new Benchmark('My Benchmark', {
+  iterations: 10,
+  action: () => {
+    console.log('foo');
+  },
+}).withVariation('with NO_DAEMON', (v) =>
+  v.withAction(() => {
+    console.log('bar');
+  }),
+);
+```
+
+##### Callbacks
+
+If the process you are benchmarking is available as a JavaScript function, you can pass it directly to the `action` property or execute it within the `action` callback. For example, if you are benchmarking a function that sorts an array, you can do the following:
+
+```javascript
+const { benchmark, setup } = require('flexi-bench');
+
+benchmark('My Benchmark', (b) => {
+  let array;
+
+  setup(() => {
+    array = Array.from({ length: 1000 }, () => Math.random());
+  });
+
+  b.withIterations(10).withAction(() => {
+    array.sort();
+  });
+});
+```
+
+##### Commands
+
+If the process you are benchmarking is not available as a JavaScript function, you can run it as a command. This can be done by passing a string to the `action` property. For example, if you are benchmarking the runtime of a cli command, you can do the following:
+
+```javascript
+const { Benchmark } = require('flexi-bench');
+
+const benchmark = new Benchmark('My Benchmark', {
+  iterations: 10,
+  action: 'echo "Hello, World!"',
+});
+
+await benchmark.run();
+```
+
+While it would be possible to write an action that runs a command using `child_process` methods, using the syntactic sugar provided by flexi-bench is more convenient and provides a few benefits.
+
+Utilizing the syntactic sugar also means that flexi-bench is aware that you are indeed running a command. This sounds obvious, but opens up some neat possibilities since we are running the command from within flexi-bench. For example, we can add variations based on tailoring CLI options which would not be possible if you ran your command directly using `child_process` methods on your own.
+
+```javascript
+const { Benchmark } = require('flexi-bench');
+
+const benchmark = new Benchmark('My Benchmark', {
+  iterations: 10,
+  action: 'echo',
+})
+  .withVariation('with argument', (v) => v.withArgument('Hello, Earth!'))
+  .withVariation('with argument', (v) => v.withArgument('Hello, Mars!'));
+
+await benchmark.run();
+```
+
+When running commands via the syntactic sugar, the command is invoked with a function similar to below:
+
+```typescript
+const child = spawn(action, variation.cliArgs, {
+  shell: true,
+  windowsHide: true,
+});
+child.on('exit', (code) => {
+  if (code === 0) {
+    resolve();
+  } else {
+    reject(`Action failed with code ${code}`);
+  }
+});
+```
+
+For more information on the simple command API, see the example: ./examples/simple-command.ts
+
+### Suites
+
+A suite is a collection of benchmarks that can be run together:
+
+```javascript
+const { Benchmark, Suite } = require('flexi-bench');
+
+const suite = new Suite('My Suite')
+  .addBenchmark(
+    new Benchmark('My Benchmark 1', {
+      iterations: 10,
+      action: () => {
+        // some action to benchmark
+      },
+    }),
+  )
+  .addBenchmark(
+    new Benchmark('My Benchmark 2', {
+      iterations: 10,
+      action: () => {
+        // some action to benchmark
+      },
+    }),
+  );
+```
+
+Suites can also be created using the runner API:
+
+```javascript
+const { suite, benchmark } = require('flexi-bench');
+
+suite('My Suite', () => {
+  benchmark('My Benchmark 1', (b) => {
+    b.withIterations(10).withAction(() => {
+      // some action to benchmark
+    });
+  });
+
+  benchmark('My Benchmark 2', (b) => {
+    b.withIterations(10).withAction(() => {
+      // some action to benchmark
+    });
+  });
+});
+```
+
+### Variations
+
+Variations allow running the same benchmark with different configurations:
+
+```javascript
+const { Benchmark, Variation } = require('flexi-bench');
+
+const benchmark = new Benchmark('My Benchmark', {
+  iterations: 10,
+  action: () => {
+    // some action to benchmark
+  },
+}).withVariation('with NO_DAEMON', (v) =>
+  v.withEnvironmentVariable('NO_DAEMON', 'true'),
+);
+```
+
+Variations can do most things that the main benchmark can do, including having their own setup and teardown functions, or even a custom action.
+
+Some helper functions are provided on the `Variation` class to make it easier to set up variations:
+
+```javascript
+const { Benchmark, Variation } = require('flexi-bench');
+
+const benchmark = new Benchmark('My Benchmark', {
+  iterations: 10,
+  action: () => {
+    // some action to benchmark
+  },
+}).withVariations(
+  // Adds 4 variations with all possible combinations of the given environment variables
+  Variation.FromEnvironmentVariables([
+    ['NO_DAEMON', ['true', 'false']],
+    ['OTHER_VAR', ['value1', 'value2']],
+  ]),
+);
+```
+
+Variations can also be added to suites. Variations added to a suite will be applied to all benchmarks in the suite.
+
+For example, the below suite would run each benchmark with 'NO_DAEMON' set to true, and then with 'OTHER_VAR' set to 'value1' for a total of 4 benchmark runs in the suite:
+
+```javascript
+const { Benchmark, Suite, Variation } = require('flexi-bench');
+
+const suite = new Suite('My Suite')
+  .addBenchmark(
+    new Benchmark('My Benchmark 1', {
+      iterations: 10,
+      action: () => {
+        // some action to benchmark
+      },
+    }),
+  )
+  .addBenchmark(
+    new Benchmark('My Benchmark 2', {
+      iterations: 10,
+      action: () => {
+        // some action to benchmark
+      },
+    }),
+  )
+  .withVariation('with NO_DAEMON', (v) =>
+    v.withEnvironmentVariable('NO_DAEMON', 'true'),
+  )
+  .withVariation('with OTHER_VAR', (v) =>
+    v.withEnvironmentVariable('OTHER_VAR', 'value1'),
+  );
+```
+
+## Examples
+
+See examples folder.
+
+- ./examples/benchmark.ts is the motivation for this project. It benchmarks the performance of Nx commands with and without a daemon.
+- ./examples/performance-observer.ts is a simple example of how to use the PerformanceObserver API to measure the performance of a function.
+- ./examples/simple-command.ts demonstrates how to benchmark a simple command.

package/dist/api-types.d.ts

@@ -1,17 +1,13 @@
 import { Variation } from './variation';
 import { Benchmark } from './benchmark';
+import { Result } from './results';
 export type MaybePromise<T> = T | Promise<T>;
 export type SetupMethod = (variation: Variation) => MaybePromise<void>;
 export type TeardownMethod = (variation: Variation) => MaybePromise<void>;
 export type ActionMethod = (variation: Variation) => MaybePromise<void>;
+export type ActionCommand = string;
+export type Action = ActionMethod | ActionCommand;
 export type EnvironmentVariableOptions = readonly (readonly [key: string, values: readonly string[]])[] | [key: string, values: string[]][];
-export type Result = {
-    label: string;
-    min: number;
-    max: number;
-    average: number;
-    p95: number;
-};
 export interface ProgressContext {
     totalIterations?: number;
     completedIterations: number;
@@ -25,3 +21,28 @@ export interface BenchmarkReporter {
 export interface SuiteReporter {
     report: (results: Record<string, Result[]>) => void;
 }
+/**
+ * The strategy to use when an error occurs during a benchmark run.
+ */
+export declare enum ErrorStrategy {
+    /**
+     * Continue running the benchmark. Errors will be collected and reported at the end. This is the default behavior.
+     */
+    Continue = "continue",
+    /**
+     * Abort the benchmark run immediately when an error occurs.
+     */
+    Abort = "abort",
+    /**
+     * Delay the error until the end of the benchmark run. This is useful when you want to see all the errors at once
+     */
+    DelayedThrow = "delayed-throw"
+}
+export declare class AggregateBenchmarkError extends Error {
+    results: Result[];
+    constructor(results: Result[]);
+}
+export declare class AggregateSuiteError extends Error {
+    results: Record<string, Result[]>;
+    constructor(results: Record<string, Result[]>);
+}

package/dist/api-types.js

@@ -1,2 +1,37 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
+exports.AggregateSuiteError = exports.AggregateBenchmarkError = exports.ErrorStrategy = void 0;
+/**
+ * The strategy to use when an error occurs during a benchmark run.
+ */
+var ErrorStrategy;
+(function (ErrorStrategy) {
+    /**
+     * Continue running the benchmark. Errors will be collected and reported at the end. This is the default behavior.
+     */
+    ErrorStrategy["Continue"] = "continue";
+    /**
+     * Abort the benchmark run immediately when an error occurs.
+     */
+    ErrorStrategy["Abort"] = "abort";
+    /**
+     * Delay the error until the end of the benchmark run. This is useful when you want to see all the errors at once
+     */
+    ErrorStrategy["DelayedThrow"] = "delayed-throw";
+})(ErrorStrategy || (exports.ErrorStrategy = ErrorStrategy = {}));
+class AggregateBenchmarkError extends Error {
+    results;
+    constructor(results) {
+        super('[AggregateBenchmarkError]: One or more benchmarks failed. Check the results for more information.');
+        this.results = results;
+    }
+}
+exports.AggregateBenchmarkError = AggregateBenchmarkError;
+class AggregateSuiteError extends Error {
+    results;
+    constructor(results) {
+        super('[AggregateSuiteError]: One or more benchmarks failed. Check the results for more information.');
+        this.results = results;
+    }
+}
+exports.AggregateSuiteError = AggregateSuiteError;

package/dist/benchmark-runner.d.ts

@@ -0,0 +1,79 @@
+import { SetupMethod, TeardownMethod } from './api-types';
+import { Benchmark } from './benchmark';
+import { Suite } from './suite';
+import { Variation } from './variation';
+/**
+ * Registers a new suite to run.
+ * @param name The name of the suite.
+ * @param fn Callback to register benchmarks and update the suite.
+ * @returns The results of the suite. `Record<string, Result[]>`
+ */
+export declare function suite(name: string, fn: (suite: Suite) => Suite | void): Promise<Record<string, import("./results").Result[]>>;
+/**
+ * Registers a new benchmark to run. If inside a {@link suite} callback, it will be added to the suite. Otherwise, it will run immediately.
+ * @param name The name of the benchmark.
+ * @param fn Callback to register variations and update the benchmark.
+ * @returns If not inside a suite, the results of the benchmark. `Result[]`. Else, `void`.
+ */
+export declare function benchmark(name: string, fn: (benchmark: Pick<Benchmark, Extract<keyof Benchmark, `with${string}`>>) => Benchmark | void): Promise<import("./results").Result[]> | undefined;
+/**
+ * Registers a new variation to run. Must be inside a {@link benchmark} or {@link suite} callback.
+ * @param name The name of the variation.
+ * @param fn A callback to update the variation.
+ * @returns `void`
+ */
+export declare function variation(name: string, fn: (variation: Variation) => Variation | void): void;
+/**
+ * Registers a setup method to run before the benchmark or variation. Ran once per benchmark or variation.
+ * @param fn The setup method.
+ */
+export declare function setup(fn: SetupMethod): void;
+/**
+ * Registers a setup method to run before each iteration of the benchmark or variation.
+ * @param fn The setup method.
+ */
+export declare function setupEach(fn: SetupMethod): void;
+/**
+ * Registers a teardown method to run after the benchmark or variation. Ran once per benchmark or variation.
+ * @param fn The teardown method.
+ */
+export declare function teardown(fn: TeardownMethod): void;
+/**
+ * Registers a teardown method to run after each iteration of the benchmark or variation.
+ * @param fn The teardown method.
+ */
+export declare function teardownEach(fn: TeardownMethod): void;
+/**
+ * Alias for `setup`.
+ */
+export declare const beforeAll: typeof setup;
+/**
+ * Alias for `setupEach`.
+ */
+export declare const beforeEach: typeof setupEach;
+/**
+ * Alias for `teardown`.
+ */
+export declare const afterAll: typeof teardown;
+/**
+ * Alias for `teardownEach`.
+ */
+export declare const afterEach: typeof teardownEach;
+/**
+ * Alias for `suite`.
+ */
+export declare const describe: typeof suite;
+/**
+ * Alias for `benchmark`.
+ */
+export declare const test: typeof benchmark;
+/**
+ * Alias for `benchmark`.
+ */
+export declare const it: typeof benchmark;
+export declare const xsuite: typeof suite;
+export declare const xdescribe: typeof suite;
+export declare const xbenchmark: typeof benchmark;
+export declare const xtest: typeof benchmark;
+export declare const xit: typeof benchmark;
+export declare const xvariation: typeof variation;