mocha-distributed 0.8.0 → 0.9.2

package/README.md

@@ -1,159 +1,267 @@
 # mocha-distributed
 
+Run mocha tests faster.
+
+Speed up your mocha tests by running them in parallel in multiple machines all
+at once without changing a single line of code. You only need a redis server.
+
+## Purpose
+
 The aim of this project is to provide a simple way of running distributed mocha
-tests without having to change too many lines of code, nor having to decide
-what to run where.
+tests without having to change any line of code, nor having to decide
+what to run where. Tests spread automatically according to the nodes you have.
+
+The concept is very simple, basically you spawn as many runners as you wish
+on as many nodes as you wish, and each node decides whether they should run
+a test or the test has already been executed or is being executed somewhere
+else.
 
 It does not matter if you run the tests in one machine as subprocesses or in
 many machines with multiple processes each.
 
-Hopefully you will only need to include a single line of code on each of your
-mocha files in order for them to run in parallel.
+Because you don't need to change a single line of code, which means that you
+can still run mocha locally as usual.
 
-Your test files will still be 100% compatible with mocha, so you can run them
-without any side-effects locally, using mocha, as if nothing was changed.
+## Quick start
 
-A idea for the future is to create a mocha-compatible runner so that you don't
-even have to change source files in any way. But let's go one step at a time.
+You don't need to change a single line of code on your tests, this project uses
+mocha hooks in order to work, so the only thing you'll need to do in preparation
+is:
 
-## How it works
+  ```bash
+  $ npm install -s mocha-distributed
+  ```
 
-### How it works: Overview
+Make sure you have a redis running somewhere with IP visibility from the machine
+or machines where you want to run the tests on. 
 
-The concept is very simple, this module hooks all mocha calls and does some magic
-to allow running tests across machines without you having to decide what runs
-where, or splitting tests beforehand, etc...
+Finally, on each of the runners just run:
 
-To distribute tests you only need to create several processess across one
-or more machines (system won't care), and set one of them as the master,
-and the rest as the runners. Only one master allowed.
+  ```bash
+    $ export MOCHA_DISTRIBUTED_EXECUTION_ID="execution__2021-01-01__20:10" 
+    $ export MOCHA_DISTRIBUTED="redis://redis.address" 
+    $ mocha --require mocha-distributed test/**/*.js
+  ```
 
-Runners connect to the master and for each suite they ask whether they are
-the 'owners' to run the tests on that suite or not. If they are, they run it.
-If they are not, they just skip the tests and continue running the next suite.
+There are several environment variables that allow you to control the behaviour
+of distributed tests, but this is the simplest way to launch them.
 
-For simplicity this is all granularity you'll get for now. If you need two
-suites to run one after another on the same machine, then create a suite
-that encloses those.
+MOCHA_DISTRIBUTED is the one holding the redis address, this is the only 
+requirement to make mocha-distributed work.
 
-All test results (with error information) are sent to the master, and the master
-will display the output of all tests.
+MOCHA_DISTRIBUTED_EXECUTION_ID is the other variable you want to pay attention
+to. Make sure you use a different value for each group of runners every time
+you launch a test. This variable is what makes possible to make a runner know
+whether a test has already been executed or not by other of their peers.
 
-If you like to save test results, etc... run the master with the right mocha
-parameters. Using those parameters on the runners won't hurt either.
+## Environment Variables
 
-### How it works: In practice
+  - **MOCHA_DISTRIBUTED** (required)
 
-There is a magic environment variable called MOCHA_DISTRIBUTED.
+    Right now this variable is the one used to specify the node that will hold
+    information about tests being run. This project only supports redis right
+    now. This variable can take the form:
+    
+      redis[s]://[[username][:password]@][host][:port]
 
-When unset or empty, this module does nothing at all. Mocha runs normally,
-as if you would have not installed this module.
+    Please make sure it has visibility to the desired redis server.
 
-When set to the special keyword 'master', the mocha will automatically create
-an HTTP server and listen to other processes or machines to connect to it and
-ask/inform about running the tests.
+  - **MOCHA_DISTRIBUTED_EXECUTION_ID** (required)
 
-For the runners, you would need to set MOCHA_DISTRIBUTED variable with the IP
-of the master computer that is running the test. If you are running all the
-tests in multiple processes you can set it to 127.0.0.1, otherwise it should
-be the IP of that machine in your private network, or the public IP if the
-machines are distributed around the world.
+    Make sure this value is different every time you launch your tests. You can
+    use any string here, but it should be different across test executions or
+    your tests will just be skipped after the second execution.
 
-You can also append the port to both the master and the runners, but in
-that case, the port must match.
+    Execution ID is used in order to differentiate different runs of the same
+    tests among parallel executions. If you launch 10 instances and you want
+    tests to be distributed among them, all need to have the same value for this
+    variable, otherwise each of them will run all the tests on its own.
 
-#### Brief example
+    Reusing this variable in different executions will cause your tests to be
+    skipped.
 
-First, add the following line in each of your test files you want to distribute.
+    Use a random uuid or other random value, a kubernetes job_name, your 
+    build system job id, ...
 
-If you try to run the tests without this line, the tests will run on ALL machines,
-including the master node.
+  - **MOCHA_DISTRIBUTED_GRANULARITY** = test
+  
+    - test (default)
+      Potentially all tests can be executed by any runner in any order. This
+      is the default, but if you have trouble running your tests in parallel
+      please use "suite" instead
 
-  ```javascript
-  require('mocha-distributed');
-  ```
+    - suite (safest)
 
-Then, if you want to run mocha in one computer (e.g your dev computer):
+      Launch all tests from the same suite in the same runner. This prevents
+      some parallelization errors if your tests are not prepared for full
+      paralelization.
 
-  ```bash
-  $ mocha test/\*\*/\*.js
-  ```
+  - **MOCHA_DISTRIBUTED_RUNNER_ID** = random-id
 
-To run mocha distributed:
+    By default this value is initialized automatically with a different random
+    string in each machine, BUT you can override this in case you need it for
+    whatever reason, although in theory you probably shouldn't.
 
-  - Execute as master in only one machine (imagine it is running on 1.2.3.4 IP address):
+  - **MOCHA_DISTRIBUTED_EXPIRATION_TIME** = 86400
 
-    ```bash
-    $ MOCHA_DISTRIBUTED="master" mocha test/\*\*/\*.js
-    ```
+    Configures to how long the data is kept in redis before it expires (in 
+    seconds). The amount of data in redis is minimal, so you probably don't want
+    to play with it.
 
-  - Execute runners on one or a thousand machines/processes as:
+    It might be helpful to increase it though, if you want to build some sort of
+    reporting on top of it, because you can directly explore test results in
+    redis. See Tests results in Redis for more info.
 
-    ```bash
-    $ MOCHA_DISTRIBUTED="1.2.3.4" mocha test/\*\*/\*.js
-    ```
 
-## Examples
+  - **MOCHA_DISTRIBUTED_VERBOSE** = false
+    - false (default)
+      Avoid printing verbose information
 
-### Run tests in one machine and one process
+    - true
+      Prints some extra information about the variables, the server, ...
+      that might be useful for debugging issues and/or informational.
 
-Just don't use the MOCHA_DISTRIBUTED variable, or set it to empty string.
+## Reading test results from Redis
 
-  ```bash
-  $ mocha test/\*\*/\*.js
+All runners write the test result in JSON format in a specific redis list.
+
+The list is basically the execution ID from the variable 
+MOCHA_DISTRIBUTED_EXECUTION_ID concatenated to ':test_result'
+
+For example, if you are using: MOCHA_DISTRIBUTED_EXECUTION_ID="abcdefg"
+
+Then the key you should look at in redis will be "abcdefg:test_result"
+
+You can access this list and explore the result of all tests. Each item
+on the list will contain information about the test suite, test id, ...
+test name, if it timed out or not, duration of the test, result of the test,
+if there were any errors, ... all that info is extracted from mocha itself.
+
+You will see something like this on each of the items of the list:
+
+  ```json
+  {
+    "id": [
+      "suite-1-async",
+      "test-1.1-async"
+    ],
+    "type": "test",
+    "title": "test-1.1-async",
+    "timedOut": false,
+    "startTime": 1642705594300,
+    "endTime": 1642705594802,
+    "duration": 502,
+    "file": "/home/psanchez/github/mocha-distributed/example/suite-1.js",
+    "state": "passed",
+    "failed": false,
+    "speed": "slow",
+    "err": 0
+  }
   ```
 
-### Run tests in one machine, multiple processes
+The JSON formatting will differ since it is saved in a single line.
 
-To keep things simple, do something like this:
+Keep in mind that:
+
+* Duration and start/end times are in milliseconds.
+* Some fields are duplicated in a way, like "state" and "failed" by design
+  because sometimes is handy to have this when reading results back.
+* You can access test_result, passed_count and failed_count in redis
+* Skipped tests are never saved in redis by design, unfortunately
+
+You might have a look at list-tests-from-redis.js for an example on how to
+query redis and list all tests.
+
+## Examples
+
+### Environment-agnostic
+
+Make sure at least the following variables are set:
 
   ```bash
-  $ MOCHA_DISTRIBUTED="master" mocha test/\*\*/\*.js
-  $ MOCHA_DISTRIBUTED="localhost" mocha test/\*\*/\*.js > /dev/null &
-  $ MOCHA_DISTRIBUTED="localhost" mocha test/\*\*/\*.js > /dev/null &
-  ...
-  $ MOCHA_DISTRIBUTED="localhost" mocha test/\*\*/\*.js > /dev/null &
+  MOCHA_DISTRIBUTED="redis://1.2.3.4"
+  MOCHA_DISTRIBUTED_EXECUTION_ID="a5ce4d8a-5b06-4ec8-aea2-37d7e4b2ffe1"
   ```
 
-Run as many processes as you'd like
-
-### Run tests in several processes across several machines
+Again, execution ID should be a different random number each time you want to
+launch tests in parallel.
 
-On one machine do:
+Example:
 
   ```bash
-  $ MOCHA_DISTRIBUTED="master" mocha test/\*\*/\*.js
+  $ mocha --require mocha-distributed test/**/*.js
   ```
 
-You can also run some runners in that machine if you wish (see previous example).
+Of course, this assumes you have already installed mocha-distributed.
 
-Figure out the IP address of the master. For this example let's say the master
-IP address is 1.2.3.4. Now on the rest of machines, just do:
+### Run tests in parallel in the same machine
+
+To keep things simple, do something like this:
 
   ```bash
-  $ MOCHA_DISTRIBUTED="1.2.3.4" mocha test/\*\*/\*.js > /dev/null &
-  $ MOCHA_DISTRIBUTED="1.2.3.4" mocha test/\*\*/\*.js > /dev/null &
+  $ MOCHA_DISTRIBUTED_EXECUTION_ID=`uuidgen`
+  $ MOCHA_DISTRIBUTED="redis://redis-server"
+
+  $ mocha --require mocha-distributed test/**/*.js > output01.txt &
+  $ mocha --require mocha-distributed test/**/*.js > output02.txt &
   ...
-  $ MOCHA_DISTRIBUTED="1.2.3.4" mocha test/\*\*/\*.js > /dev/null &
+  $ mocha --require mocha-distributed test/**/*.js > output0N.txt &
   ```
 
-Again, spawn as many processes as you'd like.
+Run as many processes as you'd like.
+
+### Using kubernetes parallel jobs to launch tests
+
+If you plan to use kubernetes to launch parallel jobs, make sure the backoff 
+limit is set to 1, so it does not retry the job after it fails, and make sure
+you set execution ID to a different value each time (but common across all
+parallel executions).
+
+The easiest is to use the job ID (not the pod ID). You can do that by exposing
+pod metadata information as environment variables.
+
+See https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/
+
+### Conceptual overview
+
+The concept is very simple, this module hooks all mocha calls and does some magic
+to allow running tests across machines without you having to decide what runs
+where, or splitting tests beforehand, etc...
+
+To distribute tests you only need to create several processess across one
+or more machines (this method won't care how you spawn your runners), and either
+set one of them as the master or use a redis database, and launch as many runners
+as you wish.
+
+Each runners connects to the redis instance and for each suite or test,
+depending on the granularity, they ask whether they are the 'owners' to run the
+tests on that suite or not. If they are, they run it. If they are not, they just
+skip the tests and continue running the next suite/tests.
+
+### Caveats
+
+When running with redis, all tests are executed by independent runners, which
+means you need to take a look at the output of all the runners and see which
+ones were skipped and which ones were executed for you to see if some of those
+executed failed.
+
+Also the exit code of the different mocha runners will differ. The
+ones whose tests fail, will return an error, and the ones whose tests work well
+or have been skipped will return 0.
 
 ## Build systems
 
 ### jenkins, bamboo, circle-ci, gitlab, travis...
 
-If you use jenkins, bamboo or any other build system, only one runner should
-be defined as the master. The master never runs tests, only waits.
+If you use jenkins, bamboo or any other build system, make sure
+one redis is installed somewhere and all runners can access to it.
 
-You should create more processes or launch more docker or kubernetes instances
-or spread test on several nodes... do it as you wish, but for each of the
-runners that you create, make sure they have visibility to the master (e.g
-make sure you can send a ping from all the runners to the master).
+Create as many processes, nodes, dockers, kubernetes pods as you wish,
+but for each of the runners that you create, make sure each of them can connect 
+to the redis instance (e.g are in the same network).
 
-In case you run multiple masters on the same machine, make sure you setup
-a different port each time, otherwise runner's from different projects will
-inform the wrong master.
+You can use the project name and build ID or job id as the execution ID for
+mocha-distributed. Use something unique among the builds of all your projects.
 
 ## MIT License
 

package/docker-compose.yml

@@ -0,0 +1,27 @@
+version: "3.7"
+
+services:
+
+  redis:
+    image: redis:latest
+    ports:
+      - 6379:6379
+
+    networks:
+      - mordor_gh_network
+
+  redis-commander:
+    image: rediscommander/redis-commander:latest
+    environment:
+      - REDIS_HOSTS=local:redis:6379
+    ports:
+      - 8081:8081
+    depends_on:
+      - redis
+
+    networks:
+      - mordor_gh_network
+
+networks:
+  mordor_gh_network:
+    external: true

package/index.js

@@ -1,49 +1,170 @@
 // -----------------------------------------------------------------------------
-// index.js
+// Copyright (c) 2018 Pau Sanchez
 //
-// Copyright(c) 2019 Pau Sanchez - MIT License
+// MIT Licensed
 // -----------------------------------------------------------------------------
+const redis = require("redis");
+const crypto = require("crypto");
 
-// Three scenarios for MOCHA_DISTRIBUTED:
-//   - master (MOCHA_DISTRIBUTED = 'master[:PORT]')
-//     Runs the server as master node, waiting for connections and waiting
-//     others to run the tests. For simplicity, this node won't execute
-//     any tests, but gather the result of all of them.
+const GRANULARITY = {
+  TEST: "test",
+  SUITE: "suite",
+};
+
+// Initialize variables from environment
+const g_redisAddress = process.env.MOCHA_DISTRIBUTED || "";
+const g_testExecutionId = process.env.MOCHA_DISTRIBUTED_EXECUTION_ID || "";
+const g_expirationTime =
+  process.env.MOCHA_DISTRIBUTED_EXPIRATION_TIME || `${24 * 3600}`;
+
+// Generate a unique random id for this runner (with almost 100% certainty
+// to be different on any machine/environment).
+const _randomRunnerBuf = Buffer.alloc(16);
+const _randomRunnerId = crypto.randomFillSync(_randomRunnerBuf).toString("hex");
+const g_runnerId = process.env.MOCHA_DISTRIBUTED_RUNNER_ID || _randomRunnerId;
+let g_granularity =
+  process.env.MOCHA_DISTRIBUTED_GRANULARITY || GRANULARITY.TEST;
+const g_mochaVerbose = process.env.MOCHA_DISTRIBUTED_VERBOSE === "true";
+
+if (g_granularity !== GRANULARITY.TEST) {
+  g_granularity = GRANULARITY.SUITE;
+}
+
+let g_redis = null;
+
+// -----------------------------------------------------------------------------
+// getTestPath
 //
-//   - runner (MOCHA_DISTRIBUTED='<master-address>[:PORT]'  IP/DNS of the master node)
-//     IP/DNS of the master node (a.k.a runner), this node will get a unique
-//     name from the master, and will ask, before each suite or orphaned test
-//     if it needs to run it or not
+// Returns an array with the test suites and test name from a test context
+// as found in the hooks
 //
-//   - Empty or not defined
-//     Runs mocha normally, we don't redefine any variable at all.
-//     Everything will run locally, if spawned on many machines or processes
-//     tests will run on all of them as if this module did not exist.
-const master = require('./master.js');
-const runner = require('./runner.js');
-
-// Initialize mode & port from environment variable MOCHA_DISTRIBUTED
-const DEFAULT_PORT = 12421;
-let mode = (process.env.MOCHA_DISTRIBUTED || '').toLowerCase();
-let port = DEFAULT_PORT;
-
-if (mode.indexOf (':') >= 0) {
-  const splitted = mode.split(':');
-  mode = splitted[0];
-  port = parseInt (splitted[1], 10);
-}
+// Example:
+//
+//    >>> getTestPath(ctxt)
+//
+// -----------------------------------------------------------------------------
+function getTestPath(testContext) {
+  const path = [testContext.title];
 
-// let's get the party started
-if (!mode) {
-  // run as normal mocha
-}
-else if (mode === 'master') {
-  master (port);
-}
-else {
-  const masterAddress = mode;
-  runner (masterAddress, port);
+  while (!testContext.root && testContext.parent) {
+    testContext = testContext.parent;
+
+    if (testContext && !testContext.root) {
+      path.push(testContext.title);
+    }
+  }
+
+  return path.reverse();
 }
 
-// remove from cache, so it is always reinitialized
-delete require.cache[require.resolve('./index.js')];
+// -----------------------------------------------------------------------------
+// Initialize redis once before the tests
+// -----------------------------------------------------------------------------
+exports.mochaGlobalSetup = async function () {
+  if (g_mochaVerbose) {
+    const redisNoCredentials = g_redisAddress.replace(
+      /\/\/[^@]*@/,
+      "//***:***@"
+    );
+    console.log("---------------------------------------------------");
+    console.log(" Mocha Distributed");
+    console.log("   - Runner Id                :", g_runnerId);
+    console.log("   - Redis Address            :", redisNoCredentials);
+    console.log("   - Execution Id             :", g_testExecutionId);
+    console.log("   - Data Expiration Time     :", g_expirationTime);
+    console.log("   - Test Parallel Granularity:", g_granularity);
+    console.log("---------------------------------------------------");
+  }
+
+  if (!g_redisAddress || !g_testExecutionId) {
+    console.log (g_redisAddress, g_testExecutionId)
+    console.error(
+      "You need to set at least the following environment variables:\n" +
+        "  - MOCHA_DISTRIBUTED\n" +
+        "  - MOCHA_DISTRIBUTED_EXECUTION_ID\n"
+    );
+    process.exit(-1);
+  }
+
+  g_redis = redis.createClient({ url: g_redisAddress });
+  g_redis.on("error", (err) => {
+    console.log("Redis Client Error", err);
+    console.log("Closing application!");
+    process.exit(-1);
+  });
+  await g_redis.connect();
+};
+
+// -----------------------------------------------------------------------------
+// Quit from redis
+// -----------------------------------------------------------------------------
+exports.mochaGlobalTeardown = async function () {
+  if (g_redis) {
+    await g_redis.quit();
+  }
+};
+
+// -----------------------------------------------------------------------------
+// Hook tests
+//
+// Please note that we run skip before each test if the ownership of it has
+// already been defined by another runner.
+// -----------------------------------------------------------------------------
+exports.mochaHooks = {
+  beforeEach: async function () {
+    const testPath = getTestPath(this.currentTest);
+    const testKeyFullPath = `${g_testExecutionId}:${testPath.join(":")}`;
+    const testKeySuite = `${g_testExecutionId}:${testPath[0]}`;
+
+    const testKey =
+      g_granularity === GRANULARITY.TEST ? testKeyFullPath : testKeySuite;
+
+    // Atomically set/get the runner id associated to this test. Only the first
+    // runner to get there will set the value to its own runner id.
+    const [_, assignedRunnerId] = await g_redis
+      .multi()
+      .set(testKey, g_runnerId, { EX: g_expirationTime, NX: true })
+      .get(testKey)
+      .exec();
+
+    if (assignedRunnerId !== g_runnerId) {
+      this.currentTest.title += " (skipped by mocha_distributted)";
+      this.skip();
+    }
+  },
+  afterEach(done) {
+    const SKIPPED = "pending";
+    const FAILED = "failed";
+
+    // Save all data in redis in a way it can be retrieved and aggregated
+    // easily for all test by an external reporter
+    if (this.currentTest.state !== SKIPPED) {
+      const testResult = {
+        id: getTestPath(this.currentTest),
+        type: this.currentTest.type,
+        title: this.currentTest.title,
+        timedOut: this.currentTest.timedOut,
+        duration: this.currentTest.duration,
+        startTime: Date.now() - (this.currentTest.duration || 0),
+        endTime: Date.now(),
+        file: this.currentTest.file,
+        state: this.currentTest.state,
+        failed: this.currentTest.state === FAILED,
+        speed: this.currentTest.speed,
+        err: this.currentTest.err || null,
+      };
+
+      // save results as single line on purpose
+      const key = `${g_testExecutionId}:test_result`;
+      g_redis.rPush(key, JSON.stringify(testResult));
+      g_redis.expire(key, g_expirationTime);
+
+      // increment passed_count/failed_count & set expiry time
+      const countKey = `${g_testExecutionId}:${this.currentTest.state}_count`
+      g_redis.incr(countKey);
+      g_redis.expire(countKey, g_expirationTime);
+    }
+
+    done();
+  },
+};