mocha-distributed 0.9.4 → 0.9.6

package/README.md

@@ -1,6 +1,6 @@
 # mocha-distributed
 
-Run mocha tests faster.
+Run mocha tests in parallel.
 
 Speed up your mocha tests by running them in parallel in multiple machines all
 at once without changing a single line of code. You only need a redis server.
@@ -12,15 +12,16 @@ tests without having to change any line of code, nor having to decide
 what to run where. Tests spread automatically according to the nodes you have.
 
 The concept is very simple, basically you spawn as many runners as you wish
-on as many nodes as you wish, and each node decides whether they should run
+on as many nodes as you wish, and each process decides whether they should run
 a test or the test has already been executed or is being executed somewhere
 else.
 
-It does not matter if you run the tests in one machine as subprocesses or in
-many machines with multiple processes each.
+It does not matter if you run the tests in one machine in multiple processes or
+in multiple machines with multiple processes each. It will just work.
 
-Because you don't need to change a single line of code, which means that you
-can still run mocha locally as usual.
+You don't need to change a single line of code, thus, this library it allows you
+to continue developing tests as usual and launch them in parallel whenever you
+want. No strings attached.
 
 ## Quick start
 
@@ -38,7 +39,7 @@ or machines where you want to run the tests on.
 Finally, on each of the runners just run:
 
   ```bash
-    $ export MOCHA_DISTRIBUTED_EXECUTION_ID="execution__2021-01-01__20:10" 
+    $ export MOCHA_DISTRIBUTED_EXECUTION_ID="execution__2024-01-01__20:10"
     $ export MOCHA_DISTRIBUTED="redis://redis.address" 
     $ mocha --require mocha-distributed test/**/*.js
   ```
@@ -102,11 +103,11 @@ whether a test has already been executed or not by other of their peers.
     string in each machine, BUT you can override this in case you need it for
     whatever reason, although in theory you probably shouldn't.
 
-  - **MOCHA_DISTRIBUTED_EXPIRATION_TIME** = 86400
+  - **MOCHA_DISTRIBUTED_EXPIRATION_TIME** = 604800
 
     Configures to how long the data is kept in redis before it expires (in 
-    seconds). The amount of data in redis is minimal, so you probably don't want
-    to play with it.
+    seconds). 7 days is the default. The amount of data in redis is minimal,
+    so you probably don't want to play with it.
 
     It might be helpful to increase it though, if you want to build some sort of
     reporting on top of it, because you can directly explore test results in
@@ -172,6 +173,65 @@ Keep in mind that:
 You might have a look at list-tests-from-redis.js for an example on how to
 query redis and list all tests.
 
+### Mark tests to run serially
+
+If you'd like some of your tests to run serially you can use a magic string with
+this framework.
+
+Simply add "[serial]" or "[serial-<ID OF YOUR CHOICE>]" to the title of your
+test or test suite and all those tests will execute serially by the same runner.
+
+The important part is that the test title contains "[serial" and ends with "]"
+
+It's easier to explain with a couple of examples:
+
+The following tests, regardless of whether they are on the same file or spreaded
+in multiple files, will be executed all by the same runner one after another.
+
+Might run in parallel to other tests that don't contain the "[serial]" word,
+but will run sequentially for this group.
+
+```javascript
+it('Test id 1 [serial]', function() { /* ... */})
+it('Test id 2 [serial]', function() { /* ... */})
+it('Test id 3 [serial]', function() { /* ... */})
+it('Test id 4 [serial]', function() { /* ... */})
+```
+
+See this other example below. Again, regardless of whether the tests are on the
+same file or spreaded in multiple files, will be executed by two sets of
+runners.
+
+```javascript
+it('Test id 1 [serial-worker]', function() { /* ... */})
+it('Test id 2 [serial-worker]', function() { /* ... */})
+it('Test id 3 [serial-another worker]', function() { /* ... */})
+it('Test id 4 [serial-another worker]', function() { /* ... */})
+```
+
+Test 1 and 2 will be executed by one runner, whereas test 3 and 4 will be
+executed by another. In both cases 1 and 2 will be executed sequentially and 3
+and 4 also sequentially, but since they have different serial IDs, those two
+subgroups of tests can run in parallel (e.g 1 and 2 in parallel with 3 and 4).
+
+And now last example below:
+
+```javascript
+describe('[serial-my test id] test multiple things sequentially', function () {
+  it('Test id 1', function() { /* ... */})
+  it('Test id 2', function() { /* ... */})
+  it('Test id 3', function() { /* ... */})
+  it('Test id 4', function() { /* ... */})
+})
+```
+
+The suite contains "[serial-my test id]", but the tests don't contain any serial
+magic id. In this case, ALL those tests will run sequentially because the suite
+contains the magic word.
+
+Long story short. Add "[serial-whatever you want]" on the title but make sure
+that "whatever you want" is the same for the stuff you want to run sequentially.
+
 ## Examples
 
 ### Environment-agnostic

package/example/suite-5-serial.js

@@ -0,0 +1,34 @@
+const util = require('./util.js');
+
+// require('mocha-distributed')
+require('../index.js');
+
+describe ('suite-5.serial', async function () {
+  beforeEach (function () {
+    // console.log ('before each');
+  });
+
+  it ('[serial] test-5.0', async function () {
+    await util.serialTestConcurrency('serial5')
+  });
+
+  it ('[serial:odd-worker] test-5.1', async function () {
+    await util.serialTestConcurrency('odd-worker')
+  });
+
+  it ('[serial:even-worker] test-5.2', async function () {
+    await util.serialTestConcurrency('even-worker')
+  });
+
+  it ('[serial:odd-worker] test-5.3', async function () {
+    await util.serialTestConcurrency('odd-worker')
+  });
+
+  it ('[serial:even-worker] test-5.4', async function () {
+    await util.serialTestConcurrency('even-worker')
+  });
+
+  it ('[serial] test-5.5', async function () {
+    await util.serialTestConcurrency('serial5')
+  });
+});

package/example/suite-6-serial.js

@@ -0,0 +1,35 @@
+const util = require('./util.js');
+
+// require('mocha-distributed')
+require('../index.js');
+
+// NOTE: since "describe" has [serial] on it, all will be executed with that ID
+describe ('[serial] suite-6.serial', async function () {
+  beforeEach (function () {
+    // console.log ('before each');
+  });
+
+  it ('[serial] test-6.0', async function () {
+    await util.serialTestConcurrency('serial6')
+  });
+
+  it ('[serial:odd-worker] test-6.1', async function () {
+    await util.serialTestConcurrency('serial6')
+  });
+
+  it ('[serial:even-worker] test-6.2', async function () {
+    await util.serialTestConcurrency('serial6')
+  });
+
+  it ('[serial:odd-worker] test-6.3', async function () {
+    await util.serialTestConcurrency('serial6')
+  });
+
+  it ('[serial:even-worker] test-6.4', async function () {
+    await util.serialTestConcurrency('serial6')
+  });
+
+  it ('[serial] test-6.5', async function () {
+    await util.serialTestConcurrency('serial6')
+  });
+});

package/example/util.js

@@ -1,3 +1,4 @@
+const fs = require('fs');
 const util = require('./util.js');
 
 async function sleep (seconds) {
@@ -6,6 +7,22 @@ async function sleep (seconds) {
   });
 }
 
+// -----------------------------------------------------------------------------
+// Append data to given file name, wait for some time, and then continue
+// adding end of data.
+//
+// This will easily make visible on the file whether two tests have been
+// executed concurrently or not, since it will mess up the lines on the file
+// -----------------------------------------------------------------------------
+async function serialTestConcurrency(name) {
+  const fname = `tmp-${name}.tmp`
+
+  fs.appendFileSync(fname, `${name}: [Start: ${Date.now()} -> `)
+  await sleep(1);
+  fs.appendFileSync(fname, ` End:${Date.now()}]\n`)
+}
+
 module.exports = {
-  sleep
+  sleep,
+  serialTestConcurrency
 }

package/index.js

@@ -15,7 +15,7 @@ const GRANULARITY = {
 const g_redisAddress = process.env.MOCHA_DISTRIBUTED || "";
 const g_testExecutionId = process.env.MOCHA_DISTRIBUTED_EXECUTION_ID || "";
 const g_expirationTime =
-  process.env.MOCHA_DISTRIBUTED_EXPIRATION_TIME || `${24 * 3600}`;
+  process.env.MOCHA_DISTRIBUTED_EXPIRATION_TIME || `${7 * 24 * 3600}`;
 
 // Generate a unique random id for this runner (with almost 100% certainty
 // to be different on any machine/environment).
@@ -59,6 +59,29 @@ function getTestPath(testContext) {
   return path.reverse();
 }
 
+// -----------------------------------------------------------------------------
+// getSerialGranularity
+//
+// Returns the full string or the "serial string" which is whatever finds that
+// follows this regex "[serial.*]" on the string. Only first instance is
+// returned.
+//
+// This will allow serializing tests with given serial name.
+// -----------------------------------------------------------------------------
+function getSerialGranularity(testKey) {
+  // NOTE: a regular expression might be trickier to get right, since you can
+  //       have multiple instances of [serialxxxx] on the same string
+  let index = testKey.indexOf('[serial')
+  if (index === -1)
+    return testKey
+
+  let index2 = testKey.indexOf(']', index)
+  if (index2 === -1)
+    return testKey
+
+  return testKey.substring(index, index2+1)
+}
+
 // -----------------------------------------------------------------------------
 // captureStream
 // -----------------------------------------------------------------------------
@@ -66,18 +89,18 @@ function captureStream(stream) {
   var oldWrite = stream.write;
   var buf = [];
 
-  stream.write = function(chunk, encoding, callback){
-    buf.push (chunk.toString()); // chunk is a String or Buffer
+  stream.write = function (chunk, encoding, callback) {
+    buf.push(chunk.toString()); // chunk is a String or Buffer
     oldWrite.apply(stream, arguments);
-  }
+  };
 
   return {
-    unhook(){
-     stream.write = oldWrite;
+    unhook() {
+      stream.write = oldWrite;
     },
-    captured(){
+    captured() {
       return buf;
-    }
+    },
   };
 }
 
@@ -101,7 +124,7 @@ exports.mochaGlobalSetup = async function () {
   }
 
   if (!g_redisAddress || !g_testExecutionId) {
-    console.log (g_redisAddress, g_testExecutionId)
+    console.log(g_redisAddress, g_testExecutionId);
     console.error(
       "You need to set at least the following environment variables:\n" +
         "  - MOCHA_DISTRIBUTED\n" +
@@ -137,8 +160,8 @@ exports.mochaGlobalTeardown = async function () {
 exports.mochaHooks = {
   beforeEach: async function () {
     const testPath = getTestPath(this.currentTest);
-    const testKeyFullPath = `${g_testExecutionId}:${testPath.join(":")}`;
-    const testKeySuite = `${g_testExecutionId}:${testPath[0]}`;
+    const testKeyFullPath = `${g_testExecutionId}:${getSerialGranularity(testPath.join(":"))}`;
+    const testKeySuite = `${g_testExecutionId}:${getSerialGranularity(testPath[0])}`;
 
     const testKey =
       g_granularity === GRANULARITY.TEST ? testKeyFullPath : testKeySuite;
@@ -154,29 +177,32 @@ exports.mochaHooks = {
     if (assignedRunnerId !== g_runnerId) {
       this.currentTest.title += " (skipped by mocha_distributted)";
       this.skip();
-    }
-    else {
+    } else {
       g_capture.stdout = captureStream(process.stdout);
       g_capture.stderr = captureStream(process.stderr);
     }
   },
+
   afterEach(done) {
     const SKIPPED = "pending";
     const FAILED = "failed";
     const PASSED = "passed";
 
-    let capturedStdout = '';
-    let capturedStderr = '';
+    let capturedStdout = "";
+    let capturedStderr = "";
     if (g_capture.stdout) {
       const stdoutArray = g_capture.stdout.captured();
-      capturedStdout = stdoutArray.join('');
-      capturedStdout = capturedStdout.replace(/\s*\u001b\[3[12]m[^\n]*\n$/g, '');
+      capturedStdout = stdoutArray.join("");
+      capturedStdout = capturedStdout.replace(
+        /\s*\u001b\[3[12]m[^\n]*\n$/g,
+        ""
+      );
       g_capture.stdout.unhook();
       g_capture.stdout = null;
     }
 
     if (g_capture.stderr) {
-      capturedStderr = g_capture.stderr.captured().join('');
+      capturedStderr = g_capture.stderr.captured().join("");
       g_capture.stderr.unhook();
       g_capture.stderr = null;
     }
@@ -184,7 +210,28 @@ exports.mochaHooks = {
     // Save all data in redis in a way it can be retrieved and aggregated
     // easily for all test by an external reporter
     if (this.currentTest.state !== SKIPPED) {
-      const stateFixed = this.currentTest.state || (this.currentTest.timedOut ? FAILED : PASSED)
+      const retryAttempt = this.currentTest._currentRetry || 0;
+      const retryTotal = this.currentTest._retries || 1;
+
+      // adjust state value accounting for exceptions, timeouts & retries
+      let stateFixed = PASSED;
+      if (
+        this.currentTest.state === FAILED ||
+        this.currentTest.timedOut ||
+        (typeof this.currentTest.state === "undefined" &&
+          retryAttempt < retryTotal)
+      ) {
+        stateFixed = FAILED;
+      }
+
+      // Error objects cannot be properly serialized with stringify, thus
+      // we need to use this hack to make it look like a normal object.
+      // Hopefully this should work as well with other sort of objects
+      const err = this.currentTest.err || null;
+      const errObj = JSON.parse(
+        JSON.stringify(err, Object.getOwnPropertyNames(err || {}))
+      );
+
       const testResult = {
         id: getTestPath(this.currentTest),
         type: this.currentTest.type,
@@ -193,15 +240,15 @@ exports.mochaHooks = {
         duration: this.currentTest.duration,
         startTime: Date.now() - (this.currentTest.duration || 0),
         endTime: Date.now(),
-        retryAttempt: this.currentTest._currentRetry || 0,
-        retryTotal: this.currentTest._retries || 1,
+        retryAttempt: retryAttempt,
+        retryTotal: retryTotal,
         file: this.currentTest.file,
         state: stateFixed,
         failed: stateFixed === FAILED,
         speed: this.currentTest.speed,
-        err: this.currentTest.err || null,
+        err: errObj,
         stdout: capturedStdout,
-        stderr: capturedStderr
+        stderr: capturedStderr,
       };
 
       // save results as single line on purpose
@@ -210,7 +257,7 @@ exports.mochaHooks = {
       g_redis.expire(key, g_expirationTime);
 
       // increment passed_count/failed_count & set expiry time
-      const countKey = `${g_testExecutionId}:${stateFixed}_count`
+      const countKey = `${g_testExecutionId}:${stateFixed}_count`;
       g_redis.incr(countKey);
       g_redis.expire(countKey, g_expirationTime);
     }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mocha-distributed",
-  "version": "0.9.4",
+  "version": "0.9.6",
   "description": "Run multiple mocha suites and tests in parallel, from different processes and different machines. Results available on a redis database.",
   "main": "index.js",
   "scripts": {
@@ -23,12 +23,12 @@
   ],
   "repository": {
     "type": "git",
-    "url": "https://github.com/pausan/mocha-distributed.git"
+    "url": "git+https://github.com/pausan/mocha-distributed.git"
   },
   "author": "Pau Sanchez",
   "license": "MIT",
   "dependencies": {
-    "redis": "^4.0.2"
+    "redis": "^4.0.3"
   },
   "devDependencies": {
     "chai": "^4.2.0",

package/test-example-nworkers.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+export MOCHA_DISTRIBUTED_EXECUTION_ID="eid_$(date +%s)"
+export MOCHA_DISTRIBUTED="redis://127.0.0.1"
+
+npm install > /dev/null 2>&1
+
+N=$1
+COMMAND="mocha --require ./index.js example/**/*.js"
+
+# cleanup tmp files
+rm tmp-*
+
+echo "Spawning $N commands in parallel with Execution ID: $MOCHA_DISTRIBUTED_EXECUTION_ID"
+
+# Run the command N times in the background, saving stdout/stderr
+for ((i = 1; i <= N; i++)); do
+    $COMMAND >"tmp-output-$i.log" 2>&1 &
+done
+
+wait
+
+echo "All tests finished!"