mocha-distributed 0.9.5 → 0.9.7

package/README.md

@@ -1,6 +1,6 @@
 # mocha-distributed
 
-Run mocha tests faster.
+Run mocha tests in parallel.
 
 Speed up your mocha tests by running them in parallel in multiple machines all
 at once without changing a single line of code. You only need a redis server.
@@ -12,15 +12,16 @@ tests without having to change any line of code, nor having to decide
 what to run where. Tests spread automatically according to the nodes you have.
 
 The concept is very simple, basically you spawn as many runners as you wish
-on as many nodes as you wish, and each node decides whether they should run
+on as many nodes as you wish, and each process decides whether they should run
 a test or the test has already been executed or is being executed somewhere
 else.
 
-It does not matter if you run the tests in one machine as subprocesses or in
-many machines with multiple processes each.
+It does not matter if you run the tests in one machine in multiple processes or
+in multiple machines with multiple processes each. It will just work.
 
-Because you don't need to change a single line of code, which means that you
-can still run mocha locally as usual.
+You don't need to change a single line of code, thus, this library it allows you
+to continue developing tests as usual and launch them in parallel whenever you
+want. No strings attached.
 
 ## Quick start
 
@@ -38,7 +39,7 @@ or machines where you want to run the tests on.
 Finally, on each of the runners just run:
 
   ```bash
-    $ export MOCHA_DISTRIBUTED_EXECUTION_ID="execution__2021-01-01__20:10" 
+    $ export MOCHA_DISTRIBUTED_EXECUTION_ID="execution__2024-01-01__20:10"
     $ export MOCHA_DISTRIBUTED="redis://redis.address" 
     $ mocha --require mocha-distributed test/**/*.js
   ```
@@ -102,11 +103,11 @@ whether a test has already been executed or not by other of their peers.
     string in each machine, BUT you can override this in case you need it for
     whatever reason, although in theory you probably shouldn't.
 
-  - **MOCHA_DISTRIBUTED_EXPIRATION_TIME** = 86400
+  - **MOCHA_DISTRIBUTED_EXPIRATION_TIME** = 604800
 
     Configures to how long the data is kept in redis before it expires (in 
-    seconds). The amount of data in redis is minimal, so you probably don't want
-    to play with it.
+    seconds). 7 days is the default. The amount of data in redis is minimal,
+    so you probably don't want to play with it.
 
     It might be helpful to increase it though, if you want to build some sort of
     reporting on top of it, because you can directly explore test results in
@@ -172,6 +173,65 @@ Keep in mind that:
 You might have a look at list-tests-from-redis.js for an example on how to
 query redis and list all tests.
 
+## Run tests serially
+
+If you'd like some of your tests to run serially you can use a magic string with
+this framework.
+
+Simply add "[serial]" or "[serial-<ID OF YOUR CHOICE>]" to the title of your
+test or test suite and all those tests will execute serially by the same runner.
+
+The important part is that the test title contains "[serial" and ends with "]"
+
+It's easier to explain with a couple of examples:
+
+The following tests, regardless of whether they are on the same file or spreaded
+in multiple files, will be executed all by the same runner one after another.
+
+Might run in parallel to other tests that don't contain the "[serial]" word,
+but will run sequentially for this group.
+
+```javascript
+it('Test id 1 [serial]', function() { /* ... */})
+it('Test id 2 [serial]', function() { /* ... */})
+it('Test id 3 [serial]', function() { /* ... */})
+it('Test id 4 [serial]', function() { /* ... */})
+```
+
+See this other example below. Again, regardless of whether the tests are on the
+same file or spreaded in multiple files, will be executed by two sets of
+runners.
+
+```javascript
+it('Test id 1 [serial-worker]', function() { /* ... */})
+it('Test id 2 [serial-worker]', function() { /* ... */})
+it('Test id 3 [serial-another worker]', function() { /* ... */})
+it('Test id 4 [serial-another worker]', function() { /* ... */})
+```
+
+Test 1 and 2 will be executed by one runner, whereas test 3 and 4 will be
+executed by another. In both cases 1 and 2 will be executed sequentially and 3
+and 4 also sequentially, but since they have different serial IDs, those two
+subgroups of tests can run in parallel (e.g 1 and 2 in parallel with 3 and 4).
+
+And now last example below:
+
+```javascript
+describe('[serial-my test id] test multiple things sequentially', function () {
+  it('Test id 1', function() { /* ... */})
+  it('Test id 2', function() { /* ... */})
+  it('Test id 3', function() { /* ... */})
+  it('Test id 4', function() { /* ... */})
+})
+```
+
+The suite contains "[serial-my test id]", but the tests don't contain any serial
+magic id. In this case, ALL those tests will run sequentially because the suite
+contains the magic word.
+
+Long story short. Add "[serial-whatever you want]" on the title but make sure
+that "whatever you want" is the same for the stuff you want to run sequentially.
+
 ## Examples
 
 ### Environment-agnostic

package/example/suite-5-serial.js

@@ -0,0 +1,34 @@
+const util = require('./util.js');
+
+// require('mocha-distributed')
+require('../index.js');
+
+describe ('suite-5.serial', async function () {
+  beforeEach (function () {
+    // console.log ('before each');
+  });
+
+  it ('[serial] test-5.0', async function () {
+    await util.serialTestConcurrency('serial5')
+  });
+
+  it ('[serial:odd-worker] test-5.1', async function () {
+    await util.serialTestConcurrency('odd-worker')
+  });
+
+  it ('[serial:even-worker] test-5.2', async function () {
+    await util.serialTestConcurrency('even-worker')
+  });
+
+  it ('[serial:odd-worker] test-5.3', async function () {
+    await util.serialTestConcurrency('odd-worker')
+  });
+
+  it ('[serial:even-worker] test-5.4', async function () {
+    await util.serialTestConcurrency('even-worker')
+  });
+
+  it ('[serial] test-5.5', async function () {
+    await util.serialTestConcurrency('serial5')
+  });
+});

package/example/suite-6-serial.js

@@ -0,0 +1,35 @@
+const util = require('./util.js');
+
+// require('mocha-distributed')
+require('../index.js');
+
+// NOTE: since "describe" has [serial] on it, all will be executed with that ID
+describe ('[serial] suite-6.serial', async function () {
+  beforeEach (function () {
+    // console.log ('before each');
+  });
+
+  it ('[serial] test-6.0', async function () {
+    await util.serialTestConcurrency('serial6')
+  });
+
+  it ('[serial:odd-worker] test-6.1', async function () {
+    await util.serialTestConcurrency('serial6')
+  });
+
+  it ('[serial:even-worker] test-6.2', async function () {
+    await util.serialTestConcurrency('serial6')
+  });
+
+  it ('[serial:odd-worker] test-6.3', async function () {
+    await util.serialTestConcurrency('serial6')
+  });
+
+  it ('[serial:even-worker] test-6.4', async function () {
+    await util.serialTestConcurrency('serial6')
+  });
+
+  it ('[serial] test-6.5', async function () {
+    await util.serialTestConcurrency('serial6')
+  });
+});

package/example/util.js

@@ -1,3 +1,4 @@
+const fs = require('fs');
 const util = require('./util.js');
 
 async function sleep (seconds) {
@@ -6,6 +7,22 @@ async function sleep (seconds) {
   });
 }
 
+// -----------------------------------------------------------------------------
+// Append data to given file name, wait for some time, and then continue
+// adding end of data.
+//
+// This will easily make visible on the file whether two tests have been
+// executed concurrently or not, since it will mess up the lines on the file
+// -----------------------------------------------------------------------------
+async function serialTestConcurrency(name) {
+  const fname = `tmp-${name}.tmp`
+
+  fs.appendFileSync(fname, `${name}: [Start: ${Date.now()} -> `)
+  await sleep(1);
+  fs.appendFileSync(fname, ` End:${Date.now()}]\n`)
+}
+
 module.exports = {
-  sleep
+  sleep,
+  serialTestConcurrency
 }

package/index.js

@@ -15,7 +15,7 @@ const GRANULARITY = {
 const g_redisAddress = process.env.MOCHA_DISTRIBUTED || "";
 const g_testExecutionId = process.env.MOCHA_DISTRIBUTED_EXECUTION_ID || "";
 const g_expirationTime =
-  process.env.MOCHA_DISTRIBUTED_EXPIRATION_TIME || `${24 * 3600}`;
+  process.env.MOCHA_DISTRIBUTED_EXPIRATION_TIME || `${7 * 24 * 3600}`;
 
 // Generate a unique random id for this runner (with almost 100% certainty
 // to be different on any machine/environment).
@@ -34,6 +34,11 @@ let g_redis = null;
 
 let g_capture = { stdout: null, stderr: null };
 
+// Cache errors from intermediate retry attempts. Mocha only sets test.err via
+// the reporter on the final EVENT_TEST_FAIL; for non-final retries it emits
+// EVENT_TEST_RETRY instead and never stores the error on the test object.
+const g_retryErrors = new Map();
+
 // -----------------------------------------------------------------------------
 // getTestPath
 //
@@ -59,6 +64,29 @@ function getTestPath(testContext) {
   return path.reverse();
 }
 
+// -----------------------------------------------------------------------------
+// getSerialGranularity
+//
+// Returns the full string or the "serial string" which is whatever finds that
+// follows this regex "[serial.*]" on the string. Only first instance is
+// returned.
+//
+// This will allow serializing tests with given serial name.
+// -----------------------------------------------------------------------------
+function getSerialGranularity(testKey) {
+  // NOTE: a regular expression might be trickier to get right, since you can
+  //       have multiple instances of [serialxxxx] on the same string
+  let index = testKey.indexOf('[serial')
+  if (index === -1)
+    return testKey
+
+  let index2 = testKey.indexOf(']', index)
+  if (index2 === -1)
+    return testKey
+
+  return testKey.substring(index, index2+1)
+}
+
 // -----------------------------------------------------------------------------
 // captureStream
 // -----------------------------------------------------------------------------
@@ -85,6 +113,11 @@ function captureStream(stream) {
 // Initialize redis once before the tests
 // -----------------------------------------------------------------------------
 exports.mochaGlobalSetup = async function () {
+  // `this` is the Mocha Runner — store errors from non-final retry attempts
+  // so afterEach can record them (Mocha never sets test.err for those).
+  this.on('retry', (test, err) => {
+    g_retryErrors.set(test.fullTitle(), err);
+  });
   if (g_mochaVerbose) {
     const redisNoCredentials = g_redisAddress.replace(
       /\/\/[^@]*@/,
@@ -137,8 +170,8 @@ exports.mochaGlobalTeardown = async function () {
 exports.mochaHooks = {
   beforeEach: async function () {
     const testPath = getTestPath(this.currentTest);
-    const testKeyFullPath = `${g_testExecutionId}:${testPath.join(":")}`;
-    const testKeySuite = `${g_testExecutionId}:${testPath[0]}`;
+    const testKeyFullPath = `${g_testExecutionId}:${getSerialGranularity(testPath.join(":"))}`;
+    const testKeySuite = `${g_testExecutionId}:${getSerialGranularity(testPath[0])}`;
 
     const testKey =
       g_granularity === GRANULARITY.TEST ? testKeyFullPath : testKeySuite;
@@ -160,7 +193,7 @@ exports.mochaHooks = {
     }
   },
 
-  afterEach(done) {
+  afterEach: async function () {
     const SKIPPED = "pending";
     const FAILED = "failed";
     const PASSED = "passed";
@@ -204,7 +237,10 @@ exports.mochaHooks = {
       // Error objects cannot be properly serialized with stringify, thus
       // we need to use this hack to make it look like a normal object.
       // Hopefully this should work as well with other sort of objects
-      const err = this.currentTest.err || null;
+      const err = this.currentTest.err
+        || g_retryErrors.get(this.currentTest.fullTitle())
+        || null;
+      g_retryErrors.delete(this.currentTest.fullTitle());
       const errObj = JSON.parse(
         JSON.stringify(err, Object.getOwnPropertyNames(err || {}))
       );
@@ -229,16 +265,16 @@ exports.mochaHooks = {
       };
 
       // save results as single line on purpose
-      const key = `${g_testExecutionId}:test_result`;
-      g_redis.rPush(key, JSON.stringify(testResult));
-      g_redis.expire(key, g_expirationTime);
-
-      // increment passed_count/failed_count & set expiry time
+      const resultKey = `${g_testExecutionId}:test_result`;
       const countKey = `${g_testExecutionId}:${stateFixed}_count`;
-      g_redis.incr(countKey);
-      g_redis.expire(countKey, g_expirationTime);
-    }
 
-    done();
+      await g_redis
+        .multi()
+        .rPush(resultKey, JSON.stringify(testResult))
+        .expire(resultKey, g_expirationTime)
+        .incr(countKey)
+        .expire(countKey, g_expirationTime)
+        .exec();
+    }
   },
 };

package/package.json

@@ -1,10 +1,10 @@
 {
   "name": "mocha-distributed",
-  "version": "0.9.5",
+  "version": "0.9.7",
   "description": "Run multiple mocha suites and tests in parallel, from different processes and different machines. Results available on a redis database.",
   "main": "index.js",
   "scripts": {
-    "test": "echo \"Error: no test specified\" && exit 1"
+    "test": "mocha test/test-retry.js"
   },
   "keywords": [
     "mocha",
@@ -23,7 +23,7 @@
   ],
   "repository": {
     "type": "git",
-    "url": "https://github.com/pausan/mocha-distributed.git"
+    "url": "git+https://github.com/pausan/mocha-distributed.git"
   },
   "author": "Pau Sanchez",
   "license": "MIT",

package/test/test-retry.js

@@ -0,0 +1,142 @@
+// -----------------------------------------------------------------------------
+// test/test-retry.js
+//
+// Mocha test suite verifying that mocha-distributed records err, stdout and
+// stderr on ALL retry attempts, not just the last one.
+//
+// Run with: mocha test/test-retry.js
+// -----------------------------------------------------------------------------
+'use strict';
+
+const assert = require('assert');
+const Mocha  = require('mocha/lib/mocha');
+const Suite  = require('mocha/lib/suite');
+const Test   = require('mocha/lib/test');
+
+// -----------------------------------------------------------------------------
+// Mock redis helpers
+// -----------------------------------------------------------------------------
+const redisResolved = require.resolve('redis');
+
+const mockMulti = (writtenResults) => () => {
+  const cmds = [];
+  const chain = {
+    set:    (...a) => { cmds.push(['set',    ...a]); return chain; },
+    get:    (...a) => { cmds.push(['get',    ...a]); return chain; },
+    rPush:  (...a) => {
+      cmds.push(['rPush', ...a]);
+      writtenResults.push(JSON.parse(a[1]));
+      return chain;
+    },
+    expire: (...a) => { cmds.push(['expire', ...a]); return chain; },
+    incr:   (...a) => { cmds.push(['incr',   ...a]); return chain; },
+    exec: async () => {
+      // beforeEach pipeline: SET NX + GET — this runner always wins ownership
+      if (cmds[0] && cmds[0][0] === 'set') {
+        return [null, process.env.MOCHA_DISTRIBUTED_RUNNER_ID];
+      }
+      // afterEach pipeline: rPush + expire + incr + expire
+      return [1, 1, 1, 1];
+    }
+  };
+  return chain;
+};
+
+function injectMockRedis(writtenResults) {
+  require.cache[redisResolved] = {
+    id:       redisResolved,
+    filename: redisResolved,
+    loaded:   true,
+    exports:  {
+      createClient: () => ({
+        on:      () => {},
+        connect: async () => {},
+        quit:    async () => {},
+        multi:   mockMulti(writtenResults),
+      })
+    },
+  };
+}
+
+function restoreRedis() {
+  delete require.cache[redisResolved];
+}
+
+function loadFreshLib() {
+  const libPath = require.resolve('../index.js');
+  delete require.cache[libPath];
+  return require('../index.js');
+}
+
+// -----------------------------------------------------------------------------
+// Tests
+// -----------------------------------------------------------------------------
+describe('mocha-distributed', function () {
+
+  describe('retry attempt recording', function () {
+    let writtenResults;
+    let lib;
+
+    before(function () {
+      writtenResults = [];
+      injectMockRedis(writtenResults);
+
+      process.env.MOCHA_DISTRIBUTED              = 'redis://mock';
+      process.env.MOCHA_DISTRIBUTED_EXECUTION_ID = 'test-exec-retry';
+      process.env.MOCHA_DISTRIBUTED_RUNNER_ID    = 'runner-test';
+
+      lib = loadFreshLib();
+    });
+
+    after(function () {
+      restoreRedis();
+      delete require.cache[require.resolve('../index.js')];
+    });
+
+    it('records err, stdout and stderr on every retry attempt', async function () {
+      this.timeout(10000);
+
+      // Build the inner mocha instance with a flaky test that fails on
+      // attempts 0 and 1, and passes on attempt 2
+      const m = new Mocha({ reporter: 'min' });
+      m.rootHooks(lib.mochaHooks);
+      m.globalSetup([lib.mochaGlobalSetup]);
+      m.globalTeardown([lib.mochaGlobalTeardown]);
+
+      const suite = Suite.create(m.suite, 'retry-suite');
+      suite.retries(2);
+
+      let attempt = 0;
+      suite.addTest(new Test('flaky-test', function () {
+        attempt++;
+        console.log('stdout from attempt ' + attempt);
+        console.error('stderr from attempt ' + attempt);
+        if (attempt < 3) throw new Error('intentional failure on attempt ' + attempt);
+      }));
+
+      await new Promise(resolve => m.run(resolve));
+
+      // Sort by retryAttempt for stable assertions
+      const results = writtenResults.slice().sort((a, b) => a.retryAttempt - b.retryAttempt);
+
+      assert.strictEqual(results.length, 3, '3 results written to Redis (one per attempt)');
+
+      // Attempts 0 and 1: failed, err populated, stdout and stderr present
+      for (const i of [0, 1]) {
+        const r = results[i];
+        assert.strictEqual(r.retryAttempt, i,             `attempt ${i}: retryAttempt`);
+        assert.strictEqual(r.state, 'failed',             `attempt ${i}: state`);
+        assert.ok(r.err && r.err.message,                 `attempt ${i}: err.message should be set`);
+        assert.ok(r.stdout.includes(`attempt ${i + 1}`), `attempt ${i}: stdout`);
+        assert.ok(r.stderr.includes(`attempt ${i + 1}`), `attempt ${i}: stderr`);
+      }
+
+      // Attempt 2: passed, stdout and stderr present
+      const r2 = results[2];
+      assert.strictEqual(r2.retryAttempt, 2,    'attempt 2: retryAttempt');
+      assert.strictEqual(r2.state, 'passed',     'attempt 2: state');
+      assert.ok(r2.stdout.includes('attempt 3'), 'attempt 2: stdout');
+      assert.ok(r2.stderr.includes('attempt 3'), 'attempt 2: stderr');
+    });
+  });
+});

package/test-example-nworkers.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+export MOCHA_DISTRIBUTED_EXECUTION_ID="eid_$(date +%s)"
+export MOCHA_DISTRIBUTED="redis://127.0.0.1"
+
+npm install > /dev/null 2>&1
+
+N=$1
+COMMAND="mocha --require ./index.js example/**/*.js"
+
+# cleanup tmp files
+rm tmp-*
+
+echo "Spawning $N commands in parallel with Execution ID: $MOCHA_DISTRIBUTED_EXECUTION_ID"
+
+# Run the command N times in the background, saving stdout/stderr
+for ((i = 1; i <= N; i++)); do
+    $COMMAND >"tmp-output-$i.log" 2>&1 &
+done
+
+wait
+
+echo "All tests finished!"