codeceptjs 4.0.0-rc.20 → 4.0.0-rc.22

package/docs/parallel.md

@@ -5,581 +5,183 @@ title: Parallel Execution
 
 # Parallel Execution
 
-CodeceptJS has multiple approaches for running tests in parallel:
+Two built-in ways to run tests in parallel:
 
-- **Test Sharding** - distributes tests across multiple machines for CI matrix execution
-- `run-workers` - which spawns [NodeJS Worker](https://nodejs.org/api/worker_threads.html) in a thread. Tests are split by scenarios, scenarios are mixed between groups, each worker runs tests from its own group.
-- `run-multiple` - which spawns a subprocess with CodeceptJS. Tests are split by files and configured in `codecept.conf.js`.
+- **`run-workers`** — split tests across worker threads on one runner.
+- **`--shard`** — split test files across CI machines in a matrix build.
 
-Workers are faster and simpler to start, while `run-multiple` requires additional configuration and can be used to run tests in different browsers at once.
+For anything more specific — your own grouping, a config per group, several browsers at once — drive the `Workers` API from a script (see [Custom parallelization](#custom-parallelization)).
 
-## Test Sharding for CI Matrix
+## Workers
 
-Test sharding allows you to split your test suite across multiple machines or CI workers without manual configuration. This is particularly useful for CI/CD pipelines where you want to run tests in parallel across different machines.
-
-Use the `--shard` option with the `run` command to execute only a portion of your tests:
-
-```bash
-# Run the first quarter of tests
-npx codeceptjs run --shard 1/4
-
-# Run the second quarter of tests
-npx codeceptjs run --shard 2/4
-
-# Run the third quarter of tests
-npx codeceptjs run --shard 3/4
-
-# Run the fourth quarter of tests
-npx codeceptjs run --shard 4/4
-```
-
-### CI Matrix Example
-
-Here's how you can use test sharding with GitHub Actions matrix strategy:
-
-```yaml
-name: Tests
-on: [push, pull_request]
-
-jobs:
-  test:
-    runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        shard: [1/4, 2/4, 3/4, 4/4]
-
-    steps:
-      - uses: actions/checkout@v2
-      - uses: actions/setup-node@v2
-      - run: npm install
-      - run: npx codeceptjs run --shard ${{ matrix.shard }}
-```
-
-This approach ensures:
-
-- Each CI job runs only its assigned portion of tests
-- Tests are distributed evenly across shards
-- No manual configuration or maintenance of test lists
-- Automatic load balancing as you add or remove tests
-
-### Shard Distribution
-
-Tests are distributed evenly across shards using a round-robin approach:
-
-- If you have 100 tests and 4 shards, each shard runs approximately 25 tests
-- The first shard gets tests 1-25, second gets 26-50, third gets 51-75, fourth gets 76-100
-- If tests don't divide evenly, earlier shards may get one extra test
-
-## Parallel Execution by Workers
-
-It is easy to run tests in parallel if you have a lots of tests and free CPU cores. Just execute your tests using `run-workers` command specifying the number of workers to spawn:
-
-```
-npx codeceptjs run-workers 2
-```
-
-> ℹ Workers require NodeJS >= 11.7
-
-This command is similar to `run`, however, steps output can't be shown in workers mode, as it is impossible to synchronize steps output from different processes.
-
-Each worker spins an instance of CodeceptJS, executes a group of tests, and sends back report to the main process.
-
-By default, the tests are assigned one by one to the available workers this may lead to multiple execution of `BeforeSuite()`. Use the option `--suites` to assign the suites one by one to the workers.
+`run-workers <N>` spawns `N` [worker threads](https://nodejs.org/api/worker_threads.html), each an independent CodeceptJS instance running a slice of the suite, and merges the results:
 
 ```sh
-npx codeceptjs run-workers --suites 2
+npx codeceptjs run-workers 4
 ```
 
-### Test Distribution Strategies
+Steps are not streamed to the console in this mode — output from separate threads can't be interleaved cleanly. While workers run, CodeceptJS sets `process.env.RUNS_WITH_WORKERS=true`, so plugins and helpers can branch on it. All `run` options work here too: `--grep "@smoke"`, `-c codecept.conf.js`, `--debug`, and the rest.
 
-CodeceptJS supports three different strategies for distributing tests across workers:
+### Distribution strategies
 
-#### Default Strategy (`--by test`)
+`--by` controls how tests spread across workers:
 
-Tests are pre-assigned to workers at startup, distributing them evenly across all workers. Each worker gets a predetermined set of tests to run.
+| `--by` | How tests are assigned | Use when |
+| --- | --- | --- |
+| `test` (default) | each test pinned to a worker up front | tests take roughly equal time |
+| `suite` | each suite pinned to a worker; its tests stay together | suites share a `BeforeSuite` you don't want repeated |
+| `pool` | workers pull the next test from a shared queue as they free up | test durations vary — best load balancing |
 
 ```sh
-npx codeceptjs run-workers 3 --by test
-```
-
-#### Suite Strategy (`--by suite`)
-
-Test suites are pre-assigned to workers, with all tests in a suite running on the same worker. This ensures better test isolation but may lead to uneven load distribution.
-
-```sh
-npx codeceptjs run-workers 3 --by suite
-```
-
-#### Pool Strategy (`--by pool`) - **Recommended for optimal performance**
-
-Tests are maintained in a shared pool and distributed dynamically to workers as they become available. This provides the best load balancing and resource utilization.
-
-```sh
-npx codeceptjs run-workers 3 --by pool
-```
-
-## Dynamic Test Pooling Mode
-
-The pool mode enables dynamic test distribution for improved worker load balancing. Instead of pre-assigning tests to workers at startup, tests are stored in a shared pool and distributed on-demand as workers become available.
-
-### Benefits of Pool Mode
-
-- **Better load balancing**: Workers never sit idle while others are still running long tests
-- **Improved performance**: Especially beneficial when tests have varying execution times
-- **Optimal resource utilization**: All CPU cores stay busy until the entire test suite is complete
-- **Automatic scaling**: Workers continuously process tests until the pool is empty
-
-### When to Use Pool Mode
-
-Pool mode is particularly effective in these scenarios:
-
-- **Uneven test execution times**: When some tests take significantly longer than others
-- **Large test suites**: With hundreds or thousands of tests where load balancing matters
-- **Mixed test types**: When combining unit tests, integration tests, and end-to-end tests
-- **CI/CD pipelines**: For consistent and predictable test execution times
-
-### Usage Examples
-
-```bash
-# Basic pool mode with 4 workers
 npx codeceptjs run-workers 4 --by pool
-
-# Pool mode with grep filtering
-npx codeceptjs run-workers 3 --by pool --grep "@smoke"
-
-# Pool mode in debug mode
-npx codeceptjs run-workers 2 --by pool --debug
-
-# Pool mode with specific configuration
-npx codeceptjs run-workers 3 --by pool -c codecept.conf.js
 ```
 
-### How Pool Mode Works
-
-1. **Pool Creation**: All tests are collected into a shared pool of test identifiers
-2. **Worker Initialization**: The specified number of workers are spawned
-3. **Dynamic Assignment**: Workers request tests from the pool when they're ready
-4. **Continuous Processing**: Each worker runs one test, then immediately requests the next
-5. **Automatic Completion**: Workers exit when the pool is empty and no more tests remain
-
-### Performance Comparison
-
-```bash
-# Traditional mode - tests pre-assigned, some workers may finish early
-npx codeceptjs run-workers 3 --by test   # ✓ Good for uniform test times
-
-# Suite mode - entire suites assigned to workers
-npx codeceptjs run-workers 3 --by suite  # ✓ Good for test isolation
+`--suites` is shorthand for `--by suite`.
 
-# Pool mode - tests distributed dynamically
-npx codeceptjs run-workers 3 --by pool   # ✓ Best for mixed test execution times
-```
+### Multiple browsers
 
-## Test stats with Parallel Execution by Workers
+Define browser profiles under `multiple` in `codecept.conf.js`:
 
 ```js
-import { event } from 'codeceptjs';
-
-export default function() {
+multiple: {
+  default: { browsers: ['chrome', 'firefox'] },
+}
+```
 
-  event.dispatcher.on(event.workers.result, function (result) {
+Then run a profile across workers — by name, or `all` for every profile:
 
-    console.log(result);
+```sh
+npx codeceptjs run-workers 3 default
+npx codeceptjs run-workers 3 all
+```
 
-  });
-}
+(`run-multiple` runs the same profiles in separate subprocesses instead of threads — see `npx codeceptjs run-multiple --help`.)
 
-// in console log
-FAIL  | 7 passed, 1 failed, 1 skipped   // 2s
-{
-  "tests": {
-  "passed": [
-    {
-      "type": "test",
-      "title": "Assert @C3",
-      "body": "() => { }",
-      "async": 0,
-      "sync": true,
-      "_timeout": 2000,
-      "_slow": 75,
-      "_retries": -1,
-      "timedOut": false,
-      "_currentRetry": 0,
-      "pending": false,
-      "opts": {},
-      "tags": [
-        "@C3"
-      ],
-      "uid": "xe4q1HdqpRrZG5dPe0JG+A",
-      "workerIndex": 3,
-      "retries": -1,
-      "duration": 493,
-      "err": null,
-      "parent": {
-        "title": "My",
-        "ctx": {},
-        "suites": [],
-        "tests": [],
-        "root": false,
-        "pending": false,
-        "_retries": -1,
-        "_beforeEach": [],
-        "_beforeAll": [],
-        "_afterEach": [],
-        "_afterAll": [],
-        "_timeout": 2000,
-        "_slow": 75,
-        "_bail": false,
-        "_onlyTests": [],
-        "_onlySuites": [],
-        "delayed": false
-      },
-      "steps": [
-        {
-          "actor": "I",
-          "name": "amOnPage",
-          "status": "success",
-          "args": [
-            "https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST"
-          ],
-          "startedAt": 1698760652610,
-          "startTime": 1698760652611,
-          "endTime": 1698760653098,
-          "finishedAt": 1698760653098,
-          "duration": 488
-        },
-        {
-          "actor": "I",
-          "name": "grabCurrentUrl",
-          "status": "success",
-          "args": [],
-          "startedAt": 1698760653098,
-          "startTime": 1698760653098,
-          "endTime": 1698760653099,
-          "finishedAt": 1698760653099,
-          "duration": 1
-        }
-      ]
-    }
-  ],
-    "failed": [],
-    "skipped": []
-}
-}
-```
+### Reading worker results
 
-CodeceptJS also exposes the env var `process.env.RUNS_WITH_WORKERS` when running tests with `run-workers` command so that you could handle the events better in your plugins/helpers
+When all workers finish, the run fires `event.workers.result` with the merged result:
 
 ```js
 import { event } from 'codeceptjs'
 
 export default function () {
-  // this event would trigger the  `_publishResultsToTestrail` when running `run-workers` command
-  event.dispatcher.on(event.workers.result, async () => {
-    await _publishResultsToTestrail()
-  })
-
-  // this event would not trigger the  `_publishResultsToTestrail` multiple times when running `run-workers` command
-  event.dispatcher.on(event.all.result, async () => {
-    // when running `run` command, this env var is undefined
-    if (!process.env.RUNS_WITH_WORKERS) await _publishResultsToTestrail()
+  event.dispatcher.on(event.workers.result, result => {
+    console.log(result.hasFailed() ? 'FAILED' : 'PASSED', result.stats)
+    for (const test of result.tests) {
+      console.log(test.title, test.duration, 'ms', `worker ${test.workerIndex}`)
+    }
   })
 }
 ```
 
-## Parallel Execution by Workers on Multiple Browsers
+For end-of-run work like publishing to a test-management tool, listen on `event.workers.result` (fires once) rather than `event.all.result` (fires in every worker).
 
-To run tests in parallel across multiple browsers, modify your `codecept.conf.js` file to configure multiple browsers on which you want to run your tests and your tests will run across multiple browsers.
+## Sharding
 
-Start with modifying the `codecept.conf.js` file. Add multiple key inside the config which will be used to configure multiple profiles.
+`--shard <index>/<total>` runs only a slice of your **test files**: the file list is cut into `total` even chunks and this run executes chunk `index`. It is built for CI matrices — one machine per shard, each running `run`:
 
-```
-export const config = {
-  helpers: {
-    WebDriver: {
-      url: 'http://localhost:3000',
-      desiredCapabilties: {}
-    }
-  },
-  multiple: {
-    profile1: {
-      browsers: [
-        {
-          browser: "firefox",
-          desiredCapabilties: {
-            // override capabilties related to firefox
-          }
-        },
-        {
-          browser: "chrome",
-          desiredCapabilties: {
-            // override capabilties related to chrome
-          }
-        }
-      ]
-    },
-    profile2: {
-      browsers: [
-        {
-          browser: "safari",
-          desiredCapabilties: {
-            // override capabilties related to safari
-          }
-        }
-      ]
-    }
-  }
-};
-```
-
-To trigger tests on all the profiles configured, you can use the following command:
-
-```
-npx codeceptjs run-workers 3 all -c codecept.conf.js
+```sh
+npx codeceptjs run --shard 1/4
+npx codeceptjs run --shard 2/4
 ```
 
-This will run your tests across all browsers configured from profile1 & profile2 on 3 workers.
+GitHub Actions:
 
-To trigger tests on specific profile, you can use the following command:
-
-```
-npx codeceptjs run-workers 2 profile1 -c codecept.conf.js
+```yaml
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        shard: ['1/4', '2/4', '3/4', '4/4']
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+      - run: npm ci
+      - run: npx codeceptjs run --shard ${{ matrix.shard }}
 ```
 
-This will run your tests across 2 browsers from profile1 on 2 workers.
-
-## Custom Parallel Execution
-
-To get a full control of parallelization create a custom execution script to match your needs.
-This way you can configure which tests are matched, how the groups are formed, and with which configuration each worker is executed.
-
-Start with creating file `bin/parallel.js`.
-
-On MacOS/Linux run following commands:
-
-```
-mkdir bin
-touch bin/parallel.js
-chmod +x bin/parallel.js
-```
+Add or remove tests freely — shards rebalance automatically.
 
-> Filename or directory can be customized. You are creating your own custom runner so take this paragraph as an example.
+## Custom parallelization
 
-Create a placeholder in file:
+When the built-in commands aren't enough, build a runner with the `Workers` API: decide which tests go to which group, give each group its own config, and listen for results.
 
 ```js
 #!/usr/bin/env node
 import { Workers, event } from 'codeceptjs'
-// here will go magic
-```
-
-Now let's see how to update this file for different parallelization modes:
-
-### Example: Running tests in 2 browsers in 4 threads
 
-```js
-const workerConfig = {
-  testConfig: './test/data/sandbox/codecept.customworker.js',
-}
-
-// don't initialize workers in constructor
-const workers = new Workers(null, workerConfig)
-// split tests by suites in 2 groups
-const testGroups = workers.createGroupsOfSuites(2)
-
-const browsers = ['firefox', 'chrome']
-
-const configs = browsers.map(browser => {
-  return {
-    helpers: {
-      WebDriver: { browser },
-    },
-  }
-})
+const workers = new Workers(null, { testConfig: './codecept.conf.js' })
 
-for (const config of configs) {
-  for (group of testGroups) {
+// split the suite into 2 groups, run each group on two browsers
+const groups = workers.createGroupsOfSuites(2)
+for (const browser of ['chromium', 'firefox']) {
+  for (const group of groups) {
     const worker = workers.spawn()
     worker.addTests(group)
-    worker.addConfig(config)
+    worker.addConfig({ helpers: { Playwright: { browser } } })
   }
 }
 
-// Listen events for failed test
-workers.on(event.test.failed, failedTest => {
-  console.log('Failed : ', failedTest.title)
-})
-
-// Listen events for passed test
-workers.on(event.test.passed, successTest => {
-  console.log('Passed : ', successTest.title)
-})
-
-// test run status will also be available in event
-workers.on(event.all.result, () => {
-  // Use printResults() to display result with standard style
-  workers.printResults()
-})
-
-// run workers as async function
-runWorkers()
-
-async function runWorkers() {
-  try {
-    // run bootstrapAll
-    await workers.bootstrapAll()
-    // run tests
-    await workers.run()
-  } finally {
-    // run teardown All
-    await workers.teardownAll()
-  }
-}
-```
+workers.on(event.test.failed, t => console.log('FAIL', t.title))
+workers.on(event.all.result, () => workers.printResults())
 
-Inside `event.all.result` you can obtain test results from all workers, so you can customize the report:
-
-```js
-workers.on(event.all.result, result => {
-  // print output
-  console.log('Test status : ', result.hasFailed() ? 'Failed' : 'Passed');
-
-  // print stats
-  console.log(`Total tests : ${result.stats.tests}`);
-  console.log(`Passed tests : ${result.stats.passes}`);
-  console.log(`Failed test tests : ${result.stats.failures}`);
-
-  // If you don't want to listen for failed and passed test separately, use `tests` array
-  for (const test of result.tests) {
-    console.log(`Test status: ${test.err===null}, `, `Test : ${test.title}`);
-  }
+await workers.bootstrapAll()
+try {
+  await workers.run()
+} finally {
+  await workers.teardownAll()
 }
 ```
 
-### Example: Running Tests Split By A Custom Function
-
-If you want your tests to split according to your need this method is suited for you. For example: If you have 4 long running test files and 4 normal test files there chance all 4 tests end up in same worker thread. For these cases custom function will be helpful.
-
-```js
-/*
- Define a function to split your tests.
-
- function should return an array with this format [[file1, file2], [file3], ...]
-
- where file1 and file2 will run in a worker thread and file3 will run in a worker thread
-*/
-const splitTests = () => {
-  const files = [['./test/data/sandbox/guthub_test.js', './test/data/sandbox/devto_test.js'], ['./test/data/sandbox/longrunnig_test.js']]
-
-  return files
-}
-
-const workerConfig = {
-  testConfig: './test/data/sandbox/codecept.customworker.js',
-  by: splitTests,
-}
-
-// don't initialize workers in constructor
-const customWorkers = new Workers(null, workerConfig)
-
-customWorkers.run()
-
-// You can use event listeners similar to above example.
-customWorkers.on(event.all.result, () => {
-  workers.printResults()
-})
-```
+Building blocks:
 
-### Emitting messages to the parent worker
+- `new Workers(N, { testConfig, options })` — `N` workers; pass `null` to spawn them yourself with `spawn()`.
+- `createGroupsOfTests(n)` / `createGroupsOfSuites(n)` — split the suite into `n` groups.
+- `worker.addTests(group)` / `worker.addConfig(partialConfig)` — assign tests and config overrides to a spawned worker.
+- `bootstrapAll()` → `run()` → `teardownAll()` — lifecycle (wrap `run()` in `try/finally` so teardown always runs).
+- Events on the `workers` object: `event.test.passed`, `event.test.failed`, `event.all.result`, plus `'message'` for anything a child worker sends. `printResults()` prints the standard summary; `result.hasFailed()` and `result.stats` give the totals.
 
-Child workers can send non-test events to the main process. This is useful if you want to pass along information not related to the tests event cycles itself such as `event.test.success`.
+To split by your own rule, pass a function as `by` — it receives the worker count and returns an array of file groups:
 
 ```js
-// inside main process
-// listen for any non test related events
-workers.on('message', data => {
-  console.log(data)
-})
-
-workers.on(event.all.result, result => {
-  // logic
-})
-```
-
-## Sharing Data Between Workers
+const splitTests = () => [
+  ['./test/login_test.js', './test/signup_test.js'],  // group 1
+  ['./test/slow_checkout_test.js'],                    // group 2
+]
 
-NodeJS Workers can communicate between each other via messaging system. CodeceptJS allows you to share data between different worker processes using the `share()` and `inject()` functions.
+const workers = new Workers(2, { testConfig: './codecept.conf.js', by: splitTests })
+workers.on(event.all.result, () => workers.printResults())
+await workers.run()
+```
 
-### Basic Usage
+## Sharing data between workers
 
-You can share data directly using the `share()` function and access it using `inject()`:
+Worker threads don't share memory. `share()` publishes a value that any worker reads with `inject()`:
 
 ```js
-// In one test or worker
-share({ userData: { name: 'user', password: '123456' } })
+// in any test or hook
+share({ user: { name: 'jane', password: 's3cret' } })
 
-// In another test or worker
-const testData = inject()
-console.log(testData.userData.name) // 'user'
-console.log(testData.userData.password) // '123456'
+// anywhere else, even in another worker
+const { user } = inject()
 ```
 
-### Initializing Data in Bootstrap
-
-For complex scenarios where you need to initialize shared data before tests run, you can use the bootstrap function:
+Seed shared state before tests run from `bootstrap()`:
 
 ```js
-// inside codecept.conf.js
+// codecept.conf.js
 export const config = {
   bootstrap() {
-    // Initialize shared data container
-    share({ userData: null, config: { retries: 3 } })
+    share({ user: null })
   },
 }
 ```
 
-Then in your tests, you can check and update the shared data:
-
-```js
-const testData = inject()
-if (!testData.userData) {
-  // Update shared data - both approaches work:
-  share({ userData: { name: 'user', password: '123456' } })
-  // or mutate the injected object:
-  testData.userData = { name: 'user', password: '123456' }
-}
-```
-
-### Working with Proxy Objects
-
-Since CodeceptJS 3.7.0+, shared data uses Proxy objects for synchronization between workers. The proxy system works seamlessly for most use cases:
-
-```js
-// ✅ All of these work correctly:
-const data = inject()
-console.log(data.userData.name) // Access nested properties
-console.log(Object.keys(data)) // Enumerate shared keys
-data.newProperty = 'value' // Add new properties
-Object.assign(data, { more: 'data' }) // Merge objects
-```
-
-**Important Note:** Avoid reassigning the entire injected object:
-
-```js
-// ❌ AVOID: This breaks the proxy reference
-let testData = inject()
-testData = someOtherObject // This will NOT work as expected!
-
-// ✅ PREFERRED: Use share() to replace data or mutate properties
-share({ userData: someOtherObject }) // This works!
-// or
-Object.assign(inject(), someOtherObject) // This works!
-```
-
-### Local Data (Worker-Specific)
-
-If you want to share data only within the same worker (not across all workers), use the `local` option:
+Shared data is a Proxy. Don't reassign the injected object itself (`let d = inject(); d = {…}` breaks the link); mutate it or call `share()` again. Pass `{ local: true }` to keep a value inside one worker:
 
 ```js
-share({ localData: 'worker-specific' }, { local: true })
+share({ tmpFile: '/tmp/run-1' }, { local: true })
 ```