codeceptjs 3.2.1 → 3.2.2

package/CHANGELOG.md

@@ -1,3 +1,13 @@
+## 3.2.2
+
+* [Playwright] Reverted removal of retry on context errors. Fixes #3130
+* Timeout improvements by @nikocanvacom:
+  * Added priorites to timeouts
+  * Added `overrideStepLimits` to [stepTimeout plugin](https://codecept.io/plugins/#steptimeout) to override steps timeouts set by `limitTime`.
+  * Fixed step timeout not working due to override by NaN by test timeout #3126
+* [Appium] Fixed logging error when `manualStart` is true. See #3140 by @nikocanvacom
+
+
 ## 3.2.1
 
 > ♻️ This release fixes hanging of tests by reducing timeouts for automatic retries on failures.

package/docs/advanced.md

@@ -186,7 +186,7 @@ You can use this options for build your own [plugins](https://codecept.io/hooks/
   });
 ```
 
-### Timeout <Badge text="Updated in 3.2" type="warning"/>
+## Timeout <Badge text="Updated in 3.2" type="warning"/>
 
 Tests can get stuck due to various reasons such as network connection issues, crashed browser, etc.
 This can make tests process hang. To prevent these situations timeouts can be used. Timeouts can be set explicitly for flaky parts of code, or implicitly in a config.

package/docs/basics.md

@@ -41,7 +41,6 @@ Refer to following guides to more information on:
 * [▶ Playwright](/playwright)
 * [▶ WebDriver](/webdriver)
 * [▶ Puppeteer](/puppeteer)
-* [▶ Protractor](/angular)
 * [▶ Nightmare](/nightmare)
 * [▶ TestCafe](/testcafe)
 
@@ -317,6 +316,32 @@ var assert = require('assert');
 assert.equal(title, 'CodeceptJS');
 ```
 
+It is important to understand the usage of **async** functions in CodeceptJS. While non-returning actions can be called without await, if an async function uses `grab*` action it must be called with `await`:
+
+```js
+// a helper function
+async function getAllUsers(I) {
+   const users = await I.grabTextFrom('.users');
+   return users.filter(u => u.includes('active'))
+}
+
+// a test
+Scenario('try helper functions', async ({ I }) => {
+  // we call function with await because it includes `grab`
+  const users = await getAllUsers(I);
+});
+```
+
+If you miss `await` you get commands unsynchrhonized. And this will result to an error like this:
+
+```
+(node:446390) UnhandledPromiseRejectionWarning: ...
+    at processTicksAndRejections (internal/process/task_queues.js:95:5)
+(node:446390) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
+```
+
+If you face that error please make sure that all async functions are called with `await`.
+
 ## Running Tests
 
 To launch tests use the `run` command, and to execute tests in [multiple browsers](/advanced/#multiple-browsers-execution) or [multiple threads](/advanced/#parallel-execution) use the `run-multiple` command.

package/docs/bdd.md

@@ -5,7 +5,7 @@ title: Behavior Driven Development
 
 # Behavior Driven Development
 
-Behavior Driven Development (BDD) is a popular software development methodology. BDD is considered an extension of TDD, and is greatly inspired by [Agile](http://agilemanifesto.org/) practices. The primary reason to choose BDD as your development process is to break down communication barriers between business and technical teams. BDD encourages the use of automated testing to verify all documented features of a project from the very beginning. This is why it is common to talk about BDD in the context of test frameworks (like CodeceptJS). The BDD approach, however, is about much more than testing - it is a common language for all team members to use during the development process.
+Behavior Driven Development (BDD) is a popular software development methodology. BDD is considered an extension of TDD, and is greatly inspired by [Agile](https://agilemanifesto.org/) practices. The primary reason to choose BDD as your development process is to break down communication barriers between business and technical teams. BDD encourages the use of automated testing to verify all documented features of a project from the very beginning. This is why it is common to talk about BDD in the context of test frameworks (like CodeceptJS). The BDD approach, however, is about much more than testing - it is a common language for all team members to use during the development process.
 
 ## What is Behavior Driven Development
 
@@ -55,7 +55,7 @@ I should see that total number of products I want to buy is 2
 And my order amount is $1600
 ```
 
-As we can see this simple story highlights core concepts that are called *contracts*. We should fulfill those contracts to model software correctly. But how we can verify that those contracts are being satisfied? [Cucumber](http://cucumber.io) introduced a special language for such stories called **Gherkin**. Same story transformed to Gherkin will look like this:
+As we can see this simple story highlights core concepts that are called *contracts*. We should fulfill those contracts to model software correctly. But how we can verify that those contracts are being satisfied? [Cucumber](https://cucumber.io) introduced a special language for such stories called **Gherkin**. Same story transformed to Gherkin will look like this:
 
 ```gherkin
 Feature: checkout process

package/docs/build/Playwright.js

@@ -355,6 +355,17 @@ class Playwright extends Helper {
   }
 
   async _before() {
+    recorder.retry({
+      retries: 5,
+      when: err => {
+        if (!err || typeof (err.message) !== 'string') {
+          return false;
+        }
+        // ignore context errors
+        return err.message.includes('context');
+      },
+    });
+
     if (this.options.restart && !this.options.manualStart) await this._startBrowser();
     if (!this.isRunning && !this.options.manualStart) await this._startBrowser();
 

package/docs/build/WebDriver.js

@@ -582,7 +582,7 @@ class WebDriver extends Helper {
     this.context = this.root;
     if (this.options.restart && !this.options.manualStart) return this._startBrowser();
     if (!this.isRunning && !this.options.manualStart) return this._startBrowser();
-    this.$$ = this.browser.$$.bind(this.browser);
+    if (this.browser) this.$$ = this.browser.$$.bind(this.browser);
     return this.browser;
   }
 

package/docs/changelog.md

@@ -7,6 +7,16 @@ layout: Section
 
 # Releases
 
+## 3.2.2
+
+* **[Playwright]** Reverted removal of retry on context errors. Fixes [#3130](https://github.com/codeceptjs/CodeceptJS/issues/3130)
+* Timeout improvements by **[nikocanvacom](https://github.com/nikocanvacom)**:
+  * Added priorites to timeouts
+  * Added `overrideStepLimits` to [stepTimeout plugin](https://codecept.io/plugins/#steptimeout) to override steps timeouts set by `limitTime`.
+  * Fixed step timeout not working due to override by NaN by test timeout [#3126](https://github.com/codeceptjs/CodeceptJS/issues/3126)
+* **[Appium]** Fixed logging error when `manualStart` is true. See [#3140](https://github.com/codeceptjs/CodeceptJS/issues/3140) by **[nikocanvacom](https://github.com/nikocanvacom)**
+
+
 ## 3.2.1
 
 > ♻️ This release fixes hanging of tests by reducing timeouts for automatic retries on failures.

package/docs/pageobjects.md

@@ -191,6 +191,8 @@ module.exports = new AttachFile();
 module.exports.AttachFile = AttachFile;
 ```
 
+> ⚠ While building complex page objects it is important to keep all `async` functions to be called with `await`. While CodeceptJS allows to run commands synchronously if async function has `I.grab*` or any custom function that returns a promise it must be called with `await`. If you see `UnhandledPromiseRejectionWarning` it might be caused by async page object function that was called without `await`.
+
 ## Page Fragments
 
 Similarly, CodeceptJS allows you to generate **PageFragments** and any other abstractions

package/docs/plugins.md

@@ -898,7 +898,7 @@ Run tests with plugin enabled:
 #### Configuration:
 
 -   `timeout` - global step timeout, default 150 seconds
--   `force` - whether to use timeouts set in plugin config to override step timeouts set in code with I.limitTime(x).action(...), default false
+-   `overrideStepLimits` - whether to use timeouts set in plugin config to override step timeouts set in code with I.limitTime(x).action(...), default false
 -   `noTimeoutSteps` - an array of steps with no timeout. Default:
 
     -   `amOnPage`
@@ -915,7 +915,7 @@ Run tests with plugin enabled:
 plugins: {
     stepTimeout: {
         enabled: true,
-        force: true,
+        overrideStepLimits: true,
         noTimeoutSteps: [
           'scroll*', // ignore all scroll steps
           /Cookie/, // ignore all steps with a Cookie in it (by regexp)

package/lib/actor.js

@@ -38,7 +38,7 @@ class Actor {
 
     event.dispatcher.prependOnceListener(event.step.before, (step) => {
       output.log(`Timeout to ${step}: ${timeout}s`);
-      step.totalTimeout = timeout * 1000;
+      step.setTimeout(timeout * 1000, Step.TIMEOUT_ORDER.codeLimitTime);
     });
 
     return this;
@@ -132,7 +132,7 @@ function recordStep(step, args) {
       step.startTime = Date.now();
     }
     return val = step.run(...args);
-  }, false, undefined, step.totalTimeout);
+  }, false, undefined, step.getTimeout());
 
   event.emit(event.step.after, step);
 

package/lib/helper/Playwright.js

@@ -355,6 +355,17 @@ class Playwright extends Helper {
   }
 
   async _before() {
+    recorder.retry({
+      retries: 5,
+      when: err => {
+        if (!err || typeof (err.message) !== 'string') {
+          return false;
+        }
+        // ignore context errors
+        return err.message.includes('context');
+      },
+    });
+
     if (this.options.restart && !this.options.manualStart) await this._startBrowser();
     if (!this.isRunning && !this.options.manualStart) await this._startBrowser();
 

package/lib/helper/WebDriver.js

@@ -582,7 +582,7 @@ class WebDriver extends Helper {
     this.context = this.root;
     if (this.options.restart && !this.options.manualStart) return this._startBrowser();
     if (!this.isRunning && !this.options.manualStart) return this._startBrowser();
-    this.$$ = this.browser.$$.bind(this.browser);
+    if (this.browser) this.$$ = this.browser.$$.bind(this.browser);
     return this.browser;
   }
 

package/lib/listener/timeout.js

@@ -3,6 +3,7 @@ const output = require('../output');
 const recorder = require('../recorder');
 const Config = require('../config');
 const { timeouts } = require('../store');
+const TIMEOUT_ORDER = require('../step').TIMEOUT_ORDER;
 
 module.exports = function () {
   let timeout;
@@ -49,14 +50,14 @@ module.exports = function () {
     if (typeof timeout !== 'number') return;
 
     if (timeout < 0) {
-      step.totalTimeout = 0.01;
+      step.setTimeout(0.01, TIMEOUT_ORDER.testOrSuite);
     } else {
-      step.totalTimeout = timeout;
+      step.setTimeout(timeout, TIMEOUT_ORDER.testOrSuite);
     }
   });
 
   event.dispatcher.on(event.step.finished, (step) => {
-    timeout -= step.duration;
+    if (typeof timeout === 'number' && !Number.isNaN(timeout)) timeout -= step.duration;
 
     if (typeof timeout === 'number' && timeout <= 0 && recorder.isRunning()) {
       if (currentTest && currentTest.callback) {

package/lib/plugin/stepTimeout.js

@@ -1,8 +1,9 @@
 const event = require('../event');
+const TIMEOUT_ORDER = require('../step').TIMEOUT_ORDER;
 
 const defaultConfig = {
   timeout: 150,
-  force: false,
+  overrideStepLimits: false,
   noTimeoutSteps: [
     'amOnPage',
     'wait*',
@@ -33,7 +34,7 @@ const defaultConfig = {
  * #### Configuration:
  *
  * * `timeout` - global step timeout, default 150 seconds
- * * `force` - whether to use timeouts set in plugin config to override step timeouts set in code with I.limitTime(x).action(...), default false
+ * * `overrideStepLimits` - whether to use timeouts set in plugin config to override step timeouts set in code with I.limitTime(x).action(...), default false
  * * `noTimeoutSteps` - an array of steps with no timeout. Default:
  *     * `amOnPage`
  *     * `wait*`
@@ -49,7 +50,7 @@ const defaultConfig = {
  * plugins: {
  *     stepTimeout: {
  *         enabled: true,
- *         force: true,
+ *         overrideStepLimits: true,
  *         noTimeoutSteps: [
  *           'scroll*', // ignore all scroll steps
  *           /Cookie/, // ignore all steps with a Cookie in it (by regexp)
@@ -85,6 +86,6 @@ module.exports = (config) => {
       }
     }
     stepTimeout = stepTimeout === undefined ? config.timeout : stepTimeout;
-    step.totalTimeout = stepTimeout * 1000;
+    step.setTimeout(stepTimeout * 1000, config.overrideStepLimits ? TIMEOUT_ORDER.stepTimeoutHard : TIMEOUT_ORDER.stepTimeoutSoft);
   });
 };

package/lib/recorder.js

@@ -178,7 +178,8 @@ module.exports = {
     if (process.env.DEBUG) debug(`${currentQueue()}Queued | ${taskName}`);
 
     return promise = Promise.resolve(promise).then((res) => {
-      const retryOpts = this.retries.slice(-1).pop();
+      // prefer options for non-conditional retries
+      const retryOpts = this.retries.sort((r1, r2) => r1.when && !r2.when).slice(-1).pop();
       // no retries or unnamed tasks
       if (!retryOpts || !taskName || !retry) {
         const [promise, timer] = getTimeoutPromise(timeout, taskName);

package/lib/step.js

@@ -13,6 +13,27 @@ const STACK_LINE = 4;
  * @param {string} name
  */
 class Step {
+  static get TIMEOUT_ORDER() {
+    return {
+      /**
+       * timeouts set with order below zero only override timeouts of higher order if their value is smaller
+       */
+      testOrSuite: -5,
+      /**
+       * 0-9 - designated for override of timeouts set from code, 5 is used by stepTimeout plugin when stepTimeout.config.overrideStepLimits=true
+       */
+      stepTimeoutHard: 5,
+      /**
+       * 10-19 - designated for timeouts set from code, 15 is order of I.setTimeout(t) operation
+       */
+      codeLimitTime: 15,
+      /**
+       * 20-29 - designated for timeout settings which could be overriden in tests code, 25 is used by stepTimeout plugin when stepTimeout.config.overrideStepLimits=false
+       */
+      stepTimeoutSoft: 25,
+    };
+  }
+
   constructor(helper, name) {
     /** @member {string} */
     this.actor = 'I'; // I = actor
@@ -38,8 +59,40 @@ class Step {
     this.metaStep = undefined;
     /** @member {string} */
     this.stack = '';
-    /** @member {number} */
-    this.totalTimeout = undefined;
+
+    const timeouts = new Map();
+    /**
+     * @method
+     * @returns {number|undefined}
+     */
+    this.getTimeout = function () {
+      let totalTimeout;
+      // iterate over all timeouts starting from highest values of order
+      new Map([...timeouts.entries()].sort().reverse()).forEach((timeout, order) => {
+        if (timeout !== undefined && (
+          // when orders >= 0 - timeout value overrides those set with higher order elements
+          order >= 0
+
+          // when `order < 0 && totalTimeout === undefined` - timeout is used when nothing is set by elements with higher order
+          || totalTimeout === undefined
+
+          // when `order < 0` - timeout overrides higher values of timeout or 'no timeout' (totalTimeout === 0) set by elements with higher order
+          || timeout > 0 && (timeout < totalTimeout || totalTimeout === 0)
+        )) {
+          totalTimeout = timeout;
+        }
+      });
+      return totalTimeout;
+    };
+    /**
+     * @method
+     * @param {number} timeout - timeout in milliseconds or 0 if no timeout
+     * @param {number} order - order defines the priority of timeout, timeouts set with lower order override those set with higher order.
+     *                         When order below 0 value of timeout only override if new value is lower
+     */
+    this.setTimeout = function (timeout, order) {
+      timeouts.set(order, timeout);
+    };
 
     this.setTrace();
   }
@@ -231,6 +284,8 @@ class MetaStep extends Step {
   }
 }
 
+Step.TIMEOUTS = {};
+
 /** @type {Class<MetaStep>} */
 Step.MetaStep = MetaStep;
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "codeceptjs",
-  "version": "3.2.1",
+  "version": "3.2.2",
   "description": "Supercharged End 2 End Testing Framework for NodeJS",
   "keywords": [
     "acceptance",

package/typings/types.d.ts

@@ -10311,7 +10311,13 @@ declare namespace CodeceptJS {
         args: any[];
         metaStep: MetaStep;
         stack: string;
-        totalTimeout: number;
+        getTimeout(): number | undefined;
+        /**
+         * @param timeout - timeout in milliseconds or 0 if no timeout
+         * @param order - order defines the priority of timeout, timeouts set with lower order override those set with higher order.
+         *                         When order below 0 value of timeout only override if new value is lower
+         */
+        setTimeout(timeout: number, order: number): void;
         setTrace(): void;
         setArguments(args: any[]): void;
         run(...args: any[]): any;
@@ -10889,3 +10895,23 @@ declare namespace CodeceptJS {
     }
 }
 
+/**
+ * timeouts set with order below zero only override timeouts of higher order if their value is smaller
+ */
+declare var testOrSuite: any;
+
+/**
+ * 0-9 - designated for override of timeouts set from code, 5 is used by stepTimeout plugin when stepTimeout.config.overrideStepLimits=true
+ */
+declare var stepTimeoutHard: any;
+
+/**
+ * 10-19 - designated for timeouts set from code, 15 is order of I.setTimeout(t) operation
+ */
+declare var codeLimitTime: any;
+
+/**
+ * 20-29 - designated for timeout settings which could be overriden in tests code, 25 is used by stepTimeout plugin when stepTimeout.config.overrideStepLimits=false
+ */
+declare var stepTimeoutSoft: any;
+