codeceptjs 3.1.3 → 3.2.3

package/CHANGELOG.md

@@ -1,3 +1,76 @@
+## 3.2.3
+
+* Documentation improvements by @maojunxyz
+* Guard mocha cli reporter from registering step logger multiple times #3180 by @nikocanvacom 
+* [Playwright] Fixed "tracing.stop: tracing.stop: ENAMETOOLONG: name too long" by @hatufacci
+* Fixed #2889: return always the same error contract from simplifyTest. See #3168 by @andremoah 
+
+## 3.2.2
+
+* [Playwright] Reverted removal of retry on context errors. Fixes #3130
+* Timeout improvements by @nikocanvacom:
+  * Added priorites to timeouts
+  * Added `overrideStepLimits` to [stepTimeout plugin](https://codecept.io/plugins/#steptimeout) to override steps timeouts set by `limitTime`.
+  * Fixed step timeout not working due to override by NaN by test timeout #3126
+* [Appium] Fixed logging error when `manualStart` is true. See #3140 by @nikocanvacom
+
+
+## 3.2.1
+
+> ♻️ This release fixes hanging of tests by reducing timeouts for automatic retries on failures.
+
+* [retryFailedStep plugin] **New Defaults**: retries steps up to 3 times with factor of 1.5 (previously 5 with factor 2)
+* [Playwright] - disabled retry on failed context actions (not needed anymore)
+* [Puppeteer] - reduced retries on context failures to 3 times.
+* [Playwright] Handling `crash` event to automatically close crashed pages.
+
+## 3.2.0
+
+🛩️ Features:
+
+**[Timeouts](https://codecept.io/advanced/#timeout) implemented**
+  * global timeouts (via `timeout` config option). 
+    * _Breaking change:_ timeout option expects **timeout in seconds**, not in milliseconds as it was previously.
+  * test timeouts (via `Scenario` and `Feature` options)
+    * _Breaking change:_ `Feature().timeout()` and `Scenario().timeout()` calls has no effect and are deprecated
+
+```js
+// set timeout for every test in suite to 10 secs
+Feature('tests with timeout', { timeout: 10 });
+
+// set timeout for this test to 20 secs
+Scenario('a test with timeout', { timeout: 20 }, ({ I }) => {});
+```  
+
+  * step timeouts (See #3059 by @nikocanvacom)
+
+```js
+// set step timeout to 5 secs
+I.limitTime(5).click('Link');
+```  
+ * `stepTimeout` plugin introduced to automatically add timeouts for each step (#3059 by @nikocanvacom).
+
+[**retryTo**](/plugins/#retryto) plugin introduced to rerun a set of steps on failure:
+
+```js
+// editing in text in iframe
+// if iframe was not loaded - retry 5 times
+await retryTo(() => {
+  I.switchTo('#editor frame');
+  I.fillField('textarea', 'value');
+}, 5);
+```
+
+* [Playwright] added `locale` configuration
+* [WebDriver] upgraded to webdriverio v7
+
+🐛 Bugfixes:
+
+* Fixed  allure plugin "Unexpected endStep()" error in #3098 by @abhimanyupandian 
+* [Puppeteer] always close remote browser on test end. See #3054 by @mattonem
+* stepbyStepReport Plugin: Disabled screenshots after test has failed. See #3119 by @ioannisChalkias
+
+
 ## 3.1.3
 
 🛩️ Features:
@@ -15,7 +88,7 @@ Given('I have an employee card', (table) => {
   // rows = [['Harry', 'Potter', Seeker]];
   }
 ```
-See updated [BDD section](https://codecept.io/bdd/) for more API options.
+See updated [BDD section](https://codecept.io/bdd/) for more API options. Thanks to @EgorBodnar
 
 * Support `cjs` file extensions for config file: `codecept.conf.cjs`. See #3052 by @kalvenschraut
 * API updates: Added `test.file` and `suite.file` properties to `test` and `suite` objects to use in helpers and plugins. 
@@ -28,6 +101,7 @@ See updated [BDD section](https://codecept.io/bdd/) for more API options.
 * [Playwright] `I.haveRequestHeaders` affects all tabs. See #3049 by @jancorvus
 * BDD: Fixed unhandled empty feature files. Fix #3046 by @abhimanyupandian 
 * Fixed `RangeError: Invalid string length` in `recorder.js` when running huge amount of tests.  
+* [Appium] Fixed definitions for `touchPerform`, `hideDeviceKeyboard`, `removeApp` by @mirao 
 
 📖 Documentation:
 

package/README.md

@@ -29,7 +29,7 @@ Scenario('check Welcome page on site', ({ I }) => {
 
 CodeceptJS tests are:
 
-* **Synchronous**. You don't need to care about callbacks, or promises, test scenarios are linear, your test should be too.
+* **Synchronous**. You don't need to care about callbacks or promises or test scenarios which are linear. But, your tests should be linear.
 * Written from **user's perspective**. Every action is a method of `I`. That makes test easy to read, write and maintain even for non-tech persons.
 * Backend **API agnostic**. We don't know which WebDriver implementation is running this test. We can easily switch from WebDriverIO to Protractor or PhantomJS.
 
@@ -38,11 +38,10 @@ CodeceptJS uses **Helper** modules to provide actions to `I` object. Currently C
 * [**Playwright**](https://github.com/codeceptjs/CodeceptJS/blob/master/docs/helpers/Playwright.md) - is a Node library to automate the Chromium, WebKit and Firefox browsers with a single API.
 * [**Puppeteer**](https://github.com/codeceptjs/CodeceptJS/blob/master/docs/helpers/Puppeteer.md) - uses Google Chrome's Puppeteer for fast headless testing.
 * [**WebDriver**](https://github.com/codeceptjs/CodeceptJS/blob/master/docs/helpers/WebDriver.md) - uses [webdriverio](http://webdriver.io/) to run tests via WebDriver protocol.
-* [**Protractor**](https://github.com/codeceptjs/CodeceptJS/blob/master/docs/helpers/Protractor.md) - helper empowered by [Protractor](http://protractortest.org/) to run tests via WebDriver protocol.
 * [**TestCafe**](https://github.com/codeceptjs/CodeceptJS/blob/master/docs/helpers/TestCafe.md) - cheap and fast cross-browser test automation.
 * [**Nightmare**](https://github.com/codeceptjs/CodeceptJS/blob/master/docs/helpers/Nightmare.md) - uses Electron and NightmareJS to run tests.
 * [**Appium**](https://github.com/codeceptjs/CodeceptJS/blob/master/docs/helpers/Appium.md) - for **mobile testing** with Appium
-* [**Detox**](https://github.com/codeceptjs/CodeceptJS/blob/master/docs/helpers/Detox.md) - This is a wrapper on top of Detox library, aimied to unify testing experience for CodeceptJS framework. Detox provides a grey box testing for mobile applications, playing especially good for React Native apps.
+* [**Detox**](https://github.com/codeceptjs/CodeceptJS/blob/master/docs/helpers/Detox.md) - This is a wrapper on top of Detox library, aimed to unify testing experience for CodeceptJS framework. Detox provides a grey box testing for mobile applications, playing especially well for React Native apps.
 
 And more to come...
 

package/bin/codecept.js

@@ -105,6 +105,7 @@ program.command('run [test]')
   .option('-c, --config [file]', 'configuration file to be used')
   .option('--features', 'run only *.feature files and skip tests')
   .option('--tests', 'run only JS test files and skip features')
+  .option('--no-timeouts', 'disable all timeouts')
   .option('-p, --plugins <k=v,k2=v2,...>', 'enable plugins, comma-separated')
 
   // mocha options

package/docs/advanced.md

@@ -97,10 +97,14 @@ Scenario('update user profile', ({  }) => {
 }).tag('@slow').tag('important');
 ```
 
-All tests with `@tag` could be executed with `--grep @tag` option.
+All tests with `@tag` could be executed with `--grep '@tag'` option.
 
 ```sh
-codeceptjs run --grep @slow
+<<<<<<< HEAD
+npx codeceptjs run --grep @slow
+=======
+codeceptjs run --grep '@slow'
+>>>>>>> 435964dff7770cb3cd58c78a8579b6221a000d24
 ```
 
 Use regex for more flexible filtering:
@@ -119,24 +123,30 @@ CodeceptJS provides a debug mode in which additional information is printed.
 It can be turned on with `--debug` flag.
 
 ```sh
-codeceptjs run --debug
+npx codeceptjs run --debug
 ```
 
 to receive even more information turn on `--verbose` flag:
 
 ```sh
-codeceptjs run --verbose
+npx codeceptjs run --verbose
 ```
 
-And don't forget that you can pause execution and enter **interactive console** mode by calling `pause()` inside your test.
+> You can pause execution and enter **interactive console** mode by calling `pause()` inside your test.
 
-For advanced debugging use NodeJS debugger. In WebStorm IDE:
+To see a complete internal debug of CodeceptJS use `DEBUG` env variable:
+
+```sh
+DEBUG=codeceptjs:* npx codeceptjs run
+```
+
+For an interactive debugging use NodeJS debugger. In **WebStorm**:
 
 ```sh
 node $NODE_DEBUG_OPTION ./node_modules/.bin/codeceptjs run
 ```
 
-For Visual Studio Code, add the following configuration in launch.json:
+For **Visual Studio Code**, add the following configuration in launch.json:
 
 ```json
 {
@@ -180,29 +190,99 @@ You can use this options for build your own [plugins](https://codecept.io/hooks/
   });
 ```
 
-### Timeout
+## Timeout <Badge text="Updated in 3.2" type="warning"/>
 
-By default there is no timeout for tests, however you can change this value for a specific suite:
+Tests can get stuck due to various reasons such as network connection issues, crashed browser, etc.
+This can make tests process hang. To prevent these situations timeouts can be used. Timeouts can be set explicitly for flaky parts of code, or implicitly in a config.
+
+> Previous timeout implementation was disabled as it had no effect when dealing with steps and promises. 
+
+### Steps Timeout
+
+It is possible to limit a step execution to specified time with `I.limitTime` command. 
+It will set timeout in seconds for the next executed step:
 
 ```js
-Feature('Stop me').timeout(5000); // set timeout to 5s
+// limit clicking to 5 seconds
+I.limitTime(5).click('Link')
 ```
 
-or for the test:
+It is possible to set a timeout for all steps implicitly (except waiters) using [stepTimeout plugin](/plugins/#steptimeout).
+
+### Tests Timeout
+
+Test timeout can be set in seconds via Scenario options:
 
 ```js
-// set timeout to 1s
-Scenario("Stop me faster",({ I }) => {
-  // test goes here
-}).timeout(1000);
+// limit test to 20 seconds
+Scenario('slow test that should be stopped', { timeout: 20 }, ({ I }) => {
+  // ...
+})
+```
 
-// alternative
-Scenario("Stop me faster", {timeout: 1000},({ I }) => {});
+This timeout can be set globally in `codecept.conf.js` in seconds:
 
-// disable timeout for this scenario
-Scenario("Don't stop me", {timeout: 0},({ I }) => {});
+```js
+exports.config = {
+
+  // each test must not run longer than 5 mins
+  timeout: 300,
+
+}
 ```
 
+### Suites Timeout
+
+A timeout for a group of tests can be set on Feature level via options. 
+
+```js
+// limit all tests in this suite to 30 seconds
+Feature('flaky tests', { timeout: 30 })
+```
+
+### Sum Up
+
+Let's list all available timeout options.
+
+Timeouts can be set globally in config:
+
+```js
+// in codecept.confg.js:
+{ // ...
+   timeout: 30, // limit all tests in all suites to 30 secs
+
+   plugins: {
+     stepTimeout: {
+       enabled: true,
+       timeout: 10, // limit all steps except waiters to 10 secs
+     }
+   }
+} 
+
+```
+
+or inside a test file:
+
+```js
+// limit all tests in this suite to 10 secs
+Feature('tests with timeout', { timeout: 10 });
+
+// limit this test to 20 secs
+Scenario('a test with timeout', { timeout: 20 }, ({ I }) => {
+   // limit step to 5 seconds
+   I.limitTime(5).click('Link');
+});
+```
+
+Global timeouts will be overridden by explicit timeouts of a test or steps.
+
+### Disable Timeouts
+
+To execute tests ignoring all timeout settings use `--no-timeouts` option:
+
+```
+npx codeceptjs run --no-timeouts
+```
 
 ## Dynamic Configuration
 
@@ -249,45 +329,3 @@ Please note that some config changes can't be applied on the fly. For instance,
 
 Configuration changes will be reverted after a test or a suite.
 
-
-### Rerunning Flaky Tests Multiple Times <Badge text="Since 2.4" type="warning"/>
-
-End to end tests can be flaky for various reasons. Even when we can't do anything to solve this problem it we can do next two things:
-
-* Detect flaky tests in our suite
-* Fix flaky tests by rerunning them.
-
-Both tasks can be achieved with [`run-rerun` command](/commands/#run-rerun) which runs tests multiple times until all tests are passed.
-
-You should set min and max runs boundaries so when few tests fail in a row you can rerun them until they are succeeded.
-
-```js
-// inside to codecept.conf.js
-exports.config = { // ...
-  rerun: {
-    // run 4 times until 1st success
-    minSuccess: 1,
-    maxReruns: 4,
-  }
-}
-```
-
-If you want to check all your tests for stability you can set high boundaries for minimal success:
-
-```js
-// inside to codecept.conf.js
-exports.config = { // ...
-  rerun: {
-    // run all tests must pass exactly 5 times
-    minSuccess: 5,
-    maxReruns: 5,
-  }
-}
-```
-
-Now execute tests with `run-rerun` command:
-
-```
-npx codeceptjs run-rerun
-```
-

package/docs/basics.md

@@ -41,7 +41,6 @@ Refer to following guides to more information on:
 * [▶ Playwright](/playwright)
 * [▶ WebDriver](/webdriver)
 * [▶ Puppeteer](/puppeteer)
-* [▶ Protractor](/angular)
 * [▶ Nightmare](/nightmare)
 * [▶ TestCafe](/testcafe)
 
@@ -108,7 +107,7 @@ I.seeElement({name: 'password'});
 I.seeElement({react: 'user-profile', props: {name: 'davert'}});
 ```
 
-In [mobile testing](http://codecept.io/mobile/#locating-elements) you can use `~` to specify the accessibility id to locate an element. In web application you can locate elements by their `aria-label` value.
+In [mobile testing](https://codecept.io/mobile/#locating-elements) you can use `~` to specify the accessibility id to locate an element. In web application you can locate elements by their `aria-label` value.
 
 ```js
 // locate element by [aria-label] attribute in web
@@ -317,6 +316,32 @@ var assert = require('assert');
 assert.equal(title, 'CodeceptJS');
 ```
 
+It is important to understand the usage of **async** functions in CodeceptJS. While non-returning actions can be called without await, if an async function uses `grab*` action it must be called with `await`:
+
+```js
+// a helper function
+async function getAllUsers(I) {
+   const users = await I.grabTextFrom('.users');
+   return users.filter(u => u.includes('active'))
+}
+
+// a test
+Scenario('try helper functions', async ({ I }) => {
+  // we call function with await because it includes `grab`
+  const users = await getAllUsers(I);
+});
+```
+
+If you miss `await` you get commands unsynchrhonized. And this will result to an error like this:
+
+```
+(node:446390) UnhandledPromiseRejectionWarning: ...
+    at processTicksAndRejections (internal/process/task_queues.js:95:5)
+(node:446390) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
+```
+
+If you face that error please make sure that all async functions are called with `await`.
+
 ## Running Tests
 
 To launch tests use the `run` command, and to execute tests in [multiple browsers](/advanced/#multiple-browsers-execution) or [multiple threads](/advanced/#parallel-execution) use the `run-multiple` command.

package/docs/bdd.md

@@ -5,7 +5,7 @@ title: Behavior Driven Development
 
 # Behavior Driven Development
 
-Behavior Driven Development (BDD) is a popular software development methodology. BDD is considered an extension of TDD, and is greatly inspired by [Agile](http://agilemanifesto.org/) practices. The primary reason to choose BDD as your development process is to break down communication barriers between business and technical teams. BDD encourages the use of automated testing to verify all documented features of a project from the very beginning. This is why it is common to talk about BDD in the context of test frameworks (like CodeceptJS). The BDD approach, however, is about much more than testing - it is a common language for all team members to use during the development process.
+Behavior Driven Development (BDD) is a popular software development methodology. BDD is considered an extension of TDD, and is greatly inspired by [Agile](https://agilemanifesto.org/) practices. The primary reason to choose BDD as your development process is to break down communication barriers between business and technical teams. BDD encourages the use of automated testing to verify all documented features of a project from the very beginning. This is why it is common to talk about BDD in the context of test frameworks (like CodeceptJS). The BDD approach, however, is about much more than testing - it is a common language for all team members to use during the development process.
 
 ## What is Behavior Driven Development
 
@@ -55,7 +55,7 @@ I should see that total number of products I want to buy is 2
 And my order amount is $1600
 ```
 
-As we can see this simple story highlights core concepts that are called *contracts*. We should fulfill those contracts to model software correctly. But how we can verify that those contracts are being satisfied? [Cucumber](http://cucumber.io) introduced a special language for such stories called **Gherkin**. Same story transformed to Gherkin will look like this:
+As we can see this simple story highlights core concepts that are called *contracts*. We should fulfill those contracts to model software correctly. But how we can verify that those contracts are being satisfied? [Cucumber](https://cucumber.io) introduced a special language for such stories called **Gherkin**. Same story transformed to Gherkin will look like this:
 
 ```gherkin
 Feature: checkout process