codeceptjs 3.2.2 → 3.3.0-beta.3

package/CHANGELOG.md

@@ -1,3 +1,60 @@
+## 3.3.0
+
+🛩️ Features:
+
+* [**API Testing introduced**](/api)
+  * Introduced [`JSONResponse`](/helpers/JSONResponse) helper which connects to REST, GraphQL or Playwright helper
+  * [REST] Added `amBearerAuthenticated` method
+  * [REST] Added `haveRequestHeaders` method
+  * Added dependency on `joi` and `chai`
+* [Playwright] Added `timeout` option to set [timeout](https://playwright.dev/docs/api/class-page#page-set-default-timeout) for all Playwright actions. If an action fails, Playwright keeps retrying it for a time set by timeout.
+* [Playwright] **Possible breaking change.** By default `timeout` is set to 1000ms. *Previous default was set by Playwright internally to 30s. This was causing contradiction to CodeceptJS retries, so triggered up to 3 retries for 30s of time. This timeout option was lowered so retryFailedStep plugin would not cause long delays.*
+* [Playwright] Extended methods to provide more options from engine. These methods were updated so additional options can be be passed as the last argument:
+  * [`click`](/helpers/Playwright#click)
+  * [`dragAndDrop`](/helpers/Playwright#dragAndDrop)
+  * [`checkOption`](/helpers/Playwright#checkOption)
+  * [`uncheckOption`](/helpers/Playwright#uncheckOption)
+
+```js
+// use Playwright click options as 3rd argument
+I.click('canvas', '.model', { position: { x: 20, y: 40 } })
+// check option also has options
+I.checkOption('Agree', '.signup', { position: { x: 5, y: 5 } })
+```
+
+* `eachElement` plugin introduced. It allows you to iterate over elements and perform some action on them using direct engines API
+
+```js
+await eachElement('click all links in .list', '.list a', (el) => {
+  await el.click();
+})
+```
+* [Playwright] Added support to `playwright-core` package if `playwright` is not installed. See #3190, fixes #2663.
+* [Playwright] Added `makeApiRequest` action to perform API requests. Requires Playwright >= 1.18
+* Added support to `codecept.config.js` for name consistency across other JS tools. See motivation at #3195 by @JiLiZART 
+* [ApiDataFactory] Added options arg to `have` method. See #3197 by @JJlokidoki
+* Improved pt-br translations to include keywords: 'Funcionalidade', 'Cenário', 'Antes', 'Depois', 'AntesDaSuite', 'DepoisDaSuite'. See #3206 by @danilolutz 
+* [allure plugin] Introduced `addStep` method to add comments and attachments. See #3104 by @EgorBodnar 
+
+🐛 Bugfixes:
+
+* Fixed #3212: using Regex flags for Cucumber steps. See #3214 by @anils92
+
+📖 Documentation
+
+* Added [Testomat.io reporter](/reporters#testomatio)
+* Added [api testing](/api) guides
+* Added [internal api](/internal-api) guides
+* [Appium] Fixed documentation for `performSwipe`
+* [Playwright] update docs for `usePlaywrightTo` method by @dbudzins 
+
+## 3.2.3
+
+* Documentation improvements by @maojunxyz
+* Guard mocha cli reporter from registering step logger multiple times #3180 by @nikocanvacom 
+* [Playwright] Fixed "tracing.stop: tracing.stop: ENAMETOOLONG: name too long" by @hatufacci
+* Fixed #2889: return always the same error contract from simplifyTest. See #3168 by @andremoah 
+
 ## 3.2.2
 
 * [Playwright] Reverted removal of retry on context errors. Fixes #3130

package/docs/advanced.md

@@ -97,10 +97,10 @@ Scenario('update user profile', ({  }) => {
 }).tag('@slow').tag('important');
 ```
 
-All tests with `@tag` could be executed with `--grep @tag` option.
+All tests with `@tag` could be executed with `--grep '@tag'` option.
 
 ```sh
-npx codeceptjs run --grep @slow
+codeceptjs run --grep '@slow'
 ```
 
 Use regex for more flexible filtering:

package/docs/api.md

@@ -1,265 +1,304 @@
 ---
-permalink: /internal-api
-title: Internal API
+permalink: /api
+title: API Testing
 ---
 
-## Concepts
+## API Testing
 
-In this guide we will overview the internal API of CodeceptJS.
-This knowledge is required for customization, writing plugins, etc.
+CodeceptJS provides a way to write tests in declarative manner for REST and GraphQL APIs. 
 
-CodeceptJS provides an API which can be loaded via `require('codeceptjs')` when CodeceptJS is installed locally. Otherwise, you can load codeceptjs API via global `codeceptjs` object:
+Take a look:
 
 ```js
-// via module
-const { recorder, event, output } = require('codeceptjs');
-// or using global object
-const { recorder, event, output } = codeceptjs;
+I.sendGetRequest('/users/1');
+// returns { "user": { "name": "jon" }, "projects": [] }
+I.seeResponseCodeIsSuccessful();
+I.seeResponseContainsKeys(['user', 'projects']);
+I.seeResponseContainsJson({ user: { name: 'jon' } });
+I.seeResponseMatchesJsonSchema($ => {
+  return $.object(
+    user: $.object({
+      name: $.string(),
+    }),
+    projects: $.array()
+  )
+});
 ```
+In this code we checked API request for:
 
-These internal objects are available:
+* status code
+* data inclusion
+* data structure
 
-* [`codecept`](https://github.com/Codeception/CodeceptJS/blob/master/lib/codecept.js): test runner class
-* [`config`](https://github.com/Codeception/CodeceptJS/blob/master/lib/config.js): current codecept config
-* [`event`](https://github.com/Codeception/CodeceptJS/blob/master/lib/event.js): event listener
-* [`recorder`](https://github.com/Codeception/CodeceptJS/blob/master/lib/recorder.js): global promise chain
-* [`output`](https://github.com/Codeception/CodeceptJS/blob/master/lib/output.js): internal printer
-* [`container`](https://github.com/Codeception/CodeceptJS/blob/master/lib/container.js): dependency injection container for tests, includes current helpers and support objects
-* [`helper`](https://github.com/Codeception/CodeceptJS/blob/master/lib/helper.js): basic helper class
-* [`actor`](https://github.com/Codeception/CodeceptJS/blob/master/lib/actor.js): basic actor (I) class
+These are the things you should generally test your APIs for.
 
-[API reference](https://github.com/Codeception/CodeceptJS/tree/master/docs/api) is available on GitHub.
-Also please check the source code of corresponding modules.
+> 🤓 It is recommended to check only invariable parts of responses. Check for required fields and only values you control. For instance, it is not recommended to check id fields, date fields, as they can be frequently changed.
 
-### Container
+## Installation
 
-CodeceptJS has a dependency injection container with helpers and support objects.
-They can be retrieved from the container:
+Install CodeceptJS if it is not installed yet.
 
-```js
-const { container } = require('codeceptjs');
-
-// get object with all helpers
-const helpers = container.helpers();
+```
+npm i codeceptjs --save-dev
+```
 
-// get helper by name
-const { WebDriver } = container.helpers();
+Initialize CodeceptJS and select REST or GraphQL helper when asked for a helper:
 
-// get support objects
-const supportObjects = container.support();
+```
+npx codeceptjs init
+```
 
-// get support object by name
-const { UserPage } = container.support();
+## Configuration
 
-// get all registered plugins
-const plugins = container.plugins();
-```
+Ensure that inside `codecept.conf.js` in helpers section `REST` or `GraphQL` helpers are enabled.
 
-New objects can also be added to container in runtime:
+* If you use `REST` helper add `JSONResponse` helper below with no extra config:
 
 ```js
-const { container } = require('codeceptjs');
+// inside codecept.conf.js
+// ...
+  helpers: {
+    REST: {
+      endpoint: 'http://localhost:3000/api'
+    },
+    // .. add JSONResponse helper here
+    JSONResponse: {}
+  }
+```
+* If you use `GraphQL` helper add `JSONResponse` helper, configuring it to use GraphQL for requests:
 
-container.append({
-  helpers: { // add helper
-    MyHelper: new MyHelper({ config1: 'val1' });
-  },
-  support: { // add page object
-    UserPage: require('./pages/user');
+```js
+  helpers: {
+    GraphQL: {
+      endpoint: 'http://localhost:3000/graphql'
+    },
+    // .. add JSONResponse helper here
+    JSONResponse: {
+      requestHelper: 'GraphQL',
+    }
   }
-})
 ```
 
-> Use this trick to define custom objects inside `boostrap` script
+Originally, REST and GraphQL helpers were not designed for API testing. 
+They were used to perform API requests for browser tests. As so, they lack assertion methods to API responses.
 
-The container also contains the current Mocha instance:
+[`JSONResponse`](/helpers/JSONResponse/) helper adds response assertions. 
 
-```js
-const mocha = container.mocha();
-```
+> 💡 In CodeceptJS assertions start with `see` prefix. Learn more about assertions by [opening reference for JSONResponse](/helpers/JSONResponse/) helper.
 
-### Event Listeners
+After helpers were configured, you can start writing first API test. By default, CodeceptJS saves tests in `tests` directory and uses `*_test.js` suffix. The `init` command created the first test for you to start.
 
-CodeceptJS provides a module with an [event dispatcher and set of predefined events](https://github.com/Codeception/CodeceptJS/blob/master/lib/event.js).
 
-It can be required from codeceptjs package if it is installed locally.
+## Requests
+
+[REST](/helpers/REST/) or [GraphQL](/helpers/GraphQL/) helpers implement methods for making API requests.
+Both helpers send requests via HTTP protocol from CodeceptJS process. 
+For most cases, you will need to have authentication. It can be passed via headers, which can be added to helper's configuration in `codecept.conf.js`. 
 
 ```js
-const { event } = require('codeceptjs');
+helpers: {
+  REST: {
+    defaultHeaders: {
+      // use Bearer Authorization
+      'Authorization': 'Bearer 11111',
+      'Content-Type': 'application/json',
+      'Accept': 'application/json',
+    },
+  }
+}
+```
 
-module.exports = function() {
+Or you can use the browser cookies if you are running browser session.
+In this case use `setSharedCookies()` from `@codeceptjs/configure` package:
 
-  event.dispatcher.on(event.test.before, function (test) {
+```js
+const { setSharedCookies } = require('@codeceptjs/configure');
 
-    console.log('--- I am before test --');
+// add this before exports.config
+setSharedCookies();
 
-  });
+exports.config = {
+  // ...
+  helpers: {  
+    // also works with Playwright or Puppeteer
+    WebDriver: {
+      //... 
+    },
+
+    REST: { 
+      // ...
+    }
+  }
 }
 ```
 
-Available events:
+### REST
 
-* `event.test.before(test)` - *async* when `Before` hooks from helpers and from test is executed
-* `event.test.after(test)` - *async* after each test
-* `event.test.started(test)` - *sync* at the very beginning of a test.
-* `event.test.passed(test)` - *sync* when test passed
-* `event.test.failed(test, error)` - *sync* when test failed
-* `event.test.finished(test)` - *sync* when test finished
-* `event.suite.before(suite)` - *async* before a suite
-* `event.suite.after(suite)` - *async* after a suite
-* `event.step.before(step)` - *async* when the step is scheduled for execution
-* `event.step.after(step)`- *async* after a step
-* `event.step.started(step)` - *sync* when step starts.
-* `event.step.passed(step)` - *sync* when step passed.
-* `event.step.failed(step, err)` - *sync* when step failed.
-* `event.step.finished(step)` - *sync* when step finishes.
-* `event.step.comment(step)` - *sync* fired for comments like `I.say`.
-* `event.all.before` - before running tests
-* `event.all.after` - after running tests
-* `event.all.result` - when results are printed
-* `event.workers.before` - before spawning workers in parallel run
-* `event.workers.after` - after workers finished in parallel run
+REST helper can send GET/POST/PATCH/etc requests to REST API endpoint:
 
+* [`I.sendGetRequest()`](/helpers/REST#sendGetRequest)
+* [`I.sendPostRequest()`](/helpers/REST#sendPostRequest)
+* [`I.sendPutRequest()`](/helpers/REST#sendPutRequest)
+* [`I.sendPatchRequest()`](/helpers/REST#sendPatchRequest)
+* [`I.sendDeleteRequest()`](/helpers/REST#sendDeleteRequest)
+* ...
 
-> *sync* - means that event is fired in the moment of the action happening.
- *async* - means that event is fired when an action is scheduled. Use `recorder` to schedule your actions.
+Authentication headers can be set in [helper's config](https://codecept.io/helpers/REST/#configuration) or per test with headers or special methods like `I.amBearerAuthenticated`.
 
-For further reference look for [currently available listeners](https://github.com/Codeception/CodeceptJS/tree/master/lib/listener) using the event system.
+Example:
 
+```js
+// this way we pass Bearer token
+I.amBearerAuthenticated(secret('token-is-here'));
+// for custom authorization with headers use
+// I.haveRequestHeaders method
+
+// here we send a POST request
+const response = await I.sendPostRequest('/users', {
+  name: 'joe',
+  email: 'joe@mail.com'
+});
+// usually we won't need direct access to response object for API testing 
+// but you can obtain it from request
+
+// check the last request was successful
+// this method introduced by JSONResponse helper
+I.seeResponseCodeIsSuccessful();
+```
 
-### Recorder
-
-To inject asynchronous functions in a test or before/after a test you can subscribe to corresponding event and register a function inside a recorder object. [Recorder](https://github.com/Codeception/CodeceptJS/blob/master/lib/recorder.js) represents a global promises chain.
+### GraphQL
 
-Provide a function in the first parameter, a function must be async or must return a promise:
+GraphQL have request format different then in REST API, but the response format is the same.
+It's plain old JSON. This why `JSONResponse` helper works for both API types.
+Configure authorization headers in `codecept.conf.js` and make your first query:  
 
 ```js
-const { event, recorder } = require('codeceptjs');
-
-module.exports = function() {
-
-  event.dispatcher.on(event.test.before, function (test) {
-
-    const request = require('request');
-
-    recorder.add('create fixture data via API', function() {
-      return new Promise((doneFn, errFn) => {
-        request({
-          baseUrl: 'http://api.site.com/',
-          method: 'POST',
-          url: '/users',
-          json: { name: 'john', email: 'john@john.com' }
-        }), (err, httpResponse, body) => {
-          if (err) return errFn(err);
-          doneFn();
-        }
-      });
-    }
-  });
-}
+// make GraphQL query or mutation
+const resp = await I.sendQuery('{ user(id: 0) { id name email }}');
+I.seeResponseCodeIsSuccessful();
+
+// GraphQL always returns key data as part of response
+I.seeResponseContainsKeys(['data']);
+
+// check data for partial inclusion
+I.seeResponseContainsJson({
+  data: {
+    user: {
+      name: 'john doe',
+      email: 'johnd@mutex.com',
+    },
+  },
+});
 ```
 
-### Config
+GraphQL helper has two methods available:
+
+* [`I.sendQuery()`](/helpers/GraphQL#sendQuery)
+* [`I.sendMutation()`](/helpers/GraphQL#sendMutation)
 
-CodeceptJS config can be accessed from `require('codeceptjs').config.get()`:
+## Assertions
+
+`JSONResponse` provides set of assertions for responses in JSON format. These assertions were designed to check only invariable parts of responses. So instead of checking that response equals to the one provided, we will check for data inclusion and structure matching.
+
+For most of cases, you won't need to perform assertions by accessing `response` object directly. All assretions are performed under hood inside `JSONResponse` module. It is recommended to keep it that way, to keep tests readable and make test log to contain all assertions.
 
 ```js
-const { config } = require('codeceptjs');
+Scenario('I make API call', ({ I }) => {
+  // request was made by REST 
+  // or by GraphQL helper
 
-// config object has access to all values of the current config file
+  // check that response code is 2xx
+  I.seeResponseCodeIsSuccessful();
 
-if (config.get().myKey == 'value') {
-  // run something
-}
+  // check that response contains keys
+  I.seeResponseContainsKeys(['data', 'pages', 'meta']);
+});
 ```
 
+### Response Status Codes
 
-### Output
+[Response status codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) can be checked to be equal to some value or to be in a specific range. 
+To check that response code is `200` call `I.seeResponseCodeIs`:
 
-Output module provides four verbosity levels. Depending on the mode you can have different information printed using corresponding functions.
+```js
+I.seeResponseCodeIs(200);
+```
+But because other response codes in 2xx range are also valid responses, you can use `seeResponseCodeIsSuccessful()` which will match 200 (OK), 201 (Created), 206 (Partial Content) and others. Methods to check 3xx, 4xx, 5xx response statuses also available.
 
-* `default`: prints basic information using `output.print`
-* `steps`: toggled by `--steps` option, prints step execution
-* `debug`: toggled by `--debug` option, prints steps, and debug information with `output.debug`
-* `verbose`: toggled by `--verbose` prints debug information and internal logs with `output.log`
+```js
+// matches 200, 201, 202, ... 206
+I.seeResponseCodeIsSuccessful();
 
-It is recommended to avoid `console.log` and use output.* methods for printing.
+// matches 300...308
+I.seeResponseCodeIsRedirection();
 
-```js
-const output = require('codeceptjs').output;
+// matches 400..451
+I.seeResponseCodeIsClientError();
 
-output.print('This is basic information');
-output.debug('This is debug information');
-output.log('This is verbose logging information');
+// matches 500-511
+I.seeResponseCodeIsServerError();
 ```
 
-#### Test Object
+### Structure
 
-The test events are providing a test object with following properties:
+The most basic thing to check in response is existence of keys in JSON object. Use [`I.seeResponseContainsKeys()`](/helpers/JSONResponse#seeResponseContainsKeys) method for it:
 
-* `title` title of the test
-* `body` test function as a string
-* `opts` additional test options like retries, and others
-* `pending` true if test is scheduled for execution and false if a test has finished
-* `tags` array of tags for this test
-* `artifacts` list of files attached to this test. Screenshots, videos and other files can be saved here and shared accross different reporters
-* `file` path to a file with a test
-* `steps` array of executed steps (available only in `test.passed`, `test.failed`, `test.finished` event)
-* `skipInfo` additional test options when test skipped 
-* * `message` string with reason for skip
-* * `description` string with test body
-and others
+```js
+// response is { "name": "joe", "email": "joe@joe.com" }
+I.seeResponseContainsKeys(['name', 'email']);
+```
 
-#### Step Object
+However, this is a very naive approach. It won't work for arrays or nested objects.
+To check complex JSON structures `JSONResponse` helper uses [`joi`](https://joi.dev) library. 
+It has rich API to validate JSON by the schema defined using JavaScript. 
 
-Step events provide step objects with following fields:
+```js
+// require joi library, 
+// it is installed with CodeceptJS
+const Joi = require('joi');
+
+// create schema definition using Joi API
+const schema = Joi.object().keys({
+  email: Joi.string().email().required(),
+  phone: Joi.string().regex(/^\d{3}-\d{3}-\d{4}$/).required(),
+  birthday: Joi.date().max('1-1-2004').iso()
+});
+
+// check that response matches that schema
+I.seeResponseMatchesJsonSchema(schema);
+```
 
-* `name` name of a step, like 'see', 'click', and others
-* `actor` current actor, in most cases it is `I`
-* `helper` current helper instance used to execute this step
-* `helperMethod` corresponding helper method, in most cases is the same as `name`
-* `status` status of a step (passed or failed)
-* `prefix` if a step is executed inside `within` block contain within text, like: 'Within .js-signup-form'.
-* `args` passed arguments
+### Data Inclusion
 
-Whenever you execute tests with `--verbose` option you will see registered events and promises executed by a recorder.
+To check that response contains expected data use `I.seeResponseContainsJson` method. 
+It will check the response data for partial match. 
 
-## Custom Runner
+```js
+I.seeResponseContainsJson({
+  user: {
+    email: 'user@user.com'
+  }
+})
+```
 
-You can run CodeceptJS tests from your script.
+To perform arbitrary assertions on a response object use `seeResponseValidByCallback`. 
+It allows you to do any kind of assertions by using `expect` from [`chai`](https://www.chaijs.com) library.
 
 ```js
-const { codecept: Codecept } = require('codeceptjs');
-
-// define main config
-const config = { 
-  helpers: { 
-    WebDriver: { 
-      browser: 'chrome', 
-      url: 'http://localhost' 
-    }
-  }
-};
-
-const opts = { steps: true };
-
-// run CodeceptJS inside async function
-(async () => {
-  const codecept = new Codecept(config, options);
-  codecept.init(__dirname);
-
-  try {
-    await codecept.bootstrap();
-    codecept.loadTests('**_test.js');
-    // run tests
-    await codecept.run(test);
-  } catch (err) {
-    printError(err);
-    process.exitCode = 1;
-  } finally {
-    await codecept.teardown();
-  }    
-})();
+I.seeResponseValidByCallback({ data, status, expect } => {
+  // we receive data and expect to combine them for good assertion
+  expect(data.users.length).to.be.gte(10);
+})
 ```
 
-> Also, you can run tests inside workers in a custom scripts. Please refer to the [parallel execution](/parallel) guide for more details.
+## Extending JSONResponse
+
+To add more assertions it is recommended to create a custom helper.
+Inside it you can get access to latest JSON response:
+
+```js
+// inside a custom helper
+makeSomeCustomAssertion() {
+  const response = this.helpers.JSONResponse.response;
+}
+```

package/docs/build/ApiDataFactory.js

@@ -257,13 +257,16 @@ class ApiDataFactory extends Helper {
    * // create user with defined email
    * // and receive it when inside async function
    * const user = await I.have('user', { email: 'user@user.com'});
+   * // create a user with options that will not be included in the final request
+   * I.have('user', { }, { age: 33, height: 55 })
    * ```
    *
    * @param {*} factory factory to use
    * @param {*} params predefined parameters
+   * @param {*} options options for programmatically generate the attributes
    */
-  have(factory, params) {
-    const item = this._createItem(factory, params);
+  have(factory, params, options) {
+    const item = this._createItem(factory, params, options);
     this.debug(`Creating ${factory} ${JSON.stringify(item)}`);
     return this._requestCreate(factory, item);
   }
@@ -277,21 +280,25 @@ class ApiDataFactory extends Helper {
    *
    * // create 3 posts by one author
    * I.haveMultiple('post', 3, { author: 'davert' });
+   *
+   * // create 3 posts by one author with options
+   * I.haveMultiple('post', 3, { author: 'davert' }, { publish_date: '01.01.1997' });
    * ```
    *
    * @param {*} factory
    * @param {*} times
    * @param {*} params
+   * @param {*} options
    */
-  haveMultiple(factory, times, params) {
+  haveMultiple(factory, times, params, options) {
     const promises = [];
     for (let i = 0; i < times; i++) {
-      promises.push(this.have(factory, params));
+      promises.push(this.have(factory, params, options));
     }
     return Promise.all(promises);
   }
 
-  _createItem(model, data) {
+  _createItem(model, data, options) {
     if (!this.factories[model]) {
       throw new Error(`Factory ${model} is not defined in config`);
     }
@@ -303,7 +310,7 @@ class ApiDataFactory extends Helper {
         modulePath = path.join(global.codecept_dir, modulePath);
       }
       const builder = require(modulePath);
-      return builder.build(data);
+      return builder.build(data, options);
     } catch (err) {
       throw new Error(`Couldn't load factory file from ${modulePath}, check that