codeceptjs 3.5.12-beta.1 → 3.5.12-beta.2

package/docs/advanced.md

@@ -0,0 +1,351 @@
+---
+permalink: /advanced
+title: Advanced Usage
+---
+
+# Advanced Usage
+
+## Data Driven Tests
+
+Execute the same scenario on a different data set.
+
+Let's say you want to test login for different user accounts.
+In this case, you need to create a datatable and fill it in with credentials.
+Then use `Data().Scenario` to include this data and generate multiple scenarios:
+
+```js
+// Define data table inside a test or load from another module
+let accounts = new DataTable(['login', 'password']); //
+accounts.add(['davert', '123456']); // adding records to a table
+accounts.add(['admin', '123456']);
+
+// You can skip some data. But add them to report as skipped (just like with usual scenarios):
+accounts.xadd(['admin', '23456'])
+
+// Pass dataTable to Data()
+// Use special param `current` to get current data set
+Data(accounts).Scenario('Test Login', ({ I, current }) => {
+  I.fillField('Username', current.login); // current is reserved!
+  I.fillField('Password', current.password);
+  I.click('Sign In');
+  I.see('Welcome '+ current.login);
+});
+
+
+// Also you can set only for Data tests. It will launch executes only the current test but with all data options
+Data(accounts).only.Scenario('Test Login', ({ I, current }) => {
+  I.fillField('Username', current.login); // current is reserved!
+  I.fillField('Password', current.password);
+  I.click('Sign In');
+  I.see('Welcome '+ current.login);
+});
+```
+
+*Important: you can't use name `current` for pageObjects or helpers in data scenarios*
+
+This will produce 2 tests with different data sets.
+Current data set is appended to a test name in output:
+
+```sh
+✓ Test Login | {"login":"davert","password":"123456"}
+✓ Test Login | {"login":"admin","password":"123456"}
+S Test Login | {"login":"admin","password":"23456"}
+```
+
+```js
+// You can filter your data table
+Data(accounts.filter(account => account.login == 'admin')
+.Scenario('Test Login', ({ I, current }) => {
+  I.fillField('Username', current.login);
+  I.fillField('Password', current.password);
+  I.click('Sign In');
+  I.see('Welcome '+ current.login);
+});
+```
+
+This will limit data sets accoring passed function:
+
+```sh
+✓ Test Login | {"login":"admin","password":"123456"}
+S Test Login | {"login":"admin","password":"23456"}
+```
+
+Data sets can also be defined with array, generator, or a function.
+
+```js
+Data(function*() {
+  yield { user: 'davert'};
+  yield { user: 'andrey'};
+}).Scenario() // ...
+```
+
+*HINT: If you don't use DataTable. add `toString()` method to each object added to data set, so the data could be pretty printed in a test name*
+
+## Tags
+
+Append `@tag` to your test name, so
+
+```js
+Scenario('update user profile @slow')
+```
+
+Alternativly, use `tag` method of Scenario to set additional tags:
+
+```js
+Scenario('update user profile', ({  }) => {
+  // test goes here
+}).tag('@slow').tag('important');
+```
+
+All tests with `@tag` could be executed with `--grep '@tag'` option.
+
+```sh
+codeceptjs run --grep '@slow'
+```
+
+Use regex for more flexible filtering:
+
+* `--grep '(?=.*@smoke2)(?=.*@smoke3)'` - run tests with @smoke2 and @smoke3 in name
+* `--grep "\@smoke2|\@smoke3"` - run tests with @smoke2 or @smoke3 in name
+* `--grep '((?=.*@smoke2)(?=.*@smoke3))|@smoke4'` - run tests with (@smoke2 and @smoke3) or @smoke4 in name
+* `--grep '(?=.*@smoke2)^(?!.*@smoke3)'` - run tests with @smoke2 but without @smoke3 in name
+* `--grep '(?=.*)^(?!.*@smoke4)'` - run all tests except @smoke4
+
+
+
+## Debug
+
+CodeceptJS provides a debug mode in which additional information is printed.
+It can be turned on with `--debug` flag.
+
+```sh
+npx codeceptjs run --debug
+```
+
+to receive even more information turn on `--verbose` flag:
+
+```sh
+npx codeceptjs run --verbose
+```
+
+> You can pause execution and enter **interactive console** mode by calling `pause()` inside your test.
+
+To see a complete internal debug of CodeceptJS use `DEBUG` env variable:
+
+```sh
+DEBUG=codeceptjs:* npx codeceptjs run
+```
+
+For an interactive debugging use NodeJS debugger. In **WebStorm**:
+
+```sh
+node $NODE_DEBUG_OPTION ./node_modules/.bin/codeceptjs run
+```
+
+For **Visual Studio Code**, add the following configuration in launch.json:
+
+```json
+{
+  "type": "node",
+  "request": "launch",
+  "name": "codeceptjs",
+  "args": ["run", "--grep", "@your_test_tag"],
+  "program": "${workspaceFolder}/node_modules/codeceptjs/bin/codecept.js"
+}
+```
+
+
+## Test Options
+
+Features and Scenarios have their options that can be set by passing a hash after their names:
+
+```js
+Feature('My feature', {key: val});
+
+Scenario('My scenario', {key: val},({ I }) => {});
+```
+
+You can use this options for build your own [plugins](https://codecept.io/hooks/#plugins) with [event listners](https://codecept.io/hooks/#api). Example: 
+
+```js
+  // for test
+  event.dispatcher.on(event.test.before, (test) => {
+    ...
+    if (test.opts.key) {
+      ...
+    }
+    ...
+  });
+  // or for suite
+  event.dispatcher.on(event.suite.before, (suite) => {
+    ...
+    if (suite.opts.key) {
+      ...
+    }
+    ...
+  });
+```
+
+## Timeout
+
+Tests can get stuck due to various reasons such as network connection issues, crashed browser, etc.
+This can make tests process hang. To prevent these situations timeouts can be used. Timeouts can be set explicitly for flaky parts of code, or implicitly in a config.
+
+> Previous timeout implementation was disabled as it had no effect when dealing with steps and promises. 
+
+### Steps Timeout
+
+It is possible to limit a step execution to specified time with `I.limitTime` command. 
+It will set timeout in seconds for the next executed step:
+
+```js
+// limit clicking to 5 seconds
+I.limitTime(5).click('Link')
+```
+
+It is possible to set a timeout for all steps implicitly (except waiters) using [stepTimeout plugin](/plugins/#steptimeout).
+
+### Tests Timeout
+
+Test timeout can be set in seconds via Scenario options:
+
+```js
+// limit test to 20 seconds
+Scenario('slow test that should be stopped', { timeout: 20 }, ({ I }) => {
+  // ...
+})
+```
+
+This timeout can be set globally in `codecept.conf.js` in seconds:
+
+```js
+exports.config = {
+
+  // each test must not run longer than 5 mins
+  timeout: 300,
+
+}
+```
+
+### Suites Timeout
+
+A timeout for a group of tests can be set on Feature level via options. 
+
+```js
+// limit all tests in this suite to 30 seconds
+Feature('flaky tests', { timeout: 30 })
+```
+
+### Timeout Confguration 
+
+<Badge text="Updated in 3.4" type="warning"/>
+
+Timeout rules can be set globally via config.
+
+To set a timeout for all running tests provide a **number of seconds** to `timeout` config option:
+
+
+```js
+// inside codecept.conf.js or codecept.conf.ts
+timeout: 30, // limit all tests in all suites to 30 secs
+```
+
+It is possible to tune this configuration for a different groups of tests passing options as array and using `grep` option to filter tests:
+
+```js
+// inside codecept.conf.js or codecept.conf.ts
+
+timeout: [
+  10, // default timeout is 10secs  
+
+  // but increase timeout for slow tests
+  {
+    grep: '@slow',
+    Feature: 50
+  },
+]
+```
+
+> ℹ️ `grep` value can be string or regexp
+
+It is possible to set a timeout for Scenario or Feature:
+
+```js
+// inside codecept.conf.js or codecept.conf.ts
+timeout: [
+
+  // timeout for Feature with @slow in title
+  {
+    grep: '@slow',
+    Feature: 50
+  },
+  
+  // timeout for Scenario with 'flaky0' .. `flaky1` in title
+  {
+    // regexp can be passed to grep
+    grep: /flaky[0-9]/,
+    Scenario: 10
+  },
+
+  // timeout for all suites
+  {    
+    Feature: 20
+  }
+]
+```
+
+Global timeouts will be overridden by explicit timeouts of a test or steps.
+
+### Disable Timeouts
+
+To execute tests ignoring all timeout settings use `--no-timeouts` option:
+
+```
+npx codeceptjs run --no-timeouts
+```
+
+## Dynamic Configuration
+
+Helpers can be reconfigured per scenario or per feature.
+This might be useful when some tests should be executed with different settings than others.
+In order to reconfigure tests use `.config()` method of `Scenario` or `Feature`.
+
+```js
+Scenario('should be executed in firefox', ({ I }) => {
+  // I.amOnPage(..)
+}).config({ browser: 'firefox' })
+```
+
+In this case `config` overrides current config of the first helper.
+To change config of specific helper pass two arguments: helper name and config values:
+
+```js
+Scenario('should create data via v2 version of API', ({ I }) => {
+  // I.amOnPage(..)
+}).config('REST', { endpoint: 'https://api.mysite.com/v2' })
+```
+
+Config can also be set by a function, in this case you can get a test object and specify config values based on it.
+This is very useful when running tests against cloud providers, like BrowserStack. This function can also be asynchronous.
+
+```js
+Scenario('should report to BrowserStack', ({ I }) => {
+  // I.amOnPage(..)
+}).config((test) => {
+  return { desiredCapabilities: {
+    project: test.suite.title,
+    name: test.title,
+  }}
+});
+```
+
+Config changes can be applied to all tests in suite:
+
+```js
+Feature('Admin Panel').config({ url: 'https://mysite.com/admin' });
+```
+
+Please note that some config changes can't be applied on the fly. For instance, if you set `restart: false` in your config and then changing value `browser` won't take an effect as browser is already started and won't be closed untill all tests finish.
+
+Configuration changes will be reverted after a test or a suite.
+

package/docs/ai.md

@@ -0,0 +1,248 @@
+---
+permalink: /ai
+title: Testing with AI 🪄 
+---
+
+# 🪄 Testing with AI
+
+**CodeceptJS is the first open-source test automation framework with AI** features to improve the testing experience. CodeceptJS uses OpenAI GPT to auto-heal failing tests, assist in writing tests, and more...
+
+Think of it as your testing co-pilot built into the testing framework
+
+> 🪄 **AI features for testing are experimental**. AI works only for web based testing with Playwright, WebDriver, etc. Those features will be improved based on user's experience.
+
+
+## How AI Improves Automated Testing
+
+ChatGPT can write automated tests for you. What you need is just ask it "How to write CodeceptJS test" and it will generate the code that can be executed but not the actual code you need. ChatGPT misses the context, of your application. 
+
+CodeceptJS uses OpenAI API to send the prompt and share the HTML context of a page. So, it can ask: how to write a test for **this** page. GPT model knows how to write CodeceptJS code, how to build good-looking semantic locators and how to analyze HTML to match them. Even more, GPT suggestions can be tested in real-time in a browser, making a feedback loop.
+
+CodeceptJS AI can do the following:
+
+* 🏋️‍♀️ **assist writing tests** in `pause()` or interactive shell mode
+* 🚑 **self-heal failing tests** (can be used on CI)
+* 💬 send arbitrary prompts to GPT from any tested page attaching its HTML contents
+
+![](/img/fill_form.gif)
+
+### How it works
+
+As we can't send a browser window with ChatGPT we are not be able to fully share the context. As only text information is accepted, we can send HTML of the currently tested page. GPT doesn't know what elements are hidden, or what elements are visible. It can operate only on the HTML provided.
+
+GPT models have limits on information passed, and HTML pages can be huge. And some information may be irrelevant for testing. For instance, if you test a reading application, you won't probably need text contents of a book inside your HTML, as they won't be used in locators. That's why CodeceptJS send HTML with **all non-interactive HTML elements removed**. So, only links, buttons, fields, and all elements that are set as a link, button, etc will be used by GPT for analysis. In case you have clickable `<div>` but with no `role="button"` it will be ignored. Also, we minify HTML before sending.
+
+Even though, the HTML is still quite big and may exceed the token limit. So we recommend using **gpt-3.5-turbo-16k** model, as it accepts 16K tokens (approx. 50K of HTML text), which should be enough for most web pages. It is possible to strictly limit the size of HTML to not exceed GPT tokens limit.
+
+> ❗AI features require sending HTML contents to OpenAI. If you use it in enterprise, ensure that your company requirements match [OpenAI complience](https://openai.com/security). If you work on public web applications this should be ok, as HTML code is genrally available to all web application users.
+
+
+### Getting Started
+
+[Install CodeceptJS 3.5](/installation):
+
+```
+npx create-codeceptjs .
+```
+[Obtain API token](https://platform.openai.com/account/api-keys) from OpenAI. Please check out [their pricing](https://openai.com/pricing) first, as unlike ChatGPT using OpenAI API requires a paid account.
+
+Add `OPENAI_API_KEY` environment variable with a key when running codeceptjs:
+
+```
+OPENAI_API_KEY=sk-******** npx codeceptjs run
+```
+
+This will enable AI features in CodeceptJS.
+
+> When running on CI set `OPENAI_API_KEY` as a secured environment variable
+
+### Writing Tests with AI Copilot
+
+If AI features are enabled when using [interactive pause](/basics/#debug) with `pause()` command inside tests:
+
+For instance, let's create a test to try ai features via `gt` command:
+
+```
+npx codeceptjs gt
+```
+
+Name a test and write the code. We will use `Scenario.only` instead of Scenario to execute only this exact test.
+
+```js
+Feature('ai');
+
+Scenario.only('test ai features', ({ I }) => {
+  I.amOnPage('https://getbootstrap.com/docs/5.1/examples/checkout/')
+  pause();
+});
+```
+
+Now run the test in debug mode:
+
+```
+OPENAI_API_KEY=sk-******** npx codeceptjs run --debug
+```
+
+When pause mode started you can ask GPT to fill in the fields on this page. Use natural language to describe your request, and provide enough details that AI could operate with it. It is important to include at least a space char in your input, otherwise, CodeceptJS will consider the input to be JavaScript code.
+
+
+```
+ I.fill checkout form with valid values without submitting it
+```
+
+![](/img/fill_form_1.png)
+
+GPT will generate code and data and CodeceptJS will try to execute its code. If it succeeds, the code will be saved to history and you will be able to copy it to your test.
+
+![](/img/fill_form2.png)
+
+This AI copilot works best with long static forms. In the case of complex and dynamic single-page applications, it may not perform as well, as the form may not be present on HTML page yet. For instance, interacting with calendars or inputs with real-time validations (like credit cards) can not yet be performed by AI.
+
+Please keep in mind that GPT can't react to page changes and operates with static text only. This is why it is not ready yet to write the test completely. However, if you are new to CodeceptJS and automated testing AI copilot may help you write tests more efficiently. 
+
+> 👶 Enable AI copilot for junior test automation engineers. It may help them to get started with CodeceptJS and to write good semantic locators.
+
+### Self-Healing Tests
+
+In large test suites, the cost of maintaining tests goes exponentially. That's why any effort that can improve the stability of tests pays itself. In CodeceptJS 3.5 we introduced a new [heal plugin](/plugins#heal) that will use AI to automatically fix a failing test.
+
+Heal plugin can solve exactly one problem: if a locator of an element has changed, and an action can't be performed, **it matches a new locator, tries a command again, and continues executing a test**. For instance, if the "Sign in" button was renamed to "Login" or changed its class, it will detect a new locator of the button and will retry execution.
+
+Heal actions **work only on actions like `click`, `fillField`**, etc, and won't work on assertions, waiters, grabbers, etc. Assertions can't be guessed by AI, the same way as grabbers, as this may lead to unpredictable results.
+
+If Heal plugin successfully fixes the step, it will print a suggested change at the end of execution. Take it as actionable advice and use it to update the codebase. Heal plugin is supposed to be used on CI, and works automatically without human assistance.
+
+To start, enable `heal` plugin in `codecept.conf.js` or `codecept.conf.ts`:
+
+```js
+plugins: {
+  heal: {
+    enabled: true
+  }
+}
+```
+
+and run tests in AI mode with `OPENAI_API_KEY` provided:
+
+```
+OPENAI_API_KEY=sk-******** npx codeceptjs run
+```
+
+![](/img/heal.png)
+
+
+### Arbitrary GPT Prompts
+
+What if you want to take ChatGPT on the journey of test automation and ask it questions while browsing pages?
+
+This is possible with the new `OpenAI` helper. Enable it in your config and it will automatically attach to Playwright, WebDriver, or another web helper you use. It includes the following methods:
+
+* `askGptOnPage` - sends GPT prompt attaching the HTML of the page. Large pages will be split into chunks, according to `chunkSize` config. You will receive responses for all chunks.
+* `askGptOnPageFragment` - sends GPT prompt attaching the HTML of the specific element. This method is recommended over `askGptOnPage` as you can reduce the amount of data to be processed.
+* `askGptGeneralPrompt` - sends GPT prompt without HTML.
+
+OpenAI helper won't remove non-interactive elements, so it is recommended to manually control the size of the sent HTML.
+
+Here are some good use cases for this helper:
+
+* get page summaries
+* inside pause mode navigate through your application and ask to document pages
+* etc...
+
+```js
+// use it inside test or inside interactive pause
+// pretend you are technical writer asking for documentation
+const pageDoc = await I.askGptOnPageFragment('Act as technical writer, describe what is this page for', '#container');
+```
+
+As of now, those use cases do not apply to test automation but maybe you can apply them to your testing setup. 
+
+## Configuration
+
+AI features can be configured inside `codecept.conf` file under `ai` section:
+
+```js
+ai: {
+  model: 'gpt-3.5-turbo-16k',
+  temperature: 0.1,
+  html: // {}
+}
+```
+
+Available options are:
+
+* `model` - [OpenAI model](https://platform.openai.com/docs/models), `gpt-3.5-turbo-16k` is recommended. You may switch to another GPT model, however, consider the speed of processing and size of the input. Models with less than 16K tokens won't be able to process complete HTML even reduced to interactive elements.
+* `temperature` - [temperature](https://platform.openai.com/docs/api-reference/chat/create#chat/create-temperature) is a measure of randomness. Use the lowest value possible for test automation purposes.
+* `html` - configures how HTML is processed before sending it to GPT. This section is highly important to tune to adapt to your application. For instance, the default strategy may remove some important elements, or contrary keep some elements that have no practical usage in test automation.
+
+Here is the default config:
+
+```js
+ai: {
+  html: {
+    maxLength: 50000,
+    simplify: true,
+    minify: true,
+    interactiveElements: ['a', 'input', 'button', 'select', 'textarea', 'option'],
+    textElements: ['label', 'h1', 'h2'],
+    allowedAttrs: ['id', 'for', 'class', 'name', 'type', 'value', 'tabindex', 'aria-labelledby', 'aria-label', 'label', 'placeholder', 'title', 'alt', 'src', 'role'],
+    allowedRoles: ['button', 'checkbox', 'search', 'textbox', 'tab'],
+  }
+}
+```
+
+* `maxLength`: the size of HTML to cut to not reach the token limit. 50K is the current default but you may try to increase it or even set it to null.
+* `simplify`: should we process HTML before sending to GPT. This will remove all non-interactive elements from HTML.
+* `minify`: shold HTML be additionally minified. This removed empty attributes, shortens notations, etc.
+* `interactiveElements`: explicit list of all elements that are considered interactive.
+* `textElements`: elements that contain text which can be used for test automation.
+* `allowedAttrs`: explicit list of attributes that may be used to construct locators. If you use special `data-` attributes to enable locators, add them to the list.
+* `allowedRoles`: list of roles that make standard elements interactive.
+
+It is recommended to try HTML processing on one of your web pages before launching AI features of CodeceptJS.
+
+
+To do that open the common page of your application and using DevTools copy the outerHTML of `<html>` element. Don't use `Page Source` for that, as it may not include dynamically added HTML elements. Save this HTML into a file and create a NodeJS script:
+
+```js
+const { removeNonInteractiveElements } = require('codeceptjs/lib/html');
+const fs = require('fs');
+
+const htmlOpts = {
+  interactiveElements: ['a', 'input', 'button', 'select', 'textarea', 'label', 'option'],
+  allowedAttrs: ['id', 'for', 'class', 'name', 'type', 'value', 'aria-labelledby', 'aria-label', 'label', 'placeholder', 'title', 'alt', 'src', 'role'],
+  textElements: ['label', 'h1', 'h2'],
+  allowedRoles: ['button', 'checkbox', 'search', 'textbox', 'tab'],
+};
+
+html = fs.readFileSync('saved.html', 'utf8');
+const result = removeNonInteractiveElements(html, htmlOpts);
+
+console.log(result);
+```
+
+Tune the options until you are satisfied with the results and use this as `html` config for `ai` section inside `codecept.conf` file.
+It is also recommended to check the source of [removeNonInteractiveElements](https://github.com/codeceptjs/CodeceptJS/blob/3.x/lib/html.js) and if needed propose improvements to it.
+
+For instance, if you use `data-qa` attributes to specify locators and you want to include them in HTML, use the following config:
+
+```js
+{
+  // inside codecept.conf.js
+  ai: {
+    html: {
+      allowedAttrs: [
+        'data-qa', 'id', 'for', 'class', 'name', 'type', 'value', 'aria-labelledby', 'aria-label', 'label', 'placeholder', 'title', 'alt', 'src', 'role'
+      ]
+    }
+  }
+}
+```
+
+## Debugging
+
+To debug AI features run tests with `DEBUG="codeceptjs:ai"` flag. This will print all prompts and responses from OpenAI
+
+```
+DEBUG="codeceptjs:ai" OPENAI_API_KEY=sk-******** npx codeceptjs run
+```