codeceptjs 4.0.2-beta.8 → 4.0.2

package/docs/heal.md

@@ -0,0 +1,213 @@
+# Self-Healing Tests
+
+Browser and Mobile tests can fail for vareity of reasons. However, on a big projects there are about 5-10 causes of flaky tests. The more you work and understand your end-to-end tests the more you learn patterns of failure. And after the research you understand how a test could have been fixed: to reload a page, to click that button once again, restart API request. If by looking into a failure you understand what, as a user, you would do to fix that error, then maybe you could teach your tests to heal themselves.
+
+## What is Healing
+
+**Healing defines the way how a test reacts to failure**. You can define multiple healing recipes that could take all needed information: error message, failed test, step, page URL, HTML, etc. A healing recipe can perform some action to fix the failing test on the fly and continue its execution.
+
+![](/img/healing.png)
+
+Let's start with a common scenario. If a user suddenly becomes unauthorized and is moved to a sign-in page, or receives an unauthorized message on the page, we can heal by navigating to the `/login` page and trying to enter credentials again.
+
+```js
+heal.addRecipe('loginOnUnauthorized', {
+  priority: 10,
+  steps: ['click', 'see', 'amOnPage'],
+  prepare: {
+    url: ({ I }) => I.grabCurrentUrl(),
+    html: ({ I }) => I.grabHTMLFrom('body'),
+  },
+  fn: async ({ url, error, step, html }) => {
+    if (!url.includes('/login') && !error.message.toLowerCase().includes('unauthorized') && !html.toLowerCase().includes('unauthorized')) return;
+    
+    return ({ I }) => {
+      I.amOnPage('/login');
+      I.fillField('Email', 'test@example.com');
+      I.fillField('Password', '123456');
+      I.click('Sign in');
+      I[step.name](...step.args);
+    };
+  },
+});
+```
+
+Another example is a very basic healing recipe. If after a click test has failed, try to reload page, and continue.
+
+```js
+heal.addRecipe('reload', {
+  priority: 10,
+  steps: ['click'],
+  fn: async ({ step }) => {
+    return ({ I }) => {
+      I.refreshPage();
+      I[step.name](...step.args);
+    };
+  },
+});
+```
+
+Sure, this won't always work and probably won't be useful on every project. But let's follow the idea: if a click has failed, probably the button is not on a page, maybe it is an issue of rendering, maybe some other element overlapped our button, so if we try to reload page we can continue test execution. At least, this is what manual QA would do if they will run the following test in a browser. They will try to reload a page before reporting "it has failed".
+
+So if it is a long end-2-end test that implements user journey, it is more valuable to continue its execution when possible, then fixing a minor issues like overlapping elements. Healing like this can improve the stability of a test.
+
+The example above is only one way a test can be healed. But you can define as many heal recipes as you like. What heal recipe would be effective in your case is depends on a system you test, so **there are no pre-defined heal recipes**. 
+
+## Healing Patterns
+
+There are some ideas where healing can be useful to you:
+
+* **Authorization**. If a user suddenly becomes unauthorized and is moved to a sign-in page, or receives an unauthorized message on the page, we can heal by navigating to the `/login` page and trying to enter credentials.
+* **Networking**. If a test depends on a remote resource, and fails because this resource is not available, you may try to send API request to restore that resource before throwing an error.
+* **Data Consistency**. A test may fail because you noticed the data glitch in a system. Instead of failing a test you may try to clean up the data and try again to proceed.
+* **UI Change**. If there is a planned UI migration of a component, for instance Button was changed to Dropdown. You can prepare test so if it fails clicking Button it can try to do so with Dropdown.
+* **Do it again**. If you know, that going one step back and trying to do same actions may solve the issue, you can do so from healers. For instance, a modal didn't render correctly, so you can close it and try to click to open it again.
+
+## Healing vs Retries
+
+Unlike retries heal recipes has following benefits:
+
+* Heal recipes are **declarative**, they are not added directly into into the test code. This keeps test clean and scenario-focused,
+* Retry can only re-run failed step(s), but heal recipe can **perform wide set of actions**
+* Heal recipe **can react to any step of any test**. So if you catch a common error and you can heal it, you won't need to guess where it can be thrown.
+
+## How to Start Healing
+
+To enable healing, you need to define healing recipes and enable heal plugin.
+
+Create basic healing recipes using this command:
+
+```
+npx codeceptjs generate:heal
+```
+
+or
+
+```
+npx codeceptjs gr
+```
+
+this will generate `recipes.js` (or `recipes.ts`) in the root directory. Provided default recipe include [AI healing](#ai-healing) and `clickAndType` recipe that replaces `fillField` with `click`+`type`. Use them as examples to write your own heal recipes that will fit for application you are testing.
+
+Require `recipes` file and add `heal` plugin to `codecept.conf` file:
+
+```js
+
+import './heal';
+
+export const config = {
+  // ...
+  plugins: {
+    heal: {
+      enabled: true
+    }
+  }
+}
+```
+
+> Please note that, healing has no sense while developing tests, so it won't work in `--debug` mode. 
+
+## Writing Recipes
+
+Custom heal recipes can be added by running `heal.addRecipe()` function. By default it should be added to `recipes.js` (or `recipes.ts`) file. 
+
+Let's see what recipe consist of:
+
+```js
+heal.addRecipe('reloadPageOnUserAccount', {
+  // recipe priority
+  // which recipe should be tried first 
+  priority: 10,
+
+  // an array of steps which may cause the error
+  // after which a recipe should be activate
+  steps: [
+    'click',
+  ],
+
+  // if you need some additional information like URL of a page,
+  // or its HTML, you can add this context to healing function by 
+  // defining `prepare` list of variable
+  prepare: {
+    url: ({ I }) => I.grabCurrentUrl(),
+    html: ({ I }) => I.grabHTMLFrom('body'),
+    // don't add variables that you won't use inside the recipe
+  },
+
+  // probably we want to execute recipes only on some tests
+  // so you can set a string or regex which will check if a test title matches the name
+  // in this case we execute recipe only on tests that have "@flaky" in their name
+  grep: '@flaky',
+
+  // function to launch healing process 
+  fn: async ({ 
+    // standard context variables
+    step, test, error, prevSteps,
+
+    // variables coming from prepare function
+    html, url,
+    
+    }) => {
+    const stepArgs = step.args;
+
+    // at this point we can decide, should we provide a healing recipe or not
+    // for instance, if URL is not the one we can heal at, we should not provide any recipes
+    if (!url.includes('/user/acccount')) return;
+  
+    // otherwise we return a function that will be executed
+    return ({ I }) => {
+      // this is a very basic example action
+      // probably you should do something more sophisticated
+      // to heal the test
+      I.reloadPage();
+      I.wait(1);
+      I[step.name](...stepArgs);
+    };
+  },
+});
+```
+
+Let's briefly sum up the properties of a recipe:
+
+* `grep` - selects tests by their name to apply heal to
+* `steps` - defines on which steps a recipe should react
+* `priority` - sets the order of recipes being applied
+* `prepare` - declare variables from a context, which can be used for healing
+* `fn` - a function to be applied for healing. It takes all context params: `test`, `step`, `error`, `prevSteps` and returns return either a function or a markdown text with recipes (used by AI healers). If no recipes match the context should not return anything;
+
+
+## AI Healing
+
+AI can be used to heal failed tests. Large Language Models can analyze HTML of a failed test and provide a suggestion what actions should be performed instead. This can be helpful when running tests on CI as AI can make basic decisions to stabilize failing tests. 
+
+> Use **OpenAI, Azure OpenAI, Claude**, or any of other LLM that can take a prompt, analyze request and provide valid JS code which can be executed by CodeceptJS as a healing suggestion.
+
+AI healing recipe is created within `recipes.js` file:
+
+```js
+heal.addRecipe('ai', {
+  priority: 10,
+  prepare: {
+    html: ({ I }) => I.grabHTMLFrom('body'),
+  },
+  steps: [
+    'click',
+    'fillField',
+    'appendField',
+    'selectOption',
+    'attachFile',
+    'checkOption',
+    'uncheckOption',
+    'doubleClick',
+  ],
+  fn: async (args) => {
+    return ai.healFailedStep(args);
+  },
+});
+```
+
+As you use, it will be activated on failed steps and will use HTML of a page as additional information. The prompt, error, and the HTML will be sent to AI provider you configured. 
+
+Learn more how you can [configure AI provider](./ai).
+
+To activate the AI healer don't forget to run tests with `--ai` flag.

package/docs/helpers/ApiDataFactory.md

@@ -0,0 +1,267 @@
+---
+permalink: /helpers/ApiDataFactory
+editLink: false
+sidebar: auto
+title: ApiDataFactory
+---
+
+<!-- Generated by documentation.js. Update this documentation by updating the source code. -->
+
+## ApiDataFactory
+
+**Extends Helper**
+
+Helper for managing remote data using REST API.
+Uses data generators like [rosie][1] or factory girl to create new record.
+
+By defining a factory you set the rules of how data is generated.
+This data will be saved on server via REST API and deleted in the end of a test.
+
+## Use Case
+
+Acceptance tests interact with a websites using UI and real browser.
+There is no way to create data for a specific test other than from user interface.
+That makes tests slow and fragile. Instead of testing a single feature you need to follow all creation/removal process.
+
+This helper solves this problem.
+Most of web application have API, and it can be used to create and delete test records.
+By combining REST API with Factories you can easily create records for tests:
+
+```js
+I.have('user', { login: 'davert', email: 'davert@mail.com' });
+let id = await I.have('post', { title: 'My first post'});
+I.haveMultiple('comment', 3, {post_id: id});
+```
+
+To make this work you need
+
+1.  REST API endpoint which allows to perform create / delete requests and
+2.  define data generation rules
+
+### Setup
+
+Install [Rosie][1] and [Faker][2] libraries.
+
+```sh
+npm i rosie @faker-js/faker --save-dev
+```
+
+Create a factory file for a resource.
+
+See the example for Posts factories:
+
+```js
+// tests/factories/posts.js
+
+const { Factory } = require('rosie');
+const { faker } = require('@faker-js/faker');
+
+module.exports = new Factory()
+   // no need to set id, it will be set by REST API
+   .attr('author', () => faker.person.findName())
+   .attr('title', () => faker.lorem.sentence())
+   .attr('body', () => faker.lorem.paragraph());
+```
+
+For more options see [rosie documentation][1].
+
+Then configure ApiDataHelper to match factories and REST API:
+
+### Configuration
+
+ApiDataFactory has following config options:
+
+*   `endpoint`: base URL for the API to send requests to.
+*   `cleanup` (default: true): should inserted records be deleted up after tests
+*   `factories`: list of defined factories
+*   `returnId` (default: false): return id instead of a complete response when creating items.
+*   `headers`: list of headers
+*   `REST`: configuration for REST requests
+
+See the example:
+
+```js
+ ApiDataFactory: {
+   endpoint: "http://user.com/api",
+   cleanup: true,
+   headers: {
+     'Content-Type': 'application/json',
+     'Accept': 'application/json',
+   },
+   factories: {
+     post: {
+       uri: "/posts",
+       factory: "./factories/post",
+     },
+     comment: {
+       factory: "./factories/comment",
+       create: { post: "/comments/create" },
+       delete: { post: "/comments/delete/{id}" },
+       fetchId: (data) => data.result.id
+     }
+   }
+}
+```
+
+It is required to set REST API `endpoint` which is the baseURL for all API requests.
+Factory file is expected to be passed via `factory` option.
+
+This Helper uses [REST][3] helper and accepts its configuration in "REST" section.
+For instance, to set timeout you should add:
+
+```js
+"ApiDataFactory": {
+   "REST": {
+     "timeout": "100000",
+  }
+}
+```
+
+### Requests
+
+By default to create a record ApiDataFactory will use endpoint and plural factory name:
+
+*   create: `POST {endpoint}/{resource} data`
+*   delete: `DELETE {endpoint}/{resource}/id`
+
+Example (`endpoint`: `http://app.com/api`):
+
+*   create: POST request to `http://app.com/api/users`
+*   delete: DELETE request to `http://app.com/api/users/1`
+
+This behavior can be configured with following options:
+
+*   `uri`: set different resource uri. Example: `uri: account` => `http://app.com/api/account`.
+*   `create`: override create options. Expected format: `{ method: uri }`. Example: `{ "post": "/users/create" }`
+*   `delete`: override delete options. Expected format: `{ method: uri }`. Example: `{ "post": "/users/delete/{id}" }`
+
+Requests can also be overridden with a function which returns [axois request config][4].
+
+```js
+create: (data) => ({ method: 'post', url: '/posts', data }),
+delete: (id) => ({ method: 'delete', url: '/posts', data: { id } })
+
+```
+
+Requests can be updated on the fly by using `onRequest` function. For instance, you can pass in current session from a cookie.
+
+```js
+ onRequest: async (request) => {
+    // using global codeceptjs instance
+    let cookie = await codeceptjs.container.helpers('WebDriver').grabCookie('session');
+    request.headers = { Cookie: `session=${cookie.value}` };
+  }
+```
+
+### Responses
+
+By default `I.have()` returns a promise with a created data:
+
+```js
+let client = await I.have('client');
+```
+
+Ids of created records are collected and used in the end of a test for the cleanup.
+If you need to receive `id` instead of full response enable `returnId` in a helper config:
+
+```js
+// returnId: false
+let clientId = await I.have('client');
+// clientId == 1
+
+// returnId: true
+let clientId = await I.have('client');
+// client == { name: 'John', email: 'john@snow.com' }
+```
+
+By default `id` property of response is taken. This behavior can be changed by setting `fetchId` function in a factory config.
+
+```js
+   factories: {
+     post: {
+       uri: "/posts",
+       factory: "./factories/post",
+       fetchId: (data) => data.result.posts[0].id
+     }
+   }
+```
+
+## Methods
+
+### Parameters
+
+*   `config` &#x20;
+
+### _requestCreate
+
+Executes request to create a record in API.
+Can be replaced from a in custom helper.
+
+#### Parameters
+
+*   `factory` **any**&#x20;
+*   `data` **any**&#x20;
+
+### _requestDelete
+
+Executes request to delete a record in API
+Can be replaced from a custom helper.
+
+#### Parameters
+
+*   `factory` **any**&#x20;
+*   `id` **any**&#x20;
+
+### have
+
+Generates a new record using factory and saves API request to store it.
+
+```js
+// create a user
+I.have('user');
+// create user with defined email
+// and receive it when inside async function
+const user = await I.have('user', { email: 'user@user.com'});
+// create a user with options that will not be included in the final request
+I.have('user', { }, { age: 33, height: 55 })
+```
+
+#### Parameters
+
+*   `factory` **any** factory to use
+*   `params` **any?** predefined parameters
+*   `options` **any?** options for programmatically generate the attributes
+
+Returns **[Promise][5]<any>**&#x20;
+
+### haveMultiple
+
+Generates bunch of records and saves multiple API requests to store them.
+
+```js
+// create 3 posts
+I.haveMultiple('post', 3);
+
+// create 3 posts by one author
+I.haveMultiple('post', 3, { author: 'davert' });
+
+// create 3 posts by one author with options
+I.haveMultiple('post', 3, { author: 'davert' }, { publish_date: '01.01.1997' });
+```
+
+#### Parameters
+
+*   `factory` **any**&#x20;
+*   `times` **any**&#x20;
+*   `params` **any?**&#x20;
+*   `options` **any?**&#x20;
+
+[1]: https://github.com/rosiejs/rosie
+
+[2]: https://www.npmjs.com/package/faker
+
+[3]: http://codecept.io/helpers/REST/
+
+[4]: https://github.com/axios/axios#request-config
+
+[5]: https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Promise