codeceptjs 4.0.2-beta.9 → 4.0.2

package/docs/puppeteer.md

@@ -0,0 +1,314 @@
+---
+permalink: /puppeteer
+title: Testing with Puppeteer
+---
+
+# Testing with Puppeteer
+
+Among all Selenium alternatives the most interesting emerging ones are tools developed around Google Chrome [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/). And the most prominent one is [Puppeteer](https://github.com/puppeteer/puppeteer). It operates over Google Chrome directly without requiring additional tools like ChromeDriver. So tests setup with Puppeteer can be started with npm install only. If you want get faster and simpler to setup tests, Puppeteer would be your choice.
+
+CodeceptJS uses Puppeteer to improve end to end testing experience. No need to learn the syntax of a new tool, all drivers in CodeceptJS share the same API.
+
+Take a look at a sample test:
+
+```js
+I.amOnPage('https://github.com')
+I.click('Sign in', '//html/body/div[1]/header')
+I.see('Sign in to GitHub', 'h1')
+I.fillField('Username or email address', 'something@totest.com')
+I.fillField('Password', '123456')
+I.click('Sign in')
+I.see('Incorrect username or password.', '.flash-error')
+```
+
+It's readable and simple and works using Puppeteer API!
+
+## Setup
+
+To start you need CodeceptJS with Puppeteer packages installed
+
+```bash
+npm install codeceptjs puppeteer --save
+```
+
+Or see [alternative installation options](https://codecept.io/installation/)
+
+> If you already have CodeceptJS project, just install `puppeteer` package and enable a helper it in config.
+
+And a basic project initialized
+
+```sh
+npx codeceptjs init
+```
+
+You will be asked for a Helper to use, you should select Puppeteer and provide url of a website you are testing.
+
+> Puppeteer can also work with Firefox. [Learn how to set it up](/helpers/Puppeteer-firefox)
+
+## Configuring
+
+Make sure `Puppeteer` helper is enabled in `codecept.conf.js` config:
+
+```js
+{ // ..
+  helpers: {
+    Puppeteer: {
+      url: "http://localhost",
+      show: true
+    }
+  }
+  // ..
+}
+```
+
+> Turn off the `show` option if you want to run test in headless mode.
+
+Puppeteer uses different strategies to detect if a page is loaded. In configuration use `waitForNavigation` option for that:
+
+By default it is set to `domcontentloaded` which waits for `DOMContentLoaded` event being fired. However, for Single Page Applications it's more useful to set this value to `networkidle0` which waits for all network connections to be finished.
+
+```js
+  helpers: {
+    Puppeteer: {
+      url: "http://localhost",
+      show: true,
+      waitForNavigation: "networkidle0"
+    }
+  }
+```
+
+When a test runs faster than application it is recommended to increase `waitForAction` config value.
+It will wait for a small amount of time (100ms) by default after each user action is taken.
+
+> ▶ More options are listed in [helper reference](https://codecept.io/helpers/Puppeteer/).
+
+## Writing Tests
+
+CodeceptJS test should be created with `gt` command:
+
+```sh
+npx codeceptjs gt
+```
+
+As an example we will use `ToDoMvc` app for testing.
+
+### Actions
+
+Tests consist with a scenario of user's action taken on a page. The most widely used ones are:
+
+- `amOnPage` - to open a webpage (accepts relative or absolute url)
+- `click` - to locate a button or link and click on it
+- `fillField` - to enter a text inside a field
+- `selectOption`, `checkOption` - to interact with a form
+- `wait*` to wait for some parts of page to be fully rendered (important for testing SPA)
+- `grab*` to get values from page sources
+- `see`, `dontSee` - to check for a text on a page
+- `seeElement`, `dontSeeElement` - to check for elements on a page
+
+> ℹ All actions are listed in [Puppeteer helper reference](https://codecept.io/helpers/Puppeteer/).\*
+
+All actions which interact with elements **support CSS and XPath locators**. Actions like `click` or `fillField` by locate elements by their name or value on a page:
+
+```js
+// search for link or button
+I.click('Login')
+// locate field by its label
+I.fillField('Name', 'Miles')
+// we can use input name
+I.fillField('user[email]', 'miles@davis.com')
+```
+
+You can also specify the exact locator type with strict locators:
+
+```js
+I.click({ css: 'button.red' })
+I.fillField({ name: 'user[email]' }, 'miles@davis.com')
+I.seeElement({ xpath: '//body/header' })
+```
+
+### Interactive Pause
+
+It's easy to start writing a test if you use [interactive pause](/basics#debug). Just open a web page and pause execution.
+
+```js
+Feature('Sample Test')
+
+Scenario('open my website', ({ I }) => {
+  I.amOnPage('http://todomvc.com/examples/react/')
+  pause()
+})
+```
+
+This is just enough to run a test, open a browser, and think what to do next to write a test case.
+
+When you execute such test with `codeceptjs run` command you may see the browser is started
+
+```
+npx codeceptjs run --steps
+```
+
+After a page is opened a full control of a browser is given to a terminal. Type in different commands such as `click`, `see`, `fillField` to write the test. A successful commands will be saved to `./output/cli-history` file and can be copied into a test.
+
+A complete ToDo-MVC test may look like:
+
+```js
+Feature('ToDo')
+
+Scenario('create todo item', ({ I }) => {
+  I.amOnPage('http://todomvc.com/examples/react/')
+  I.dontSeeElement('.todo-count')
+  I.fillField('What needs to be done?', 'Write a guide')
+  I.pressKey('Enter')
+  I.see('Write a guide', '.todo-list')
+  I.see('1 item left', '.todo-count')
+})
+```
+
+### Grabbers
+
+If you need to get element's value inside a test you can use `grab*` methods. They should be used with `await` operator inside `async` function:
+
+```js
+import assert from 'assert'
+Scenario('get value of current tasks', async ({ I }) => {
+  I.fillField('.todo', 'my first item')
+  I.pressKey('Enter')
+  I.fillField('.todo', 'my second item')
+  I.pressKey('Enter')
+  let numTodos = await I.grabTextFrom('.todo-count strong')
+  assert.equal(2, numTodos)
+})
+```
+
+### Within
+
+In case some actions should be taken inside one element (a container or modal window or iframe) you can use `within` block to narrow the scope.
+Please take a note that you can't use within inside another within in Puppeteer helper:
+
+```js
+await within('.todoapp', () => {
+  I.fillField('.todo', 'my new item')
+  I.pressKey('Enter')
+  I.see('1 item left', '.todo-count')
+  I.click('.todo-list input.toggle')
+})
+I.see('0 items left', '.todo-count')
+```
+
+### Each Element <Badge text="Since 3.3" type="warning"/>
+
+Usually, CodeceptJS performs an action on the first matched element.
+In case you want to do an action on each element found, use the special function `eachElement` which comes from [eachElement](https://codecept.io/plugins/#eachelement) plugin.
+
+`eachElement` function matches all elements by locator and performs a callback on each of those element. A callback function receives [ElementHandle instance](https://pptr.dev/#?product=Puppeteer&show=api-class-elementhandle) from Puppeteer API. `eachElement` may perform arbitrary actions on a page, so the first argument should by a description of the actions performed. This description will be used for logging purposes.
+
+Usage example
+
+```js
+await eachElement(
+  'click all checkboxes',
+  'input.custom-checkbox',
+  async (el, index) => {
+    await el.click();
+  });
+);
+```
+
+> ℹ Learn more about [eachElement plugin](/plugins/#eachelement)
+
+## Mocking Network Requests <Badge text="Since 3.5.16" type="warning"/>
+
+Network requests & responses can be mocked and modified. Use `mockRoute` which strictly follows [Puppeteer's `setRequestInterception` API](https://pptr.dev/next/api/puppeteer.page.setrequestinterception).
+
+```js
+I.mockRoute('https://reqres.in/api/comments/1', request => {
+  request.respond({
+    status: 200,
+    headers: { 'Access-Control-Allow-Origin': '*' },
+    contentType: 'application/json',
+    body: '{"name": "this was mocked" }',
+  });
+})
+
+I.mockRoute('**/*.{png,jpg,jpeg}', route => route.abort());
+
+// To disable mocking for a route call `stopMockingRoute`
+// for previously mocked URL
+I.stopMockingRoute('**/*.{png,jpg,jpeg}'
+```
+
+To master request intercepting [use `HTTPRequest` object](https://pptr.dev/next/api/puppeteer.httprequest) object passed into mock request handler.
+
+## Accessing Puppeteer API
+
+To get Puppeteer API inside a test use [`I.usePupepteerTo`](/helpers/Puppeteer/#usepuppeteerto) method with a callback.
+To keep test readable provide a description of a callback inside the first parameter.
+
+```js
+I.usePuppeteerTo('emulate offline mode', async ({ page, browser }) => {
+  await page.setOfflineMode(true)
+})
+```
+
+> Puppeteer commands are asynchronous so a callback function must be async.
+
+A Puppeteer helper is passed as argument for callback, so you can combine Puppeteer API with CodeceptJS API:
+
+```js
+I.usePuppeteerTo('emulate offline mode', async Puppeteer => {
+  // access internal objects browser, page, context of helper
+  await Puppeteer.page.setOfflineMode(true)
+  // call a method of helper, await is required here
+  await Puppeteer.click('Reload')
+})
+```
+
+## Capturing Code Coverage
+
+Code coverage can be captured, by enabling the `coverage` plugin in `codecept.config.js`.
+
+```js
+{
+  plugins: {
+    coverage: {
+      enabled: true
+    }
+  }
+}
+```
+
+Once all the tests are completed, `codecept` will create and store coverage in `output/coverage` folder, as shown below.
+
+![](<(https://github.com/codeceptjs/CodeceptJS/assets/7845001/3b8b81a3-7c85-470c-992d-ecdc7d5b4a1e)>)
+
+Open `index.html` in your browser to view the full interactive coverage report.
+
+![](https://github.com/codeceptjs/CodeceptJS/assets/7845001/f45607ed-dbe8-4ed4-9b21-01ce25288d22)
+
+![](https://github.com/codeceptjs/CodeceptJS/assets/7845001/c821ce45-6590-4ace-b7ae-2cafb3a4e532)
+
+## Extending Helper
+
+To create custom `I.*` commands using Puppeteer API you need to create a custom helper.
+
+Start with creating an `MyPuppeteer` helper using `generate:helper` or `gh` command:
+
+```sh
+npx codeceptjs gh
+```
+
+Then inside a Helper you can access `Puppeteer` helper of CodeceptJS.
+Let's say you want to create `I.renderPageToPdf` action. In this case you need to call `pdf` method of `page` object
+
+```js
+// inside a MyPuppeteer helper
+async renderPageToPdf() {
+  const page = this.helpers['Puppeteer'].page;
+  await page.emulateMedia('screen');
+  return page.pdf({path: 'page.pdf'});
+}
+```
+
+The same way you can also access `browser` object to implement more actions or handle events.
+
+> [▶ Learn more about Helpers](https://codecept.io/helpers/)

package/docs/quickstart.md

@@ -0,0 +1,120 @@
+---
+permalink: quickstart
+title: Quickstart
+layout: Section
+sidebar: true
+---
+
+# Quickstart
+
+Install CodeceptJS into your project:
+
+```
+npm install codeceptjs playwright --save-dev
+```
+
+Then install the browser binaries:
+
+```
+npx playwright install --with-deps
+```
+
+The `--with-deps` flag also installs required system dependencies for the browsers.
+
+> Prefer WebDriver or Appium? See [installation options](/installation/) for all supported helpers.
+
+---
+
+### Init
+
+Initialize CodeceptJS to set up the config file and test directory:
+
+```
+npx codeceptjs init
+```
+
+This command walks you through a short setup wizard and creates `codecept.conf.js`, a sample test file, and any required browser binaries.
+
+Answer the questions, accepting defaults to get started quickly:
+
+| Question | Default Answer  | Alternative
+|---|---|---|
+| Do you plan to write tests in TypeScript?  | **n** (No)  | or [learn how to use TypeScript](/typescript)
+| Where are your tests located? | `**./*_test.js` | or any glob pattern like `**.spec.js`
+| What helpers do you want to use? | **Playwright** | See options for [web testing](https://codecept.io/basics/#architecture), [mobile testing](https://codecept.io/mobile/), [API testing](https://codecept.io/api/)
+| Where should logs, screenshots, and reports be stored? | `./output` | path to store artifacts and temporary files
+
+For Playwright, you'll also be asked about the site and browser:
+
+| Question | Default Answer  | Alternative
+|---|---|---|
+| Base url of site to be tested | http://localhost | URL of the site you plan to test
+| Show browser window | **y** Yes | or run in **headless mode**
+| Browser | **chromium** | or `firefox`, `webkit` (open-source Safari), or `electron`
+
+Sample output:
+
+```
+? Do you plan to write tests in TypeScript? No
+? Where are your tests located? **./*_test.js
+? What helpers do you want to use? Playwright
+? Where should logs, screenshots, and reports be stored? ./output
+? [Playwright] Base url of site to be tested http://localhost
+? [Playwright] Show browser window Yes
+? [Playwright] Browser in which testing will be performed chromium
+```
+
+When asked, create your first feature and test file.
+
+---
+
+### Write Your First Test
+
+Open the generated test file. It will look like this:
+
+```js
+Feature('My First Test');
+
+Scenario('test something', ({ I }) => {
+
+});
+```
+
+Add a simple scenario:
+
+```js
+Feature('My First Test');
+
+Scenario('test something', ({ I }) => {
+  I.amOnPage('https://github.com');
+  I.see('GitHub');
+});
+```
+
+---
+
+### Run Tests
+
+```
+npx codeceptjs run
+```
+
+Expected output:
+
+```bash
+My First Test --
+  test something
+     I am on page "https://github.com"
+     I see "GitHub"
+ ✓ OK
+```
+
+Run in headless mode:
+
+```
+npx codeceptjs run --p browser:hide
+```
+
+See all available commands in the [CLI reference](https://codecept.io/commands/).
+
+> [▶ Next: CodeceptJS Basics](/basics/)

package/docs/reports.md

@@ -0,0 +1,195 @@
+---
+permalink: /reports
+title: Reporters
+---
+
+# Reporters
+
+CodeceptJS prints test results to the console by default (see [CLI output](#cli-output)). For an HTML report, a pull-request comment, JUnit XML, or a hosted dashboard, use **[Testomat.io Reporter](https://github.com/testomatio/reporter)**.
+
+Testomat.io Reporter is open source. It can send create reports of various types:
+
+- **Local files** — HTML, CSV, and Markdown reports written to disk.
+- **CI/CD pipeline reports** — reports added as comments to GitHub, GitLab, or Bitbucket pull requests.
+- **Cloud Services** — a hosted dashboard on [app.testomat.io](https://testomat.io) to keep track of results history and get analytics.
+
+You choose the destinations. The reporter collects steps, screenshots, videos, traces, and logs once and sends them everywhere you enabled.
+
+### Install
+
+```sh
+npm install @testomatio/reporter --save-dev
+```
+
+Enable the reporter plugin and turn on the local reports you want:
+
+```js
+// codecept.conf.js
+plugins: {
+  testomatio: {
+    enabled: true,
+    require: '@testomatio/reporter/codecept',
+    html: true,
+    // markdown: true,
+    // csv: true,
+    // reportDir: 'output/report',
+  },
+}
+```
+
+> `html`, `markdown`, and `csv` are config switches for local reports. Local reports are stored in filesystem in `output/report` dir by default.
+
+### Local reports and remote destinations
+
+Local reports are turned on with config switches and saved to disk:
+
+| Config switch | File in `reportDir` |
+| --- | --- |
+| `html: true` | `testomatio-report.html` |
+| `markdown: true` | `testomatio-report.md` |
+| `csv: true` | `report.csv` |
+
+Remote destinations need an access token. A token is a secret, so pass it through an environment variable or CI secret instead of committing it to the config file:
+
+| Destination | Token variable |
+| --- | --- |
+| Run result on [app.testomat.io](https://testomat.io) | `TESTOMATIO` |
+| Comment on a GitHub Pull Request | `GH_PAT` (`${{ github.token }}` in Actions) |
+| Comment on a GitLab Merge Request | `GITLAB_PAT` (token with `api` scope) |
+| Comment on a Bitbucket Pull Request | `BITBUCKET_ACCESS_TOKEN` (repo access token) |
+
+One run feeds every destination you enabled. On CI, keep the report switches in the config and pass only the tokens as environment variables:
+
+```yaml
+- run: npx codeceptjs run
+  env:
+    GH_PAT: ${{ github.token }}             # → Print report as PR comment
+    # TESTOMATIO: ${{ secrets.TESTOMATIO }} # → Send report to testomat.io
+- uses: actions/upload-artifact@v4
+  if: always()
+  with:
+    name: codeceptjs-output
+    path: output/
+```
+
+The GitHub destination also needs the job to grant `permissions: pull-requests: write`.
+
+For the full list of options and environment variables, see the [reporter configuration reference](https://github.com/testomatio/reporter/blob/master/docs/configuration.md).
+
+### HTML Report
+
+A local, self-contained HTML file saved to `output/report/testomatio-report.html`. It holds the run summary and, per test, its steps, screenshots, logs, and error. It needs no API key and no service, so it works anywhere — open it locally or attach it to a CI build.
+
+![HTML report](./images/testomatio-html-report.png)
+
+Enable it with `html: true` in `plugins.testomatio` and run `npx codeceptjs run`. To change the file name or folder, see the [HTML pipe docs](https://github.com/testomatio/reporter/blob/master/docs/pipes/html.md).
+
+### CSV Report
+
+A local CSV file saved to `output/report/report.csv`, with one row per test — suite, title, and status. Use it in spreadsheets or data pipelines.
+
+Enable it with `csv: true` in `plugins.testomatio`. See the [CSV pipe docs](https://github.com/testomatio/reporter/blob/master/docs/pipes/csv.md).
+
+### Testomat.io Cloud Report
+
+A remote destination: the run is sent to [app.testomat.io](https://testomat.io), a hosted dashboard with run history, flaky-test detection, parallel-run merging, re-running failed tests, and notifications. Free for small teams.
+
+![Testomat.io report](https://user-images.githubusercontent.com/220264/151728836-b52d2b2b-56e1-4640-8d3a-b39de817b1fd.png)
+
+Set the `TESTOMATIO` environment variable to your project API key and run the tests. Run titles, run groups, shared runs, and publishing options: [Testomat.io pipe docs](https://github.com/testomatio/reporter/blob/master/docs/pipes/testomatio.md).
+
+To view artifacts on the cloud, upload them to S3 storage. Images come from the [`screenshot`](/plugins#screenshot) plugin, videos from the [`screencast`](/plugins#screencast) plugin (or the Playwright helper's `video` and `trace`). Any S3 provider works: AWS S3, Cloudflare R2, Google Cloud Storage (interoperability mode), DigitalOcean Spaces, MinIO.
+
+### GitHub Report
+
+A remote destination: a comment on the Pull Request with run status, pass/fail/skip counts, stack traces of the failures, screenshots, and the slowest tests. Re-running the workflow replaces the previous comment.
+
+![GitHub report](https://raw.githubusercontent.com/testomatio/reporter/master/docs/pipes/images/github.png)
+
+Set `GH_PAT` to a GitHub token (`${{ github.token }}` works in Actions) and grant the job `permissions: pull-requests: write`. More options: [GitHub pipe docs](https://github.com/testomatio/reporter/blob/master/docs/pipes/github.md).
+
+### GitLab Report
+
+A remote destination: a comment on the Merge Request with the same summary. Run it in merge-request pipelines (`$CI_PIPELINE_SOURCE == "merge_request_event"`).
+
+![GitLab report](https://raw.githubusercontent.com/testomatio/reporter/master/docs/pipes/images/gitlab.png)
+
+Set `GITLAB_PAT` to a Personal or Project Access Token with `api` scope. More options: [GitLab pipe docs](https://github.com/testomatio/reporter/blob/master/docs/pipes/gitlab.md).
+
+### Bitbucket Report
+
+A remote destination: a comment on the Pull Request with the same summary. Comments are created only in `pull-requests` pipelines.
+
+![Bitbucket report](https://raw.githubusercontent.com/testomatio/reporter/master/docs/pipes/images/bitbucket.png)
+
+Set `BITBUCKET_ACCESS_TOKEN` to a repository access token with `Pull requests: Write` and `Repository: Read`. More options: [Bitbucket pipe docs](https://github.com/testomatio/reporter/blob/master/docs/pipes/bitbucket.md).
+
+### Markdown Report
+
+A local, self-contained Markdown file saved to `output/report/testomatio-report.md`. It renders in PR comments, CI job summaries, and Slack, and is convenient for AI agents reading test results. It needs no API key.
+
+![Markdown report](./images/testomatio-markdown-report.png)
+
+Enable it with `markdown: true` in `plugins.testomatio`. To change the file name, folder, or title, see the [Markdown pipe docs](https://github.com/testomatio/reporter/blob/master/docs/pipes/markdown.md).
+
+On GitHub Actions, append it to the job summary: `cat output/report/testomatio-report.md >> "$GITHUB_STEP_SUMMARY"`.
+
+## JUnit XML
+
+For CI servers that read JUnit XML (Jenkins, GitLab CI, CircleCI, the GitHub Actions test tab), enable the [`junitReporter`](/plugins#junitreporter) plugin. It writes `output/report.xml` with CodeceptJS steps included — unlike `mocha-junit-reporter`.
+
+```js
+plugins: {
+  junitReporter: { enabled: true },
+}
+```
+
+Options (`outputName`, `output`, `testGroupName`, `attachMeta`, `attachSteps`, `stepsInFailure`): [plugin docs](/plugins#junitreporter).
+
+## ReportPortal
+
+[ReportPortal](https://reportportal.io) is an open-source self-hosted dashboard for test reports. Publish with the [CodeceptJS Agent for ReportPortal](https://github.com/reportportal/agent-js-codecept/).
+
+## Custom reporter
+
+The [`customReporter`](/plugins#customreporter) plugin hooks into test events:
+
+```js
+plugins: {
+  customReporter: {
+    enabled: true,
+    onTestFailed: (test, err) => console.log('FAIL', test.title, err.message),
+    onResult: result => {
+      // result.stats, result.tests
+    },
+  },
+}
+```
+
+Hooks: `onHookFinished`, `onTestBefore`, `onTestPassed`, `onTestFailed`, `onTestSkipped`, `onTestFinished`, `onResult`.
+
+For built-in Mocha reporters, use `--reporter`:
+
+```sh
+npx codeceptjs run --reporter dot
+```
+
+> The bundled `Mochawesome` helper was removed in 4.x. For an HTML report use the [Testomat.io Reporter](https://github.com/testomatio/reporter) HTML pipe (see above); for JUnit XML use the [`junitReporter`](#junit-xml) plugin. Wiring multiple Mocha reporters through `mocha-multi`/`cmr` is not recommended — prefer these instead.
+
+Plugins exist for [TestRail](https://www.npmjs.com/package/codeceptjs-testrail) and [Tesults](https://www.npmjs.com/package/codeceptjs-tesults).
+
+## CLI output
+
+By default CodeceptJS prints test names and failures. Add `--steps` to see each step, `--debug` for runner notices, or `--verbose` for full stack traces and events (use this when reporting bugs).
+
+```sh
+npx codeceptjs run --steps
+npx codeceptjs run --debug
+npx codeceptjs run --verbose
+```
+
+`dry-run` lists tests and steps without running them:
+
+```sh
+npx codeceptjs dry-run --steps
+```