codeceptjs 4.0.0-rc.8 → 4.0.0

package/docs/puppeteer.md

@@ -0,0 +1,314 @@
+---
+permalink: /puppeteer
+title: Testing with Puppeteer
+---
+
+# Testing with Puppeteer
+
+Among all Selenium alternatives the most interesting emerging ones are tools developed around Google Chrome [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/). And the most prominent one is [Puppeteer](https://github.com/puppeteer/puppeteer). It operates over Google Chrome directly without requiring additional tools like ChromeDriver. So tests setup with Puppeteer can be started with npm install only. If you want get faster and simpler to setup tests, Puppeteer would be your choice.
+
+CodeceptJS uses Puppeteer to improve end to end testing experience. No need to learn the syntax of a new tool, all drivers in CodeceptJS share the same API.
+
+Take a look at a sample test:
+
+```js
+I.amOnPage('https://github.com')
+I.click('Sign in', '//html/body/div[1]/header')
+I.see('Sign in to GitHub', 'h1')
+I.fillField('Username or email address', 'something@totest.com')
+I.fillField('Password', '123456')
+I.click('Sign in')
+I.see('Incorrect username or password.', '.flash-error')
+```
+
+It's readable and simple and works using Puppeteer API!
+
+## Setup
+
+To start you need CodeceptJS with Puppeteer packages installed
+
+```bash
+npm install codeceptjs puppeteer --save
+```
+
+Or see [alternative installation options](https://codecept.io/installation/)
+
+> If you already have CodeceptJS project, just install `puppeteer` package and enable a helper it in config.
+
+And a basic project initialized
+
+```sh
+npx codeceptjs init
+```
+
+You will be asked for a Helper to use, you should select Puppeteer and provide url of a website you are testing.
+
+> Puppeteer can also work with Firefox. [Learn how to set it up](/helpers/Puppeteer-firefox)
+
+## Configuring
+
+Make sure `Puppeteer` helper is enabled in `codecept.conf.js` config:
+
+```js
+{ // ..
+  helpers: {
+    Puppeteer: {
+      url: "http://localhost",
+      show: true
+    }
+  }
+  // ..
+}
+```
+
+> Turn off the `show` option if you want to run test in headless mode.
+
+Puppeteer uses different strategies to detect if a page is loaded. In configuration use `waitForNavigation` option for that:
+
+By default it is set to `domcontentloaded` which waits for `DOMContentLoaded` event being fired. However, for Single Page Applications it's more useful to set this value to `networkidle0` which waits for all network connections to be finished.
+
+```js
+  helpers: {
+    Puppeteer: {
+      url: "http://localhost",
+      show: true,
+      waitForNavigation: "networkidle0"
+    }
+  }
+```
+
+When a test runs faster than application it is recommended to increase `waitForAction` config value.
+It will wait for a small amount of time (100ms) by default after each user action is taken.
+
+> ▶ More options are listed in [helper reference](https://codecept.io/helpers/Puppeteer/).
+
+## Writing Tests
+
+CodeceptJS test should be created with `gt` command:
+
+```sh
+npx codeceptjs gt
+```
+
+As an example we will use `ToDoMvc` app for testing.
+
+### Actions
+
+Tests consist with a scenario of user's action taken on a page. The most widely used ones are:
+
+- `amOnPage` - to open a webpage (accepts relative or absolute url)
+- `click` - to locate a button or link and click on it
+- `fillField` - to enter a text inside a field
+- `selectOption`, `checkOption` - to interact with a form
+- `wait*` to wait for some parts of page to be fully rendered (important for testing SPA)
+- `grab*` to get values from page sources
+- `see`, `dontSee` - to check for a text on a page
+- `seeElement`, `dontSeeElement` - to check for elements on a page
+
+> ℹ All actions are listed in [Puppeteer helper reference](https://codecept.io/helpers/Puppeteer/).\*
+
+All actions which interact with elements **support CSS and XPath locators**. Actions like `click` or `fillField` by locate elements by their name or value on a page:
+
+```js
+// search for link or button
+I.click('Login')
+// locate field by its label
+I.fillField('Name', 'Miles')
+// we can use input name
+I.fillField('user[email]', 'miles@davis.com')
+```
+
+You can also specify the exact locator type with strict locators:
+
+```js
+I.click({ css: 'button.red' })
+I.fillField({ name: 'user[email]' }, 'miles@davis.com')
+I.seeElement({ xpath: '//body/header' })
+```
+
+### Interactive Pause
+
+It's easy to start writing a test if you use [interactive pause](/basics#debug). Just open a web page and pause execution.
+
+```js
+Feature('Sample Test')
+
+Scenario('open my website', ({ I }) => {
+  I.amOnPage('http://todomvc.com/examples/react/')
+  pause()
+})
+```
+
+This is just enough to run a test, open a browser, and think what to do next to write a test case.
+
+When you execute such test with `codeceptjs run` command you may see the browser is started
+
+```
+npx codeceptjs run --steps
+```
+
+After a page is opened a full control of a browser is given to a terminal. Type in different commands such as `click`, `see`, `fillField` to write the test. A successful commands will be saved to `./output/cli-history` file and can be copied into a test.
+
+A complete ToDo-MVC test may look like:
+
+```js
+Feature('ToDo')
+
+Scenario('create todo item', ({ I }) => {
+  I.amOnPage('http://todomvc.com/examples/react/')
+  I.dontSeeElement('.todo-count')
+  I.fillField('What needs to be done?', 'Write a guide')
+  I.pressKey('Enter')
+  I.see('Write a guide', '.todo-list')
+  I.see('1 item left', '.todo-count')
+})
+```
+
+### Grabbers
+
+If you need to get element's value inside a test you can use `grab*` methods. They should be used with `await` operator inside `async` function:
+
+```js
+import assert from 'assert'
+Scenario('get value of current tasks', async ({ I }) => {
+  I.fillField('.todo', 'my first item')
+  I.pressKey('Enter')
+  I.fillField('.todo', 'my second item')
+  I.pressKey('Enter')
+  let numTodos = await I.grabTextFrom('.todo-count strong')
+  assert.equal(2, numTodos)
+})
+```
+
+### Within
+
+In case some actions should be taken inside one element (a container or modal window or iframe) you can use `within` block to narrow the scope.
+Please take a note that you can't use within inside another within in Puppeteer helper:
+
+```js
+await within('.todoapp', () => {
+  I.fillField('.todo', 'my new item')
+  I.pressKey('Enter')
+  I.see('1 item left', '.todo-count')
+  I.click('.todo-list input.toggle')
+})
+I.see('0 items left', '.todo-count')
+```
+
+### Each Element <Badge text="Since 3.3" type="warning"/>
+
+Usually, CodeceptJS performs an action on the first matched element.
+In case you want to do an action on each element found, use the special function `eachElement` which comes from [eachElement](https://codecept.io/plugins/#eachelement) plugin.
+
+`eachElement` function matches all elements by locator and performs a callback on each of those element. A callback function receives [ElementHandle instance](https://pptr.dev/#?product=Puppeteer&show=api-class-elementhandle) from Puppeteer API. `eachElement` may perform arbitrary actions on a page, so the first argument should by a description of the actions performed. This description will be used for logging purposes.
+
+Usage example
+
+```js
+await eachElement(
+  'click all checkboxes',
+  'input.custom-checkbox',
+  async (el, index) => {
+    await el.click();
+  });
+);
+```
+
+> ℹ Learn more about [eachElement plugin](/plugins/#eachelement)
+
+## Mocking Network Requests <Badge text="Since 3.5.16" type="warning"/>
+
+Network requests & responses can be mocked and modified. Use `mockRoute` which strictly follows [Puppeteer's `setRequestInterception` API](https://pptr.dev/next/api/puppeteer.page.setrequestinterception).
+
+```js
+I.mockRoute('https://reqres.in/api/comments/1', request => {
+  request.respond({
+    status: 200,
+    headers: { 'Access-Control-Allow-Origin': '*' },
+    contentType: 'application/json',
+    body: '{"name": "this was mocked" }',
+  });
+})
+
+I.mockRoute('**/*.{png,jpg,jpeg}', route => route.abort());
+
+// To disable mocking for a route call `stopMockingRoute`
+// for previously mocked URL
+I.stopMockingRoute('**/*.{png,jpg,jpeg}'
+```
+
+To master request intercepting [use `HTTPRequest` object](https://pptr.dev/next/api/puppeteer.httprequest) object passed into mock request handler.
+
+## Accessing Puppeteer API
+
+To get Puppeteer API inside a test use [`I.usePupepteerTo`](/helpers/Puppeteer/#usepuppeteerto) method with a callback.
+To keep test readable provide a description of a callback inside the first parameter.
+
+```js
+I.usePuppeteerTo('emulate offline mode', async ({ page, browser }) => {
+  await page.setOfflineMode(true)
+})
+```
+
+> Puppeteer commands are asynchronous so a callback function must be async.
+
+A Puppeteer helper is passed as argument for callback, so you can combine Puppeteer API with CodeceptJS API:
+
+```js
+I.usePuppeteerTo('emulate offline mode', async Puppeteer => {
+  // access internal objects browser, page, context of helper
+  await Puppeteer.page.setOfflineMode(true)
+  // call a method of helper, await is required here
+  await Puppeteer.click('Reload')
+})
+```
+
+## Capturing Code Coverage
+
+Code coverage can be captured, by enabling the `coverage` plugin in `codecept.config.js`.
+
+```js
+{
+  plugins: {
+    coverage: {
+      enabled: true
+    }
+  }
+}
+```
+
+Once all the tests are completed, `codecept` will create and store coverage in `output/coverage` folder, as shown below.
+
+![](<(https://github.com/codeceptjs/CodeceptJS/assets/7845001/3b8b81a3-7c85-470c-992d-ecdc7d5b4a1e)>)
+
+Open `index.html` in your browser to view the full interactive coverage report.
+
+![](https://github.com/codeceptjs/CodeceptJS/assets/7845001/f45607ed-dbe8-4ed4-9b21-01ce25288d22)
+
+![](https://github.com/codeceptjs/CodeceptJS/assets/7845001/c821ce45-6590-4ace-b7ae-2cafb3a4e532)
+
+## Extending Helper
+
+To create custom `I.*` commands using Puppeteer API you need to create a custom helper.
+
+Start with creating an `MyPuppeteer` helper using `generate:helper` or `gh` command:
+
+```sh
+npx codeceptjs gh
+```
+
+Then inside a Helper you can access `Puppeteer` helper of CodeceptJS.
+Let's say you want to create `I.renderPageToPdf` action. In this case you need to call `pdf` method of `page` object
+
+```js
+// inside a MyPuppeteer helper
+async renderPageToPdf() {
+  const page = this.helpers['Puppeteer'].page;
+  await page.emulateMedia('screen');
+  return page.pdf({path: 'page.pdf'});
+}
+```
+
+The same way you can also access `browser` object to implement more actions or handle events.
+
+> [▶ Learn more about Helpers](https://codecept.io/helpers/)

package/docs/quickstart.md

@@ -0,0 +1,120 @@
+---
+permalink: quickstart
+title: Quickstart
+layout: Section
+sidebar: true
+---
+
+# Quickstart
+
+Install CodeceptJS into your project:
+
+```
+npm install codeceptjs playwright --save-dev
+```
+
+Then install the browser binaries:
+
+```
+npx playwright install --with-deps
+```
+
+The `--with-deps` flag also installs required system dependencies for the browsers.
+
+> Prefer WebDriver or Appium? See [installation options](/installation/) for all supported helpers.
+
+---
+
+### Init
+
+Initialize CodeceptJS to set up the config file and test directory:
+
+```
+npx codeceptjs init
+```
+
+This command walks you through a short setup wizard and creates `codecept.conf.js`, a sample test file, and any required browser binaries.
+
+Answer the questions, accepting defaults to get started quickly:
+
+| Question | Default Answer  | Alternative
+|---|---|---|
+| Do you plan to write tests in TypeScript?  | **n** (No)  | or [learn how to use TypeScript](/typescript)
+| Where are your tests located? | `**./*_test.js` | or any glob pattern like `**.spec.js`
+| What helpers do you want to use? | **Playwright** | See options for [web testing](https://codecept.io/basics/#architecture), [mobile testing](https://codecept.io/mobile/), [API testing](https://codecept.io/api/)
+| Where should logs, screenshots, and reports be stored? | `./output` | path to store artifacts and temporary files
+
+For Playwright, you'll also be asked about the site and browser:
+
+| Question | Default Answer  | Alternative
+|---|---|---|
+| Base url of site to be tested | http://localhost | URL of the site you plan to test
+| Show browser window | **y** Yes | or run in **headless mode**
+| Browser | **chromium** | or `firefox`, `webkit` (open-source Safari), or `electron`
+
+Sample output:
+
+```
+? Do you plan to write tests in TypeScript? No
+? Where are your tests located? **./*_test.js
+? What helpers do you want to use? Playwright
+? Where should logs, screenshots, and reports be stored? ./output
+? [Playwright] Base url of site to be tested http://localhost
+? [Playwright] Show browser window Yes
+? [Playwright] Browser in which testing will be performed chromium
+```
+
+When asked, create your first feature and test file.
+
+---
+
+### Write Your First Test
+
+Open the generated test file. It will look like this:
+
+```js
+Feature('My First Test');
+
+Scenario('test something', ({ I }) => {
+
+});
+```
+
+Add a simple scenario:
+
+```js
+Feature('My First Test');
+
+Scenario('test something', ({ I }) => {
+  I.amOnPage('https://github.com');
+  I.see('GitHub');
+});
+```
+
+---
+
+### Run Tests
+
+```
+npx codeceptjs run
+```
+
+Expected output:
+
+```bash
+My First Test --
+  test something
+     I am on page "https://github.com"
+     I see "GitHub"
+ ✓ OK
+```
+
+Run in headless mode:
+
+```
+npx codeceptjs run --headless
+```
+
+See all available commands in the [CLI reference](https://codecept.io/commands/).
+
+> [▶ Next: CodeceptJS Basics](/basics/)

package/docs/reports.md

@@ -0,0 +1,198 @@
+---
+permalink: /reports
+title: Reporters
+---
+
+# Reporters
+
+CodeceptJS prints test results to the console by default (see [CLI output](#cli-output)). For an HTML report, a pull-request comment, JUnit XML, or a hosted dashboard, it is recommeded to use **[Testomat.io Reporter](https://github.com/testomatio/reporter)**. It sends results to whichever destinations you turn on with steps, screenshots, videos, traces, and logs.
+
+### Install
+
+```sh
+npm install @testomatio/reporter --save-dev
+```
+
+Enable reporter plugin:
+
+```js
+// codecept.conf.js
+plugins: {
+  testomatio: {
+    enabled: true,
+    require: '@testomatio/reporter/codecept',
+    html: true,
+    markdown: true,
+    csv: true,
+    reportDir: 'output/report',
+  },
+}
+```
+
+The local reports above are enabled directly from CodeceptJS config. If `reportDir` is omitted, reports are written to `output/report` using the CodeceptJS `output` directory.
+
+### Enable an output
+
+Each output can also be enabled with environment variables. Run your tests as usual and one run feeds every output you enabled.
+
+| To get… | Set | Details |
+| --- | --- | --- |
+| HTML report | `TESTOMATIO_HTML_REPORT_SAVE=1` | [HTML Report](#html-report) |
+| Markdown report | `TESTOMATIO_MARKDOWN_REPORT_SAVE=1` | [Markdown Report](#markdown-report) |
+| Run Result on [app.testomat.io](https://testomat.io) | `TESTOMATIO` (project API key) | [Cloud Report](#cloud-report) |
+| A comment on a GitHub Pull Request | `GH_PAT` (`${{ github.token }}` in Actions) | [GitHub Report](#github-report) |
+| A comment on a GitLab Merge Request | `GITLAB_PAT` (token with `api` scope) | [GitLab Report](#gitlab-report) |
+| A comment on a Bitbucket Pull Request | `BITBUCKET_ACCESS_TOKEN` (repo access token) | [Bitbucket Report](#bitbucket-report) |
+
+Screenshots and videos in these reports are uploaded to your own storage — see [Artifacts](#artifacts).
+
+Put the variables on CI when running tests:
+
+```yaml
+- run: npx codeceptjs run
+  env:
+    TESTOMATIO_HTML_REPORT_SAVE: 1                  # → output/reports/testomatio-report.html
+    TESTOMATIO_HTML_REPORT_FOLDER: output/reports   # keep it with the rest of output/
+    GH_PAT: ${{ github.token }}                     # → PR comment
+    # TESTOMATIO: ${{ secrets.TESTOMATIO }}         # → testomat.io run
+- uses: actions/upload-artifact@v4
+  if: always()
+  with:
+    name: codeceptjs-output
+    path: output/
+```
+
+The GitHub pipe also needs the job to grant `permissions: pull-requests: write`.
+
+
+### HTML Report
+
+A single self-contained HTML file with the run summary and, per test, its steps, screenshots, logs, and error. It needs no API key and no service, so it works anywhere — open it locally or attach it to a CI build.
+
+![HTML report](./images/testomatio-html-report.png)
+
+- Preferred in CodeceptJS 4: enable `html: true` in `plugins.testomatio` and run `npx codeceptjs run`
+
+- `TESTOMATIO_HTML_REPORT_SAVE=1` — enable the report
+- `TESTOMATIO_HTML_REPORT_FOLDER=output/reports` — keep it inside CodeceptJS's `output/` dir (default folder is `html-report`)
+- `TESTOMATIO_HTML_FILENAME` — file name, must end in `.html` (default `testomatio-report.html`)
+
+### Cloud Report
+
+Sends the run to [app.testomat.io](https://testomat.io) — a hosted dashboard with run history, flaky-test detection, parallel-run merging, re-running failed tests, and notifications. Free for small teams.
+
+![Testomat.io report](https://user-images.githubusercontent.com/220264/151728836-b52d2b2b-56e1-4640-8d3a-b39de817b1fd.png)
+
+- `TESTOMATIO` — project API key; enables the pipe
+- `TESTOMATIO_CREATE=1` — create tests in Testomat.io that were not imported beforehand
+- `TESTOMATIO_TITLE` — report title
+- `TESTOMATIO_RUNGROUP_TITLE` — add the run to a group (e.g. `"Build ${BUILD_ID}"`)
+- `TESTOMATIO_PUBLISH=1` — make the report publicly accessible
+
+More options (shared runs, rungroups, run management): [Testomat.io pipe](https://github.com/testomatio/reporter/blob/master/docs/pipes/testomatio.md).
+
+To view artifacts on cloud they must be uploaded to S3 storages. Images from [`screenshot`](/plugins#screenshot) plugin, videos from the [`screencast`](/plugins#screencast) plugin (or the Playwright helper's `video` and `trace`). Can be used with any S3 provider: AWS S3, Cloudflare R2, Google Cloud Storage (interoperability mode), DigitalOcean Spaces, MinIO. 
+
+### GitHub Report
+
+Posts a comment to the Pull Request: run status, pass/fail/skip counts, stack traces of the failures, screenshots, and the slowest tests. Re-running the workflow replaces the previous comment.
+
+![GitHub report](https://raw.githubusercontent.com/testomatio/reporter/master/docs/pipes/images/github.png)
+
+- `GH_PAT` — GitHub token; `${{ github.token }}` works in Actions
+- the job must grant `permissions: pull-requests: write`
+- `GH_KEEP_OUTDATED_REPORTS=1` — keep previous comments instead of deleting them
+
+### GitLab Report
+
+Posts a comment to the Merge Request with the same summary. It needs Merge Request context, so run it in merge-request pipelines.
+
+![GitLab report](https://raw.githubusercontent.com/testomatio/reporter/master/docs/pipes/images/gitlab.png)
+
+- `GITLAB_PAT` — Personal or Project Access Token with `api` scope
+- run in merge-request pipelines (`$CI_PIPELINE_SOURCE == "merge_request_event"`)
+- `GITLAB_KEEP_OUTDATED_REPORTS=1` — keep previous comments
+- `GITLAB_REMOVE_ALL_OUTDATED_REPORTS=1` — remove all previous comments, not just the latest
+
+### Bitbucket Report
+
+Posts a comment to the Pull Request with the same summary. Comments are created only in `pull-requests` pipelines.
+
+![Bitbucket report](https://raw.githubusercontent.com/testomatio/reporter/master/docs/pipes/images/bitbucket.png)
+
+- `BITBUCKET_ACCESS_TOKEN` — repository access token with `Pull requests: Write` and `Repository: Read`
+- run in `pull-requests` pipelines
+- `BITBUCKET_KEEP_OUTDATED_REPORTS=1` — keep previous comments
+
+### Markdown Report
+
+- Preferred in CodeceptJS 4: enable `markdown: true` in `plugins.testomatio` and run `npx codeceptjs run`
+
+![Markdown report](./images/testomatio-markdown-report.png)
+
+A single self-contained Markdown file — renders in PR comments, CI job summaries, and Slack, and is convenient for AI agents reading test results. Needs no API key.
+
+- `TESTOMATIO_MARKDOWN_REPORT_SAVE=1` — enable the report
+- `TESTOMATIO_MARKDOWN_REPORT_FOLDER=output/reports` — keep it inside CodeceptJS's `output/` dir (default folder is `md-report`)
+- `TESTOMATIO_MARKDOWN_FILENAME` — file name, must end in `.md` (default `testomatio-report.md`)
+- `TESTOMATIO_TITLE` — document title (default `Test Results`)
+
+On GitHub Actions, append it to the job summary: `cat output/reports/testomatio-report.md >> "$GITHUB_STEP_SUMMARY"`.
+
+## JUnit XML
+
+For CI servers that read JUnit XML (Jenkins, GitLab CI, CircleCI, the GitHub Actions test tab), enable the [`junitReporter`](/plugins#junitreporter) plugin. It writes `output/report.xml` with CodeceptJS steps included — unlike `mocha-junit-reporter`.
+
+```js
+plugins: {
+  junitReporter: { enabled: true },
+}
+```
+
+Options (`outputName`, `output`, `testGroupName`, `attachMeta`, `attachSteps`, `stepsInFailure`): [plugin docs](/plugins#junitreporter).
+
+## ReportPortal
+
+[ReportPortal](https://reportportal.io) is an open-source self-hosted dashboard for test reports. Publish with the [CodeceptJS Agent for ReportPortal](https://github.com/reportportal/agent-js-codecept/).
+
+## Custom reporter
+
+The [`customReporter`](/plugins#customreporter) plugin hooks into test events:
+
+```js
+plugins: {
+  customReporter: {
+    enabled: true,
+    onTestFailed: (test, err) => console.log('FAIL', test.title, err.message),
+    onResult: result => {
+      // result.stats, result.tests
+    },
+  },
+}
+```
+
+Hooks: `onHookFinished`, `onTestBefore`, `onTestPassed`, `onTestFailed`, `onTestSkipped`, `onTestFinished`, `onResult`.
+
+For built-in Mocha reporters, use `--reporter`:
+
+```sh
+npx codeceptjs run --reporter dot
+```
+
+> The bundled `Mochawesome` helper was removed in 4.x. For an HTML report use the [Testomat.io Reporter](https://github.com/testomatio/reporter) HTML pipe (see above); for JUnit XML use the [`junitReporter`](#junit-xml) plugin. Wiring multiple Mocha reporters through `mocha-multi`/`cmr` is not recommended — prefer these instead.
+
+Plugins exist for [TestRail](https://www.npmjs.com/package/codeceptjs-testrail) and [Tesults](https://www.npmjs.com/package/codeceptjs-tesults).
+
+## CLI output
+
+By default CodeceptJS prints test names and failures. Add `--steps` to see each step, `--debug` for runner notices, or `--verbose` for full stack traces and events (use this when reporting bugs).
+
+```sh
+npx codeceptjs run --steps
+```
+
+`dry-run` lists tests and steps without running them:
+
+```sh
+npx codeceptjs dry-run --steps
+```