codeceptjs 4.0.0-rc.21 → 4.0.0-rc.22

package/docs/plugins.md

@@ -569,6 +569,49 @@ More config options are available:
 
 *   `config`   (optional, default `{}`)
 
+## junitReporter
+
+Generates a JUnit-compatible XML report after a test run.
+
+Unlike Mocha's `mocha-junit-reporter`, this plugin understands CodeceptJS steps and substeps.
+For every `<testcase>` it includes:
+
+*   `<properties>` — the test's meta information: every `meta` key from `Scenario('...', { meta })`, plus its `tags` and `retries`
+*   `<system-out>` — an indented step/substep log (substeps are nested under their meta step); only failed steps are marked
+*   `<failure>` — for failed tests: the error message, type, stack trace and (optionally) the step trace
+
+The produced file is consumable by Jenkins, GitLab CI, CircleCI, GitHub Actions test reporters, etc.
+
+#### Configuration
+
+```js
+"plugins": {
+   "junitReporter": {
+     "enabled": true
+   }
+ }
+```
+
+Possible config options:
+
+*   `outputName`: file name for the report. Default: `report.xml`.
+*   `output`: directory where the report is stored, relative to the project root. Default: the `output` directory.
+*   `testGroupName`: value of the `name` attribute on the root `<testsuites>` element. Default: `CodeceptJS`.
+*   `attachMeta`: add the test's meta information (`meta` keys, `tags`, `retries`) as `<properties>`. Default: true.
+*   `attachSteps`: add the step/substep log as `<system-out>`. Default: true.
+*   `stepsInFailure`: append the step trace to the `<failure>` body. Default: true.
+
+CLI examples:
+
+    npx codeceptjs run -p junitReporter
+    npx codeceptjs run -p junitReporter:outputName=junit.xml
+
+> ℹ When running with `run-workers`, steps are serialized between processes and substep nesting is flattened.
+
+### Parameters
+
+*   `config` **any**  (optional, default `{}`)
+
 ## pageInfo
 
 Collects information from web page after each failed test and adds it to the test as an artifact.

package/docs/reports.md

@@ -5,479 +5,180 @@ title: Reporters
 
 # Reporters
 
-## Cli
+CodeceptJS prints test results to the console by default (see [CLI output](#cli-output)). For an HTML report, a pull-request comment, JUnit XML, or a hosted dashboard, it is recommeded to use **[Testomat.io Reporter](https://github.com/testomatio/reporter)**. It sends results to whichever destinations you turn on with steps, screenshots, videos, traces, and logs.
 
-By default, CodeceptJS provides cli reporter with console output.
-Test names and failures will be printed out on screen.
+### Install
 
 ```sh
-GitHub --
- ✓ search in 2577ms
- ✓ signin in 2170ms
- ✖ register in 1306ms
-
--- FAILURES:
-
-  1) GitHub: register:
-      Field q not found by name|text|CSS|XPath
-
-  Scenario Steps:
-
-  - I.fillField("q", "aaa") at examples/github_test.js:29:7
-  - I.fillField("user[password]", "user@user.com") at examples/github_test.js:28:7
-  - I.fillField("user[email]", "user@user.com") at examples/github_test.js:27:7
-  - I.fillField("user[login]", "User") at examples/github_test.js:26:7
-
-
-
-  Run with --verbose flag to see NodeJS stacktrace
-
-```
-
-output steps use `--steps` option:
-
-```
-npx codeceptjs run --steps
-```
-
-Output:
-
-```sh
-GitHub --
- search
- • I am on page "https://github.com"
- • I am on page "https://github.com/search"
- • I fill field "Search GitHub", "CodeceptJS"
- • I press key "Enter"
- • I see "Codeception/CodeceptJS", "a"
- ✓ OK in 2681ms
-
- signin
- • I am on page "https://github.com"
- • I click "Sign in"
- • I see "Sign in to GitHub"
- • I fill field "Username or email address", "something@totest.com"
- • I fill field "Password", "123456"
- • I click "Sign in"
- • I see "Incorrect username or password.", ".flash-error"
- ✓ OK in 2252ms
-
- register
- • I am on page "https://github.com"
-   Within .js-signup-form:
-   • I fill field "user[login]", "User"
-   • I fill field "user[email]", "user@user.com"
-   • I fill field "user[password]", "user@user.com"
-   • I fill field "q", "aaa"
- ✖ FAILED in 1260ms
-```
-
-To get additional information about test execution use `--debug` option.
-
-```
-npx codeceptjs run --debug
-```
-
-This will show execution steps
-as well as notices from test runner. To get even more information with more technical details like error stack traces,
-and global promises, or events use `--verbose` mode.
-
-```
-npx codeceptjs run --verbose
-```
-
-```sh
-GitHub --
- register
-   [1] Starting recording promises
-   Emitted | test.before
- > WebDriver._before
-   [1] Queued | hook WebDriver._before()
-   [1] Queued | amOnPage: https://github.com
-   Emitted | step.before (I am on page "https://github.com")
- • I am on page "https://github.com"
-   Emitted | step.after (I am on page "https://github.com")
-   Emitted | test.start ([object Object])
-...
-```
-
-Please use verbose output when reporting issues to GitHub.
-
-### Dry Run
-
-There is a way to list all tests and their steps without actually executing them. Execute tests in `dry-run` mode to see all available tests:
-
-```
-npx codeceptjs dry-run
-```
-
-Output:
-
-```
-Tests from /home/davert/projects/codeceptjs/examples:
-
-Business rules --
-  ☐ do something
-Google --
-  ☐ test @123
-GitHub -- /home/davert/projects/codeceptjs/examples/github_test.js
-  ☐ Visit Home Page @retry
-  ☐ search @grop
-  ☐ signin @normal @important @slow
-  ☐ signin2
-  ☐ register
-
-  Total: 3 suites | 7 tests
-
---- DRY MODE: No tests were executed ---
-```
-
-Pass `--steps` or `--debug` option as in `run` command to also get steps and substeps to be printed. In this mode **tests will be executed** but all helpers and plugins disabled, so no real actions will be performed.
-
-```
-npx codecepjs dry-run --debug
+npm install @testomatio/reporter --save-dev
 ```
 
-> ℹ If you use custom JavaScript code inside tests, or rely on values from `grab*` commands, dry-run may produce error output.
-
-## Testomat.io
-
-[Testomat.io](https://testomat.io) is a modern test management tool focused on CodeceptJS and **created by CodeceptJS team**.
-Testomat.io is commercial SaaS service that can receive run reports from local runs or CI. Out of box Testomat.io supports parallel runs, uploading of screenshots and videos.
-
-![](https://user-images.githubusercontent.com/220264/151728836-b52d2b2b-56e1-4640-8d3a-b39de817b1fd.png)
-
-> 😻 **Testomat.io is free** for small teams, so you can use its reporting features with CodeceptJS.
-
-To receive run reports you should:
-
-- [Sign up](https://app.testomat.io/users/sign_up) at Testomat.io
-- Create a new "Classical" project (select "BDD" project if you use CodeceptJS in BDD mode)
-- Select "Import from Source Code"
-- Select "CodeceptJS" as testing framework and JavaScript or TypeScript as a language. If you use BDD select "Gherkin" as language.
-- Execute provided command in a terminal with your project. This will be "check-tests" or "check-cucmber" command. It scans all your test files and imports them into Testomat.io. This way all your e2e tests will be visible in one UI.
-- After tests are imported, go to Runs tab and select "Setup automated tests".
-- Follow the instructions:
-
-![image](https://user-images.githubusercontent.com/77803888/151834217-5da44d92-a59a-458d-8856-64ce61bf3a38.png)
-
-- You will need to install `@testomatio/reporter` package and enable it as a plugin in codeceptjs config:
+Enable reporter plugin:
 
 ```js
+// codecept.conf.js
 plugins: {
   testomatio: {
     enabled: true,
-    require: '@testomatio/reporter/lib/adapter/codecept',
-    apiKey: process.env.TESTOMATIO,
-  }
-}
-```
-
-- Run tests with `TESTOMATIO=` env variable and API key provided by Testomat.io
-- See the run report is created and updated in realtime.
-
-[Testomat.io](https://testomat.io) reporter works in the cloud, so it doesn't require you to install additional software. It can be integrated with your CI service to rerun only failed tests, launch new runs from UI, and send report notifications by email or in Slack, MS Teams, or create issue in Jira.
-
-## ReportPortal
-
-For enterprise grade we reporting we recommend using [ReportPortal](https://reportportal.io).
-
-![](https://camo.githubusercontent.com/6550c0365f1d0ce1e29c53f1860b12957d1fc529/68747470733a2f2f692e6962622e636f2f516d353247306e2f53637265656e73686f742d323031392d30342d31312d61742d31352d35372d34302e706e67)
-
-[ReportPortal](https://reportportal.io) is open-source self-hosted service for aggregating test execution reports.
-Think of it as Kibana but for test reports.
-
-Use official [CodeceptJS Agent for ReportPortal](https://github.com/reportportal/agent-js-codecept/) to start publishing your test results.
-
-## XML
-
-Use default xunit reporter of Mocha to print xml reports. Provide `--reporter xunit` to get the report to screen.
-It is recommended to use more powerful [`mocha-junit-reporter`](https://www.npmjs.com/package/mocha-junit-reporter) package
-to get better support for Jenkins CI.
-
-Install it via NPM (locally or globally, depending on CodeceptJS installation type):
-
-```sh
-npm i mocha-junit-reporter
-```
-
-Additional configuration should be added to `codecept.conf.js` to print xml report to `output` directory:
-
-```json
-  "mocha": {
-    "reporterOptions": {
-        "mochaFile": "output/result.xml"
-    }
-  },
-```
-
-Execute CodeceptJS with JUnit reporter:
-
-```sh
-codeceptjs run --reporter mocha-junit-reporter
-```
-
-Result will be located at `output/result.xml` file.
-
-Alternatively, the reporter name can be added to the configuration file as well. In this case, CodeceptJS can be executed without additional CLI options.
-
-```json
-  "mocha": {
-    "reporter": "mocha-junit-reporter",
-    "reporterOptions": {
-        "mochaFile": "output/result.xml"
-    }
+    require: '@testomatio/reporter/codecept',
   },
+}
 ```
 
-```sh
-codeceptjs run
-```
+### Enable an output
 
-## Html
+Each output turns on when you set its environment variable. Run your tests as usual — one run feeds every output you enabled.
 
-### Testomat.io HTML Reporter
+| To get… | Set | Details |
+| --- | --- | --- |
+| HTML report | `TESTOMATIO_HTML_REPORT_SAVE=1` | [HTML Report](#html-report) |
+| Markdown report | `TESTOMATIO_MARKDOWN_REPORT_SAVE=1` | [Markdown Report](#markdown-report) |
+| Run Result on [app.testomat.io](https://testomat.io) | `TESTOMATIO` (project API key) | [Cloud Report](#cloud-report) |
+| A comment on a GitHub Pull Request | `GH_PAT` (`${{ github.token }}` in Actions) | [GitHub Report](#github-report) |
+| A comment on a GitLab Merge Request | `GITLAB_PAT` (token with `api` scope) | [GitLab Report](#gitlab-report) |
+| A comment on a Bitbucket Pull Request | `BITBUCKET_ACCESS_TOKEN` (repo access token) | [Bitbucket Report](#bitbucket-report) |
 
-For modern HTML reports, use **[@testomatio/reporter](https://github.com/testomatio/reporter)** package.
+Screenshots and videos in these reports are uploaded to your own storage — see [Artifacts](#artifacts).
 
-#### Installation
+Put the variables on CI when running tests:
 
-```sh
-npm install @testomatio/reporter --save-dev
+```yaml
+- run: npx codeceptjs run
+  env:
+    TESTOMATIO_HTML_REPORT_SAVE: 1                  # → output/reports/testomatio-report.html
+    TESTOMATIO_HTML_REPORT_FOLDER: output/reports   # keep it with the rest of output/
+    GH_PAT: ${{ github.token }}                     # → PR comment
+    # TESTOMATIO: ${{ secrets.TESTOMATIO }}         # → testomat.io run
+- uses: actions/upload-artifact@v4
+  if: always()
+  with:
+    name: codeceptjs-output
+    path: output/
 ```
 
-#### Configuration
+The GitHub pipe also needs the job to grant `permissions: pull-requests: write`.
 
-Add the `testomatio` plugin to your `codecept.conf.js`:
 
-```js
-export const config = {
-  plugins: {
-    testomatio: {
-      enabled: true,
-      require: '@testomatio/reporter/lib/adapter/codecept',
-    },
-  },
-}
-```
+### HTML Report
 
-#### Usage
+A single self-contained HTML file with the run summary and, per test, its steps, screenshots, logs, and error. It needs no API key and no service, so it works anywhere — open it locally or attach it to a CI build.
 
-Generate HTML reports with environment variable:
+![HTML report](https://raw.githubusercontent.com/testomatio/reporter/master/docs/pipes/images/html-pipe.png)
 
-```sh
-TESTOMATIO_HTML_REPORT_SAVE=1 npx codeceptjs run
-```
+- `TESTOMATIO_HTML_REPORT_SAVE=1` — enable the report
+- `TESTOMATIO_HTML_REPORT_FOLDER=output/reports` — keep it inside CodeceptJS's `output/` dir (default folder is `html-report`)
+- `TESTOMATIO_HTML_FILENAME` — file name, must end in `.html` (default `testomatio-report.html`)
 
-Report is saved to `html-report/testomatio-report.html`.
+### Cloud Report
 
-#### Configuration Options
+Sends the run to [app.testomat.io](https://testomat.io) — a hosted dashboard with run history, flaky-test detection, parallel-run merging, re-running failed tests, and notifications. Free for small teams.
 
-Customize report location:
+![Testomat.io report](https://user-images.githubusercontent.com/220264/151728836-b52d2b2b-56e1-4640-8d3a-b39de817b1fd.png)
 
-```sh
-# Custom folder
-TESTOMATIO_HTML_REPORT_SAVE=1 TESTOMATIO_HTML_REPORT_FOLDER=./reports npx codeceptjs run
+- `TESTOMATIO` — project API key; enables the pipe
+- `TESTOMATIO_CREATE=1` — create tests in Testomat.io that were not imported beforehand
+- `TESTOMATIO_TITLE` — report title
+- `TESTOMATIO_RUNGROUP_TITLE` — add the run to a group (e.g. `"Build ${BUILD_ID}"`)
+- `TESTOMATIO_PUBLISH=1` — make the report publicly accessible
 
-# Custom filename
-TESTOMATIO_HTML_REPORT_SAVE=1 TESTOMATIO_HTML_FILENAME=my-report.html npx codeceptjs run
+More options (shared runs, rungroups, run management): [Testomat.io pipe](https://github.com/testomatio/reporter/blob/master/docs/pipes/testomatio.md).
 
-# With Testomat.io cloud
-TESTOMATIO_HTML_REPORT_SAVE=1 TESTOMATIO=your_api_key npx codeceptjs run
-```
+To view artifacts on cloud they must be uploaded to S3 storages. Images from [`screenshot`](/plugins#screenshot) plugin, videos from the [`screencast`](/plugins#screencast) plugin (or the Playwright helper's `video` and `trace`). Can be used with any S3 provider: AWS S3, Cloudflare R2, Google Cloud Storage (interoperability mode), DigitalOcean Spaces, MinIO. 
 
-For more details, see [documentation](https://docs.testomat.io/test-reporting/pipes/html/).
-### Mochawesome
+### GitHub Report
 
-Best HTML reports could be produced with [mochawesome](https://www.npmjs.com/package/mochawesome) reporter.
+Posts a comment to the Pull Request: run status, pass/fail/skip counts, stack traces of the failures, screenshots, and the slowest tests. Re-running the workflow replaces the previous comment.
 
-![mochawesome](/img/mochawesome.png)
+![GitHub report](https://raw.githubusercontent.com/testomatio/reporter/master/docs/pipes/images/github.png)
 
-Install it via NPM:
+- `GH_PAT` — GitHub token; `${{ github.token }}` works in Actions
+- the job must grant `permissions: pull-requests: write`
+- `GH_KEEP_OUTDATED_REPORTS=1` — keep previous comments instead of deleting them
 
-```sh
-npm i mochawesome
-```
+### GitLab Report
 
-If you get an error like this
+Posts a comment to the Merge Request with the same summary. It needs Merge Request context, so run it in merge-request pipelines.
 
-```sh
-"mochawesome" reporter not found
+![GitLab report](https://raw.githubusercontent.com/testomatio/reporter/master/docs/pipes/images/gitlab.png)
 
-invalid reporter "mochawesome"
-```
+- `GITLAB_PAT` — Personal or Project Access Token with `api` scope
+- run in merge-request pipelines (`$CI_PIPELINE_SOURCE == "merge_request_event"`)
+- `GITLAB_KEEP_OUTDATED_REPORTS=1` — keep previous comments
+- `GITLAB_REMOVE_ALL_OUTDATED_REPORTS=1` — remove all previous comments, not just the latest
 
-Make sure to have mocha installed or install it:
+### Bitbucket Report
 
-```sh
-npm i mocha -D
-```
+Posts a comment to the Pull Request with the same summary. Comments are created only in `pull-requests` pipelines.
 
-Configure it to use `output` directory to print HTML reports:
+![Bitbucket report](https://raw.githubusercontent.com/testomatio/reporter/master/docs/pipes/images/bitbucket.png)
 
-```json
-  "mocha": {
-    "reporterOptions": {
-        "reportDir": "output"
-    }
-  },
-```
-
-Execute CodeceptJS with mochawesome reporter:
-
-```sh
-codeceptjs run --reporter mochawesome
-```
+- `BITBUCKET_ACCESS_TOKEN` — repository access token with `Pull requests: Write` and `Repository: Read`
+- run in `pull-requests` pipelines
+- `BITBUCKET_KEEP_OUTDATED_REPORTS=1` — keep previous comments
 
-Result will be located at `output/index.html` file.
+### Markdown Report
 
-Alternatively, the reporter name can be added to the configuration file as well. In this case, CodeceptJS can be executed without additional CLI options.
+A single self-contained Markdown file — renders in PR comments, CI job summaries, and Slack, and is convenient for AI agents reading test results. Needs no API key.
 
-```json
-  "mocha": {
-    "reporter": "mochawesome",
-    "reporterOptions": {
-        "reportDir": "output"
-    }
-  },
-```
+- `TESTOMATIO_MARKDOWN_REPORT_SAVE=1` — enable the report
+- `TESTOMATIO_MARKDOWN_REPORT_FOLDER=output/reports` — keep it inside CodeceptJS's `output/` dir (default folder is `md-report`)
+- `TESTOMATIO_MARKDOWN_FILENAME` — file name, must end in `.md` (default `testomatio-report.md`)
+- `TESTOMATIO_TITLE` — document title (default `Test Results`)
 
-```sh
-codeceptjs run
-```
+On GitHub Actions, append it to the job summary: `cat output/reports/testomatio-report.md >> "$GITHUB_STEP_SUMMARY"`.
 
-### Advanced usage
+## JUnit XML
 
-Want to have screenshots for failed tests?
-Then add Mochawesome helper to your config:
+For CI servers that read JUnit XML (Jenkins, GitLab CI, CircleCI, the GitHub Actions test tab), enable the [`junitReporter`](/plugins#junitreporter) plugin. It writes `output/report.xml` with CodeceptJS steps included — unlike `mocha-junit-reporter`.
 
-```json
-  "helpers": {
-    "Mochawesome": {
-        "uniqueScreenshotNames": "true"
-    }
-  },
+```js
+plugins: {
+  junitReporter: { enabled: true },
+}
 ```
 
-Then tests with failure will have screenshots.
-
-### Configuration
+Options (`outputName`, `output`, `testGroupName`, `attachMeta`, `attachSteps`, `stepsInFailure`): [plugin docs](/plugins#junitreporter).
 
-This helper should be configured in codecept.conf.ts
-
-- `uniqueScreenshotNames` (optional, default: false) - option to prevent screenshot override if you have scenarios with the same name in different suites. This option should be the same as in common helper.
-- `disableScreenshots` (optional, default: false) - don't save screenshot on failure. This option should be the same as in common helper.
+## ReportPortal
 
-Also if you will add Mochawesome helper, then you will able to add custom context in report:
+[ReportPortal](https://reportportal.io) is an open-source self-hosted dashboard for test reports. Publish with the [CodeceptJS Agent for ReportPortal](https://github.com/reportportal/agent-js-codecept/).
 
-#### addMochawesomeContext
+## Custom reporter
 
-Adds context to executed test in HTML report:
+The [`customReporter`](/plugins#customreporter) plugin hooks into test events:
 
 ```js
-I.addMochawesomeContext('simple string')
-I.addMochawesomeContext('http://www.url.com/pathname')
-I.addMochawesomeContext('http://www.url.com/screenshot-maybe.jpg')
-I.addMochawesomeContext({
-  title: 'expected output',
-  value: {
-    a: 1,
-    b: '2',
-    c: 'd',
+plugins: {
+  customReporter: {
+    enabled: true,
+    onTestFailed: (test, err) => console.log('FAIL', test.title, err.message),
+    onResult: result => {
+      // result.stats, result.tests
+    },
   },
-})
-```
-
-##### Parameters
-
-- `context` string, url, path to screenshot, object. See [this](https://www.npmjs.com/package/mochawesome#adding-test-context)
-
-## Multi Reports
-
-Want to use several reporters in the same time? Try to use [mocha-multi](https://www.npmjs.com/package/mocha-multi) reporter
-
-Install it via NPM:
-
-```sh
-npm i mocha-multi
+}
 ```
 
-Configure mocha-multi with reports that you want:
-
-```json
-  "mocha": {
-    "reporterOptions": {
-      "codeceptjs-cli-reporter": {
-        "stdout": "-",
-        "options": {
-          "verbose": true,
-          "steps": true,
-        }
-      },
-      "mochawesome": {
-        "stdout": "./output/console.log",
-        "options": {
-          "reportDir": "./output",
-          "reportFilename": "report"
-        }
-      },
-      "mocha-junit-reporter": {
-        "stdout": "./output/console.log",
-        "options": {
-          "mochaFile": "./output/result.xml",
-          "attachments": true //add screenshot for a failed test
-        }
-      }
-    }
-  }
-```
+Hooks: `onHookFinished`, `onTestBefore`, `onTestPassed`, `onTestFailed`, `onTestSkipped`, `onTestFinished`, `onResult`.
 
-Execute CodeceptJS with mocha-multi reporter:
+For Mocha reporters, use `--reporter`:
 
 ```sh
-npx codeceptjs run --reporter mocha-multi
+npx codeceptjs run --reporter mochawesome --reporter-options reportDir=output
 ```
 
-This will give you cli with steps in console and HTML report in `output` directory.
+Plugins exist for [TestRail](https://www.npmjs.com/package/codeceptjs-testrail) and [Tesults](https://www.npmjs.com/package/codeceptjs-tesults).
 
-Alternatively, the reporter name can be added to the configuration file as well. In this case, CodeceptJS can be executed without additional CLI options.
+## CLI output
 
-```json
-  "mocha": {
-    "reporter": "mocha-multi",
-    "reporterOptions": {
-      ...
-    }
-  }
-```
+By default CodeceptJS prints test names and failures. Add `--steps` to see each step, `--debug` for runner notices, or `--verbose` for full stack traces and events (use this when reporting bugs).
 
 ```sh
-codeceptjs run
-```
-
-## Testrail
-
-Testrail integration with CodeceptJS is now so seamless. The test run is created automatically afterwards. The screenshots of failed tests are also attached to test results.
-
-Try to use [codeceptjs-testrail](https://www.npmjs.com/package/codeceptjs-testrail) plugin
-
-Install it via NPM:
-
-```sh
-npm i codeceptjs-testrail --save
+npx codeceptjs run --steps
 ```
 
-![Attachemnt for failed case](http://g.recordit.co/ajaa2QRlnW.gif)
-
-Now there is new feature, add the configuration to test run of test plan
-![Attachemnt for failed case](http://g.recordit.co/uQLvQUq7cT.gif)
-
-## Tesults
-
-Submit test results data from CodeceptJS to [Tesults](https://www.tesults.com) easily with the [codeceptjs-tesults](https://www.npmjs.com/package/codeceptjs-tesults) plugin. Test results data is submitted automatically after a test run completes.
+`dry-run` lists tests and steps without running them:
 
 ```sh
-npm i codeceptjs-tesults --save
+npx codeceptjs dry-run --steps
 ```
-
-Once installed, follow the [quick and easy integration instructions](https://www.tesults.com/docs/codeceptjs) to get setup in no time.