testeiya 0.2.0

package/skills/testeiya/testomatio-docs/docs/test-reporting/frameworks.md

@@ -0,0 +1,490 @@
+---
+title: NodeJS Test Frameworks
+description: The Testomat.io framework integration guide explains how to implement different testing frameworks and reporting tools. It details configuration options for test reporting, integrating frameworks like Mocha, Jasmine, and Jest, and the generation of reports from automated tests. Additionally, the guide covers debugging, error handling, and customizing test reports to fit specific project needs.
+type: article
+url: https://docs.testomat.io/test-reporting/frameworks
+head:
+  - tag: meta
+    attrs:
+      name: keywords
+      content: Testomat.io, testing frameworks, automated testing, test reports, Mocha, Jasmine, Jest, integration, debugging, error handling, test automation
+---
+## JavaScript Frameworks
+
+- [CodeceptJS](#codeceptjs)
+- [Cucumber](#cucumber)
+- [Cucumber Java](#cucumber-java)
+- [Cypress](#cypress)
+- [Detox](#detox)
+- [Java Frameworks](#java-frameworks)
+- [Jest](#jest)
+- [JUnit](#junit)
+- [Mocha](#mocha)
+- [Newman/Postman](#newman)
+- [Playwright](#playwright)
+- [Protractor](#protractor)
+- [Selenide](#selenide)
+- [TestCafe](#testcafe)
+- [Vitest](#vitest)
+- [WebdriverIO](#webdriverio)
+
+Testomat.io reporter is NodeJS package that can be applied to all popular JavaScript test runners.
+
+### CodeceptJS
+
+> 📐 When used with [Testomat.io Application](https://app.testomat.io) it is recommended to import automated tests first via [check-tests](https://github.com/testomatio/check-tests#cli). To create items on the fly set `TESTOMATIO_CREATE=1` env variable.
+
+Add plugin to [codecept conf](https://github.com/testomatio/reporter/blob/master/example/codecept/codecept.conf.js#L23):
+
+```javascript
+plugins: {
+  testomatio: {
+    enabled: true,
+    require: '@testomatio/reporter/lib/adapter/codecept',
+  }
+}
+```
+
+Run the following command from you project folder:
+
+```bash
+TESTOMATIO={API_KEY} npx codeceptjs run
+```
+
+> 🖼 Screenshots of failed tests and videos (for Playwright helper) will be automatically uploaded as [Artifacts](./artifacts.md)
+
+#### CodeceptJS Parallel Run
+
+If tests run parallel, like workers in CodeceptJS use `@testomatio/reporter run` command to get proper reports:
+
+```bash
+TESTOMATIO={API_KEY} npx @testomatio/reporter run 'npx codeceptjs run-workers 2'
+```
+
+> Specify a command to run as argument to `@testomatio/reporter run`
+
+Use `--env-file <envfile>` option to load environment variables from .env file. Inside env file TESTOMATIO credentials like `TESTOMATIO` api key or [S3 config for artifacts](./artifacts.md).
+
+Command `@testomatio/reporter run` is used to initiate a single run report before all workers are started. Each worker will report to the same Run, and after all workers and codeceptjs finishes, this will finish the run report.
+
+> 📑 [Example Project](https://github.com/testomatio/examples/tree/master/codeceptJS) | 🗄 [CodeceptJS API Example](https://github.com/testomatio/examples/tree/master/codeceptJSApi) | 🥒 [CodeceptJS Cucumber Example](https://github.com/testomatio/examples/tree/master/codeceptjs-cucumber)
+
+> 📺 [Video](https://www.youtube.com/watch?v=f_pCe3wPRPs)
+
+### Playwright
+
+> 📐 When used with [Testomat.io Application](https://app.testomat.io) it is recommended to import automated tests first via [check-tests](https://github.com/testomatio/check-tests#cli). To create items on the fly set `TESTOMATIO_CREATE=1` env variable.
+
+Add a reporter to Playwright config:
+
+```javascript
+reporter: [
+  ['list'],
+  [
+    '@testomatio/reporter/lib/adapter/playwright.js',
+    {
+      apiKey: process.env.TESTOMATIO,
+    },
+  ],
+];
+```
+
+Run the following command from you project folder:
+
+```bash
+TESTOMATIO={API_KEY} npx playwright test
+```
+
+> 🖼 Screenshots of failed tests and videos will be automatically uploaded as [Artifacts](./artifacts.md)
+
+> 📑 [Example Project](https://github.com/testomatio/examples/tree/master/playwright) | 🥒 [Playwright + Cucumber Example](https://github.com/testomatio/examples/tree/master/playwright-cucumber-js)
+
+### Cypress
+
+> 📐 When used with [Testomat.io Application](https://app.testomat.io) it is recommended to import automated tests first via [check-tests](https://github.com/testomatio/check-tests#cli). To create items on the fly set `TESTOMATIO_CREATE=1` env variable.
+
+<details>
+  <summary> For Cypress <code>< 10.0.0</code> <i>(click to expand)</i></summary>
+  Register `cypress-plugin` in `cypress/plugins/index.js`:
+
+```javascript
+const testomatioReporter = require('@testomatio/reporter/lib/adapter/cypress-plugin');
+
+/**
+ * @type {Cypress.PluginConfig}
+ */
+module.exports = (on, config) => {
+  // `on` is used to hook into various events Cypress emits
+  // `config` is the resolved Cypress config
+
+  testomatioReporter(on, config);
+
+  return config;
+};
+```
+
+</details>
+
+For Cypress >= `10.0.0` use `setupNodeEvents` in `cypress.config.js(ts)`
+
+```javascript
+setupNodeEvents(on, config) {
+  return require('@testomatio/reporter/lib/adapter/cypress-plugin')(on, config)
+}
+```
+
+Run the following command from you project folder:
+
+```bash
+TESTOMATIO={API_KEY} npx cypress run
+```
+
+> 🖼 Screenshots of failed tests and videos will be automatically uploaded as [Artifacts](./artifacts.md)
+
+> 📑 [Example Project](https://github.com/testomatio/examples/tree/master/cypress) | 🥒 [Cypress + Cucumber Example](https://github.com/testomatio/examples/tree/master/cypress-cucumber)
+
+### Mocha
+
+> 📐 When used with [Testomat.io Application](https://app.testomat.io) it is recommended to import automated tests first via [check-tests](https://github.com/testomatio/check-tests#cli). To create items on the fly set `TESTOMATIO_CREATE=1` env variable.
+
+Run the following command from you project folder:
+
+```bash
+mocha --reporter ./node_modules/@testomatio/reporter/lib/adapter/mocha.js --reporter-options apiKey={API_KEY}
+```
+
+> 📑 [Example Project](https://github.com/testomatio/examples/tree/master/mocha-ts-multi-reporters)
+
+### Jest
+
+> 📐 When used with [Testomat.io Application](https://app.testomat.io) it is recommended to import automated tests first via [check-tests](https://github.com/testomatio/check-tests#cli). To create items on the fly set `TESTOMATIO_CREATE=1` env variable.
+
+Add the following line to [jest.config.js](https://github.com/testomatio/reporter/blob/master/example/jest/jest.config.js#L100):
+
+```javascript
+reporters: ['default', ['@testomatio/reporter/lib/adapter/jest.js', { apiKey: process.env.TESTOMATIO }]],
+```
+
+Run tests:
+
+```
+TESTOMATIO={API_KEY} npx jest
+```
+
+> **Warning**
+> Do not use `bail` option in your jest config or testrun script. (It cause issues with updating testrun status).
+
+> 📑 [Example Project](https://github.com/testomatio/examples/tree/master/jest)
+
+> 📺 [Video](https://www.youtube.com/watch?v=RKfIfnEuGys)
+
+### Vitest
+
+When used with [Testomat.io](https://app.testomat.io) it is recommended to import automated tests first via [check-tests](https://github.com/testomatio/check-tests#cli). To create items on the fly set `TESTOMATIO_CREATE=1` env variable.
+
+1. Install Testomatio reporter:
+
+```bash
+npm install @testomatio/reporter --save-dev
+```
+
+2. Add Testomatio reporter to [vitest.config.ts](https://vitest.dev/config/)
+
+```typescript
+// import reporter
+import TestomatioReporter from '@testomatio/reporter/lib/adapter/vitest';
+
+export default defineConfig({
+  test: {
+    // add Testomatio reporter to reporters list
+    reporters: ['verbose', new TestomatioReporter()],
+  },
+});
+```
+
+> You may also pass your API key
+
+```typescript
+new TestomatioReporter({ apiKey: process.env.TESTOMATIO });
+```
+
+or pass it later via environment variable when running tests.
+
+3. Run tests:
+
+```bash
+TESTOMATIO={API_KEY} npx vitest
+```
+
+> 📑 [Example Project](https://github.com/testomatio/examples/tree/master/vitest)
+
+Vitest reporter has some limitations:
+
+- it does not support [testomatio functions](./functions.md) (but supports adding console.logs and [vitest metadata](https://vitest.dev/advanced/metadata) to report)
+- tests can't be uploaded "on the fly", they sent to Testomat.io only after the run is finished (but vitest is supposed to be used as unit test framework, so your run should not take much time)
+
+### WebdriverIO
+
+> 📐 When used with [Testomat.io Application](https://app.testomat.io) it is recommended to import automated tests first via [check-tests](https://github.com/testomatio/check-tests#cli). To create items on the fly set `TESTOMATIO_CREATE=1` env variable.
+
+Add the following lines to [wdio.conf.js](https://webdriver.io/docs/configurationfile/):
+
+```javascript
+const testomatio = require('@testomatio/reporter/lib/adapter/webdriver');
+
+exports.config = {
+  // ...
+  reporters: [
+    [testomatio, {
+      apiKey: $ {
+        process.env.TESTOMATIO
+      }
+    }]
+  ]
+}
+```
+
+For making screenshots on failed tests add the following hook to `wdio.conf.js`:
+
+```js
+    afterTest: function (test, context, { error, result, duration, passed, retries }) {
+        if (error) {
+            browser.takeScreenshot()
+        }
+    },
+```
+
+Run the following command from you project folder:
+
+```bash
+TESTOMATIO={API_KEY} npx @testomatio/reporter run 'npx wdio wdio.conf.js'
+```
+
+> 📑 [Example Project](https://github.com/testomatio/examples/tree/master/webdriverio-mocha)
+
+> 📺 [Video](https://www.youtube.com/watch?v=cjVZzey-lto)
+
+### Cucumber
+
+> 📐 When used with [Testomat.io Application](https://app.testomat.io) it is recommended to import feature files via [check-cucumber](https://github.com/testomatio/check-cucumber).
+
+Run the following command from you project folder:
+
+```bash
+TESTOMATIO={API_KEY} npx cucumber-js --format ./node_modules/@testomatio/reporter/lib/adapter/cucumber.js
+```
+
+> **Note**
+> If you use Cucumber with Playwright, Cypress, CodeceptJS, please refer to corresponding framework. This reference is required only if you run tests via `cucumber-js` CLI.
+
+> 📑 [Example Project](https://github.com/testomatio/examples/tree/master/cucumber)
+
+> 📺 [Video](https://www.youtube.com/watch?v=qf83AtII-LI)
+
+### TestCafe
+
+> 📐 When used with [Testomat.io Application](https://app.testomat.io) it is recommended to import automated tests first via [check-tests](https://github.com/testomatio/check-tests#cli). To create items on the fly set `TESTOMATIO_CREATE=1` env variable.
+
+Run the following command from you project folder:
+
+```bash
+TESTOMATIO={API_KEY} npx testcafe chrome -r testomatio
+```
+
+### Newman
+
+To report Newman tests a separate package is required:
+
+```bash
+npm i newman-reporter-testomatio --save-dev
+```
+
+> **Note** > `newman` and `newman-reporter-testomatio` should be installed in the same directory. If you run your tests using globally installed newman (`newman run ...`), intall `newman-reporter-testomatio` globally too (`npm i newman-reporter-testomatio -g`). If you use locally installed newman (within the project) (`npx newman run ...`), install `newman-reporter-testomatio` locally (`npm i newman-reporter-testomatio`).
+> You can verify installed packages via `npm list` or `npm list -g`.
+
+Run collection and specify `testomatio` as reporter:
+
+```bash
+TESTOMATIO={API_KEY} npx newman run {collection_name.json} -r testomatio
+```
+
+> 📑 [Example Project](https://github.com/testomatio/examples/tree/master/newman)
+
+### Detox
+
+> 📐 When used with [Testomat.io Application](https://app.testomat.io) it is recommended to import automated tests first via [check-tests](https://github.com/testomatio/check-tests#cli). To create items on the fly set `TESTOMATIO_CREATE=1` env variable.
+
+Run detox tests sptcifying configuration name:
+
+```
+TESTOMATIO={API_KEY} npx detox test -c {configuration_name}
+```
+
+> **Warning**
+> Do not use `bail` option in your jest config or testrun script. (It cause issues with updating testrun status).
+
+### Protractor
+
+> 📐 When used with [Testomat.io Application](https://app.testomat.io) it is recommended to import automated tests first via [check-tests](https://github.com/testomatio/check-tests#cli). To create items on the fly set `TESTOMATIO_CREATE=1` env variable.
+
+Add the following lines to [conf.js](https://github.com/angular/protractor/blob/5.4.1/example/conf.js):
+
+```javascript
+const JasmineReporter = require('@testomatio/reporter/lib/adapter/jasmine');
+
+exports.config = {
+  onPrepare: () => {
+    jasmine.getEnv().addReporter(new JasmineReporter({ apiKey: process.env.TESTOMATIO }));
+  },
+};
+```
+
+Run the following command from you project folder:
+
+```bash
+TESTOMATIO={API_KEY} npx @testomatio/reporter run 'npx protractor conf.js'
+```
+
+> 📑 [Example Project](https://github.com/testomatio/examples/tree/master/protractor)
+
+## Java Frameworks
+
+> 📐 This section describes reporting into [Testomat.io Application](https://app.testomat.io)
+
+Reporting from Java Frameworks is done via JUnit XML report. Install `@testomatio/reporter` NodeJS package to process reports:
+
+```
+npm init -y
+npm install @testomatio/reporter --save-dev
+```
+
+JUnit XML report can be created by test framework you use:
+
+### JUnit
+
+If you run JUnit tests via Maven you can use [Surefire Report Plugin](https://maven.apache.org/surefire/maven-surefire-report-plugin/usage.html) to generate JUnit XML report.
+
+For example, with Surefire Report Plugin in Maven, you can add the following configuration to your project's `pom.xml`:
+
+```xml
+<build>
+    <plugins>
+        <plugin>
+            <groupId>org.apache.maven.plugins</groupId>
+            <artifactId>maven-surefire-plugin</artifactId>
+            <version>3.0.0-M5</version>
+        </plugin>
+    </plugins>
+</build>
+```
+
+In this case JUnit XML will be saved into `target/surefire-reports/`
+
+So you can import reports to Testomat.io by running:
+
+```
+TESTOMATIO={API_KEY} npx @testomatio/reporter xml "target/surefire-reports/**.xml" --java-tests
+```
+
+> **Note**
+> If your tests are located in a folder other other than `src/test/java`, specify a path to test files using `--java-tests` option: `--java-tests="path/to/tests"`
+
+This will import test reports into Testomat.io. If a reporter can access source code of Java tests, the source code will also be imported into Testomat.io
+
+#### Test IDS
+
+It is possible to attach reported tests to the current tests in Testomat.io by their IDs. Copy Test ID of a test you want to match, and put it as a comment into a Java test you want to import.
+
+In this example, we added ID as a comment to `negativeNumbersCanBeSubtracted` test:
+
+```java
+  @Test
+  public void negativeNumbersCanBeSubtracted() throws Exception {
+      // @T8acca9eb
+      assertThat(calc.Calculate(-1.0, -3.0, "-"), equalTo(2.0));
+  }
+```
+
+To make this feature work, please ensure that source code of Java tests is accessible to `npx @testomatio/reporter xml` command, use `--java-tests` option to specify the correct path. To check if source code of tests is available run reporter with DEBUG mode:
+
+```
+DEBUG=@testomatio/reporter:* TESTOMATIO={API_KEY} npx @testomatio/reporter xml "target/surefire-reports/**.xml" --java-tests
+```
+
+Is a source code is not available, test IDs can be set from output. To set Test ID to a test, copy test ID and print it from a test:
+
+```java
+System.out.println("tid://" + TID);
+```
+
+For example, if your test id is `@T8acca9eb` you can print it:
+
+```java
+System.out.println("tid://@T8acca9eb");
+```
+
+#### Artifacts
+
+Screenshots or videos from tests are uploaded if test contains output with a path to file of following format:
+
+```
+file://path/to/screenshot.png
+```
+
+Use `System.out.println` to print an absulute path to file that should be uploaded as a screenshot.
+
+```java
+System.out.println("file://" + pathToScreenshot);
+```
+
+This will produce XML report which contains path to a file:
+
+```xml
+<testcase>
+  <system-out><![CDATA[
+    file://path/to/scrrenshot.png
+  ]]></system-out>
+</testcase>
+```
+
+When XML report is uploaded, all files from `file://` will be uploaded to corresponding tests.
+
+> 🖼 Read more how [Artifacts](./artifacts.md) work
+
+### Selenide
+
+Please refer to [JUnit](#junit) if you use JUnit as a test runner for Selenide tests.
+However, it is important to note, that Selenide automatically adds artifacts into JUnit reports printing them as `file://` into XML report. This means that no code changes should be made to publish artifacts to Testomat.io.
+
+### Cucumber Java
+
+If you use Java version of Cucumber Java you should import your feature files first using [check-cucumber](https://www.npmjs.com/package/check-cucumber).
+Provide a path to directory containing feature files by using `-d` option:
+
+```
+TESTOMATIO={API_KEY} npx check-cucumber -d features
+```
+
+It is recommended to set Test IDs for Cucumber scenarios to make reports match imported tests.
+
+```
+TESTOMATIO_TITLE_IDS=1 TESTOMATIO={API_KEY} npx check-cucumber -d features --update-ids
+```
+
+We use `--update-ids` option to write test IDs obtained from Testomat.io into source code
+Also we use `TESTOMATIO_TITLE_IDS=1` to write test IDs into scenario titles instead of scenario tags. This is important so JUnit report would contain test IDs.
+
+To generate JUnit reports, you can use the built-in Cucumber JUnit plugin. When you run your Cucumber tests the JUnit reports will be generated in the default directory (`build/reports/tests/test`) in XML format.
+
+To submit report to Testomat.io use `npx @testomatio/reporter xml` command from `@testomatio/reporter` NodeJS package:
+
+```
+TESTOMATIO={API_KEY} npx @testomatio/reporter xml "build/reports/tests/**/*.xml"
+```
+
+If you want to have artifacts attached, use `System.out.println` to print an absulute path to file that should be uploaded as a screenshot.
+
+```java
+System.out.println("file://" + pathToScreenshot);
+```
+

package/skills/testeiya/testomatio-docs/docs/test-reporting/functions.md

@@ -0,0 +1,144 @@
+---
+title: Functions
+description: Explore Testomat.io's flexible reporting functions to enhance your test reports. Learn to use functions like artifact, log, and step for adding files, logging additional information, and describing test actions. Integrates seamlessly with frameworks like Playwright, Jest, and Mocha, supporting custom reporting needs.
+type: article
+url: https://docs.testomat.io/test-reporting/functions
+head:   
+  - tag: meta
+    attrs:
+      name: keywords
+      content: Testomat.io, test reporting functions, artifact, log, step, Playwright, Jest, Mocha, CodeceptJS, test automation, test customization
+---
+Testomat functions gives you more flexibility in reporting and make your reports more powerful.
+
+### Usage example
+
+```javascript
+import testomat from '@testomatio/reporter'; // or const testomat = require('@testomatio/reporter');
+
+test('my test', async () => {
+  testomat.artifact('path/to/file');
+});
+```
+
+Or import only required functions:
+
+```javascript
+import { artifact, log, meta } from '@testomatio/reporter'; // or const { artifact, log } = require('@testomatio/reporter');
+
+test('my test', async () => {
+  meta('ISSUE', 'MY-123');
+  await page.login();
+  log`I was logged in with user ${user}`;
+  artifact(await saveScreenshot());
+  assert(something);
+});
+```
+
+After you import and invoke `testomat`, autocompletion will help you to find the right function.
+
+### Available functions
+
+- [artifact](#artifact)
+- [log](#log)
+- [step](#step)
+- [meta (key:value)](#meta)
+
+### Artifact
+
+Adds file to the test report (text, image, video, etc.)
+
+```javascript
+import { artifact } from '@testomatio/reporter';
+
+test('my test', async () => {
+  const pathToFile = await saveScreenshot();
+  artifact(pathToFile);
+});
+```
+
+> Artifacts, generated by testrunners (like screenshots/videos by Playwright) are uploaded automatically, you don't need to add them manually using artifact function. Just setup [S3 bucket](./artifacts.md).
+
+### Log
+
+Similar to [step](#step) function, intended to log any additional info to the test report (including text, arrays, complex objects).
+
+##### Usage examples:
+
+```javascript
+testomat.log('your message');
+testomat.log(`your message ${variable}`);
+testomat.log('your message', variable1, variable2);
+```
+
+```javascript
+const testomat = require('@testomatio/reporter');
+test('Your test @T12345678', async () => {
+  await page.login(user);
+  testomat.log`I was logged in with user ${user}`;
+  assert(loggedIn);
+
+  log`I was logged in with user ${user}`; // < shorter syntax
+});
+```
+
+### Step
+
+Adds step to the test report. Step is a human-readable description of the test action. It is used to describe the test flow. This function is similar to [log](#log) function, but looks differently in the report.
+
+```javascript
+const testomat = require('@testomatio/reporter');
+describe('Your suite @S12345678', () => {
+  test('Your test @T12345678', async () => {
+    await page.login();
+    testomat.step`Login successful`;
+    assert(something);
+  });
+});
+```
+
+### Meta
+
+Meta information is a key:value pair(s), which is used to add additional information to the test report. E.g. browser, environment, etc.
+
+```js
+import { meta } from '@testomatio/reporter';
+
+test('my test', () => {
+  // use it inside tests as key, value
+  meta('browser', 'chrome');
+
+  // or use it as an object
+  meta({
+    browser: 'chrome',
+    server: 'staging',
+  });
+})
+```
+
+Or in CommonJS style:
+
+```javascript
+const { meta } = require('@testomatio/reporter');
+
+test('Your test @T12345678', async () => {
+  await page.login();
+  testomat.meta({
+    browser: 'chrome',
+    testType: 'smoke',
+  });
+  assert(something);
+});
+```
+
+---
+
+Supported frameworks:
+
+- 🟢 CodeceptJS
+- 🟢 Cucumber
+- 🟢 Jest
+- 🟢 Mocha
+- 🟢 Playwright
+- 🟢 WDIO (everything, except artifacts)
+

package/skills/testeiya/testomatio-docs/docs/test-reporting/index.mdx

@@ -0,0 +1,48 @@
+---
+title: Testomat.io Test Reporting
+description: The Testomat.io Reporter is a Node.js package designed to send test results directly to Testomat.io. It supports real-time reports, artifact uploads, and logs for Node.js frameworks like Jest and Playwright. It can also generate reports from JUnit files for other languages like Java, PHP, and Ruby. The package is configured via environment variables and integrates with CI workflows.
+type: article
+url: https://docs.testomat.io/test-reporting/overview
+head:
+  - tag: meta
+    attrs:
+      name: keywords
+      content: Testomat.io, Node.js, Jest, Playwright, JUnit, test reporting, real-time reports, CI integration, test artifacts, automated testing, test management
+---
+
+import DirectoryLinks from '/src/components/DirectoryLinks.astro';
+
+<DirectoryLinks directory="test-reporting" />
+
+Testomat.io reporter is NodeJS package that sends test results to [app.testomat.io](https://app.testomat.io)
+
+:::note
+Testomatio Reporter is a standalone package that can be used without Testomat.io application.
+[Open it on GitHub](https://github.com/testomatio/reporter)
+:::
+
+## Installation
+
+To enable Testomat.io Reporter install `@testomatio/reporter` package
+
+
+Use one of your favorite package managers:
+
+```bash
+npm install @testomatio/reporter --save-dev
+```
+
+```bash
+pnpm install @testomatio/reporter --save-dev
+```
+
+```bash
+yarn add @testomatio/reporter --dev
+```
+
+Reporter works differenttely depending on your current setup: 
+
+* If you use **NodeJS** framework like Jest, Playwright, etc, you get realtime reports, automatic artifacts uploads, logs. [Learn how to set up frameworks reporter](./frameworks.md) 
+* If you use **Java, Ruby, PHP, or other languages**, reports are created from JUnit file. [Learn how to set up JUnit reporter](./junit.md)
+
+Reporter is configured through environment variables. [Learn more about configuration](./pipes/testomatio.md).