testeiya 0.2.0

package/skills/testeiya/testomatio-docs/docs/test-reporting/artifacts.md

@@ -0,0 +1,388 @@
+---
+title: Artifacts
+description: Understand how to manage and upload test artifacts in Testomat.io using external S3-compatible storage. This guide covers configuring public and private access to test artifacts, setting up an S3 bucket, and automating the upload of screenshots, videos, and other test-related files. Learn how to securely store and manage artifacts to improve test analysis and reporting.
+type: article
+url: https://docs.testomat.io/test-reporting/artifacts
+head:
+  - tag: meta
+    attrs:
+      name: keywords
+      content: Testomat.io, test artifacts, S3 storage, public access, private access, test management, test reports, automation, artifact upload, S3 configuration, QA tools
+---
+
+
+To inspect a failing test additional files (usually screenshots, videos or traces) can be attached to a test. These files are called test artifacts.
+
+Testomat.io Reporter uses a custom S3 bucket for artifacts which can be obtained from any S3 provider, like AWS, DigitalOcean and others. This makes artifacts storage to be independent from Testomat.io Application. In case you decide to stop using it, you still control your data. It is also up to you to clean old artifacts when you don't need them.
+
+By using external storage Testomat.io allows getting full control over how the storage is used. You can either clean up old test artifacts or contrary extend storage to store all history for all periods. S3 was chosen as a de-facto standard for file storage so all cloud providers support it. If you don't have S3 storage yet, you can purchase one from your favorite cloud provider and create a new bucket (storage space) on it. Testomat.io has no direct access to artifacts storage.
+
+## Set Up S3 Bucket
+
+To have test artifacts uploaded you need to create S3 Object Storage bucket on AWS, DigitalOcean, or Google Cloud Storage in interoperability mode.
+
+<Aside type="caution" title="Important">
+
+You need to obtain the following credentials: `ACCESS_KEY_ID`, `SECRET_ACCESS_KEY`, `BUCKET`, `REGION`, `ENDPOINT` (not required for AWS) to access S3 bucket. Then to to [Configuration](#configuration) section to enable S3 access.
+
+</Aside>
+
+## Overview
+
+Test artifacts can be uploaded with **public (default) or private** access to a bucket
+
+![Test Artifacts](./images/artifacts.png)
+
+1. Reporter sends a file to S3 storage and sends link back
+2. Testomat.io displays artifacts by the link in public mode or pre-signed links in private mode
+
+> Testomat.io Application won't bill you for stored artifacts, as they are stored in your bucket. If you don't have S3 bucket yet, it's not more than 5$ a month to purchase them one of the cloud providers. S3 was chosen as it is de-facto standard for file storage for cloud hosting. All popular hosting except Microsoft Azure support S3 protocol for storing files.
+
+If a test fails, a test runner may produce screenshot, video, or trace of a failed test. These files will be picked up by a reporter and uploaded to S3 bucket and attached to test.
+
+## Privacy
+
+### Public Access
+
+**By default tests artifacts are uploaded to bucket with `public-read` permission.**
+
+In this case uploaded files will be publicly accessible in Internet for any user via a permalink. These public links will be used by [Testomat.io Application](https://testomat.io) as well as [GitHub](./pipes/github.md#github-pipe) and [GitLab](./pipes/gitlab.md#gitlab-pipe) Pipes to display images.
+
+A bucket can be configured to disable the public listing of files, an arbitrary user can't access a file without knowing that permalink.
+
+Stored files will be served publicly via an appropriately configured ACL (Access Control List) or ACP (Access Control Policy).
+
+### Private Access
+
+If you need a more secure approach use **private access** mode. In this case, you will need to configure access credentials for Testomat.io, so it could access files inside a bucket.
+
+First, add `TESTOMATIO_PRIVATE_ARTIFACTS=1` environment value.
+Then provide the same S3 credentials in "Settings > Artifacts" section of a project, so [Testomat.io Application](https://testomat.io) could connect to the same bucket and fetch uploaded artifacts.
+
+Links to files will be pre-signed and expire automatically in 10 minutes.
+
+![Testomatio - Private Access](./images/Test_artifacts.png)
+
+> Test Artifacts settings are available only to managers and owners of a project!
+
+## Configuration
+
+There are two options for setting S3 bucket credentials:
+
+- via environment variables
+- by connecting to Testomat.io Application
+
+Recommended way is to set S3 bucket credentials as environment variables:
+
+- `S3_ACCESS_KEY_ID` - Access key.
+- `S3_SECRET_ACCESS_KEY` - Secret.
+- `S3_REGION` - Region your bucket lies.
+- `S3_BUCKET` - Bucket name.
+- `S3_ENDPOINT` - for providers other than AWS
+
+If you use Testomat.io Application, you can set those variables inside **Settings > Artifacts** page: enable "Share credentials" toggle to pass credentials into reporter and fill in S3 credentials into the displayed form.
+
+![Testomatio - Configuration](./images/Configuration.png)
+
+In this case Testomat.io Reporter will obtain S3 credentials for server and use them to save artifacts.
+
+Alternatively, you can configure reporter by using environment variables. If `S3_BUCKET` environment variable is defined reporter will start uploading test artifacts.
+
+We recommend storing S3 configuration in `.env` files when running tests locally and using job configuration when running on the Continuous Integration server.
+Please keep `S3_SECRET_ACCESS_KEY` in secret.
+
+```
+
+TESTOMATIO_PRIVATE_ARTIFACTS=1
+
+
+S3_ACCESS_KEY_ID=
+S3_SECRET_ACCESS_KEY=
+S3_REGION=
+S3_BUCKET=
+S3_ENDPOINT=
+
+```
+
+> `S3_ENDPOINT` is required only if you use S3 provider other than AWS
+
+To disable publishing of artifacts use `TESTOMATIO_DISABLE_ARTIFACTS=1` environment variable.
+
+Examples of upload configuration defined in environment variables are displayed below.
+
+## Providers
+
+### AWS
+
+Upload to AWS with private access mode:
+
+```bash
+TESTOMATIO_PRIVATE_ARTIFACTS=1
+S3_ACCESS_KEY_ID=11111111111111111111
+S3_SECRET_ACCESS_KEY=2222222222222222222222222222222222222222222
+S3_BUCKET=artifacts
+S3_REGION=us-west-1
+```
+
+To allow Testomat.io access stored files it is recommended to apply this policy to the bucket:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "VisualEditor0",
+      "Effect": "Allow",
+      "Action": ["s3:PutObject", "s3:GetObjectAcl", "s3:GetObject", "s3:GetObjectVersionAcl"],
+      "Resource": "arn:aws:s3:::<bucket_name>/*"
+    },
+    {
+      "Sid": "VisualEditor1",
+      "Effect": "Allow",
+      "Action": ["s3:ListBucket", "s3:PutBucketCORS", "s3:GetBucketAcl"],
+      "Resource": "arn:aws:s3:::<bucket_name>"
+    },
+    {
+      "Sid": "VisualEditor2",
+      "Effect": "Allow",
+      "Action": "s3:ListAllMyBuckets",
+      "Resource": "*"
+    }
+  ]
+}
+```
+
+If you use **Playwright** and you want to enable trace viewing ensure that CORS policy is enabled for the bucket:
+
+```
+aws s3api put-bucket-cors \
+    --bucket YOUR_BUCKET_NAME \
+    --cors-configuration '{
+    "CORSRules": [
+        {
+            "AllowedHeaders": ["*"],
+            "AllowedMethods": ["GET"],
+            "AllowedOrigins": ["https://app.testomat.io"],
+            "ExposeHeaders": ["Access-Control-Allow-Origin"],
+            "MaxAgeSeconds": 3000
+        },
+        {
+            "AllowedHeaders": ["*"],
+            "AllowedMethods": ["GET"],
+            "AllowedOrigins": ["https://trace.playwright.dev"],
+            "ExposeHeaders": ["Access-Control-Allow-Origin"],
+            "MaxAgeSeconds": 3000
+        }
+    ]
+}'
+```
+
+### DigitalOcean
+
+Upload to DigitalOcean in private access mode:
+
+```bash
+TESTOMATIO_PRIVATE_ARTIFACTS=1
+S3_ENDPOINT=https://ams3.digitaloceanspaces.com
+S3_ACCESS_KEY_ID=11111111111111111111
+S3_SECRET_ACCESS_KEY=2222222222222222222222222222222222222222222
+S3_BUCKET=artifacts
+S3_REGION=ams3
+```
+
+### Minio
+
+Upload to Minio in public access mode:
+
+```bash
+S3_ENDPOINT=http://company.storage.com
+S3_ACCESS_KEY_ID=minio
+S3_SECRET_ACCESS_KEY=minio123
+S3_BUCKET=testomatio
+S3_FORCE_PATH_STYLE=true
+```
+
+> It is important to add S3_FORCE_PATH_STYLE var for minio setup
+
+### Google Cloud Storage
+
+Google Cloud Storage can work via S3 protocol if **Interoperability mode** is enabled.
+
+![Testomatio - Google Cloud Storage](./images/Google_Cloud.png)
+
+Open Cloud Storage. Create a bucket for artifacts, then inside Settings select "Interoperability". Create `ACCESS_KEY` and `SECRET_ACCESS_KEY` for the current user.
+
+```bash
+S3_ENDPOINT=https://storage.googleapis.com
+S3_ACCESS_KEY_ID=11111111
+S3_SECRET_ACCESS_KEY=2222222
+S3_BUCKET=testomatio
+S3_REGION=us-east1
+```
+
+Please note, that you need to enable [Use Private URLs for Test Artifacts](https://docs.testomat.io/project/runs/reporter/artifacts/#privacy) in Testomat.io Project Settings if you configure bucket credentials on Testomat.io side.
+
+#### Cloudflare R2 and integration with Testomatio
+
+##### 1. Creating a Bucket in Cloudflare R2
+
+- **Cloudflare Dashboard:** [https://dash.cloudflare.com/](https://dash.cloudflare.com/)
+- **Steps:**
+
+  - Navigate to the **R2** section and select the option to create a new bucket.
+  - Choose a unique name and a region for the bucket.
+
+    ![Create a new bucket](./images/cloudr2-01.png)
+
+    ![Set bucket name](./images/cloudr2-02.png)
+
+## 2. Creating API Keys for the Bucket
+
+- **Purpose:** To ensure secure access to the bucket.
+- **Steps:**
+
+  - Go to the access management or API section.
+  - Generate a new API key with read/write permissions.
+  - Save and verify the generated key.
+
+    ![Generate API keys for bucket](./images/cloudr2-03.png)
+
+    ![Select Api variant](./images/cloudr2-04.png)
+
+    ![Create Api key](./images/cloudr2-05.png)
+
+    ![Set permissions for Api key](./images/cloudr2-06.png)
+
+    ![Copy Api key](./images/cloudr2-07.png)
+
+## 3. Connecting the Bucket to Testomatio
+
+- **Overview:** Integration with R2 streamlines testing and data management.
+- **Configuration:**
+
+  - In the Testomatio panel, enter the API keys and bucket address.
+  - Configure the paths and access parameters as required.
+
+    ![Set bucket credentials](./images/cloudr2-08.png)
+
+## Adding Artifacts
+
+### JavaScript Frameworks
+
+If you successfully configured `@testomatio/reporter` to work with the current test runner you can set up S3 credentials via environment variables.
+
+We recommend using [dotenv](https://www.npmjs.com/package/dotenv) library in development mode to store all config variables in a file. Create .env files in the same folder where a test configuration is stored and add require('dotenv').config(); to the top of the file:
+
+```js
+// config file
+require('dotenv').config();
+
+// ...
+```
+
+Testomat.io Reporter has built-in support and automatically uploads saved artifacts for the following test frameworks:
+
+- Playwright
+- CodeceptJS
+- Сypress
+- WebdriverIO
+
+If a screenshot, a video, or a trace was saved and attached to test, Testomat.io Reporter will automatically upload any of these as artifacts.
+
+Usually, artifacts are stored automatically for failed test cases. To send arbitrary files as artifacts use [`artifact`](./functions.md#artifact) method from Testomat.io functions `@testomatio/reporter` package.
+
+If you use other test runners or you want to add more files to test report you can add this via `@testomatio/reporter` API. To attach a file inside a test, add a filename of an existing file to `global.testomatioArtifacts` array:
+
+```js
+// inside a test:
+
+// attach a local file to a report
+global.testomatioArtifacts.push(fileName);
+
+// attach a picture inside a test
+
+global.testomatioArtifacts.push('img/file.png');
+
+// attach a picture and add a name to it
+global.testomatioArtifacts.push({ name: 'Screenshot', path: 'img/file.png' });
+```
+
+Artifacts will be uploaded for the current test when it is finished:
+
+If everything was configured correctly test artifacts will be uploaded on the next run.
+
+If S3 credentials are not set, artfact file will be ignored.
+
+### Other Frameworks
+
+For tests in **Java**, **Python**, **C#** and other languages, which are not (yet) integrated into Testomat.io Reporter, **use [JUnit format](./junit.md) for reporting**.
+
+To attach a file to a test as an artifact print the file name into console with `file://` prefix and upload XML report with Testomat.io.
+
+If S3 credentials are set, files will be uploaded to bucket and attached to test in a report.
+
+##### Java Example:
+
+Attaching a screenshot to the Java test
+
+```java
+// inside Java test
+// assume you created a screenshot
+// which is located at `pathToScreenshot`
+System.out.println("file://" + pathToScreenshot);
+```
+
+##### C# Example:
+
+Attaching a screenshot to the C# test
+
+```c#
+// assume you created a screenshot
+// which is located at `pathToScreenshot`
+Console.WriteLine("file://" + pathToScreenshot);
+```
+
+#### Python
+
+Attaching a screenshot to the Python test
+
+```python
+print("file://" + pathToScreenshot)
+```
+
+#### Ruby
+
+Attaching a screenshot to the Ruby test
+
+```python
+puts "file://" + path_to_screenshot
+```
+
+## Troubleshooting
+
+#### I don't have a S3 Bucket
+
+Well then, just get it. Even if your company doesn't provide one, you can purchase a S3 storage by yourself.
+
+#### Publishing Artifacts from Docker Container
+
+If your tests are running within Docker container pass all environment variables explicitly
+
+```
+docker run -e TESTOMATIO_PRIVATE_ARTIFACTS=1 \
+-e S3_ACCESS_KEY_ID=11111111111111111111 \
+-e S3_SECRET_ACCESS_KEY=2222222222222222222222222222222222222222222 \
+-e S3_BUCKET=artifacts \
+-e S3_REGION=us-west-1 \
+run-tests
+```
+
+#### Environment variables for S3 are not working
+
+This can be caused by various reasons. As an alternative approach, you can try to set S3 credentials inside Testomat.io Application and enable shared credentials.
+
+#### How to cleanup old artifacts?
+
+At this moment we don't provide any tools for the cleanup of files.
+It is recommended to write a custom cleanup tool on a S3 bucket.
+

package/skills/testeiya/testomatio-docs/docs/test-reporting/cli.md

@@ -0,0 +1,215 @@
+---
+title: CLI
+description: Learn how to use the Testomat.io Reporter CLI, a powerful command-line tool for managing and automating test runs. This guide covers installation and general usage, along with specific commands for starting, finishing, and executing test runs. It also explains how to parse XML reports and upload artifacts, making it a versatile tool for integrating Testomat.io into CI/CD pipelines.
+type: article
+url: https://docs.testomat.io/test-reporting/cli
+head: 
+  - tag: meta
+    attrs:
+      name: keywords
+      content: Testomat.io Reporter, CLI, command-line tool, test runs, managing test runs, CI/CD pipelines, test reporting, parse XML reports, upload artifacts, @testomatio/reporter, API key, environment variables, JUnit, NUnit, xUnit, TRX, S3 artifacts configuration, test runner, test automation, software testing
+---
+
+The Testomat.io Reporter CLI is a powerful tool for managing test runs, parsing XML reports, and uploading artifacts. CLI can be used to start and finish test runs, run tests, parse XML reports, and upload artifacts. It can be used in CI/CD pipelines or locally.
+
+Reporter is designed to work with [Testomat.io](https://testomat.io) service but not exclusively
+
+## Installation
+
+```bash
+npm install -g @testomatio/reporter
+```
+
+## General Usage
+
+```bash
+npx @testomatio/reporter <command> [options]
+```
+
+## Available Commands
+
+### 1. start
+
+Starts a new test run and returns its ID. This requires an API key to be set in the `TESTOMATIO` environment variable.
+
+**Usage:**
+
+```bash
+npx @testomatio/reporter start [options]
+```
+
+**Environment Variables:**
+
+- `TESTOMATIO`: Your Testomat.io API project key in format: `tstmt_*` (required).
+
+**Options:**
+
+- `--env-file <envfile>`: Load environment variables from a specific env file. If none specified, it will look for `.env` file.
+
+> Previously known as: `npx start-test-run --launch` _(before 1.6.0)_
+
+### 2. finish
+
+Finishes a run by its ID.
+
+**Usage:**
+
+```bash
+npx @testomatio/reporter finish [options]
+```
+
+**Environment Variables:**
+
+- `TESTOMATIO`: Your Testomat.io API key in format: `tstmt_*` (required).
+
+**Options:**
+
+- `--env-file <envfile>`: Load environment variables from a specific env file. If none specified, it will look for `.env` file.
+
+> Previously known as: `npx start-test-run --finish` _(before 1.6.0)_
+
+### 3. run
+
+Creates a run, and runs tests with the specified command, then finishes the run. This command ensures that if test runner spawns parallel workers or processes and reports it will report to the same run.
+
+> Use `npx @testomatio/reporter run` if multiple run reports are created on the same launch
+
+**Usage:**
+
+```bash
+npx @testomatio/reporter run [options]
+```
+
+**Environment Variables:**
+
+- `TESTOMATIO`: Your Testomat.io API key in format: `tstmt_*` (required).
+
+**Options:**
+
+- `-c, --command <cmd>`: Test runner command (required).
+- `--filter <filter>`: [Filter executed tests](./pipes/testomatio.md#filter-tests) by tag, label, jira, plan.
+- `--env-file <envfile>`: Load environment variables from a specific env file.
+
+**Examples:**
+
+```bash
+npx @testomatio/reporter run "npm test"
+npx @testomatio/reporter run "npx jest" --filter "testomatio:tag=frontend"
+npx @testomatio/reporter run "npx jest" --filter "testomatio:label=Smoke"
+npx @testomatio/reporter run "npx jest" --filter "testomatio:jira=TC-123"
+npx @testomatio/reporter run "npx jest" --filter "testomatio:plan=a123fb12"
+npx @testomatio/reporter run "mocha tests/" --env-file .env.test
+```
+
+> Previously known as: `npx start-test-run -c "command"` _(before 1.6.0)_
+
+### 4. xml
+
+Parses XML reports in JUnit (NUnit, xUnit, TRX) XML format and uploads them to Testomat.io or uses GitHub / GitLab / ButBucket pipe to create a mini-report
+
+**Usage:**
+
+```bash
+npx @testomatio/reporter xml <pattern> [options]
+```
+
+**Environment Variables:**
+
+- `TESTOMATIO`: Your Testomat.io API key in format: `tstmt_*` if you want to upload reports to Testomat.io (optional).
+
+**Arguments:**
+
+- `<pattern>`: XML file pattern (required).
+
+**Options:**
+
+- `-d, --dir <dir>`: Project directory.
+- `--java-tests [java-path]`: Load Java tests from path (default: src/test/java).
+- `--lang <lang>`: Language used (python, ruby, java).
+- `--timelimit <time>`: Default time limit in seconds to kill a stuck process.
+- `--env-file <envfile>`: Load environment variables from a specific env file.
+
+**Examples:**
+
+```bash
+npx @testomatio/reporter xml "test-results/**.xml" --lang python
+npx @testomatio/reporter xml "junit-reports/**.xml" -d ./project --lang java
+npx @testomatio/reporter xml "pytest-results.xml" --timelimit 300 --env-file .env.test
+```
+
+> Previously known as: `npx report-xml` _(before 1.6.0)_
+
+### 5. upload-artifacts
+
+Testomat.io reporter automatically uploads artifacts during run. However, either some artifacts failed to upload or you intentioanlly disabled file upload during tests to speed up reporting. In this case you can use this command to upload artifacts after the run.
+
+It is important to have the `TESTOMATIO_RUN` environment variable set to the run ID.
+
+**Usage:**
+
+```bash
+npx @testomatio/reporter upload-artifacts [options]
+```
+
+**Environment Variables:**
+
+- `TESTOMATIO`: Your Testomat.io API key (required).
+- `TESTOMATIO_RUN`: The previous run ID you want to upload artifacts (optional). If none set, latest run will be used.
+
+**Options:**
+
+- `--force`: Re-upload artifacts even if they were uploaded before.
+- `--env-file <envfile>`: Load environment variables from a specific env file.
+
+You still need [S3 artifacts configuration](./artifacts.md) to be set to upload artifacts to storage. In order to disable artifacts upload during tests you can use `TESTOMATIO_DISABLE_ARTIFACTS=1` while running tests.
+
+**Examples:**
+
+```bash
+npx @testomatio/reporter upload-artifacts
+npx @testomatio/reporter upload-artifacts --force
+npx @testomatio/reporter upload-artifacts --env-file .env.prod
+```
+
+With Playwright:
+
+```bash
+TESTOMATIO=tstmt_* TESTOMATIO_DISABLE_ARTIFACTS=1 npx playwright test
+TESTOMATIO=tstmt_* npx @testomatio/reporter upload-artifacts
+```
+
+With webdriverio:
+
+```bash
+TESTOMATIO=tstmt_* TESTOMATIO_DISABLE_ARTIFACTS=1 npx @testomatio/reporter run "npx wdio"
+TESTOMATIO=tstmt_* npx @testomatio/reporter upload-artifacts
+```
+
+You can also upload small artifacts during the run, while large files can be uploaded after. For instance, all files uploading larger than 10MB will be skipped during the run.
+
+```bash
+TESTOMATIO=tstmt_* TESTOMATIO_ARTIFACT_MAX_SIZE_MB=10 npx playwright test
+TESTOMATIO=tstmt_* npx @testomatio/reporter upload-artifacts
+```
+
+However, `upload-artifacts` command will upload all files after the run, without blocking the final result.
+
+## Environment Variables
+
+Many commands rely on environment variables. You can set these in a command line, in a `.env` file, or use the `--env-file` option to specify a custom env file. Important variables include:
+
+- `TESTOMATIO`: Your Testomat.io API key.
+- `TESTOMATIO_RUN`: The current run ID (usually set automatically by the `start` command).
+- `TESTOMATIO_TITLE`: Title for the test run (optional).
+- [more..](./configuration.md)
+
+## Tips
+
+1. Always ensure your Testomat.io API key is properly set before running commands.
+2. Use the `--env-file` option when you have different configurations for various environments.
+3. The `run` command is versatile and can be used with various test runners like Jest, Mocha, or custom scripts.
+4. When using the `xml` command, ensure your XML reports are in a supported format.
+5. The `upload-artifacts` command is useful for adding additional files or logs to your test runs.
+
+For more detailed information or troubleshooting, please refer to the Testomat.io documentation or contact support.
+