testeiya 0.2.0

package/skills/testeiya/testomatio-docs/docs/project/runs/running-automated-tests.md

@@ -0,0 +1,256 @@
+---
+title: Running Automated Tests
+description: Learn how to run automated tests using Testomat.io. This guide walks through the process of importing tests, generating run reports, and configuring automated test frameworks. It also covers advanced reporting features like parallel test execution, managing stack traces, and handling terminated runs for optimized test management and reporting.
+type: article
+url: https://docs.testomat.io/project/runs/running-automated-tests
+head:
+  - tag: meta
+    attrs:
+      name: og:image
+      content: https://docs.testomat.io/_astro/image-7.BWXewamn_Z27T85y.webp
+
+  - tag: meta
+    attrs:
+      name: keywords
+      content: automated tests, test reporting, test management, parallel testing, CI/CD, stack trace, Playwright, Testomat.io, test import, run reports, test automation frameworks
+---
+
+Testomat.io can receive and store test run reports from various test frameworks.
+You can use Testomat.io as a test management system or as a rich reporting tool.
+
+Depending on how you plan to use it we can look from different perspectives:
+
+- In case of **Test Management System**, the key entity is a Test. So before reporting tests it is **required to import your tests** first. Test has its history and a lifecycle, so each test report will be attached to a corresponding test in a project. If a reported test doesn't exist, you will see message "Tests Not Matched". By default, new tests are not created from Run report, to avoid duplicates, or cases when you accidentally reported wrong tests.
+
+- In case of **Reporting Tool**, you are more focused on getting reports rather than managing tests. In this case you may not need test history, so **importing tests is not required**. You can run tests with `TESTOMATIO_CREATE=1` option enabled, and all tests will be created from a test result.
+
+If your project contains only automated tests, you may prefer using Testomat.io as reporting tool only. However, to unleash the full power of Testomat.io we recommend using it as a test management system, that means keep tests synchronized with the codebase.
+
+![Alt text](./images/image-7.png)
+
+### Why Do I need to Import my Tests?
+
+As described in a section above, Testomat.io require tests to be imported first in order to synchonize them wth the codebase and to keep track the history of tests between reports. If test was not found in a project it is marked as "Unmatched" in report. We use this notice to avoid duplicates or accidentally added tests. Read more about [importing tests](../import-export/import-tests-from-source-code.md).
+
+![Unnatched tests](./images/unmatched_tests.png)
+
+This rule doesn't apply to test reported via JUnit XML. In this case all tests from XML file are created automatically on report.
+
+### How to Receive Run Report
+
+You have already [imported automated tests](https://docs.testomat.io/getting-started/import-tests-from-source-code/) and you wonder what's next? Testomat.io will help you to generate a run command to use on your machine or on your CI. As a result, you will get human-readable Run Reports with lots of additional information.
+
+Testomat.io has a friendly UI that helps to set up automated test reporting. So there is no need to learn all technical documentation in the very first steps. Let's see how it works.
+
+### Launch Automated Tests
+
+First, go to the Runs page, click on the Extra button and pick **Setup Automated**
+
+![Import automated tests](./images/2023-08-04_23.00.40@2x.png)
+
+Then you need to pick your framework from the dropdown.
+
+![Choose test automation framework](./images/2023-08-04_23.02.20@2x.png)
+
+Testomat.io will generate instructions regarding your framework. You may need to make few changes to your code, namely, add the Testomat.io plugin.
+
+![Make configuration seetings](./images/2023-08-04_23.05.14@2x-1.png)
+
+Please note that you need to start generated commands in your terminal from your project folder.
+
+![Execute test cases](./images/2023-08-04_23.11.43@2x.png)
+
+> Also, you can add advanced options to your command to enable extra options. For example, you can give a title to your report by passing it as environment variable to `TESTOMATIO_TITLE` or can add environments to run by providing `TESTOMATIO_ENV`.
+> Learn more about [advanced reporting options here](https://docs.testomat.io/reference/reporter/#advanced-usage).
+
+If you have successfully launched your automated tests, a new Test Run will appear on Runs page.
+
+![Automated test run](./images/2023-08-04_23.13.11@2x.png)
+
+You can see tests results in real-time in the Run Report.
+
+![Simplified test report](./images/2023-08-04_23.21.08@2x.png)
+
+As soon as all tests are completed you can check Run Report with details.
+
+![Detailed test report](./images/2023-08-04_23.21.47@2x.png)
+
+### Adding Multi-Select Suite IDs to CI Settings
+
+With Testomat.io you have the ability to select and pass **multiple suite IDs to CI** when creating a new run. This option is available in the New Run view as a multi-select input, allowing you to define which test suites should be executed as part of a run.
+
+The multi-select is implemented as a dropdown, where you can choose the required suites by their IDs. The selected suite IDs are then passed to your CI system.
+
+**To use this feature, you need:**
+
+1. Adjust your CI workflow by adding `grep by suites` - (Learn more about [Environment Configuration](https://docs.testomat.io/integrations/continuous-integration/#environment-configuration)).
+
+![Testomat.io - Multi-Select Suite IDs](./images/suite_id_ci_1.png)
+
+2. Update your Testomat CI configuration to accept and process the passed suite IDs.
+
+![Testomat.io - Multi-Select Suite IDs](./images/suite_id_ci_2.png)
+
+When everything is configured, previously added suites will be available under **'Suites'** dropdown list in the New Run view.
+
+![Testomat.io - Multi-Select Suite IDs](./images/suite_id_ci_3.png)
+
+:::note
+
+When setting up automated tests, selecting a suite linked to a CI configuration restricts further selection to only those suites within that same CI configuration.
+
+![Testomat.io - Multi-Select Suite IDs](./images/suite_id_ci_4.gif)
+
+:::
+
+## Advanced Reporting
+
+Testomat.io reporter can be configured to add additional information for Run report. For instance, you can specify:
+
+- run title
+- rungroup
+- environemnt
+
+[Learn more of all possible options](../runs/reporter/pipes/testomatio.md).
+
+### Reporting Parallel Tests
+
+When you enable reporing for tests running in parallel, you might end with multiple reports per each executed process. There are few options to deal with this case, which you can use depending on your setup.
+
+#### **Strategy 1: Use shared run**
+
+In this case multiple independent launches will report data to the report matched by the same Run title.
+
+![Shared run strategy](./images/image-12.png)
+
+Pick the unique name for this run and use `TESTOMATIO_SHARED_RUN=1` environement variable to enable shared report:
+
+```
+TESTOMATIO_SHARED_RUN=1 TESTOMATIO_TITLE="UniqTitleForThisBuild" <actual run command>
+```
+
+For instance, if you run tests on CI as a title you can use pipeline/workflow ID:
+
+```
+TESTOMATIO_TITLE="Pipeline $CI_PIPELINE_ID" TESTOMATIO_SHARED_RUN=1 <actual run command>
+```
+
+If you prefer you can use Git commit as unique identifier:
+
+```
+TESTOMATIO_TITLE="Commit $GIT_COMMIT" TESTOMATIO_SHARED_RUN=1 <actual run command>
+```
+
+Please refer to documentation of your CI system and pick the variable which and be unique to all runs of this build. This approach **fits perfectly for sharded tests when you run tests on different jobs, different containers, different machines**.
+
+We recommend to also append some more info into the `TESTOMATIO_TITLE`
+
+#### **Strategy 2: Use @testomatio/reporter run**
+
+Run tests via `npx @testomatio/reporter run` command:
+
+```
+npx @testomatio/reporter run "<actual run command>"
+```
+
+Under hood `@testomatio/reporter run` creates a new empty run and passes its ID as environment variable into all spawned processes. So no matter how many parallel processes are started they will report to the single Run report.
+
+![Reporter run strategy](./images/image-10.png)
+
+However, this might not work in all cases.
+
+> If you use sharding (running tests on multiple machines) you should use strategy 1 or 3.
+
+#### **Strategy 3: Manually create and close run**
+
+In this case you create a run, receive its ID and manually close it after all runs are finished.
+
+![Manual run strategy](./images/image-9.png)
+
+Create a run via `@testomatio/reporter start`:
+
+```
+export TESTOMATIO_RUN=$(TESTOMATIO=xxx npx @testomatio/reporter start | tail -1)
+```
+
+then execute tests passing the `TESTOAMTIO_PROCEED` variable:
+
+```
+TESTOMATIO=xxx TESTOMATIO_RUN=$RUN_ID TESTOMATIO_PROCEED=1 <actual-run-command>
+```
+
+Once tests are finished close the run with `@testomatio/reporter finish`:
+
+```
+TESTOMATIO=xxx npx @testomatio/reporter finish
+```
+
+If you have a complex pipeline, you can start Run on the stage #1, execute tests in parallel on stage #2, and close the run on stage #3.
+
+### Terminated Runs
+
+Sometimes, during test automation, unexpected issues may arise, or a test can be stopped for various reasons.
+
+![Terminated test run](./images/terminated-test-run.png)
+
+For example, during the execution of the problematic test case, the some gateway becomes unresponsive due to a server issue. This issue was unforeseen and not within the control of the testing team. Testomat.io detects the problem and initiates a termination of the problematic test case. The custom timeout you defined (min 30 minutes) comes into play. If the test case does not complete within this time frame, it is terminated automatically.
+
+So you can terminate test runs without causing disruptions and you can set custom timeouts for terminated runs.
+
+![setup terminated timeout](./images/setup-terminated.png)
+
+### Stack Traces
+
+Testomat.io provides detailed information about the active stack frames during the execution of a program.
+
+It extracts URLs from user logs and convert them into clickable links within the test result view, making it easy for users to access the URLs directly from the test results.
+
+![Testomat.io - Clickable links in automated tests report](./images/New_fE0hyJn9_2024-07-30.png)
+
+Stacktrace lets you search for text and regular expressions. This powerful tool enables you to quickly locate specific data within extensive logs or codebases, saving you valuable time and effort. By using text searches, you can easily pinpoint exact matches, while regular expressions allow for more complex queries.
+
+![Testomat.io - Stacktrace search](./images/New_FJSHQ5EX_2024-08-17.gif)
+
+## Running Manual and Automated Tests
+
+Testomat.io has a separate category for working with tests called **Mixed**. It is a combination of automated and manual tests. You can create a **Mixed Plan** and run manual and automated tests in parallel.
+
+To start the Mixed Run, you need to do a few things:
+
+- Tests need to have **IDs**. This can be done by adding the `--update-ids` option when importing tests.
+
+- You need to configure **Continuous Integration**. To learn how to set up CI in Testomat.io, visit the [dedicated page](https://docs.testomat.io/usage/continuous-integration/).
+
+- Create a **Mixed Plan** that contains both automated and manual tests. You can learn how to create a Mixed Plan by visiting the page [dedicated page](https://docs.testomat.io/getting-started/test-plans/).
+
+To get started, open the **Runs** page and select **Mixed Run** in the menu.
+
+![Testomat.io - Start the Mixied Run](./images/New_G4KYCW6P_2024-09-15.png)
+
+Next, name the run (optional), select a Profile (this is your configured CI), select a pre-created Mixed Plan and click on the **Launch** button.
+
+![Testomat.io - Set Up the Mixied Run](./images/New_DSZjGZmH_2024-09-15.png)
+
+While you are running manual tests, your automated tests are running in parallel.
+
+![Testomat.io - Run Tests](./images/New_rzMWPX3a_2024-09-15.png)
+
+![Testomat.io - GitHub Actions](./images/New_bwcoXKfZ_2024-09-15.png)
+
+As a result, you receive the combined report:
+
+![Testomat.io - Mixed Run Report](./images/New_cHhvni5r_2024-09-15.png)
+
+## How To Enable Playwright Trace Viewer
+
+Traces in Playwright refer to detailed logs or recordings of browser interactions during a test script's execution. They capture a chronological sequence of events, including browser actions, network requests, and other relevant information.
+
+Here are steps how to enable Playwright trace viewer for uploaded artifacts in Test Run:
+
+1. setup S3 Bucket (see [dedicated documentation](https://docs.testomat.io/usage/test-artifacts#set-up-s3-bucket))
+2. enable third-party cookies in your browser
+3. run your tests
+4. click on a test
+5. click on the trace.zip
+
+![open playwright trace viewer](./images/Open-Playwright-Trace.gif)

package/skills/testeiya/testomatio-docs/docs/project/runs/running-manual-and-automated-tests.md

@@ -0,0 +1,331 @@
+---
+title: Running Manual and Automated Tests
+description: Learn how to execute manual and automated tests together using Mixed Run. This guide explains execution types (CI or local), launch flows from different pages, configuration options, and how to manage and complete mixed test runs.
+type: article
+url: https://docs.testomat.io/project/runs/running-manual-and-automated-tests
+head:
+  - tag: meta
+    attrs:
+      name: keywords
+      content: manual testing, automated tests, mixed run, mixed plan, mixed tests, test plans, manual test execution, automated test execution, multi-environment testing, QA testing, CI, pipelines, CLI
+---
+
+**Mixed Run** allows teams to execute manual and automated tests together within a single execution session, combining results into one unified report.
+
+It is designed for real-world QA workflows where releases require validation across both automated pipelines and manual verification.
+
+Key Benefits:
+
+- Execute manual and automated testing in one place
+- Unified reporting across execution types
+- Real-time visibility into release readiness
+- Flexible execution via CI pipelines or local CLI
+- Supports exploratory and structured testing flows
+- Centralized results for stakeholders and QA teams
+
+In a Mixed Run:
+
+- Manual tests are executed directly in the Testomat.io interface
+- Automated tests run via CI or locally using CLI
+- All results are placed into a single report
+
+## Execution Types
+
+When launching a Mixed Run, you must choose **how automated tests will be executed**.
+
+Mixed Run supports two execution types.
+
+| Execution Type           | Automated Tests           | Requirements                  |
+| ------------------------ | ------------------------- | ----------------------------- |
+| **Mixed Run on CI**      | Executed in a CI pipeline | CI profile must be configured |
+| **Mixed Run without CI** | Executed locally via CLI  | No CI setup required          |
+
+### Mixed Run on CI
+
+Automated tests are executed through a configured **CI pipeline**.
+
+Characteristics:
+
+- automated tests start automatically after the run is launched
+- results are reported from CI to Testomat.io
+- manual tests are completed in the UI
+
+Typical use cases:
+
+- regression pipelines
+- build validation
+- automated testing triggered by commits or pull requests
+
+Learn more in the [Continuous Integration guide](https://docs.testomat.io/integrations/continuous-integration/).
+
+### Mixed Run without CI
+
+Automated tests are executed **locally using the Testomat.io CLI**.
+
+Characteristics:
+
+- no CI setup required
+- tests run from a local machine or testing environment
+- results are reported to Testomat.io via CLI
+
+Typical use cases:
+
+- early automation adoption
+- local debugging of automated tests
+- hybrid workflows combining manual and automated testing
+
+## How to Launch Mixed Runs
+
+Mixed Runs can be started in **2 ways** depending on how much configuration is required.
+
+- from the **Tests** page - for quick execution; ideal when you need to run specific tests or suites quickly, with minimal configuration
+- from the **Runs** page - for fully configured execution; allows selecting test plans, assigning users, setting environments, and other advanced options before launch
+
+### Launch Mixed Run from Tests Page
+
+You can start a Mixed Run directly from the **Tests** page for quick execution of selected tests or suites.
+
+- Run **a single suite** via **'...'** menu
+- Run **multiple suites/tests** using **bulk selection**
+
+Both options follow the same launch flow.
+
+#### Launch a Suite from Tests Page
+
+1. Go to the **Tests** page
+2. Open a suite with manual and automated tests
+3. Click the **'...'** menu
+4. Select the **Run Tests** option
+
+![Run Tests](./images/att4_415.png)
+
+In the **Run** modal window,
+
+5. Open the **Mixed Run** tab
+6. Choose execution type:
+   - **On CI** → Select [CI Profile](https://docs.testomat.io/integrations/continuous-integration/)
+   - **Without CI** → disable CI build
+7. Click the **Launch** button
+
+![Select mixed tests without CI](images/gif1_415.gif)
+
+#### Launch Tests Using Bulk Selection
+
+1. Go to the **Tests** page
+2. Enable the **multi-select** mode
+3. Select tests or suites
+4. Click the **Run** button
+
+![Select mixed tests](./images/att1_415.png)
+
+In the **Run** modal window,
+
+5. Open the **Mixed Run** tab
+6. Choose execution type:
+   - **On CI** → Select [CI Profile](https://docs.testomat.io/integrations/continuous-integration/)
+   - **Without CI** → disable CI build
+7. Click the **Launch** button
+
+![Select mixed tests without CI](images/gif1_415.gif)
+
+### Launch Mixed Run from Runs Page
+
+You can create and configure Mixed Runs from the **Runs page** when more control is required.
+
+Launching a Mixed Run typically consists of **three steps:**
+
+1. Create a new Mixed Run
+2. Select Execution Type ([CI](https://docs.testomat.io/integrations/continuous-integration/)/ without CI)
+3. Configure Mixed Run
+
+Let's dive deeper into each step below.
+
+#### Create a Mixed Run
+
+1. Open the **Runs** page
+2. Click the **'⌄'** button next to the 'Manual Run'
+3. Select **Mixed Run** from dropdown
+
+![Mixed Run](./images/att6_415.png)
+
+#### Select Execution Type
+
+Choose how automated tests should run:
+
+- **Enable CI build** — automated tests run via [CI pipeline](https://docs.testomat.io/integrations/continuous-integration/)
+- **Disable CI build** — automated tests run locally via CLI
+
+This selection determines how automated test execution will start.
+
+![Select Execution Type](./images/gif4_415.gif)
+
+#### Configure Mixed Run
+
+When creating a mixed run from the Runs page, the following options are available for configuration:
+
+**Options Overview:**
+
+- **All tests** – launch all manual and automated tests available in the project
+- **Test plan** – select from existing test plans
+  - Bulk selection is NOT supported, only one plan can be selected
+  - You can also create a new test plan if needed (Learn more [here](https://docs.testomat.io/project/plans/#ways-to-create-a-test-plan))
+- **Select tests** – manually select tests from the tree structure, use search, or apply filters to narrow down your selection
+  - Supports bulk selection
+  - Filters and collections allow for advanced selection and grouping of tests
+  - Learn more about working with test collections & filters [here](https://docs.testomat.io/project/plans/#how-to-work-with-test-collections)
+- **Without tests** – create a mixed run first and populate it later
+
+![Options Overview](./images/gif3_415.gif)
+
+:::note
+
+These options are mutually exclusive — you can select only one at a time — but you can still use bulk actions within ‘Select tests’ option.
+
+:::
+
+**Additional Configurations:**
+
+- **Assign to** – define who will execute the tests (See: [How to Assign Users to the Run](https://docs.testomat.io/project/runs/running-tests-manually/#how-to-assign-users-to-the-run))
+- **Title** (optional) – give your run a descriptive name
+- **Rungroup** – group the run within a specific RunGroup for better organization (See: [How to Run Tests in RunGroups](https://docs.testomat.io/project/runs/running-tests-manually/#how-to-run-tests-in-rungroups))
+- **Set environment for execution** – choose one or multiple environments for the run (See: [How to Run Environments](https://docs.testomat.io/project/runs/environments/#how-to-run-with-multi-environments))
+- **Run as checklist** – hides test descriptions for faster execution, ideal for experienced testers (See: [How to Run Tests As Checklist](https://docs.testomat.io/project/runs/running-tests-manually/#how-to-run-tests-as-checklist))
+- **Description** (optional) – provide context for this run
+
+![Additional Configurations](./images/att7_415.png)
+
+**After configuration:**
+
+- **Launch** – immediately start the run with the configured settings
+- **Save** – store the configured run without launching it, so it can be executed later
+- **Cancel** - exit without saving; all changes will be lost
+
+![After configuration](./images/att8_415.png)
+
+## How to Execute Mixed Run
+
+After launching a Mixed Run, manual and automated tests can be executed independently depending on the selected execution type.
+
+### Mixed Run without CI
+
+When a Mixed Run is launched and **CI build is disabled**, follow these steps:
+
+1. Copy the command in the launched Mixed Run
+
+![CLI command](./images/att9_415.png)
+
+2. Paste it in your terminal
+
+![Terminal](./images/att3_415.png)
+
+Example command:
+
+`TESTOMATIO_RUN={Run_ID} TESTOMATIO={Project Reporting API key} npx @testomatio/reporter run "npx codeceptjs run" --filter "testomatio:plan={Plan_ID}"`
+
+The CLI will:
+
+- execute automated tests
+- collect results
+- send them to Testomat.io
+- attach results to the current run
+
+Meanwhile, manual tests can be executed directly in the UI.
+
+3. Set up test case results for manual tests
+4. Update test case results for automated tests or leave them as they are
+5. Click the **Finish Run** button
+
+![Finish Mixed Run](./images/gif2_415.gif)
+
+See how to set up test case results [here.](https://docs.testomat.io/project/runs/running-tests-manually/#how-to-set-test-case-results-in-manual-run)
+
+### Mixed Run on CI
+
+When a Mixed Run is launched and **CI build is enabled**:
+
+1. CI pipeline starts automatically
+2. Automated tests run in the CI environment
+3. Test results are reported back to Testomat.io
+4. manual tests must be executed in the Testomat.io UI in the same way as they are executed without CI
+5. Finish the run when testing is complete
+
+![Mixed Run on CI](./images/att5_415.png)
+
+### How to Make Changes in Ongoing Mixed Run
+
+During testing, new scenarios may appear that need immediate validation. Testomat.io allows you to edit an active Mixed Run without stopping the testing process.
+
+This ensures flexibility, smooth testing workflow, and that no important scenarios are missed.
+
+1. Go to the **Runs** page
+2. Open the unfinished mixed run you want to edit
+3. Click the **Edit** button
+
+![Edit button](./images/att10_415.png)
+
+Once the **Edit mixed run** view is opened, here’s what you can do:
+
+- **Assign to** - Remove existing users or assign more to the run
+- **Title** - Enter or change the run title
+- **Environment** - Add or remove testing environments
+- **Description** - Add or update the run description
+- **details** link - Check configuration details of the run
+- **Current tests** – Shows all tests currently included in the run. Remove unnecessary tests by clicking the **trash** icon
+- **+ Tests** – Similar to the 'Select tests' tab in a new mixed run. You can browse the test tree, use search or filters, or expand suites to select manual and automated tests to add to this run
+- **+ Plans** – Select tests from existing test plans. You can include one or multiple plans in the current run
+
+![Edit mixed run](./images/att11_415.png)
+
+:::note
+
+You can add tests from plans and additional tests at the same time. For example: select one or more plans, then select specific suites or individual tests — everything will be added to the run instantly.
+
+:::
+
+## Managing Mixed Runs
+
+Testomat.io provides several ways to manage and reuse existing runs. Understanding each action ensures efficient test management and reduces the risk of errors in your QA workflow.
+
+### Options only for Mixed Runs with CI
+
+When the original run was executed via CI, two extra relaunch options appear:
+
+1. **Relaunch Failed**
+
+- Re-runs only the automated tests that failed in the previous run via CI
+- A new CI job is created, but the Mixed Run ID stays the same
+- Manual tests are skipped but can be executed again in the Testomat.io UI
+
+2. **Relaunch All**
+
+- Re-runs all automated tests via CI
+- Previous results are reset, and a new CI job is created
+- Manual tests still need to be executed manually in the UI
+
+These options help teams to:
+
+- repeat regression cycles efficiently
+- validate fixes quickly
+- re-run only failed automated tests when needed
+- reuse existing run configurations
+- maintain consistent and flexible QA workflows
+
+### Options for all Mixed Runs
+
+- **Relaunch Manually**
+
+Restart the Mixed Run manually. All test cases (both manual and automated) will need to be executed manually in the Testomat.io UI, without triggering CI or CLI execution.
+
+- **Advanced Relaunch**
+
+Modify parameters before restarting execution. Learn more here about [Advanced Relaunch](https://docs.testomat.io/project/runs/managing-runs/#advanced-relaunch).
+
+- **Launch a Copy Manually**
+
+Create a copy of an existing mixed run and execute it manually. The copy allows independent execution without affecting the original run.
+
+- **Launch a Copy**
+
+Automatically create a copy of the mixed run and start it immediately with the same settings as the original.
+
+![All relaunch options](./images/att12_415.png)