testeiya 0.2.1 → 0.3.9

package/skills/testomatio/test-management/qa-test-code-coverage/SKILL.md

@@ -1,174 +0,0 @@
----
-name: qa-test-code-coverage
-description: Map test cases — manual (markdown) and automated e2e (Playwright, Cypress, WebdriverIO, CodeceptJS, Puppeteer, Appium) — to source code files and produce a `coverage.tests.yml` mapping consumed by `@testomatio/reporter --filter "coverage:..."`. Use this skill when the user wants to run only the tests affected by a code change, generate a coverage file (manual, e2e, or both), build a traceability matrix between tests and source code, or set up change-aware regression.
-license: MIT
-metadata:
-  author: Testomat.io
-  version: 1.0.0
----
-
-# Tests to Code Coverage
-
-Analyze the project's tests — manual markdown cases and automated e2e tests — and its source code. Produce `coverage.tests.yml` — a map from source files (or globs) to test identifiers: suite IDs, test IDs, tags. Testomat.io keeps manual and automated tests in one project, so one coverage file serves both. `@testomatio/reporter` consumes it to run only the tests affected by a diff (Step 6).
-
-## Critical Constraints
-
-- **Map the tests that exist — manual markdown cases, automated e2e tests, or both.** No unit tests. Do not suggest creating new tests.
-- You may write to exactly three places, nothing else:
-  - the coverage file — default `coverage.tests.yml`, or the path the user gave;
-  - the `.testeiya/` cache dir — clone automated test repos, pull manual cases from Testomat.io (Step 1);
-  - `.gitignore` — add a `.testeiya/` line if it is missing.
-- **Never modify a source or test file.**
-- Never pull or clone tests into a tracked folder — only into the gitignored `.testeiya/`.
-- **Everything in the coverage file must come from this project:** file keys from its source tree, identifiers from the Step 2 inventory. Never copy a path or ID from this skill's docs — they are placeholders.
-- **No ad-hoc scripts, no parsers, never Python.** File reads and `grep` only; the single exception is the bundled checker (Step 5); beyond that a one-line `node -e '…'` is the limit.
-- **Be proactive — never stop at a representative sample.** A few entries per domain is a failed run. Keep mapping without asking permission to continue until the completion criterion in Step 4 is met.
-- Stop if you cannot write the output file, or no tests are found and the user declines every option in Step 1.
-
-## Workflow
-
-### Step 1: Discover the project
-
-- Run the `scan-automation-project` skill to inventory the project. Use its output as the source of truth — do not duplicate the scan.
-- Capture from its result:
-  - Manual Tests — the `.test.md` files and their suite/test titles.
-  - Automated Tests — the e2e test files and framework(s). See [E2E Frameworks](./references/E2E_FRAMEWORKS.md) if the scan is ambiguous.
-  - Project Overview — languages, frameworks, complexity (drives the subagent split in Step 4).
-- Map whatever exists: both kinds go into one coverage file; a single kind is fine too.
-- ❓ If the scan finds no tests at all (it checks the `.testeiya/` cache too), ask the user:
-  1. Pull manual cases from Testomat.io — have `sync-test-cases-with-tms` pull into the cache: `npx check-tests pull -d .testeiya/manual-tests`, add `.testeiya/` to `.gitignore` if missing, re-run the scan.
-  2. Clone the automated tests repo: `git clone <url> .testeiya/e2e-tests`, add `.testeiya/` to `.gitignore` if missing, re-run the scan.
-  3. Point to a directory the scan missed, then re-run the scan there.
-  4. Stop.
-- ❓ If two automated frameworks are detected (say Jest and Playwright), ask which one is the e2e framework — coverage filtering runs per runner.
-
-### Step 2: Extract test information
-
-Build one inventory of identifiers across both kinds:
-
-- Suite IDs — `@S` + 8 chars.
-- Test IDs — `@T` + 8 chars.
-- Tags — other `@word` markers.
-- Context — titles, steps, page objects, routes hit. This drives the mapping in Step 4.
-
-Where IDs live:
-
-- Manual cases follow the [Classical Tests Markdown Format](../qa-write-test-cases/references/test-case-format.md) — IDs in `<!-- suite ... id: @S... -->` / `<!-- test ... id: @T... -->` blocks, tags in titles and `tags:` lines.
-- Automated tests carry IDs in `describe` / `it` / `test` / `Feature` / `Scenario` titles — see [E2E Frameworks](./references/E2E_FRAMEWORKS.md).
-
-Read the files, or `grep` the directories that hold the tests:
-
-```bash
-grep -rnE 'id:[[:space:]]*@[ST]' <dir>          # manual suite/test IDs
-grep -rnoE '@[ST][0-9a-f]{8}' <dir>             # IDs in automated test titles
-grep -rhoE '@[A-Za-z0-9_-]+' <dir> | sort -u    # every @token, tags included
-```
-
-Missing IDs mean the tests were never synced with Testomat.io — the reporter cannot select them:
-
-- ❓ Manual files without IDs: ask whether to push them first via `sync-test-cases-with-tms`, or skip those files.
-- Automated tests without IDs in most files: stop and instruct the user to run `npx check-tests@latest <Framework> "<glob>" --update-ids` first (per-framework commands: [E2E Frameworks](./references/E2E_FRAMEWORKS.md)).
-
-### Step 3: Plan the coverage map by domain
-
-**Do not crawl the codebase file by file.** Identify the application's domain areas first; Step 4 then maps area by area.
-
-- Derive the areas from both sides:
-  - the suite tree from Step 2 — the suite hierarchy is the QA team's own map of the product;
-  - the source structure — modules, routes, services, entry points.
-- Pair each area with the suites that test it and the source folders that implement it.
-- Order the areas by criticality, most critical first: payments/billing, auth and access control, data integrity, core business flows — then supporting features, then peripheral UI. Suite size is a signal: what QA tests heavily, they consider critical.
-- Show the ordered plan to the user before mapping.
-- Business code only: skip test code, manual test directories, dependency/build/vendor folders, framework configs, lock files.
-- **Templates and views are mappable source** (`.vue`, `.erb`, `.blade.php`, …) — tests check the rendered UI, so map them like code.
-- ❓ If areas or their priority are unclear, ask the user.
-
-### Step 4: Map source files to tests
-
-**The Step 2 inventory is the work list. The map is complete only when every suite in it is either present in the coverage file or recorded as unmapped with a reason.** Work through the areas in Step 3's order — most critical first, so even an interrupted run covers what matters most. Run two passes — always both:
-
-- Code → tests: for each area, map its source files and subtrees to the identifiers of the tests that check them.
-- Tests → code: walk the inventory; for every suite still absent from the map, find the source it exercises and add it — or record why it cannot be mapped (feature has no code here, external system, needs user input).
-
-Scale the passes to the project — codebase size and suite count from Step 1:
-
-- Small codebase and few suites — run both passes in this session.
-- Large codebase or dozens of suites — spawn subagents in parallel: one per domain area for the code→tests pass, then one per batch of still-unmapped suites for the tests→code pass. Give each subagent:
-  - its slice — one area (its suites and source folders), or a batch of suites;
-  - the full test inventory from Step 2 — subagents must not re-extract it;
-  - the mapping rules below and the [Coverage File Format](./references/COVERAGE_FILE_FORMAT.md);
-  - the instruction to return a YAML fragment for its slice only.
-- Merge the fragments into one map, drop duplicate keys and empty entries. Only the main session writes the file (Step 5).
-
-For each candidate source file, pick the identifier that keeps selecting the right tests as the test suite grows:
-
-- Suite (`@S`) — the default. Most tests in the suite relate to the file.
-- Tag (`@word`) — the relevant tests live across several suites.
-- Test (`@T`) — the exception. Use only when just one or two tests in a suite match the file.
-
-Rules:
-
-- **More than a few test IDs from one suite or category → replace them with the suite or tag.** Tests added there later are picked up automatically; a list of test IDs goes stale.
-- Never list a test ID whose suite is already in the same entry.
-- Keys are source file paths or globs, relative to the repo root; use a glob when a whole subtree maps to the same identifiers.
-- Manual and automated identifiers mix freely in one entry.
-- No empty entries.
-- Annotate each identifier with a `#` comment naming its suite/test/tag title.
-
-YAML grammar: [Coverage File Format](./references/COVERAGE_FILE_FORMAT.md).
-
-### Step 5: Save and validate the coverage file
-
-- Write the YAML to the output path, keeping the `#` comments.
-- Keys are paths to source files in this repo — never `.testeiya/...` paths.
-- Check it with the bundled checker. It ships in this skill's `scripts/` directory (next to this SKILL.md), not in the project — resolve its path from the skill location:
-
-```bash
-npx js-yaml coverage.tests.yml | node <path-to-this-skill>/scripts/check-coverage.mjs
-```
-
-- The checker flags keys missing on disk and empty entries, and prints every identifier referenced — cross-check them against the Step 2 inventory; only you know which are real.
-- Reconcile the other direction too: every suite in the inventory must appear in the file or in the unmapped list. Anything missing from both means the tests→code pass is not finished — go back to Step 4.
-
-### Step 6: Report coverage completeness
-
-Show the user:
-
-- Suites mapped vs. total suites in the inventory; same for tests and tags where meaningful.
-- Every unmapped suite with its reason.
-- The produced YAML (or its path and entry count when large).
-
-### Step 7: Show next steps
-
-Tell the user how to use the file with `@testomatio/reporter`:
-
-```bash
-# Automated tests — the project's runner command must be passed in
-npx @testomatio/reporter run "npx playwright test" \
-  --filter "coverage:file=coverage.tests.yml,diff=main"
-
-# Manual tests — creates a pending run with only the affected cases
-npx @testomatio/reporter run --kind manual \
-  --filter "coverage:file=coverage.tests.yml,diff=main"
-```
-
-- Per-framework runner commands and a GitHub Actions example: [Coverage File Format](./references/COVERAGE_FILE_FORMAT.md).
-- To group manual and automated runs of one regression cycle, set the same `TESTOMATIO_RUNGROUP` env var on both commands.
-- Recommend committing the coverage file so CI and teammates use the same mapping.
-- In CI, diff against the base branch — `diff=origin/main` — and checkout with `fetch-depth: 0`.
-- Pulled or cloned tests stay in the gitignored `.testeiya/` cache for re-runs — don't delete it or move it into a tracked folder.
-
-### Step 8: Suggest follow-ups
-
-- Wire the map into CI — runs created per PR and launched on preview/merge: delegate to `setup-pr-testing`.
-- Run the affected tests right now from the terminal: delegate to `run-tests-with-testomatio-reporter`.
-- Coverage gaps — source features no test maps to. On approval, propose new cases (delegate to `qa-write-test-cases`).
-- Dead tests — tests whose features no longer exist in source.
-- Answer questions like "do we have tests for X?" from the inventory.
-- Editing pulled manual cases: edit them in `.testeiya/manual-tests/` and push back with `sync-test-cases-with-tms`.
-
-## References
-
-- [Coverage File Format](./references/COVERAGE_FILE_FORMAT.md) — coverage YAML grammar, reporter contract, GitHub Actions example.
-- [E2E Frameworks](./references/E2E_FRAMEWORKS.md) — framework detection signals, where IDs live, per-framework `check-tests` commands.
-- [Manual test markdown format](../qa-write-test-cases/references/test-case-format.md) — canonical, owned by `qa-write-test-cases`.

package/skills/testomatio/test-management/qa-test-code-coverage/references/COVERAGE_FILE_FORMAT.md

@@ -1,138 +0,0 @@
-# Coverage File Format
-
-`coverage.tests.yml` (any path works) maps source files to manual and automated test identifiers — one grammar for both kinds. It is read by `@testomatio/reporter run --filter "coverage:file=<path>,diff=<branch>"`.
-
-Every path and identifier in this document is a placeholder showing the shape. Real file keys come from the project's source tree; real IDs come from its test inventory. Never copy a value from here into a coverage file.
-
-## Top-level shape
-
-A YAML map. Keys are file paths or globs relative to the repository root. Values are lists of identifiers — Suite IDs, Test IDs, or tags.
-
-```yaml
-<file or glob>:
-  - "@S<8 chars>"   # Suite ID
-  - "@T<8 chars>"   # Test ID
-  - "@<tag>"        # Tag (matches anything tagged with that label)
-```
-
-## Identifier types
-
-| Identifier | Format       | Shape         | Meaning                                  |
-| ---------- | ------------ | ------------- | ---------------------------------------- |
-| Suite ID   | `@S` + 8 hex | `@S1a2b3c4d`  | All tests inside the suite are selected. |
-| Test ID    | `@T` + 8 hex | `@T5e6f7a8b`  | Only this specific test is selected.     |
-| Tag        | `@<word>`    | `@smoke`      | All tests tagged with the label.         |
-
-For automated tests, IDs live in describe / it / test names:
-
-```javascript
-describe('<suite title> @S1a2b3c4d', () => {
-  it('<test title> @T5e6f7a8b', () => { ... });
-});
-```
-
-They are populated by `npx check-tests <Framework> "<glob>" --update-ids` after the tests are imported into Testomat.io.
-
-## File keys
-
-Either an exact path or a glob:
-
-```yaml
-# Exact file
-<path/to/file>:
-  - "@S1a2b3c4d"
-
-# Glob — any file under the directory
-<path/to/dir>/**:
-  - "@<tag>"
-
-# Multiple identifiers per file
-<path/to/file>:
-  - "@S1a2b3c4d"
-  - "@T5e6f7a8b"
-```
-
-## Comments
-
-Use `#` to annotate each identifier with its human-readable title or purpose:
-
-```yaml
-<path/to/file>:
-  - "@S1a2b3c4d"  # Suite: <suite title>
-  - "@T5e6f7a8b"  # Test: <test title>
-```
-
-## Reporter usage
-
-For automated tests the runner command **must** be the first positional argument — `--filter` generates a `--grep` that the runner consumes.
-
-```bash
-# Playwright
-npx @testomatio/reporter run "npx playwright test" \
-  --filter "coverage:file=coverage.tests.yml,diff=main"
-
-# Cypress
-npx @testomatio/reporter run "npx cypress run" \
-  --filter "coverage:file=coverage.tests.yml,diff=main"
-
-# WebdriverIO / Mocha / Jest / CodeceptJS — pass the corresponding runner command
-npx @testomatio/reporter run "npx codeceptjs run" \
-  --filter "coverage:file=coverage.tests.yml,diff=main"
-```
-
-For manual tests pass `--kind manual` instead of a runner — the reporter creates a pending run in Testomat.io with only the affected cases:
-
-```bash
-npx @testomatio/reporter run --kind manual \
-  --filter "coverage:file=coverage.tests.yml,diff=main"
-```
-
-The `diff` value must be a stable branch (`main`, `origin/main`) the reporter can run `git diff` against.
-
-## GitHub Actions example
-
-```yaml
-name: Run Tests Based on Changes
-
-on:
-  pull_request:
-    branches: [main]
-
-jobs:
-  e2e-tests:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
-
-      # install browsers and prepare app for testing here
-
-      - name: Run affected E2E tests
-        env:
-          TESTOMATIO: ${{ secrets.TESTOMATIO }}
-        run: |
-          npx @testomatio/reporter run "npx playwright test" \
-            --filter "coverage:file=coverage.tests.yml,diff=origin/main"
-```
-
-## Checking the coverage file
-
-One command — no parser of your own. Run it with the project root as working directory (file keys resolve against it), but the checker itself lives in the skill's `scripts/` directory, not in the project:
-
-```bash
-npx js-yaml coverage.tests.yml | node <path-to-this-skill>/scripts/check-coverage.mjs
-```
-
-`npx js-yaml` parses the YAML (it fails loudly on a malformed file, so a broken one never reaches the script). `check-coverage.mjs` (~25 lines, zero deps) reads that parsed map on stdin, flags any key whose path is missing on disk, flags any key with no identifiers, lists every `@S…` / `@T…` / tag the file references, and exits non-zero on a problem.
-
-The script can't tell which identifiers are real — check the listed ones against the set you extracted earlier in the workflow. Don't re-parse the test set, don't write your own YAML parser, and never use `python`.
-
-## Authoring tips
-
-- Prefer Suite IDs when most of a suite relates to a file.
-- Use tags for cross-cutting concerns that span suites.
-- Use Test IDs only when one or two tests in a suite are relevant — a long test-ID list goes stale as tests are added, a suite or tag keeps selecting new ones.
-- Never list a test ID whose suite is already in the entry — the suite ID already selects it.
-- Avoid empty entries.
-- Commit the coverage file to the repo so CI and teammates use the same mapping.

package/skills/testomatio/test-management/qa-test-code-coverage/references/E2E_FRAMEWORKS.md

@@ -1,65 +0,0 @@
-# E2E Framework Detection
-
-How to detect the framework in use and where Testomatio IDs live in each.
-
-## Detection signals
-
-| Framework    | Filename patterns                                         | Imports                              | Test syntax                              |
-| ------------ | --------------------------------------------------------- | ------------------------------------ | ---------------------------------------- |
-| Playwright   | `*.spec.ts`, `*.spec.js`, `*.test.ts`                     | `@playwright/test`                   | `test(...)`, `test.describe(...)`        |
-| Cypress      | `*.cy.js`, `*.cy.ts`                                      | `cypress`                            | `describe(...)`, `it(...)`, `context(...)` |
-| WebdriverIO  | `*.test.js`, `*.e2e.js`                                   | `@wdio/cli`, `webdriverio`           | `describe(...)`, `it(...)`               |
-| Puppeteer    | `*.test.js`                                               | `puppeteer`                          | `describe(...)`, `it(...)`               |
-| CodeceptJS   | `*_test.js`, `*.test.js`                                  | `codeceptjs`                         | `Feature(...)`, `Scenario(...)`          |
-| Appium       | `*.spec.js`, `*.e2e.js`                                   | `appium`, `webdriverio`              | `describe(...)`, `it(...)`               |
-| Mocha (e2e)  | `*.test.js`, `*.spec.js`                                  | `mocha`                              | `describe(...)`, `it(...)`               |
-| Jest (e2e)   | `*.test.js`, `*.spec.js`                                  | `jest`                               | `describe(...)`, `it(...)`, `test(...)`  |
-
-Look at filename patterns first, then confirm with imports. Configuration files (`playwright.config.*`, `cypress.config.*`, `wdio.conf.*`, `codecept.conf.*`) are the strongest tiebreaker.
-
-## Where Testomatio IDs live
-
-IDs are appended to test/suite names — never inside code blocks:
-
-```javascript
-// Playwright / WebdriverIO / Mocha / Jest / Puppeteer / Cypress
-describe('<suite title> @S1a2b3c4d', () => {
-  it('<test title> @T5e6f7a8b', () => { ... });
-});
-
-test('<test title> @smoke @T5e6f7a8b', async ({ page }) => { ... });
-```
-
-```javascript
-// CodeceptJS
-Feature('<suite title> @S1a2b3c4d');
-
-Scenario('<test title> @T5e6f7a8b', ({ I }) => { ... });
-```
-
-Tags use the same `@word` form (`@smoke`, `@regression`, `@jira-123`).
-
-## Populating IDs
-
-If tests do not yet contain `@S` / `@T` markers, run `check-tests` with the matching framework adapter and `--update-ids`:
-
-```bash
-# Playwright
-npx check-tests@latest Playwright "**/*{.,_}{test,spec}.{js,ts}" --update-ids
-
-# Cypress
-npx check-tests@latest Cypress "**/*.cy.{js,ts}" --update-ids
-
-# CodeceptJS
-npx check-tests@latest CodeceptJS "**/*_test.js" --update-ids
-
-# WebdriverIO
-npx check-tests@latest WebdriverIO "**/*.{test,e2e}.js" --update-ids
-```
-
-`check-tests` rewrites the test files in place, inserting the IDs assigned by Testomat.io. Commit the changes before running `qa-test-code-coverage`.
-
-## Related skills
-
-- `qa-e2e-tests-reporting` — install `@testomatio/reporter` and import tests via `check-tests`.
-- `sync-test-cases-with-tms` — pull/push manual cases. See its [Testomat.io CLI reference](../../sync-test-cases-with-tms/references/TESTOMATIO_CLI.md) for the full `check-tests` command set, including `--update-ids`.

package/skills/testomatio/test-management/qa-test-code-coverage/scripts/check-coverage.mjs

@@ -1,40 +0,0 @@
-#!/usr/bin/env node
-// Sanity-check a coverage YAML mapping (manual and automated — same format).
-//
-// `js-yaml` does the YAML parsing; this script just checks the result.
-// Run it from the project root and pipe the parsed file in:
-//
-//   npx js-yaml coverage.tests.yml | node check-coverage.mjs
-//
-// (`npx js-yaml` prints JSON, and fails loudly if the YAML is malformed,
-// so a broken file never reaches this script.)
-//
-// Reports: keys whose path is missing on disk, keys with no identifiers,
-// and the list of @S/@T/tag IDs referenced — cross-check those against
-// the test set you already extracted. Exits non-zero on any problem.
-// (Never use python.)
-
-import { existsSync, readFileSync } from 'node:fs';
-
-const map = JSON.parse(readFileSync(0, 'utf8')); // fd 0 = stdin
-if (!map || typeof map !== 'object' || Array.isArray(map)) {
-  console.error('expected a JSON object of "path": [ids] — pipe `npx js-yaml <file>` into me');
-  process.exit(1);
-}
-
-const ids = new Set();
-let problems = 0;
-
-for (const [key, value] of Object.entries(map)) {
-  const list = Array.isArray(value) ? value : value == null ? [] : [value];
-  if (list.length === 0) { console.log('empty:  ', key); problems++; }
-  list.forEach((id) => ids.add(String(id)));
-
-  if (key.startsWith('tag:')) continue;                     // tag selector, not a path
-  const base = key.replace(/[\/\\][^\/\\]*[*?[\]].*$/, ''); // a glob → its non-glob prefix
-  if (base && !existsSync(base)) { console.log('missing:', key); problems++; }
-}
-
-console.log('\nidentifiers:', [...ids].sort().join(', ') || '(none)');
-console.log(problems ? `\n${problems} problem(s) above — fix them` : '\nall keys resolve, no empty entries');
-process.exit(problems ? 1 : 0);

package/skills/testomatio/test-management/qa-thinking/SKILL.md

@@ -1,36 +0,0 @@
----
-name: qa-thinking
-description: Analyze a feature a developer is building from a QA perspective — edge cases, negative flows, abuses, unobvious scenarios — and write acceptance criteria. Use when asked "what could go wrong?", "what am I missing?", "review this as QA", or for acceptance criteria before tests exist.
----
-
-
-Look at the feature in development and think about it like a senior QA engineer.
-The user can explain the idea, or you can take it from the current branch or current PR.
-
-Think:
-
-- Find feature positive scenarios
-- Find feature negative scenarios: what are possible negative usage scenarios in this feature?
-- Discover unobvious ways of using the feature (multiple actions, boundary values, cancellations)
-- Combinations: look how this feature aligns with other features. What can users do?
-- What are possible abuses of this feature?
-- Data consistency: think how user inputs may create inconsistent state and how it can affect the feature in the future or the overall system
-- What are possible security vulnerabilities in this feature?
-- What can be unit tested, what can't be, what should be verified manually? How to verify manually? (do not recommend manually verifying things already unit tested)
-- What end-to-end test scenarios can you recommend?
-- What test coverage do you recommend (percent of unit, e2e, manual)?
-
-Output:
-
-- list number and scope of unit/e2e tests if they are available (Section: ☑️ What is already tested)
-- ask questions to resolve important ambiguities of the feature (Section: 👓 What must be clarified)
-- propose up to 10 important test scenarios first (not more) (Section: 🔬 What must be verified)
-- prefer simple wording and short sentences
-- when asked for more, propose another bunch of test scenarios
-- do not list all automated unit/e2e test scenarios (prefer to generalize them and expand when the user asks you to)
-
-Outcomes:
-
-- Testing plan for unit/e2e/manual testing
-- Propose checklist for manual testing with acceptance criteria (`action → expected result`).
-- Offer to turn it into test cases with `qa-write-test-cases` skill.