testeiya 0.4.3 → 0.4.5

package/skills/testomatio/test-management/migrate-to-testomatio/SKILL.md

@@ -0,0 +1,52 @@
+---
+name: migrate-to-testomatio
+description: Migrate tests to Testomat.io from TestRail, XRay, Testmo, QMetry, Allure TestOps, TestCaseLabs, Qase, Zephyr, QTest, CSV/XLSX, or an unsupported TMS needing a custom converter or API script. Use when user wants to import tests from another TMS, move test suites to Testomat.io, or convert an export file to Testomat.io format.
+license: MIT
+metadata:
+  author: Testomat.io
+  version: 1.0.0
+---
+
+# Migrate to Testomat.io
+
+Migrate test suites from another TMS into Testomat.io via UI import, API migration script, or CSV converter.
+
+## Pick Strategy
+
+- Identify source: ask for TMS name only if not given.
+- Identify input: live instance with API access, or exported CSV/XLSX file.
+- Route by source:
+  - TestRail + API access, >1000 tests or needs attachments/runs: API migration script ([TESTRAIL_MIGRATION.md](./references/TESTRAIL_MIGRATION.md)).
+  - TestRail, <1000 tests, no attachments: built-in UI import (CSV or TestRail API option in Imports window).
+  - XRay (Jira): migration script producing Testomat.io CSV ([XRAY_MIGRATION.md](./references/XRAY_MIGRATION.md)).
+  - Testmo, QMetry, TestCaseLabs, Allure TestOps export file: CSV converter script ([CSV_MIGRATION.md](./references/CSV_MIGRATION.md)).
+  - Qase, QTest, Zephyr, other TMS export file: direct UI CSV import, no script ([CSV_MIGRATION.md](./references/CSV_MIGRATION.md)).
+  - Unsupported or broken TMS support: build a custom converter or API script ([CUSTOM_MIGRATION.md](./references/CUSTOM_MIGRATION.md)).
+  - Screenshots or file attachments needed: use the API script path, CSV import cannot create attachments.
+- **Never invent converter output columns; converted files always import with format `Testomat.io`.**
+- **API scripts need source credentials plus a Testomat.io General Token; never hardcode tokens, use `.env`.**
+
+## Scope
+
+- Migration covers test cases; runs, defects, and requirements are optional extras via API.
+- Upload run results only after test cases are uploaded.
+- User fields with no Testomat.io equivalent are not dropped silently: map them to Labels/Tags or extend the converter script.
+
+## Workflow
+
+- Create empty Testomat.io project for the import target.
+- Get source access: API credentials (TestRail/XRay path) or export file (CSV path).
+- Run the routed path:
+  - API script path: clone repo to a temp dir, configure `.env`, dry-run, run full migration.
+  - CSV path: convert if a converter exists, then import via UI.
+  - Custom path: build converter or API script first, then follow the matching path above.
+- UI import in all cases ends at: Tests tab > (`...`) > Import from other TMS > Import > Import from CSV > pick source format > Choose file > Create.
+- Verify: check Tests page count, suite nesting, steps formatting, priorities/tags.
+- Offer post-migration cleanup: `detect-duplicate-test-cases`, `improve-test-cases`.
+
+## References
+
+- [TESTRAIL_MIGRATION.md](./references/TESTRAIL_MIGRATION.md) — TestRail UI options, API script env vars, runs and attachments migration.
+- [XRAY_MIGRATION.md](./references/XRAY_MIGRATION.md) — XRay token extraction, env vars, folder-scoped import.
+- [CSV_MIGRATION.md](./references/CSV_MIGRATION.md) — converter scripts, direct UI imports, custom Testomat.io XLSX columns.
+- [CUSTOM_MIGRATION.md](./references/CUSTOM_MIGRATION.md) — custom converter or API v2 script for unsupported or broken TMS support.

package/skills/testomatio/test-management/migrate-to-testomatio/references/CSV_MIGRATION.md

@@ -0,0 +1,72 @@
+# CSV Migration (Testmo, QMetry, TestCaseLabs, Allure, Others)
+
+Two paths: converter script (normalizes export to Testomat.io CSV), or direct UI import.
+
+Docs: https://docs.testomat.io/project/import-export/import/import-tests-from-csv-xlsx
+
+## UI Import (All Sources)
+
+- Open project > Tests tab > (`...`) > Import from other TMS > Import > Import from CSV.
+- Sidebar: pick the tool the file came from (Qase, QTest, Zephyr, TestRail, XRay, Testmo, QMetry, Allure TestOps, TestCaseLabs, Testomatio) > Choose file > Create.
+- Same flow works for BDD projects; rows map Precondition > Given, Step > When, Expected Result > Then.
+- Converted files always import with format `Testomatio`.
+
+## Converter Scripts
+
+- Requires NodeJS 18+, each produces `*_Testomatio.csv`.
+- Clone outside the project repo, one per source:
+
+```bash
+git clone https://github.com/testomatio/migrate-testmo.git <temp-dir>/migrate-testmo
+git clone https://github.com/testomatio/migrate-qmetry.git <temp-dir>/migrate-qmetry
+git clone https://github.com/testomatio/migrate-testcaselabs.git <temp-dir>/migrate-testcaselabs
+git clone https://github.com/testomatio/migrate-allure.git <temp-dir>/migrate-allure
+```
+
+- Run inside the cloned repo:
+
+```bash
+npm install
+node convert.js <path-to-export-csv>
+```
+
+- Example: `node convert.js TestCases.csv` produces `TestCases_Testomatio.csv`.
+- Allure output is `allure_Testomatio.csv`; maps Feature/Epic/Story to Folder hierarchy, `automated` flag to Status, `jira-*` columns to Issues.
+- No options to invent; script takes only the input file path.
+- Edit `convert.js` in the cloned repo to adjust column mapping.
+
+## Sources Without Converter
+
+- Qase, QTest, Zephyr: export CSV/XLSX from the source tool, import directly with matching format dropdown.
+- Qase walkthrough: https://docs.testomat.io/tutorials/migration-from-qase
+
+## Custom Testomat.io XLSX
+
+- Build a file with these columns (ID left empty): ID, Title, Status, Folder, Emoji, Priority, Tags, Owner, Description, Labels, Issues.
+- `Folder` nesting: `/suite/sub-suite`.
+- `Priority`: normal, important, high, critical, low.
+- `Status`: manual or automated, blank allowed.
+- `Description` supports Markdown.
+- `Issues`: Jira keys (`ABC-123`), comma-separated.
+- Example file: https://testomatiofiles.ams3.cdn.digitaloceanspaces.com/Testomat_example.xlsx
+
+## ID Compatibility
+
+- Testomat.io public IDs are 8 alphanumeric chars, case-insensitive.
+- Keep old IDs recognizable: zero-pad the numeric part to 8 chars (`TC-1` becomes `tc000001`).
+- Put the mapped ID in the `ID` column of the Testomat.io CSV.
+
+## Suites and Tests
+
+- A source suite holding both suites and tests must be split: suite-folder holds only suites, suite-file holds only tests (they differ by `file_type`).
+- Every test needs a suite: always fill the `Folder` column, using `/` nesting (`/suite/sub-suite`).
+
+## Descriptions and Images
+
+- Descriptions are pure Markdown; format steps per [test-case-format.md](../qa-write-test-cases/references/test-case-format.md).
+- Images inside test cases are not carried by CSV: upload them via Testomat.io API, or switch to the API script path.
+
+## Recovery
+
+- Import fails: first row must hold column names.
+- Wrong suites: check `Folder` column uses `/` nesting.

package/skills/testomatio/test-management/migrate-to-testomatio/references/CUSTOM_MIGRATION.md

@@ -0,0 +1,11 @@
+# Custom Migration (Unsupported or Broken TMS)
+
+When no converter or script fits the source, build one. Two options, cheapest first.
+
+- CSV-to-Testomat.io converter: Node script reading the source export, writing Testomat.io CSV columns (see [CSV_MIGRATION.md](./CSV_MIGRATION.md) for column spec, ID padding, and suite splitting).
+- API v2 migration script: pushes cases directly, required when attachments or run results must migrate.
+- API reference: https://app.testomat.io/docs/openapi
+- Base new API scripts on https://github.com/testomatio/migrate-testrail; for Jira-backed sources also use https://github.com/testomatio/migrate-xray.
+- Map unmigrated user fields to Labels/Tags rather than dropping them; extend the script when the user flags an important field.
+- Keep ID compatibility, suite-file vs suite-folder split, Markdown descriptions, and image uploads per [CSV_MIGRATION.md](./CSV_MIGRATION.md).
+- Upload run results only after test cases are uploaded.

package/skills/testomatio/test-management/migrate-to-testomatio/references/TESTRAIL_MIGRATION.md

@@ -0,0 +1,87 @@
+# TestRail Migration
+
+Three import methods. Pick by test count and attachment need.
+
+- CSV file: simple import, no attachments.
+- Built-in UI tool (TestRail API option): up to 1000 tests, no attachments.
+- Migration script: over 1000 tests, attachments, test runs with results.
+
+Docs: https://docs.testomat.io/project/import-export/import/import-tests-from-testrail
+
+## UI Import
+
+- Open project > Tests tab > (`...`) > Import from other TMS > Import > Import From TestRail.
+- CSV option: Import > Import from CSV > dropdown `TestRail` > Choose file > Create.
+- API option: enable API in TestRail (Administration > Site Settings > API toggle), enter TestRail credentials in Testomat.io Imports tab > Import Tests.
+- Sample export for comparison: https://testomatio-artifacts.ams3.cdn.digitaloceanspaces.com/documentation/TestRail.csv
+
+## Migration Script
+
+Repo: https://github.com/testomatio/migrate-testrail
+
+- Requires NodeJS 20+.
+- Clone outside the project repo:
+
+```bash
+git clone https://github.com/testomatio/migrate-testrail.git <temp-dir>/migrate-testrail
+cp .env.example .env
+npm i
+npm start
+```
+
+- `.env` vars:
+
+```env
+TESTRAIL_URL=
+TESTRAIL_USERNAME=
+TESTRAIL_PASSWORD=
+TESTRAIL_PROJECT_ID=
+# TESTRAIL_SUITE_ID= # optional, single suite only
+TESTOMATIO_TOKEN=testomat_****
+TESTOMATIO_PROJECT=
+# TESTOMATIO_HOST=https://app.testomat.io # custom instance only
+# DRY_RUN=1 # dry run, no import
+```
+
+- `TESTOMATIO_PROJECT` is the URL slug: `https://app.testomat.io/projects/<slug>`.
+- `TESTOMATIO_TOKEN` is a General Token from https://app.testomat.io/account/access_tokens.
+- Debug flags: `DEBUG="testomatio:testrail:*" npm start` (`:in` source data, `:out` posted data, `:migrate` processing).
+- Single case debug (run after full migration): `TESTRAIL_CASE_ID=12345 npm start`.
+- Edit `migrate.js` to customize sections, suites, steps mapping.
+
+## Test Runs with Results
+
+- Requires Project Reporting API key (project Settings > API section) as `TESTOMATIO_REPORT_TOKEN`.
+- Import all cases first, then:
+
+```bash
+npm run migrate-run-results
+```
+
+- Single run: `TESTRAIL_RUN_ID=<id> npm run migrate-run-results`.
+- Already-imported runs are skipped via `@id:<run_id>` tag in run title.
+- Artifacts need S3 bucket plus same S3 creds in Testomat.io project Settings:
+
+```env
+S3_ACCESS_KEY_ID=
+S3_SECRET_ACCESS_KEY=
+S3_REGION=
+S3_BUCKET=
+# S3_ENDPOINT= # non-AWS only
+```
+
+## Attachments Fix
+
+- Fixes orphaned `index.php?/attachments/get/123` URLs left by earlier imports.
+- Needs `TESTRAIL_SESSION` (browser cookie `tr_session` from DevTools > Application > Cookies after TestRail login).
+
+```bash
+npm run migrate-attachments:dry-run
+npm run migrate-attachments
+```
+
+## Recovery
+
+- Duplicated steps after TestRail template change: `git checkout opt/template-fields-sync`, rerun.
+- UI tool cannot connect: API disabled in TestRail.
+- CSV fails: first row must hold column names.

package/skills/testomatio/test-management/migrate-to-testomatio/references/XRAY_MIGRATION.md

@@ -0,0 +1,56 @@
+# XRay Migration
+
+Script pulls from XRay API, produces a Testomat.io-compatible CSV, imported via UI.
+
+Docs: https://docs.testomat.io/project/import-export/import/import-tests-from-xray
+Repo: https://github.com/testomatio/migrate-xray
+
+## Setup
+
+- Requires NodeJS 20+.
+- Clone outside the project repo:
+
+```bash
+git clone https://github.com/testomatio/migrate-xray.git <temp-dir>/migrate-xray
+cp .env.example .env
+npm i
+npm start
+```
+
+- `.env` vars:
+
+```env
+JIRA_URL=
+JIRA_USERNAME=
+JIRA_TOKEN=
+JIRA_PROJECT_ID=
+XRAY_URL=
+XRAY_INTERNAL_TOKEN=
+# XRAY_FOLDER_ID= # optional, single folder from Test Repository URL (?selectedFolder=...)
+TESTOMATIO_TOKEN=testomat_****
+TESTOMATIO_PROJECT=
+# DRY_RUN=1 # dry run, no import
+```
+
+- `JIRA_TOKEN` is an Atlassian API token: https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/
+- `TESTOMATIO_TOKEN` is a General Token from https://app.testomat.io/account/access_tokens.
+- `TESTOMATIO_PROJECT` is the URL slug: `https://app.testomat.io/projects/<slug>`.
+
+## XRay Token Extraction
+
+- XRay has no public token endpoint; extract manually.
+- Open XRay app > F12 DevTools > Network tab > filter `xray.cloud.getxray.app`.
+- Open any XRay request Headers, copy `X-Acpt` header value into `XRAY_INTERNAL_TOKEN`.
+- Copy request Origin into `XRAY_URL` (usually `https://eu.xray.cloud.getxray.app` or `https://us.xray.cloud.getxray.app`).
+- Token expires: on `401 ... Authentication request has expired` reopen XRay and fetch a fresh token.
+
+## Import Result
+
+- Tests tab > Imports > Import from CSV > dropdown `XRay` (or `Testomatio` for script output) > Choose file > Create.
+- Debug: `DEBUG="testomatio:xray:*" npm start` (`:in` source, `:out` posted, `:migrate` processing).
+- Edit `migrate.js` to customize steps or field mapping.
+
+## Known Limitations
+
+- Test params are not exported by XRay API.
+- Tests referencing steps from another XRay test import as `[steps from a missing XRay test]` unless the referenced test is imported first.

package/skills/testomatio/test-management/scan-automation-project/SKILL.md

@@ -1,15 +1,15 @@
 ---
 name: scan-automation-project
-description: Scan project source code to inventory languages, frameworks, and existing tests (manual `*.test.md` and automated test files). Use this skill whenever analyzing a codebase for test planning, detecting test frameworks, or preparing for test automation. Specifically, when the user mentions "scan project", "what tests exist", "analyze codebase", "detect frameworks", "test matrix", or needs a codebase inventory before a QA workflow.
+description: Scan a project and inventory what is available for QA work - project type, languages, frameworks, manual `*.test.md` cases, automated tests and automated references, and how Testomat.io is reachable (token, MCP, project settings). Use this skill whenever analyzing a codebase for test planning, detecting test frameworks, counting tests, or preparing for test automation. Specifically, when the user mentions "scan project", "what tests exist", "how many tests", "analyze codebase", "detect frameworks", "test matrix", or needs an inventory before a QA workflow.
 license: MIT
 metadata:
   author: Testomat.io
-  version: 1.0.0
+  version: 1.1.0
 ---
 
 # Scan Automation Project
 
-Scan project source code and return a QA-focused inventory: languages, frameworks, and existing tests.
+Scan the project and return a QA-focused inventory: project type, languages, frameworks, existing tests, and what Testomat.io access is available.
 
 **Shallow scan only.** No full import resolution, no line-by-line parsing, no reading full test implementations. High-level structure only.
 
@@ -27,26 +27,42 @@ Where is the source code?
 ```
 
 - Wait for the user's reply.
-- If the user gives a Git repo or another folder, clone or symlink it into `.testeiya/code/`. No-op if it is already there.
+- If the user gives a Git repo or another folder, clone or symlink it into `.testeiya/<name>/` (the source's basename; `extra-` prefix if taken). No-op if it is already there.
+
+### The `.testeiya/` folder
+
+All persistent QA metadata lives in `.testeiya/` in the project root.
+
+- `<name>/`: a linked or cloned external project, named after its source. Read-only reference.
+- `requirements/`: user stories and acceptance criteria (pdfs, docs, images).
+- `docs/`: feature explanations, test planning and strategy files. Prose for humans, never data dumps.
+- `manual-tests/`: markdown test cases pulled from Testomat.io.
+- `auto-tests/`: references to the relevant automated (e2e) tests.
+- `exploratory/`: explorbot setup.
+- `project-info.json`: the project's settings as Testomat.io reports them, when cached.
+
+Rules:
+
+- It is a dot-folder. File-search tools skip it by default: search it with hidden:true or an explicit `.testeiya/` path prefix. A linked folder is a symlink; wildcard searches do not descend into it, so search it with its own `.testeiya/<name>/` prefix.
+- It excludes itself from git: when you create `.testeiya/`, also create `.testeiya/.gitignore` containing a single `*`. **Never add `.testeiya/` to the repo's own `.gitignore`**: a repo-level entry hides the folder from search tools, so the next scan reports "no manual tests" while the cases sit right there.
 
 ### Rule for pulled data
 
 **When you need to pull external data** (manual test cases, app code, e2e tests from another repo) into this project:
 
-| This project has | Store pulled data in | Gitignored? |
-| ------------------ | ---------------------- | ----------- |
-| **Source code** (`src/` folder) or **big project** | `.testeiya/…` | **Yes** |
-| **Only test infrastructure** (e2e dirs like `tests/`, `playwright/`, `cypress/` but no `src/`) | `manual-tests/` or `e2e-tests/` | No |
-| **Empty / manual-only** | `manual-tests/` | No |
+| This project has | Store pulled data in |
+| ------------------ | ---------------------- |
+| **Source code** (`src/` folder) or **big project** | `.testeiya/…` |
+| **Only test infrastructure** (e2e dirs like `tests/`, `playwright/`, `cypress/` but no `src/`) | `manual-tests/` or `e2e-tests/` |
+| **Empty / manual-only** | `manual-tests/` |
 
 Detection logic:
 1. Has `src/` folder or is a monorepo? → Use `.testeiya/`
 2. Has e2e test dirs (`tests/`, `playwright/`, `cypress/`, `e2e/`)? → Use tracked folder (`manual-tests/`, `e2e-tests/`)
 3. Otherwise → Default to `manual-tests/`
 
-- Any skill that creates `.testeiya/...` must also add `.testeiya/` to the project's `.gitignore` — but only if it is not there yet. Never add it twice.
 - The rule covers *pulled* data only. Output a skill *produces* and the user wants — a `coverage.*.yml`, new `*.test.md` files — goes in the repo as normal.
-- On a source repo with no manual tests, this skill changes no tracked file: it only creates the gitignored `.testeiya/` directory and, if needed, adds one line to `.gitignore`.
+- On a source repo with no manual tests, this skill changes no tracked file: it only creates the self-ignored `.testeiya/` directory.
 - **Only touch `.testeiya/` and tracked folders — never pollute the repo with cache in a wrong location.**
 
 ## Step 2: Project Analysis
@@ -59,11 +75,15 @@ Include:
 
 Exclude:
 - Dependencies, build output, coverage, reports, caches, config, lock, and environment files.
-- Paths ignored by `.gitignore` — but **not** `.testeiya/code/`. In a manual-tests repo the app code lives there; scan it as source.
+- Paths ignored by `.gitignore` — but **not** `.testeiya/<name>/`. In a manual-tests repo the app code lives there; scan it as source.
 - Testeiya internal files (e.g. `session-factory.ts`, `system-prompt.ts`).
 - **If in doubt**, exclude.
 
 From the file list:
+- Classify the **project type**, one of:
+  - `source`: application source code. Never change it; use it for discovery.
+  - `e2e`: a test automation project (e2e dirs, a test framework config, little or no app source). You can write tests for it.
+  - `manual`: empty, or mostly `*.test.md` files. Test cases go into the project itself.
 - Detect languages and frameworks. Collect one `frameworks` list with ALL application and testing frameworks.
 - Extract the project name from a project config file (`package.json`, `Cargo.toml`, `pom.xml`, ...); fall back to the root directory name.
 - Rate complexity by source file count:
@@ -75,34 +95,81 @@ From the file list:
 | 151-500    | `large`      |
 | 500+       | `very-large` |
 
-## Step 3: Test Inventory (optional)
+## Step 3: Test Inventory
 
 Detect existing tests, automated and manual. Stay shallow.
 
-Automated tests:
-- Detect frameworks via config files (`jest.config.*`, `playwright.config.*`, `vitest.config.*`, `pytest.ini`, `pom.xml`, ...), project dependencies, and test file patterns (`*.test.*`, `*.spec.*`, `*_test.*`).
-- For each framework: identify its test file pattern and count matching files.
+### Manual tests
 
-Manual tests:
-- Find all `.test.md` files and parse test titles. `find .` also looks inside `.testeiya/manual-tests/`, so a re-run after a pull finds the cached cases instead of reporting "no manual tests":
+A `*.test.md` file is a suite. Each test inside it is a `<!-- test ... -->` block followed by an `#`/`##` heading (the title). The block's `type:` field says which kind of test it is:
+
+- `type: manual`: a manual test case, authored and maintained here as markdown.
+- `type: automated`: a **reference** to an automated test (see below).
+- No `type:` line at all: treat as manual. This is common, so never count manual tests by grepping for `type: manual`; it silently undercounts.
+
+Know the counts before you answer anything about project scope, coverage or progress. Read them straight from the files; this mirrors how Testomat.io parses them. `find .` also looks inside `.testeiya/manual-tests/`, so a re-run after a pull finds the cached cases instead of reporting "no manual tests":
 
 ```bash
+# one line per suite or test: SUITE <title>, or <type> <title>
 find . -name "*.test.md" -exec awk '
-  /^<!-- test/   { in_block=1; kind="TEST";  next }
   /^<!-- suite/  { in_block=1; kind="SUITE"; next }
+  /^<!-- test/   { in_block=1; kind="manual"; next }
+  in_block && /^type:[[:space:]]*automated/ { kind="automated"; next }
   in_block && /^-->/ { in_block=0; expect=1; next }
   expect && /^#+[[:space:]]+/ {
     title=$0; sub(/^#+[[:space:]]+/, "", title)
     if (kind == "SUITE") printf "SUITE: %s\n", title
-    else             printf "|- %s\n",     title
+    else                 printf "|- [%s] %s\n", kind, title
     expect=0
   }
 ' {} +
 ```
 
+Pipe the output into `grep -c '\[manual\]'` or `grep -c '\[automated\]'` to count one kind.
+
 - If the only `.test.md` files are under `.testeiya/manual-tests/`, say so in the inventory: they came from Testomat.io, not from this repo.
+- A source checkout with no `*.test.md` files has nothing to count. Say so; do not estimate.
+
+### Automated tests
+
+- Detect frameworks via config files (`jest.config.*`, `playwright.config.*`, `vitest.config.*`, `pytest.ini`, `pom.xml`, ...), project dependencies, and test file patterns (`*.test.*`, `*.spec.*`, `*_test.*`).
+- For each framework: identify its test file pattern and count matching files.
+- Unit and integration tests are inventory, not QA scope: note them, but flag e2e and acceptance suites as the ones QA work targets.
+
+### Automated references
+
+`type: automated` entries in `*.test.md` are references, not runnable code. They arrive via `check-tests pull --export-automated`, which exports the automated tests Testomat.io knows about as markdown. The real implementation lives in a **different repository**, the automation project, which is usually not this one.
+
+- Never claim you ran an automated test because you found its reference here. Nothing here executes.
+- Do not edit a reference's markdown to change the test's behaviour. The code is elsewhere, the edit gets overwritten on the next pull, and pushing it can clobber TMS data. Locate the real test in the automation repo instead.
+- To actually run them, use a CI profile (Step 4): if the project has one configured, a run can be triggered through Testomat.io.
+- They are still valuable context: they tell you what is already automated versus what is still manual, which is what coverage gaps and automation candidates need.
+
+## Step 4: Testomat.io Access
+
+Record how Testomat.io is reachable from this session. Check, do not assume:
+
+- **Token:** `TESTOMATIO` in the environment (or in `.env`). With it, `npx check-tests` can pull and push test cases and the REST API answers under `$TESTOMATIO_URL/api/v2`.
+- **MCP:** Testomat.io MCP tools are present (tools prefixed `testomatio-`, direct tools such as `tests_list` and `runs_list`, or one `mcp` tool that searches and calls operations). MCP needs `TESTOMATIO_PROJECT_ID` as well as the token; a token alone identifies no project. Setup lives in the `testomatio-mcp` skill.
+- **Project settings:** `.testeiya/project-info.json`, when present: framework, language, environments, labels, tags, and **CI profiles** (runs can be triggered through them).
 
-## Step 4: Output
+Never ask the user for the Testomat.io token when one is configured. Secrets the app under test needs are a different thing; a missing one blocks a run.
+
+### Which source answers what
+
+| Question is about | First action |
+|---|---|
+| Test content: steps, gherkin, description, file path, tags | `read`, `find`, `grep` in the project |
+| Suite hierarchy, structure, which files exist | `ls`, `find` |
+| Individual test metadata: priority, status flag, labels | the files; if missing, get the test from Testomat.io |
+| Runs, testruns, plans, labels, issues, analytics | Testomat.io (MCP or REST); these are not files |
+| Creating or updating tests or suites | edit the markdown file, then `npx check-tests push` (see `sync-test-cases-with-tms`) |
+| Launching an automated or mixed run on CI | `npx -y @testomatio/reporter@latest run --remote <profile>`; raise an error if no CI profile is configured |
+| Running automated tests locally | the local test runner with the Testomat.io reporter attached |
+
+Statuses and counts in Testomat.io are live. Runs change them at any time, so fresh query results supersede numbers from earlier in the conversation.
+
+## Step 5: Output
 
 Return one structured markdown result directly. **Do NOT save to a file.**
 
@@ -111,14 +178,16 @@ Return one structured markdown result directly. **Do NOT save to a file.**
 
 - **Project Name:** acme-web-app
 - **Description:** A React-based customer dashboard with an Express API.
+- **Project Type:** source
 - **Languages:** TypeScript, SQL
 - **Frameworks:** React, Express, Jest, Playwright
 - **Complexity:** small (12 files)
 
 ## Test Inventory
 
-- **Automated Tests:** 10 files
-- **Manual Tests:** 37 cases
+- **Automated Tests:** 10 files (Playwright)
+- **Manual Tests:** 37 cases (`.testeiya/manual-tests/`, pulled from Testomat.io)
+- **Automated References:** 12 (implementation lives in another repository)
 
 ### Manual Tests (20 of 37 shown)
 
@@ -133,17 +202,25 @@ Return one structured markdown result directly. **Do NOT save to a file.**
 
 - home.page.spec.ts
 ...
+
+## Testomat.io
+
+- **Token:** set (`TESTOMATIO`)
+- **MCP:** available (`testomatio-*` tools)
+- **Project settings:** `.testeiya/project-info.json`, framework playwright, CI profiles: "e2e" (github)
 ```
 
 Field rules:
 - **Description:** 1-2 sentences based only on detected source code and folder structure (include the domain area if that makes sense).
+- **Project Type:** one of `source` | `e2e` | `manual`.
 - **Complexity:** one of `small` | `moderate` | `large` | `very-large`, plus the file count.
-- **Manual Tests:** preserve hierarchy as plain strings — SUITE items as parent bullets, test titles (`|-`) as nested children.
+- **Manual Tests:** preserve hierarchy as plain strings — SUITE items as parent bullets, test titles (`|-`) as nested children. Say where they live.
 - **Automated Tests:** list of detected test files.
-- All values must come from observable files. Do NOT guess missing data or add fields not shown above.
+- **Testomat.io:** one line per access path, `available`/`not available`; omit the section only when none is present, and then say the project is not connected.
+- All values must come from observable files and tools. Do NOT guess missing data or add fields not shown above.
 
 Sections:
-- If Step 3 was skipped or found no tests, omit `## Test Inventory` and note that the project contains no tests; the `# Project Overview` section is still useful for next steps.
+- If Step 3 found no tests, omit `## Test Inventory` and note that the project contains no tests; the `# Project Overview` section is still useful for next steps.
 - If one test type is absent, note it with a blockquote, e.g. ``> No manual tests (`.test.md`) found in the project.``
 
 Test listing truncation:

package/dist/prompt/context.js

@@ -1,79 +0,0 @@
-import dedent from "dedent";
-import { TESTEIYA_DIR_NAME } from "./vocab.js";
-/** System-prompt section describing the workspace context; "" when none. */
-export function contextPromptSection(entries, folders = []) {
-    if (entries.length === 0 && folders.length === 0)
-        return "";
-    return dedent `
-  <workspace-context>
-    This workspace has context attached — the user (or you, earlier) put it there for a purpose. Consult it before answering, and prefer it over assumptions about the project.
-
-  ${contextLines(entries, folders)}
-
-    ALL of it lives under the hidden \`${TESTEIYA_DIR_NAME}/\` dir: file-search tools skip hidden dirs by default, so search it explicitly (pass hidden:true, or prefix the path with \`${TESTEIYA_DIR_NAME}/\`). Linked folders and cloned repositories are reference material: read them, never modify them. A linked folder is a symlink — wildcard searches do not descend into it; search it with its own path prefix (e.g. \`${TESTEIYA_DIR_NAME}/<name>/\`). Documents under \`${TESTEIYA_DIR_NAME}/requirements\` and \`${TESTEIYA_DIR_NAME}/docs\` are specs and plans — use them when writing or reviewing tests.
-  </workspace-context>
-  `;
-}
-/** Per-prompt notice when the context changed mid-session; appended to the next prompt. */
-export function contextUpdateNotice(entries, folders = []) {
-    if (entries.length === 0 && folders.length === 0) {
-        return dedent `
-    <workspace-context-update>
-      All extra workspace context was removed. Do not rely on it anymore.
-    </workspace-context-update>
-    `;
-    }
-    return dedent `
-  <workspace-context-update>
-    The workspace context just changed. It now contains:
-
-  ${contextLines(entries, folders)}
-
-    This was done for a purpose — take it into account for this and future requests. Remember \`${TESTEIYA_DIR_NAME}/\` is hidden: search it with hidden:true or an explicit path prefix.
-  </workspace-context-update>
-  `;
-}
-function contextLines(entries, folders) {
-    const lines = [];
-    const covered = new Set(entries.map((e) => parentDir(e.path)));
-    for (const f of folders) {
-        if (covered.has(f.path))
-            continue;
-        lines.push(`- \`${f.path}/\` — ${folderPurpose(f.name)}`);
-    }
-    for (const e of entries)
-        lines.push(`- ${describeEntry(e)}`);
-    return lines.join("\n");
-}
-function folderPurpose(name) {
-    if (name === "manual-tests") {
-        return "the project's manual test suites pulled from Testomat.io (*.test.md — THE manual tests of this project)";
-    }
-    if (name === "code")
-        return "reference code added as context";
-    if (name === "requirements")
-        return "requirements and specs";
-    if (name === "docs")
-        return "test planning and strategy documents";
-    if (name === "auto-tests")
-        return "automated test references";
-    return "exploratory testing setup";
-}
-function parentDir(rel) {
-    const idx = rel.lastIndexOf("/");
-    if (idx < 0)
-        return "";
-    return rel.slice(0, idx);
-}
-function describeEntry(e) {
-    const date = e.addedAt?.split("T")[0] ?? "";
-    let from = "";
-    if (e.origin && e.origin !== e.name)
-        from = ` from ${e.origin}`;
-    if (e.kind === "repo")
-        return `\`${e.path}\` — git repository${from} (${date})`;
-    if (e.kind === "folder")
-        return `\`${e.path}\` — local folder${from} (${date})`;
-    return `\`${e.path}\` — document${from} (${date})`;
-}
-//# sourceMappingURL=context.js.map

package/dist/prompt/context.js.map

@@ -1 +0,0 @@
-{"version":3,"file":"context.js","sourceRoot":"","sources":["../../prompt/context.ts"],"names":[],"mappings":"AAAA,OAAO,MAAM,MAAM,QAAQ,CAAC;AAC5B,OAAO,EAAE,iBAAiB,EAAE,MAAM,YAAY,CAAC;AAE/C,4EAA4E;AAC5E,MAAM,UAAU,oBAAoB,CAClC,OAAuB,EACvB,UAA2B,EAAE;IAE7B,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,CAAC;IAC5D,OAAO,MAAM,CAAA;;;;IAIX,YAAY,CAAC,OAAO,EAAE,OAAO,CAAC;;yCAEO,iBAAiB,gIAAgI,iBAAiB,0NAA0N,iBAAiB,kCAAkC,iBAAiB,yBAAyB,iBAAiB;;GAEhhB,CAAC;AACJ,CAAC;AAED,2FAA2F;AAC3F,MAAM,UAAU,mBAAmB,CACjC,OAAuB,EACvB,UAA2B,EAAE;IAE7B,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACjD,OAAO,MAAM,CAAA;;;;KAIZ,CAAC;IACJ,CAAC;IACD,OAAO,MAAM,CAAA;;;;IAIX,YAAY,CAAC,OAAO,EAAE,OAAO,CAAC;;kGAEgE,iBAAiB;;GAEhH,CAAC;AACJ,CAAC;AAED,SAAS,YAAY,CAAC,OAAuB,EAAE,OAAwB;IACrE,MAAM,KAAK,GAAa,EAAE,CAAC;IAC3B,MAAM,OAAO,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAC/D,KAAK,MAAM,CAAC,IAAI,OAAO,EAAE,CAAC;QACxB,IAAI,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC;YAAE,SAAS;QAClC,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,IAAI,SAAS,aAAa,CAAC,CAAC,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IAC5D,CAAC;IACD,KAAK,MAAM,CAAC,IAAI,OAAO;QAAE,KAAK,CAAC,IAAI,CAAC,KAAK,aAAa,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;IAC7D,OAAO,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAC1B,CAAC;AAED,SAAS,aAAa,CAAC,IAAY;IACjC,IAAI,IAAI,KAAK,cAAc,EAAE,CAAC;QAC5B,OAAO,yGAAyG,CAAC;IACnH,CAAC;IACD,IAAI,IAAI,KAAK,MAAM;QAAE,OAAO,iCAAiC,CAAC;IAC9D,IAAI,IAAI,KAAK,cAAc;QAAE,OAAO,wBAAwB,CAAC;IAC7D,IAAI,IAAI,KAAK,MAAM;QAAE,OAAO,sCAAsC,CAAC;IACnE,IAAI,IAAI,KAAK,YAAY;QAAE,OAAO,2BAA2B,CAAC;IAC9D,OAAO,2BAA2B,CAAC;AACrC,CAAC;AAED,SAAS,SAAS,CAAC,GAAW;IAC5B,MAAM,GAAG,GAAG,GAAG,CAAC,WAAW,CAAC,GAAG,CAAC,CAAC;IACjC,IAAI,GAAG,GAAG,CAAC;QAAE,OAAO,EAAE,CAAC;IACvB,OAAO,GAAG,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC;AAC3B,CAAC;AAED,SAAS,aAAa,CAAC,CAAe;IACpC,MAAM,IAAI,GAAG,CAAC,CAAC,OAAO,EAAE,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;IAC5C,IAAI,IAAI,GAAG,EAAE,CAAC;IACd,IAAI,CAAC,CAAC,MAAM,IAAI,CAAC,CAAC,MAAM,KAAK,CAAC,CAAC,IAAI;QAAE,IAAI,GAAG,SAAS,CAAC,CAAC,MAAM,EAAE,CAAC;IAChE,IAAI,CAAC,CAAC,IAAI,KAAK,MAAM;QAAE,OAAO,KAAK,CAAC,CAAC,IAAI,sBAAsB,IAAI,KAAK,IAAI,GAAG,CAAC;IAChF,IAAI,CAAC,CAAC,IAAI,KAAK,QAAQ;QAAE,OAAO,KAAK,CAAC,CAAC,IAAI,oBAAoB,IAAI,KAAK,IAAI,GAAG,CAAC;IAChF,OAAO,KAAK,CAAC,CAAC,IAAI,gBAAgB,IAAI,KAAK,IAAI,GAAG,CAAC;AACrD,CAAC"}

package/dist/prompt/index.js

@@ -1,60 +0,0 @@
-import { getSystemPrompt } from "./system-prompt.js";
-import { testomatioTms, testomatioConnection, testomatioNotConnected, projectSettings, } from "./testomatio.js";
-import { briefAnswer, nonInteractive, reportOutput } from "./print.js";
-import { contextPromptSection, } from "./context.js";
-export function buildSystemPrompt(options) {
-    const mode = options?.mode ?? "tui";
-    const interactive = mode !== "print";
-    // The base prompt (role, workspace, tools, goals) comes first — it defines who
-    // the agent is. Specialized rules layer on after it, dynamic context goes last.
-    // Anything only one harness can do — its own question tool, a browser it can
-    // open for a watching user — reaches the prompt through `sections`,
-    // `toolBullets` and `rules`, never from here.
-    const parts = [
-        getSystemPrompt(options?.cwd, {
-            interactive,
-            toolBullets: options?.toolBullets,
-            rules: options?.rules,
-            connectedClis: options?.connectedClis,
-            connectedMcps: options?.connectedMcps,
-        }),
-    ];
-    const tokenSlugs = options?.tokens ? Object.keys(options.tokens) : [];
-    // The TUI passes no `connection`, so it keeps the managed-tokens gating.
-    const tokenAvailable = options?.connection?.tokenAvailable ?? tokenSlugs.length > 0;
-    const tms = options?.tms ?? "mcp-direct";
-    // Gated on the same condition as the connection section below: both describe
-    // the same connection, and the rules are just as needed when the token comes
-    // from the environment rather than from a per-project session token.
-    if (tokenAvailable || tokenSlugs.length > 0)
-        parts.push(testomatioTms(tms));
-    if (mode === "print")
-        parts.push(nonInteractive);
-    // Whatever this harness alone can offer — its chat-app UI tools, a browser it
-    // can open for a watching user. A one-shot CLI contributes none of it, and
-    // `nonInteractive` tells the agent to report such a gap rather than try.
-    for (const section of options?.sections ?? [])
-        parts.push(section);
-    if (options?.promptContext) {
-        parts.push(`## Project Test Context\n\n${options.promptContext}`);
-    }
-    if (options?.contextEntries?.length || options?.contextFolders?.length) {
-        parts.push(contextPromptSection(options.contextEntries ?? [], options.contextFolders ?? []));
-    }
-    if (options?.projectInfo) {
-        parts.push(projectSettings(options.projectInfo));
-    }
-    if (tokenAvailable || tokenSlugs.length > 0) {
-        parts.push(testomatioConnection(tokenSlugs, options?.backendUrl, options?.connection, tms));
-    }
-    else if (mode === "web" || mode === "print") {
-        parts.push(testomatioNotConnected());
-    }
-    // The answer contract goes last so it is the final instruction the model reads.
-    if (options?.brief)
-        parts.push(briefAnswer);
-    if (options?.outputFile)
-        parts.push(reportOutput(options.outputFile));
-    return parts.join("\n\n");
-}
-//# sourceMappingURL=index.js.map